# Hanging Pictures or Searching the Web

N. Yiannoutsou, N. Otero, Wolfgang Müller, C. Neofytou, M. Miltiadous, T. Hadzilacos

► To cite this version:

N. Yiannoutsou, N. Otero, Wolfgang Müller, C. Neofytou, M. Miltiadous, et al. "Hanging Pictures or Searching the Web. Open Conference on Computers in Education (OCCE), Jun 2018, Linz, Austria. pp.112-121, 10.1007/978-3-030-23513-0_11." [hal-02370918]

HAL Id: hal-02370918

https://inria.hal.science/hal-02370918

Submitted on 19 Nov 2019

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire **HAL**, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Hanging Pictures or Searching the Web

## Informing the Design of a Decision-making System that empowers Teachers to appropriate Educational Resources to their School's Infrastructure

N. Yiannoutsou

Open University, Cyprus, University College London – Knowledge Lab, UK,
n.yiannoutsou@ucl.ac.uk

N. Otero

Linnaeus University, Sweden and ISCTE-IUL, CIS-IUL, Portugal, nuno.otero@lnu.se

W. Müller

⁴University of Education Weingarten, Weingarten, Germany, mueller@md-phw.de

C. Neofytou, M. Miltiadous

Open University, Cyprus, chrystalla.neofytou@ouc.ac.cy, miltos.miltiadous@st.ouc.ac.cy

T. Hadzilacos

Open University, Cyprus, The Cyprus Institute, Cyprus, Thanasis.Hadzilacos@ouc.ac.cy

**Abstract.** In this paper, we report work in designing a decision-making system that aims to support teachers in appropriating to their practice innovative scenarios that employ uses of information and communication technologies (ICT) in teaching and learning. To this end, we break down educational scenarios into micro-activities, and connect them to required and alternative infrastructure. We argue that micro-activities is a unit of analysis of educational scenarios that is compatible with the role of teachers as designers who select, decompose, combine, enact and revise different pieces of resources. This paper offers a reflective viewpoint on integrating ICT in existing scenarios and investigates how teaching objectives make use, or not, of the potential of digital technologies.

**Keywords.** Educational scenarios, micro-activities, educational innovation

## 1 Introduction

Educational technology moves much faster than pedagogical innovation. This leads to the paradox of schools never having enough information and communication technology (ICT) while this very ICT is underutilised. Teachers willing to at least try out innovative technology-enhanced educational scenarios are often stopped by perceived lack of necessary equipment. However, the question “can I do this with my existing school infrastructure?” may be unnecessarily getting negative answers, as obvious and non-obvious substitutes exist. While most teachers know about open source alternatives to a piece of software they do not have, they may need to be told
that a shared document (e.g., Google Docs, cryptpad) can play the role of an interactive whiteboard, thus representing a not-so-obvious replacement for a piece of hardware described as being essential in a learning scenario identified by a teacher as desirable. In this paper, we present a systematic approach to answer the question “Can I do this (ICT-enhanced lesson) with my school’s infrastructure?” We start with a structural analysis (break up) of the educational scenario, leading to a sequence (or web) of ‘micro-activities’, where alternatives with other equipment may exist for each. It is up to the teacher, and depends on the learning context and goals, whether each of these alternatives is an acceptable alternative or not. Our analysis is backed by an ontology-based knowledge base system that provides the means to propose alternative implementations of scenarios on the technical level, potentially allowing for more sophisticated inference mechanisms in the future.

## 2 Theoretical background

In early discussions about integration of ICT in education, availability of resources was one of the contextual forces impeding the use of digital technologies in the classroom [1]. Today, the situation is very different, as teachers are exposed to numerous learning resources through platforms, be they open (PhET, i2geo, LeMill, Curriki, EduTags), from textbook or learning tool publishers, or more social network oriented (e.g., OpenDiscoverySpace, eTwinning, YouTube for Schools, Canvas LMS). While these platforms offer widely-available learning scenarios and, sometimes, reports of experiential use in different contexts, their current impact on schools and teachers remains low [2]. Our observation is that each of these contributions is quite isolated and the deployment within the school infrastructure is often inexplicit. Lack of infrastructure used to be, and in some cases still is, a problem for ICT integration [3]. However, today, the problem of infrastructure has been transformed to an issue regarding type of infrastructure available and teachers’ access to it, making infrastructure an issue of school and/or national policy [1, 3, 4].

The paradox of choice (i.e., more is less) that applies in the availability and use of resources has another facet, which is related to the grainsize of resources available, and the way teachers use these resources [5]. A fully-fledged scenario (lasting several hours) or a lesson plan, may be difficult to implement in another classroom for reasons related not only to curriculum and context (i.e., classroom, school, country), but also to a teacher’s personal epistemologies and pedagogies (such as factors influencing the use of resources by the teachers [5]). Furthermore, appropriation of this type of resources is often time-consuming and requires a lot of effort to overcome cultural, contextual and methodological barriers. This is not to say that a scenario or a lesson plan is not useful as a resource; instead, for the teachers to be able to use it, we argue that it is important to address teachers as designers, and not just as users.

These observations are backed by the work of Gueudet and Trouche [6], which highlights that the use of resources by the teachers does not simply involve implementation of what they (the teachers) find available. Instead, it is a complex and demanding process involving a continuous dialogue between design and enactment. More specifically, teachers, using existing knowledge and influenced by the institution
and the community they belong to, select resources, combine different pieces of
resources together, test them in their class and revise the initial use (ibid). To capture
this complexity, Gueudet and Trouche (ibid), describe the use of resources in practice
as **documentational genesis**. Documentational genesis consists of two elements: a) the
resource; and b) the development of a utilisation scheme. The latter involves a process
of appropriation and transformation of the resource in order to solve a specific problem
or to achieve a type of task (ibid). Documentational genesis is mediated by two
intertwined processes. Instrumentalisation: where teachers appropriate and shape the
resources (i.e., in our case educational scenarios/lesson plans) using their existing
knowledge. Instrumentation: where a teacher's interaction with the resources (e.g.,
inspection, appropriation) enriches and shapes a teacher's knowledge and practice.

Our approach for the structural analysis of scenarios and the design of the
recommendation system, is informed by the theoretical analysis of teachers’ use of
resources, in the following ways:

* We break up the scenario into micro-activities (which can lead back to the initial scenario) in order to facilitate the process of appropriation, selection and combination of different pieces of resources;

* We provide connections of micro-activities to different types of infrastructures in order to facilitate the instrumentalisation process (i.e., adaptation of resources by the teacher);

* We design recommendations for adaptation of micro-activities based on technology functionalities and different contexts of use, aiming to support the instrumentation process (enrichment of teacher knowledge). The purpose of the latter is to attend to the creative dimension of teaching and address teachers (also) as designers.

Fischer et al. [7] highlight that creativity can emerge in contexts where people
experience breakdowns (i.e., when they experience something they cannot do).
Considering that our overarching question “Can I do this?” is also a fertile ground for
creativity [8], we do not provide ready-made solutions, but instruments to trigger a
teacher’s creativity, i.e., choosing one or more micro-activities from a scenario,
showing how the same micro-activity can be transformed in different contexts and
supporting the investigation of alternatives.

## 3 Motivation

The motivation for our work stems from two observations. First, the use of ICT in the classroom being tool-centred – as opposed to affordances-centred – very often results in short-sighted and trivial uses of digital technologies, which could be replaced by ‘low tech’ alternatives if seen from the point of view of the instructional goal they support. Second, looking into tool affordances - instead of specific pieces of software - can help facilitate the implementation of ICT scenarios with the available infrastructure, and also support teachers in harnessing the potential of ICT in the scenarios they apply in their classroom.

Based on these two observations, we argue that in order to answer the question “Can I do this” we need to adopt a critical stance both when we look at the uses of ICT in
educational scenarios and when we look at the instructional goals underlying each activity. Next, we use the example of a simple activity, that of hanging pictures on the classroom wall, to demonstrate how focusing on the affordances of a tool and being critical of the instructional goals supported by the specific tool, can lead to a number of feasible (in terms of infrastructure available) and suitable (in terms of tool affordances) uses of technology.

## 3.1 Hanging pictures on the classroom wall – an outsider's view

We adapt an outsider's view to discuss the activity of hanging pictures on the classroom wall. Being an outsider that observes a classroom activity through the window of the class frees us from accepting contextual assumptions about the instructional goals, and directs us to explore the context by asking a very important question: why are they doing this? The exploration of possible answers to this question allows us to create a "locus of potentiality" populated with various instructional goals behind a single micro-activity, each of which is re-examined in relation to the infrastructure it requires to be achieved. In our example, the micro-activity is the following: "The teacher asks the student to put up the picture on the class wall". The necessary infrastructure for this activity is: a) a framed picture; b) a hammer; and c) a nail.

Now let us investigate "why are they doing this", i.e., what are the potential instructional goals behind this activity. In a kindergarten class, each pupil is asked to put a picture on the wall for the whole class to see everybody's work. In this context, the required infrastructure to perform this activity is shaped as follows: the picture does not need to be framed; pupils can use blue tack to put an unframed picture on the wall; hammer and nail are not necessary.

In a vocational education setting, the goal might be to show what type of hanging is suitable for each type of wall surface. In this case, the absolute specific infrastructure is needed (i.e., framed picture, nail and hammer), as they are essential for achieving the goal of the specific learning activity. A screw and a screwdriver might provide a useful alternative in terms of infrastructure, depending on the type of wall, or it could be used as a counter-example of what should not be done.

In a high-school classroom, the picture might be needed on the wall in order to analyse its content in a whole class discussion. In this case the goal is to make the picture visible to the whole class for the duration of the specific lesson. To achieve this goal, we might use a stone and a nail, instead of a hammer, a screw and a screwdriver if they are available, blue tack, or a computer and a projector. In this case the nail, hammer and framed picture are not essential.

In an examination context, at high school, the picture needs to be put up on the wall in order for the students to analyse it individually, responding to one or more test questions. In this case, the goal again is to make the picture visible to the whole class for the duration of the examination. All the solutions to replace hammer and nail mentioned in the previous paragraph are applicable here. Furthermore, considering the context of the examination, we might prefer to provide students with a printed picture allowing them to observe it closely and to comment on it in order to structure their response to the test. Alternatively, and if students have their computers or mobile
telephones with them, they could access a common digital picture or slightly different pictures slightly changing the initial scenario. Again, nail, hammer and framed picture are not essential.

## 4 Method of work: Reverse engineering of educational scenarios

We mentioned earlier in this paper that our aim is to design a recommendation system supporting teachers to adapt existing scenarios to their classroom infrastructure. To this end we built a knowledge base consisting of 200 educational scenarios (accessible at: www.esit4sip.eu) drawn from the web and provided by the schools we are collaborating with. The next step was to select certain scenarios and ICT tools to focus on, in order to be able at a later stage to create a more general model to be applied in all the scenarios of the knowledge base. At this first stage, we chose scenarios of sufficient complexity based on teacher suggestions and on diversity of educational contexts. We analysed the selected scenarios using qualitative research methods and tools as to how and why ICT is used in each one. For this, we broke down the scenarios extracting ICT micro-activities, which are smaller units of learner and teacher ICT activity. The way ICT is used in a micro-activity corresponds to ICT tool affordances. The educational effect to which ICT is used in a micro-activity corresponds to educational functionality. For each micro-activity, we considered technological alternatives with educationally equivalent functionality.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Method of scenario analysis - example of micro-activity with whiteboard</figcaption></figure>

As long as “Can I do this?” refers to whole educational scenarios, answering it remains very complex. By breaking up a scenario into micro-activities, we reduce the question to finding equivalent and alternative micro-activities using different ICT infrastructure. Educational equivalence depends on the exact context and learning goals. A stone can replace the hammer for driving the nail in the wall, except if the purpose of hanging the picture was the very use of the hammer. Another word processor can replace Microsoft (MS) Word for writing a text, except if the purpose of writing the text was learning the specific characteristics of MS Word 2016.

We would need to know the educational rationale of the micro-activity in order to find proper equivalents using alternative infrastructure. But the educational rationale
may be hidden and certainly not explicitly stated in an educational scenario. What our
system can do is discern patterns of use (instructional approaches) and propose
alternatives for the teacher to decide if they are sufficiently suitable substitutes - some
may be unacceptable, others may modify the learning results, others may be equivalent,
still others may offer something quite different but quite acceptable. To better
demonstrate our methodology, we present in the following the procedure we follow to
generate recommendations:

1. **From the scenario abstract micro-activities** ("hang a picture") and related 'infrastructure' (hammer, nail, picture)

2. **Consider diverse possible instructional rationales** of each micro-activity (diverse educational contexts): “why would you want students to hang pictures on the wall?”

3. **Analyse functionality/affordances of infrastructure** - Hammer: can drive nails into walls; Nail: can hold framed pictures on walls; Picture on wall: can be seen by whole class

4. **Consider other infrastructure with** similar functionality (stone ~ hammer, screw ~ nail) or similar result (glue ~ hang, project digital ~ hang physical, directly observe single physical object ~ through ICT observe digital copies)

Next, we show how we use these steps to analyse an ICT-based scenario in order to
provide recommendations for equivalent and alternative activities for one of its micro-
activities (collect material).

5 “Searching the web”: Analysis - recommendations for equivalent and alternative activities

The work we report here takes place in the context of the Erasmus+ project “eSIT4SIP” (empowering the School IT infrastructures for the implementation of Sustainable Instructional Patterns, www.esit4sip.eu). In section 4, we mentioned that part of the project’s outputs is an ontology-based knowledge-base consisting of 200 scenarios. From this knowledge-base, we extracted the scenario: “How to revive the story” (Authors: Nada Stojičević, Nikola Ćurčin). The scenario is designed for 15-18-year-old students. The subject matter is not mentioned. Following the analytic scheme presented in section 4, our analysis of this scenario takes the following form:

**Micro-activities:** The micro-activities (coded as MA) extracted from the scenario are the following - MA1: Prepare a story that triggers student interest about QR codes and animated maps; MA2: Discuss the story with the students; MA3: Create groups of students (different roles: photographers, researchers, coordinators, animators, web designers); MA4: Taking pictures of selected sites (topics in the original); MA5: Collect material [interpretation: information for the locations] that will be integrated in the animated map; MA6: Create animated maps. [Subject: students]; MA7: Upload the finished materials to the site (Wordpress); MA8: create and print quick response (QR) codes [Subject: students]; MA9: Students present their work [Subject: students].

**Infrastructure:** The infrastructure mentioned in the scenario involves: at least 15 computers with internet access; mobile telephones with cameras and QR code scanner; digital photo cameras; software: Animaps, Wordpress, Panorama, QR code generator.
**Functionalities/affordances of infrastructure:** Web-search: Find information around a subject using relevant keywords, offer access to various types of information regarding the topic of interest and sort the information found from the most relevant to the least relevant.

**Other infrastructure.** Similar result: take interviews, use information from an accredited source. Similar functionality: use teacher laptop and projector instead of computer laboratory.

To demonstrate our approach, we will focus only on the “Collection of information” micro-activity (MA5). We chose this activity for two reasons. The first is that collection of information is a common element of a wide variety of scenarios, especially in social sciences and humanities. The other reason is that information seeking is part of a new set of skills acknowledged as digital competences [9]. In the next two sub-sections, we show how we generate recommendations for alternative and equivalent activities based on an analysis of the affordances of technology (i.e., what is its potential) in relation to the teaching objectives it serves (i.e., what a teacher would intend to do with this technology).

## 5.1 Diverse instructional rationales (alternative activities)

The scenario mentions that students collect information to add to the interactive map without specifying the means or the type of the collected material. However, in the introductory session, the authors describe the actual output of students’ work: “*In addition to photography, the user will be reading an explanation of the museum building, centre, municipal building, park, church ....*”. Situating this in the context of the overall scenario - i.e., the creation of an interactive map - helps us to assume that the collected information will consist of short texts describing the sites of the map.

The scenario neither explains the means nor the tools students are going to use to collect information. Two assumptions are drawn from this: a) the use of web search is so widespread that it is not necessary to be mentioned; b) the type of information collected is not that important because the emphasis is on adding content (photographs, text) on the map. If we accept the first assumption, then web-search is a “legitimate” learning activity. In this case, the infrastructure required for this micro-activity is: computer laboratory, computers connected to the internet, browser. Next, we provide two examples of instructional goals related to the micro-activity involving web-search and respective settings.

*Setting 1 - Instructional goal 1:* Identify differences between search engines and between devices (owned by different users) - the filter bubble [10]. The infrastructure mentioned in the micro-activity is the same, but the instructional goal involves the development of digital competences. The micro-activity can be implemented with the same age group in the context of IT lessons.

*Setting 2 - Instructional goal 2:* Seek, evaluate, select and appropriate information from the web. Here the emphasis would be for the students to learn how to refine their keywords (seek information), evaluate the sources of the information provided and appropriate the information they select to their purposes. The micro-activity can be
implemented across subject topics, with younger audiences (age 12-15 years), using
the same infrastructure (as in the initial MA).

## 5.2 Infrastructure with similar functionality or similar result (equivalent activities)

In this section, we present how the focus on the technological affordances in terms of
results and/or functionalities can support the implementation of the micro-activity
“Collection of information” in the classroom with different types of infrastructure.

*Infrastructure with similar result 1:* Take an interview from someone who has the information or ask people who live or work next to the sites that are going to be included in the map, to describe the site of interest with a relevant representative phrase. The infrastructure needed here is a notebook or a sound recording device. From an instructional point of view, the use of this infrastructure will allow students to collect the information they need. This choice can be “instructionally acceptable” in the following situations: a) if the means students use to collect information does not really matter for the specific learning scenario (i.e. there are no implicit or explicit objectives regarding this activity); b) if the teacher would like to explore with the students the interview as a medium for collecting information. If a teacher decides to take this path, then he/she should envisage a time slot for the students to work on appropriating this information to be integrated in a digital map. A final step would be for the students to delve into the differences between the information collected and that existing in other digital maps. From an instructional perspective, this option might offer rich learning opportunities regarding the use of digital information, especially if compared to unstructured web-search where students type a keyword and copy-paste information from the first result coming up.

*Infrastructure with similar result 2:* The teacher provides each group with print-outs containing information about the sites they are going to include in their map. Their job is to appropriate this information so as to be interesting for the users of the map and to record it on a piece of paper. Infrastructure: printouts and notepad. From an instructional point of view this option can be acceptable if the teacher does not want, or does not have the time, to emphasise the aspect of including information on a digital map or wants her/his students to work on specific information, e.g., taken from an accredited text book. The appropriation of information, however, should consider the functionalities of a digital map (length and type of information shown). From an instructional point of view, this activity, though not making direct use of digital technologies, is done with reference to digital technologies (i.e., the functionalities of a digital map) and allows for focused work on the editing and appropriating information, which often is overlooked when simple web-search is involved.

*Infrastructure with similar functionality:* Students work in groups, each group being responsible for one site of interest, and take some time to think about the keywords they could use in order to search information on the web. When they are ready, each group takes turns in dictating their keywords to the teacher who types them on his/her laptop and the teams see the results through a video projector. Then the group, with the help of the teacher and the other groups, review the results and refine their search if
necessary. The teacher goes through the information found (i.e., reading it aloud) and the group responsible for the specific site takes notes to use them for the construction of information to be included in the map. Alternatively, the teacher can print out the information for the groups to adapt it for the map. The infrastructure that is necessary here is one computer connected to the internet, a browser, and a projector. From an instructional point of view, the use of the same technology in a different orchestration (from the computer laboratory to the teacher laptop) makes salient in the whole class a process which usually happens in a group or individually. This transfer results in a qualitatively different process, as it offers opportunities for refinement from different viewpoints (i.e., the viewpoints of the other groups), which is rarely pursued when web-search is just a small step for something else (e.g., to use the information found to construct a map).

The analysis of this scenario aimed to show the application of our analytic framework and the production of recommendations by domain experts. The recommendations stemmed from a critical-reflective analytic approach on the uses of technology. This approach involved a focus on learning objectives and technology affordances. In the example we analysed above, we found that there are uses of ICT which do not harness the potential of digital technologies to support learning (in the sense that the same learning objectives can be pursued effectively without technology). Furthermore, looking at the technology from the point of view of affordances allows us not only to rethink the infrastructure needed but also to come up with new educational activities which rely on what the technology can actually do.

# 6 Concluding remarks

In this paper, we described our approach in empowering teachers to integrate available resources (i.e., ICT enhanced scenarios and lessons in their practice) by making use of existing infrastructure in their schools. Our work is informed by two theoretical underpinnings. One is the role of infrastructure in impeding teachers to try innovative educational scenarios in their class. The second involves the role of the teacher as designer. In order to facilitate the appropriation process, we break down the educational scenarios into micro-activities, each explicitly connected to the infrastructure mentioned or implied in the scenario and to other equivalent solutions. Furthermore, considering that the use of resources is also a creative process enriching and shaping a teacher's knowledge, we explore diverse instructional rationales around the use of infrastructure exploring alternative micro-activities. We used a quite general example to illustrate the implementation of this analytic framework and to show how this analysis can be used to inform the design of an ontology and a knowledge base supporting a decision-making system for teachers.

Currently, the decision-making system provides alternatives for the 200 scenarios of the knowledge base which are tagged manually according to parts of the analysis presented here. The implementation of the system at this stage looks only at the tools and their affordances in a scenario, and based on these affordances, provides suggestions for alternative tools. The next step is to refine our system so as to include the micro-activities, the educational functionalities and a set of technology-enhanced
learning design patterns. The latter are recurring ICT-based solutions for recurring educational problems in diverse educational contexts, which will be used to conceptually group scenarios, thereby allowing integration of contextual knowledge in our system related to the use of the tools and not only on their affordances. The question “Is this alternative acceptable, i.e. does it serve the learning goals?” was put to teachers. It is the teacher of course who decides if the learning goals are met with the proposed replacements/substitutions. Seeing an active role for the teachers in this process, we expect to create a vibrant community with teachers and other stakeholders, sharing learning scenarios, providing advice regarding the implementation of the different scenarios, or even suggesting alterations based on their own experiences.

**Acknowledgments.** The research has been conducted in the context of the eSIT4SIP project, co-funded by the Erasmus+ programme of the European Union. The European Commission support for the production of this publication does not constitute an endorsement of the contents, which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

## References

1. Mumtaz, S.: Factors affecting teachers’ use of information and communications technology: a review of the literature. Journal of Information Technology for Teacher Education. 9, 319–342 (2000)

2. Mavroudi, A., Miltiadous, M., Libbrecht, P., Müller, W., Hadzilacos, T., Otero, N., Barth, K., Georgiou, K.: Let Me Do It: Towards the Implementation of Sustainable Instructional Patterns. In: Spector, J. M., Tsai, C., Sampson, D., Huang, K. R., Chen, N., Resta, P. (eds.) Advanced Learning Technologies (ICALT), IEEE 16th International Conference, 414–415 (2016)

3. Vrasidas, C.: The rhetoric of reform and teachers’ use of ICT: The rhetoric of reform and teachers’ use of ICT. British Journal of Educational Technology. 46, 370–380 (2015).

4. Tondeur, J., van Braak, J., Valcke, M.: Curricula and the use of ICT in education: Two worlds apart? British Journal of Educational Technology. 38, 962–976 (2007). doi:10.1111/j.1467-8535.2006.00680.x

5. Gueudet, G., Pepin, B., Trouche, L. eds: From Text to “Lived” Resources. Springer Netherlands, Dordrecht (2012)

6. Gueudet, G., Trouche, L.: Teachers’ Work with Resources: Documentational Geneses and Professional Geneses. In: Gueudet, G., Pepin, B., and Trouche, L. (eds.) From Text to “Lived” Resources. Springer Netherlands, Dordrecht, 23–41 (2011)

7. Fischer, G., Giaccardi, E., Eden, H., Sugimoto, M., Ye, Y.: Beyond binary choices: Integrating individual and social creativity. International Journal of Human-Computer Studies. 63, 482–512 (2005). doi:doi.org/10.1016/j.ijhcs.2005.04.014

8. Nelson, C.: Generating Transferrable Skills in STEM through Educational Robotics. In: Barker, B., Nugent, G., Grandgenett, N., and Adamchuk, V. (eds.) Robots in K-12 Education: A New Technology in Learning. Inf. Sc. Ref. (IGI Global), 54–65. (2012)

9. Vuorikari, R., Punie, Y., Carretero, S., Brande, L.V. den: DigComp 2.0: the digital competence framework for citizens. JRC-Science Hub - EC, Luxembourg (2016)

10. Pariser, E.: The filter bubble: How the new personalized web is changing what we read and how we think. Viking (2011)