# The Application of Anchoring Vignettes in the Analysis of Self-assessment of ICT Skills

Hana Voňková, Miroslava Černochová, Hasan Selcuk, Jan Hrabák, Kateřina Králová

## ► To cite this version:

Hana Voňková, Miroslava Černochová, Hasan Selcuk, Jan Hrabák, Kateřina Králová. The Application of Anchoring Vignettes in the Analysis of Self-assessment of ICT Skills. Open Conference on Computers in Education (OCCE), Jun 2018, Linz, Austria. pp.243-252, 10.1007/978-3-030-23513-0_24 hal-02370939

HAL Id: hal-02370939

https://inria.hal.science/hal-02370939

Submitted on 19 Nov 2019

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Application of Anchoring Vignettes in the Analysis of Self-assessment of ICT Skills

## A Pilot Study among Czech Secondary School Students

Hana Voňková, Miroslava Černochová, Hasan Selcuk, Jan Hrabák, Kateřina Králová

Charles University, Faculty of Education, Czech Republic, {hana.vonkova, miroslava.cernochova, hasan.selcuk}@pedf.cuni.cz, kabarh.naj@seznam.cz, katerina.kralova@pedf.cuni.cz

**Abstract.** This paper presents pilot study findings of a research project about the application of anchoring vignettes in the analysis of Czech upper secondary school students' self-assessment of ICT skills. The pilot study was conducted in December 2017 with 166 respondents from four different types of upper secondary schools. Anchoring vignettes, which are brief texts describing hypothetical people who illustrate a certain level of the trait of interest (e.g. information and communication technology (ICT) skills), is a method implemented to identify response scale differences in survey questions and to adjust self-assessments caused by response scale differences. Methodologically, as there are only a few examples of the application of this method in the self-assessment of ICT skills and also in educational research, this pilot study has enabled the researchers to test how comprehensible a questionnaire with a set of vignettes was for the upper secondary school students. This enhanced research method based on anchoring vignettes will be used for the main study in spring 2018. The pilot study findings confirmed the high variability of the use of scale for respondents' self-assessments and vignettes.

**Keywords.** Information and communication technology, self-assessment, anchoring vignette, upper secondary schools

## 1 Introduction

This paper presents pilot study findings of a research project about the application of anchoring vignettes in the analysis of Czech secondary school students' self-assessment of ICT skills. As explained by King et al. [1], anchoring vignettes is a method to identify response scale differences in survey questions and to adjust the self-assessments caused by response scale differences. For example, when students are asked to self-assess their ICT skills in a survey question on a scale of 1 to 7 (1 being the lowest and 7 the highest), students might either overvalue or undervalue their ability in the self-assessment. Hence, anchoring vignettes, which are brief texts describing hypothetical people who illustrate a certain level of the trait of interest (e.g. ICT skills), enable researchers to identify response scale differences in self-assessments and thus, to adjust them.

Since the early 2000s, the anchoring vignette method has been implemented in a variety of areas of research, such as political efficacy [1], work disability [2], job satisfaction [3], health ([4 - 6]), health system performance [7], life satisfaction [8], and
satisfaction with social contacts [9]. However, there are only a few examples of the
application of this method for educational research. Regarding one of these, Buckley
and Scheider [10] implemented anchoring vignettes when investigating charter schools
in the United States of America (USA) and parents' satisfaction with different types of
schools. Vonkova and Hrabak's study [11] focused on Czech upper secondary school
students' self-assessment of ICT knowledge and skills through the anchoring vignette
method. Moreover, Vonkova et al. [12] investigated Czech lower-level secondary
school students' self-assessment of dishonest behaviour in school by using the
anchoring vignette method. Von Davier et al. [13] examined the effects of vignette
scoring on reliability and validity of student self-assessment, according to the
Programme for International Student Assessment (PISA) 2012 dataset. He et al. [14]
conducted a study on effects of this method on comparability and the predictive validity
of student self-assessment in 64 countries based on data from PISA 2012. Vonkova et
al. [15, p.3] looked into cross-country heterogeneity in students' self-assessment of
their teacher's classroom management also based on the PISA 2012 dataset and they
found that the anchoring vignette method was potentially a useful tool to enhance the
comparability of the self-reported measures in education. To contribute to research in
the field of education, we have conducted a study aimed at designing an enhanced data
collection method based on anchoring vignettes to explore different realms in
educational research, such as digital literacy.

## 2 Literature review: The application of anchoring vignettes in the self-assessment of ICT skills

This section presents a review of existing studies that are pertinent to the application of anchoring vignettes in the self-assessment of ICT skills in the context of education. Before moving to reviewing the existing studies, it is worthwhile mentioning the European Computer Driving Licence (ECDL) foundation study [16], which highlighted how people's self-assessment of their ICT skills can be different from their actual ability. The ECDL foundation's study was conducted in five European countries, namely Austria, Denmark, Finland, Germany and Switzerland, to identify people's ICT skills. Respondents (aged 15-64 years) were given a questionnaire in two parts, with the first focusing on self-assessment of ICT skills, and the second testing their real level. The findings of the study showed that in all the countries surveyed, respondents overestimated their ICT skills. Taking into account the study of the ECDL foundation [16], we suggest that anchoring vignettes could provide dependable findings from respondents' answers with respect to self-assessment in surveys.

Regarding studies that have applied anchoring vignettes in self-assessments,
Vonkova and Hrabak [11] compared the ICT knowledge and skills of two distinct
groups of upper secondary school students by examining their self-assessed perspective
on these, both before and after the anchoring vignette adjustment for the different usage
of scale. The study findings revealed that the anchoring vignette method enabled the
researchers to distinguish between the two groups of students' differences in terms of
scale usage and showed how adjusted self-assessments corresponded to the assumed
level of students' ICT knowledge and skills. Cerna [17] investigated the self-
assessment of undergraduate university students' ICT knowledge by applying the
anchoring vignette method. Respondents were students from the Faculty of Education
at Charles University and specialised in three different study programmes (information
technology (IT), social science, and mathematics). The findings of Cerna's study
showed significant differences between respondents' self-assessment and their
assumed actual ICT knowledge. Moreover, the author found that those who study IT as
their specialisation in education have a greater tendency to exaggerate the level of their
ICT knowledge compared to other respondents studying different subject areas.

3 Pilot study methodology

**3.1 Aims of the pilot study**

Generally, a pilot study refers to “a small-scale version or trial run, done in preparation for the major study” [18, p. 467]. As described by Vogt [19], it can be considered as a ‘dress rehearsal’ to identify any possible problems before undertaking the major study. The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, which consists of a questionnaire with a set of vignettes focused on self-assessment of the ICT knowledge and skills of young people.

In the pilot study, we aimed to test how comprehensible our questionnaire was for
upper secondary school students, who are at Year 1 (age 15 years) and 4 (age 19 years).
The pilot was conducted in December 2017 among (N = 166) students from four upper
secondary schools in the Czech Republic. Experiences gained from the pilot study were
then used in the main study, conducted in spring 2018 with a representative sample of
2,600 students from 56 secondary schools in the Czech Republic.

**3.2 Procedures of developing the data collection method**

“The aim of the anchoring vignette method is to clear/correct the self-assessment of respondents so that they can be comparable” [20, p. 14]. Respondents in the area (in our case ICT skills and knowledge) use a self-assessment question, as well as evaluating hypothetical people described in a short story (anchor). The presented verbal characteristics of hypothetical people in vignettes in the case of categorical assessment scales can be evaluated by different respondents in different ways. Respondents’ answers to self-assessment can be affected by the different use of scale categories. For example, respondents with a lot of ICT experience and a deep interest in ICT can use different scale categories for evaluating a given level of ICT knowledge than beginners and ordinary ICT users. Our main data collection method was a questionnaire, which consisted of a set of anchoring vignettes and fixed-choice questions to obtain background information about each respondent (age, sex, school, grade in ICT subjects, field of study, interest about ICT, participation in programming or informatics competitions, number of hours spent on a computer at school or at home, respondents’
use of ICT within different types of activities, parents' education and parents' use of ICT in their job or free time). Briefly, in our pilot study, we used one self-assessment question (S) and three vignettes (V1, V2, V3), with a scale of 1 to 7 (1 – the lowest level and 7 – the highest level).

In the vignettes designed for our pilot study (see Table 1), we focused on five domains of computer literacy (information and data literacy, communication and collaboration, digital content creation, safety, and problem-solving) in accordance with the concept of digital literacy, as defined in DigComp (see [21, pp. 8-9]).

<table><caption>**Table 1.** Overview of the self-assessment question and three vignettes</caption><tr><td>General self-assessment question (S)</td><td>How do you evaluate your knowledge and skills in ICT?<br/>Note: the following was applied to each of the vignettes below:<br/>Use a scale of 1 to 7 ('1' being the lowest and '7' the highest).</td></tr><tr><td>Vignette 1 (V1)</td><td>Filip can work with texts and charts with the use of basic functions available from the ribbon. He saves his files on a desktop, he doesn't use sharing or cloud saving, but instead, sends the files via email. He uses the same password for the social network, email, etc. accounts. If he encounters any problem while working on a computer, he usually asks his friend for help.</td></tr><tr><td>Vignette 2 (V2)</td><td>Kristin can process texts and charts with the use of advanced functions (e.g. created personal styles, automatic table of content). She goes in for creating graphics (designing business cards) and short original footage, which she shares on YouTube. She doesn't post any sensitive information on social networks. If she encounters any problem while solving a task, she searches for an instruction on the internet and determines the solution procedure with its help.</td></tr><tr><td>Vignette 3 (V3)</td><td>Adam can process texts and charts with the use of advanced functions. He is able to program his own functions for more difficult tasks. He goes in for computer graphics (designing posters), creating footage and programming websites. He manages two Facebook groups and verifies the credibility of the shared posts. He uses multi-layered security (SMS verification) for his accounts on the internet.</td></tr></table>

In formulating the vignettes, we presented stories that would be comprehensible to students in the social sciences areas as well as the students of technical fields, including IT specialisation, in addition to being accessible to the two chosen age groups (Year 1 aged 15-16, and Year 4 aged 18-19 years). The vignettes should be understandable to all respondents also from a curricular point of view (note that in the Czech Republic, the current curriculum for ICT subjects does not cover all domains of DigComp).

The level of ICT skills and knowledge presented in vignettes is as follows: V1 describes a basic level (Filip's knowledge and skills in ICT correspond to the knowledge of a basic school graduate in the Czech Republic). V2 describes a more advanced level (Kristin uses ICT for creative activities and some problem-solving; she behaves safely on social networks); the majority of respondents from all secondary schools should achieve this level. V3 describes highly advanced ICT skills and knowledge beyond curriculum requirements (programming additional functions for more difficult tasks, programming a website, using multi-layered security, etc., which
are not included in the curriculum for general education). This vignette should mainly distinguish students with ICT specialisation. By providing a score on the self-assessment question and for all three vignettes in Table 1, students provided a personal rating and calibration points for inter-participant comparison.

### 3.3 Procedures of data collection

We gathered data in December 2017 through an on-line questionnaire from 166 students of Year 1 and 4, attending four different public upper secondary schools (see Table 2): School 1 focuses on general education (a gymnasium) and is located in Prague. School 2 focuses on humanities and is located in Beroun, whilst School 3 is a technical school specialising in IT and located in Prague, and lastly, School 4 is a technical school specialising in transport and mechanisation in Mladá Boleslav. These schools were not selected at random, so respondents do not represent a representative sample of the secondary school population. However, these schools represent different specialisations among Czech upper secondary schools. Respondents filled out the on-line questionnaire at their schools with the participation of the researcher.

<table><caption>**Table 2.** Characteristics of the pilot schools</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Number of respondents</th><th>Male</th><th>Female</th><th>Male</th><th>Female</th><th colspan="2">Respondents</th></tr><tr><th>(%)</th><th>(%)</th><th>Age</th><th>Age</th><th>Year 1 (%)</th><th>Year 4 (%)</th></tr></thead><tbody><tr><td>School 1</td><td>47</td><td>17.0</td><td>83.0</td><td>17.3</td><td>16.5</td><td>59.6</td><td>40.4</td></tr><tr><td>School 2</td><td>56</td><td>12.5</td><td>87.5</td><td>17.0</td><td>17.6</td><td>53.6</td><td>46.4</td></tr><tr><td>School 3</td><td>38</td><td>84.2</td><td>15.8</td><td>17.6</td><td>16.6</td><td>71.1</td><td>28.9</td></tr><tr><td>School 4</td><td>25</td><td>84.0</td><td>16.0</td><td>17.5</td><td>16.7</td><td>60.0</td><td>40.0</td></tr><tr><td>Total</td><td>166</td><td>41.0</td><td>59.0</td><td>17.4</td><td>16.9</td><td>60.2</td><td>39.8</td></tr></tbody></table>

### 3.4 Procedures of data analysis

For data analysis, we used Microsoft Office Excel 2011 and the statistical software called GRETL [http://gretl.sourceforge.net]. Table 3 shows how all respondents (N=166) assessed the self-assessment question and three vignettes from across scale 1 to 7 (1' was the lowest and '7' was the highest). The variability of the use of scale for both self-assessments and vignettes is high; respondents use (almost) the whole range of the scale categories. Concerning vignettes, it shows a high heterogeneity in reporting behaviour of respondents – the same level of ICT skills described in the vignettes is evaluated differently by different respondents. We expected the respondents would assess vignettes in the natural order V1<V2<V3 (the order given by a researcher). It means the expected value for V1 would be lower than the value for V2 and the value for V2 would be lower than that for V3. However, 5.4% of respondents assessed the vignettes in another way; they typically tied their assessment to two subsequent vignettes (see Table 3).
<table><caption>**Table 3.** Variability of the use of scale for self-assessment</caption><thead><tr><th>Scale</th><th>1<br>(%)</th><th>2<br>(%)</th><th>3<br>(%)</th><th>4<br>(%)</th><th>5<br>(%)</th><th>6<br>(%)</th><th>7<br>(%)</th></tr></thead><tbody><tr><td>Self-assessment (S)</td><td>4,22</td><td>18,07</td><td>27,71</td><td>25,90</td><td>19,28</td><td>4,82</td><td>0,00</td></tr><tr><td>Vignette 1 (V1)</td><td>7,23</td><td>26,51</td><td>35,54</td><td>26,51</td><td>3,61</td><td>0,60</td><td>0,00</td></tr><tr><td>Vignette 2 (V2)</td><td>0,00</td><td>1,20</td><td>3,61</td><td>15,66</td><td>34,34</td><td>35,54</td><td>9,64</td></tr><tr><td>Vignette 3 (V3)</td><td>0,60</td><td>0,60</td><td>1,20</td><td>5,42</td><td>10,24</td><td>31,93</td><td>50,00</td></tr></tbody></table>

## 4 Findings from the pilot study

For further data analysis, we analysed each school separately and tried to distinguish the differences in scale usage among different schools. Using the correction based on the non-parametric approach, which consists of how the self-assessment S of a respondent relates to his/her vignette evaluations V1, V2 and V3 "assuming that the vignettes are naturally ordered (for example, from the lowest skills level of a hypothetical vignette person to the highest)" (see [11, p. 192]), we corrected the values for each respondent for each school (see Table 4).

The findings in Table 4 show that the ranking of two schools has changed after the correction, with the most considerable change being in the ranking for School 4 – its position has decreased more, indicating its low standards for evaluating ICT skills. Table 5, and Figures 1 and 2 show comparisons of self-assessments before and after correction for School 4 and the best performing School 3. We showed there were statistically significant differences between Schools 3 and 4, not only before the self-assessment correction, but also after the correction.

<table><caption>**Table 4.** Average self-assessments of respondents in all schools before and after correction</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Number<br>of respondents</th><th colspan="2">Average value of<br>self-assessment S</th><th colspan="2">Ranking of schools</th><th rowspan="2">Comment</th></tr><tr><th>Before<br>correction</th><th>After<br>correction</th><th>Before<br>correction</th><th>After<br>correction</th></tr></thead><tbody><tr><td>School 1</td><td>47</td><td>3.02</td><td>2.22</td><td>4</td><td>4</td><td>The position is<br>the same.</td></tr><tr><td>School 2</td><td>56</td><td>3.34</td><td>2.44</td><td>3</td><td>2</td><td>The position<br>improved, high<br>standards.</td></tr><tr><td>School 3</td><td>38</td><td>4.29</td><td>3.27</td><td>1</td><td>1</td><td>The position is<br>the same.</td></tr><tr><td>School 4</td><td>25</td><td>3.72</td><td>2.43</td><td>2</td><td>3</td><td>The position<br>decreased, low<br>standards.</td></tr></tbody></table>
<table><caption>**Table 5.** Comparison of self-assessment of respondents from School 3 and School 4 before and after correction</caption><thead><tr><th rowspan="3">Scale</th><th colspan="4">Uncorrected self-assessment<br>(before correction)</th><th colspan="4">Corrected self-assessment<br>(after correction)</th></tr><tr><th colspan="2">School 3</th><th colspan="2">School 4</th><th colspan="2">School 3</th><th colspan="2">School 4</th></tr><tr><th>Absolute value</th><th>Relative value (%)</th><th>Absolute value</th><th>Relative value (%)</th><th>Absolute value</th><th>Relative value (%)</th><th>Absolute value</th><th>Relative value (%)</th></tr></thead><tbody><tr><td>1</td><td>0</td><td>0.0</td><td>0</td><td>0.0</td><td>5</td><td>13.51</td><td>4</td><td>19.0</td></tr><tr><td>2</td><td>2</td><td>5.3</td><td>2</td><td>8.0</td><td>3</td><td>8.1</td><td>7</td><td>33.3</td></tr><tr><td>3</td><td>5</td><td>13.2</td><td>10</td><td>40.0</td><td>14</td><td>37.8</td><td>7</td><td>33.3</td></tr><tr><td>4</td><td>14</td><td>36.8</td><td>7</td><td>28.0</td><td>11</td><td>29.7</td><td>3</td><td>14.3</td></tr><tr><td>5</td><td>14</td><td>36.8</td><td>5</td><td>20.0</td><td>1</td><td>2.0</td><td>0</td><td>0.0</td></tr><tr><td>6</td><td>3</td><td>7.9</td><td>1</td><td>4.0</td><td>2</td><td>5.4</td><td>0</td><td>0.0</td></tr><tr><td>7</td><td>0</td><td>0.0</td><td>0</td><td>0.0</td><td>1</td><td>2.7</td><td>0</td><td>0.0</td></tr><tr><td>Number of respondents</td><td>38</td><td></td><td>25</td><td></td><td>37</td><td></td><td>21</td><td></td></tr><tr><td>Average</td><td>4.29</td><td></td><td>3.72</td><td></td><td>3.27</td><td></td><td>2.43</td><td></td></tr><tr><td>Standard deviation</td><td>0.97</td><td></td><td>1.00</td><td></td><td>1.36</td><td></td><td>0.95</td><td></td></tr><tr><td>T-test: p value</td><td colspan="2">0.03414</td><td colspan="2"></td><td colspan="2">0.01730</td><td colspan="2"></td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Comparison of self-assessment of respondents from School 3 and School 4 before correction</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>**Figure 2.** Comparison of self-assessment of respondents from School 3 and School 4 after correction</figcaption></figure>

To understand better how respondents evaluated their skills, we compared schools using the background variables of respondents. Analyses of questions focused on respondents' interests and ways of using ICT in their free time, which showed respondents do not differ too much. Generally speaking, ICT knowledge, skills and interests of respondents correspond to topics they have learned mainly at school. Regarding respondents' grades, 69.9% of all the respondents from Schools 1-4 have obtained an average grade between 1 to 2 in ICT school subjects (1 is the best grade and 5 is the worst grade in Czech schools); only respondents from School 4 got worse grades (the average is 2.45). This corresponds to our ranking of schools based on adjusted self-assessments. Nonetheless, 14.5% of respondents reported that they had never studied such a subject before.

As already mentioned, School 3 is a technical upper secondary school specialising in ICT located in Prague and School 4 is a secondary school without any specialisation in ICT located in an industrial city. In both schools, practically the same number of respondents (about 65%) agreed with a statement "I am doing my best to have good results in ICT because I am expected to do so". Respondents from School 3 differed in some characteristics from respondents in School 4. The ratio between university-educated mothers and fathers of respondents from School 3 was 2.2 and for School 4 was 0.7. Respondents of School 3 spent 1.8-times more time on computers at weekends compared to working days, while at School 4 only 1.3-times more. On weekends, respondents in School 3 spent an average 4.2 hours/day on computers, while respondents in School 4 only 3.4 hours/day. Seventy-four per cent of respondents in Schools 3 and 4 spent practically every day surfing on the Internet for fun. Comparing respondents in School 4 with those in School 3, more enjoyed creating digital music, were active on social networks and enjoyed playing computer games. Respondents in School 3 dedicated more time to web design activities, work with graphics software, publishing on YouTube or creating digital animations than respondents in School 4. Respondents in School 3 were more interested (53%) in the latest ICT news, new technologies, computer graphics, etc., than respondents in School 4 (24%). Respondents in School 3 liked learning new things in ICT (87%) much more than respondents in School 4 (48%). Respondents in School 3 (68%) were fond of creative activities using ICT much more than respondents in School 4 (36%). Ninety-two per cent of the respondents in School 3 agreed with the statement: "To do my best in the ICT lessons pays off because it can help me to get a job I want to do in the future", while in School 4 this was only 60% of respondents. To summarise, the background
characteristics of respondents are in line with our adjusted self-assessments using
anchoring vignettes.

**5 Conclusion**

The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, consisting of a questionnaire with a set of vignettes focused on self-assessment of ICT knowledge and skills of young people. From this pilot study, we identified some problems that shed light on the main research, conducted in spring 2018 in the Czech Republic.

The pilot study showed us that in terms of data interpretation, it was very important
that researchers could visit all schools to instruct respondents what to do and how to
fill out the on-line questionnaire. The researchers could understand better some
contexts related to students' ICT knowledge, skills, motivation, and approaches to ICT.

Some questions need to be adapted for the main research (questions about type of school, number of inhabitants living in a respondent's town/village, grades from ICT subjects, arrangement and ordering vignettes on a page, questions about the family). The pilot also highlighted questions about how to organise data collection through an on-line questionnaire in school computer laboratories, how to support teacher co-operation to motivate students to answer the questionnaire responsibly and maintain classroom discipline, etc. It is necessary in the classes to ensure peace and discipline in order for the respondents to read attentively all questions, especially vignettes.

Limitations of this study are as follows. Findings obtained in the pilot cannot be
generalised; sampling does not allow this. However, findings are definitely of interest
since they indicate huge differences in scale usage between different types of schools
- students from some schools have high/low standards when evaluating their ICT skills.
However, all three vignettes V1, V2 and V3 were presented to respondents on one page
of the questionnaire - this might have affected respondents' assessment of vignettes.

**Acknowledgments.** This study is a result of the research funded by the Czech Science Foundation, project GA ČR 17-02993S "Factors influencing the ICT skill self-assessments of upper-secondary school students".

**References**

1. King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A.: Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567–583 (2004)

2. Kapteyn, A., Smith, J. P., & van Soest, A.: Vignettes and self-reports of work disability in the US and the Netherlands. American Economic Review, 97(1), 461–473 (2007)

3. Kristensen, N., & Johansson, E.: New evidence on cross-country differences in job satisfaction using anchoring vignettes. Labour Economics, 15, 96–117 (2008)

4. Bago d'Uva, T., van Doorslaer, E., Lindeboom, M., & O'Donnell, O.: Does reporting heterogeneity bias the measurement of health disparities? Health Economics, 17(3), 351–375 (2008)
5. Peracchi, F., & Rossetti C.: Heterogenity in health responses and anchoring vignettes. Empirical Economics, 42(2), 513–538 (2012)

6. Vonkova, H., & Hullegie, P.: Is the anchoring vignettes method sensitive to the domain and choice of the vignette? Journal of the Royal Statistical Society: Series A, 174(3), 597–620 (2011)

7. Sirven, N., Santos-Eggimann, B., & Spagnoli, J.: Comparability of health care responsiveness in Europe. Social Indicators Research, 105(2), 255–271 (2012)

8. Angelini, V., Cavapozzi, D., Corazzini, L., & Paccagnella, O.: Age, health and life satisfaction among older Europeans. Social Indicators Research, 105(2), 293–308 (2012)

9. Bonsang, E., & van Soest, A.: Satisfaction with social contacts of older Europeans. Social Indicators Research, 105 (2), 273–292 (2012)

10. Buckley, J., & Schneider, M.: Charter schools: Hope or hype? Princeton, NJ: Princeton University Press (2007)

11. Vonkova, H., & Hrabak, J.: The (in) comparability of ICT knowledge and skill self-assessments among upper secondary school students: The use of the anchoring vignette method. Computers & Education, 85, 191–202 (2015)

12. Vonkova, H., Bendl, S. & Papajoanu, O.: How Students Report Dishonest Behavior in School: Self-Assessment and Anchoring Vignettes, The Journal of Experimental Education, 85(1), 36-53 (2017)

13. von Davier, M., Shin, H. J., Khorramdel, L., Stankov, L.: The effects of vignette scoring on reliability and validity. Applied Psychological Measurement, 1-16 (2017)

14. He, J., Buchholz, J. & Klieme, E.: Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures. Journal of Cross-Cultural Psychology, 48(3), 319-334 (2017)

15. Vonkova, H.; Zamarro, G. & Hitt, G.: Cross-Country Heterogeneity in Students' Reporting Behavior: The Use of the Anchoring Vignette Method. Journal of Educational Measurement, 55(1), 3-31 (2018)

16. ECDL: Perception and Reality. Measuring Digital Skills in Europe. Retrieved from http://ecdl.org/media/perception_and_reality_-_annex_1.pdf (2016)

17. Černá, P.: Self-assessment of teacher students' knowledge in the field of information and communication technologies [Master's thesis]. Charles University, Prague (2017)

18. Polit, D.F., Beck, C.T.: Essentials of nursing research: Methods, Appraisal and Utilization (6th edition), Philadelphia, PA: Lippincott Williams & Wilkins (2006)

19. Vogt, W. P.: Dictionary of statistics and methodology: A nontechnical guide for the social sciences. Newbury Park: Sage (1993)

20. Vonkova, H.: Metoda ukotvujících vinět a její využití v pedagogickém výzkumu. Pedagogická fakulta, Univerzita Karlova: Praha (2017)

21. Vuorikari, R., Punki, Y., Carreto Gomes, S. & Van den Brande, G.: DigComp 2.0: The Digital Competence Framework for Citizens. Update Phase 1: The Conceptual Reference Model. Luxembourg: Publication Office of the European Union (2016)