# What Students Do While You Are Teaching – Computer and Smartphone Use in Class and Its Implication on Learning

Carli Ochs, Andreas Sonderegger

► To cite this version:

Carli Ochs, Andreas Sonderegger. What Students Do While You Are Teaching – Computer and Smartphone Use in Class and Its Implication on Learning. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.501-520, 10.1007/978-3-030-85616-8_29. hal-04196874

HAL Id: hal-04196874

https://inria.hal.science/hal-04196874v1

Submitted on 5 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# What students do while you are teaching – Computer and smartphone use in class and its implication on learning

Carli Ochs<fnref n="1" />ª & Andreas Sonderegger<fnref n="2" />, <fnref n="1" />

**Abstract.** The presence of mobile devices (e.g., smartphones, tablets and computers) in the classroom gives students the possibility of doing off-task activities during lectures. The purpose of this mixed-method field study was to learn more about students' behaviors, reasons, and opinions regarding such activities and their consequences on learning. This study is one of few to take a holistic view on this topic by taking the use of all technical devices in class into account and assessing its consequences on learning objectively. This is important to gain a full picture concerning the consequences of off-task activities in class. Right after a lecture, bachelor students (N = 125) answered a survey containing questions on their usage of mobile devices during this last class. Furthermore, they took a test on the content of that lecture. Qualitative and quantitative analysis of data revealed that students spent an average of more than 19% of their time using a digital device for non-class purposes. Interestingly, this was not significantly linked with learning, although many students reported being aware of this behavior's potential negative consequences. But there was a significant negative link between the number of received notifications and learning. These results suggest that external interruptions have a stronger negative effect than internal interruptions, allowing us to make better recommendations on how to use electronic devices in the classroom.

**Keywords:** Digital Distractions, Academic performance, Notifications.

## 1 Introduction

The days of looking at a classroom filled with students holding a pen or pencil and yellow legal notepad are long gone. While a few students still enjoy taking handwritten notes, they often seem to be in the minority, while a diversity of devices is visible in classrooms; these might be laptops, smartphones, or tablets. The question hence arises whether these changes are for better or worse in terms of the student's learning.

Research has shown that technological devices can be useful to support learning (e.g., taking notes on a computer) [1]. However, these devices can also be a source of distraction [1–4]. Students can either be interrupted and distracted by their own usage but also through the usage of others (e.g., seeing content on fellow students’ screens or receiving notifications), which may result in reduced attention in the classroom [1, 5]. Reduced attention might represent an important issue since attention seems to play an important role in learning and performance [6]. Research done on technological devices in the classroom has mainly shown a negative impact on learning when students use their devices for off-task activities [2, 7–9].

<footnote n="1">¹Department of Psychology, University of Fribourg, Fribourg, Switzerland,</footnote>

<footnote n="2">²Bern University of Applied Sciences, Bussiness School, Bern, Switzerland</footnote>

<footnote>ª Corresponding author: carli.ochs@unifr.ch</footnote>
However, most of the research on distraction in the classroom focuses on a specific kind of use (e.g., texting, using social media) or on a particular device (e.g., laptop computer or smartphone). In addition, learning was assessed often only at the end of the term, allowing for the influence of different other variables (e.g., motivation, time spent studying) on learning performance [2, 10, 11]. Also, the objective assessment used in some studies (e.g., through an app) implies that the study's goal was disclosed to the students [11]. This might lead participants to behave differently and heavily impinge on the study's ecological validity.

In addition, due to rapidly growing technological advancements, user behavior has changed significantly in recent years (e.g., an always increasing number of students using laptops), which is why the collection of up-to-date data is of importance. In the following paper, we took a holistic approach to technology usage in the classroom. We assessed the kind of usage (e.g., social media or communication), the length and frequency of use (e.g., usage time, number of unlocks), and what devices were used (e.g., laptop or smartphone) directly after class. In addition, we evaluated how the use of technology in the classroom affected learning objectively and questioned the students on their opinion regarding technology use in class.

## 1.1 Technology-induced interruptions in class

Technology in the classroom can be distracting in mainly two ways. Students can distract themselves by doing nonrelated course work activities on their technological devices. Such type of behavior is termed self-interruption [12]. Additionally, to self-interruption, individuals can be externally interrupted. External interruptions can come from other students but also from notifications [12]. Both types of distractions have been shown to have a negative impact on student's learning performance [2, 5].

**Self-interruption in class.** When students self-interrupt from following the class to do something else, this activity is called an off-task activity [13]. Examples of such activities are online chatting, playing games, social media use, and working for a different class [14, 15]. Off-task activities have been shown to be problematic because they reduce the attentional resources available for following the class [6]. In this context, it has been shown that students who engage in off-task activities on their laptop or smartphone during lectures score lower on knowledge tests [2, 7, 8, 11]. However, when asked about the potential consequences of off-task activities, many students seem to think that they do not affect their performance [16]. Others were aware of potential negative consequences, but still engaged in them [16, 17]. Reasons given by students for using technology for off-task activities in class are boredom, the need to
communicate with someone, and class-specific characteristics (e.g., number of stu-
dents in the classroom, structure, and content of the course) [15]. Supporting these ex-
planations are Tran et al.' [18] findings who reported that a trigger for smartphone us-
age during class was unoccupied moments (e.g., boredom).

In addition, it has been argued that self-interruption is often due to a fear of miss-
ing out (FOMO) [13, 19]. FOMO describes a belief that others might enjoy them-
selves without the concerned person and is characterized by the desire to remain con-
stantly in touch with others to avoid the risk of missing out [20]. It has been shown
that FOMO can cause people to check their phones on average every 30 minutes with-
out an external trigger to make sure that they are not missing any notifications [19].

**External interruptions in class.** In addition to the negative consequences of technology-induced self-interruptions in class, students can be externally interrupted. One good example of an external interruption are notifications. Notifications are generally attended to very quickly [21, 22]. They are designed to attract attention and hence likely to distract [23]. Just the sound or vibration of a notification has been shown to have a negative impact on attention by impacting the cognitive load and attentional resources of the user [24, 25]. In addition, a notification can distract not only the users of a device but also others around them.

Notifications exist on all devices and are either automated messages from applica-
tions such as calendar reminders but can also come from people through email, mes-
senger apps and video calls. Most often, notifications are not controlled by users, un-
less they explicitly disable them or enable a ‘do not disturb function’ on their techno-
logical device. While several studies addressing the impact of notifications on learn-
ing focused on the writing of text messages during class, empirical research regarding
the disturbance caused by notifications in the classroom is to our knowledge rather
scarce. Outside the educational field of research, studies have shown that constant in-
terruptions by notifications impinge on work and even social life [21, 26]. Pielot and
Rello [21] showed that users who switched off notifications for an entire day reported
higher levels of productivity and lower levels of distraction as compared to a baseline
day with notifications switched on.

The notification type with the highest affordance to react is the text message type [22]. Agrawal et al. [19] have shown that in 90% of the cases, individuals reply in the following 15 minutes after having received a text message. Text messaging has been shown to be a rather common off-task activity in class, with over 70% of students reporting texting during a lecture [15]. Addressing the link between texting and learning, Bowman et al. [27] reported that texting had a consequence on speed of reading but not on comprehension in a text comprehension test. Several other studies showed that texting during class had a negative effect on learning performance (e.g., total grade point average (GPA), or questions on a video recorded lecture that participants
watched while they received texts) [3, 5, 28, 29]. Hence texting in class can be con-
sidered to be a substantial problem for learning.

Another external source of distraction to students can be their fellow students' off-task usage of technology [8]. Seeing fellow students engaging in off-task activities has been reported to be a trigger to engage oneself in such activities [13].

An important issue of off-task activities in class is that they come with higher costs than expected. Once distracted by an off-task activity, users often report feeling sucked in or losing control of their usage [30, 31]. This implies that students risk spending more time on their device than the original interruption would have taken. Meaning that just checking, or just reading a message might have a higher price than some users expect.

1.2 Present study

In summary, students either self-interrupt their focus from the lecture by doing off-
task activities or are externally interrupted by others' usage or by notifications appear-
ing on their devices. Additional issues are that once interrupted, students tend to stay
on their device longer than intended, and once they do decide to redirect their atten-
tion towards the course, this can also demand a certain amount of cognitive load [32].

This present mixed method survey study aims to determine how much time stu-
dents spend on off-task activities during a typical 90-minute university lecture, what
kind of activities they perform, what devices they use to perform these activities, and
what the impact on learning performance is. As hypotheses, we expect off-task activi-
ties to have a negative impact on learning. This effect is expected to be particularly
pronounced if interruptions have an external source (e.g., notifications).

While previous studies addressing similar questions used different, eventually bi-
ased methodological approaches (e.g., regarding the assessment of learning and off-
task activities), focused on one device (smartphone or computer) or a specific type of
usage (texting), we chose a holistic approach to assess students’ off-task activities and
learning directly after the class. Taking a holistic approach is important to gain a full
picture of student’s activities in class as students that might partake in off-task activi-
ties on one device might not do so on the other.

2 Method

2.1 Participants

One hundred and twenty-five bachelor students (105 female, 18 male, 2 did not specify) of the University of Fribourg ranging from the age of 18 to 29 yrs (M = 21.28, SD = 1.95) were tested at the end of bachelor level lecture in different psychology
method classes (introduction to psychological methods and introduction to psychological testing) taught by different lecturers.

An a priori sample size estimation following the minimally important difference approach [33, 34] revealed that for the detection of a change in the grade of a student based on the applied grading scheme (i.e. the decrease of the grade by one point, $d = 0.86$, was defined as minimal important difference), 64 participants would be required (assuming an error probability of $\alpha = .05$) to achieve a power of $1 - \beta = .95$ for the correlational analysis, and 74 participants for the analysis of variances [35].

## 2.2 Measures

**Questions regarding computer use and smartphone use.** Participants were asked a series of questions on their computer and smartphone use during the lecture. These questions can be found in Table 1. Second order questions (e.g., 1a. How many notifications/messages did you receive on your computer, during this lecture?) were only presented if participants answered yes to the first order question (e.g., 1. Did you use your computer during this lecture?). Question 2.a, and 3.c left enough space to report up to 5 activities. At the end of the questionnaire (question 4) participants were asked to answer an open question about their opinion on the use of technology for off-task activities in the classroom.

<table><caption>**Table 1.** Questions regarding participants' computer and smartphone use during the lecture.</caption><tbody><tr><th colspan="3" style="text-align: left;">Computer use</th></tr><tr><td>1.</td><td>Did you use your computer during this lecture?</td><td>Yes*/No</td></tr><tr><td>1.a</td><td>How many notifications/messages did you receive on your computer, during this lecture?</td><td></td></tr><tr><td>2.</td><td>Have you used your computer for activities other than those related to today's lecture?</td><td>Yes*/No</td></tr><tr><td>2.a</td><td>Which off-task activities did you do on your computer?<br>Indicate the different activities and the estimated time you spent on doing them.</td><td></td></tr><tr><td></td><td>Activity:</td><td>Estimated time:</td></tr><tr><th colspan="3" style="text-align: left;">Smartphone use</th></tr><tr><td>3.</td><td>Did you use your smartphone during this lecture?</td><td>Yes*/No</td></tr><tr><td>3.a</td><td>How many times did you pick up your smartphone or clicked the Main/Home button?</td><td></td></tr><tr><td>3.b</td><td>How many notifications did you receive during this lecture?</td><td></td></tr></tbody></table>
3.c Which off-task activities did you do on your smartphone?
Indicate the different activities and the estimated time you spent on doing them.

Activity:

Estimated time:

General opinion questions

4. What is your opinion on the use of smartphones and computers for non-study related activities during classes?

**Learning score.** Six multiple choice questions on the content of the specific topic of the class were asked. Questions were selected from previous exams and thus represented a valid assessment of learning performance. These questions were of K-prime 1/0 type; with four answer options, of which one or two could be correct (cf. Table 2). Students received a point if all the answering options were responded to correctly. Based on this test, a learning score was calculated. Since data analysis revealed that difficulty level and learning performance varied between the different classes, learning scores were standardized for each class (z-scores).

<table><caption>**Table 2.** Exam example question</caption><tr><td colspan="2">1) Which statement(s) is/are correct?</td></tr><tr><td>a)</td><td>Moderation bias implies that participants tend to choose the middle of the scale to answer questions.</td></tr><tr><td>b)</td><td>Moderation bias implies that participants tend to choose the extremes of a scale.</td></tr><tr><td>c)</td><td>One way to avoid moderation bias is to standardize the scores.</td></tr><tr><td>d)</td><td>One way to avoid moderation bias is to reverse the items.</td></tr><tr><td colspan="2">2) The BOLD (Blood-Oxygenation-Level Dependent) effect is important for which psychological measurement method(s)?</td></tr><tr><td>a)</td><td>EEG</td></tr><tr><td>b)</td><td>ECG</td></tr><tr><td>c)</td><td>fMRI</td></tr><tr><td>d)</td><td>fNIRS</td></tr><tr><td colspan="2">3) Which statement(s) about advantages and disadvantages of EEG is/are false?</td></tr><tr><td>a)</td><td>EEG has a very high spatial resolution</td></tr><tr><td>b)</td><td>EEG is tolerant with regard to movement</td></tr><tr><td>c)</td><td>EEG is non-invasive and quiet</td></tr><tr><td>d)</td><td>The signal-to-noise ratio in EEG is poor</td></tr></table>
# 3 Procedure

For all classes, no specific class policy regarding the use of technology in the classroom has been imposed. Participants were recruited at the end of lectures given by staff of the psychology department. Students were asked 20 minutes before the end of a lecture if they would be willing to participate. If they agreed, they stayed seated in the lecture hall and completed the online survey that lasted about 15 minutes.

Participants were informed of the theme of the questionnaire a first time orally and a second time in written form on the first page of the survey. Great importance was attached to emphasizing that the survey is anonymous and that students should answer as honestly as possible. After the participants had confirmed their informed consent, they answered the questionnaire, took the learning test and were allowed to leave.

# 4 Data analysis and accuracy evaluation

## 4.1 Analysis of quantitative data

Since the collected data met the requirements of parametric testing (normal distribution and homogeneity of variances), quantitative data was analysed using between-groups analysis of variance (ANOVA) and correlations (Pearson).

## 4.2 Analysis of qualitative data

Since two very different types of qualitative data were collected, two different methods of analysis were used. Firstly, ten categories of activities were inductively created from the data specifying off-task activities students adhered to during class. Then two coders proceeded to code each activity into the 10 different categories (cf. Table 2 in the results section). The inter-rater reliability was measured using Cohen's kappa and showed to be satisfactory (k = .84). The few differences in coding were then solved through discussion.

Secondly the open question (4., cf. Table 1) regarding students' opinions was analyzed using the inductive thematic analysis methodology [36]. After reading all of the students' answers, a set of preliminary codes were independently produced by both coders. These codes were then compared and discussed by the two coders. The coders then proceeded to code the data with these codes while regular meetings were organized to compare and discuss coding, as well as to make sure that the analysis was comprehensive, coherent, and reflecting the actual data. Following the coding of the data, emerging themes and sub-themes were identified and discussed between the two researchers. We did not measure inter-rater reliability for this analysis, because this
would imply an unequivocally “true” way of interpreting data, which we believe is not possible for this type of dataset [36].

## 4.3 Accuracy evaluation of self-reports of technology use

Previous research has shown that participants' knowledge (e.g., telling participants that data of their usage behavior will be assessed) or guesswork about the objective of a study can influence their behavior [37]. Therefore, we decided not to log students' usage behavior to avoid influencing it. This implied that the data on usage behavior were collected subjectively after class. However, a recent meta-analysis has shown that self-reported data of media use are only moderately correlated with objective (logged) measurements [38]. Different cognitive processes such as memory biases are usually put forward as main reason for such moderate correlations [39, 40]. However, most studies analyzed in the above-mentioned meta-analysis compared self-report and objective data regarding the use of unspecific media types over a rather long period of time (e.g., days or even weeks). In contrast to this, use of technology was recorded in this study within a highly restricted timeframe (i.e., the previous 90 minutes) regarding very specific usage behaviors (i.e., off-task activities).

In order to gain a better knowledge of the reliability of such self-report data (i.e., specific, short-time usage behavior directly assessed after class), a separate study was conducted (N = 23) in which usage time and number of received notifications during a class were subjectively assessed as well as automatically logged. Analysis of these data is presented in Table 3, indicating a moderate to good reliability of self-report data [41]. This points out that the subjective assessment of usage data is reliable for such short-term evaluations as conducted in this piece of research.

<table><caption>**Table 3.** Results of interclass correlation (ICC) calculation using single-rating, absolute-agreement, 2-way mixed-effects model and Pearson correlations between self-report and log data.</caption><thead><tr><th></th><th>ICC</th><th>95% CI</th><th>F</th><th>r</th></tr></thead><tbody><tr><td>Usage time</td><td>.76</td><td>.43 to .90</td><td>F(22, 22) = 9.33, p&lt;.000</td><td>.83***</td></tr><tr><td>Number of notifications</td><td>.60</td><td>.30 to .83</td><td>F(19, 19) = 5.42, p&lt;.000</td><td>.91***</td></tr></tbody></table>

<footnote>*** significant at the 0.01 level (2-tailed), CI = confidence interval</footnote>
# 5 Results

## 5.1 Off-task activities and notifications

Data analysis revealed that more than half of the students (N = 71, 57%) used their computers and more than two thirds (N = 89, 71%) their mobile phones for off-task activities during class, while a rather small number did not use any of their devices for off-task activities (N = 16, 13%; see Table 4 for details).

Table 5 summarizes the activities reported by students (several activities could be mentioned for each device used in class). Interestingly, the most frequently mentioned activities are related to communication, e.g., writing or reading emails or text messages and the use of social networks. The use of WhatsApp might be considered in both of these categories (communication or social media)<fnref n="1" />. A closer look into the off-task activities as a function of device that was used revealed that the mobile phone was mainly used for communication and social media purposes (72.34%). In contrast, the most frequently mentioned activities for the computer were communication (23.5%, mainly emails) and the preparation for other classes (17.5%). The average time each student spent on off-task activities during a 90-minutes lecture (calculated based on the total number of participants) was 17.2 minutes (SD = 19.0) which represents 19.1% of the total time. About half of the time (8.56 minutes, SD = 11.9) was spent on the laptop computer and half (8.62 minutes, SD = 15.14) on the mobile phone.

Regarding the number of notifications students received during the class, data analysis revealed a mean value of 2.79 notifications (SD = 6.53) on the computer and 4.72 (SD = 6.16) notifications on the smartphone. For both, the measures ranged between 0 and 30 notifications.

<table><caption>**Table 4.** Use of computer and smartphone for off-task activities.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">Smartphone use</th></tr><tr><th>No (%)</th><th>Yes (%)</th><th>Total (%)</th></tr></thead><tbody><tr><td rowspan="3">Computer use</td><td>No (%)</td><td>16 (13)</td><td>38 (30)</td><td>54 (43)</td></tr><tr><td>Yes (%)</td><td>20 (16)</td><td>51 (41)</td><td>71 (57)</td></tr><tr><td>Total (%)</td><td>36 (29)</td><td>89 (71)</td><td>125 (100)</td></tr></tbody></table>

<footnote n="1">¹ There is a distinction between WhatsApp and other social media apps: WhatsApp is used primarily for communication while other social media apps offer other content. As for Facebook it is possible to make the distinction between Facebook and its messenger app, we would suggest coding the messenger app as communication and the social media app as social media when such a distinction is possible. In order to have the most fined grained analysis we chose to create a separate category for WhatsApp instead of coding it as social media or communication app.</footnote>
<table><caption>**Table 5.** Evaluation of the mentioned off-task activities ordered by different categories (average time of use was calculated for students reporting this activity).</caption><thead><tr><th rowspan="2">Activities</th><th colspan="2">On computer</th><th colspan="2">On mobile phone</th></tr><tr><th>No. of men-tions (%)</th><th>Average time of use in min. (SD)</th><th>No. of men-tions (%)</th><th>Average time of use in min. (SD)</th></tr></thead><tbody><tr><td>WhatsApp</td><td>29 (14.5)</td><td>8 (5.9)</td><td>70 (37.2)</td><td>5 (4.2)</td></tr><tr><td>Social networks (e.g., Facebook, Instagram)</td><td>18 (9)</td><td>6.5 (4.2)</td><td>50 (26.6)</td><td>6 (5.2)</td></tr><tr><td>Communication (e.g., e-mail, text messages)</td><td>47 (23.5)</td><td>5 (4)</td><td>16 (8.5)</td><td>3 (5.1)</td></tr><tr><td>Other</td><td>27 (13.5)</td><td>12 (5.9)</td><td>26 (13.8)</td><td>10 (21.2)</td></tr><tr><td>Activities for another class</td><td>35 (17.5)</td><td>23.9 (34.7)</td><td>1 (0.5)</td><td>15</td></tr><tr><td>Planning/organisation</td><td>18 (9)</td><td>4 (2.5)</td><td>9 (4.8)</td><td>2 (2.1)</td></tr><tr><td>(Online) games</td><td>6 (3)</td><td>25 (18)</td><td>8 (4.3)</td><td>11 (9.3)</td></tr><tr><td>Magazine/news</td><td>8 (4)</td><td>12 (5.9)</td><td>5 (2.7)</td><td>10.3 (19.5)</td></tr><tr><td>Online shopping</td><td>9 (4.5)</td><td>8.3 (10.8)</td><td>1 (0.5)</td><td>5</td></tr><tr><td>Videos, Netflix, YouTube</td><td>3 (1.5)</td><td>17.3 (14.2)</td><td>2 (1.1)</td><td>30 (42.4)</td></tr><tr><td>Total</td><td>200</td><td>11.9 (19.8)</td><td>188</td><td>5.2 (11.2)</td></tr></tbody></table>

## 5.2 Learning Performance

With regard to the question whether non-lecture-related activities influence learning, visual inspection of the standardised learning scores (see fig.1) indicated slightly higher scores for participants who did not use their mobile phone and computer during class. Statistical analysis however revealed only a small and not significant main effect for computer use on the learning score, $F(1, 121) = 0.77, p = .38, \eta_p^2 = .01$. Also smartphone use did not show a significant effect on students' learning score $F(1, 121) = 3.26, p = .073, \eta_p^2 = .026$. The interaction of computer and smartphone use did not reach significance level either, $F(1, 121) = 0.31, p = .58, \eta_p^2 = .00$.

Correlational analysis revealed that the standardised learning score did neither correlate with the time students spent on the smartphone nor with the time they spent on the computer for off-task activities (see Table 6). Interestingly, the only measure that was considerably correlated with learning was the number of smartphone notifications received during the lecture. The more smartphone notifications a student received, the lower was their learning score. In addition, the number of received notifications showed a significant correlation with the time students spent on their smartphone for non-lecture related activities. In a similar vein, the number of notifications received on the computer were correlated with time spent on the computer.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Learning score (scale represents range of lowest and highest scores obtained) as a function of computer and smartphone use (errors bars representing 95% CI).</figcaption></figure>

<table><caption>**Table 6.** Correlations (Pearsons's rho) between the various measures recorded in this study.</caption><thead><tr><th></th><th>Time on Computer</th><th>Time on Smartphone</th><th>Notifications on computer</th><th>Notifications on smartphone</th></tr><tr><th></th><th>n=125</th><th>n=130</th><th>n=94</th><th>n=90</th></tr></thead><tbody><tr><td>Learning Score (standardized)</td><td>-.05</td><td>-.05</td><td>-.10</td><td>-.21*</td></tr><tr><td>Time on Computer</td><td></td><td>-.03</td><td>.17*</td><td>.06</td></tr><tr><td>Time on Smartphone</td><td></td><td></td><td>-.01</td><td>.22*</td></tr><tr><td>Notifications on computer</td><td></td><td></td><td></td><td>.12</td></tr></tbody></table>

## 5.3 Students' opinions on the use of technology in class.

Qualitative analysis of students' answers to the open question regarding their opinion on the usage of technology for off-task activities in class revealed three main themes. These themes were consequences on learning and attention, judgement and reasons. All three themes and their subcomponents are described in the following sections.

<footnote>* p < .05,</footnote>
**Consequences on learning and attention.** The theme "consequences on learning and attention" summarizes the negative consequences stemming from technology use for off-task activities in class. We identified four different types of negative effects based on students' comments: less attention/distraction, hindrance to learning, out of the loop and disturbing others. These negative effects can be found in Table 7 with their definition and an example of such a mention by a participant.

<table><caption>**Table 7.** Summary of the qualitative data analysis for the theme "detrimental effects"</caption><thead><tr><td>Consequences on learning and attention</td><td>Definition</td><td>Example</td></tr></thead><tbody><tr><td>Less attention/ distraction</td><td>Distraction caused by technology / attention diverted from the course.</td><td>P.10 "In general, I see such activities as a distraction to the lecture content. You can't have your attention on the lecture and your smartphone at the same time"</td></tr><tr><td>Hindrance to learning</td><td>Loss of information related to the course due to technology.</td><td>P.29 "You should not use smartphones because you miss things."</td></tr><tr><td>Out of the loop</td><td>Loss of the thread of the course due to distraction caused by technology.</td><td>P.112 "[...] the few minutes we spend looking at the phone makes us lose track of the class."</td></tr><tr><td>Disturbing others</td><td>People being disturbed due to other people's off-task activities.</td><td>P.121 "Sometimes it can be annoying when someone in front of us does something else on their computer during class. [...]"</td></tr></tbody></table>

**Judgement.** The theme "judgment" summarizes any judgment given on the use of technology for off-task activities by a participant. We identified six different types of judgment: Individual responsibility, bad/to be avoided, no big deal, stay home, and lack of respect. These can be found in Table 8.

<table><caption>**Table 8.** Summary of the qualitative data analysis for the theme Judgement</caption><thead><tr><td>Judgement</td><td>Definition</td><td>Example</td></tr></thead><tbody><tr><td>Individual responsibility</td><td>Everyone is responsible for their own use and the possible consequences of such use.</td><td>P.118 "Since we are all adults, it is up to each of us to decide how we want to use our time, whether we think we need to be attentive or not. [...]"</td></tr></tbody></table>
<table><tr><td>Bad/to be avoided</td><td>The use of technology for anything other than the course is bad or should be avoided during the course.</td><td>P.101 "Should be avoided if something (important) is taught."</td></tr><tr><td>No big deal</td><td>The use of technology for off-task activities during the course is not a serious matter.</td><td>P.16 "[...] Personally, I don't think it's bad if you use your smartphone or computer for other things during class."</td></tr><tr><td>Useful/practical</td><td>Usefulness and convenience of the telephone or computer.</td><td>P.43 "This can be handy to organize something happening later [...]"</td></tr><tr><td>Stay home</td><td>It is better to stay at home than to use your devices in class.</td><td>P.108 "In my opinion one should concentrate mainly on the lecture, otherwise one can stay at home. [...]"</td></tr><tr><td>Lack of respect</td><td>Lack of respect for the professors.</td><td>P.68 "I find it disrespectful to the person who teaches [...]"</td></tr></table>

**Reasons.** The theme "reasons" contains all explanations/justification put forward by students for using technology for off-task activities in class. We identified two subcategories of reasons: intentional reasons and unintentional reasons. Intentional reasons refer to descriptions of a clear goal or objective participants put forward to explain their usage of technology for an off-task activity. We identified two intentional reasons: to occupy oneself and important stuff (cf. Table 9). Reasons classified as unintentional reasons were descriptions with no clear objective or explanation for the off-task activity (e.g., usage because of habit, or because it is tempting). The four nonspecific reasons identified were accessibility, tempting, habit, and addictive.

<table><caption>**Table 9.** Summary of the qualitative data analysis for the theme Reasons</caption><thead><tr><td>Intentional reasons</td><td>Definition</td><td>Example</td></tr></thead><tbody><tr><td>To occupy oneself</td><td>Used to take a break, to keep busy, to pass the time.</td><td>P.125 "[...] But sometimes, when the teacher is rambling on another subject, I often take the opportunity to look at my phone. [...]"</td></tr><tr><td>Important stuff</td><td>Used only for urgent or important personal matters.</td><td>P.108 "[...] If you're expecting an important message, I think it's okay to use your smartphone for this."</td></tr><tr><td>Unintentional reasons</td><td>Definition</td><td>Example</td></tr></tbody></table>
<table><tr><td>Accessibility</td><td>Easy access to the telephone or computer (e.g., within arm's length/on the table).</td><td>P.130 "Having my computer/smartphone handy distracts my attention faster [...]."</td></tr><tr><td>Tempting</td><td>Temptation to use the phone or computer for off-task activities.</td><td>P.110 "[...]. In theory I wouldn't like to do something else, but it's very tempting. [...]"</td></tr><tr><td>Notifications</td><td>The display of notifications that can distract or tempt the person to use their devices for off-task activities.</td><td>P. 33 "[...] especially when notifications are enabled. A notification attracts our attention and we don't pay attention to the course."</td></tr><tr><td>Habit</td><td>Use of technology by habit, automatic reflex.</td><td>P. 96 "These are activities that I do out of habit."</td></tr><tr><td>Addictive</td><td>Difficult to prevent oneself from doing off-task activities on the computer or phone during lectures.</td><td>P.50 "It's a harmful distraction. we do it anyway so it's addictive."</td></tr></table>

# 6 Discussion

Qualitative and quantitative analysis of the data revealed several interesting results. A large proportion of students engage in off-task activities during class. Taking a holistic approach, this study distinguished which devices were used for such activities. In this regard, data revealed that mobile phones were used more often for off-task activities compared to laptop computers. Furthermore, differences were found in the activities that the students performed with the devices. While mobile phones and laptop computers both were often used for communication purposes (i.e., messaging, and social media), students used their laptops during class interestingly quite often to prepare for other courses. A thematic analysis of the comments students formulated regarding their reasons and opinions with regard to off-task use of electronic devices during class revealed that there was a certain awareness of possible negative consequences of such activities on learning. Statistical analysis of learning scores revealed however that it was not the duration students engaged in off-task activities that considerably affected their learning performance but the number of notifications they received on their mobile phones.

## 6.1 What students do in class

Results of this study reveal that the mobile phone was used by a large part of the students (72,3%) for purposes of communication and social media, during an average
time of almost five minutes (in a 90-minutes class). Less often, students reported using their mobile phone for activities such as planning and organization, playing games, or reading news. These findings are in line results of another study showing that social media and communication are the most frequently used app types [17]. However, this other study focused on smartphone usage and did not assess typical computer use in class.

In general, the activities that students engaged in on their laptop computers varied somewhat more (as compared to the mobile phone). Although communication and social media were also often mentioned as off-task activities, this was the case only for 47% of the mentions and during over six minutes on average. Several students mentioned other activities such as preparing for another class, planning and organization, reading online magazines or news, online shopping, or even (online) gaming and streaming.

## 6.2 Students opinion on what they do in class

The qualitative analysis of the students' answers to the questions about their opinion on off-task technologies in the classroom has revealed various interesting results. Firstly, the theme "consequences on learning and attention" shows that students seem to be aware that using technology for off-task activity may have a negative effect on learning. Additionally, students mentioned that off-task activities were distracting, and caused them to lose track of the course content. In line with findings of previous research [e.g., 12], students also noted that usage could be disturbing to fellow students and mentioned that off-task usage was only acceptable as long as it did not disturb others.

The belief that off-task activities are inappropriate in the classroom is also reflected in the theme "judgments", with mentions of off-task activities as being something bad or something that should be avoided. Some students took a more rigid stance by saying that there was no point in coming to a lecture if it was not to follow the class actively, and a few also mentioned that following off-task activities was a lack of respect towards the lecturer. However, some participants gave more moderate judgments. Some said that it was each student's individual responsibility to follow the class or not. Some students also mentioned that off-task activities were no big deal. Lastly, some students said they thought it would be very practical to do off-task activities because it would allow them to get organizational things done.

## 6.3 The impact of what students do in class on learning

While many students seem to believe that off-task activities have a negative impact on learning, the analysis of the results of the learning test has shown that there is only a
very small and non-significant link between time spent on off-task activities and
learning performance. This seems at first to be a rather surprising result, also in view
of the fact that it contradicts the results of previous research [e.g., 2, 7, 8]. However,
the correlational analyses revealed that notifications are negatively linked with learn-
ing while usage time did not show an effect. This emphasizes the importance of noti-
fications for the interplay between off-task activities and learning, corroborating re-
sults from previous research on texting in class [3, 5, 15, 29]. Those studies showed a
negative link between notifications and learning, highlighting the detrimental effect of
notifications in the classroom.

Several assumptions can be put forward to address these unexpected findings of a
missing link between time on off-task activities and learning performance. For one,
intentionality of actions might play an important role on the link between off-task ac-
tivities and learning. It could be assumed that intentional shifts of attention from the
class to off-task activities (i.e., to complete an assignment for another class while the
professor is presenting an example for a theory the student has already understood) is
less detrimental to learning compared to unintentional shifts of attention (i.e., pro-
voked by a notification). This is because it could be assumed that intentional shifts of
attention occur in situations during the lecture when a focus on the content is of lesser
importance (e.g., the student has understood the topic or is bored). This is not the case
for unintentional shifts of attention provoked by a notification, which can arrive at
any moment during a lecture and divert the attention in phases of high importance for
the understanding of the topic (e.g., when a concept is defined which is important for
the understanding of the further development of the lecture). Below you can find an
example of a student expressing what happens when receiving a notification:

P.33 "This is detrimental to the attention given to the lesson, especially when notifica-
tions are activated. A notification attracts our attention, and we don't pay attention to
the course anymore."

However, notifications are not the only unintentional reason mentioned by students to
use their devices for off-task activities. Students also mentioned their temptation, the
accessibility, addiction to their devices and habit for use as unintentional reasons. All
these motifs differ in one way from notifications: A notification is an external inter-
ruption. Contrary to the other unintentional reasons driving students to engage in an
off-task activity, notifications are not an interruption coming from the student them-
self but from someone else.

Regarding the unexpectedly weak link between time spent on off-task activities
and learning, the question arises as to why the results in this study differ from those
presented in other work, where it has been shown that off-task activities are nega-
tively linked with learning. Explanations may be found in differences in the methodo-
logical approach chosen in this piece of research. First, several studies reporting ef-
fects of off-task activities on learning followed an experimental design. In one study
for example, students were asked to watch a lecture recorded on video. To simulate an
off-task activity during a lecture, students were asked to either send a certain number
of text messages or perform another task while following the recorded lecture [7, 8,
42]. In such an experimental setting, it is not the students' choice whether they pay at-
tention to the class or not. The off-task activities represent therefore no self-interrup-
tions, and they are also not intentional. Students are forced by an external reason (the
instructions) to interrupt their learning task, which is very similar to the situation of
being distracted by an external trigger such as a notification. In contrast, in the eco-
logically valid field study, unintentional and intentional shifts of attention can occur,
which either can be triggered internally or externally. However, there have also been
field studies in which a link between off-task activities and learning was shown. Most
of these studies did not assess learning directly after the class but through a grade
point average (GPA) score established at the end of a term [2, 10, 11]. Assessing
learning at the end of the year is a considerable difference because it allows many
other variables to influence the results. For example, it can be assumed that students
who report to engage often in off-task activities are generally less motivated for the
subject, which might influence their GPA performance.

**6.4 Limitations**

The results of this study must be interpreted taking into consideration several limita-
tions. In this respect, it is worth mentioning the relatively small and homogeneous
sample. Data was collected in five different psychology lectures for bachelor students
(group sizes varied between 30 and 60 students) on different topics held by experi-
enced lecturers who did not define specific rules regarding the use of technology dur-
ing the class. In this context, it can be assumed that similar results might be obtained
in comparable circumstances. For future research however, it might be interesting to
test whether these results can be replicated for other study domains (e.g., medicine,
engineering, economy), different types of courses (e.g., seminars, exercises), for dif-
ferent student groups (e.g., high school, undergraduate, Masters etc.), and in classes of
different student numbers (e.g., results of a previous study showed that students' off-
task use increased with class size [43]).

Another limitation might be the fact that students' off-task activities during the
class were self-reports. Although we highlighted the anonymity of the survey as well
as the importance of being honest, it cannot be excluded that social desirability could
be a potential bias leading to an underestimation of the reported frequency of off-task
activities. We are however not the first in this field to use self-reports to asses digital
distraction in class [2, 16, 44]. Previous studies have also asked students to report on the frequency and duration of off-task activities in class. However, these studies ask for mean or typical use, and the ones that also looked at performance asked for overall GPA [2, 16, 44]. By asking students directly after a lecture about the usage and testing their knowledge, we reduce potential memory biases but also avoid potential confounding variables such as motivation or time spent studying for exams. Non the less we are aware of potential biases due to inadequacies in estimating durations and frequencies of off-task technology use in class, although our preliminary study (cf., section 4.3) showed that self-report data as collected in this study seem to be reliable. However, alternative methods of tracking this behavior (e.g., observation or automatic recording via pre-installed app) would not be possible without divulging the purpose of the study and thus significantly affect the observed behavior [45, 46]. After intensive consideration of the advantages and disadvantages of the various methodological options, we concluded that the chosen method was the best fit for the question at hand.

In terms of confounding variables, even in a controlled setting there is risk of them having an impact on results. We specifically tested for digital distraction, but participants might have been distracted by something else (e.g., a peer talking to them, day-dreaming, or studying for another class with a printed document). In addition, it might have been interesting to assess the use of smartwatches in the classroom because such devices are becoming increasingly popular.

Lastly, previous research has shown that prior knowledge on the course topic might influence performance on the learning test [47, 48] or that the use of devices for off-task activities might be influenced by students’ self-efficacy [48]. Therefore, it may be of interest to include such potentially confounding variables in future research.

## 6.5 Implications for HCI

The results reported in this ecologically valid piece of research open up exciting questions for future research. While notifications are a problem, they seem to be an easy fix; notifications can be disabled manually for each application or temporarily by another application. However, while these possibilities are already available, students don't seem to use them. For this reason, more research in this domain is important. Research either needs to focus on methods to make students more aware of the problem and how to fix it or on other alternative ways to reduce disturbance through notifications.

Some alternatives to reduce the impact of notifications have already been suggested [21, 49–51]. Such suggestions encompass, for example, to batch notifications, so users only receive them three times a day [49], as well as to only send notifications
when the phone detects a break cue, something such as silence or a person standing
up, that would suggest a break or a transition moment [50]. These types of sugges-
tions should be tested in the specific context of the classroom.

7 Conclusions

To conclude, this study partially confirms results of previous research by suggesting that notifications are negatively linked with learning. In addition, it suggests that external interruptions have a stronger detrimental effect on learning than self-interruptions. This might be because students have less to no control over the interruption. When students self-interrupt, it is likely that this occurs not during the most crucial part of the class. Notifications however can divert a student's attention at every moment. So, you might not need to worry too much about your students using their devices for off-task activities during your class - just remind them to switch off notifications.
References

1. Anshari, M., Almunawar, M.N., Shahrill, M., Wicaksono, D.K., Huda, M.: Smartphones usage in the classrooms: Learning aid or interference? Education and Information Technologies. 22, 3063–3079 (2017). https://doi.org/10.1007/s10639-017-9572-7.

2. Bjornsen, C.A., Archer, K.J.: Relations between college students’ cell phone use during class and grades. Scholarship of Teaching and Learning in Psychology. 1, 326–336 (2015). https://doi.org/10.1037/stl0000045.

3. Dietz, S., Henrich, C.: Texting as a distraction to learning in college students. Computers in Human Behavior. 36, 163–167 (2014). https://doi.org/10.1016/j.chb.2014.03.045.

4. May, K.E., Elder, A.D.: Efficient, helpful, or distracting? A literature review of media multitasking in relation to academic performance. Int J Educ Technol High Educ. 15, 13 (2018). https://doi.org/10.1186/s41239-018-0096-z.

5. Mendoza, J.S., Pody, B.C., Lee, S., Kim, M., McDonough, I.M.: The effect of cellphones on attention and learning: The influences of time, distraction, and nomophobia. Computers in Human Behavior. 86, 52–60 (2018). https://doi.org/10.1016/j.chb.2018.04.027.

6. Randall, J.G., Oswald, F.L., Beier, M.E.: Mind-wandering, cognition, and performance: A theory-driven meta-analysis of attention regulation. Psychological Bulletin. 140, 1411–1431 (2014). https://doi.org/10.1037/a0037428.

7. Kuznekoff, J.H., Munz, S., Titsworth, S.: Mobile Phones in the Classroom: Examining the Effects of Texting, Twitter, and Message Content on Student Learning. Communication Education. 64, 344–365 (2015). https://doi.org/10.1080/03634523.2015.1038727.

8. Sana, F., Weston, T., Cepeda, N.J.: Laptop multitasking hinders classroom learning for both users and nearby peers. Computers & Education. 62, 24–31 (2013). https://doi.org/10.1016/j.compedu.2012.10.003.

9. Aaron, L.S., Lipton, T.: Digital Distraction: Shedding Light on the 21st-Century College Classroom. Journal of Educational Technology Systems. 46, 363–378 (2018). https://doi.org/10.1177/0047239517736876.

10. Junco, R.: In-class multitasking and academic performance. Computers in Human Behavior. 28, 2236–2243 (2012). https://doi.org/10.1016/j.chb.2012.06.031.
11. Ravizza, S.M., Uitvlugt, M.G., Fenn, K.M.: Logged In and Zoned Out: How Laptop Internet Use Relates to Classroom Learning. Psychol Sci. 28, 171–180 (2017).
https://doi.org/10.1177/0956797616677314.

12. Dabbish, L., Mark, G.: Why do i keep interrupting myself?: environment, habit and self-interruption. 4 (2011).

13. Parry, D.A., le Roux, D.B.: Media multitasking and cognitive control: A systematic review of interventions. Computers in Human Behavior. 92, 316–327 (2019).
https://doi.org/10.1016/j.chb.2018.11.031.

14. Barak, M., Lipson, A., Lerman, S.: Wireless Laptops as Means For Promoting Active Learning In Large Lecture Halls. Journal of Research on Technology in Education. 38, 245–263 (2006).
https://doi.org/10.1080/15391523.2006.10782459.

15. Kornhauser, Z.G.C., Paul, A.L., Siedlecki, K.L.: An Examination of Students’ Use of Technology for Non-Academic Purposes in the College Classroom. Journal of Teaching and Learning with Technology. 5, 1–15 (2016).
https://doi.org/10.14434/jotlt.v5n1.13781.

16. McCoy, B.: Digital Distractions in the Classroom Phase II: Student Classroom Use of Digital Devices for Non-Class Related Purposes. Journal of Media Education. (2016).

17. Brennan, A., Dempsey, M.: The student voice: the students own views on smartphone usage and impact on their academic performance. Presented at the March 5 (2018).
https://doi.org/10.21125/inted.2018.1836.

18. Tran, J.A., Yang, K.S., Davis, K., Hiniker, A.: Modeling the Engagement-Disengagement Cycle of Compulsive Phone Use. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. pp. 1–14. ACM Press, Glasgow, Scotland Uk (2019).
https://doi.org/10.1145/3290605.3300542.

19. Agrawal, P., Sahana, H.S., De', R.: Digital Distraction. In: Proceedings of the 10th International Conference on Theory and Practice of Electronic Governance - ICEGOV ’17. pp. 191–194. ACM Press, New Delhi AA, India (2017).
https://doi.org/10.1145/3047273.3047328.

20. Przybylski, A.K., Weinstein, N.: A Large-Scale Test of the Goldilocks Hypothesis: Quantifying the Relations Between Digital-Screen Use and the Mental Well-Being of Adolescents. Psychological Science. (2017).
https://doi.org/10.1177/0956797616678438.
21. Pielot, M., Rello, L.: Productive, anxious, lonely: 24 hours without push notifications. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI '17. pp. 1-11. ACM Press, Vienna, Austria (2017). https://doi.org/10.1145/3098279.3098526.

22. Sahami Shirazi, A., Henze, N., Dingler, T., Pielot, M., Weber, D., Schmidt, A.: Large-scale assessment of mobile notifications. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 3055-3064. Association for Computing Machinery, Toronto, Ontario, Canada (2014). https://doi.org/10.1145/2556288.2557189.

23. Pielot, M., Vradi, A., Park, S.: Dismissed!: a detailed exploration of how mobile phone users handle push notifications. In: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI '18. pp. 1-11. ACM Press, Barcelona, Spain (2018). https://doi.org/10.1145/3229434.3229445.

24. End, C.M., Worthman, S., Mathews, M.B., Wetterau, K.: Costly Cell Phones: The Impact of Cell Phone Rings on Academic Performance: Teaching of Psychology. (2009). https://doi.org/10.1080/00986280903425912.

25. Kushlev, K., Proulx, J., Dunn, E.W.: "Silence Your Phones": Smartphone Notifications Increase Inattention and Hyperactivity Symptoms. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. pp. 1011-1020. ACM Press, Santa Clara, California, USA (2016). https://doi.org/10.1145/2858036.2858359.

26. Mark, G., Czerwinski, M., Iqbal, S.T.: Effects of Individual Differences in Blocking Workplace Distractions. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. pp. 1-12. ACM Press, Montreal QC, Canada (2018). https://doi.org/10.1145/3173574.3173666.

27. Bowman, L.L., Levine, L.E., Waite, B.M., Gendron, M.: Can students really multitask? An experimental study of instant messaging while reading. Computers & Education. 54, 927-931 (2010). https://doi.org/10.1016/j.compedu.2009.09.024.

28. Froese, A.D., Carpenter, C.N., Inman, D.A., Schooley, J.R., Barnes, R.B., Brecht, P.W., Chacon, J.D.: Effects of classroom cell
phone use on expected and actual learning. College Student Journal. 46, 323–332 (2012).

29. Larry D., R., Alex F., L., L. Mark, C., Nancy A., C.: An Empirical Examination of the Educational Impact of Text Message-Induced Task Switching in the Classroom: Educational Implications and Strategies to Enhance Learning. Revista de Psicología Educativa. 17, 163–177 (2011). https://doi.org/10.5093/ed2011v17n2a4.

30. Lukoff, K., Yu, C., Kientz, J., Hiniker, A.: What Makes Smartphone Use Meaningful or Meaningless? Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2, 1–26 (2018). https://doi.org/10.1145/3191754.

31. Mutchler, L.A., Shim, J.P., Ormond, D.: Exploratory Study on Users’ Behavior: Smartphone Usage. 11 (2011).

32. Liefooghe, B., Barrouillet, P., Vandierendonck, A., Camos, V.: Working memory costs of task switching. J Exp Psychol Learn Mem Cogn. 34, 478–494 (2008). https://doi.org/10.1037/0278-7393.34.3.478.

33. Anderson, S.F.: Best (but oft forgotten) practices: sample size planning for powerful studies. The American Journal of Clinical Nutrition. 110, 280–295 (2019). https://doi.org/10.1093/ajcn/nqz058.

34. Lakens, D.: Sample Size Justification, https://psyarxiv.com/9d3yf/, (2021). https://doi.org/10.31234/osf.io/9d3yf.

35. Faul, F., Erdfelder, E., Lang, A.-G., Buchner, A.: G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods. 39, 175–191 (2007). https://doi.org/10.3758/BF03193146.

36. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Research in Psychology. 3, 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa.

37. Ochs, C., Sauer, J.: Curtailing smartphone use: a field experiment. submitted for publication. (2020).

38. Parry, D.A., Davidson, B.I., Sewall, C.J.R., Fisher, J.T., Mieczkowski, H., Quintana, D.S.: A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nature Human Behaviour. 1–13 (2021). https://doi.org/10.1038/s41562-021-01117-5.
39. Schwarz, N., Oyserman, D.: Asking questions about behavior: cognition, communication, and questionnaire construction. The American Journal of Evaluation. 22, 127–160 (2001).
https://doi.org/10.1016/S1098-2140(01)00133-3.

40. Jobe, J.B.: Cognitive psychology and self-reports: Models and methods. Qual Life Res. 12, 219–227 (2003).
https://doi.org/10.1023/A:1023279029852.

41. Koo, T.K., Li, M.Y.: A Guideline of Selecting and Reporting Intra-class Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine. 15, 155–163 (2016).
https://doi.org/10.1016/j.jcm.2016.02.012.

42. Flanigan, A.E., Babchuk, W.A.: Digital distraction in the classroom: exploring instructor perceptions and reactions. Teaching in Higher Education. 0, 1–19 (2020).
https://doi.org/10.1080/13562517.2020.1724937.

43. Zaza, C., Neiterman, E.: Does Size Matter? Instructors' and Students' Perceptions of Students' Use of Technology in the Classroom. Journal of Information Technology Education: Research. 18, 379–393 (2019).

44. Leysens, J.-L., le Roux, D.B., Parry, D.A.: Can I Have Your Attention, Please?: An Empirical Investigation of Media Multitasking during University Lectures. In: Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists on - SAICSIT '16. pp. 1–10. ACM Press, Johannesburg, South Africa (2016).
https://doi.org/10.1145/2987491.2987498.

45. Haynes, S.N., Horn, W.F.: Reactivity in behavioral observation: A review. Behavioral Assessment. 4, 369–385 (1982).

46. Gittelsohn, J., Shankar, A., West, K., Jr., Ram, R., Gnywali, T.: Estimating Reactivity in Direct Observation Studies of Health Behaviors. Human Organization. 56, 182–189 (2008).
https://doi.org/10.17730/humo.56.2.c7x0532q2u86m207.

47. Beland, L.-P., Murphy, R.: Ill Communication: Technology, distraction & student performance. Labour Economics. 41, 61–76 (2016).
https://doi.org/10.1016/j.labeco.2016.04.004.

48. Lepp, A., Barkley, J.E., Karpinski, A.C.: The Relationship Between Cell Phone Use and Academic Performance in a Sample of U.S. College Students. SAGE Open. 5, 2158244015573169 (2015).
https://doi.org/10.1177/2158244015573169.
49. Fitz, N., Kushlev, K., Jagannathan, R., Lewis, T., Paliwal, D., Ari-
ely, D.: Batching smartphone notifications can improve well-being.
Computers in Human Behavior. 101, 84–94 (2019).
https://doi.org/10.1016/j.chb.2019.07.016.

50. Park, C., Lim, J., Kim, J., Lee, S.-J., Lee, D.: Don’t Bother Me. I’m Socializing!: A Breakpoint-Based Smartphone Notification System. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW ’17. pp. 541–554. ACM Press, Portland, Oregon, USA (2017).
https://doi.org/10.1145/2998181.2998189.

51. Shin, I., Seok, J., Lim, Y.: Ten-Minute Silence: A New Notification UX of Mobile Instant Messenger. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. p. 442:1-442:13. ACM, New York, NY, USA (2019).
https://doi.org/10.1145/3290605.3300672.