## 3 Dataset  
To investigate users' behavior changes over long-term periods, we collected additional data from the same user group for 68 days, approximately 2 years after the previous experiment was conducted [17]. As shown in Figure 1, we used the existing framework consisting of an Android application, data server, and data analyzer for data collection and analysis. We briefly recap the existing framework and describe the sensor data capture and features extraction process.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Original Model** & **New Dataset** \\ \hline
**Precision** & 80.2\% & 53.1\% (\(\Delta-\)27.1\%) \\
**Recall** & 76.2\% & 41.7\% (\(\Delta-\)34.5\%) \\
**F-Score** & 77.2\% & 44.3\% (\(\Delta-\)32.9\%) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results of accuracy drop against a new dataset collected over long-term periods.  
* **Data Collection Application.** We reused a data collection application called _nCollector_, which determines the label of notifications and senses user contexts when a notification appears. Specifically, the contexts sensed by _nCollector_ are notification contents, users' physical activities, time, and phone status. Subsequently, the 10 features listed in Table 2 were extracted from the sensed data. As listed in Table 2, the label of the notification type is determined based on _whether the application that triggered the notification was launched at a certain time. Auto-labeling_ in Figure 1 provides an overview of how the notification type is determined. Therefore, _nCollector_ keeps track of whether an application that triggers a notification is launched before or after this notification is removed. Two time variables (\(T_{launched}\) and \(T_{removed}\)) are relevant to user decisions. Subsequently, the elapsed time \(T_{elapsed}\) was computed as \(|T_{removed}-T_{launched}|\). If \(T_{elapsed}\) is less than or equal to 10 s (\(T_{reaction}\)), _nCollector_ judges that a user has used the smartphone to handle further interaction with the notification. We carefully defined a heuristic parameter (\(T_{reaction}\)) of 10 s because 25% of the response time for notifications is less than 10 s, and the curve of the response time stabilizes at this point. Even though the quality of modeling strongly depends on the accuracy of auto-labeling, we bring it into this study as a practical method in a real-world scenario because an on-body sensor or user interruption for questionnaire answering is not realistic.
* **Data Collection Server.** We reused the developed _Data Collection Server_ using Java Sever Pages (JSP) and a Bash Shell script on Ubuntu Linux. The labeled notification and context were conveyed to the server. To avoid frequent data transmission and vast data sizes, data are formatted by JSON, and its transmission occurs once at the end of a day.
* **Data Analyzer.** We reused _Data Analyzer_ on Windows 7 using several R packages: JSON parer (jsonlite), parallel computing (doParallel), text handling (text-mining), and machine learning (caret).  
We requested 11 users who participated in the previous experiment join the new experiment. Of the 11 users, 5 agreed to take part in this experiment. Table 3 shows the participant demographics for the additional experiments. Their ages differ from those of the previous experiment because the new data collection started two years after the original experiment.  
Table 4 shows details of collected data across two periods. During 68 days on average, we collected an additional 11,258 notifications. For analysis, we used 18,407 notifications, including the previous dataset (7149 notifications on only 5 users). The whole period for data collection lies across 2 years. Active days for data collection are more than 100 days on average. All the data was collected from the multi-device environment where users use a smartphone and a smartwatch.