# A Novel Protocol Using Captive Portals for FIDO2 Network Authentication +
## 6 Results
### FIDO2 CAP Compatibility and Usability
#### 6.3.2. Usability Experiment with Users  
As explained in Section 5.4.2, we designed and conducted a usability experiment with 15 subjects with their own smartphone devices. Users were asked to connect to the Wi-Fi network running the developed captive-portal authentication service, and authenticate with a registered security key. From the 13 users with compatible smartphones, 69.23% completed the task of connecting to the Wi-Fi network and successfully authenticating in the captive portal with security keys. Table 2 shows the completeness of every step of the task. As we can see, 84.62% reached a compatible web browser to start authentication. For those who had a compatible web browser, 81.82% completed authentication. On the other hand, we detected several errors of different categories that constituted different types of obstacles during the execution of the task. Namely, we found (1) use errors, produced by users; (2) OS errors, caused by the software of the user device; (3) connectivity issues, while connecting the security key via USB or NFC; and (4) compatibility errors, as incompatibility with captive portal on iPhone devices. Table 3 shows the error rate of 1.93. Most of the detected errors were operating system-related errors. Specifically, the most common error was the Android credential manager menu that delayed to appear, which confused some users. Even further, in some cases, we detected that the Android credential manager menu blocks the completion of the operation by becoming frozen, causing the user to abandon the task. On the other hand, there were errors caused by the use. We observed that some users tried to connect the key directly to the unlocked smartphone, expecting the Wi-Fi network to be automatically connected and authenticated without any additional step. Finally, some of the errors were caused by connectivity of the USB using an adaptor. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Step** & **Total Completeness** & **Step Completeness** \\ \hline Connect to Wi-Fi network & 100.00\% & 100.00\% \\ \hline Follow OS instructions & 92.31\% & 92.31\% \\ \hline Browser redirection & 84.62\% & 91.67\% \\ \hline Authentication & 69.23\% & 81.82\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Completeness of every step of the proposed task. Sample of 13 users with compatible smartphones. Figure 10: Prototype UI warning the user that the OS or the browser are not compatible. The button opens a new browser. The target browser depends on the specific OS running the request. This screenshot was taken on an Android 13 phone running the default mini-browser for captive portals, and the button should open the Google Chrome app with the same URL. Most of the observed errors did not cause the task to fail, although they delayed its completion. For measuring the efficiency, we calculated Time-Based Efficiency (TBE) and Overall Relative Efficiency (ORE), shown in Figures 11 and 12. In the first try, users completed \(6.18\times 10^{-4}\) objectives per second, while in the second try, where the users had learnt from the errors, users completed \(1.571\times 10^{-3}\) objectives per second, more than the double. Also, we measured the Overall Relative Efficiency as the ratio of time taken by the users who successfully completed the task in relation to the total time taken by all users. The ORE of the first try is 54.38%, and 35.95% in the second try. In the second try, the users who failed the first time spent more time than the ones that completed the task the first time. Finally, we measured satisfaction through different questions during the experiment. According to Figure 13, 53.33% of users found it easy to complete the process of connecting to the Wi-Fi with security keys, and 57.14% found it easy to try again when some error occurred. At the end of the experiment, as shown in Figure 14, 66.67% of users declared that they would have no problem in using security keys for connecting to a Wi-Fi network, but none of them preferred using keys.