# Usability Evaluation of SSI Digital Wallets

Abylay Satybaldy

► To cite this version:

Abylay Satybaldy. Usability Evaluation of SSI Digital Wallets. 17th IFIP International Summer School on Privacy and Identity Management (Privacy and Identity), Aug 2022, Virtual, Germany. pp.101-117, 10.1007/978-3-031-31971-6_9. hal-05315950

HAL Id: hal-05315950

https://inria.hal.science/hal-05315950v1

Submitted on 15 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Usability Evaluation of SSI Digital
## Wallets

Abylay Satybaldy(✉)

Computer Science Department, NTNU, Gjøvik, Norway

abylay.satybaldy@ntnu.no

**Abstract.** Self-sovereign identity (SSI) is a new decentralized ecosystem for secure and private identity management. In contrast to most previous identity management systems where the service provider was at the center of the identity model, SSI is user-centric and eliminates the need for a central authority. It allows the user to own their identity and carry it around in a form of digital identity wallet, for example, on their mobile device or through a cloud service. The digital wallet supports mechanisms for key generation, backup, credential issuance and validation, as well as selective disclosure that protects the user from unintended sharing of user's personal data. In this article we evaluate the usability of existing SSI digital wallets: Trinsic, Connect.me, Esatus and Jolocom Smartwallet. We study how these early experiments with SSI address the usability challenges. We aim to identify the potential obstacles and usability issues, which might hinder wide-scale adoption of these wallets. Applying the analytical cognitive walkthrough usability inspection method, we analyse common usability issues with these mobile-based wallets. Our results reveal that wallets lack good usability in performing some fundamental tasks which can be improved significantly. We summarize our findings and point out the aspects where the issues exist so that improving those areas can result in better user experience and adoption.

**Keywords:** self-sovereign identity · digital wallet · usability study

## 1 Introduction

Identity management is the key component of information flow, provenance, information protection as well as enabler for many digital services. At present, identity data and credentials have become increasingly centralized by organizations and some of the larger corporations. The aggregation of personal data in centralized databases and data silos is undesirable for many reasons, including security, data lock-ins, as well as personal tracking and targeted advertising (e.g. for political influence). The security flaws in design incur an enormous risk of data breaches in the traditional systems. The latest evidence of such data breaches are the cases of Equifax [4], Cambridge Analytica [16], and First American Financial [8], where the identity information of millions of individuals was exposed. Moreover, identity data of users is fragmented among multiple providers. Consequently users must establish and manage a large number of

© IFIP International Federation for Information Processing 2023

Published by Springer Nature Switzerland AG 2023

F. Bieker et al. (Eds.): Privacy and Identity 2022, IFIP AICT 671, pp. 101–117, 2023.

https://doi.org/10.1007/978-3-031-31971-6_9
accounts, IDs and passwords to interact with the large number of repositories, service providers and verifiers. Lack of adequate privacy controls, and visibility to how user identity data is generated, managed and shared by third parties is yet another major concern.

These challenges have led the industry and the academia to explore innovative approaches to the management of digital identity information and cryptographic secrets. Self-sovereign identity (SSI) is a new architecture for privacy-preserving and user-centric identity management. Christopher Allen proposed a set of ten properties for SSI, which are often referred to in the literature, and are as follows: “existence, control, access, transparency, persistence, portability, interoperability, consent, minimization, and protection” [1]. SSI aims to avoid a single point of dependency and allows individuals to take ownership of their digital identities.

SSI is a set of technologies that move control of digital identity from third parties directly to individuals. The foundation concepts of SSI were officially brought to life when the Credentials Community Group was created under the umbrella of the international organization, the World Wide Web Consortium (W3C). The group defined two fundamental standards for the development of a new decentralized identity architecture: Decentralized Identifiers (DIDs) [38] and Verifiable Credentials (VCs) [39]. DIDs are a new type of identifier that is decentralized, resolvable and cryptographically verifiable. It differs from other types of identifiers in that it can exist without the involvement of any certificate authorities, third parties, providers, or centralized identity registers. VCs are typically used in conjunction with DIDs and are used to build trust between the involved parties in an SSI ecosystem, which includes an issuer, a holder, and a verifier as shown in Fig. 1.

In SSI, there is no central authority, so users hold their own digital keys and have full control over their personal information. This information is typically carried around by the user in the digital identity wallet on their mobile device. A digital wallet, in the context of SSI, is a software application and encrypted database that stores credentials, keys, and other secrets necessary for self-sovereign identity management to operate. It enables the user to have data sovereignty and complete control as well as data portability. A user can establish relationships and interact with third parties in a trusted manner. The digital identity wallet contains public-private key pairs that can be used to sign transactions, statements, credentials, documents or claims. Each participant in the SSI ecosystem needs a digital wallet in order to issue, hold, and verify VCs.

Users use a personal digital wallet to manage their keys and data which means that unlike typical systems that store data centrally, credentials and identity data would be saved in digital wallets that would be distributed all over the edges of the network giving individual users full control over their personal data. This would also vastly increase the complexity of any kind of attack, and even if certain systems were penetrated, it would no longer be a massive honeypot containing millions of individuals’ personal information. The data can be delegated and managed by a third party service, and in the ideal case it is up to the user to choose the provider of those services. Even though the delegation models will be the most convenient for novice users, in this research we focus on
<figure><img src="image_3.png" /><figcaption>Fig. 1. Self-Sovereign Identity Architecture.</figcaption></figure>

the case in which the users hold the identity data directly themselves on their mobile devices.

The concept of European Digital Identity Wallet was published by the European Commission in 2022 [12]. This initiative from the EU resulted in an enormous increase in interest in SSI, decentralized identity and wallet-based systems. In this paper we aim to contribute to this novel technology by evaluating usability of existing digital identity wallets.

## 1.1 Problem Statement

Self-sovereign identity systems should be designed to solve the challenges faced by end users. So far, we have seen that the existing implementations mainly focus on the underlying technology, not the user interaction [28,36]. Usable interface and privacy implications for users are not addressed yet in sufficient depth [29]. The future SSI schemes with a novel technological underpinning but developed with impractical end user interaction are unlikely to create widespread uptake.

## 1.2 Research Objectives

The research objectives are following:

- To evaluate the usability of digital identity wallet apps among its users.

- To derive recommendations from the user study to enhance the usability of digital identity wallets.

# 2 Related Work

Several researchers have pointed out that usability remains a pressing unknown in the existing SSI solutions [6,35]. However, there have been a few existing works which studied the usability issues related to SSI systems [7]. Zaeem et al. [42]
studied the most commonly used SSI solutions with respect to their usability. The paper reports several usability issues related to heavy reliance of wallets on QR code scanning and proposes potential solutions to replace the QR code when applicable. Another usability pitfall the study found was the backup and recovery of identity credentials. Although they identified some concrete usability problems, the study was based on general observations rather than any quantitative analysis and did not involve any external participants.

We have not found any study that has applied the analytical cognitive walk-through inspection method to investigate common usability issues with SSI wallets. However, there are several studies which relied on this methodology to examine usability problems in cryptocurrency wallets. For example, Moniruzza-man et al. [20] used the analytical cognitive walk-through usability inspection method to show that both desktop and mobile-based cryptocurrency wallets lack good usability in performing the fundamental tasks. The usability study identified several common usability issues with wallets such as the use of difficult technical language, lack of interface cues and action guides, and user restrictions due to abstractions. Eskandari et al. [11] also applied a cognitive walk-through inspection method to evaluate the usability and security issues in several bitcoin wallets and found that users performing tasks involving key management can be stuck with complex security issues. Moreover, the study revealed that the metaphors and abstractions used in the analysed wallets are subject to misinterpretations, and that the wallets do not do enough to support their users.

As there are significant differences in terms of core functionalities between cryptocurrency and identity wallets, we aim to fill in this gap in this study and evaluate the usability of identity wallets by applying an analytical usability inspection method, the cognitive walkthrough method.

## 3 Methodology

There exist different usability inspection methods such as the heuristic evaluation (HE) [2], the cognitive walkthrough (CW) [32], the pluralistic walkthrough (PW) [26], and the formal usability inspections [15]. HE is done by expert evaluators examining the design of a user interface and judging its compliance with a list of predefined principles (heuristics). HE is a low-cost and intuitive method for evaluating the user interface early in the design process. On the other hand, CW is a task-oriented and structured method that focuses on the learnability of a system for new users and can be performed on an interface at any time during the development process, from the original mock-ups through the final release [25]. It rests on the assumption that evaluators are capable of taking the perspective of the user and can apply this user perspective to a task scenario to identify user-interface problems. PW adapted the traditional usability walk-through to incorporate representative users, product developers, members of the product team, and usability experts in the process. This approach offers feedback from users even if the interface is not fully developed and enables rapid iteration of the design cycle [15]. Formal usability inspections are structured activities
with defined steps and trained inspectors. This method is most appropriate for
more complex software where product teams want to track usability defects and
establish a process to detect and eliminate major usability bugs [41].

Our main goal was to investigate the usability issues digital identity wallet
users face. Considering that the main focus of the study is on new users and
the selected wallet applications for testing are all already released, the cognitive
walkthrough method has been selected for evaluating the usability of wallets.

**3.1 Procedure**

The cognitive walkthrough is conducted following the procedure summarized
below:

1. Determine what tasks are most appropriate for the walkthrough.

2. Select a group of evaluators and give them basic training on the process.

3. Evaluators walk through the action sequences for each task and answer the following four standard questions which were first mentioned in the usability book by C. Wharton et al. [40] and currently is considered as a standard procedure in the cognitive walkthrough questionnaires. These questions essentially require each participant to answer in a binary yes/no fashion where a yes signifies that the user has been able to perform the respective task whereas a no signifies the user has failed.

Q1. Will the user understand how to start the task?

Q2. Is the control (e.g. button) for the action clearly visible?

Q3. Will the user associate the correct action with the outcome they expect to achieve? (link of the control with the action).

Q4. If the correct action is performed, will the user see that progress is being made towards their intended outcome?

4. If the evaluator fails to complete the task, the short interview was conducted to identify the exact problem.

5. Record issues and design suggestions that were not the direct output of the walkthrough, assumptions about users, comments about the tasks, and other information that may be useful in improving the usability of wallet apps.

**3.2 Usability Evaluation**

The study collected qualitative (interview notes) and quantitative (questionnaire
results) data during the cognitive walkthrough usability evaluation. The main
focus of the study was to identify usability issues users encounter when using dig-
ital wallets. Therefore short interviews were conducted whenever the evaluator
failed or struggled to perform the task during the cognitive walkthrough.

The data from the questionnaire results was used to measure the effectiveness
and the number of errors per task. Errors are defined as mistakes that are made
by the participant when attempting a task. Effectiveness can be measured using
the completion rate of tasks and can be defined as a percentage by utilising the simple equation represented below [3].

$$ Effectiveness = \frac{Number of tasks completed successfully}{Total number of tasks undertaken} * 100\% $$

## 3.3 Tasks and Scenario

For our experiment, we have selected five major tasks (Table 1) based on the core functions of an SSI wallet specified in the literature [21, 23, 24]. To make it easy for evaluators, we focus on a scenario where a user (Alice) will receive a college diploma and will prove to her new employer she graduated college to get a job.

**T1: Configure.** User installs a wallet for the first time in a new device, creates a new account and saves his/her wallet recovery phrase.

**T2: Establish a connection with third parties.** The SSI wallet can create an encrypted communication channel to exchange information between a user and a third party (issuer, verifier or another user). This communication channel is based on a unique identifier, which the user controls. The user receives connection invitations which are usually transmitted through QR codes or deeplinks. The user scans the QR code and accepts the connection invitation that shows up in the wallet. The connection will be added to the user's wallet.

**T3: Receive a credential from issuer and store it in a wallet.** Issuer offers a credential to the user. The user opens the credential offer and inspects the attributes. If everything looks correct, the user accepts the offer. The credential will be issued to the user's wallet, and can be viewed in the corresponding tab.

**T4: Share a credential with verifier and complete the verification process.** The user gets the verification request from the verifier. From the home screen, the user selects the verification request to respond, customizes the information and shares only required attributes (if a wallet supports the selective disclosure). If the information looks correct, the user shares the credential and gets confirmation whether the credential proof was successful or not.

**T5: Backup and restore a wallet.** This task will require the user to set up a backup for the wallet and then restore it. The recovery phrase is usually used to recover the encrypted wallet in case the device is lost or destroyed.

<table><caption>Table 1. Fundamental tasks and sub-tasks for the experiment.</caption><tr><td>T1: Configure</td><td>T1.1 Download and install the wallet app<br>T1.2 View and save wallet recovery phrase</td></tr><tr><td>T2: Establish a connection with third parties</td><td>T2.1 Scan QR code and accept the invitation<br>T2.2 View the added connection in the wallet</td></tr><tr><td>T3: Receive a credential from issuer and store it in the wallet</td><td>T3.1 Inspect the attributes and accept the credential offer<br>T3.2 View the new credential in the wallet</td></tr><tr><td>T4: Share a credential with verifier and complete the verification process</td><td>T4.1 Select the verification request to respond and present the credential that can satisfy the request</td></tr><tr><td>T5: Backup and restore the wallet</td><td>T5.1 Set up a local or cloud backup for the wallet<br>T5.2 Restore the wallet</td></tr></table>
## 3.4 Selected Wallets

Table 2 presents an overview of the eight analysed SSI wallet providers. The study results show that all except Civic base their development work on open SSI standards and specifications by W3C and DIF. This highlights the positive development in SSI space as following common standards considerably help achieving interoperability between existing SSI systems. The three of the reviewed SSI systems does not provide a mobile application for wallet. To meet the study's goal of examining usability issues, four different digital identity wallet applications that satisfy our requirements (Table 2) were selected. The wallets selected for this study are Trinsic [37], Connect.me (Sovrin) [13], Esatus [10] and Jolocom Smartwallet [17]. We selected these wallets because they follow the SSI principles and common standards. They provide mobile wallet applications on both iOS and Android platforms and regularly maintain them. The selected wallets support core SSI functionalities with minimum feature sets maturity of level 1 (basic) as presented by Rosendahl (2022) [27] and can be used to conduct usability tests. Additionally, they support several distributed ledgers, have sizable online communities, and provide tutorials and documentation with the most technical details of their designs.

## 3.5 Participant Requirement

The participants were recruited via project advert asking their interest and engagement. Upon receiving expression of interest from the participants, we contacted them via email to book a time as per their availability. Prior to the evaluation, each participant was supplied with the documentation of our designed cognitive walk-through method and other necessary tutorials.

The main focus of study was to test new users who have not used identity wallet apps before. Therefore, we have excluded users with prior experience. According to the literature [14,15], it is best to narrow down the target audience to the users most likely to use the product. As identity wallets are at early stage of development and there is no data available on user base, we looked into similar mobile applications like Apple Wallet app and cryptocurrency wallets. Based on our analysis, Apple Wallet usage is the highest among Millenials and Generation Z consumers [19,30] and cryptocurrency wallets are most popular with young adults [5,34]. Therefore, we have selected graduate students and young professionals within the age group of 20 to 35 years.

<table><caption>Table 2. Overview of the SSI wallet providers.</caption><thead><tr><th>Characteristics</th><th>Esatus</th><th>Civic</th><th>Jolocom</th><th>Trinsic</th><th>Sovrin</th><th>Spruce</th><th>Affinidi</th><th>Veramo</th></tr></thead><tbody><tr><td>Align with SSI standards (W3C and DIF)</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Mobile application for wallet (iOS and Android)</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>No</td></tr><tr><td>Regularly maintained (mobile wallet app)</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Core SSI features maturity (mobile wallet app)</td><td>Rich</td><td>Rich</td><td>Basic</td><td>Rich</td><td>Basic</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Supported distributed ledgers</td><td>Esatus, Sovrin, IUnion</td><td>Ethereum, Solana</td><td>Jolocom</td><td>Sovrin, BCGov, Indy</td><td>Sovrin</td><td>ION, Timos, Ethereum</td><td>Ethereum</td><td>Ethereum</td></tr></tbody></table>
It is widely accepted that testing with small group of users is capable of finding almost as many usability issues as testing with many users, specifically when the study focuses on one target group as in our case [18, 22]. Moreover, we aim to conduct a case study for a relatively new concept and carry out an in-depth investigation employing quantitative and qualitative methods. Therefore, a small group of five participants were recruited. Each experiment lasted between 60 to 80 min in average and was conducted with mobile wallet apps on Apple iPhone powered by iOS 15.

# 4 Results

In this section, we describe the qualitative and quantitative results obtained from the study. As part of the qualitative analysis short interviews were conducted whenever the evaluator failed or struggled to perform the task during the cognitive walkthrough. Overall, 35 different usability problems were captured from evaluation sessions.

## 4.1 Qualitative Analysis

### 1. Trinsic

**T1: Configure.** Trinsic app uses onboarding screens to introduce users with various features of the app and to inform them that their identity data is encrypted and stored locally. The welcome page also contains *Recover wallet* button that enables existing users to restore their wallet. Evaluators reported that the onboarding process was smooth and informative. The app does not offer a password or passcode options and only relies on biometric authentication to protect the wallet. However, although all participants enabled a biometric security during the configuration phase, the app has not shown a biometric authentication dialog afterwards and basically allows anyone to access the app.

**T2: Establish a connection.** All participants were able to scan QR code and accept the invitation (T2.1) as the app's home page contains a large *Scan code* button with additional animation on top. Two evaluators could not find the *Connections* (failed Q1 and Q2) to view the added connection (T2.2) from the first attempt. The *Connections* tab is two taps away from the home page which features three primary tabs (Home, Credentials, and Settings) but these tabs are represented by descriptive icons only. It might be helpful to add texts under icons to avoid confusion among end users.

**T3 and T4: Receive and share a credential.** The app allows a user to scan a QR code, inspect the attributes and accept the credential offer from a issuer. If the user does not choose the correct blockchain network before accepting the credential, "A network mismatch has occurred. Please select the correct network" message shows up. It would be helpful if the message also notifies the user which network to choose as Trinsic supports five
different blockchain networks. Moreover, Trinsic users have to use QR code scanning each time they receive and share a credential.

**T5: Backup and restore a wallet.** Users can navigate to *Backup and restore* tab under *Settings* menu and can choose local or cloud backup options. Trinsic supports a wallet recovery phrase which is displayed as a 12-word phrase. The app uses in-app notification to confirm that the user has already written down his/her phrase and warns that it is not possible to recover the wallet without this phrase. If the wallet was successfully backed up, it shows a notification to the user that this has occurred. All evaluators answered with *Yes* to Q4 and confirmed it. To restore the wallet users need to enter their recovery phrase into the box and click "Recover wallet" button. The app does not explain the correct format the phrases should be entered.

## 2. Jolocom

**T1: Configure.** Passcode protection is mandatory in Jolocom, and the user is notified about the importance of securing the wallet on first run. Finding the generated recovery phrase was not challenging for evaluators and T1.2 was accomplished successfully by all evaluators. The user must navigate to the *Settings* tab located in the navigation bar. The interface displays a warning about the importance of saving the secret phrase. However, the app allows only one opportunity to view and write down the phrase. Although there is a warning about this feature, it might not be the best practice as users might easily make a mistake during the process or lose it afterwards.

**T2: Establish a connection.** Although the QR code scan button is represented by only icon, all evaluators reported that it is clearly visible and they understand how to start the task. When the connection is added to the user's wallet, the app shows the notification "Success! Your interaction was successful" to let the user know that the process is complete. But the app gave the same notification message for all other processes. Participants found it too generic and less informative. Moreover, there is no a separate tab for connections. Once the user establish a connection, it will show up under *History* tab with other processes such as issuance and verification requests. The organization of information on this screen was not very clear to the study participants.

**T3 and T4: Receive and share a credential.** The app uses in-app pop-up notifications to show the incoming credential offers. The notification message shows a sender, credential and corresponding attribute names, and provides the user with *Accept* and *Ignore* options. However, the attribute values are hidden which does not allow the user to fully inspect the credential before accepting it. The issued credential can be viewed from the *Documents* tab.

**T5: Backup and restore a wallet.** Jolocom wallet supports seed phrases for private key recovery. To recover the wallet users need to click *Import identity* button and insert 12-word recovery phrase one by one. Evaluators reported that the recovery option is only available during the
configuration phase. However, restoring the wallet content was not possi-
ble as the app does not backup properly.

3. Esatus

**T1: Configure.** Esatus does not have onboarding screens that guide users on how to use an app's various features as in other wallets. The apps only requires the user to create a PIN and enable notifications. Biometrics could be enabled from *Settings* menu. Esatus generates a new set of seed phrases during the setup of wallet, but shows no notification to the user that this has occurred. The wallet recovery phrase can be found under the *Create/Export backup* tab in *Settings* menu. One participant struggled to find the correct tab (failed Q2).

**T2: Establish a connection.** Evaluators reported that all the tabs for primary functions (Connections, Credentials, Scan, and Settings) can be easily accessed from the home screen and represented by recognizable icon and text. When the user establish a connection with a third party it shows up in *Connections* tab. For each connection the app provides a transparent history of interactions between parties. Participants have not experienced issues with T2.2.

**T3 and T4: Receive and share a credential.** All received credentials are visible in *Credentials* tab which is also the home page of the app. The app uses horizontal scrolling list to display credentials. However, horizontal scroll works best only when there are few credentials available and is not very convenient when the wallet stores many credentials. *Info* section displays some redundant information such as credential definition and schema (e.g. `KuXsPLZAsxgjbaAeQd4rr8:2:schema:2.0`) which are useful for developers but not for the average app users. There is also *Proofs* tab in the home screen which allows to create proofs but the app does not provide any explanation about this feature and how to use it. The *Proofs* screen just displays "No Gateways found" when the user opens it first time. The terminology is too technical for users.

**T5: Backup and restore a wallet.** As with other wallets, recovery is not possible if no backup of the wallet file was made. Creating a backup is straightforward by using the *Create/Export backup* menu and then following the instructions. The app also verifies that users saved the recovery phrase correctly.

4. Connect.me

**T1: Configure.** The evaluators claimed that the onboarding process is very limited and features only user approval for enabling biometrics and notifications. Connect.me generates private keys for its users, but the process is hidden and abstracted from users. Moreover, the study participants could not view their recovery phrases.

**T2: Establish a connection.** Connect.me features a separate *My connections* tab for displaying the list of added connections. The app gave clear error messages when the process of establishing a connection failed. The wallet relies on QR code scanning to initially connect an identity
owner with other entities. But the app heavily relies on QR code scanning and requires it for every interaction such as receiving issuance and verification requests which might hinder the user experience. Moreover, it was reported that the *Scan* button is too small and its icon's colour is same as its background (white) which makes it hardly visible.

**T3 and T4: Receive and share a credential.** Credential offer notification message displays the name of the issuer and all attributes which can be inspected by simply scrolling down. After inspection the user can *Accept* or *Reject* the credential. Users also get notified when the process is complete and the credential is issued. The verification request transparently shows the name of the verifier and which attributes is being requested. Users can view added credentials under the tab *My Credentials* which has a colorful and graphical user interface. The study participants agreed that it was easy to use the wallet for receiving and sharing the credential.

**T5: Backup and restore a wallet.** It was found that Connect.me wallet app still does not support this fundamental feature and it was recorded as a failure in our experiment. Users of this wallet will not be able to restore their personal data, credentials and connections in case of mobile device loss or theft.

## 4.2 Quantitative Analysis

Analysis of the results is performed mainly on the metrics such as user's effectiveness performance and the number of errors made per task. There were 5 main tasks (T1-T5) with a total of 9 sub-tasks (Table 1). For each sub-task evaluators had to complete 4 steps (total 36 steps) and answer usability questions (Q1-Q4) mentioned in Sect. 3.1. During the experiment a total of 36 *yes-no* answers from each evaluator were recorded. A *yes* signifies that the user has been able to perform the respective step for sub-task whereas a *no* signifies the user has failed. We assumed that a task is successful if the user was able to perform all four steps (answered Q1-Q4 with *yes*).

Table 3 shows the effectiveness performance average for selected wallet applications. The methodology for calculating the effectiveness is described in Sect. 3.2. The average effectiveness for Connect.me and Jolocom wallets were quite low, 56.11% and 75.00% respectively. Users had much smoother experience with Esatus and Trinsic mobile wallets and consequently higher effectiveness performance was observed, 91.11% and 90.56% respectively. As we can see, users

<table><caption>**Table 3.** Effectiveness average for the selected wallets.</caption><thead><tr><th>Wallets</th><th>Esatus</th><th>Connect.me</th><th>Jolocom</th><th>Trinsic</th></tr></thead><tbody><tr><td>Effectiveness average (%)</td><td>91.11</td><td>56.11</td><td>75.00</td><td>90.56</td></tr></tbody></table>
failed to complete some fundamental tasks in every examined digital identity
wallet.

Figure 2 better represents the difficulties faced while completing the tasks for all the selected wallets where the graphs illustrate the average number of failed steps per task (signified by a *no answer*) by participants. We can observe that receiving a credential (T3) and sharing it with verifier (T4) were the most easy tasks to accomplish by the participants as the lowest number of steps were failed. For every selected wallet, users struggled with task T5 most - backing up and restoring a wallet.

# 5 Discussion

In this section, we summarize and discuss the main usability issues that were identified from our study and propose recommendations to enhance the usability of digital identity wallets.

All the four studied SSI wallet apps rely on QR code scanning to initially connect an identity owner with an issuer or a verifier. One of the major advantages of using QR code in SSI wallet apps is its ease of use and mobile friendliness which was again confirmed by our study participants. However, several participants reported that some examined wallets heavily rely on QR code scanning and require it for every interaction such as receiving issuance and verification requests which might hinder the user experience. This issue was also reported by Zaeem et al. [42] and our results confirm it. Based on this feedback, we recommend using QR code scanning for only opening a private and secure channel between the two parties. Then for further interactions use message exchange through already established secure connection (i.e. using DIDComm Messaging [9]) and in-app notifications to show incoming requests to the user.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** The average number of failed steps per each task for the selected wallets.</figcaption></figure>
SSI digital wallets do not support third-party centralized authentication methods and do not require a new user to create a login with password for the app during the configuration phase which was confirmed in this study. Instead wallets allow a user to create its own DIDs, receive credentials from different issuers and form his digital identity. Users will be using their wallets to authenticate to other services. Therefore it is important to ensure that only authorized users can access the wallet and its data. The study revealed that some wallets do not support passcodes/passwords and in one of the evaluated wallets the biometrics authentication was not working properly. Digital identity wallets should support both the passcode and biometrics authentication to protect the sensitive personal data stored in the wallet.

Several participants observed a little longer delays when retrieving or verifying credentials and it was not very clear to them why it usually happens. The reason behind this is that identity wallet apps in most cases need to synchronize with a public blockchain network and update the state in the ledger. Therefore, in some cases, decentralized apps<fnref n="1" /> like wallets might be slower than traditional apps which we use daily. Moreover, all of the selected wallets support several blockchain networks. Setting the correct network before retrieving or verifying the credential is important and was a challenge for some participants. Wallets should have more intuitive interfaces with the interface cues enabling non-technical users to get comfortable with the app.

On first run, all of the evaluated wallets generate the recovery phrase without informing the user and do not provide any information on it. The abstractions made in identity wallets are sometimes beneficial for users, such as in the case of hiding the credential definition and schemas. However, the recovery phrase is the key to the wallet and the user should understand it and be warned about the importance of not forgetting the secret phrase. This information could be included in onboarding screens.

The study participants reported that some wallets used difficult technical languages. Examples of such languages are “out of sync”, “no gateways found”, and some others mentioned in Sect. 4.1. Highly technical language often becomes incomprehensible to average users due to the unfamiliar terminology that’s simply part of the technical domain. These technical terms should be replaced by simpler instructions.

Traditional identity management models provide a key management protocol that rely on a trusted third party, while in the context of the SSI model the responsibility of key management is assigned to the identity owners themselves. The three wallets out of four presented here support seed phrases which enables the user to recover his/her wallet account (a private key). However, if you lose your device you may still lose all your existing credentials and connections stored in a wallet, even if you have recovered the private key. Thus, users are advised to backup encrypted wallet data. Our study reveals that only two of examined wallets (Trinsic and Esatus) allow the user to restore his/her personal data stored

<footnote n="1">¹ Decentralized applications (DApps) are applications that run on top of blockchain networks.</footnote>
<table><caption>Table 4. Problems and recommendations</caption><thead><tr><td>Usability Issues</td><td>Recommendations</td></tr></thead><tbody><tr><td>The wallet app heavily relies on QR code scanning</td><td>Use QR code scanning for only opening a private and secure channel between two parties. Then for further interactions use message exchange through already established secure channel</td></tr><tr><td>Users experience longer delays when a wallet interacts with a blockchain</td><td>Use a graphical view indicator that displays the status of work being done. The visual cue allows users to see progress, which encourages them to wait</td></tr><tr><td>The user faces difficulty in setting the correct blockchain network</td><td>Wallets should have more intuitive interfaces with the interface cues and instructions enabling non-technical users to get comfortable with the app</td></tr><tr><td>The wallet app uses a difficult technical language</td><td>Replace highly technical terms with simpler instructions</td></tr><tr><td>The user is unable to backup the wallet content</td><td>The app should offer cloud and local backup options for its users</td></tr><tr><td>Seed phrases are a single point of failure</td><td>Decentralized key recovery solutions should be considered.</td></tr><tr><td>The user has problem finding the location of the recovery phrase</td><td>The wallet should have a separate visible tab that allows to view and save the recovery phrase</td></tr><tr><td>The user is unable to understand the functionality of the recovery phrase</td><td>The wallet should provide an explanation as to why it is important to securely save the recovery phrase and how to use it. This message should be easily accessed (e.g. a welcome page), along with links to further instructions</td></tr><tr><td>The app uses only icons for primary tabs</td><td>It might be helpful to add texts under icons to avoid confusion among end users.</td></tr><tr><td>The mobile authentication is not working properly</td><td>The app should support mobile authentication methods that are secure, reliable and protect the sensitive personal data stored in the wallet</td></tr></tbody></table>

in the wallet through cloud or local backup. Jolocom does provide this feature but it does not work properly. Connect.me wallet does not provide any options to recover and restore the wallet. The wallets also should notify the user about the risks of not taking a backup. Based on experiment results, only one of four selected wallets has done it.

Although evaluators have not reported critical usability challenges regarding the use of seed phrases during the experiment, using these seeds is not the most safe way for recovery as it presents a single point of failure. To improve the usability of backup and recovery, decentralized key recovery solutions should be considered. For example, a social recovery method which employs entities trusted by the identity owner called 'trustees' who store recovery data on an identity owner's behalf typically in their own wallets. There are several research papers that proposed decentralized key backup and recovery solutions for SSI wallets based on Shamir's Secret Sharing algorithm [31,33] which could be potentially implemented in existing identity wallets.

Table 4 contains examples of usability problems frequently encountered in our evaluation and the recommendations from us to resolve them.

## 6 Conclusions

In this paper we presented the usability study of some existing digital identity wallets focusing on a series of tasks. The cognitive walkthrough inspection method has been used to identify the usability issues and to measure the effectiveness performance of users. Based on the review of the current state-of-the-art
and comparison analysis, several wallets were selected. In our evaluation, we have observed that some existing digital identity wallets already provide a decent level of user experience and are good in terms of effectiveness, however there are still a number of usability issues in fundamental tasks that need improvements. We summarized the main challenges and proposed recommendations that may be helpful to improve the overall usability and accessibility among the users. Our findings suggest that the key management and wallet recovery methods need to be improved. It is crucial for digital identity wallets to have a secure and usable management of keys and personal data. Wallets should have more intuitive interfaces with the interface cues enabling new users to get comfortable with the app. Resolving these issues will considerably improve the user acceptance and global adoption of digital identity wallets.

All of the participants in this study were young adults employed/studying within the same university. While this likely introduces bias, our subjects were from different departments and well fit for the existing user base of digital wallets. Nonetheless, one future direction from this work would be to perform similar evaluations with older participants from different sectors. As the SSI technology matures and gains traction digital wallets might become integral part of the general population in near future.

## References

1. Allen, C.: The path to SSI (2016), bit.ly/3W7KXV0. Accessed 20 May 2022

2. Alonso-Ríos, D., Mosqueira-Rey, E., Moret-Bonillo, V.: A systematic and generalizable approach to the heuristic evaluation of user interfaces. Int. J. Human-Comput. Interact. 34(12), 1169–1182 (2018)

3. Alturki, R., Gay, V.: Usability testing of fitness mobile application: case study aded surat app. Int. J. Comput. Sci. Inform. Technol. (IJCSIT) 9, (2017)

4. Berghel, H.: Equifax and the latest round of identity theft roulette. Computer 50(12), 72–76 (2017)

5. Bohr, J., Bashir, M.: Who uses bitcoin? an exploration of the bitcoin community. In: 12th Annual International Conference on Privacy, Security and Trust, pp. 94–101. IEEE (2014)

6. Čučko, Š, Bećirović, Š, Kamišalić, A., Mrdović, S., Turkanović, M.: Towards the classification of SSI properties. IEEE Access 10, 88306–88329 (2022)

7. Čučko, Š, Turkanović, M.: Decentralized and self-sovereign identity: Systematic mapping study. IEEE Access 9, 139009–139027 (2021)

8. Dellinger, A.: Understanding the first american financial data leak: How did it happen and what does it mean? https://bit.ly/3QvjRG2 Accessed 1 May 2022

9. DIF: DIDComm Messaging. https://bit.ly/3CBJ1Mk Accessed 23 June 2022

10. Esatus: Esatus wallet, https://esatus.com/ Accessed 20 June 2022

11. Eskandari, S., Clark, J., Barrera, D., Stobert, E.: A first look at the usability of bitcoin key management. arXiv preprint arXiv:1802.04351 (2018)

12. European Commission: European Digital Identity Architecture and Reference Framework. Tech. rep. (2022), https://bit.ly/3kaFIXe

13. Evernym: Connect.me wallet, https://connect.me, Accessed 27 June 2022

14. Edmunds, M., Hass, C., Holve, E. (eds.): Springer, Cham (2019). https://doi.org/10.1007/978-3-319-96906-0
15. Hollingsed, T., Novick, D.G.: Usability inspection methods after 15 years of research and practice. In: Proceedings of the 25th Annual ACM Conference on Design of Communication, pp. 249–255 (2007)

16. Isaak, J., Hanna, M.J.: User data privacy: Facebook, cambridge analytica, and privacy protection. Computer **51**(8), 56–59 (2018)

17. Jolocom: Jolocom SmartWallet, https://jolocom.io/ Accessed 20 May 2022

18. Khajouei, R., Zahiri Esfahani, M., Jahani, Y.: Comparison of heuristic and cognitive walkthrough usability methods for evaluating health information systems. J. Am. Med. Inform. Assoc. **24**(e1), e55–e60 (2017)

19. Liébana-Cabanillas, F., García-Maroto, I., Muñoz-Leiva, F., Ramos-de Luna, I.: Mobile payment adoption in the age of digital transformation: The case of apple pay. Sustainability **12**(13), 5443 (2020)

20. Moniruzzaman, Md., Chowdhury, F., Ferdous, M.S.: Examining Usability Issues in Blockchain-Based Cryptocurrency Wallets. In: Bhuiyan, T., Rahman, M.M., Ali, M.A. (eds.) ICONCS 2020. LNICST, vol. 325, pp. 631–643. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52856-0-50

21. Mühle, A., Grüner, A., Gayvoronskaya, T., Meinel, C.: A survey on essential components of a self-sovereign identity. Comput. Sci. Rev. **30**, 80–86 (2018)

22. Nielsen, J.: How many test users in a usability study?, https://www.nngroup.com/articles/how-many-test-users Accessed 1 May 2022

23. Podgorelec, B., Alber, L., Zefferer, T.: What is a (digital) identity wallet? a systematic literature review. In: 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC). pp. 809–818. IEEE (2022)

24. Preukschat, A., Reed, D.: Self-sovereign identity. Manning Publications (2021)

25. Privitera, M.B.: Heuristic analysis, cognitive walkthroughs & expert reviews. In: Applied Human Factors in Medical Device Design, pp. 165–180. Elsevier (2019)

26. Riihiaho, S.: The pluralistic usability walk-through method. Ergonom. Design **10**(3), 23–27 (2002)

27. Rosendahl, R.: Understand your application’s feature set maturity (2022), bit.ly/3XlABSn, Accessed 24 November 2022

28. Satybaldy, A., Nowostawski, M., Ellingsen, J.: SSI systems. In: IFIP International Summer School on Privacy and Identity Management. pp. 447–461. Springer (2019)

29. Schardong, F., Custódio, R.: Self-sovereign identity: a systematic review, mapping and taxonomy. Sensors **22**(15), 5641 (2022)

30. Shopify: How many people use apple pay in us?, https://bit.ly/3Xmtt8l, Accessed 23 November 2022

31. Singh, H.P., Stefanidis, K., Kirstein, F.: A private key recovery scheme using partial knowledge. In: 2021 11th IFIP International Conference on New Technologies, Mobility and Security (NTMS), pp. 1–5. IEEE (2021)

32. Smith-Jackson, T.L.: Cognitive walk-through method (CWM). In: Handbook of Human Factors and Ergonomics Methods, pp. 785–793. CRC Press (2004)

33. Soltani, R., Nguyen, U.T., An, A.: Practical key recovery model for self-sovereign identity based digital wallets. In: 2019 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, pp. 320–325. IEEE (2019)

34. Statista: Crypto ownership among consumers in the united states, by age and gender, https://bit.ly/3Zu0eC Accessed 13 November 2022

35. Stokkink, Q., Ishmaev, G., Epema, D., Pouwelse, J.: A truly SSI system. In: 2021 IEEE 46th Conference on Local Computer Networks (LCN), pp. 1–8. IEEE (2021)

36. Toth, K.C., Anderson-Priddy, A.: Self-sovereign digital identity: a paradigm shift for identity. IEEE Secur. Privacy **17**(3), 17–27 (2019)
37. Trinsic: A full-stack SSI platform, https://trinsic.id/, Accessed 23 November 2022

38. W3C: Decentralized Identifiers (2022), bit.ly/3GXbCj9 Accessed 13 June 2022

39. W3C: Verifiable Credentials (2022), bit.ly/3XpGv5g, Accessed 8 June 2022

40. Wharton, C., Rieman, J., Lewis, C., Polson, P.: The cognitive walkthrough method: A practitioner's guide. In: Usability inspection methods, pp. 105–140 (1994)

41. Wilson, C.: User Interface Inspection Methods. O’Reilly, Morgan Kaufmann (2013)

42. Zaeem, R.N., Khalil, M.M., Lamison, M.R., Pandey, S., Barber, K.S.: On the usability of SSI solutions. Tech. rep. (2021). https://bit.ly/3ivOzCt