Remote Usability Testing in an Academic Library
By: Jung Mee Park
Tags: Internship Experience, Libraries, Usability Testing, UX Research, Academia

Remote Usability Testing in an Academic Library

What remote usability testing can look like. Image source: https://uxpacleveland.org/event-3909977

Leaving behind academic research

Before my summer internship in 2021, to me research meant spending months reading and writing journal articles that require months of peer review. At times, academic research seemed intentionally slow. In my past career, I was postdoctoral scholar and lecturer in sociology, who focused on comparative historical analysis of 19th century East Asian societies. After a few years of teaching, I found myself drawn to graduate studies again. This time in Library and Information Sciences. I liked libraries. And every librarian I met was a totally awesome human being.

As part of my graduate program at the University of Arizona, I had to find an internship. I met a User Experience Librarian, Rebecca Blakiston, virtually through a friend.

UX research is a different beast from academic social science research! I always thought, research is research and in many ways it is, but here is how my summer internship taught me to approach research differently. First at a personal level, transitioning from academic research to UX research meant different expectations and goals for the study. Second, at the global, pandemic level, interactions between researchers and participants that were once in person had to go remote.

Here are a few of my takeaways after successfully conducting remote usability tests with the UX Team at the University of Arizona’s University Libraries.

Setting up the notion.so page to manage the project notes and outcomes.

Embracing UX research

I first had to learn that UX researchers work on a shorter time scale with quicker turnarounds. The shorter timeline also means greater flexibility. Another facet of academic research is its rigidity. In my doctoral program, I was taught that if I wanted to change the wording of a survey, I would have to get the new survey approved through the Institutional Review Board (IRB) or funding agency before I could enact the changes. In UX research, if something was not working, then fix it sooner not later. This was refreshing.

During the summer of 2021, I conducted usability testing on the library’s website. Some of the specific tasks we asked our participants to complete included finding books, articles, and films using the library’s discovery tool called Primo. Usability testing gets users to work through probable scenarios they could encounter on the website. Usability studies can be conducted in-person or remotely. We chose to conduct the tests remotely because the Covid-19 pandemic meant we should be distancing ourselves. Furthermore, the tests were conducted over the summer when fewer students were on campus, so it made sense to do the tests remotely over Zoom to expand our participant pool.

Moderated, remotely moderated versus unmoderated usability testing. image source: https://www.testingtime.com/en/blog/moderated-vs-unmoderated-remote-user-tests/

Usability testing can be either moderated or unmoderated. Moderated tests require more personnel, such as the moderator and notetaker. We chose the moderated option for our remote usability tests.

What worked well

During the pandemic, the advantages of remote testing included health benefits. Even if we conducted our tests outside, the summer heat in Tucson is a safety concern. By providing folks with a Zoom link, we avoided any discussion of Covid-19 safety procedures and vaccination status.

The expanded geographic reach was another benefit. Conducting research over the summer on a college campus had been tough even in non-Covid years. Most students move off campus and return home during the summers. As a result, several participants we recruited for the study were outside the United States during our sessions.

During moderated usability tests, users could ask clarifying questions. We were directing users to search specific terms, so typing them into the chat eliminated confusion. Users also tended not to give up so easily when someone was there encouraging them.

User completes a task and finds the article. Image from Zoom on June 21, 2021.

Now, there are so many great tools that are free such as Calendly, Mailchimp, Notion, and Zoom that can help you facilitate your remote usability testing. We used Notion to manage the project. I created Notion pages to track the documents and findings from the research. Calendly integrates with Zoom to make scheduling simple.

Calendly is a great tool for scheduling. Screenshot of a page participants could use to sign up for a time slot.

Challenges

Recruitment can be a challenge since emails, even those offering gift cards, can go unread or sent to spam. You lose a lot of potential participants this way. For our study, we emailed participants gift cards to the University BookStores which ranged from $5 (for 15 minutes) to $15 (for 30 minutes). This requires you to find more money within your research budget, which is not always feasible. That said, we think the gift card incentive is a big part of why students signed up. The UX team had failed in the past to get participants when no incentive was offered.

Scheduling can be another challenge to moderated testing. You need the participant, moderator, and notetaker to all agree on a time. Unmoderated studies only need the user to find the time. And even when appointments are made, people sometimes forget to log on. So you have to send reminder emails regarding upcoming sessions.

Sometimes people log on but might not have a strong Wi-Fi connection. Remote testing from recruitment onward presumes that users will have access to a phone or computer plus reliable Internet. Researchers cannot control the users’ environment, including Wi-Fi connection, as much in remote sessions.

Another disadvantage I noticed with remote usability testing is that fewer participants are sending back signed media release consent forms. You are not standing right next to them with a free candy bar and a clipboard. People are more willing to sign a piece of paper right then and there.

How we adapted

In total, we conducted 10 usability tests over the summer. We recruited six undergraduate students for the initial study. The testing got underway in late June 2021.

The first three participants (out of the six we planned for the initial study) successfully completed all the tasks quickly. That’s great! But we were not learning anything new. So we decided to add another task which stumped many of our users. Again, that wasn’t our intent, but seeing users struggle helped the library find room for improvement.

Using Notion.so to track of results such as number of attempts, minutes per task, navigation path and user information.

After our study, we learned that most tasks take about 2 and a half minutes to complete. The most challenging task of finding a DVD through WorldCat took over 10 minutes for successful attempts. As a result, we updated the language on the help page to inform the users on additional tips.

Previous version of the library’s help page.

Updated library’s help page to include links to WorldCat and interlibrary loan under the heading of “How can I get better results?” https://new.library.arizona.edu/find/librarysearch

After we updated the help page, we recruited four more users for a quick follow-up study, which focused on one task. The single task involved users finding a film outside of the library’s holding and requesting the DVD through interlibrary loan. We also used this opportunity to redirect users to the updated help page.

The takeaways

After conducting my first UX research project, I learned nuanced points such as rewriting scripts to avoid leading questions or sending email reminders to participants. However, three main takeaways I came away with from the summer can be broadly applied to most research projects.

Plan ahead as much as you can. Write a research plan and script for each of the usability tests. Scripts help you stay on task and avoid leading language. Make sure to ask for help from moderators and note-takers well in advance. Send reminders to the users to reduce the number of no-shows. Know what you can and cannot control. Sometimes emergencies come up and you might have to reschedule a usability test. People’s Wi-Fi might cut out. So it may be a good idea to have one or two extra tests scheduled. Be flexible because even the best laid plans might not work. Plans change, but changes often make your project better. Tasks might be too easy (or too hard). Make modifications so that the study’s findings lead to actionable improvements.

There are several things I would do differently if doing similar studies again. For one, I would send more email reminders, and I would rephrase my scripts to avoid leading questions at the outset. Moving forward, I will keep in mind that all research requires preparation. Some work might become a multi-year process like my prior academic projects. My UX research experiment taught me how to divide up projects into smaller and more manageable units. I learned so much as the research progressed. After every usability test, I had moments of “Oh, I wish I did that differently.” But over time, I learned that I cannot control the past but can choose to learn from it.

Having recently started a new position as a fully-remote data analyst at the University of Arizona, I’ve benefited from my experience of a fully-remote internship. I can now work effectively on a remote team, pace myself when the workload seems overwhelming, and be adaptable and flexible whatever comes my way.