Comparative Survey of Blind, Low-Vision
and Sighted Workers on Crowdsourcing
Platforms
Ying Zhong, Makoto Kobayashi, Masaki Matsubara and Atsuyuki
Morishima
University of Tsukuba, Tsukuba University of Technology
zhong.ying.wp@alumni.tsukuba.ac.jp, koba@cs.k.tsukuba-tech.ac.jp,
masaki@slis.tsukuba.ac.jp, morishima-office@ml.cc.tsukuba.ac.jp
Abstract
Crowdsourcing offers a new form of work that makes it easy for people to workfrom home on their computers. The completion of tasks, however, can be difficult for those
who have certain disabilities, such as visual impairment. There is a lack of research oncomparing visually impaired and sighted crowd workers on non-English crowdsourcing
platforms. This research aims to determine the best ways to support visually impaired
workers with AI and change the modality to improve the performance of visually impaired
crowd workers, especially screen reader users. This study used a questionnaire with 10
visually impaired and 20 sighted crowd workers to investigate the problems encountered byvisually impaired people on Japanese crowdsourcing platforms. The key findings are as
follows. (1) The number of task types for visually impaired workers is limited compared tothe types of tasks finished by sighted workers on crowdsourcing platforms. (2) Moreaccessible platforms and tasks for screen readers are crucial for visually impaired crowd
workers. (3) The time taken by visually impaired crowd workers to perform tasks is a
crucial problem. We also include suggestions for future work as part of this study andprovide future directions.

Keywords
Crowdsourcing, Visually-Impaired Workers, Workers with Disabilities, Labor
Platforms, Online Work, Remote Work, Gig Work.

Journal on Technology and Persons with Disabilities
Robles, A.: CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

138

Introduction
Crowdsourcing offers a new form of work that makes it easy for people to work fromhome on their computers. Crowdsourcing platforms are emerging in increasing numbers. These
crowdsourcing platforms provide functionalities to match tasks with workers.
Crowd workers can choose the tasks they can perform on the crowdsourcing platform.
Performing online tasks can however be problematic, for some people with disabilities, such as
visually impaired people (Bourne et al., Zyskowski et al.).
In this study, an online survey was conducted to investigate the current status ofcrowdsourcing platforms and the problems encountered by visually impaired workers when
performing these tasks. We also compared their working conditions with those of sighted crowd
workers. A possible solution to increase the number of potential workers, including visually
impaired workers, who can obtain new jobs in a crowdsourcing platform, has been provided
herein.
The key findings of this paper are as follows:• The number of task types for visually impaired workers is limited compared to thetypes of tasks finished by sighted workers on crowdsourcing platforms. This suggeststhat it will be beneficial to develop methods that can transform the tasks into thosemore accessible to visually impaired workers.
• Most visually impaired crowd workers are screen reader users. This suggests thatvisually impaired crowd workers need more accessible platforms and tasks for screen
readers.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

139

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms• The time taken by visually impaired crowd workers to perform tasks is a crucial

problem. This suggests the importance of the development of technologies for reducing
the time required to complete tasks.
Related Work
Online Microtasks
Online microtasks are tasks such as categorizing images and drafting articles on
crowdsourcing platforms, such as Amazon Mechanical Turk (AMT). Most online tasks are visualand can be run on a personal computer.
Luz et al. provided some types of online tasks including categorization, classification,
data verification, data extraction, moderation and tagging of multimedia content, transcription
from multimedia content, sentiment analysis and surveys and search relevance.
A few researchers have focused on task design and user interface to enhance the worker
performance. Sampath et al. claimed that using cognitively inspired features for task design is a
powerful technique to maximize the performance of crowd workers. Rahmanian et al. studied theeffects of different user interface designs on the performance of crowdsourcing systems. Manino
et al. provided the first theoretical explanation of the accuracy gap between the most popularcollection policies.
Calvo et al. evaluated the accessibility of crowdsourcing tasks on Amazon Mechanical
Turk and found some problems for visually impaired crowd workers, such as pages that lacked
sufficient metadata describing the page language, which could cause a screen reader to havedifficulty pronouncing page text.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

140

Visually Impaired Crowd Workers
BSpeak is an accessible crowdsourcing marketplace that enables the visually impaired indeveloping regions to earn money by transcribing audio files containing speech (Vashistha etal.). Zyskowski et al. presented the first formal study of crowd workers with disabilities via indepth open-ended interviews of 17 people (disabled crowd workers and job coaches for people
with disabilities) and a survey of 631 adults with disabilities. They found several accessibility
and usability barriers such as inaccessible task features, inability to find accessible tasks, andtime restrictions to complete tasks, limiting the adoption of crowdsourcing marketplaces by blind
people.
Research Method
Participants
An online survey was conducted on crowdsourcing platforms for visually impaired andsighted crowd workers named SunnyBank and Crowdworks, which are the main crowdsourcing
platforms in Japan. The online survey was approved by the IRB at the Faculty of Library,
Information and Media Science at the University of Tsukuba. (approval number: 22-23)
A total of 10 visually impaired and 20 sighted crowd workers were recruited. Eachvisually impaired crowd worker received compensation of 3,000 Japanese yen for participatingin the survey. Each sighted crowd worker received compensation of 500 Japanese yen forparticipating in the survey. The reward for visually impaired crowd workers was higher because
the platform needed more commission to check the visually impaired crowd workers’
information, and visually impaired crowd workers were required to answer more questions.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

141

Questions
The survey contained 31 and 26 questions for the visually impaired crowd workers andsighted crowd workers, respectively (See Table 1). The questions were related to the following
topics: demographics, visual impairment, use of the crowdsourcing platform, and accessibility
problems.
Questions on demographics: They included questions on age, gender, education level,
and working conditions.
Table 1: Table of Questions.
Question
Number (Q)Question

Q1

Age

Q2

GenderQ3

If you don’t mind, please tell us what kind of visual impairments you have?

Q4

Select letter(s) you can read. (multiple answers is ok.)Q5

If you select magnified printed letters in Q4, please tell us how you zoom in?

Q6

If you select magnified printed letters in Q4, please tell us what is thepreferable number of points?

Q7

Do you use a screen reader?

Q8

If you choose ’Yes’ in Q7, please tell us which screen reader you use?

Q9

What is your final education?

Q10

Do you have a regular (daily) job?

Q11

Do you have another job except for crowd-working or a regular job?

Q12
Q13
Q14

What kind of device are you using when you take part in Amazon
Mechanical Turk?
Which crowdsourcing platform did you use other than Amazon MechanicalTurk?
Why do you choose these crowdsourcing platforms?

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

Question
Number (Q)142

Question

Q15

How did you know the existence of Amazon Mechanical Turk?

Q16

If you choose ’others’ in Q15, please write in detail.

Q17

How did you learn these crowdsourcing platforms?

Q18

If you choose ’others’ in Q17, please write in detail.

Q19

How long did you use a crowdsourcing platform including other platforms(total experience)?

Q20

How often do you finish tasks in a crowdsourcing platform?

Q21

Please write the details about Q20.

Q22

On average, how long do you do crowdsourcing tasks?

Q23

Please write the details about Q22.

Q24

How many tasks did you finish in a crowdsourcing platform including otherplatforms?

Q25

What kind of tasks did you choose (you could finish)?

Q26

Why do you choose these tasks?

Q27

How did you find the tasks?

Q28

If you write ’searching keywords’ in Q27, please tell us what kind of
keywords you searched for?

Q29

What kind of functionality do you expect in a crowdsourcing platform?

Q30

What problems do you have when you are doing the tasks?

Q31

Do you have any suggestions for these problems?

Questions of visual impairment: They include questions on the type of visual impairment,
letters they can read, and the use of screen readers. This part is only for visually impaired
workers.
Questions of use of the crowdsourcing platform: They included questions on the reasons
why they chose a crowdsourcing platform and their experience on crowdsourcing platforms.
Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

143

Questions of accessibility problems: They included questions on the method of searchingfor accessibility tasks, the problems that they overcome, a function that they hope for, andsuggestions.
Results
DemographicsAge
The visually impaired and sighted participants were aged between 10 and 59 years, withmost people between the ages of 40 and 59 for visually impaired crowd workers. There are no
participants above the age of 60. Thus, it can be concluded that the visually impaired crowd
workers belonged to the “middle age category” (See Table 2).
Table 2. Age
Visually Impaired
Workers

SightedWorkers

10-19 years old

0%0%20-29 years old

20%

20%

30-39 years old

10%

40%

40-49 years old

40%

35%

50-59 years old

0%5%more than 60 years
old

0%

0%

Age range

GenderOn SunnyBank, the number of male and female participants was approximately the same.
On Crowdworks, there were more male participants than females (See Table 3).

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

144

Table 3. Gender
GenderMale
FemaleVisually
ImpairedWorkers

SightedWorkers

50%

60%

50%

40%

Education Level
Table 4 summarizes the education level of the visually impaired and sighted participants.
Approximately 70% of the visually impaired participants and 80% of the sighted participants had
a university degree at least. Thus, we can conclude that the majority of visually impaired
participants were educated, which implies that they possess the ability to perform tasks on
crowdsourcing platforms.
Table 4. Education Level
Visually Impaired
Workers

SightedWorkers

High school

30%

20%

University

50%

75%

Graduated school

10%

5%

Other

0%

0%

Education Level

Working Conditions
Table 5 details the working conditions of the visually impaired and sighted crowd
workers and shows whether visually impaired and sighted crowd workers have regular jobs.
Approximately 50% of the visually impaired and 70% of the sighted crowd workers have regularjobs.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

145

Table 5. Working Style
Visually Impaired
Workers

SightedWorkers

Having regular work

50%

70%

Having no regular
work

50%

30%

Working Style

Visual Impairment
Types of Visual Impairment
All visually impaired participants had a disability certification and all were at level 3 orhigher, with vision below 0.07. 80% of the participants were at level 1, with vision below 0.01.
Letters
A total of 80% of the participants could read the braille script, 30% could read magnified
printed letters, and 10% could read printed letters. Methods for magnification included lens,magnifying via app or screen, and glasses.
Use of Screen Readers
There were nine screen-reader users. Various screen readers included those by NVDA,
PC-Talker, Voice-Over and JAWS.
Use of the Crowdsourcing Platform
Devices
Table 6 indicates that approximately 60% of the visually impaired participants and 95%of the sighted participants used PCs to perform tasks on crowdsourcing platforms.
Table 6. The devices visually impaired and sighted crowd workers use.
DevicePC
TabletVisually Impaired
Workers
60%
10%

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Sighted Workers
95%
5%Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

DeviceSmartphone

Visually Impaired
Workers30%

146

Sighted Workers
0%

Crowdsourcing Platforms
Furthermore, we investigated other crowdsourcing platforms apart from Sunnybank andCrowdworks. The visually impaired crowd workers also used Crowdworks, Sagoworks andFreelancer. The sighted crowd workers also used Yahoo! Crowdsourcing, Lancers andZaitakuworks. We discovered there are limited crowdsourcing platforms in Japan. Table 7
summarizes how the participants discovered the crowdsourcing platforms. Table 8 summarizes
how the visually impaired and sighted participants learned to use these crowdsourcing platforms.
Table 7. The way visually impaired and sighted crowd workers know the existence of
crowdsourcing platforms
SourceFrom other people
Searching by yourself
OthersVisually Impaired
Workers
80%
10%
10%

Sighted Workers
10%
90%
0%

Table 8. The way how visually impaired crowd workers learn these crowdsourcing
platforms
SourceFrom other people
Searching by yourself

Visually Impaired
Workers20%
80%

Sighted Workers
5%95%

ReasonWe investigated the reasons why the participants selected a specific crowdsourcing
platform. For visually impaired crowd workers, there were three main criteria: the platform, thereasons included consideration for people with disabilities and simplicity of operation. For the
Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

147

task, the reasons included finding a task that they could do and working with peace of mind evenin the current situation, including disabilities and physical conditions. For others, the reason wasthat more friends used the platform.
For sighted crowd workers, the same three criteria were used: platform, task, and others.
For the platform, the reasons included lack of time limit, lots of tasks, matching tasks, major
companies and relatively high rewards. For the task, the reasons included the numerous tasks forfemales. For others, the reasons included being introduced from blogs, YouTube and friends.
Time
Table 9 lists the working hours of the participants per day; 80% of the visually impaired
crowd workers worked 1h to 5h per day. 20% of the visually impaired crowd workers worked
less than 1h per day. 75% of sighted crowd workers worked 1h to 5h per day. 25% of the sighted
crowd workers worked less than 1h per day. We discover that visually impaired crowd workers
work longer hours than sighted crowd workers.
Table 9. Working hours of visually impaired and sighted crowd workers per day
Visually Impaired
Workers

Sighted Workers

Less than 1 hour per day

20%

25%

1-5 hours per day

80%

75%

Frequency

Frequency
Table 10 lists the number of tasks performed by the crowd workers on the crowdsourcing
platforms per month; 60% of visually impaired crowd workers finished 1 to 5 tasks per monthand 40% of visually impaired crowd workers finished less than 1 task per month. The reason isthat visually impaired workers cannot perform too many tasks. 85% of sighted crowd workers
finished more than 10 tasks per month. Six sighted crowd workers finished more than 30 tasks
Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

148

per month. We determine that visually impaired crowd workers finish much fewer tasks thansighted crowd workers.
Table 10: The frequency visually impaired crowd workers finish tasks in a crowdsourcing
platform
Visually Impaired
Workers

Sighted Workers

Less than 1 task per month

40%

0%

1-5 tasks per month

60%

5%

5-10 tasks per month

0%

10%

More than 10 tasks per month

0%

85%

Frequency

Tasks
For the number of tasks performed by the crowd workers in crowdsourcing platforms
including other platforms; 90% of the participants finished less than 20 tasks on thecrowdsourcing platforms. These tasks are mainly related to web accessibility tests, surveys,
writing, etc.
95% of the sighted crowd workers finished less than 50 tasks in the crowdsourcing
platforms. These tasks are mainly related to survey, writing, data input, interview, report ofsearch results, experiments, investigations, etc. We observed that visually impaired crowd
workers completed fewer tasks than sighted workers.
Accessibility Problems
The Method of Searching for Accessible Tasks
Methods used by the visually impaired participants for finding tasks included searching,looking for the list on the homepage of crowdsourcing platforms, and recommendations fromothers. The keywords used by them to search for a task on a crowdsourcing platform included
Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

149

teleworking, disability, simple, beginner, side job, accessibility, writing, questionnaire, point,
article and translation. It can be concluded that the participants focused on the type and content
of tasks.
Problems
70% of the visually impaired crowd workers mentioned no problems with finishing tasks.
However, 40% of the visually impaired participants mentioned almost no suitable tasks for them.
Visually impaired crowd workers work longer hours than sighted crowd workers.
Functionalities and Suggestion
The participants desired functionalities such as pausing the time, functionality expansion,
screen-reader features, easy understanding of the content, typing error checks, and task
recommendation by AI. Other suggestions included more detailed descriptions and simplifying
the tasks available.
Discussion and Conclusion
In this study, the survey was conducted in Japanese on crowdsourcing platforms. Most of
the Japanese visually impaired crowd workers only performed tasks such as taking surveys,
writing articles and accessibility tests. The survey results suggest several ways for using AI to
support visually impaired workers. For example, most tasks are simple text-based tasks. Morecomplex text tasks, such as entity-matching tasks (Zhong et al.), and more media-type tasks, suchas image tasks (Zhong et al.), can be extended.
Additionally, time is a significant factor to consider. Visually impaired crowd workers
spend more time than sighted crowd workers. Reducing the working time is an important issue.
The completion time of visually impaired crowd workers can be reduced using several methods.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

150

Zhong et al. suggested that the completion time can be reduced by removing unimportant images
from the tasks that use web images and consistency-conscious design of tabular tasks.
The proposals for the future direction are as follows:• We can support visually impaired workers with AI. For example, most tasks are simple

text-based tasks. More complex text tasks, such as entity-matching tasks, and more
media-type tasks, such as image tasks, can be included.
• We can extend the time required for task completion for visually impaired crowd

workers and provide the estimated completion time.
• We can reduce the completion time of visually impaired crowd workers. For example,

completion time can be reduced by removing unimportant images on the task whichuses web images and consistency-conscious design of tabular tasks.
• We can change the modality (e.g., visual information to auditory information, text tospeech technology, alt text generation) to improve the performance of visually
impaired crowd workers, especially screen reader users.
Acknowledgments
This work was partially supported by JSPS KAKENHI Grant Number 22H00508 and
21H03552, Japan. This work was approved by IRB of University of Tsukuba.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

151

Works Cited
Bourne, R., Price, H., Stevens, G. Global burden of visual impairment and blindness. Arch
Ophthalmol, 2012.
Calvo, R., Kane, S. K., Hurst, A. Evaluating the accessibility of crowdsourcing tasks on
Amazon’s mechanical turk. Proceedings of the 16th international ACM SIGACCESS
conference on computers accessibility, pp. 257–258, 2014.
Luz, N., Silva, N., Novais, P. A survey of task-oriented crowdsourcing Artificial Intelligence
Review, no. 44, pp. 187–213, 2015.
Manino, E., Tran-Thanh, L., Jennings, N. On the efficiency of data collection for crowdsourced
classification. Proceedings of the Twenty-Seventh International Joint Conference on
Artificial Intelligence (IJCAI-18), pp. 1568-1575, 2018.
Rahmanian, B., Davis, J. G. User interface design for crowdsourcing systems. Proceedings of the2014 International Working Conference on Advanced Visual Interfaces (AVI 2014), pp.
405-408, 2014.
Sampath, H. A., Rajeshuni, R., Indurkhya, B. Cognitively inspired task design to improve user
performance on crowdsourcing platforms. International Conference for Human-computer
Interaction (CHI2014), pp. 3665-3674, 2014.
Vashistha, A., Sethi, P., Anderson, R. Bspeak: An accessible crowdsourcing marketplace forlow-income blind people. International Conference for Human-computer Interaction (CHI2018), 2018.
Zhong, Y., Kobayashi, M., Matsubara, M., Morishima, A. Effects of crowd-in-the-loop alt text
addition on the performance of visually impaired workers in online microtasks. The
Transaction of Human Interface Society (2020), vol. 20, pp 251-262, 2020.
Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge

Comparative Survey of Blind, Low-Vision and Sighted Workers on Crowdsourcing Platforms

152

Zhong, Y., Kobayashi, M., Matsubara, M., Morishima, A. Usability of audio matrix microtasks
for screen reader users. The Journal on Technology and Persons with Disabilities (2021),
vol. 9, pp 54-70, 2021.
Zhong, Y., Matsubara, M., Kobayashi, M., Morishima, A. Understandable microtasks with novisual representation. The Tenth AAAI Conference on Human Computation and
Crowdsourcing (HCOMP 2019), 2019.
Zyskowski, K., Morris, M., Bigham, J., Gray, M., Kane, S. Accessible crowdwork?
understanding the value in and challenge of microtask employment for people withdisabilities. Proceedings of the 18th ACM Conference on Computer-Supported
Cooperative Work Social Computing (CSCW2015), pp. 1682-1693, 2015.

Journal on Technology and Persons with Disabilities
Robles, A. (Eds): CSUN Assistive Technology Conference© 2023 California State University, Northridge