Proceedings of the 2016

International Conference on

Crowd Science and Engineering
25-30 July 2016
Vancouver, Canada
Contact: ICCSE2016@crowdscience.org

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Table of Contents
Introduction ............................................................................................................................................ 2
International Advisory Committee ......................................................................................................... 3
Organizing Committee ............................................................................................................................ 4
Program Committee ............................................................................................................................... 6
Conference Program (25th-27th July 2016) .............................................................................................. 7
Conference Program (28th-30th July 2016) .............................................................................................. 8
Keynotes.................................................................................................................................................. 9
Tutorials ................................................................................................................................................ 12Awards .................................................................................................................................................. 15
Accepted Papers ................................................................................................................................... 16
Demo/Posters ..................................................................................................................................... 196© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

1

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Introduction
The 2016 International Conference on Crowd Science and Engineering (ICCSE 2016) offers a wonderful
opportunity for academics, industry practitioners and policy makers to gather and explore thetransformative potential of crowd science research, with an aim to find ways to engineer efficient systems
which can combine the strengths of both humans and machines for new innovative possibilities. ICCSE
2016 provides a global forum for scientists, engineers and educators to present and discuss the latest
Crowd Science and Engineering research, emerging technologies and their future applications. ICCSE
2016 has received 100 paper submissions from 12 countries. Among the 100 submissions, 27 papers havebeen selected as full paper to be presented at the conference.
The ICCSE 2016 program includes 3 keynote sessions given by world renowned speakers such as Kevin
Leyton-Brown, Andrew Mao and Chunyan Miao; 2 workshop sessions on Crowd Wisdom for Healthy
Living and Life-Long Learning conducted jointly by Nanyang Technological University and Peking
University, 3 tutorials given by experts in their respective fields, 4 technical sessions on emerging
research areas, 4 special sessions on Crowd Decision Making and Crowd Computing & Human
Computation for Ageless Aging, and a number of Poster/Demo/Exhibition sessions.
ICCSE 2016 would like to thank the authors in our and nearby communities for submitting a large
number of top quality papers and the PC members, Session Chairs, International Advisory Committee
and Organizing Committee members for their great contributions to ensure the success of the conference.
ICCSE 2016 also thanks The University of British Columbia for providing the conference venue.

ICCSE 2016 General Chairs
Cyril Leung (The University of British Columbia, Canada)
Chunyan Miao (Nanyang Technological University, Singapore)
Chengqi Zhang (University of Technology Sydney, Australia)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

2

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

International Advisory Committee

Prof Rabab Ward

Prof Victor Lesser

The University of British Columbia,
CanadaUniversity of Massachusetts Amherst,
USA

Prof Yueting Chai

Prof Qiang Yang

Tsinghua University,
China

Hong Kong University of Science andTechnology, Hong Kong

Prof Xiaoming Li

Prof Ah-Hwee Tan

Peking University,
China

Nanyang Technological University,
Singapore© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

3

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Organizing Committee
General Chairs
Cyril Leung, The University of British Columbia, Canada
Chunyan Miao, Nanyang Technological University, Singapore
Chengqi Zhang, University of Technology Sydney, Australia

Programme Committee (PC) Chair
Bo An, Nanyang Technological University, Singapore

PC Vice-Chairs
Dong Xu, University of Sydney, Australia
Takayuki Ito, Nagoya Institute of Technology, Japan
Yiqiang Chen, University of Chinese Academy of Sciences, China
Quan Bai, Auckland University of Technology, New ZealandFinance Chairs
Jane Wang, The University of British Columbia, Canada
Liang Zhang, Nanyang Technological University, Singapore

Demo/Poster Chairs
Richard Smith, Simon Fraser University, Canada
Zhiqi Shen, Nanyang Technological University, Singapore

Workshop Chairs
Di Wang, Nanyang Technological University, Singapore
Simon Fauvel, Nanyang Technological University, Singapore© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

4

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Organizing Committee
Tutorial Chair
Jane Wang, The University of British Columbia, Canada

Publicity Chairs
Jun Lin, Nanyang Technological University, Singapore
Lizhen Cui, Shandong University, China

Publication Chairs
Baowei Sun, Central University of Finance and Economics, China
Minting Huang, Nanyang Technological University, Singapore

Webmaster, Registration, enquiries & special requests
Jun Lin, Nanyang Technological University, Singapore

Awards Committee Chairs
Thomas W. Calvert, Simon Fraser University, Canada
Yonggang Wen, Nanyang Technological University, Singapore© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

5

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Program Committee
Bo An, Nanyang Technological University, Singapore and Chinese Academy of Sciences, China
Yueting Chai, Tsinghua University, China
Yiqiang Chen, University of Chinese Academy of Sciences, China
Chengqi Zhang, University of Technology Sydney, Australia
Lizhen Cui, Shandong University, China
Daqing Zhang, Institut Mines-Télécom/Télécom SudPais, France
Han Yu, Nanyang Technological University, Singapore
Haoming Zhong, WeBank Tencent, China
Joy Zhang, Google, USA
Jun Lin, Nanyang Technological University, Singapore
Qingzhong Li, Shandong University, China
Xiaoming Li, Peking University, China
Yinsheng Li, Fudan University, China
Martin McKeown, University of British Columbia, Canada
Minjie Zhang, University of Wollongong, Australia
Rabab Ward, The University of British Columbia, Canada
Roger Jiao, Georgia Institute of Technology, USA
Simon Fauvel, The University of British Columbia, Canada
Siyuan Liu, Nanyang Technological University, Singapore
Baowen Sun, Central University of Finance and Economics, China
Tiong Chee Tay, Dell Singapore
Tom Calvert, Simon Fraser University, Canada
Wei-Tek Tsai, Arizona State University, USA
Xi Jessie Yang, Massachusetts Institute of Technology, USA
Xiuyi Fan, Nanyang Technological University, Singapore
Chenglei Yang, Shandong University, China
Yuan Miao, Victoria University, Australia
Yuting Zheng, Baidu, China
Zhen Hai, Institute for Infocomm Research, A*Star, Singapore
Zhiqi Shen, Nanyang Technological University, Singapore© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

6

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Conference Program (25th-27th July 2016)
Monday, July 25, 2016 (MCLD 214, Macleod Building, UBC)
9:00 AM-17:45 PM
15:00 PM-16:30 PM
16:30 PM-17:00 PM
17:00 PM-17:45 PM

Registration (MCLD 214)
Tutorial I and II
Tea BreakTutorial III

Tuesday, July 26, 2016 (KAIS 5530, Kaiser Building, UBC)
9:00 AM-16:30 PM
10:00 AM-15:00 PM
17:30 PM-21:00 PM

Registration (KAIS Lobby)
UBC Campus Visit
Welcome Reception (KAIS 5530)

Wednesday, July 27, 2016 (ESB 1013, Earth Sciences Building, UBC)
9:00 AM-10:00 AM
10:00 AM-10:30 AM
10:30 AM-12:00 PM
12:00 PM-13:00 PM

Welcome Speech/Opening Address
Tea BreakTechnical Sessions (Full/Short)
Poster/Demo/Exhibition

Lunch Timein conjunction with lunch session

13:00 PM-14:30 PM
14:30 PM-15:00 PM
15:00 PM-16:30 PM
Evening

Joint NTU-PKU Workshop on Crowd Wisdom for Healthy Living
and Life-Long Learning (Session I)
Tea Break
Joint NTU-PKU Workshop on Crowd Wisdom for Healthy Living
and Life-Long Learning (Session II)
Vancouver Summer Fireworks (free and easy)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

7

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Conference Program (28th-30th July 2016)
Thursday, July 28, 2016 (ESB 1013, Earth Sciences Building, UBC)
9:00 AM-10:00 AM
10:00 AM-10:30 AM
10:30 AM-12:00 PM
12:00 PM-13:00 PM

Keynote (Chunyan Miao)
Tea BreakTechnical Sessions (Full/Short)
Poster/Demo/Exhibition

Lunch Timein conjunction with lunch session

13:00 PM-14:30 PM
14:30 PM-15:00 PM
15:00 PM-16:30 PM
18:30 PM-21:30 PM

Special Session on Crowd Decision Making (Session I)
Tea BreakSpecial Session on Crowd Decision Making (Session II)
Award Banquet

Friday, July 29, 2016 (ESB 1013, Earth Sciences Building, UBC)
9:00 AM-10:00 AM
10:00 AM-10:30 AM
10:30 AM-12:00 PM
12:00 PM-13:00 PM

Keynote (Andrew Mao)
Tea BreakTechnical Sessions (Full/Short)
Poster/Demo/Exhibition

Lunch Timein conjunction with lunch session

13:00 PM-14:30 PM
14:30 PM-15:00 PM
15:00 PM-16:30 PM

Special Session on Crowd Computing and Human Computation forAgeless Aging (Session I)
Tea BreakSpecial Session on Crowd Computing and Human Computation forAgeless Aging (Session II)

Saturday, July 30, 2016 (ESB 1013, Earth Sciences Building, UBC)
9:00 AM-10:00 AM
10:00 AM-10:30 AM
10:30 AM-12:00 PM
12:00 PM-13:00 PM

Keynote (Kevin Leyton-Brown)
Tea BreakTechnical Sessions (Full/Short)
Conference Room

Lunch Time

Poster/Demo/Exhibition in conjunction with lunch session

13:00 PM-16:30 PM
Evening

Open Networking/Discussion
Vancouver Summer Fireworks (free and easy)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

8

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Keynotes
Chunyan Miao
Director, NTU-UBC Research Centre of Excellence in Active Living for the
Elderly & Professor, School of Computer Science and Engineering, Nanyang
Technological University, Singapore

Title: Unobtrusive Crowd Sensing and Gamification forAgeless Ageing
Thursday, July 28, 2016
9:00 AM - 10:00 AM

Abstract:
In recent years, interactive digital games are starting to move from mainly relying on the mouse andkeyboard for user inputs towards using body movements as inputs. Examples of such game platforms
include the Nintendo Wii, the Microsoft Kinect, and the Leap Motion device. These devices have changed
the nature of computer games to make it possible for people who are not technically savvy, such as seniorcitizens, to enjoy the game play experience. Researchers are starting to leverage body gesture-based
games to unobtrusively study the progression of medical conditions and long term individual wellness
trends. Many interesting results have emerged.
In this presentation, I will share with the audience on some of the methods, results and lessons learnt fromour research in wellness games and aging-in-place IoT technologies for senior citizens. I will draw upon
the research experience from the Joint NTU-UBC Research Centre of Excellence in Active Living for the
Elderly (LILY), Singapore, to illustrate how technologies could help to advance the field of personalized
healthcare at a population scale.
LILY focuses on applying research advances in Artificial Intelligence, Human Computation, Data
Analytics and Digital Media for helping senior citizens. An impactful example of LILY’s innovative
research which exploits the combined power of AI, human wisdom and data analytics is an engaging
game for health and wellness – Pumpkin Garden. It is a population level screening and rehabilitation
game for neurodegenerative diseases such as Parkinson and dementia. In this game, data are continuously
collected and analyzed to provide health professionals with insights into the progress of their patients. I
will also introduce one of our latest works in using motor patterns for stroke detection and rehabilitations.

Biography:
Dr Chunyan Miao is an Associate Professor in the School of Computer Science and Engineering,
Nanyang Technological University and the Director of the Joint NTU-UBC Research Centre of
Excellence in Active Living for the Elderly (LILY) co-initiated by IDMPO/NRF, MDA, NTU and UBC.
Over the last ten years, she has successfully led a number of national research projects in AI, intelligent
interactive digital media, smart health, data analytics, aging in place, ageless aging design & technologies
with funding from both government agencies and industry.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

9

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

KeynotesAndrew Mao
Researcher, the Computational Social Science group, Microsoft Research

Title: Studying Teamwork and Cooperation in the
Virtual Lab
Friday, July 29, 2016
9:00 AM - 10:00 AM

Abstract:
For decades, physical behavioral labs have been a primary, yet limited, method for controlled
experimental studies of human behavior. Now, software-based "virtual labs" on the Internet allow forstudies of increasing complexity, size, and scope. In this talk, I highlight the potential of virtual lab
experiments for studying social interaction and coordination. First, we explore collective intelligence anddigital teamwork in "crisis mapping", where digital volunteers organize to assess and pinpoint damage in
the aftermath of humanitarian crises. By simulating a crisis mapping scenario to study self-organization inteams of varying size, and find a tradeoff between individual effort in small groups and collective
coordination in larger teams. We also conduct a study of cooperation in a social dilemma over a month ofreal time, using crowdsourcing participants to overcome the time constraints of behavioral labs. Our studyof about 100 participants over 20 consecutive weekdays finds that a group of resilient altruists sustain ahigh level of cooperation across the entire population. Together, our work motivates the potential ofcontrolled, highly instrumented studies of social interaction; the importance of behavioral experiments onlonger timescales; and how open-source software both can speed up the iteration and improve thereproducibility of experimental work.
* based on joint work with Lili Dworkin, Winter Mason, Siddharth Suri, and Duncan Watts.

Biography:
Andrew Mao is currently a postdoctoral researcher in Computational Social Science at Microsoft
Research in NYC. He received his PhD from Harvard University in 2015. His research focuses on
studying collective intelligence and social interaction on the Internet, such as teamwork in online
communities and coordination in crowdsourcing systems. Andrew specializes in designing and gathering
data from real-time, interactive, web-based behavioral experiments, and he is the designer of TurkServer(http://turkserver.readthedocs.io/), an open-source platform for building such experiments. He received
the Yahoo! Key Scientific Challenges award in 2011.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

10

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Keynotes
Kevin Leyton-Brown
Professor, Computer Science, University of British Columbia,
CanadaTitle: Peer Grading: From Theory to PracticeSaturday, July 30, 2016
9:00 AM - 10:00 AM

Abstract:
This talk describes a research effort on the peer grading problem, running the gamut from theory topractice. First, I'll describe a theoretical investigation of different approaches to incentivizing students toinvest the effort to perform thoughtful and honest peer evaluations. One promising approach is to draw on
the large literature on so-called "peer prediction". The mechanisms proposed in this literature can
incentivize truth telling in equilibrium, but also give rise to equilibria in which agents do not pay the costs
required to evaluate accurately, and hence fail to elicit useful information. We show that this problem isunavoidable whenever agents are able to coordinate using low-cost signals about the items being
evaluated (e.g., superficially glancing at the text to be graded, focusing on grammar and formatting).
The second part of the talk will describe our experiences with using this simpler mechanism in practice.
No existing peer review software made it possible to leverage TA grades, so we built our own, which we
called "Mechanical TA" and published as open-source software. In this system, human TAs both evaluate
the peer reviews of students who have not yet demonstrated reviewing proficiency and spot check thereviews of students who have. Mechanical TA also features "calibration" reviews, allowing students toquickly gain experience with the peer-review process. We used Mechanical TA for three years to run
weekly essay assignments in a class of about 70 students, a course design that would have been
impossible if every assignment had had to be graded by a TA. We show evidence that Mechanical TA
helped to support student learning, leading us to believe that it may also be useful to others.

Biography:
Kevin Leyton-Brown is a professor of computer science at The University of British Columbia. He holds
a PhD and M.Sc. from Stanford University (2003; 2001) and a B.Sc. from McMaster University (1998).
He studies the intersection of computer science and microeconomics, addressing computational problems
in economic contexts and incentive issues in multiagent systems. He also applies machine learning to theautomated design and analysis of algorithms for solving hard computational problems.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

11

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Tutorials
Tutorial I
Title: Incentives and Mechanism Design for Human Computation Systems
Date and Time: Monday, 25 July 2016, 15:00pm to 15:45pm
Location: MCLD 214, Macleod Building, 2356 Main Mall, UBC
Presenter: Simon Fauvel, NTU LILY Research Centre

Abstract:
Human computation systems (HCSs) have gained tremendous popularity in the last decade. Their primary
purpose is to offer an ecosystem that harnesses human intelligence to solve computational problems thatmodern computers cannot solve alone. One of the main challenges in ensuring the success and sustainable
operation of HCSs is how to incentivize a broad range of users to participate in the system and provide
their best efforts. In this tutorial, we will discuss HCSs from the perspective of incentives and mechanism
design. We will take a look at the different kinds of HCSs, focusing on how incentives are provided tousers in these systems. We will then explore mechanisms derived from state-of-the-art HCSs as well asclassic mechanisms that have been successfully deployed in HCSs. We will also explore how game theory
tools can be used to theoretically analyse the strength of different incentives. Finally, we will outline afew potential research directions in this field.

Presenter Bio:
Simon Fauvel is an electrical and computer engineer from Montréal, Canada. He obtained his B. Eng.
from McGill University, Montréal, Canada and his MASc from The University of British Columbia,
Vancouver, Canada. He is currently a research engineer in the Joint NTU-UBC Research Centre of
Excellence in Active Living for the Elderly (LILY), Singapore. His broad research interests are in thefields of signal processing, artificial intelligence and data mining for learning at scale and ageless aging.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

12

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Tutorials
Tutorial II
Title: Deep Learning Meets Signal Processing
Date and Time: Monday, July 25, 15:45pm to 16:30pm
Location: MCLD 214, Macleod Building, 2356 Main Mall, UBC
Presenter: Angshul Majumdar, IIIT-Delhi

Abstract:
Deep learning has recently attracted a lot of attention from academia and industry. It has been highly
successful in applications ranging from speech processing to computer vision to information retrieval.
Dictionary learning on the other hand has been successful in solving inverse problems (arising in signal
processing) and learning problems (arising in computer vision). In general, both deep learning anddictionary learning fall under the broader purview of representation learning. In this talk, we briefly
review these two topics; we will see how they are connected. Building on these connections, we willshow how deeper architectures can be built from dictionary learning.
Presenter Bio:
Angshul Majumdar did his Master's and Ph.D. at The University of British Columbia in 2009 and 2012
respectively. He did his Bachelor's from Bengal Engineering College, Shibpur. Currently he is anassistant professor at Indraprastha Institute of Information Technology, Delhi. His research interests arebroadly in the areas of signal processing and machine learning. He has co-authored over 120 papers injournals and reputed conferences. He is the author of Compressed Sensing for Magnetic Resonance Image
Reconstruction published by Cambridge University Press and co-editor of MRI: Physics, Reconstruction
and Analysis published by CRC Press. He is currently serving as the chair of the IEEE SPS Chapter'scommittee and the chair of the IEEE SPS Delhi Chapter.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

13

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Tutorials
Tutorial III
Title: Wearable Computing for e-Health
Date and Time: Monday, July 25, 17:00pm to 17:45pm
Location: MCLD 214, Macleod Building, 2356 Main Mall, UBC
Presenter: Yiqiang Chen, Chinese Academy of Sciences

Abstract:E-health is one of the most life critical application domains for wearable computing. It is going to be a
new frontier of research and development which brings a lot of challenges in unobtrusive sensing, highaccuracy wearable recognition model and cognitive e-Health applications. In this talk, the main
challenges in wearable computing technologies for e-Health will be analyzed first. Then, several related
research topics will be introduced in detail, including wearable unobtrusive sensing, light-weight androbust incremental recognition model and motor pattern mining for brain diseases. At last, one integrated
system named WearableHUB for healthcare provision for senior citizens will be introduced.
Presenter Bio:
Dr Yiqiang Chen is a Professor and the Founding Director of the Pervasive Computing Research Center,
Institute of Computing Technology (ICT), the Chinese Academy of Sciences. He received his PhD degreefrom ICT in 2002. In 2004, he was a Post-Doctoral in the Department of Computer Science, Hong Kong
University of Science and Technology (HKUST). His research focuses on human computer interaction
and pervasive computing and wearable computing, especially on cognitive wearable computing--learningand understanding human behaviors (physiological, gesture, activity and context etc) in unobtrusive way.
He has published over 100 papers in ACM/IEEE TMC, TKDE, TNN, TIST, TCSVT and PR, as well as
International conferences such as ACM Multimedia, Ubicomp, AAAI, IJCAI, and PerCom. He has also
initiated several collaborations with large multinational corporations such as Huawei, Legend, Changhong(the biggest TV manufacturer in China) and France Télécom (the biggest telecom company in France).
His research works have been translated into several real world applications and are wildly used in China.
One of them is the Multimodal Chinese Sign Language Interaction system which has been used in over
1,000 schools for the hearing impaired and several TV stations around China. He has many received
international accolades including Best Paper/Contest/Video/Poster Awards, including PRICAI’05 、Gamenet’14 ICDM’07 and AAAI-16, etc. He is the chair of the Wearable Multimedia TechnicalCommittee of the National Audio-Video Standard Group which is responsible for the wearable e-Healthdata exchange standard in China, and also a TC member of IEEE MSA and IEEE IWCD.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

14

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

AwardsBEST PAPER AWARDS
Crowd Science and Engineering: Concept and Research Framework
Yueting Chai, Chunyan Miao, Sun Baowen, Yongqing Zheng and Qingzhong Li

Towards Intelligent Crowd Decision Support: A Preliminary Result on Large-scale Discussion Suppor based on the
Discussion Tree
Akihisa Sengoku,Takayuki Ito, Katsuhide Fujita, Shun Shiramatsu, Takanori Ito and Eizo Hideshima

Crowd Wisdom Drives Intelligent Manufacturing
Jiaqi Lu, Shijun Liu, Lizhen Cui, Li Pan and Lei Wu

Modeling of Crowd Science Network: An Approach Based on Cyber-Anima
Jianping Shen, Yadong Huang and Yueting Chai

Towards Data-driven Software Engineering Skills Assessment
Jun Lin, Han Yu, Zhengxiang Pan, Zhiqi Shen and Lizhen Cui

BEST STUDENT PAPER AWARDS
Computerizing More Repeatable Trail Making Test for Longitudinal Cognitive Assessment
Zhiwei Zeng, Chunyan Miao, Cyril Leung and Zhiqi Shen

Motivation Mechanism of Gamification in Crowdsourcing Projects
Zhiyuan Zeng, Jian Tang and Tianmei Wang

BEST POSTER AWARDS
Efficient Crowd-Powered Active Learning for Reliable Review Evaluation
Xinping Min, Lizhen Cui, Qingzhong Li and Han Yu
Game Theoretical Approach to Prevent Illegal Network Flow
Qingyu Guo, Bo An and Yair Zick

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

15

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Accepted Papers
An Approach to Discussion Facilitators' Action Selection based on Expected Utility Calculated with Random Forest
Regression
Shun Shiramatsu and Yuto Ikeda
Automatic Headline Generation using Templates for Electronic Discussion Bulletin Board in Japanese
Katsuhide Fujita and Ryosuke Watanabe
Collaborative Innovation Modes for Small and Micro Enterprises
Aidan Li, Shuangxi Huang and Yueting Chai
Computerizing More Repeatable Trail Making Test for Longitudinal Cognitive Assessment
Zhiwei Zeng, Chunyan Miao, Cyril Leung and Zhiqi Shen
Crowd Computing for Population Aging Challenges
Han Yu, Zhengxiang Pan, Chunyan Miao and Cyril Leung
Crowd Science and Engineering: Concept and Research Framework
Yueting Chai, Chunyan Miao, Sun Baowen, Yongqing Zheng and Qingzhong Li
Crowd Wisdom Drives Intelligent Manufacturing
Jiaqi Lu, Shijun Liu, Lizhen Cui, Li Pan and Lei Wu
Design on Information Providing Service for Temporary Organizations
Tetsuya Oishi, Rieko Fujita and Tokuro Matsuo
Design Tradeoffs for Cloud-Based Ambient Assisted Living Systems
Yi Dong, Yonggang Wen, Han Hu, Chunyan Miao and Cyril Leung
Distributed ACO Based on Crowdingsourcing Model for Multiobjective Problem
Jiaqi Lu, Li Pan and Shijun Liu
Efficient Crowd-Powered Active Learning for Reliable Review Evaluation
Xinping Min, Lizhen Cui, Qingzhong Li and Han Yu
ETTF: A Trusted Trading Framework Using Blockchain in E-commerce
Xinping Min, Qingzhong Li, Lei Liu and Lizhen Cui
Evaluating Complex Task through Crowdsourcing: Multiple Views Approach
Lingyu Lyu and Mehmed Kantardzic
Familiarity Design in Exergames for Elderly
Hao Zhang, Chunyan Miao and Di Wang
Investigating the Influences of Gamification in Citizen Science Systems: A Motivational Affordances Perspective
Jian Tang and Yuxiang Zhao
Modeling of Crowd Science Network: An Approach Based on Cyber-Anima
Jianping Shen, Yadong Huang and Yueting Chai
Motivation Mechanism of Gamification in Crowdsourcing Projects
Zhiyuan Zeng, Jian Tang and Tianmei Wang

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

16

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Accepted Papers
Research on the Characteristics and Operation Mechanism of the Maker Space in the Perspective of Complex Adaptive
System
Xiao Hou, Xin Jin and Jing Wu
Role Fulﬁlment Model for Motivating Participation in Mundane Crowdsourcing Tasks
Xinjia Yu, Chunyan Miao, Cyril Leung and Charles Thomas Salmon
Role of Human Emotions in Social Influence
Pankaj Mishra, Rafik Hadfi and Takayuki Ito
Sharing Economy: Sustainable Development Driven by Internet
Xinyue Zhu, Rong Guo and Xuankai Zhao
The existence of herding in P2P lending and the rationality of it - An empirical study from China
Xiangyu Wang and Yan Tu
Toward a Multiagent Reinforcemnet Learning Approach for Cooperative Game Playing with Users on a Sugoroku-like
Board Game
Kihaya Sugiura and Naoki Fukuta
Towards Age-friendly E-commerce Through Crowd-improved Speech Recognition, Multimodal Search, and Personalized
Speech Feedback
Lei Meng, Quy Hy Nguyen, Xiaohai Tian, Zhiqi Shen, Eng Siong Chng, Yunqing Guan, Chunyan Miao and Cyril Leung
Towards Data-driven Software Engineering Skills Assessment
Jun Lin, Han Yu, Zhengxiang Pan, Zhiqi Shen and Lizhen Cui
Towards Intelligent Crowd Decision Support: A Preliminary Result on Large-scale Discussion Suppor based on the
Discussion Tree
Akihisa Sengoku, Takayuki Ito, Katsuhide Fujita, Shun Shiramatsu, Takanori Ito and Eizo Hideshima
Understanding the Factors that Influence the Implemented New Product Ideas in Organization’s Crowdsourcing
Communities
Qian Liu and Bao Wen Sun© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

17

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

An Approach to Discussion Facilitators’ Action
Selection based on Expected Utility Calculated with
Random Forest Regression
Shun Shiramatsu∗ , Yuto Ikeda∗∗ Department of Computer Science, Nagoya Institute of Technology, Nagoya 466–8555, Japan

Abstract—Although facilitation is important in large-scaleconsensus building, keeping Web-based discussions as productive
as face-to-face discussions is sometimes difficult. We focus on facilitators’ action selection on a Web-based forum system. Clarifying
the principle of facilitators’ action selection is needed for alleviating cognitive loads of human facilitators during Web-based
discussion. We assume that a facilitator selects an action thatmaximizes expected utility corresponding to his/her intention.
This paper particularly discribes a particular utility function,i.e., the number of non-facilitators’ utterances in succeedingutterances within a certain period of time, corresponding to afacilitators’ intention to promote active discussion. The expectedutility can be estimated with Random Forest Regression trainedby a discussion corpus. The experimental results showed that
actions selected by the expected utility were consistent with theintention represented by the expected utility. However, humanfacilitators’ actual actions were not consistent with the actions
selected by the expected utility. These results indicated that weneed to investigate facilitators’ diverse intentions through trying
diverse utility functions.

I.

I NTRODUCTION

Although Web-based discussion support systems have major potential for large-scale public involvement, keeping Webbased discussions as productive as face-to-face discussions issometimes difficult. Discussion facilitators play an importantrole in keeping public debate productive and in building better
consensus because diverse stakeholders addressing public issues generally have diverse viewpoints. This diversity should
be increased as more people participate using the Web. This is
an important problem to be addressed for achieving large-scale
crowd decision making.
We aim to develop autonomous facilitator agents to leadbetter quality of consensus building. We particularly consider
the action selection of the facilitator agents. Clarifying theprinciple of facilitators’ action selection is needed for alleviating cognitive loads of human facilitators on Web-based
discussion. In our previous work [1], we statistically analyzed
facilitators’ utterances to investigate what kind of linguistic
expressions are used for facilitation in Web-based discussion.
This statistical analysis used a corpus of Web-based discussionthat is collected by using a Web-based forum system called
COLLAGREE [2], [3]. In this paper, we also use this discussion corpus.
To generate autonomous facilitators’ utterances, we needed
to clarify the following five criteria through analyzing theactual facilitators’ utterances.

•

Why: Why the facilitator says it, i.e., the intention
or purpose of the facilitator’s action.

•

What: What the facilitator says, i.e., the semantic
content of the facilitator’s utterance.

•

How: How the facilitator says it, i.e., the linguistic
expression of the facilitator’s utterance.

•

When: When the facilitator says it, i.e., the timingof the facilitator’s action.

•

Whom: To whom the facilitator sayz it, i.e., the targetof the facilitator’s utterance.

The action selection of facilitator, the focus in this paper,is related to “Why,” “What,” and “When.” The intention or
purpose of facilitators’ action, i.e., “Why,” can be dealt withusing an expected utility function corresponding to his/her
intention or purpose. The semantic content of facilitators’
utterance, i.e., “What,” can be dealt with using our definitionof eight types of facilitators’ utterances while referring toour previous work [1]. The timing of facilitators’ action, i.e.,“When,” can be dealt with by preparing the action type to wait.
The timing of when the expected utility of waiting becomes
lower than that of other actions is the timing to say something.
II.

ACTION S ELECTION BASED ON E XPECTED U TILITY

If we can define and calculate a metric for quality ofconsensus in discussion, such a metric can be used as anutility function of discussion facilitators. If it can be calculated
and accurately predicted, even the intention or purpose of the
facilitators’ actions can be selected according to the expectedutility. However, defining it for the quality of consensus is aquite difficult and far-reaching challenge.
A. Candidate Utility Function
As a realistic approach, we consider using easily observable
metrics as a utility function of facilitators. Our proposed model
assumes that the facilitator can use different utility functions
depending on his/her intention, e.g., promoting diversity ofidea viewpoints, promoting integration of ideas, or promoting
participants’ mutual understanding. Facilitators select theiractions according to the expected utility of the candidate
actions.
For example, such easily observable metrics can be enumerated as follows.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

1

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

1)

2)

3)

4)

5)

Number of non-facilitators’ utterances in succeeding utterances within a certain period of time.
This metric can be selected as a utility function whenfacilitators intend to promote active discussion.
Number of different non-facilitators in succeedingutterances within a certain period of time. This
metric can be selected as a utility function when facilitators intend to promote the diversity of participants.
Number of novel terms in succeeding utterances
within a certain period of time. This metric canbe selected as a utility function when facilitators
intend to promote the completeness or diversity ofparticipants’ viewpoints.
Increase in the content similarities among utterance sets for each non-facilitator in succeedingutterances within a certain period of time. This
metric can be selected as a utility function when facilitators intend to promote the mutual understandingof participants.
Ratio of positive expressions in succeeding utterances within a certain period of time. This metriccan be selected as a utility function when facilitators
intend to promote positive discussion.

Metrics (3) and (4) are inspired by Takahashi et al. [4], whodeal with an incentive mechanism for discussion participantsin the COLLAGREE system.
In this paper, we particularly focus on metric (1), i.e., thenumber of non-facilitators’ utterances in succeeding utterances
within a certain period of time, because this metric is considered to be the simplest to analyze, predict, and interpret among
these five examples.
B. Notation and Formulation
Hereafter, let A = {Wait, utyp 1 , utyp 2 , · · · } be a set of
facilitators’ candidate actions where Wait refers to postingnothing and utyp i refers to posting the i-th utterance type. We
assume that a facilitator agent can choose an action a ∈ A
while maximizing the expected utility EU t (a) at time t.

TABLE I.

ACTION T YPES

Action type

Samples

Feature expressions(Morpheme N-grams in Japanese)

Wait
Post something
Summarization(utyp 1 )

717
506
37

Greeting(utyp 2 )71

Paraphrase(utyp 3 )Gratitude(utyp 4 )Compliment(utyp 5 )Agreement(utyp 6 )Prompting(utyp 7 )Questioning(utyp 8 )

32

Other
(utyp 9 )184

xP (Ut (a) = x|st ) dx,(1)

where Ut (a) is a utility of action a from the viewpoint offacilitation and st is a situation at time t, i.e., a vector of
preceding contextual features at time t.
The utility Ut (a) can be estimated using regression analysis
because an observable metric is used as Ut (a) in this paper.
Explanatory variables of regression are features represening sy
and a. The probability P (Ut (a) = x|st ) can be estimated from
distributions of residual errors of the regression analysis.
Ut (a) =EU t (a) ==yt (a) + ϵ∫ +∞(yt (a) + ϵ)P (ϵ) dϵ−∞∫ +∞yt (a) +ϵP (ϵ) dϵ,
−∞

36
92
97

C. Action Types
The set of action types A = {Wait, utyp 1 , utyp 2 , · · · }are determined by referring to our previous work [1]. Table I
shows action types including eight types of facilitators’ utterance. These eight utterance types (Summarization, Greeting,
Paraphrase, Gratitude, Compliment, Agreement, Prompting,and Questioning) are derived from feature expressions offacilitators’ utterances extracted from the discussion corpus ina statistical way. The column “Samples” means the number ofsamples for each action type. These eight types are not disjointed because one facilitator’s utterance can contain multiple
feature expressions.
III.

∫ +∞
−∞

28

where let yt (a) is a predicted value∫ +∞ by regression and ϵ is a
residual error. That is, the term −∞ ϵP (ϵ) dϵ is the expectedvalue of residual error.

The expected utility EU t (a) can be calculated as

EU t (a) =

56

(2)

(3)

–
–
SHUYAKU SHI MASHI TA
GIRON NO MATOME DESU
NITSUITE MATOME MASHI TATAISEKI SHI MASUTAISHUTSU SHI MASUASHITA MO YOROSHIKU ONEGAI SHI MASUTOIU KOTO DESU NE
RIKAI DE YOROSHII DESHOU KA
ARIGATOU GOZAI MASUGO IKEN ARIGATOU GOZAI MASUSUTEKI DESU NE
II DESU NE
SOU DESU NE
OSSHARU YOU NIGO IKEN OOMACHI SHI TE ORI MASUDONO YOU NANO DE SHOU KADONNA MONO GA–

E XPERIMENTS

In this paper, we conducted preliminary experiments regarding metric (1) as a utility function. This utility function, thenumber of non-facilitators’ utterances in succeeding utterances
within a certain period of time, corresponds to facilitators’
intention to promote active discussion.
In the experiments, we used the discussion corpus gathered
on COLLAGREE in a social experiment in 2013. This corpus
consists of 1223 utterances written in Japanese posted by 266
participants. Each utterance consists of one or more sentences
and belongs to a particular discussion thread. The total numberof discussion threads is 214. The utterance set contains 506
utterances posted by nine facilitators.
A. Feature Selection for Regression
To apply regression analysis, feature selection for representing the action type a and the situation st is needed. We
selected 276 features (explanatory variables) for regression
analysis, i.e., ten binary features for a (Wait, Summarization, Greeting, Paraphrase, Gratitude, Compliment, Agreement,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

2

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Prompting, and Questioning), six quantitative features for st ,and 260 binary features as feature expressions for st .
We used Random Forest [5] for regression analysis because
it is robust against multicollinearity and can deal with a lotof explanatory variables (features). Random Forest Regression
is one of the ensemble machine learning algorithms thatgenerates a lot of regression trees while randomly choosing asmall set of features. In concrete terms, we use the R package“randomForest” [6], the implementation of the Random Forest
algorithm on the statistical software R.

means that most samples were classified into “Wait.” Onlya few samples were classified into the eight utterance types.
Intrinsically, the facilitators’ action selection is not a suitableapplication of classification because it cannot be divided into
a correct action and an incorrect one.
TABLE II.

RANDOM FOREST CLASSIFICATION

Action type
Wait
Post something
Summarization
Greeting
Paraphrase
Gratitude
Compliment
Agreement
Prompting
Questioning
Other
All data

Contextual features representing st , the situation at time t,
are extracted from the preceding context at t in a discussion
thread. The features for st are listed as follows.
•

timeint: Time interval from the latest utterance posted
by non-facilitators•

timeint faci: Time interval from the latest utterance
posted by facilitators•

lastlen: Number of morphemes included in the latest
utterance posted by non-facilitators•

precutt: Number of non-facilitators’ utterances in thepreceding context•

precutt faci: Number of facilitators’ utterances in thepreceding context•

partnum: Number of non-facilitator participants inthe preceding context•

fi : Feature expressions (morpheme N-grams) includedin the latest utterance posted by non-facilitators

Although the former six features are one-dimensional andquantitative features, the last one, feature expressions in thelatest utterance, corresponds to over a thousand binary dimensions representing the appearance of over a thousand feature
expressions.
For the experiments, we selected 260 feature expressions
using the information gain (IG) calculated as follows.
IG(C|Fi ) = H(C) − H(C|Fi ),

(4)

with
H(C) = −p(c+ ) log p(c+ )−p(c− ) log p(c− ),(5)
H(C|Fi ) = −p(c+ |fi+ ) log p(c+ |fi+ )−p(c− |fi+ ) log p(c− |fi+ )−p(c+ |fi− ) log p(c+ |fi− )−p(c− |fi− ) log p(c− |fi− ),(6)where C = {c+ , c− }, c+ is an immediate action type of
facilitator, c− is a comlement set of c+ , F = {fi+ , fi− }, fi+
is a latest utterance including feature expression fi , fii is acomplement set of fi+ , and H(·) denotes an entropy function.
We selected fi where IG(C|Fi ) ≥ 0.0025 for at least one
action type. The number of selected feature expressions was
260.

ACCURACY OF CLASSIFICATION RESULTS USING THE

Precision(%)
69.8
31.5
0
0
0
40
100
60
0
22.2
32.1
61.8

Recall(%)
94.3
15.8
0
0
0
5.4
8.7
10.3
0
7.1
36.4
–

C. Calculating Expected Utility∫ +∞In Equation (3), the term −∞ ϵP (ϵ) dϵ denotes the
expected value of the residual errors. Calculating an expectedutility requires a predicted value of the utility yt (a) and the
expected residual error.
First, parameter settings are needed for the regression of
the utility Ut (a), i.e., the number of non-facilitators’ utterancesin succeeding utterances within a certain period of time.
We empirically set “a certain period of time” to ten hours.
Parameters ntree , the number of regression trees, and mtry ,the number of features randomly chosen, also need to be
determined for the Random Forest. Figures 1 and 2 show
the change in prediction ability of the obtained regression
model when varying these two parameters. Both the correlation
coefficient and the root mean squared errors (RMSE) became
best when ntree = 600 and mtry = 20.
Figure 3 shows the correlation between the actual utilities
and the predicted ones. The correlation coefficient +0.796
shows a relatively high ability of the regression model topredict the utility.
TABLE III.

F ITNESS OF THE REGRESSION MODEL : THE CORRELATION
COEFFICIENTS AND THE ROOT MEAN SQUARED ERRORS (RMSE) FOR
EACH ACTION TYPE OF FACILITATORS

B. Classification Trial
Before the regression experiment, we tried to classify theaction types using the Random Forest Classification with the
276 features. Table II shows the results of the classification
accuracy. The high recall and middle precision of “Wait”© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

Action type
Wait
Post something
Summarization
Greeting
Paraphrase
Gratitude
Compliment
Agreement
Prompting
Questioning
Other
All data

Correlation+0.738+0.802-0.332+0.802+0.527+0.737+0.870+0.539+0.814+0.748+0.853+0.796

RMSE
1.83
1.26
0.98
0.80
1.36
1.05
1.39
0.94
1.34
1.79
1.17
1.62

3

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IV.

Fig. 1. Correlation coefficients by varying the number of regression trees
and number of variables randomly sampled

Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
21
22
65
92
102
104
141
155
167

I MPORTANCE OF FEATURES IN THE R ANDOM F OREST
R EGRESSION

Featurea: Otherlastlentimeint
timeint faci
precutt faci
a: Waitprecutt
fi : “DA TO OMOI MASU”
fi : “KAIGAI KARA NO”
fi : “SUNDE IRU”
fi : “NO DE SHOU KA”
fi : “KOEN YA DOURO NO”
fi : “GA HITSUYOU DA TO”
fi : “SUSUME MASU TO”
fi : “DOU DE SHOU KA”
a: Questioning
partnuma: Prompting
a: Complimenta: Agreementa: Greeting
a: Gratitude
a: Paraphrase
a: Summarization

Importance
366.57359.24330.45317.99308.22295.07165.15
86.68
70.98
68.08
66.04
64.99
64.52
64.43
63.23
54.68
54.27
29.72
19.66
17.39
16.80
9.57
8.40
6.95

facilitators’ summarization. It is not a major problem because
the RMSE in summarization samples is less than 1, that is,
equivalent legel of other action types.

Fig. 2. Root mean squared errors by varying the number of regression trees
and number of variables randomly sampled

Table IV shows the importance of features in the Random
Forest Regression. The result indicates that the contextual
features extracted from the preceding context relatively contributed to predicting the number of succeeding utterances.
In the action types, “Other,” “Wait,” and “Questioning” wererelatively important.
The expected residual error is calculated for each action
type because the differences between action types need to beconsidered when comparing action types by EU t (a). Figure
4 shows the probability distribution of the residual errors foreach action type. The expected residual errors are calculated
from this probability distribution as shown in Table V. This
result shows the expected utilities of “Wait” and “Questioning”
relatively increased above the utility predicted by the regression analysis.

Fig. 3.
Correlation between the actual measured counts of succeedingutterances and predicted values

Table III shows more detailed results for each action type of
facilitators. The action types “Wait,” “Greeting,” “Gratitude,”“Compliment,” “Prompting,” and “Questioning” resulted in a
relatively high fitness of the regression model. It’s notablethat the action type “Summarization” shows a totally different
result, i.e., a negative correlation coefficient. This difference isbecause non-facilitators’ generally make few utterances after

We calculated EU t (a) for all action types a and compared action types by EU t (a) for each sample. Table VI
shows frequencies where an action type a was top-ranked by
EU t (a). These results indicated that when facilitators intend
to promote active discussion, only “Wait,” “Questioning,” and“Promoting” are selected by the expected utility. Moreover,the frequency where the expected utility selected to post
“Questioning” of “Promoting” was only 41 times. In mostcases (1,147 samples), the expected utility selected “Wait,” i.e.,
to say nothing. These results are consistent with the tendencyof the training corpus because the average utility of “Wait”
is greatest and that of “Questioning” is greater than that ofother utterance types, as shown in Table VII. These results arealso consistent with the intention to promote active discussion
corresponding to the utility function. However, the selected© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

4

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE V.

E XPECTED RESIDUAL ERROR FOR EACH ACTION TYPE

Action type
Wait
Post something
Summarization
Greeting
Paraphrase
Gratitude
Compliment
Agreement
Prompting
Questioning
Other
All data
TABLE VI.

Expected
residual error+0.100-0.115+0.081+0.070-0.469-0.393-0.536-0.472+0.011+0.113-0.179+0.011

T OP - RANKED FREQUENCIES BY THE EXPECTED UTILITYFOR EACH ACTION TYPE

Action type
Wait
Post something
Summarization
Greeting
Paraphrase
Gratitude
Compliment
Agreement
Prompting
Questioning
Other
TABLE VII.

Top-ranked
by EU t (a)
1147
–
0
0
0
0
0
0
1
40
–

Precision

Recall

0.576
–
0
0
0
0
0
0
1
0.15
–

0.922
–
0
0
0
0
0
0
0.011
0.062
–

AVERAGE NUMBER OF SUCCEEDING UTTERANCESWITHIN TEN HOURS

Action type
Wait
Post something
Summarization
Greeting
Paraphrase
Gratitude
Compliment
Agreement
Prompting
Questioning
Other
All data

Averageutility
2.13
1.12
0.43
0.31
1.19
1.02
1.46
0.89
1.02
1.98
1.12
1.55

utterance types were quite different from the actual facilitators’
utterance types according to the precision and recall shown in
Table VI.

Fig. 4.

Probability distribution of the residual errors

This difference between the expected utility that we focus
on in this paper and the actual facilitators’ action selection
indicates that we need to consider other intentions of facilitators. The expected utility we focus on this time corresponds tothe intention to promote active discussion. As aforementioned,
we are considering other intentions, i.e., the intention topromote the diversity of participants, the intention to promote
the completeness or diversity of participants’ viewpoints, theintention to promote participants’ mutual understanding, and© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

5

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

the intention to promote positive discussion.
Although the contribution of this finding to developing
autonomous facilitator agents is not major, it can be applied tosupporting human facilitators. That is, a Web-based forum system provides various utility functions, and human facilitators
select one utility function according to their intention. Afterthe human facilitator selects the utility function, the system
can recommend action selection according to the regression
analysis and calculated expected utility. Such a facilitator
support function potentially decreases the cognitive load of
the human facilitators.
IV.

[6] A. Liaw, M. Wiener, L. Breiman, and A. Cutler, “randomforest: Breimanand cutler’s random forests for classification and regression,” https://cran.r-project.org/web/packages/randomForest/index.html (accessed on
May. 23, 2016), 2015.

C ONCLUSION

We tried to estimate the number of succeeding utterances ina certain period of time using Random Forest Regression, andwe calculated the expected utility from the regression results.
The regression model to predict the number of succeedingutterances was well-trained because the correlation coefficient
between the predicted values and the actual observed value was+0.796. The action selection experiment showed the results
were consistent with the intention to promote active discussion
corresponding to the utility function. “Wait” and “Questioning”
were selected to increase the number of succeeding nonfacilitators’ utterances. However, these results are quite different from the actual human facilitators’ action selection.
These results also indicated that actual human facilitators
have various intentions other than the intention to promote
active discussion. As future work, we are planning to investigate other utility functions, e.g., the number of novel terms insucceeding utterances, the increase in the content similarities
among utterance sets for each non-facilitator, and the ratio of
positive expressions in succeeding utterances. We also aim
to develop an action recommending function based on thesystem’s calculation of expected utility and human facilitator’s
selection of a suitable utility function matching with his/her
intention.
ACKNOWLEDGMENTS
This work was partially supported by JST CREST and a
Grant-in-Aid for Young Scientists (B) (No. 25870321) fromJSPS.
R EFERENCES[1] S. Shiramatsu, T. Nishida, T. Ito, and K. Fujita, “Feature expression
extraction from discussion facilitators’ utterances in web-based forum
system towards autonomous facilitator agents,” in Proceedings of the
5th IIAI International Congress on Advanced Applied Informatics, 2016
(to appear).
[2] T. Ito, Y. Imi, T. Ito, and E. Hideshima, “COLLAGREE: A faciliatormediated largescale consensus support system,” in Collective Intelligence
2014, 2014.
[3] T. Ito, Y. Imi, M. Sato, T. Ito, and E. Hideshima, “Incentive Mechanism for Managing Large-Scale Internet-Based Discussions on COLLAGREE,” in Collective Intelligence 2015, 2015.
[4] K. Takahashi, T. Ito, T. Ito, E. Hideshima, S. Shiramatsu, A. Sengoku,
and K. Fujita, “Incentive mechanism based on quality of opinion forlarge-scale discussion support,” in Proceedings of the Collective Intelligence 2016, 2016.
[5] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
5–32, 2001.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

6

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Automatic Headline Generation using Templates forElectronic Discussion Bulletin Boards in Japanese
Katsuhide Fujita and Ryosuke Watanabe
Faculty of Engineering,Tokyo University of Agriculture and Technology,
Koganei, Tokyo,184-8588, Japan,
Email: katfuji@cc.tuat.ac.jp

Abstract—Recently, the opportunity to discuss topics on avariety of online discussion bulletin boards has been increasing.
However, it can be difficult to understand the discussion as
the number of posts increases. In addition, it is important togenerate headlines that can automatically summarize each postat a glance to easily understand the discussion’s structure. In
this paper, we propose a method to automatically extract andgenerate post headlines for online discussion bulletin boards. We
propose templates with multiple patterns to extract importantsentences from the posts. In addition, we propose a method togenerate headlines by matching the templates with the patterns.
Then, we evaluate the effectiveness of our proposed method using
questionnaires.

I. I NTRODUCTION
In recent years, the general public has taken to exchanging opinions using various online platforms, such as online
discussion bulletin board systems (BBSs), microblogs, andother promising collective intelligence platforms ([1] etc.). The
number of opinions on the web is huge, which has not yet been
fully considered. On web-based platforms, crowds can freely
post their opinions without constraints of time or place. In thenear future, large-scale discussion aggregations, discussions,and negotiations will become possible on supporting webbased platforms, and further developments in social/crowd
engineering are expected ([2], [3], [4], [5]). With an increasein the amount of information and posts on online discussion
platforms, reading and understanding all posts can be difficult.
Therefore, platforms that automatically construct a structured
visualization of discussions have been proposed [1]. In particular, an open web-based forum system called COLLAGREE([6], [7]), which has facilitator support functions, has beendeveloped and deployed for an internet-based town meetingin Japan. In COLLAGREE, a discussion tree is employed anda method that generates headlines for each post is required[8].
In addition to social and academic environments, linguistic
resources have become increasingly rich via web developments
and the technology of natural language processing (NLP)
has been developed. However, it is still difficult to obtainaccurate information from large-scale discussion platforms.
This is because the amount of unnecessary information has
also increased with the development of linguistic resources. In
addition, argumentation mining aimed at automatically extract-

ing structured arguments from unstructured textual documents
has become important for internet-based discussions ([9], [10],[11]). This topic has recently become of broad interest due
to its potential to process information originating on theWeb, in particular, from social media. Automatic structuringof discussions and argumentations is enabling breakthroughapplications in social and economic sciences, policy-making,and crowd engineering. Argumentation models, methods, andapplications have been proposed to conjugate representational
needs with user-related cognitive models and computationalmodels for automated reasoning.
Automatic headline generation is necessary for visualizinguser-related cognitive models and computational models. Despite the number of studies focused on automatic headline
generation ([12] etc.), previous approaches have not taken
into account the structure of the discussions because theirtargets have been newspapers or books, not online large-scale
discussion platforms.
In this paper, we propose a method to automatically extract
important sentences and generate headlines from posts fordiscussion BBSs. In our proposed method, templates withmultiple patterns are used to extract important sentences fromthe posts. In addition, we propose a method to generate aheadline by matching the templates with the patterns. Then,
we evaluate the effectiveness of our proposed method using
questionnaires. The datasets for the evaluations are log datafrom an internet-based town meeting in Nagoya, which was acity project for an actual town meeting of the Nagoya Next
Generation Total City Planning for 2014-2018 [6]. In theseevaluations, we compare our proposed approach with various
existing methods of automatically generating headlines using
a questionnaire.
The remainder of this paper is organized as follows. In
section 2, related studies concerning automatic headline generation methods are shown. In section 3, the approach forautomatic headline extraction using templates is proposed. In
Section 4, we describe the results of the evaluation experiments
and discussions. Finally, we present our conclusions.
II. R ELATED W ORKS
Multiple studies concerning headline generation have focused on many languages and document types ([13], [14], [15],© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

7

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[16] etc.). According to Gattani, three primary techniques canbe identified [15].
Statistics-based approaches
These methods apply statistical models to learn correlations
between words in headlines and documents. The majorityof models are used in supervised learning environments;
therefore, large amounts of training datasets with labels arenecessary. For example, a method viewing summarization asa problem analogous to statistical machine translation basedon Naive Bayes has been proposed [17]. The use of statisticalmodels for learning pruning-rules for parse trees has also beenstudied ([18] etc.).
Summarization-based approaches
Headlines can be thought of as very short summaries.
Therefore, traditional summarization methods can be used togenerate one-line summaries ([19] etc.). The primary difficultywith these approaches is that they use techniques that were
not initially devised for generating compressions of less than
10% of the original content, which directly affects the qualityof the resulting summary [17]. They generate a headline byreusing words presented in the article; therefore, the generated
headline risks losing or changing the contextual meaning of
the reused words [20].
Template-based approaches
These methods apply handcrafted linguistic rules to extract
or compress important sentences in a document [21]. They
are simple and lightweight; however, they fail to understandcomplex relationships in the text, and it is not easy to prepare
templates for them. For example, Alfonseca et al. proposed an
approach mining the syntactic patterns that a Noisy-OR model
generalizes into event descriptions [22]. At the inference time,the method searches the model with the observed patterns
in an unseen news collection and identifies the event thatbest captures the gist of the collection to retrieve the mostappropriate pattern to generate a headline.
However, original templates for each language are necessary
in this approach. In other words, studies focusing on theJapanese language and online BBSs are few. In this study,
we employ this approach because it is lightweight and does
not require training datasets to generate templates.
For automatic headline generation approaches for multiple
sentence compression
The above three approaches used in the previous studies
focused on a single article or document. However, this study
focuses on texts in online discussion BBSs; in other words, thisstudy considers the task of summarizing a cluster of related
sentences with a short sentence, which we call multi-sentencecompression, to present a simple headline. The advantage
and novelty of our proposed method is that we can use thecharacteristics of the discussions and the Japanese language.
One popular approach to multi-sentence compression was
described by Filippova, who presented a simple and robust

word graph-based method to generate succinct compressions
that require just a part of speech tagger and a list of stop
words [12]. The advantage and the novelty of the proposed
method is that it is syntax lean and requires little more than atokenizer and a tagger. However, the approach of Filippova isfor English and Spanish and is completely different from our
proposed method.
III. AUTOMATIC H EADLINE G ENERATION USINGT EMPLATESBBSs for discussions, such as COLLAGREE ([6], [7]) are
given the main themes and comprise several threads thatdeal with a single related topic. A thread refers to a set of
opinions related to a particular topic or issue. In COLLAGREE
interface, which has several functions, the first opinion of
the user who creates the thread becomes the parent opinion,and other users reply to it as child opinions. There are also
cases where some users reply to a child opinion as grandchild
opinions.
Our goal is to propose a method to automatically generatea headline for each post in the discussion BBS. The processof the proposed method is as follows.
1) Pre-processing2) Extraction of Headline
3) Generation of Headline4) Complement of Headline
5) Post-processing
A. Pre-processing
To arrange the text of the BBS for the discussion, thefollowing preprocessing step is conducted. The parentheses
and URL are removed. This unimportant information cancause the generated summary to be redundant and should
therefore be deleted before summarization. It is also essential
to clarify anaphors in the original post. In reference to theresult of an anaphora analysis using CaboCha [23], only
anaphors that refer to a named entity are retained. Furthermore,
only nouns and verbs are extracted from each post (e.g., an
opinion or a sentence) based on the result of a morphologicalanalysis.
B. Extraction of Headline
The sentence that will become the headline is extracted
from the posts, which are divided into sentences. In previous
studies focusing on conference notes, headlines were extracted
by focusing on expressions such as “about” [24]. However, aclear structure does not exist for online discussion bulletinboards. Therefore, we propose novel templates with several
patterns of sentences that seem to express the intent of theparticipants. Table I shows a proposed template with threepatterns to extract important sentences in the discussion BBS.
Table I shows the infinitive only; however, other inflected
word forms are applied. We can match infinitive words witha morphological analysis using CaboCha on the proposed
templates. The details of each pattern are as follows.
The details of each pattern are as follows:© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

8

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IT EMPLATE FOR EXTRACTING THE SENTENCES

pattern 1
pattern 2

PriorityHigh
Mediumpattern 3

Low

Pattern“need” or “important”“think,” “believe” or “regard”
agreements (including positive keywords)
disagreement (including negative keywords)

Pattern 1: Sentence including “need” or “important”
Sentences including “need’ or “important” are more important to the participants. Therefore, their priority is thehighest in the template.
• Pattern 2: Sentence including “think” or “regard”
Because the discussion BBS is a place for participants
to post their opinions, often sentences including “think,”“believe,” or “regard” occur. Therefore, their priority is
set to medium.
• Pattern 3: Agreements and Disagreements
Sentences including agreements and disagreements areimportant as headlines because participants express theiropinions when having a discussion. Therefore, a sentence for which the absolute sum of all positive andnegative words is the highest is extracted as Pattern 3.
The negative and positive words are judged using the
Evaluative Expressions Dictionary ([25], [26]). The score
for words including positive words is “+1” and that forwords including negative words is “-1.”The sentence with the highest priority is extracted when
multiple patterns are matched in Table I. In addition, thesentence with a higher cosine similarity between it and theparent post is extracted when the same patterns match multiple
sentences. We assume that the set of m nouns in the parent post
is P = {p1 , p2 , ..., pm }, the set of n nouns in the matchingsentence is Q = {q1 , q2 , ..., qn }, and each element means thefrequency of occurrence. The cosine similarity is calculatedby the following expression.
•

P ·Q
cos(P, Q) =|P ||Q|(1)

The sentence with the highest score is then extracted. In otherwords, sentences with similar words to those in the parent postare extracted. When the sentences do not match the patternsin Table I, our method outputs “no pattern.”
C. Automatic Headline Generation
The sentence is extracted by the method proposed in theprevious section when the length of the sentence is less thantwenty letters. The headline is generated automatically basedon the following template when the length of the sentence ismore than twenty letters. It is proposed by analyzing the posts
in any actual BBSs for discussions.
Fig. 1 shows the proposed template1 .
1 The part of the template is written in Japanese because it used the original

Japanese grammars.

Fig. 1. The Proposed Template for Automatic Headline Generation

When multiple patterns match the sentence, the pattern withthe highest average weights of BM25 [27] of the noun is used.
The expression of BM25 of a word (w) in n posts of set(D = {d1 , d2 , ..., dn }) is defined as follows:score(w, D) =n∑f (w, di ) · (k1 + 1)
idf (w) ·
|di |
f
(w,
d
)
i + k1 · (1 − b + b · avgdl )i=1
idf (w) = log(2)

N − df (w) + 0.5
df (w) + 0.5(3)

In the above equations, f (w, di ) is the frequency of theword(w) in the post(di ), |di | is the number of words in di .
avgdl is the average number of words in the post set(D), k1
and b are previously decided parameters. In this study, weset, k1 = 2.0, b = 0.75 are used because they are used in a
general way. N is the number of posts and df (w) is the numberof posts including the word w in all posts. In our proposed
method, we used BM25 because the BM25 of characteristic
words in the entire BBS tends to be higher.
D. Complement of Headline
The headline generated using the automatic headline generation method in the previous section is insufficient because
there is no clause modified by the extracted clause to explain it.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

9

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE II
ACCURACY RATES OF EACH METHOD
MethodProposedFirst Sentence
Random

Accuracy rates
0.64
0.37
0.32

TABLE III
ACCURACY RATES IN EACH PATTERN OF OUR PROPOSED METHOD
Pattern
The number of applicable
Accuracy
Accuracy rates

Fig. 2. Removal and Replace Words Lists

Therefore, modified clauses are added using the modification
parsing of CaboCha [23]. Concretely speaking, a clause tomodify the clause extracted by the method in the previous
section is complemented when there is an extracted clause,and the clause with the highest BM25 is complemented when
there are many extracted clauses. This complement is repeatedby a constant number of characters. In this study, the constantnumber of characters is 20.
E. Post-processing
This step examines the generated headline to improve it.
A particle remains at the end of a sentence when a headline
is extracted by matching each clause. Therefore, unnecessary
clauses, such as particles at the end of a sentence, are replaced
and removed based on the list of Fig. 22 . Removal andmatching are repeated for the word lists for the end of thesentence until the headline no longer changes.
IV. E XPERIMENTAL R ESULTS
A. Datasets for the experiments
We conducted comparative studies to evaluate the proposed
method by extracting headlines using questionnaires. The
datasets for the evaluations are log data from an internetbased town meeting in Nagoya, which was a city project
for an actual town meeting of the Nagoya Next Generation
Total City Planning for 2014-2018 [6]. In this town meeting,there were 266 participants and 9 facilitators. The topics of
discussion were “human rights,” “the environment,” “attractive,” and “disasters” for the Nagoya Next Generation Total
City Planning for 2014-2018. There were a total of 1151
posts. These experiments did not address the posts from thefacilitators because their models are completely different fromthose of the participants.
2 The part of the template is written in Japanese because it used the original
Japanese grammars.

1
22
19
0.86

2
48
29
0.60

3
30
16
0.53

N/A
0
0
-

Total
100
64
0.64

B. Evaluations for extracting the headline
The correct datasets for the extraction of the sentence thatis suitable for the headline were generated by questionnaires.
The sentence that was most effective as a headline wasselected from all posts in each thread by ten undergraduate
and master students at the Tokyo University of Agriculture
and Technology. If the optimal headline differed between thestudents, most of the selected sentences were considered toprovide the correct answer.
The comparative methods for the experiment are as follows:• Proposed: The headline extraction method proposed inthis paper• First Sentence: The method of selecting the first sentence
as the headline• Random: The headline extraction method of selecting asentence randomly“First Sentence” is a heuristic method of extracting aheadline. This method is effective for extracting headlines innewspapers, where important contents tend to be placed at
the beginning of the article [28]. However, this method is
not always effective for online BBSs because the lengths ofposts in BBSs are shorter than those in articles. “Random” isa baseline method for the comparative study.
Table II shows the accuracy rates of the three methods
based on the correct dataset. Our proposed method has thebest accuracy rate because the tendency of placing important
content at the beginning of a post, as in newspapers, is notapplicable in discussion BBSs. The beginning of a post instead
tends to be a simple approval or greeting (e.g., “I agree with
his idea” or “Hello”). In other words, the first sentence tends
to be completely different from the core of the commentsor questions. In fact, simple replies such as “yes” and “ofcourse” are commonly found. Our proposed method is also
better than Random (the baseline); therefore, our proposed
method is effective for discussion BBSs.
Table III shows the accuracy rates for each pattern of our
proposed method. For Pattern 1, the number of applicable
cases is small and the accuracy rate is high. posts including“necessary” and “important” generally reflect the participant’s
intentions. Conversely, the number of applicable cases is high© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

10

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IVE VALUATIONS OF R EADABILITY
Score
5
4
3
2
1

Description
The readability of the headline is extremely well. The meaning of the headline can understand, easily.
The headline is mostly readable. The meaning of the headline
can understand without any difficulty.
The readability of the headline is tolerable but not great. The
meaning of the headline can understand, somehow.
The readability of the headline is badly. It is difficult tounderstand the meaning of the headline.
The readability of the headline is extremely unnatural. The
meaning of the headline can’t understand.

TABLE VIAVERAGE OF FIVE - GRADE EVALUATION SCORES BY THE QUESTIONNAIRES

ProposedFirst Twenty Letters

Readability
3.09
3.74

Completeness
2.73
3.06

TABLE VIIAVERAGE OF FIVE - GRADE EVALUATION SCORES BY THE
QUESTIONNAIRES IN EACH PATTERN (R EADABILITY )Pattern
ProposedFirst Twenty Letters

1
3.26
3.68

2
3.00
3.65

3
2.54
3.53

N/A
3.42
4.04

TABLE VE VALUATIONS OF C OMPLETENESS
Score
5
4
3
2
1

Description
The headline covers all of the post.
The headline covers most of the post.
The headline covers some of the post.
The headline covers parts of the post.
The headline covers little of the post.

and the accuracy rate is low for Pattern 2. This is because“think” and “believe” are commonly used expressions andappear in multiple sentences. For Pattern 3, the sentences
including positive words are extracted as the important phrases
in the post. All posts match our three proposed patterns inthis experiment; therefore, our proposed method has effective
templates to extract the important sentences.
C. Evaluations for generating the headline
We evaluated the headlines generated by our proposed
method using questionnaires. The evaluation criteria for theheadlines were its ”readability” and ”completeness” because
the purpose of our proposed method was to generate headlines
with high readability and core content. Each evaluator gradeda headline on a five-grade evaluation scale following Tables
IV and V.
The comparative methods for these experiments are as
follows:• Proposed: Our proposed headline generation method• First Twenty Letters: Extracting the first twenty lettersas the headline from the posts
In the proposed method, the clauses were extracted basedon the template, and complemented clauses with high BM25
were generated using dependency parsing. In the First Twenty
Letters method, the first 20 letters were simply extracted as theheadline from the posts. The number of letters for the headline
was therefore equal to or less than 20.
The headlines of our proposed method and the baseline
method were evaluated by 11 undergraduate and master’s students at the Tokyo University of Agriculture and Technology.
Table VI shows the average of the five-grade evaluation
scores from the questionnaires. Our proposed method has
lower scores than the baseline method for readability andcompleteness according to the questionnaires. Tables VII andTABLE VIIIAVERAGE OF FIVE - GRADE EVALUATION SCORES BY THE
QUESTIONNAIRES IN EACH PATTERN (C OMPLETENESS )Pattern
ProposedFirst Twenty Letters

1
3.00
2.95

2
2.61
3.02

3
2.17
2.97

N/A
3.07
3.29

VIII show the average values of the five-grade evaluation
scores from the questionnaires in each pattern for readability
completeness, respectively In our proposed method, the score
for Pattern 1 is the highest for readability and completeness.
Next, a detailed analysis of the results of headline generationis shown.
• The extracted sentence matches Pattern 1 or Pattern
2: The scores of the First Twenty Letters method are
higher than those of our proposed method for readability
and completeness when the extracted sentence matches
Pattern 1 or Pattern 2. This is because the clauses thatare regarded as unnecessary are removed when generating
and complementing the headline. When the unnecessary
words are used to maintain a natural sentence, theseprocesses reflect negatively on the headline. Conversely,
the headline generated by the method of extracting theFirst Twenty Letters method is unaffected because theextracted sentences do not remove any clauses.
• The extracted sentence matches Pattern 3: Scores of
the First Twenty Letters method are higher than thoseof our proposed method in readability and completenesswhen the extracted sentence matches Pattern 3. This isbecause sentences including positive or negative words
are meaningless.
• Sentences including a conjunction or subject: Whenthe sentence includes a conjunction and a subject, ourproposed method can remove them by determining
the unnecessary words. In this situation, our method cancomplete some important words while maintaining highreadability.
V. C ONCLUSION
t is important to automatically generate headlines so thatreaders can understand each post at a glance and easily© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

11

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

understand the discussion structure. In this paper, we proposeda method of automatically extracting and generating headlines
from posts by focusing on the online discussion bulletinboard systems. We proposed templates with multiple patterns
to extract important sentences from posts. In addition, we
proposed a method to generate a headline by matching thetemplates with the patterns. We evaluated the effectivenessof our proposed method using questionnaires. In the task ofextracting the headline from the posts, our proposed method
had a higher accuracy rate than the baseline method. In thetask of generating the headline, our proposed method needs
improvement in terms of readability and completeness.
The possible future work includes improving the headline
generation from the extracted sentence. A hybrid method of
our proposed method and the heuristic method can be effective.
In addition, more evaluation criteria are necessary to evaluate
the generated headline using questionnaires. Other evaluation
criteria, such as the diffuseness and the essence of the posts,
will be useful in evaluating the headlines.
ACKNOWLEDGMENT
This work was supported by CREST, JST.
R EFERENCES[1] A. Gürkan, L. Iandoli, M. Klein, and G. Zollo, “Mediating debate
through on-line large-scale argumentation: Evidence from the field,” Inf.
Sci., vol. 180, no. 19, pp. 3686–3702, 2010.
[2] M. Klein, “Enabling large-scale deliberation using attention-mediation
metrics,” Computer Supported Cooperative Work, vol. 21, no. 4-5, pp.
449–473, 2012.
[3] ——, “Achieving collective intelligence via large-scale on-line argumentation,” in Proceedings of the Second International Conference on
Internet and Web Applications and Services (ICIW-2007), 2007, pp. 58–
58.
[4] M. Klein and L. Iandoli, “Supporting collaborative deliberation using alarge-scale argumentation system: The mit collaboratorium,” MIT Sloan
Research Paper, no. 4691-08, 2008.
[5] J. Introne, R. Laubacher, G. M. Olson, and T. W. Malone, “Theclimate colab: Large scale model-based collaborative planning,” in 2011
International Conference on Collaboration Technologies and Systems,
CTS-2011, 2011, pp. 40–47.
[6] T. Ito, Y. Imi, T. Ito, and E. Hideshima, “Multi-sentence compression:
finding shortest paths in word graphs,” in Proceedings of the International Conference on Collective Intelligence 2014 (CI-2014), 2014, pp.
322–330.
[7] T. Ito, Y. Imi, M. Sato, T. Ito, and E. Hideshima, “Incentive mechanism
for managing large-scale internet-based discussions on collagree,” inProceedings of the International Conference on Collective Intelligence2015 (CI-2015), 2015.
[8] A. Sengoku, T. Ito, K. Takahashi, S. Shiramatsu, T. Ito, E. Hideshima,
and K. Fujita, “Discussion tree for managing large-scale internetbased discussions,” in Proceedings of the International Conference on
Collective Intelligence 2016 (CI-2016), 2016.
[9] M. Lippi and P. Torroni, “Argumentation mining: State of the art andemerging trends,” ACM Transactions on Internet Technology, vol. 16,
no. 2, pp. 10:1–10:25, 2016.
[10] B. K. Bal and P. S. Dizier, “Towards building annotated resources foranalyzing opinions and argumentation in news editorials,” in Proceedings of the Seventh International Conference on Language Resources
and Evaluation (LREC-2010), 2010.
[11] P. Baroni, M. Romano, F. Toni, M. Aurisicchio, and G. Bertanza, “Automatic evaluation of design alternatives with quantitative argumentation,”Argument and Computation, vol. 6, no. 1, pp. 24–49, 2015.
[12] K. Filippova, “Multi-sentence compression: finding shortest paths inword graphs,” in Proceedings of the 23rd International Conference on
Computational Linguistics (COLING-2010), 2010, pp. 322–330.

[13] M. Gambhir and V. Gupta, “Recent automatic text summarization
techniques: a survey,” Artificial Intelligence Review, pp. 1–66, 2016.
[Online]. Available: http://dx.doi.org/10.1007/s10462-016-9475-9[14] V. Gupta and G. Lehal, “A survey of text summarization extractive
techniques,” Journal of Emerging Technologies in Web Intelligence,
vol. 2, no. 3, 2010.
[15] A. Gattani, Automated Natural Language Headline Generation Using
Discriminative Machine Learning Models, ser. Thesis (School of Computing Science). Simon Fraser University, 2007.
[16] C. A. Colmenares, M. Litvak, A. Mantrach, and F. Silvestri,“Heads: Headline generation as sequence prediction using an
abstract feature-rich space,” in Human Language Technologies:
The 2015 Annual Conference of the North American Chapter ofthe ACL (NAACL-2015), 2015, pp. 133–142. [Online]. Available:
http://www.aclweb.org/anthology/N15-1014[17] M. Banko, V. O. Mittal, and M. J. Witbrock, “Headline generation basedon statistical translation,” in Proceedings of the 38th Annual Meeting on
Association for Computational Linguistics (ACL-2000). Association for
Computational Linguistics, 2000, pp. 318–325.
[18] Y. Unno, T. Ninomiya, Y. Miyao, and J. Tsujii, “Trimming cfg parsetrees for sentence compression using machine learning approaches,” inProceedings of the COLING/ACL on Main Conference Poster Sessions(COLING-2006). Association for Computational Linguistics, 2006, pp.
850–857.
[19] H. Morita, R. Sasano, H. Takamura, and M. Okumura, “Subtree extractive summarization via submodular maximization,” in Proceedings of the
51st Annual Meeting of the Association for Computational Linguistics(ACL-2013) - Volume 1. Association for Computational Linguistics,
2013, pp. 1023–1032.
[20] B. K. Boguraev and M. S. Neff, “Discourse segmentation in aid of document summarization,” in Proceedings of the 33rd Hawaii International
Conference on System Sciences (HICSS-2000)-Volume 3, 2000.
[21] B. Dorr, D. Zajic, and R. Schwartz, “Hedge trimmer: A parse-and-trim
approach to headline generation,” in Proceedings of the HLT-NAACL
03 on Text Summarization Workshop - Volume 5.
Association for
Computational Linguistics, 2003, pp. 1–8.
[22] E. Alfonseca, D. Pighin, and G. Garrido, “Heady: News headline
abstraction through event pattern clustering,” in Proceedings of 51th
Annual Meeting on Association for Computational Linguistics (ACL2013), 2013.
[23] Y. M. Taku Kudo, “Japanese dependency analysis using cascaded
chunking,” in Proceedings of the 6th Conference on Natural Language
Learning 2002 (COLING 2002 Post-Conference Workshops), 2002, pp.
63–69.
[24] M. Koizumi, S. Shiramatsu, T. Ozono, and T. Shintani, “Implementationof minutes browsing support system based on structured question andanswers by relationship of discourse,” 2012 Information Processing
Society of Japan, vol. 2012, no. 1, pp. 657–659, 2012.
[25] N. Kobayashi, K. Inui, Y. Matsumoto, K. Tateishi, and T. Fukushima,“Collecting evaluative expressions for opinion extraction,” in Proceedings of the First International Joint Conference on Natural Language
Processing (IJCNLP-2004), 2005, pp. 596–605.
[26] K. Inui, S. Abe, K. Hara, H. Morita, C. Sao, M. Eguchi, A. Sumida,
K. Murakami, and S. Matsuyoshi, “Experience mining: Building alarge-scale database of personal experiences and opinions from web
documents,” in Proceedings of the 2008 IEEE/WIC/ACM International
Conference on Web Intelligence and Intelligent Agent Technology (WIIAT 2008), vol. 1, 2008, pp. 314–321.
[27] S. Robertson and H. Zaragoza, The probabilistic relevance framework:
BM25 and beyond. Now Publishers Inc, 2009.
[28] R. Brandow, K. Mitze, and L. F. Rau, “Automatic condensation ofelectronic publications by sentence selection,” Information Processing& Management, vol. 31, no. 5, pp. 675–685, 1995.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

12

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Collaborative Innovation Modes for Small and MicroEnterprises
Aidan Li

Shuangxi Huang*, Yueting Chai

School of Management and Economics,

Department of Automation,

Beijing Institute of Technology Beijing, China

Tsinghua University, Beijing, China

liaidan28@hotmail.com

Huangsx*, caiyt@mail.tsinghua.edu.cn

Abstract—Small and micro enterprises (SMEs),

the vast

September, 2013, China has had a total number ofmajority of Chinese businesses, constitute the main force of

146,903,100

technological innovation in China. This paper is a preliminary

99.7%. SMEs in China make not only the largest number,study on the collaborative innovation between SMEs withbut contribute 85% of increased jobs, 60% of GDP, 50% of

low-technology and low value-added products. A cooperative

tax revenue and 50% of total enterprise profit. Besides, 75%

game model based on Cournot model is used to analyze theof technical innovations are made by SMEs and 80% of new

profit function of SMEs’ collaborative innovation. It is

product R&D come from the SMEs. Judging from theconcluded that when an SME is set up the more realistic

number of technological innovations, SMEs have become

approach is to choose a cooperative partner who owns productthe main driving force of technological innovation inbrands and R&D resources for collaborative innovation. In

enterprises in China.

thecurrent

situation,

as

only

the

low

technology

manufacturing enterprises have the financial capabilities orsupport resources, they could have realized the transformationof enterprises and enhanced the capability of independent
innovation.

enterprises, ofwhich SMEs

account

for

Manufacturing industry plays a dominant role in Chinese
economy. There have been a large number of SME
manufacturers, most of which are short of innovation ability
and the production is kept at a relatively low level. There aremany acute problems in technical innovation and the key

Keywords— SMEs, low technology, collaborative innovation

problem is that SMEs are short of talent and capital support.
And

I． INTRODUCTION
Accordingto "The Small

there

are

other

problems

likelow patenttransformation rate.
and

Micro

(1) Lack of innovative personnel

Enterprises Development Report", published by the State
Administration for Industry and Commerce in Chinese, as of

At present, most SME employees in manufacturing are

the end of March, 2013, China has had a total number ofrelatively low in knowledge level. The SMEs are short of

11,698,700 small and micro enterprises (SMEs), accounting

technical

for76.57%

the 44,362,900

of

the

total

individual

enterprises

industrial

and

in

China. Ifcommercial

technicians

personnel, especially
who

have

seniorengineers

visionand innovation

marketandconsciousness. According to the "China SME Humanhouseholds are counted into the statistics, Chinese SMEs

Resource Management

Whitewill account for 94.15% of the total number of businesses

CHINA HRKEY Research Center in September, 2012,

registered in China. SMEs refer to those with no more thanChinese SMEs only have an average life of 2.5 years. Their

300 employees, annual revenues of 20,000,000 yuan, which,technical

are the vast majority of Chinese enterprises. As ofshort-term interests but do not take long-term interests into

innovations mainly

aim

Paper" released

at

the

by

current andaccount, which ends up with a shortage of long-term© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

13

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

personnel training and reserve. The current training is notcredit market, andadequate, resulting in drainage of brains.

bond issuance requirements.

SMEs canhardlyTherefore,

meet
it

is

the
almostimpossible for to raise funds through bond financing.

(2) Lack of enterprise innovation funds(3) Low patent conversion rate

Most SMEs in China only have a few years of history.
As they are small, they usually lack of enough money to run

The implementation rate of patent transformation in

the business. Normally SMEs first use their money toChina is averaged only 10% and the profit margin is only

purchase production materials while the money for

10%-15%, which are far below the level of developed

employees, technical innovation and personnel training iscountries’ 60%-70%[1].

often insufficient. If technical innovation needs extra funds,
that will worsen the situation. They will not have enoughfunds to buy technology, patents and equipment. The capitalturnover is often a major problem confronting SMEs in their
daily management. As purchasing fixed assets and severalrawmaterials for

sufficient circulating

the

daily

funds, theproduction

shortage

of

need

funds

is

restricting the SMEs’ daily production, management and
R&D activities.

in

China

have

strengthened

internal

rectification, and the bank credit scale is shrinking. To savecosts the state-owned commercial banks are unwilling tolend moneytocommercial

banks,

SMEs. Althoughcreditsmall-sized local

cooperatives

and

local

commercial banks have become the main force in supporting
SME development. But they are not capable of meetingthe capital requirements of SMEs.
An

collaborative

innovation isfor low-technology SMEs

a

to

catch

realistic

way

up

the

with

world advanced technologies with minimum cost and fastest
speed, which is an inevitable stage before finally developingthecapabilityofindependent

technical

innovation. Collaborative innovation is an important means
to optimize the allocation of resources and improve
the innovation efficiency for SMEs[2].

Under the impact of the financial crisis, financial
institutions

The experience of developed countries shows that: theenterprise’s

Xie,

Xuemei (2010)

made

a

surveythroughquestionnaires of 188 SMEs in manufacturing, by adopting
the structural equation model to explore the relationshipbetween the different collaborative innovation network andinnovation performance. Xie thinks that the collaborative
innovation network, such as "enterprise to enterprise","enterprise toagency" and "enterprise toresearch organization" have significantly positive effect on
the enterprise innovation performance. The "enterprise todirect financing isbond financing

government" collaborative

innovation networks

do

not

through the equity financing and the issuance of enterprise

have a direct effect on innovation performance, but it showsbonds as well as stocks. For the equity financing, as thea significant indirect effect. There are significant differences

listed entry threshold is too high, the most majority ofwithSMEs can hardly raise money they want this way. China

cooperative innovation networks

Growthperformance,

EnterpriseMarket/GrowthEnterprisetheimpactamong

degreeoftodifferent
the innovation

which the "enterprise to

Board (CGEM) was launched in October, 2009 and it hasenterprise" collaborative innovation network shows the most

lowered the entry barriers and strictly controlled thesignificant effect in promoting the enterprise innovation

financing operation in this market. It can somewhat help

performance[3].

SMEs

with

potential ability

toobtain financing

opportunities, but the CGEM is still at the infant stage
and there are still some problems. But, anyway CGEM
might relatively

alleviatethefinancingproblems. For

the bond financing, because the development of corporate

Many SMEs in China have obtained patents in thestart-up stage. But the patent quality is generally not high,with low added value. And their ability to apply the patenttechnology to the product is not strong. Especially the utility
model patent and the design patent are often eliminatedbond market is far behind stock market and bank

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

14

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

during one or two years for the short market cycles. Some

constants, a is the highest market price of the product. The

SMEs have transformed their patent technologies into theenterprise cost function: C = c + dqt + m, c is the fixed

products and earned first local or nationwide market shares.

cost, dqt is the variable cost, m is the opportunity cost to

Unfortunately once the market response is good, there willproduce the product. c, d, m are constants. The profit

soon emerge a large number of imitations on for the low

function of the enterprise in time period t:

entry barriers .

𝐿𝑡 = 𝑞𝑡 (𝑎 − 𝑏𝑞𝑡 ) − (𝑐 + 𝑑𝑞t + 𝑚)

The protection level of intellectual property is positively
related to technical innovation in developing countries[4].
The protection of intellectual property is an important

The enterprise wants the products to enter the market assoon as possible, but if the enterprise complete the technicalinnovation through its own, it need a lot money for the fixed

determinant contributing to the economic growth and it

cost.

shows positive impact more significantly in economic

B)

Collaborative innovation

growth in developing countries[5]. There are two different

In this mode, the enterprise needs to choose aattitudes about the protection enforcement of intellectual

cooperative partner that owns product brands and R&D

property in China between Chinese scholars. One is that theresources

protection intensity of intellectual property has gone beyond

distribution ratio depends on the conventions of the two

what the Chinese economy can take, thus hampering theenterprisesforcollaborative

innovation.

The

profits

economic development in China. Another is that there still
exists a big gap between the protection intensity ofintellectual property and the required level of technicalinnovation. Therefore, a strengthened the intellectual

The outputs of product decision-making occurs indiscrete time period t, t = 0,1,2,3, …，qi,t is the productoutput of enterprise i in time period t, i = 1,2. t The priceof time period t is depend on the output of product:

property protection is badly needed.

p = a − b(q1,t + q2,t ) ，a, b are constants, a is the

The protection of intellectual property is not very

highest market price of the product. The enterprise cost

important for SMEs. Chinese SMEs in manufacturing arefunction: C = ci + di qi,t + ei t + mi , i = 1,2, ci is the fixedlack of personnel and financial resources in the early stage

cost of the enterprise i, di qi,t is the variable cost of theof development, a more realistic approach is to let theirenterprise i , t is the total technological investment,

products go to the market as soon as possible to meet theincluding the protection of intellectual property costs. ei isrelated market demand and earn the initial higher profit

the technological investment proportion of the enterprise i,

margin in the market.

mi is the opportunity cost of the enterprise i to produce theII．

product. e2 = 1 − e1 The profit functions of the enterprises

GAME MODEL

Now if Chinese SMEs have low technology level for
production, they have three alternative development modes:

in time period t are respectively:
𝐿1,𝑡 = 𝑞1,𝑡 [𝑎 − 𝑏(𝑞1,𝑡 + 𝑞2,𝑡 )]itself;− (𝑐1 + 𝑑1 𝑞1,𝑡collaborative innovation with a partner who owns product+ 𝑒1 𝑡 + 𝑚1 )technical

innovation

throughtheenterprisebrands and R&D resources; sell technology to a large

𝐿2,𝑡 = 𝑞2,𝑡 [𝑎 − 𝑏(𝑞1,𝑡 + 𝑞2,𝑡 )]enterprise.

− [𝑐2 + 𝑑2 𝑞2,𝑡A)

+ (1 − 𝑒1 )𝑡 + 𝑚2 ]Technical innovation through the enterprise itself
The outputs of product decision-making occurs in

The profits distribution ratio depends on the convention

discrete time period t，t = 0,1,2,3, …, qt is the output ofof the two enterprises and the convention is about technicalproduct in time period t . The price of time period t

investment. Assuming the ratio of technical investment for

depends on the output of product: p = a − bqt , a, b areeach of the two enterprises is consistent. The profit margin© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

15

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

𝐿1 ∗ = 𝑞1 ∗ [𝑎 − 𝑏(𝑞1 ∗ + 𝑞2 ∗ )]of the first enterprise extract from the second enterprise isε1 , and the profit margin of the second enterprise extract− (𝑐1 + 𝑑1 𝑞1 ∗from the first enterprise isε2 . The final profit functions of the+ 𝑒1 𝑡 + 𝑚1 )two enterprises in time period t are respectively:
𝐿2 ∗ = 𝑞2 ∗ [𝑎 − 𝑏(𝑞1 ∗ + 𝑞2 ∗ )]𝐿1,𝑡 ’ = (1 − 𝜀2 )𝐿1,𝑡 + 𝜀1 𝐿2,𝑡

− [𝑐2 + 𝑑2 𝑞2 ∗𝐿2,𝑡 ’ = (1 − 𝜀1 )𝐿2,𝑡 + 𝜀2 𝐿1,𝑡
Repeatedgame

production

dynamic+ (1 − 𝑒1 )𝑡 + 𝑚2 ]adjustment

𝐿1 ∗ is related to 𝑎, 𝑏, 𝑑1 , 𝑑2 , 𝑐1 , 𝑒1 , ε1 , ε2 .

mechanism can be expressed as:
𝑞1,𝑡+1 = 𝑞1,𝑡 + 𝛼1 𝑞1,𝑡

𝜕𝐿1,𝑡 ′
𝜕𝑞1,𝑡

𝜕𝐿2,𝑡 ′
𝑞2,𝑡+1 = 𝑞2,𝑡 + 𝛼2 𝑞2,𝑡𝜕𝑞2,𝑡α1 , α2 are the adjustment speed of the two enterprise.
𝜕𝐿1,𝑡 ′= (1 − 𝜀2 )(𝑎 − 𝑏𝑞2,𝑡 − 2𝑏𝑞1,𝑡
𝜕𝑞1,𝑡− 𝑑1 ) − 𝜀1 𝑏𝑞2,𝑡
𝜕𝐿1,𝑡 ′= (𝜀2 − 𝜀1 − 1)𝑏𝑞2,𝑡
𝜕𝑞1,𝑡

The two enterprises get the equilibrium profit after
repeated games. The relationship between the twoenterprises is cooperation and competition at the same time,the enterprises take into account the long-term interests ofthe enterprises, so they do not rely on one-sided increase
production to obtain short-term gains, but more emphasis onbalancing the profit that can be obtained after repeatedgames. However, the enterprises in cooperation should
adjust the distribution of profits ε1 and ε2 between them,because the value of ε1 and ε2 might impact the total
output and profit. When one of the enterprises feel unfair,− 2𝑏(1 − 𝜀1 )𝑞1,𝑡

the balance may be upset.

+ (𝑎 − 𝑑1 )(1 − 𝜀1 )C)

𝑞1,𝑡+1 = 𝑞1,𝑡 + 𝛼1 𝑞1,𝑡 [(𝜀2 − 𝜀1− 1)𝑏𝑞2,𝑡− 2𝑏(1 − 𝜀2 )𝑞1,𝑡Sell technology to a large enterprise
In this situation, the enterprise sells technology to a large

enterprise, and earns the value of the technology f. Whenthe money resources is accumulated to a higher level, it will

+ (𝑎 − 𝑑1 )(1enable SMEs to employ and keep higher level technical− 𝜀2 )]

talents, to pay for the R&D, to increase the added-value of

𝑞2,𝑡+1 = 𝑞2,𝑡 + 𝛼2 𝑞2,𝑡 [(𝜀1 − 𝜀2the products, earn more money and finally form the− 1)𝑏𝑞1,𝑡enterprise into the high-technology one.

− 2𝑏(1 − 𝜀1 )𝑞2,𝑡D)+ (𝑎 − 𝑑2 )(1− 𝜀1 )]Nash Equilibrium Point(q1 ∗ , q2 ∗ ):𝑞1 ∗(1 − 𝜀1 )[(𝑎 − 𝑑2 )(𝜀2 − 𝜀1 − 1) + 2(𝑎 − 𝑑1 )(1 − 𝜀2 )]=4𝑏(1 − 𝜀1 )(1 − 𝜀2 ) − (𝜀1 − 𝜀2 − 1)(𝜀2 − 𝜀1 − 1)𝑏∗𝑞2(1 − 𝜀2 )[(𝑎 − 𝑑1 )(𝜀1 − 𝜀2 − 1) + 2(𝑎 − 𝑑2 )(1 − 𝜀1 )]=4𝑏(1 − 𝜀1 )(1 − 𝜀2 ) − (𝜀1 − 𝜀2 − 1)(𝜀2 − 𝜀1 − 1)𝑏

Analysis
As the enterprises face different situation, they canchoose different modes to earn the profits.
Technical innovation through the enterprise own:
L_t1 = q_t (a − bq_t ) − (c + dq_t + m)
Collaborative innovation with a cooperative partner who
owns product brands and R&D resources:

q1 ∗ is related to a, b, d1 , d2 , ε1 , ε2 .

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

Lt2 = q1 ∗ [a − b(q1 ∗ + q2 ∗ )]− (c1 + d1 q1 ∗+ e1 t + m1 )16

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Selling technology to a large enterprise:technology and low value-added product, we can draw the

Lt3 = fIf the profits of selling technology is more than the

conclusion that in the current situation, only the low

production profit obtained in the equilibrium point, it iscapabilities or support resources and they could havebetter to choose selling the technology.

realized the transformation of enterprises to enhance theComparewithtechnical

innovation

throughthetechnology manufacturing enterprises have the financial

capability of independent innovation.
The

enterprise own mode and collaborative innovation with aChinese SMEs inmanufacturing

are

lack

cooperative partner who owns product brands and R&D

of personnel and financial resources in the early stage ofresources mode, assuming that the enterprise have the samedevelopment, a more realistic approach is to put out theopportunity cost, the products have the same price andproducts in the market as soon as possible in one out of thevariable cost.

three appropriate collaborative innovation ways mentioned
in this paper to meet the related market demand and earn the

𝐿𝑡2 − 𝐿𝑡1 = (𝑎 − 𝑑)(𝑞1 ∗ − 𝑞𝑡 )initial higher profit in market. Under these circumstances, in+ 𝑏[𝑞𝑡 2 − (𝑞1 ∗ )2

the early development of a SME the more effective method− 𝑞1 ∗ 𝑞2 ∗ ]is to choose a partner that owns product brands and R&D+ (𝑐 − 𝑐1 ) − 𝑒1 𝑡

resources for collaborative innovation. When the money

Assuming that total outputs of product are equal,
q1 ∗ + q2 ∗ = qt

resources is accumulated to a higher level, it will enable
SMEs to employ and keep higher level technical talents, so
as to pay for the R&D, increase the added-value of the

𝐿𝑡2 − 𝐿𝑡1 = 𝑞2 ∗ [−𝑎 + 𝑏(𝑞1 ∗ + 𝑞2 ∗ )products, earn more money and finally develop the+ 𝑑] + (𝑐 − 𝑐1 )enterprise into a high-tech one.

− 𝑒1 𝑡If the fixed cost investment of the enterprises in the first

ACKNOWLEDGMENTS.

mode is more than the sum of profits of the second
enterprise and investment in fixed costs and technologicalThis work is supported by National Key Technology
R&D Program under Grant No. 2015AA043702.

innovation of the first enterprise in the second mode, the
REFERENCES

more effective method is to choose a partner who owns
product brands and R&D resources for collaborative[1]

innovation.
[2]

Generally speaking, China has a huge market, assumingthat the number of product sales depends on output of
product,q1 ∗ = qt

[3]
∗

∗

𝐿𝑡2 − 𝐿𝑡1 = −𝑏𝑞1 𝑞2 + (𝑐 − 𝑐1 )[4]

− 𝑒1 𝑡If the fixed cost investment of the enterprises is morethan bq1 ∗ q2 ∗ + c1 + e1 t, the more effective method is tochoose a cooperative partner who owns product brands and[5]Feng, Ying: Research on the causes and countermeasures for the
low patent transformation rate. Anhui Science and Technology, vol.
8, pp. 36-37.(2012) (in Chinese)
May, P., Ehrlich, H.C., Steinke, T.: ZIB Structure Prediction
Pipeline: Composing a Com- plex Biological Workflow throughWeb Services. In: Nagel, W.E., Walter, W.V., Lehner, W. (eds.)
Euro-Par 2006. LNCS, vol. 4128, pp. 1148–1158. Springer,
Heidelberg (2006)
Foster, I., Kesselman, C.: The Grid: Blueprint for a New Computing
Infrastructure. Mor- gan Kaufmann, San Francisco (1999)
Czajkowski, K., Fitzgerald, S., Foster, I., Kesselman, C.: Grid
Information Services for Distributed Resource Sharing. In: 10th
IEEE International Symposium on High Perfor- mance Distributed
Computing, pp. 181–184. IEEE Press, New York (2001)
Foster, I., Kesselman, C., Nick, J., Tuecke, S.: The Physiology of
the Grid: an Open Grid Services Architecture for Distributed
Systems Integration. Technical report, Global Grid Forum (2002)

R&D resources in a way of collaborative innovation.
III． CONCLUSIONS
Through the above analysis of collaborative innovation
situationofChinese SME

manufacturers

with

low

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

17

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Computerizing More Repeatable Trail Making Test
for Longitudinal Cognitive Assessment
Zhiwei Zeng† , Chunyan Miao†‡ , Cyril Leung† , Zhiqi Shen†‡† Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly,‡ School of Computer Science and Engineering

Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798
Email:{zwzeng, ascymiao, cleung, zqshen}@ntu.edu.sg

Abstract—A decline in cognitive functions may be an indicatorof increased fall risk or a precursor to other serious healthproblems such as dementia. Longitudinal neuropsychologicalassessment is often used to track changes in cognitive functions.
The Trail Making Test (TMT) adapted by Reitan is one of
the most frequently used neuropsychological tests for assessing cognitive status. However, Reitan’s TMT is unsuitable forlongitudinal assessment due to significant practice effects whenadministered repeatedly. To overcome this issue, we propose
a divide-and-combine (DAC) approach for generating different
instances of TMT that have a similar level of difficulty to Reitan’s
TMT and can be used in repeated assessments with nearly no
discernible practice effects. In the DAC approach, partial trails
are generated separately in different layers and then combined
to form a complete TMT trail. The proposed approach was
implemented in a computerized test application called iTMT. A
pilot study was conducted to evaluate the proposed approach
using this application. The results show that the instances of
TMT generated by the approach had a similar level of difficulty to
Reitan’s TMT. iTMT also achieved a stronger construct validity,
higher test-retest reliability and significantly reduced practiceeffects than existing computerized tests. These preliminary results
suggest the prospect of using iTMT in longitudinal cognitive
assessment for fall prevention and early diagnosis of dementia.

I. I NTRODUCTION
Both falls and dementia are major health concerns among
the elderly. About one third of the elderly over 65 years
old fall each year [1]. Falls also account for about 10% ofvisits to hospital emergency departments among elderly [2].
On the other hand, a new dementia case is reported every 3.2
seconds, and the cost of dementia represented 1.09% of global
GDP in 2015 [3]. A decline in cognitive functions has been
associated with increased risk of fall [4] and identified as aprecursor syndrome to dementia [5]. Thus, monitoring changes
in cognitive functions may be helpful for fall prevention, as
well as early diagnosis and intervention of dementia.
Repeated assessments are required to track the changes incognitive functions timely and effectively. The Trail Making
Test (TMT) is one of the most frequently employed neuropsychological tests for cognitive assessment [6] due to its
sensitivity, simplicity and ease of administration. The current
version of TMT is adapted by Reitan in 1955 [7] (thereafterreferred to as Reitan’s TMT), which has only one instanceand is administered using paper and pencil. However, therepeated use of Reitan’s TMT in longitudinal assessment isseverely limited by its high susceptibility to practice effects[8]. Practice effects refer to improvements in test performancethat occur when a subject is retested on the same instance,or tested repeatedly on very similar ones. It is hard tosegregate performance improvements due to practice effects
from meaningful cognitive changes. This affects test accuracy
and reliability. Increasing the test-retest interval may attenuate
practice effects, but this may obscure the timely detection of
meaningful cognitive changes [9].
Some researchers have proposed to use alternative forms[10] and mirror images [11] of TMT serially in consecutive
test administrations to reduce practice effects. However, thenumber of equivalent alternative forms and mirror images of
TMT is limited. Other researchers have proposed more systematic and divergent approaches for generating new instancesof TMT [12], [13]. Although their approaches could generatetheoretically unlimited instances of TMT, the generated instances may be less difficult than Reitan’s version – they may
have shorter average trail length and less visual interference
than the instance in Reitan’s version. Consequently, the generated instances may have poorer diagnostic efficacy and lessdiscriminating power to distinguish among different cognitive
status. Moreover, as a cognitive assessment tool, the generated
instances should be assessed for their validity and reliability,
yet to the best of our knowledge, none of these two approaches
have been validated in user studies.
In order to generate TMTs that can be used in repeated assessments, we propose a divide-and-combine (DAC) approach– a systematic approach for generating instances of TMT
which:
1) are sufficiently different from each other to reduce
practice effects when used in consecutive test administrations; and2) have a similar level of difficulty and thus diagnostic
power to Reitan’s TMT.
To achieve (1), the proposed approach uses pseudorandomized processes to generate different instances. To
achieve (2), it attempts to reproduce the spatial characteristics
of Reitan’s TMT to the greatest extent possible. Accordingto [12], trails in Reitan’s TMT are self-avoiding and gradually unwind in a clockwise or anticlockwise direction. To
reproduce these characteristics, our DAC approach gener-© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

18

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

ates sub-solutions in divided problem spaces and combines
sub-solutions to form a complete solution. In the ”divide”
phase, the test region is divided into several nested andnon-overlapping layers. Within each layer, a partial trail isgenerated with the desired spatial characteristics. Then, in the”combine” phase, the partial trails are joined together to form
a complete trail while preserving the desired characteristics.
With the increasing penetration and improved usabilityof digital devices, there has been a number of attempts toadminister TMTs using digital devices, e.g. computers. Computerized tests can facilitate more standardized and accurate
data capturing as well as support detailed analysis. They
also open the possibility to implement unlimited instances of
TMT using systematic approaches. In order to evaluate our
DAC approach, we created a test application, called iTMT,
which implements the DAC approach to generate computerized TMTs. A pilot study was conducted using this applicationand involving 10 participants with different levels of cognitive
abilities. The pilot study results suggest that no significant
difference exists between the computerized tests generated
by our DAC approach and Reitan’s TMT in terms of total
segment length and visual interference, indicating that they had
a similar level of difficulty. Moreover, iTMT also demonstrated
stronger construct validity, higher test-retest reliability and significantly reduced practice effects than existing computerized
tests. These preliminary results support the effectiveness of the
DAC approach and indicate that iTMT is a promising tool in
longitudinal cognitive assessment.
In the following sections, we first introduce the neuropsychological background of TMT. Then, we review the priorefforts made to adapt it for longitudinal assessment and tocomputerize it. To address the issues identified from prior
works, we propose our DAC approach in section IV. In section
V, we describe how we computerized our DAC approach andcreated iTMT. The pilot study results are presented in section
VI. Finally, the main findings are presented in section VII.
II. N EUROPSYCHOLOGICAL BACKGROUND OF TMTThe Trail Making Test (TMT) is one of the most frequently
employed neuropsychological tests [6] due to its sensitivity,
simplicity and ease of administration. The TMT was originallyconstructed to assess general intelligence in Army Individual
Test of General Ability [14]. It was later adapted by Reitan andincluded in the Halstead-Reitan Neuropsychological Battery[7] in its current form. It is often used as a diagnostic tool
to detect cognitive impairments due to brain damage, e.g.
dementia, stroke and traumatic brain injuries [15], [16].
During test administration of Reitan’s TMT, the subject
is instructed to connect a set of dots as quickly as possible
while maintaining accuracy. Reitan’s version only has one test
instance and uses fixed dot arrangements. It contains two parts.
As shown in Fig. 1, Part A involves connecting 25 numbereddots in increasing order. Part B involves connecting 25 dots
labelled with numbers and letters, in the alternating sequence”1-A-2-B-3-C. . . ”. The test is meant to assess cognitive
functions such as cognitive flexibility, executive functioning,(a) Part A(b) Part B

Fig. 1. Reitan’s Trail Making Test

mental processing speed, divided attention as well as visualscanning [17]. Part B is more difficult than Part A, possibly due
to differences in symbolic complexity and spatial arrangement[18]. While Part A only contains numerals, Part B involves two
symbol systems, alphabets and numerals, making it a moredifficult cognitive task. Besides, segment length and amountof visual interference are also factors affecting the difficultyof TMT. Part B has a longer total trail length and more visualinterfering stimuli than Part A, making it more demanding interms of motor speed and visual scanning [19].
Reitan’s TMT is performed using paper and pencil and isadministered by a healthcare professional. Time to completionand frequency of errors are the most common metrics which
are recorded and used to interpret TMT performance of both
Part A and Part B.
III. R ELATED W ORK
A. Adapting TMT for Longitudinal Assessment
Although Reitan’s TMT is widely used for assessing cognitive abilities, its use in longitudinal assessment is constrained
due to its high susceptibility to practice effects [8]. Eachrepeated administration of it uses the same test instance, which
decreases its sensitivity.
Atkinson and Ryan proposed to use other validated neuropsychological tests as alternative forms of TMT in longitudinal assessment [10], [20]. They identified three alternative
forms that are equivalent and can be used interchangeably in a
serial manner with a brief test-retest interval. However, due totheir similarity in content and format, the use of alternative
forms can only slightly reduce practice effects during longitudinal assessment. Instead of using other validated tests,
Wagner et al. proposed to create new instances of TMT using
its mirror images [11]. Similar to the use of alternative forms,
the mirror images also exhibited discernible practice effects
when used serially in assessment.
The number of equivalent alternative tests and mirror imagesof TMT is limited. Hence, it is still not practical to adopt
the aforementioned two approaches in longitudinal cognitive© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

19

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)assessment over a long time period. A group of researchers[12], [13] have proposed more systematic and divergent approaches for generating a theoretically unlimited number of
TMT instances. They contended that, although it is not clear
whether Reitan’s TMT was constructed according to some
implicit principles, it is vastly different from trails that are
generated by purely random processes [12]. They observed
two characteristics of Reitan’s TMT trails. Firstly, for both
Part A and Part B, the trails gradually uncoil in either aclockwise or an anticlockwise direction from the inner toouter part. Secondly, the trails are self-avoiding, i.e. the line
segments connecting consecutive points have no intersections
with one another. However, it is a non-trivial problem togenerate new instances of TMT which are endowed with thesetwo characteristics [13]. Working reversely from the desired
characteristics, two approaches to generate self-avoiding trails
were proposed in [12] and [13].
The first approach is suggested by considering the fractal
nature of TMT trails. The problem of generating new TMT
instance is transformed into the problem of generating selfavoiding fractal curve [12]. Starting with a seed element,
fractal curves are generated by repeatedly applying a set oftransformations to the seed. The second approach transforms
the problem of generating new TMT instance into a Travelling
Salesman Problem (TSP) [13]. Dots in TMT are treated as
cities in TSP which can only be visited exactly once in a trail.
The solution to TSP, the shortest path, is typically self-avoiding
as having intersections in the trail tends to lengthen the path.
Thus, given a set of dots, the solution to the corresponding
TSP can be converted into a new instance of TMT.
Although both the fractal curve and the TSP approach cangenerate a theoretically unlimited number of TMT instances,
the generated instances may have shorter average trail lengthand less visual interference than Reitan’s paper-and-pencil
TMT. Consequently, these instances may be less difficult than
Reitan’s version and less demanding in terms of the cognitive
functions assessed. For example, less visual interfering stimuli
would make the visual scanning much easier. Moreover, whentest performance is measured by time to completion, the total
segment length will also affect the test performance as it affects the drawing time. Due to their reduced level of difficulty,
the generated instances may not have enough discriminative
power to produce statistically significant performance differences between different cognitive status. Consequently, their
sensitivity and diagnostic efficacy may also be compromised.
B. Computerized TMT
Unlike manually administered tests which are susceptible to variations in test administrator and test procedures,computerized tests maintain standardized test procedures andare consistent across subjects [21]. Smith developed a computerized touch-screen version of TMT, eTrails [22], whichcontains a digital embodiment of the paper-and-pencil TMT
and four computerized variants of the standard test. Experimental results suggest that all the five computerized tests
have considerably higher test-retest reliability than TMT, pos-sibly due to standardized procedures and less administrative
errors. Another benefit brought by computerized tests is theability to capture more high-fidelity data which increase theaccuracy of performance measurements and facilitate in-depth
performance analysis. Woods et al. developed a computerized
TMT, C-TMT, which supports segment-by-segment analysis ofperformance and separates analysis of time spent on different
tasks, e.g. dwelling and moving [21].
More importantly, computerized tests also provide the ideal
format for generating a theoretically unlimited number of
TMT instances. Following algorithmic approaches, pseudorandomized dot arrangements can be generated for each test
administration. Compared to Reitan’s TMT, the practice effects
for both eTrails and C-TMT were significantly attenuated [21],[22]. Yet, it was not clearly indicated that how the instancesof TMT were generated in these two computerized tests. The
fractal curve approach and the TSP approach introduced in theprevious section are both systematic and divergent approaches
rooted in sound theoretical basis. However, to the best of ourknowledge, neither of them has been formally computerized
and validated in user studies.
We have reviewed the prior efforts made to adapt TMT for
longitudinal assessment and to computerize TMT, as well as
discussed the issues with these attempts. Next, we will propose
our solution to the issues.
IV. T HE D IVIDE - AND -C OMBINE A PPROACH
In this section, we propose a divide-and-combine (DAC)
approach for generating different instances of TMT in asystematic manner. As introduced in section I, the generated
instances need to be sufficiently different from each otherwhile having a similar level of difficulty to Reitan’s TMT.
More specifically, the aim of the DAC approach is to produce
different sets of ordered dots in a rectangular test region, sothat the trail obtained by connecting dots in each set according
to their order exhibits the following characteristics:• Uncoiling in clockwise or anticlockwise direction.
• Self-avoiding.
The DAC approach consists of two phases. In the ”divide”
phase, the test region is divided into several nested andnon-overlapping layers. Within each layer, a partial trail isgenerated with the desired spatial characteristics. Then, in the”combine” phase, the partial trails are combined together toform a complete trail.
Suppose that we want to generate instances of TMT that
contains n dots in a rectangular test region on the x-y plane.
The plane spans across the area x ∈ [a, b] and y ∈ [c, d], wherea < b and c < d.
Suppose that we use m layers, where m ∈ N+ . There areno constraints on the shape of the layers. However, the layers
defined should satisfy two requirements. Firstly, in order tomake the generated trail follow an unwinding pattern, thelayers need to be nested. Secondly, the layers should be nonoverlapping to reduce intra-layer intersections.
Definition 1: A layer Li , where i = 1, 2, · · · , m, is defined
as a 4-tuple Li = {di , Ai , Pi , Si }:© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

20

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Algorithm 1 The divide-and-combine approach
1: var Pregen2: var ppivot. The dots that need to be regenerated. The pivot dot for combing two sub trails

3: for each Li do
4:5:. The ”divide” phase

Pi ← randomDots(Ai , di × n)
Si ← sortDots(Pi )6: for each Li do. The ”combine” phase7:
while intersect(Ti−1 , Ti )! = N U LL do
8:Pregen ← intersect(Ti−1 , Ti )9:Pi ← regenerateDots(Pi , Pregen )10:Si ← sortDots(Pi )11: for every Li and Li+1 do
12:
ppivot ← pickP ivot(Si , Pi , Si+1 , Pi+1 )13:
Si+1 ← connect(Si , Si+1 , ppivot )14: for each Pi do
15:
assignLabel(Pi , Si )1) di ∈ R>0 represents the relative density of the dots inlayer Li (refer to Definition 2). di × n gives the numberof dots in layer Li .
2) Ai defines the area layer Li spans.
3) Pi = {pj = (xj , yj )|xj ∈ [a, b]; yj ∈ [c, d]; j =1, 2, · · · , di × n} represents the dots in layer Li .
4) Si = {s(k)|s(k) ∈ [1, di × n]; k = 1, 2, · · · , di × n}
represents the order of the dots in layer Li . To form apartial trail Ti , the dots are connected according to thesequence of ps(1) → ps(2) → · · · → ps(di ×n) .
Definition 2: The density of dots Di in a layer Li can be
defined as:
|Pi |
di =
(0 < di 6 1)n
di is a measure of relative density and is the ratio of numberof dots in layer Li (the cardinality of Pi ) to the total numberof dots (n).
The pseudo code of the DAC approach is outlined in
Algorithm 1. In the ”divide” phase (line 3-5), a partial trail Ti
is generated within each layer. For each layer Li , the dots are
randomly generated within the layer and then sorted to determine their connecting order. The function randomDots()
is called to generate di × n number of dots in the area definedby Ai . Then, the function sortDots() is invoked to sort thedots in layer Li with respect to an anchor point and return
their sorted order Si . For the simplicity, the bottom left dot
in the layer can be chosen as the anchor point in sorting. The
dots are sorted according to their angles with respect to theanchor point. A self-avoiding partial trail can be formed betraversing the dots by either increasing or decreasing angle.
Fig. 2 illustrates sorting by increasing angle.
In the ”combine” phase (line 6-15), the partial trails in

Fig. 2. Sorting the dots in a layer (increasing angle)

adjacent layers are combined together to form a complete trail.
Before connecting the partial trails, a round of checking is
performed to ensure that there are no intersections among them(line 7-10). The function intersect(Ti−1 , Ti ) is used to detect
intersections between the partial trails in two adjacent layers,
Ti−1 and Ti . It returns the end points of line segments in theouter layer Ti that intersects with Ti−1 . In the case when two
trails intersect, i.e. intersect(Ti−1 , Ti )! = N U LL, the trail inthe outer layer will be adjusted to eliminate the intersection.
The end points of the intersecting segments in Ti (or Pregen )are regenerated by calling the function regenerateDots().
With the newly generated dots, the dots in layer Li are resorted by calling sortDots() again. When no intersections arefound, all the partial trails are connected together to form acomplete trail. By calling connect(), each two partial trails inadjacent layers are joined through a pivot dot selected from theouter layer Li+1 . The function pickP ivot() is called to choosea pivot from Pi+1 to connect Ti and Ti+1 . The choice of pivot
dot should guarantee that no intersection will be resulted fromjoining it and the last dot in the inner layer. Fig. 3 illustrates
connecting two trails Li−1 and Li , where ps(3) in Li is chosen
as the pivot dot. Finally, alphanumerical labels are assigned tothe dots based on their order in the complete trail.

Fig. 3. Connecting two trails in adjacent layers with a pivot

With the algorithm described above, the DAC approach isable to generate TMT instances that reproduce the two desired spatial characteristics: unwinding clockwise or counter-© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

21

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

clockwise and self-avoiding. In the following part of thissection, we will explain how the design of the DAC approach
help to reproduce these characteristics.
Unwinding clockwise or counter-clockwise The DAC
approach uses both intra-layer and inter-layer designs to
reproduce this characteristic. The intra-layer strategy ensures
the partial trails exhibits the desired pattern locally within alayer, while the inter-layer strategy ensures that the pattern isembodied globally by a trail.
Within a layer (intra-layer), sorting and connecting the dots
according to their angles with respect to the anchor point
endows a partial trail with the desired unwinding pattern. As
shown in Fig. 2, the dots in a layer can also be viewed in apolar coordinate system with the anchor point as pole. Eachdot can be represented by a radial coordinate (r, θ), whereθ ∈ [0, 2π]. Connecting the dots according to the value of θ
would form a trail that starts from and then zigzags aroundthe anchor point. Visually, it can be viewed as a trail gradually
uncoiling from the anchor point. The direction of unwinding
can be controlled by the sorting order. If the angles are sorted
in ascending order, the partial trail unwinds counter-clockwise.
Conversely, the partial trail unwinds clockwise.
At the inter-layer level, the definition of the layers help tocreate a complete trail in the desired unwinding pattern. The
layers defined must be nested. This requirement ensures thatthe area of an inner layer is surrounded by its outer layers. The
trail formed by joining partial trails within nested layers also
appears to be nested, which gradually unfolds when traversing
from an inner layer to an outer layers.
Self-avoiding In order to generate self-avoiding trails, theDAC approach also employs intra- and inter-layer strategies.
The intra-layer strategy ensures each partial trail formed isself-avoiding. The inter-layer strategy generate a self-avoiding
trail by connecting the partial trails while avoiding intersections in the connecting process.
At the intra-layer level, sorting and connecting dots byangles ensures that the resultant partial trail is self-avoiding.
Referring to Fig. 2 again, the area that spans across [0, 2π] is
divided into sectors by the dotted lines connecting the anchor
point to other dots. These sectors are contiguous but nonoverlapping. Each line segment of a partial trail spans within
exactly one of the sectors. Thus, the line segments have nointersections with each other, except for the joints.
At the inter-layer level, using non-overlapping layers canhelp to reduce intersections among partial trails in different
layers. However, a partial trail formed by traversing the dots
within a layer may still cut through other layers. Intersection
checking is performed to detect and eliminate intersections insuch cases, as described by Algorithm 1 line 6 to line 10.
Moreover, the choice of pivot dot while connecting adjacent
trails also avoids creating intersections in the combined trail.
The DAC approach is a generalized approach that is suitablefor generating trails for both TMT Part A and Part B. It
maintains the modelling flexibility to accommodate the differences between Part A and Part B. The dots in Part A scatter
more evenly over the rectangular test region, while the dots

in Part B are more skewed towards the rim of the area. There
are a number of strategies (see Table IV) to manipulate the
distribution of the dots to cater to such differences.
TABLE IS TRATEGIES FOR GENERATING TMT PART A AND PART B WITH THE
DIVIDE - AND - COMBINE APPROACH

Parameter toadjustPart APart B

Relative
densityofdots in a layer,diAssign larger di toinner layers, smaller
di to outer layers

Assign smaller di toinner layers, larger dito outer layers

Area of a layer,
Ai

Define smaller Ai for
inner layers, larger
Ai for outer layers

Define larger Ai for
inner layers, smaller
Ai for outer layers

Total numberof layers, m

Use larger m for PartA

Use smaller m

In this section, we proposed the DAC approach for generating different instances of TMT. In the following sections,
we will describe how this algorithmic approach can be implemented and built into computerized TMTs.
V. C OMPUTERIZING TMT U SING THE
D IVIDE - AND -C OMBINE A PPROACH
We created a test application called iTMT, which implements the DAC approach with the parameters determined from
Reitan’s paper-and-pencil TMT.
After examining the dot arrangement in Reitan’s TMT, wechose to use 25 dots and 3 layers for both Part A and PartB, i.e. n = 25, m = 3. For the ease of modelling, we used
3 concentric layers, with each of their centres at the origin.
As illustrated in Fig. 4, L1 is a rectangle, while L2 andL3 are hollow rectangles. Collectively, the 3 non-overlapping
layers cover the test region exhaustively. The values of di arepresented in Table II. Compared to Part A, Part B has smaller
relative density in L1 and L2 , but much greater density in L3 ,which is designed in accordance with the observations madefrom the paper-and-pencil TMT.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

Fig. 4. The layers used (m = 3)

22

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IIR ELATIVE D ENSITIES di F OR E ACH L AYER IN PART A AND PART B

Test

d1

d2

d3

Part A

0.24

0.36

0.4

Part B

0.2

0.2

0.6

Fig. 5. A test screen of iTMT (Part A)

The area each layer spans can be defined as following:
|x| ≤ x1 , |y| ≤ y1
Ai spans the area x1 < |x| ≤ x2 , y1 < |y| ≤ y2
x2 < |x| ≤ x3 , y2 < |y| ≤ y3

i=1
i=2
i=3

Following the strategies in Table IV, Ai is defined differently for Part A and Part B. Comparatively, for Part B, A1
spans across larger area, while A2 and A3 are made smaller
so that more dots would be placed near the edge of the test
region.
Using the aforementioned parameters, we built the test
application, iTMT. It is a touch-screen application that canadminister computerized tests on tablets. Instead of using apencil, the subjects need to connect the dots using their fingers.
During each administration, iTMT generates a new instance of
TMT using the DAC approach. Fig. 5 shows a test screen of
iTMT, where Part A was being administered.
VI. E VALUATING C OMPUTERIZED TMT
In order to evaluate the DAC approach, a pilot study was
conducted for iTMT. The instances of computerized TMT
generated with the DAC approach was evaluated for thefollowing aspects:
Level of difficulty: whether the generated instances areequivalent to Reitan’s paper-and-pencil TMT in terms oftest difficulty (as measured by total segment length andvisual interference).
• Construct validity: whether the generated instances areable to produce test results that are comparable to Reitan’s•

paper-and-pencil TMT (as measured by time to completion).
• Test-retest reliability: whether the test results are consistent across different instances (as measured by time
to completion) and whether there are significant practiceeffects.
The study involved 10 participants aged from 33 to 84, with
8 above 50 and 4 above 65. As age is an important factor
which affects cognitive abilities, we recruited participants froma broad age range in order to include people with different
levels of cognitive abilities. Considering that some elderly
participants are illiterate or only had limited education, onlyPart A of TMT was administered. Each subject was administered 3 tests consecutively, including 1 paper-and-pencil test
and 2 computerized tests on a tablet. The computerized tests
were generated by iTMT using the DAC approach. As the sizeof test regions may affect the difficulty metrics, we used anadapted version of Part A in Reitan’s TMT for the paper-andpencil test to control this factor. Reitan’s Part A is compressed
proportionally into a smaller test region which has an equalarea as the screen of the tablet used, with the relative positionsof the dots preserved.
Before conducting the tests, the participant was shown asample TMT with 8 dots so as to become familiar with theconnecting rules and test procedures. Then, the paper-andpencil test was administered to the participant. The time taken
to complete the trail, i.e. time to completion, was recordedwith a timer. After an interval of 30 seconds, two computerized
tests were administered to the participant consecutively. The
test statistics were recorded by iTMT for later analysis.
A. Level of Difficulty
As proposed in [19], we use total segment length andvisual interference as two metrics for the difficulty of TMTs.
Total segment length is defined as the summed length of
the shortest line segments connecting successive dots. Visualinterference is quantified by summing the number of dotsthat lie within 3cm of each line segment. The two metricsof the paper-and-pencil TMT administered in the study were
calculated and used as normative values for evaluating thecomputerized tests. The paper-and-pencil TMT has a total
segment length of 135.9cm and visual interference of 49. For
each of the 20 computerized tests (2 per participant × 10)
administered during the study, its total segment length andvisual interference was calculated. Two t-tests were performed
to determine whether the computerized tests generated by
iTMT have a similar level of difficulty to the paper-and-pencil
test. One t-test compared the mean of total segment length(µseg ) of computerized tests and the total segment length of thepaper-and-pencil test with the null hypothesize µseg = 135.9.
As p = 0.33 > α = 0.05 (t0 = 0.98 < t0.05,19 = 2.093),
we accept the null hypothesize. The other t-test compared themean of visual interference (µvis ) of computerized tests andthe visual interference of the paper-and-pencil test with thenull hypothesize µvis = 49. Since p = 0.31 > α = 0.05
(t0 = 1.04 < t0.05,19 = 2.093), we also accept H0 .

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

23

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

The test results suggest that there is no significant difference
between the computerized tests and the paper-and-pencil test
in terms of total segment length and visual interference. Hence,
the results further indicate that, as measured by these two
metrics, computerized tests and the paper-and-pencil test canbe considered to have a similar level of difficulty.
TABLE IIIH YPOTHESIS T ESTS FOR S EGMENT L ENGTH AND V ISUAL I NTERFERENCE(α = 0.05)
H0x̄

p valuet scoret0

ResultSegment
lengthµseg = 135.9

141

0.33

0.98

2.093

Accept H0

Visual interferenceµvis = 49

51.19

0.31

1.04

2.093

Accept H0(a) Paper-and-pencil test vs 1st iTMT test

B. Construct Validity
The construct validity of the computerized tests generated
by iTMT was measured by correlating the time taken by thesame participant to complete a computerized test and thepaper-and-pencil test. The Pearson Product-Moment correlations between time to completion are shown in Fig. 6. The
time taken by a participant to complete the 1st computerized
test was positively correlated with the time he/she needs tocomplete the paper-and-pencil test (r = 0.89, p = 0.0006).
Positive correlation was also reported between the time tocompletion of the 2nd computerized test and the paperand-pencil test, with a higher correlation coefficient (r =0.97, p = 0). Both correlations reached statistical significance(p < 0.01), indicating significant linear relationships between
the time to completion of computerized tests and paper-andpencil test. Compared to eTrails [22], an existing computerized
test that achieved moderate correlation with the paper-andpencil test (highest r = 0.668), iTMT demonstrated a muchstronger correlation with the paper-and-pencil test (r = 0.89
and 0.97). When test performance was measured by time tocompletion, the computerized tests generated by iTMT wereable to produce test scores that are highly correlated with thepaper-and-pencil test score of the same participant, suggestinghigh construct validity of the computerized tests.
TABLE IVC OMPARISON AMONG iTMT AND OTHER EXISTING COMPUTERIZED TMT

NameConstruct Validity(Measured by r)

Test-retest Reliability(Measured by r)

Test-retest Reliability(Measured by % reduction)

iTMT

0.89, 0.97

0.93

5.58%

eTrails

0.668

0.62

10.33%

C-TMT

-

0.87

12%

C. Test-retest Reliability
The test-retest reliability of the computerized tests generated
by iTMT was calculated by correlating the time taken tocomplete the 1st and the 2nd computerized test. The Pearson
Product-Moment correlation between the time to completion(b) Paper-and-pencil test vs 2nd iTMT test
Fig. 6. Pearson correlations between the time to completion

Fig. 7. Pearson correlation between time to completion (1st vs 2nd test)of two computerized tests is shown in Fig. 7. Significant linear
relationship was observed between the time to completionof the two computerized tests (r = 0.93). iTMT achieved
considerably higher test-retest reliability than eTrails (highest
r = 0.62) [22] and C-TMT (ICC = 0.87) [21]. Moreover,
iTMT was found to have almost no discernible practiceeffects. As shown in Fig. 7, some participants took longer tocomplete the 1st test, while others took longer to complete the
2nd test on iTMT. From the first to the second administration,
the average time to completion was only reduced by 5.58%.
While the reduction in average time to completion for eTrail[22] was 10.33% and 12% for C-TMT [21]. Comparatively,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

24

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

iTMT was less susceptible to practice effects, providing further
evidence to support its high reliability when administeredrepeatedly.
VII. C ONCLUSION
In this paper, we proposed a divide-and-combine (DAC)
approach to generate instances of TMT that can be used inlongitudinal cognitive assessment. Our proposed approach isable to generated a theoretically unlimited number of different
TMT instances which can be used in consecutive test administrations. Moreover, the instances generated by the proposed
approach have a similar level of difficulty to Reitan’s paperand-pencil TMT by reproducing its spatial characteristics. We
also created a test application, iTMT, which implements the
DAC approach to generate computerized TMTs. The preliminary results from the pilot study support the effectiveness of
our DAC approach. Compared to existing computerized tests,
the instances of TMT generated by the approach produced
test results that were significantly more correlated with resultsof Reitan’s TMT. Similar difficulty and highly correlated
results suggest that these instances possess similar diagnostic
power to the Reitan’s TMT and are able to better distinguish
subjects with different level of cognitive functioning. iTMT
also demonstrated higher test-retest reliability and significantly
reduced practice effects than existing computerized tests.
Due to the illiteracy of some participants, only TMT Part
A and its computerized versions were tested in this study.
In the future, we plan to study Part B in a similar way.
Moreover, in order to validate the feasibility in employing
iTMT in assessment over a long time period, it is worthwhile
to study how test performance of it would change beyond the
2nd administration. Further experiments need to be conducted
to find out whether the test performance of iTMT is convergent
and how does the test performance converge.
ACKNOWLEDGEMENT
This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its IDM
Futures Funding Initiative. The authors would like to thank
Jun Ji for his contributions in application implementation andtesting.
R EFERENCES[1] M. E. Tinetti, M. Speechley, and S. F. Ginter, “Risk factors for falls
among elderly persons living in the community,” New England journalof medicine, vol. 319, no. 26, pp. 1701–1707, 1988.
[2] R. W. Sattin, “Falls among older persons: a public health perspective,”
Annual review of public health, vol. 13, no. 1, pp. 489–508, 1992.
[3] “Dementia statistics,” Alzheimer’s Disease International, accessible till
July 2016. [Online]. Available: https://www.alz.co.uk/research/statistics[4] S. W. Muir, K. Gopaul, and M. M. M. Odasso, “The role of cognitive
impairment in fall risk among older adults: a systematic review andmeta-analysis,” Age and ageing, vol. 41, no. 3, pp. 299–308, 2012.
[5] C. G. Lyketsos, O. Lopez, B. Jones, A. L. Fitzpatrick, J. Breitner, and
S. DeKosky, “Prevalence of neuropsychiatric symptoms in dementia andmild cognitive impairment: results from the cardiovascular health study,”
The Journal of the American Medical Association, vol. 288, no. 12, pp.
1475–1483, 2002.
[6] M. Butler, P. D. Retzlaff, and R. Vanderploeg, “Neuropsychological test
usage,” Professional psychology: Research and practice, vol. 22, no. 6,
pp. 510–512, 1991.
[7] R. M. Reitan, “The relation of the trail making test to organic brain
damage,” Journal of consulting psychology, vol. 19, no. 5, pp. 393–394,
1955.
[8] L. J. Beglinger, B. Gaydos, O. Tangphao-Daniels, K. Duff, D. A.
Kareken, J. Crawford, P. S. Fastenau, and E. R. Siemers, “Practice effects
and the use of alternate forms in serial neuropsychological testing,”
Archives of Clinical Neuropsychology, vol. 20, no. 4, pp. 517–529, 2005.
[9] K. K. Buck, T. M. Atkinson, and J. P. Ryan, “Evidence of practice effects
in variants of the trail making test during serial assessment,” Journal ofclinical and experimental neuropsychology, vol. 30, no. 3, pp. 312–318,
2008.
[10] T. M. Atkinson and J. P. Ryan, “The use of variants of the trail
making test in serial assessment: A construct validity study,” Journalof Psychoeducational Assessment, vol. 26, no. 1, pp. 42–53, 2007.
[11] S. Wagner, I. Helmreich, N. Dahmen, K. Lieb, and A. Tadić, “Reliabilityof three alternate forms of the trail making tests a and b,” Archives ofClinical Neuropsychology, vol. 26, no. 4, pp. 314–321, 2011.
[12] D. Vickers, N. Vincent, and A. Medvedev, “The geometric structure,
construction, and interpretation of path-following (trail-making) tests,”
Journal of clinical psychology, vol. 52, no. 6, pp. 651–661, 1996.
[13] D. Vickers and M. D. Lee, “Never cross the path of a traveling salesman:
The neural network generation of halstead-reitan trail making tests,”
Behavior Research Methods, Instruments, & Computers, vol. 30, no. 3,
pp. 423–431, 1998.
[14] “Army Individual Test Battery: Manual of Directions and Scoring,” War
Department, Adjutant General’s Office, Washington, DC, 1944.
[15] L. Ashendorf, A. L. Jefferson, M. K. OConnor, C. Chaisson, R. C.
Green, and R. A. Stern, “Trail making test errors in normal aging, mild
cognitive impairment, and dementia,” Archives of Clinical Neuropsychology, vol. 23, no. 2, pp. 129–137, 2008.
[16] Y. Chen, H. Yu, C. Miao, B. Chen, Y. Xiaodong, and C. Leung,“Using motor patterns for stroke detection,” Science (Advances in
Computational Psychophysiology), vol. 350, no. 6256, pp. 12–14, 2015.
[17] I. Sanchez-Cubillo, J. Perianez, D. Adrover-Roig, J. Rodriguez-Sanchez,
M. Rios-Lago, J. Tirapu, and F. Barcelo, “Construct validity of thetrail making test: role of task-switching, working memory, inhibition/interference control, and visuomotor abilities,” Journal of the International Neuropsychological Society, vol. 15, no. 3, pp. 438–450,
2009.
[18] B. Fossum, H. Holmberg, and I. Reinvang, “Spatial and symbolic factors
in performance on the trail making test.” Neuropsychology, vol. 6, no. 1,
pp. 71–75, 1992.
[19] E. A. Gaudino, M. W. Geisler, and N. K. Squires, “Construct validity
in the trail making test: what makes part b harder?” Journal of clinicaland experimental neuropsychology, vol. 17, no. 4, pp. 529–535, 1995.
[20] T. M. Atkinson, J. P. Ryan, A. Lent, A. Wallis, H. Schachter, and
R. Coder, “Three trail making tests for use in neuropsychological assessments with brief intertest intervals,” Journal of clinical and experimental
neuropsychology, vol. 32, no. 2, pp. 151–158, 2010.
[21] D. L. Woods, J. M. Wyma, T. J. Herron, and E. W. Yund, “The effects
of aging, malingering, and traumatic brain injury on computerized trailmaking test performance,” PloS one, vol. 10, no. 6, p. e0124345, 2015.
[22] B. T. Smith, “Creation of a more accurate and predictive trail making
test,” Ph.D. dissertation, University of North Carolina Wilmington, 2012.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

25

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Crowd Computing for Population Aging Challenges
Han Yu† , Zhengxiang Pan† , Chunyan Miao∗† and Cyril Leung†‡∗ School of Computer Science and Engineering (SCSE), Nanyang Technological University (NTU), Singapore† Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), NTU, Singapore‡ Department of Electrical and Computer Engineering, the University of British Columbia, Vancouver, BC, Canada{han.yu, panz0012, ascymiao}@ntu.edu.sg, cleung@ece.ubc.ca

Abstract—Today, major population centres in the world (e.g.,
U.S., China, and Japan) are experiencing population aging. The
number of senior citizens aged 65 and above will equate thosebelow the age of 15. Technological innovations are needed inorder to mitigate the emotional, cognitive, and socio-economic
challenges of population aging. As the current generation ofseniors demonstrate relative good health and willingness tocontribute to productive aging activities after retirement, theprevailing perception which used to regard seniors as being
similar to the handicapped has changed. Such a change inperception has initiated a change in the gerontechnologies
landscape from one primarily focused on assistive technologies
towards one encompassing more diverse aspects of wellbeing.
Over the years, the research community has made progress inmany areas of crowd computing including affective computing,cognitive agents, and crowdsourcing. However, little effort has
been directed at enabling intelligent agents to serve the elderly.
As seniors’ characteristics and needs can be very different fromthose of the young, many existing agent techniques must be
adapted in light of gerontology research results in order tobe re-contexted for the elderly. In this paper, we highlight thechallenges and opportunities for crowd computing research inaddressing population aging challenges. We will discuss a numberof areas where significant innovations can emerge by combining
crowd computing with gerontology research, and explore ways
for the research community to organize its resources to help
societies adjust to population aging.
Index Terms—Population aging; crowdsourcing

I. I NTRODUCTION
A. Aging in the 21st Century
In the 1900s, the life expectancy at birth in the United States
was 47. A hundred years later, this number has increased toover 76 years, and will likely reach 83 years by 2050 [1]. At
the start of the 20th Century, about 4% of the U.S. population
was over the age of 65. In 2000, that percentage has climbed
to 13%. By the year 2050, the number of senior citizens (i.e.,
those aged 65 and above) will reach around 2 billion andequate those below the age of 15 [2].
This unprecedented wave of population aging is now sweeping through many nations. Japan and Germany have alreadybeen termed “Super Aged” nations (with more than 20% of the
population over 65), with Canada, Spain, Finland and Greece
lined up to join this category by 2020 [3]. By 2030, other majorcountries such as the U.S., UK, China and Australia will also
become Super Aged nations. Such a worrying trend is putting
increasing strains on healthcare systems and public healthbudgets, and prompting many to caution about a worldwide“silver tsunami” [4].

Currently, the prevailing perception of senior citizens is thatthey are physically and cognitively frail. With this perception,much research effort in the field of gerontechnologies focuses on developing assistive technologies for the elderly [5].
Telehealth and other remote care technologies which strive
to offer an alternative model of care to hospitals and seniorhomes have received much interest from funding agencies andinvestors [6], [7], [8].
However, with improved nutrition and medical care, mostseniors are enjoying many more years of relative health thanprevious generations. Almost 90% of those aged 65 to 74 in
the U.S. report no disability. Even after the age of 85, 40%of the population is fully functional. Today, between the ageof 65 to 80, most American men will remain fully functionalfor 12 out of the 15 years on average [9].
Furthermore, there is more to successful aging than justdelaying the onset of physical or cognitive declines [10], [11].
As the demographic shift towards fewer young people takes
place, there is also some questioning of the traditional contract
between the working generation and the retired generation.
Traditionally, the intergenerational contract has been built
on the basis of intergenerational reciprocity. As the oldage support ratio around the world dwindles [12], the newnorm is likely to require seniors to bear more of the costsof their longer lives through active participation in socioeconomic development activities post retirement [2]. To age
successfully, the maintenance or enhancement of physicaland cognitive functions and full engagement in life throughproductive activities and social connections will be the key foci[13], [14], [15], [16]. Therefore, gerontechnologies should alsochange from treating seniors similar to the disabled to actively
engaging them in various aspects of successful aging.
B. Challenges and Opportunities for Crowd Computing
The challenges of global population aging coincide withthe rapid development in crowd computing in the first decadeof the 21st Century. Crowd computing is concerned with theresearch of information processing systems in which humans
participate to contribute their intelligence, effort, time and/or
resources. As an interdisciplinary and multi-faceted field,
crowd computing research has yielded many technologies thatcan potentially help seniors age successfully. In a world where
technologies permeate people’s lives, it is difficult for seniors
who tend to be technologically less savvy than the young toengage in meaningful activities. For instance, one has to be© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

26

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

familiar with such technologies as Skype to make a simple
phone call to an overseas relative with affordable costs.
It is not feasible to expect each senior to be aware of thefull range of relevant possibilities brought about by various
technologies, and be able to pull them together manually.
Thus, technologies with simple user experience which canproactively guide users’ interactions based on their objectives
and constraints are needed. In order not to dehumanize seniors
with technologies, it is also important to strike a balance ofcontrol between senior users and machines. Crowd computing
offers great promises to addressing this challenge.
Many existing individual dimensions of crowd computing
research (e.g., affect, trust and planning) [17], when re-lookedin the context of interacting with the elderly, can lead to novel
research breakthroughs. In particular, more opportunities forbreakthroughs appear in the “soft” aspects of aging which
concern seniors’ emotional and cognitive wellbeing as wellparticipation in productive aging activities. In order to make amore significant impact, an ecological approach to organize theresources across multiple research teams to build synergy isneeded to tackle system-level challenges. In this paper, we will
discuss a number of areas where breakthroughs can emerge by
combining crowd computing with gerontology research, andways for the research community to organize its resources tohelp societies adjust to population aging.
II. E MOTIONAL C OMPANIONSHIP
Due to falling birth rates and many young people seeking
opportunities away from their home towns, an increasing
number of senior citizens are being left behind. These elderly
only households with seniors living away from their caregivers
are referred to as “empty nest” households. In China, 31.8% of
the 202 million people aged above 60 years live on their own
or with a spouse [18]. For this group of empty nest seniors,
the lack of emotional companionship is a significant challenge[19].
Affective computing is an emerging research field whichis concerned with making sense of people’s emotions basedon various sources of information and enabling computational
entities to display appropriate believable emotions [20]. Overthe years, the researchers have produced many methods to impart emotional capabilities on virtual interactive technologies
to enhance the believability of artificial emotional interactions
with users [21], [22], [23], [24], [25], [26]. These techniques
serve as a promising basis for designing next generation virtualcompanions to provide emotional companionship to empty
nest seniors.
The key challenge to fulfilling this goal lies in seniors’
emotional characteristics which are different from those of thesubjects often studied in agent research (typically technologysavvy young adults [27]). For example, according to theSocioemotional Selectivity Theory (SST), when people perceive their future as open ended (e.g., when they are young),
they tend to focus on future-oriented and development-related
goals. However, when they feel that time is running out andthe opportunity to reap rewards from future-oriented goals’

realization is dwindling (e.g., at old age), their focus tends toshift towards present- and emotion-related goals [28]. In addition, seniors who have been able to maintain good wellbeing
in old age have reduced responsiveness to regret comparedto their less well-to-do peers or people from younger age
group [29]. These results affect the seniors’ objective functions
and, in turn, how human-centred technologies optimize theiremotional responses to seniors. In order for this to happen,a pre-requisite is to find suitable and efficient ways to infuse
these considerations into the computational emotional models
of senior citizens.
III. C OGNITIVE S UPPORT
Cognitively degenerative diseases, such as Parkinson’s disease, pose serious negative impact on seniors’ ability to live
actively and independently [30]. Although not yet officiallyrecognized as a form of disability, many international advocacy groups, such as Alzhemer’s Disease International (http://www.alz.co.uk/), are pushing for changes in legislations sothat dementia patients can be supported by disability benefits
provided by many governments.
In recent years, agent-oriented systems are starting to emerge to help doctors understand seniors with dementia andprovide possible rehabilitation support [31], [32]. Nevertheless, the group of seniors clinically classified as dementia
patients makes up only a small fraction (less than 1%) of
the population (http://www.alz.org/facts/). Most seniors sufferfrom general decline in cognitive functions as their age advances. The declines in cognitive functions are most significant
in terms of memory capabilities [33]. The type of memory
which poses the most challenge to active independent living
is prospective memory (PM).
Prospective memory is a form of memory that involves
remembering to perform a planned action or intention at somefuture point in time [34]. PM tasks are highly prevalent in daily
life and range from relatively simple tasks (e.g., remembering
to buy groceries) to life-or-death situations (e.g., remembering
to take the correct medications). The deterioration in PM is oneof the causes of a worldwide challenge - how to help seniors
who live on their own adhere to prescribed treatments. Smart
pillbox technology alone cannot address this challenge completely [35]. The key difficulties are: 1) distinguishing between
the action of removing medications from the containers andthe action of administering the medication; and 2) providing
effective and yet non-intrusive reminders so as not to annoy
the users and build long term trust [36], [37]. The emerging
works on personalized PM agents [38] and persuasive agents[39] hold promising potential to address these issues.
IV. P RODUCTIVE AGING
Productive aging activities (e.g., volunteering), even those
providing no enhancement to fitness, have been found to lower
the risk of mortality as much as fitness activities do [40].
Crowdsourcing presents a promising solution for bridging
seniors with productive aging activities in a flexible way.
Currently, no commercial crowdsourcing platforms provide© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

27

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

explicit support to elderly users. Nevertheless, technology
components for building crowdsourcing platforms for productive aging are starting to emerge.
In [41], [42], knowledge representation tools that provide
a human interpretable format for representing knowledge andprocesses which can also be easily interpreted by a computer
have been proposed. For motivating the elderly to participate
in crowdsourcing, works that combine human factors into
intelligent agents such as in [43] can be leveraged. Support
for collaboration among older crowdsourcing workers can beachieved through multi-agent organization technologies [44] aswell as intelligent interface agents [45], [46], [31]. Trustworthy
computing research also provides many useful models tofoster trust among the older workers [47], [48], [49], [50],[51]. In addition, intelligent technologies are emerging which
can help optimize the quality-time-cost trade-off to improve
the productivity of large crowdsourcing systems [52], [53],[54], [55], [56], [57], even in situations in which workers
with diverse skills must collaborate to accomplish tasks [58],[59]. These innovative intelligent technologies, combined withappropriate changes in policies governing the hiring of older
workers, hold the key to sustaining socio-economic growth inaging societies. With a recently published large-scale dataset
on crowdsourcing decision-making strategies demonstrated byusers [60], the research community can work towards more
intelligent and personalized crowdsourcing control strategies
to help seniors achieve productive aging.
V. G LOBAL P ROGRAMMES S UPPORTING ACTIVE AGING

them can apply for financial incentives from WorkPro. Lastly,companies which re-hire retirees can apply for grants fromWorkPro in order to re-design the workplace to make it agefriendly, and train their human resource executives to be morewell-versed in how to engage with aged workers. This resembles very much the complex workflows of inter-generational
collaborative crowdsourcing in which workers with different
skills cooperate to solve tasks that cannot be accomplishedby an individual [58]. If crowd computing can be used tocomplement the WorkPro programme, it is possible to enable
the programme to transcend organizational boundaries andenable a larger pool of retirees to work on projects they areinterested in whenever they wish to.

Government

Employees

Help
Work

Businesses

Flexible
RetireesWork
Arrangement

As the important socio-economic impact of population
aging becomes increasingly apparent, governments and organizations of countries in the front line of this demographic
shift are starting to formulate various initiatives and programmes supporting the aging populations. In this section,
we highlight several examples of such policies from aging
countries, including those initiated in a top-down manner by
the governments and those initiated in a bottom-up manner bygrassroots organizations, and how crowd computing research
can make an impact under these policy frameworks.
In Singapore, the policy follows the top-down approach. A
major policy initiative closely related to active aging is the
WorkPro programme1 which focuses on incentivizing retirees
to keep contributing to national development. Through the
WorkPro programme, the government provides the much needed monetary support to promote companies to hire qualified
retirees back to the workforce in order to tap into theirskills and experience horned over entire careers. The scheme
includes three types of monetary support as illustrated in
Figure 1. Firstly, retirees who wish to re-enter the workforce
can apply for salary top-ups and bonuses from WorkPro if
they stay in the workforce for a designated amount of time.
Secondly, current employees of companies re-hiring retirees
who are willing to provide on job training and support for

Similar to Singapore, China’s Mass Entrepreneurship andInnovation (MEI) programme2 is also a top down approachfor crowdsourcing entrepreneurship and innovative ideas. MEI
is not specifically aimed at active and productive aging. It isa more open-ended call for entrepreneurship and innovation
by mass collaboration. However, the scheme has provided
a platform for the elderly who have the experience, skills,
or capital to take up different roles in the entrepreneurshipand innovation processes. The MEI programme consists ofnine different aspects of governance reform to better support
the emergence of mass entrepreneurship and innovation. They
include:1) Institutional mechanisms reform to promote fair competition and talent recruitment;2) Optimizing financial policies and support from related
government bodies to support MEI;
3) Financial market reforms to nurture crowd-funding;4) Providing support for start-up companies;

1 http://www.mom.gov.sg/employment-practices/
schemes-for-employers-and-employees/workpro

2 http://english.gov.cn/policies/latest releases/2015/06/16/content
281475128473681.htm

Fig. 1. The WorkPro Programme in Singapore.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

28

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Under the MEI programme, many crowdsourcing based
MEI businesses are starting to emerge in China. For example,
the Weitianshi Alliance3 focuses on building a communityof experienced small scale angel investors (who are often
senior citizens) to serve as mentors for the young who wish
to start up their own businesses based on their innovativeideas. The mentors have the options to invest their own money
into start-up companies in which they see potential, and canplay an active role in helping the young technopreneurs steer
the companies towards success. In exchange, the mentors
are given shares in the companies they supervise. Feedbacks
from both the mentors and the technopreneurs are tracked inorder to help them build up a profile which can be seen by
future users when they decide which mentor/technopreneur
to approach. Major Internet companies in China, such as
Baidu4 , Tencent5,6 , and Taobao7 , have all set up their crowd
funding platforms to support MEI. As crowdsourcing based
platforms for MEI experience a period of rapid development
in China, crowd computing related research and development
will receive increasing attention accordingly.

100%
90%

Cumulative Percentage

80%
70%
60%
50%
40%
30%
20%
10%
0%

Annual Income from Crowdsourced Work

Fig. 3. Cumulative percentage distribution of annual income derived fromcrowdsourced work by people aged 50 and above in Japan.

100%
90%
Percentage of Respondants

5) Develop services (e.g., incubators) to support MEI;
6) Establish technological and regional platforms to support
MEI;7) Inspire MEI through educational reforms and the recruitment of overseas talents;
8) Boost MEI in underdeveloped rural areas to improve
local employment opportunities; and9) Inter-governmental body cooperation to oversee the implementation of the schemes under the MEI programme.

80%
70%
60%
50%
40%
30%
20%
10%
0%
Making Money in
Spare Time

Have Fun

Getting to Know Task Acquire Business Skills
Requesters

Reasons for Joining Crowdsourcing

Fig. 4. Reasons cited by people aged 50 and above in Japan for joining
crowdsourcing.

Fig. 2. An overview of the inter-generational collaborative crowdsourcing for
producing e-books in traditional Japanese scripts [61].

In contrast, Japan has embraced crowdsourcing as a tool
for active and productive aging following more of a bottomup approach. An example of collaborative crowdsourcing for
productive aging in Japan is shown in Figure 2. In this case,people across different generations contribute their knowledge
to convert traditional Japanese scripts into e-books. The elderly
3 http://www.weitianshi.cn/
4 http://8.baidu.com/rich/film/100000
5 http://gongyi.qq.com/succor/index.htm
6 http://www.webank.com/
7 https://izhongchou.taobao.com/index.htm

No. of CrowdWorks Users > 50 Years Old

3000
2500
2000
1500
1000
500
0
Mar 2012

Jun 2012

Sep 2012

Dec 2012

Mar 2013

Jun 2013

Time

Fig. 5. Growth trajectory of users aged 50 and above in CrowdWorks, Japan.

who are familiar with the intricacies of traditional Japanesescripts contribute their linguistic knowledge, while the young
assist the elderly by providing their technical knowhow (i.e.
scanning books with traditional Japanese scripts to enableelderly workers to process them) [61].

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

29

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

In Japan, crowdsourcing is starting to become a viable
source of work and income for many people aged 50 andabove. According to a recent survey8 conducted by Crowdworks – a crowdsourcing marketplace in Japan, the cumulative
distribution of annual income derived from crowdsourced workby people aged 50 and above in Japan is as illustrated in Figure
3. It can be observed that over 50% of aged crowdsourcing
workers earn between US$1,200 to US$12,000 annually fromperformance crowdsourced tasks, making it a significant supplementary component of their income. More than 10% ofthem can even earn between US$36,000 to US$70,000 per
year from crowdsourced work, significantly higher than the
average annual income by working age adults in Japan.
Various reasons are cited by the aged survey respondants
about why they join crowdsourcing (Figure 4). Over 60% ofthem did so in order to obtain more income during their sparetime. Having fun is also a compelling reason with more than
40% of them selecting it as one of the reasons they join
crowdsourcing. Socializing is also an important reason for
more than 30% of them for joining crowdsourcing. Around aquarter of them selected acquiring business skills as one of thereasons, indicating that crowdsourcing can also be perceived
as a venue for personal growth and life-long learning. So
far, the number of crowdsourcing workers aged 50 and aboveis growing at an exponential rate in CrowdWorks in Japan(Figure 5).
VI. M OVING F ORWARD
In recent years, crowd computing as a field has achieved
significant advances and demonstrated practical impact inmany application domains. Going forward, we envision crowdpowered humanized artificial intelligence (AI) beings will beneeded to fulfill many roles in serving the elderly population
in a world with decreasing younger populations. The researchcommunity has made much progress in the human-centric
aspects of crowd computing and human-agent interactions
which laid a strong foundation for addressing population aging
challenges. We believe by embracing population aging challenges as one of the foci for the research community, crowd
computing as a field will find many exciting opportunities tostrive in the coming decades.
A growing number of aging research centres focusing on
information technology are emerging around the world (e.g.,the LILY Research Centre, Singapore; AGE-WELL, Canada;
and Institute for Aging, New Castle University, UK). These
centres could create and organize clusters of crowd computing research efforts to build up inter-disciplinary research
initiatives leading to agent-based solutions to population aging challenges. In addition, these centres could also help
coordinate the involvement of seniors from diverse cultural
backgrounds into various stages of research. As such, aging
research centres would be excellent partners for the crowd
computing research community in the effort to create agefriendly crowd computing ecosystems of tomorrow.
8 http://thebridge.jp/en/2013/06/japan-seniors-employment

ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its IDM Futures Funding Initiative; the Interdisciplinary Graduate School,
Nanyang Technological University, Singapore; and the Lee
Kuan Yew Post-Doctoral Fellowship Grant.
R EFERENCES[1] E. L. Schneider, “Aging in the third millennium,” Science, vol. 283, no.
5403, pp. 796–797, 1999.
[2] S. Harper, “Economic and social implications of aging societies,”
Science, vol. 346, no. 6209, pp. 587–591, 2014.
[3] E. Duggar and A. B. V. Praagh, “Aging will reduce economic growth
worldwide in the next two decades,” Moody’s, vol. Aug, 2014.
[4] R. Das, “A silver tsunami invades the health of nations,” Forbes, vol.
Aug, 2015.
[5] H. Bouma, J. Fozard, and J. van Bronswijk, “Gerontechnology as a fieldof endeavour,” Gerontechnology, vol. 8, no. 2, pp. 68–75, 2009.
[6] K. Kroenke, “Distance therapy to improve symptoms and quality oflife: Complementing office-based care with telehealth,” Psychosomatic
Medicine, vol. 76, no. 8, pp. 578–580, 2014.
[7] R. Oosterom-Calo, K. Vice, and M. Breslow, “Experiences of older
patients with multiple chronic conditions in the intensive ambulatory
care home telehealth program,” in HCI International 2015 - Posters
Extended Abstracts, ser. Communications in Computer and Information
Science, C. Stephanidis, Ed., 2015, vol. 528, pp. 311–316.
[8] T. Russell, N. Gillespie, N. Hartley, D. Theodoros, A. Hill, and L. Gray,“Exploring the predictors of home telehealth uptake by elderly australian
healthcare consumers,” Journal of Telemedicine and Telecare, vol. 21,
no. 8, pp. 485–489, 2015.
[9] J. W. Rowe, “The new gerontology,” Science, vol. 278, no. 5337, p. 367,
1997.
[10] E. Gonzales, C. Matz-Costa, and N. Morrow-Howell, “Increasing opportunities for the productive engagement of older adults: A response
to population aging,” The Gerontologist, vol. 55, no. 2, pp. 252–261,
2015.
[11] J. M. Hilton, K. Kopera-Frye, and A. Krave, “Successful aging from theperspective of family caregivers,” The Family Journal, vol. 17, no. 1,
pp. 39–50, 2009.
[12] “World population ageing 2013,” United Nations, Tech. Rep., 2013.
[13] M. Arbesman and D. Lieberman, “Methodology for the systematic
reviews on occupation- and activity-based intervention related to productive aging,” American Journal of Occupational Therapy, vol. 66, pp.
271–276, 2012.
[14] K. J. Johnson and J. E. Mutchler, “The emergence of a positive gerontology: From disengagement to social involvement,” The Gerontologist,
vol. 54, no. 1, pp. 93–100, 2014.
[15] T. Y.-s. Lum, “Advancing research on productive aging activities ingreater chinese societies,” Ageing International, vol. 38, no. 2, pp. 171–
178, 2012.
[16] K. Thanakwang and S. arun Isaramalai, “Productive engagement in older
adults: A concept analysis,” Nursing and Health Sciences, vol. 15, no. 1,
pp. 124–130, 2013.
[17] H. Yu, Z. Shen, and B. An, “An adaptive witness selection method for
reputation-based trust models,” in Proceedings of the 15th International
Conference on Principles and Practice of Multi-Agent Systems (PRIMA’12), 2012, pp. 184–198.
[18] J.-E. Liua, J.-Y. Tiana, P. Yuea, Y.-L. Wangb, X.-P. Dub, and S.-Q.
Chen, “Living experience and care needs of Chinese empty-nest elderly
people in urban communities in Beijing, China: A qualitative study,”International Journal of Nursing Sciences, vol. 2, no. 1, pp. 15–22,
2015.
[19] Y. Chen, A. Hicks, and A. E. While, “Loneliness and social support ofolder people in china: a systematic literature review,” Health and Social
Care in the Community, vol. 22, no. 2, pp. 113–123, 2014.
[20] J. Tao and T. Tan, “Affective computing: A review,” in Proceedings ofthe 3rd International Conference on Affective Computing and Intelligent
Interaction (ACII’05), 2005, pp. 981–995.
[21] J. Mell, “Toward social-emotional virtual humans,” in Proceeding of the
14th International Conference on Autonomous Agent and Multi-Agent
Systems (AAMAS’15), 2015, pp. 2015–2016.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

30

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[22] X. Yu, C. T. Salmon, and C. Leung, “Emotional interactions between
artificial companion agents and the elderly,” in Proceeding of the
14th International Conference on Autonomous Agent and Multi-Agent
Systems (AAMAS’15), 2015, pp. 1991–1992.
[23] M. E. Foster, A. Deshmukh, S. Janarthanam, M. Y. Lim, H. Hastie,
and R. Aylett, “Influencing the learning experience through affective
agent feedback in a real-world treasure hunt,” in Proceeding of the
14th International Conference on Autonomous Agent and Multi-Agent
Systems (AAMAS’15), 2015, pp. 1711–1712.
[24] W. Lee, S. Park, and I.-C. Moon, “Modeling multiple fields of collective
emotions with brownian agent-based model,” in Proceeding of the
13th International Conference on Autonomous Agent and Multi-Agent
Systems (AAMAS’14), 2014, pp. 797–804.
[25] Y. Ding, K. Prepin, J. Huang, C. Pelachaud, and T. Artières, “Laughter
animation synthesis,” in Proceeding of the 13th International Conference
on Autonomous Agent and Multi-Agent Systems (AAMAS’14), 2014, pp.
773–780.
[26] M. Courgeon, C. Céline, and J.-C. Martin, “Modeling facial signs ofappraisal during interaction: Impact on users’ perception and behavior,”
in Proceeding of the 13th International Conference on Autonomous
Agent and Multi-Agent Systems (AAMAS’14), 2014, pp. 765–772.
[27] J. Lin, H. Yu, C. Miao, and Z. Shen, “An affective agent for studying
composite emotions,” in Proceeding of the 14th International Conference on Autonomous Agent and Multi-Agent Systems (AAMAS’15), 2015,
pp. 1947–1948.
[28] L. L. Carstensen, D. M. Isaacowitz, and S. T. Charles, “Taking time seriously: A theory of socioemotional selectivity,” American Psychologist,
vol. 54, no. 3, pp. 165–181, 1999.
[29] S. Brassen, M. Gamer, J. Peters, S. Gluth, and C. Büchel, “Don’t lookback in anger! Responsiveness to missed chances in successful andnonsuccessful aging,” Science, vol. 336, no. 6081, pp. 612–614, 2012.
[30] B. M. Bot, C. Suver, and et al, “The mPower study, Parkinson diseasemobile data collected using ResearchKit,” Scientific Data, vol. 3, no.
160011, p. doi:10.1038/sdata.2016.11, 2016.
[31] Y. Cai, Z. Shen, S. Liu, H. Yu, X. Han, J. Ji, M. J. McKeown, C. Leung,and C. Miao, “An agent-based game for the predictive diagnosis ofparkinson’s disease,” in Proceeding of the 13th International Conference
on Autonomous Agent and Multi-Agent Systems (AAMAS’14), 2014, pp.
1663–1664.
[32] S. Liu, Z. Shen, M. J. McKeown, C. Leung, and C. Miao, “A fuzzy logic
based parkinson’s disease risk predictor,” in Proceedings of the 2014
IEEE International Conference on Fuzzy Systems (FUZZ-IEEE’14),
2014, pp. 1624–1631.
[33] U. Lindenberger, “Human cognitive aging: Corriger la fortune?” Science,
vol. 346, no. 6209, pp. 572–578, 2014.
[34] M. A. Brandimonte, G. O. Einstein, and M. A. McDaniel, Prospective
Memory: Theory and Applications, 3rd ed. New York and London:
Psychology Press, 2014.
[35] M. Suárez-Varela and et al, “Study on the use of a smart pillbox toimprove treatment compliance,” Atención Primaria, vol. 41, no. 4, pp.
185–191, 2009.
[36] Z. Shen, H. Yu, C. Miao, and J. Weng, “Trust-based web-service
selection in virtual communities,” Journal for Web Intelligence andAgent Systems (WIAS), vol. 9, no. 3, pp. 227–238, 2011.
[37] S. Liu, Z. Shen, H. Yu, H. Lin, Z. Guo, Z. Pan, C. Miao, and C. Leung,“A kinect-based interactive game for improving the cognitive inhibitionof the elderly,” in Proceedings of the 15th International Conference on
Autonomous Agents and Multi-Agent Systems (AAMAS’16), 2016.
[38] H. Lin, J. Hou, H. Yu, Z. Shen, and C. Miao, “An agent-based gameplatform for exercising people’s prospective memory,” in Proceedings ofthe 2015 IEEE/WIC/ACM International Conference on Web Intelligenceand Intelligent Agent Technology (WI-IAT’15), 2015, pp. 235–236.
[39] Y. Kang, A.-H. Tan, and C. Miao, “An adaptive computational model
for personalized persuasion,” in Proceedings of the 24th International
Joint Conference on Artificial Intelligence (IJCAI’15), 2015, pp. 61–67.
[40] T. A. Glass, C. M. de Leon, R. A. Marottoli, and L. F. Berkman,“Population based study of social and productive activities as predictorsof survival among elderly americans,” British Medical Journal, vol. 319,
no. 7208, pp. 478–483, 1999.
[41] H. Yu, Z. Shen, and C. Miao, “Intelligent software agent design tool using goal net methodology,” in Proceedings of the 2007 IEEE/WIC/ACM
International Conference on Web Intelligence and Intelligent Agent
Technology (WI-IAT’07), 2007, pp. 43–46.

[42] ——, “A goal-oriented development tool to automate the incorporationof intelligent agents into interactive digital media applications,” ACM
Computers in Entertainment (CIE), vol. 6, no. 2, 2008.
[43] H. Yu, Z. Shen, C. Miao, and A.-H. Tan, “A simple curious agent to help
people be curious,” in Proceeding of the 10th International Conference
on Autonomous Agent and Multi-Agent Systems (AAMAS’11), 2011, pp.
1159–1160.
[44] B. Li, H. Yu, Z. Shen, and C. Miao, “Evolutionary organizational
search,” in Proceeding of the 9th International Conference on Autonomous Agent and Multi-Agent Systems (AAMAS’09), 2009, pp. 1329–
1330.
[45] H. Yu, Y. Cai, Z. Shen, X. Tao, and C. Miao, “Agents as intelligent
user interfaces for the net generation,” in Proceedings of the 15th
International Conference on Intelligent User Interfaces (IUI’10), 2010,
pp. 429–430.
[46] Q. Wu, X. Han, H. Yu, C. Miao, and Z. Shen, “The innovative
applications of learning companions in virtual singapura,” in Proceedingof the 12th International Conference on Autonomous Agent and MultiAgent Systems (AAMAS’13), 2013, pp. 1171–1172.
[47] H. Yu, Z. Shen, C. Miao, C. Leung, and D. Niyato, “A survey oftrust and reputation management systems in wireless communications,”
Proceedings of the IEEE (Proc. IEEE), vol. 98, no. 10, pp. 1755–1772,
2010.
[48] H. Yu, Z. Shen, C. Leung, C. Miao, and V. R. Lesser, “A survey ofmulti-agent trust management systems,” IEEE Access, vol. 1, no. 1, pp.
35–50, 2013.
[49] S. Liu, H. Yu, C. Miao, and A. C. Kot, “A fuzzy logic based reputation model against unfair ratings,” in Proceeding of the 12th International Conference on Autonomous Agent and Multi-Agent Systems(AAMAS’13), 2013, pp. 821–828.
[50] L. Qiu, H. Lin, C.-Y. Chiu, and P. Liu, “Online collective behaviors
in china: dimensions and motivations,” Analyses of Social Issues andPublic Policy, 2014.
[51] Y. Liu, J. Zhang, H. Yu, and C. Miao, “Reputation-aware continuous
double auction,” in Proceedings of the 28th AAAI Conference on
Artificial Intelligence (AAAI-14), 2014, pp. 3126–3127.
[52] H. Yu, Z. Shen, C. Miao, and B. An, “A reputation-aware decisionmaking approach for improving the efficiency of crowdsourcing systems,” in Proceeding of the 12th International Conference on Autonomous
Agent and Multi-Agent Systems (AAMAS’13), 2013, pp. 1315–1316.
[53] H. Yu, C. Miao, B. An, C. Leung, and V. R. Lesser, “A reputation management approach for resource constrained trustee agents,” inProceedings of the 23rd International Joint Conference on Artificial
Intelligence (IJCAI’13), 2013, pp. 418–424.
[54] H. Yu, C. Miao, B. An, Z. Shen, and C. Leung, “Reputation-aware
task allocation for human trustees,” in Proceeding of the 13th International Conference on Autonomous Agent and Multi-Agent Systems(AAMAS’14), 2014, pp. 357–364.
[55] H. Yu, X. Yu, S. F. Lim, J. Lin, Z. Shen, and C. Miao, “A multiagent game for studying human decision making,” in Proceeding of the
13th International Conference on Autonomous Agent and Multi-Agent
Systems (AAMAS’14), 2014, pp. 1661–1662.
[56] H. Yu, C. Miao, Z. Shen, and C. Leung, “Efficient task sub-delegation
for crowdsourcing,” in Proceedings of the 29th AAAI Conference on
Artificial Intelligence (AAAI-15), 2015, pp. 1305–1311.
[57] ——, “Quality and budget aware task allocation for spatial crowdsourcing,” in Proceeding of the 14th International Conference on Autonomous
Agent and Multi-Agent Systems (AAMAS’15), 2015, pp. 1689–1690.
[58] Z. Pan, H. Yu, C. Miao, and C. Leung, “Efficient collaborative crowdsourcing,” in Proceedings of the 30th AAAI Conference on Artificial
Intelligence (AAAI-16), 2016.
[59] H. Yu, C. Miao, S. Liu, Z. Pan, N. S. B. Khalid, Z. Shen, and C. Leung,“Productive aging through intelligent personalized crowdsourcing,” inProceedings of the 30th AAAI Conference on Artificial Intelligence(AAAI-16), 2016.
[60] H. Yu, C. Miao, Z. Shen, J. Lin, C. Leung, and Q. Yang, “Infusing
human factors into algorithmic crowdsourcing,” in Proceedings of the
28th Conference on Innovative Applications of Artificial Intelligence(IAAI-16), 2016.
[61] M. Kobayashi, T. Ishihara, T. Itoko, H. Takagi, and C. Asakawa, “Agebased task specialization for crowdsourced proofreading,” in Proceedings of the 7th international conference on Universal Access in HumanComputer Interaction (UAHCI’13), 2013, pp. 104–112.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

31

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Crowd Science and Engineering: Concept andResearch Framework
Yueting Chai1, Chunyan Miao2, Baowen Sun3, Yongqing Zheng4, Qingzhong Li4
1

National Engineering Laboratory for E-commerce Technologies, Tsinghua University
2
Nanyang Technological University, Singapore
3
Central University of Finance and Economics
4
Shangdong University
chaiyt@tsinghua.edu.cn, ascymiao@ntu.edu.sg, sunbaowen@263.net,
zhengyongqing@dareway.com.cn, lqz@sdu.edu.cn

Abstract—The synthetic application and interaction of/between
the Internet, Internet of Things (IOT), cloud computing, big data,
Industry 4.0 and other new patterns and new technologies shallbreed future web-based industrial operation system and social
operation management patterns, manifesting as a crowd cybereco-system composed of multiple interconnected intelligent
agents (enterprises, individuals and governmental agencies) and
its dynamic behaviors. To explore the basic principles and laws ofsuch a system and its behavior, we propose the concepts of crowd
science and engineering (CSE) and expound its main content,
thus to form a research framework of theories and methodologies
of crowd science, which is expected to substantially promote theformation and development of crowd science and thus lay afoundation for the advancement of web-based industrial
operation system and social operation management patterns.
Index Terms—crowd science, modeling and simulation, ecology,transaction, intelligence, evolution, stability.

I. INTRODUCTION
Following the first industrial revolution marked byinvention of steam engine and the second industrial revolution
marked by invention of electric power, man has ushered in thethird industrial revolution with the Internet as the symbol.
Compared with the first two, the third one is transforming theindustrial forms and social operation patterns of the humansociety in a more comprehensive and fundamental way. The
underlying reason is that the Internet has ignited man’s demand
and unleashed potential needs of humans, thus having formed
the motive power for Internet changing industry and social
operation mode.
With a great prospect for integrated development in various
fields, the Internet is exerting overall and strategic impact on
the economic and social development of various countries. For
this reason, major developed economies have drawn up
corresponding measures to gain the power of initiative in futurecompetition, for instance, Industry 4.0 in Germany andIndustry Internet in the U.S. Meanwhile, the Internet-based
economy represented by E-commerce has sprung up in China,
which is extending from consumption to industry, taking on
robust power in ensuring economic growth, adjusting
economic structure, stabilizing employment and promoting

innovations. Chinese government has in a timely manner madea series of policies and strategies to promote E-commerce,Internet Plus, IOT, big data, cloud computing, mass
entrepreneurship and innovation, crowd making, crowd
sourcing, crowd supporting and crowd funding, as well as
China Manufacturing 2025 Development Program, etc.
A comprehensive survey of the policies, measures andaction plans of various nations finds that most of them are
advancing the work through the perspectives of the Internet,
IOT, cloud computing, big data, Industry 4.0, etc. As a matterof fact, the synthetic application and interaction of/between theabove-mentioned new patterns and new technologies shallbreed future web-based industrial operation system and social
operation management patterns. See Figure 1.
In comparison with traditional industrial operation system,
future web-based industrial operation system takes on thefollowing essential features: initiative, personalized
consumption,
immediacy,
centralized
circulation,
decentralization, smart production, self-organization andecologicalized system.
II. CONCEPT OF CROWD SCIENCE AND ENGINEERINGA deep analysis of general types and substantive
characteristics of future web-based industrial operation system
can easily identify a smart interconnected network system thattakes on the features of large scale, openness, self-organizationand ecologicalization. Hence, the system is called “Crowd
cyber System” in short, as shown in Figure 2. In terms ofcomponents, this kind of network system take the forms ofsmart entities composed of a great many intelligently
interconnected individuals, enterprises, governmental agencies
and articles, with sort of supply-demand relationship between
them. In terms of major behaviors on the network, theinteraction between any smart entities boils down to some sortof transaction or behavior. From the perspective of system, thenetwork system is a ternary system of information, physics andsociety. Openness, interconnection, cooperation and sharing
constitute the basic logic of the network system while stability,
efficiency, innovation and development are the targets thesystem aims to fulfill.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

32

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Figure 1: Future Web-based Industrial Operation System

Figure 2: Schematic diagram of crowd cyber system

The numerous intelligent agents (enterprises, institutions,individuals and articles, etc.) in the crowd cyber system cangive full play to their wisdom and potentiality, interact witheach other and produce various kinds of unexpected nonlinear
behaviors and effects, which cannot be explained by anexisting discipline, but rather need the basic theories andfrontier achievements of multiple disciplines. Therefore, anemerging interdiscipline is needed.
Therefore, we propose the concept of “crowd science andengineering” (CSE). By use of knowledge of system theory,
information theory, cybernetics, computer science andengineering, management, economics, sociology, psychology,
as well as the new technological means of IOT, cloudcomputing, big data, artificial intelligence (AI), CSE aims to
explore the basic principles and laws of the swarm intelligence
activities of the ternary system of information, physics andsociety in the context of large-scale online interconnection. In
so doing, it seeks to establish related methods and tools, givefull play to individual and group wisdom of mankind, tap into
the potentiality and effectively advance a new web-basedindustrial operation system and social operation management
patterns.
With crowd cyber system as the object of study, crowd
science aims to establish the system of concepts of the crowd

cyber system featuring stability, efficiency, innovation anddevelopment, the basic principles, behavior patterns and related
methodologies, thus laying a solid foundation for the webbased industry.
The essential questions to be addressed include:
1) How to construct the crowd cyber system to realize asynchronic mapping in structure, information, behavior andconsciousness between individuals, enterprises, governmental
agencies and articles of the physical world and various
corresponding intelligent agents in the information world, andsupport various kind of intelligent transaction behavior
between intelligent agents.
2) What kind of eco-structure and behavior mode will themultiple intelligent agents take on to ensure the efficient
operation of the network system?
3) The originality of the crowd cyber system rests with themultitude of its intelligence. What is intelligence? How tomeasure it? What is the relationship between group intelligenceand individual intelligence of various intelligent agents? Howto make use of the group intelligence of the crowd cybersystem to measure the originality of the network system.
4) The sustainability of the crowd cyber system rests withthe evolution of various intelligent agents. What are themechanism, ways, orientations and conditions of various© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

33

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

intelligent agents in the system? How to maintain the rational
evolution of various intelligent agents so as to arrive at thesustainable development of the network system?
5) How to maintain the relative stability of the network
system and rule out breakdowns. This, in the real world, means
to avoid destructive “bubble economy”, “economic crisis” and“subversive revolution”.
III. THE RESEARCH FRAMEWORK OF CROWD SCIENCE AND
ENGINEERING
In order to answer the basic questions mentioned above, theresearch on the theories and methodologies of CSE mainly
concerns issues including: (1) research on the modeling andsimulation of crowd cyber system; (2) theoretical research on
the structure of crowd cyber eco-system; (3) theoreticalresearch on the intelligent transactions of crowd cyber system;(4) research on the intelligence measurement methods of crowd
cyber system; (5) theoretical research on the evolution of
crowd cyber system; (6) theoretical research on the stability ofcrowd cyber system.
A. Research on the Modeling and Simulation of Crowd cyber
Mainly dealing with the construction of crowd network, itaims to build the mental models of network for various
intelligent agents (individuals, enterprises, governments andthings), so as to enable all intelligent agents in cyber space toreflect individuals, enterprises, governments and things in thereal world comprehensively, authentically, correctly andsimultaneously. Such models refer to analytic model (forcomputing), static structure model (for building different kindsof systems), dynamic behavior model (for studying theoperational rules of systems), presentation model (forpresenting real individuals, enterprises, governments and thingsin cyber space), and other intelligent interconnected models
and algorithms between all intelligent agents.
B. Theoretical Research on the Structure of Crowd Cyber
Eco-System
In the future web-based industrial system, varied properties
and features of industries or services will definitely result in alarge number of operation platforms for interconnected webbased industries and thus form industrial eco-structure. Then,
how many platforms are needed to sustain efficient running ofspecific industrial systems? Should they be comprehensive,
semi-professional or professional? These actual questions
indicate the necessity of research on crowd cyber eco-system.
The eco-structure of crowd network is the relationship ofdivision and collaboration between intelligent agents that form
the network, while in terms of future web-based industries, it isindustrial organization form. Eco-structure is one of the key
factors affecting the efficiency of network operation.
In a certain industrial organization ecology, each intelligent
agent (each market entity) has its corresponding ecologicalniche, which refers to the sum of an intelligent agent’s
particular position in time and space and its functional
relationship with other agents. The ecological niche and such
relationship will change as transaction efficiency changes, thus
eco-structure changes. Specific transaction efficiency goes with

its own eco-structure which is the most efficient structure forthe operation of crowd cyber or industrial systems. Transaction
efficiency is the main inducement to the evolution of ecostructure. That an intelligent agent’s ecological niche and its
relationship with other agents change after the changing
transaction efficiency is in fact a strategy adopted by the agent
according to its ecological niche and environmental variation.
Therefore, in eco-structure theories, the focuses of research lie
in the evolutionary dynamics, mechanisms and paths of ecostructure.
C. Theoretical Research on the Intelligent Transactions of
Crowd Network
The interaction between various intelligent agents that
constitute crowd network boils down to some transaction
activities or behaviors. A transaction activity involves the
supply side, demand side, objects and rules of the transaction.
Major problems in intelligent transactions to be tackled include:1) to describe the needs of demand sides accurately and choose
the most suitable out of many supply sides; 2) to accurately
describe the needs of supply sides and choose the most suitable
out of many demand sides; 3) to quickly conclude transactions.
To cope with the above problems, the theoretical research on
intelligent transactions should focus on the following aspects:(1) intelligent methods of acquiring the law of demand andaccurate demands; (2) intelligent methods of acquiring the lawof supply and accurate supplies; (3) methods of intelligent
matching transactions; (4) mechanisms and methods offorming dynamic pricing; (5) methods of evaluating intelligent
transactions.
D. Research on the Intelligence Measurement Methods of
Crowd cyber
The purposes of research on the intelligence measurement
methods of crowd network are to evaluate the intelligence ofindividuals and groups and to offer supports for evaluating theinnovation potential of individuals and groups. The research
focuses on the concept of standardized intelligence, and themeasurement methods of intelligence of individuals and groups.
There is no standardized concept of intelligence at present.
Intelligence and its nature have been studied by lots ofphilosophers and brain scientists all around the world sinceancient times but still remain so mysterious that the occurrence
intelligence is among the four secrets of nature with other three
being the nature of matter, the origin of the universe and thenature of life. We should, on the basis of existing research
results, attach a standardize concept to intelligence, which cancontribute to quantitative evaluation of intelligence.
Generally speaking, intelligence may be high or low, but towhat degree? Related research is short of fruits. We should
build a measurement model of individual intelligence tocalculate individual intelligence quantitatively. Meanwhile,group intelligence is higher than individual intelligence in mostcases. But there is no research explaining the exact difference
and the relationships between them. We should build a crowd
cyber-based measurement model of intelligence to calculategroup intelligence quantitatively.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

34

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

E. Research on the Evolution of Crowd Network
The purpose of researching the co-evolution theories of
crowd network to support the sustainable development of webbased industries in the future with scientific evidence. To
achieve a sustainable development, an industry must, first,
create customers’ demands; second, satisfy these demands withlow cost and high efficiency; and third, get the most out of all
resources, particularly physical resources. All of these requirecontinuous innovation on products and services, improvement
in manufacturing or service process, and reduction in wastes ofresources. Such process of innovation or improvement is
manifested as a continuous evolution of specialized knowledge,
business process, ways of resource utilization in anenvironment of market competition. In other words, to sift out
those lagging behind and promote those going ahead.
Therefore, research the co-evolution theories of crowd cybershould focus on the following aspects: (1) evolution methodsof specialized knowledge; (2) co-evolution mechanisms andalgorithms of business process; (3) evolution mechanisms andalgorithms of resource utilization methods.
F. Theoretical Study on Crowd Network Stability
Stability is the foundation of everything, including thefuture network-based industry. Of course, the stability in thiscase is dynamical since the industry itself is developing. It canbe derived from existing research results that a stable network

cannot be established without the following factors: 1) Enoughnetwork nodes. Otherwise, a network with every few network
nodes could be vulnerable to external interference, which
would ruin the network. 2) Excellent network connectivity.
Otherwise, a loose connection among network nodes would
result in a quick network partition under external interference.
3) Controllable differences among network nodes. Otherwise,
the internal imbalance and insoluble contradiction
intensification would cause the collapse of the network. Given
the above-mentioned factors, the concept of crowd cyberstability should be raised and regularized while the crowd
cyber stability quantification methods are demonstrated, which
provide the basis of network stability evaluation and thus
facilitate the establishment of Crowd cyber Stability Model,
through which the solution comes out with network stability
status and relevant conditions.
REFERENCES[1] G. Harari, Y. Sapiens, Y. Noah. ‘A Brief History of
Humankind’, translated by Lin Junhong, CITIC Press Corp.,
2014[2] K. Kelly. ‘Out of Control: The New Biology of Machines,
Social Systems, and the Economic World’, Translated by
Dongxiwenku, New Star Press, 2013, Beijing.
[3] Jigang Bao. ‘The Economics of Information Entropy: the Wayof Human Development’, Economic Science Press , Beijing ,2013© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

35

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Crowd Wisdom Drives Intelligent Manufacturing
Jiaqi Lu, Shijun Liu+, Lizhen Cui, Li Pan, Lei Wu
School of Computer Science & Technology
Shandong University
Jinan, China
453429978@qq.com, {lsj, clz, panli, i_lily}@sdu.edu.cn

Abstract—A
fundamental
problemforintelligent
manufacturing is to equip the agents with the ability to makejudgments and decisions automatically. This paper introduces
the basic principle for intelligent crowds, attempts to show that
crowd wisdom could help make accurate judgments and proper
decisions, and illustrates the positive effect that crowd wisdom
could bring throughout the whole manufacturing process. Efforts
to support the critical role of crowd wisdom in intelligent
manufacturing involve theoretical explanation, includingreference to several prevailing concepts, such as C2B,
crowdfunding, an interpretation to the contemporary Big Data
mania. Finally, an empirical study with three business cases
reinforces that our ideas could well explain the current businessphenomena as well as guides the future of manufacturing.
Keywords—crowd wisdom; crowd science; big data; intelligent
manufacturing

I.

INTRODUCTION

Intelligent Manufacturing is the development andimplementation of intelligent machine in manufacturing. It
refers to the use of production process technology that can
automatically adapt to changing environments and varying
process requirements, with the capability of manufacturing
various products with minimal supervision and assistancefrom operators [1],[2].
An urgent need is to endow machines with intelligence, inorder to achieve the goal of intelligent decision-making in theprocess of manufacture. Getting more insight into our massive
customer group in the current Internet Era, there never has
been a time like this when enterprises can access such a
massive crowd, constituted by individuals with various
backgrounds, coming from different countries and districts. A
crowd of diverse strangers, what a coincidence, isn’t’ it ourdesired intelligent crowd? In fact, C2B mode is calling forindustry to make good use of crowd wisdom.
In this paper, we propose crowd wisdom working as thebrain of manufacturing. Plenty of examples prove that a
qualified crowd would make wise decisions and accurate
judgments, with the abundant information collected by theconsisting individuals. Thus we apply the theory onto
manufacturing, point out that with various data contributed by
the crowd, including customers especially, manufacturing
decision could be obtained, and we try to explicate the reason
why Big Data is key to intelligent manufacturing in theperspective of crowd wisdom.

The rest of the paper is organized as follows. Section 2
explicates the base of crowd wisdom, including thequalification for an intelligent crowd and the source of its
wisdom. Section 3 introduces the crowd participating in detailand how it contributes to intelligent manufacturing. Section 4
strengthens our points with several cases where enterprises
utilize crowd wisdom, which is of great help. Section 5
concludes the paper.
II.

THE PRINCIPLE OF CROWD WISDOM

Predecessors’ works revealed an amazing fact: the wisdomof crowds is incredibly surprising. In 1906, a British scientist,
Francis Galton invited 800 participants to evaluate the weightof an ox. Some of the participants are experts on agriculture,
while the others are sheer laymen. The average evaluation of
the crowd is only one pound less than the real weight [3].
Individuals make their own judgments on the same topic
independently. When the judgments aggregate, the crowd’s
judgment would be incredibly close to the fact. In fact, theaggregate judgment is often accurate even in the case ofpredicting future. Firstly, we analyze the key principles of
crowd wisdom.
A. Characterristics of Intelligent Crowd
Aggregating individuals’ judgment then we’ll obtain agood judgment. The approach is seemingly too naïve to be real.
But unfortunately, not all crowds possess such sharp
judgment.
J.Surowiecki
elaborated
thenecessarycharacteristics of a crowd with great wisdom in his work, thatis, under decentralization system, a diverse crowd constitutedof independent individuals would make good judgment [3].
Fig. 1 shows the characteristics and how they are related toeach other.
The base of crowd wisdom is the private information
within an individual’s judgment. The information from eachindividual partly depicts the fact more or less. When a large
amount of information from the individuals aggregate,
information counteracts and complements. Thus mistakes
cancels out, and the crowd possesses the whole information
about the truth. This guarantees the authenticity of the crowd’s
judgment, as well as the decision derived.
The wisdom comes from the crowd, as the crowd owns
abundant information. Ultimately, the wisdom comes from theinformation. And the reason why we request the diversity of© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

36

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

crowd is that we have to make sure that the information fromthe crowd is truly comprehensive rather than duplicates of thesame information in diverse expressions.

Fig. 1. Characteristics of intelligent crowd

B. Diversity of the Crowd
Generally speaking, making a crowd judgment is a twostage process. The first stage is searching for a judgment pool,which is the collection of each individual’s own judgment
utilizing his or her private information. And the second stage
is obtaining a crowd judgment, by means of aggregating the
judgment of every individual in the crowd with a particulartechnique.
It is obvious that the more extensive the judgment pool, themore preferable, as a good judgment is more likely to be
included, enter the second-stage, and impact the final
judgment rather than being ignored regretfully.
Hence now our problem is how to extend our judgment
pool?
Since judgment originates from individuals’ analysis andthinking, we need people with unlike thought patterns, expert
in different domains, owning diverse experience andbackground involved. Individual differences ensure thediversity of information and the judgment.
C. Independent Individuals
Independence guarantees a wrong judgment from anindividual would cause no impact on other individuals in thecrowd. Mistaken judgments can counteract in the process ofaggregating, therefore the accumulated information is stillcorrect.
Besides, independence encourages individuals to explore
for new private information, rather than sedentary satisfied
with the public information. Apparently, new information isadvantageous to extend our judgment pool.
Here we arrive at a paradox. If the crowd’s judgment is
intelligent, why don't we just simply adopt the crowd’s
judgment instead of contemplating our own judgment, which
is likely to be a wrong one?
The answer may sound contradictory: the mistakes ofindividuals make the crowd smart, thus leading to a good
crowd judgment. But when individuals begin adopting thecrowd’s judgment, the crowd becomes sluggish while theindividual is gaining wisdom.
Making a judgment is never a short-term process, neither
is the evaluation of a judgment. A good process should be like
this: individuals make judgments independently, then thecrowd wisdom works and a good crowd judgment is obtained.
After a period when large amount of individuals adopt thevery crowd judgment, defects of the judgment would
gradually emerge or someone would figure out a new
judgment. Then, the massive followers would stop adopting
the existing judgment and a new round of independent
exploration ensues.
Perhaps as time goes by, the defects would be exposed tothe public, so the process proceeds. Nevertheless, generally, a
promoter, the one who figures out a new judgment when mostof his or her peers adopt the crowd’s judgment, plays a role.
Those insistent independent thinkers propose a new judgment,
break the original stagnant state of the crowd, and weaken theauthenticity of the prevailing crowd’s judgment. More
individuals are encouraged to cease following and pay more
attention on private information and personal judgment. The
promoters help us from drowning in the crowd wisdom.
D. Decentralization System
The concept of crowd is always related to system design.
The art of management is designing an appropriate system,
one that vitalizes the crowd.
In our context, we expect a system that can motivate theindividuals to search for diverse judgments, maintainindividual independence as well as ensure the final crowd’s
judgment is indeed an aggregated one, instead of coming froma dominant individual.
Plenty of precedents prove the superiority ofdecentralization system in the aspect of ensuring the three very
critical points above. A decentralization system is one that isabsent of the center of power.
In a system with distinct powerful members, for example,
in a centralization system, various influences from dominant
individuals can be a problem that cannot be neglected.
Individuals may judge no longer based on facts, but
reluctantly, indifferently or blindly follow the decision ofsuperiors, experts, or elites.
The undesirable situation defies the demand forindependence. When dominant individuals spread theirinfluence, they publicize their private information, which
subtly influences the choice of other individuals. Even worse,a wrong judgment from the dominant individual would leadthe whole crowd into an abyss.
Moreover, a blind crowd can neither be tagged as diversity
nor provide a diverse judgment pool, since individuals canhardly be expected to give unique or distinct judgments. It is
predictable that the judgment pool would be partial.
A centralization system stifles the diversity of the crowd.
The crowd may come to an agreement promptly meanwhile
they may enter the dilemma of Group Think [4], where© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

37

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 2. Crowd participating in manufacturing process, and crowd data as a source of intelligent manufacturing

everyone tries to obey the majority and avoids proposing new
angles or objection. Such crowd may extremely believe in thecorrectness of crowd judgment as well as strongly object toany criticism or advice from the outsiders, as they deem thatthe final judgment is a real crowd wisdom while actually it isnot. A decentralization system helps us to avert such plight. At
least, during the first stage, a decentralization system ensures
the judgment from an individual is independently made, thus
guarantees the diversity of the judgment pool, as a result, thefinal judgment won’t be greatly influenced by someindividuals in the crowd.
III.

HOW CROWD WISDOM PARTICIPATING IN
MANUFACTURING?

Fig. 2 gives the overview of how crowd wisdom
participates in the manufacturing process. The manufacturing
process could be generally divided into Requirement Eliciting,
Design, Trial, Production, Marketing, and Upgrade. Each
phase is confronted with a main judgment and decision, whichis specific to this phase but critical to the whole manufacturing
procedure. System accomplishes the decision-making tasks
with the help of a large amount of data that contributed fromcrowds. Thus, the participation of crowds is vital to accurate
judgments and smart decisions, and the influence exists
throughout the whole process.
A. Manufacturing from Customers, for Customers
The concept of intelligent manufacturing is tremendously
extensive, among which an important proposition is productinnovation. Normally, user requirements point out thedirection for enterprises. Nevertheless, it is not easy forenterprises to fully understand users’ real demand. As a result,
it is common that contrary to its over-rhetoric advertisement,

new product is of no novelty at all, but full of trivial so-called
new functionality.
To solve the dilemma, C2B(customer to business) mode
was proposed. The core of C2B is that customers bring value
to the company, involving consumers co-creating ideas,
product or service concepts and solutions with a company
through the social media [5]. Customers put forward their
dream product and then businesses produce accordingly. In thegeneration of intelligent manufacturing, the traditional
B2C(business to customer) would naturally transform into thebottom-to-top C2B mode, where user’s desire substantiallydominates manufacturing. There is no need for business todevise new products painstakingly with its own innovationteam and then persuade the customers to buy their stuff, since
they only produce what customers desire for. Obviously, it is amuch more efficient mode and it promotes personalization andcustomization, which are also key words in the theory ofintelligent manufacturing.
A distinction between B2C and C2B is the involvement ofcustomers’ opinion. It is reasonable to ascribe the advantage
of C2B to its positive employment of the power of customer
group, in other words, the wisdom of crowd.
B. The Intelligent Crowd
The paragraphs above may create an illusion: intelligent
design is a really simple task for enterprises, all they would do
is to ask customers, then adopt customer’s choice. It seems
that innovation department can be disbanded forever.
There is a possibility that the design process of someproducts is that simple. For example, lots of brands had
encouraged their customers to design and vote, and they
produced the most popular design. But in most cases, theprocess is more complex, and the intelligent crowd involves
more individuals rather than customer group solely.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

38

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Innovation departments are needed. Most of the enterprises
cling to its own characteristics, such as the front design of
BMW series. Internal staff’s judgment would bring in suchconsideration and thus the final judgment would be a balance
between customer needs and the enterprise style.
Another group of important participants is investors, or the
management of enterprise. Judgment from investors would
take business factors into consideration, reflecting a new angle
in the final crowd judgment.
A relative concept is crowdfunding [6], that is, a presented
idea appeals for various investors’ funding on Internet, andonly with adequate funding the project can proceed. This issimilar to the concept of ‘voting with feet’ in the discipline ofcorporate finance, where you express your judgment bybuying or selling stocks. Similarly, investors provide theirpersonal judgment in financial way.
Besides, potential contributors may include various
seemingly irrelative individuals not belonging to the groups
above. Consider an example of car design: mechanics andwashers can provide useful information about car design. For
example, the existing design with superfluous details brings
plenty of inconvenience for the maintenance.
C. Data Reveals Wisdom, but Crowd Creates Data
Now we have to answer a question: when aggregating the
judgment from individuals, what are we actually aggregating?
Are we sheerly aggregating the judgments?
We mentioned that “the base of crowd wisdom is theprivate information within individuals’ judgments”. What we
actually aggregate is the information conveyed in the form of
judgment. However, when individuals are unwilling to express
their judgments or inconvenient to utilize their private
information, can we skip the step of individual making
judgment but aggregate information directly? Can we stillobtain crowd wisdom in this way? Can we paraphrase “thewisdom comes from the crowd, as the crowd owns abundant
information” as “the wisdom comes from abundant
information”?
Sure we can. To some extent, we can simply ignore thejudgment of individuals. The judgment ability of theindividuals can’t explain the superior performance of crowd’s
judgment. The good performance is the result of the quality of
input information, precisely, whether the information isabundant and diverse enough to depict the fact wholly.
And thus we can explain the importance and attention the
public puts on Big Data, which is a hit in the past years. The
industry widespreadly highlights the necessity of Big Data inthe on- going revolution to intelligent manufacturing [7]. A
more common expression is ‘Big Data drives Intelligent
Manufacturing’ rather than what we propose here. But wemust understand the reason why Big Data could provide us
smart judgment and decision. In fact, Big Data is the abundant
information generated by individuals from the crowd,
including the netizens(Internet citizens) crowd. And thus, as
our discussion above, the truth is depicted and the reliability ofan aggregated judgment based on the data is pretty guaranteed.

Therefore, we can conclude that the wisdom of Big Data is thewisdom of crowd.
It seems that something inconsistent with what we
described earlier. An individual’s judgment is no longer
necessary but we still rely on the individuals to explore
abundant and diverse information for the crowd. And thus the
final crowd judgment still reveals the crowd wisdom.
D. Data Collection and Crowd Collaboration
Information is far more comprehensive than user
requirements solely. And thus what we ask for from theindividuals in the crowd extends beyond their own judgment.
Particularly, customers are expected to contribute personal
habits of consumption, evaluation of the purchased products
and so on. Various, ordinary, even seemingly unrelated
information may tell incredible business secret when they are
aggregated. And that is why lots of enterprises are eager for
user information, even when the information is of no
relevance to its business.
The process can be described as collaboration. Informationfrom a single individual is partial, but when information fromall the individuals working as a whole, it is complete enoughfor an accurate description of the truth, assisting
manufacturing decision further than providing inspiration anddesign. For example, with abundant feedbacks, decision likeincreasing or reducing production can be made automatically,
defects feedback could be collected and used to guide betternext-generation product, the production process would gain
improvement and management technique would get upgraded.
In a word, the more individuals in the crowd contribute,the more powerful of the crowd wisdom, and thus the better
intelligent manufacturing would function.
E. Aggregation: the Emergence of Wisdom
So far, we blur the boundary between judgment anddecision. The explanation for this vagueness is that a smart
decision is usually derived from an accurate judgment to thetarget and thus they are the same in essence. Also, we haven’t
explicitly explained how a crowd make judgment.
In fact, we would not and could not explore in how thecrowd decision is made. This is a question of how to deal withthe pool. We guarantee you that wisdom does in the pool. But
as for how to extract the wisdom, the techniques vary.
In free markets, price is the method, while in diverse
evaluation experiments, which J.Surowiecki mentioned in his
book, statistical methods, such as counting votes arerecommended. A point worth noting is that the method should
ensure that all the information would be made good use of andunified to work as a component of the whole [3].
For some problems, the technique is naïve and explicit,
just as what we mentioned above, while for others, it involves
more advanced technology , and that’s the subject of anotherdiscipline, that is, Artificial Intelligence [8], an art ofextracting. Artificial intelligence, along with all of us, would
be indispensable in the trends of intelligent manufacturing.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

39

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

IV.

CROWD WISDOM DRIVES MANUFACTURING: SERVERALCASE STUDIESIn the following paragraphs, we’ll study several cases,
where enterprises successfully utilize the crowd wisdom.
A. Crowd Wisdom Inspire Design
In February 2016, Haier launched a water heater withpurification functionality. The sales of the new heater
surpassed 10 thousands only in 21 days [9].
Not long ago, Haier received complaints from over 300
thousand neitizens about unclean bathwater. After consultingwith neitizens, Haier was inspired to develop a new series ofheaters. The new functionality is the fully-purification
capacity, not only eliminating the bacteria, but also removing
the residual chlorine in tap water and sediment [10].
Complaining customers must be far more than 300
thousand and the contents must be diverse. Entering Haier’s
official website, you’ll notice that some complain about theprice, some are dissatisfied with the inconstant temperature.
But the 300 thousand must be the majority and thus statistics
works. A simple aggregate of individual judgment leads to a
crowd judgment: purification functionality is a good idea.
The crowd wisdom efficiently facilitates Haier on the issue
of novelty, thus researchers and developers can focus ontechniques.
B. Crowd Wisdom Streamlines Production
In Zara stores, there are cameras hidden in every cornerand staff carrying with PDA. The purpose is to collectcustomer’s every single piece of advice, like preference
towards patterns or the size of buttons. Besides, Zara’s online
stores contribute more data. Zara has opened six online storesin Europe. Every day, the collected information would be
aggregated to Zara’s internal global network and facilitate thedesigner. Once the design is determined, it would be
immediately sent to the production line.
The data do help a lot. Making production decision basedon collected data greatly reduces inventory thus reduces costs.
Also, analysis of the data reveals domestic prevailing colorand other information which helps discover target markets anddesign.
The Inditex Group, the parent company of Zara, prides
itself to deliver quality merchandise in as little as three weeks
from its own factories. Designers develop new models daily,
sometimes three or four a day, which are then reviewed andput into production [11].
These valuable customer information is not only applied in
the manufacturing, but also in all departments throughout theInditex Group, including customer service centers, marketing
departments, design teams, and production lines, etc. Thesemassive data forms all departments KPI to complete thevertical integration principle axe inside Zara.
And the optimization based on positive utility of massive
data does bring substantial profits. In fiscal 2014, (ended
January 31, 2015) the company had sales of $19.7 Billion

compared to Uniqlo $16.6 Billion, Gap $16.4 Billion, Mango$2.1 Billion. Inditex sales increased 8%, far stronger than its
competitors [12].
C. Crowd Wisdom Creates Business Opportunity
A challenge for enterprises is that not all the customers arewilling to provide their information. Obtaining a piece ofadvice may be easy, but for more private information, such as
birth, age, daily routine, lots of people would regard it asinvasion of privacy and strongly oppose it.
But since enterprises are desperate for the information
revealed from customers’ data, they have to resort to tricks
and strategies. The strong motivation drives enterprises tocreate new business modes, which enable them to establish
intimate relationships with their customers.
Nike transformed its sports shoes into wearable devices,and connected user data to Nike+ fitness social network byshoes itself or wristband sensor. Nike shoes would
automatically record users’ training data while iPod can storeand display the data, including motion date, time, distance,
calories burned and so on. Users upload data to the Nike
community, share and discuss with friends. Members in thecommunity can compare, analyze, and set targets for their own
training [13].
Nike accesses users’ data by means of encouraging themkeep uploading their training records. With the growing of thecrowd, Nike learns about the best jogging routine in major
cities, even grasp a list of pop songs for jog lovers.
If Nike plans to organize a marathon, in order to advertiseits new products, the data and the crowd wisdom revealed
would be of great help. Besides, the abundant data plays animportant role in efforts to understand user habits, improving
products and so on.
V. CONCLUSION
Wisdom derives from crowds, to be precise, from theinformation contributed by the crowd. Information, in theforms of various data, may be seemingly unrelated or useless,but actually tells a lot when aggregated, which we proved withseveral real cases. Crowd wisdom assists, or even partly
substitutes humans to make judgments, which could be wellapplied on manufacturing, where plenty of judgments anddecisions should be made in the process. In the blueprint of
future industries, crowd wisdom would surly drive thedevelopment of intelligent manufacturing.
ACKNOWLEDGMENTS
The authors would like to acknowledge the support
provided by the National Natural Science Foundation of China(61402263, 61503217), the National High Technology
ResearchandDevelopment
Programof
China(2014AA01A302), the special funds of Taishan scholar
construction project, the Independent Innovation Projects ofShandong Province (2014ZZCX08102, 2014ZZCX03409), theNatural Science Foundation of Shandong Province(ZR2014FQ031, ZR2014FP002), and the Science &© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

40

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Technology Development Projects of Shandong Province(2014GGX101028, 2014GGH201007).
REFERENCES
[1]

[2]

[3]

[4]
[5]
[6]

S.Yang, H.Ding, "A Research on Intelligent Manufacturing
Techonology and Intelligent Manufacturing System", China MechanicalEngineering, vol 2, 1992, pp.15-18
J.Qu, S.Liu, "The Research Based on Multi-Agent for the Pattern of theIntelligent Manufacture ", Intelligent Computation Technology andAutomation, 2009. ICICTA '09. Second International Conference on(Volume:4 ), pp.15-17.
J. Surowiecki, The Wisdom of Crowds: Why the Many Are SmarterThan the Few and How Collective Wisdom Shapes Business, Economies,
Societies and Nations. New York: Doubleday, 2004.
Z.Zhang, J.Kou, F.Chen, M.Li ， "Strategic Analysis of C2B BasedIntelligent E-commerce Model"，Soft Science, vol.4, 2010, pp.53-59
P.Hart. “Irving L. Janis' Victims of Groupthink”,Political Psychology,
vol.12.2, 1991, pp.247–278.
J.Jussila, K.Menon, R.R.Mukkamala, L.A.Lasrado, A.Hussain,
R.Vatrapu, H.Kärkkäinen, J.Huhtamäki, "Crowdfunding in theDevelopment of Social Media Fanbase -- Case Study of Two Competing

Ecosystems" ,2016 49th Hawaii International Conference on System
Sciences (HICSS), pp.4495 - 4504
[7] J. Krumeich, S. Jacobi, D. Werth, and P. Loos, “Big Data
Analytics for Predictive Manufacturing Control – A Case Study
from Process Industry,” Proc. 3rd IEEE Int. Congress on Big Data, pp.
530-537.
[8] F.Meziane, S.Vadera, K.Kobbacy, N.Proudlove, “Intelligent Systems in
Manufacturing:Current Developments and Future Prospects”, The
Internaltional Journal of Manufacturing Technology Management,11(4),pp.218-238[9] Y.Xu, "Haier Pure Water Heater co-created by 300 thousands users sold10 thousand units in 21 days",http://article.pchome.net/content1898216.html，accessed 2016-4-20.
[10] L.Su, "Healthy Bath，Pure Water Electric Heaters Leading New Trend",
Household Appliance, vol.9, 2015，pp.62-63[11] W.Chen, "Business - IT Strategic Matching :A Case Study ofZara",Economic Survey, vol.05, 2009, pp.31-34[12] W.Loeb,
"Zara
Leads
In
Fast
Fashion",http://www.forbes.com/sites/walterloeb/2015/03/30/zara-leads-in-fastfashion/#e607cbc61d79, accessed 2016-4-20.
[13] Y.Wu, J.He, "How NIKE+ is Using Big Data?", Tsinghua Business
Review, vol.12, 2014, pp.10-17.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

41

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Design on Information Providing Service for
Temporary Organizations
Tetsuya Oishi

Rieko Fujita

Tokuro Matsuo

Advanced Institute ofIndustrial Technology

Advanced Institute ofIndustrial Technology

Advanced Institute ofIndustrial Technology

Abstract—Most of all organizations try to get more benefit
by gathering and analyzing various data. Especially, in profitmaking organizations, these systems work well. In recent years,
permanently voluntary organizations such as universities became
active in order to know themselves. However, temporary voluntary organizations do not have any such systems. As a result,
such organizations cannot always decide something precisely. In
this paper, we design a mechanism to solve this problem for
temporary voluntary organizations. Using simple example, we
also simulate this mechanism.

II. P RELIMINARY
A. Cases based on Experiences
There is the influence of leadership soft skills, which include communication skill to gather various data, with school
improvement [1]. That is to say, organizations that have nosoft skills will be ruined. For instance, universities in USA
use data to raise donation for their better performance andmanagement [2].
B. Institutional Research

I. I NTRODUCTION
In recent year, we have become to be able to analyze
a large-scale data because Information and Communication
Technology (ICT for short) has been highly developing. Mostof organizations as well as end users store various data intheir computers. Computer users can analyze various data
individually by their own Personal Computers (PCs for short)which have high performance hardware. In most of all profit
organizations, their stored data have been analyzed and used
to get more benefit. For example, these data include clients
attributes and behaviors, income and profits, number of staffs,
number of branches, advertisements, and so on. Even thoughvoluntary associations and organizations have that kind ofdata, they sometimes have some difficulties to analyze suchdata and to make a strategy for their management due to few
data analysts in the organizations. Universities and research
institutes which do not pursue profits use the method of
Institutional Research (IR for short) in order to improve theirconditions, performances, managements, and so on.
This method is available for temporary voluntary associations and organizations. For example, we assume that thepeople who live in a local community have to decide to build
new institutions, to hold a festival, or to sell special products.
Then, they will create temporary organizations. Also, theseorganizations are often voluntary. There are similar temporary
voluntary organizations in the world. However, the decisions
produced from these organizations hardly contribute to othersimilar organizations. It is very hard because there are no
systems to provide a temporary voluntary organization with theinformation created by the other organizations. In this paper,
we design a method of providing such temporary organizations
with many information to obtain the best decision.

There are many permanently voluntary associations andorganizations. University is an example of these organizations.
Universities must do a self-check on aspects like education, scientific research and social contribution. Institutional
Research (IR for short) is receiving a lot of attentions byuniversities. IR is an activity to provide information andits analysis, which supports institutional planning, decisionmaking and improvement [3]. Virtual environments that enableInstitutional Researchers (IRers for short) to share and codevelop informational systems to gather various data wereintroduced in order to conduct IR [4][5]. Especially, many universities introduced questionnaire survey systems and gatheredvarious data from students and teachers [6][7].
III. D ESIGNIn this section, we show a design on information providing
service for temporary groups which mean organizations andassociations. After we explain the traditional mechanism, we
propose our design.
A. Traditional Mechanism
We assume that there is a temporary voluntary groupindependently. The group can use its own information in orderto decide something but hardly uses the the information theother groups have because there are no services providing
information one another. Actually, the information the group
can use is composed of its own attributes and its own previous decision. The situation is represented by the following
formula.
Dp (tm ) = Fp (Ap (tm ) , Dp (tm−1 ))(1)Here, the symbols used in the formula (1) are as follows.
• Dp (tm ) means the decision that the temporary voluntarygroup Gp can obtain at the time tm .

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

42

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 1. Traditional Mechanism

Fp means the function as a process of the discussion inthat a temporary voluntary group Gp using Ap (tm ) andDp (tm−1 ).
• Ap (tm ) means the attributes that the temporary voluntarygroup Gp originally has at the time tm .
• Dp (tm−1 ) means the decision that the temporary voluntary group Gp obtained at the time tm−1 .
• tm−1 means the previous time of the time tm .
•The decision Dp (tm ) is greatly affected by Ap (tm ) because
the group Gp hardly uses the information from the othergroups. Fig. 1, which consists of squares, circles, and arrows,
explains the formula (1). Here, the squares mean the decisions,the circles mean the attributes, the solid arrows mean theeffects on the decisions by the attributes, and the broken arrows
mean the effects on the decisions by the decisions of theother groups. The thick arrows mean the attributes and thedecisions affect the decision directly, for example, number andsort of participants, fund of the group, opinions created fromthe group, and so on. The thin arrows mean the attributes andthe decisions influence the decision indirectly, for example,
rumor, entries of blogs, and so on.
B. Proposed Service
We assume that there are n temporary voluntary groups.
The difference with the traditional mechanism mentioned in
Section III-A is that one of these organizations can use not
only its own data but also the data from other organizations
in order to decide something. The situation is represented by
the following formula.
n∪
Dp (tm ) = Fp Ap (tm ) ,Dj (tm−1 )
(2)
j=1Here, the symbols used in the formula (2) are as follows.
•

Dp (tm ) is the decision that a temporary voluntary group
Gp can obtain at the time tm . The group Gp is one ofsimilar temporary voluntary groups. The number of thesegroups is n.

Fig. 2. Proposal Mechanism

TABLE IA SSUMPTION AND F INAL D ECISION IN C ONCEPTUAL S IMULATION

group
number
G1
G2
G3

numberofcommittee
members
10
10
10

attributes
estimate
numberofparticipants
600
100
300

initialcapital$1,000$300$2,000

final
decisionheld
not heldheld

Fp means the function as a process of a discussion inthat a temporary voluntary group Gp using Ap (tm ) andDj (tm−1 ) where j = 1, · · · , n.
• Ap (tm ) means the attributes that a temporary voluntarygroup Gp originally has at the time tm .
• Dj (tm−1 ) means the decision that a temporary voluntarygroup Gj obtained at the time tm−1 .
• tm−1 means the previous time of the time tm
The decision Dp (tm ) is greatly affected by Ap (tm ) andDj (tm−1 ) where j = 1, · · · , n. In our proposed service, whena group decides something at the time tm , the group can use
the data from own and other decisions at the time tm−1 . Fig.
2, where the symbols used in the figure are same as the ones
mentioned in Section III-A, explains formula (2). Our service
enables a group to decide something precisely by providing
not only own data but also data obtained from other groups.
•

IV. C ONCEPTUAL S IMULATIONIn this section, we show a conceptual simulation using ourservice. We assume that there are three temporary voluntarygroups (G1 , G2 , and G3 shown in TABLE I) which intend to
decide whether their reunion should be held. Each group has
number of committee members, estimate number of participants, and initial capital as attributes .
A. Discussion in G1 at time t0
The formula (3) shows that the group G1 starts discussing
at time t0 with only its own attributes (A1 (t0 )) because there
are no own and other decisions.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

D1 (t0 ) = F1 (A1 (t0 ) , ∅)

(3)

43

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 3. Discussion in G1 at time t0

Fig. 5. Discussion in G3 at time t2

Fig. 4. Discussions in G1 and G2 at time t1

Fig. 6. Discussion in G3 at time t3

Fig. 3 shows that G1 could not decide anything at t0 , that is
to say, D1 (t0 ) means “undecided”.
B. Discussions in G1 and G2 at time t1
The formula (4) and (5) show that the group G1 and G2
start discussing at time t1 with these own attributes (A1 (t1 )and A2 (t1 )) and the decision (D1 (t0 )), respectively.
D1 (t1 ) = F1 (A1 (t1 ) , D1 (t0 ))(4)

D2 (t1 ) = F2 (A2 (t1 ) , D1 (t0 ))(5)

Fig 4 shows that G1 could decide that its reunion would beheld at t1 , that is to say, D1 (t1 ) means “held”. Also, thisfigure shows that G2 could decide that its reunion would not
be held at t1 , that is to say, D2 (t1 ) means “not held”. The
reason why the reunion for G2 will not be held, for example,
is that there is some fear to be in deficit after holding thereunion because of small estimate number of participants.
C. Discussion in G3 at time t2
The formula (6) shows that the group G3 starts discussing
at time t2 with these own attributes (A3 (t2 )) and the decisions(D1 (t1 ) and D2 (t1 )).
D3 (t2 ) = F3 (A3 (t2 ) , D1 (t1 ) ∪ D2 (t1 ))(6)

Fig. 5 shows that G3 could not decide anything at t2 , that is
to say, D3 (t2 ) means “undecided” because D1 (t1 ) is “held”
while D2 (t1 ) is “not held”. Then the group G3 put off adecision next time.
D. Discussion in G3 at time t3
The formula (7) shows that the group G3 starts discussing
at time t3 with these own attributes (A3 (t3 )) and the decisions(D1 (t2 ), D2 (t2 ), and D3 (t2 )).
D3 (t3 ) = F3 (A3 (t3 ) , D1 (t2 ) ∪ D2 (t2 ) ∪ D3 (t2 ))(7)Here, the formulas D1 (t2 ) = D1 (t1 ) and D2 (t2 ) = D2 (t1 )are valid becauseD1 (t2 ) and D2 (t2 ) are same as decisions at
t1 . Fig 6 shows that G3 could decide that its reunion would beheld at t3 , that is to say, D3 (t3 ) means “held”. The reason why
the reunion for G3 will be held, for example, is that there is a
firm belief to be in surplus after holding the reunion becauseof enough estimate number of participants and initial capital.
As mentioned above, our design enables a group to decide
something by using the decisions of the other groups. Using
traditional mechanism, the group G3 might decide not to beheld reunion even though there are potential to be held reunion
in its attributes.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

44

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IIA SSUMPTION AND F INAL D ECISION IN S IMULATION BY C OSINES IMILARITY (xi,j )Fig. 7. Schematic diagram of simulation by using cosine similarity

V. C ALCULATIVE S IMULATIONIn this section, we show a calculative simulation using ourservice. We assume that there are twenty groups that havethree attributes respectively as shown in TABLE II. Here, firstnineteen groups from G1 to G19 have already decided whether
they should hold their reunions at time tm−1 . In this condition,
we will explain how group G20 can decide whether its reunion
should be held at time tm . In Fig. 7, we show a schematic
diagram of this condition. Moreover, what we will explain isshown in the formula (8).

19∪D20 (tm ) = F20 A20 (tm ) ,Dj (tm−1 )
(8)

group
number
G1
G2
G3
G4
G5
G6
G7
G8
G9
G10
G11
G12
G13
G14
G15
G16
G17
G18
G19
G20

numberofcommittee
members
a1
10
10
10
10
13
18
4
12
12
15
18
14
10
12
19
2
12
4
18
19

attributes
estimate
numberofparticipants
a2
600
100
300
572
604
429
82
488
613
402
230
259
743
170
215
224
700
737
372
191

initialcapitala3$1,000$300$2,000$1,325$1,109$256$1,449$443$1,686$1,722$245$1,146$288$451$1,077$1,047$460$1,975$1,129$414

finaldecisionheld
not held
held
held
held
not held
not held
not heldheld
held
not held
not heldheld
not held
not held
not held
held
held
held
undecided

v
u ∑
u1 n
2
si = t(xi,j − xi )
n j=1

(10)

j=1

We use the cosine similarity as F20 in the formula (8). In
order to use the cosine similarity, we have to define vectors.
We will show how to make vectors before we explain thecosine similarity.
A. Making Vector
We treat values shown in TABLE II as numeric values
simply. First, we define that xi,j is the value of the attribute
ai for group Gj . Here, a1 , a2 , and a3 means the number ofcommittee members, the estimate number of participants, andthe initial capital respectively as shown in TABLE II.
The cosine similarity we will mention later can be influenced by high value shown in a vector. Since we would liketo make vectors for every group, the elements of the vector
are the values of attributes. For example, the cosine similarity
will be influenced by attribute a3 because the values shownin a3 are higher than the other attributes in TABLE II. Then
we calculate x′i,j after normalizing xi,j .
We have decided to use the standard deviation in order tonormalize xi,j . In this method, all values of attributes become
from −1 to 1. First we have to calculate xi which is theaverage of values for attribute ai and si which is the standard
deviation of values for attribute ai . xi and si are calculatedby the formula (9) and (10) respectively.
xi =

n
1∑

n j=1

xi,j

Here, n is the total number of groups, that is, n = 20. x′i,j iscalculated by using xi,j , xi , and si as shown in formula (11).
x′i,j =

xi,j − xisi(11)

We make vectors VGj (j = 1, · · · , 20) which are consisted by
x′i,j for group Gj as shown in TABLE III. VGj is shown informula (12) because there are three attributes which are a1 ,
a2 , and a3 in our situation.
VGj = (x′1,j , x′2,j , x′3,j )(12)

B. Cosine Similarity
We calculate Sim(VGp , VGq ) which is a cosine similarity
between group Gp and group Gq . Sim(VGp , VGq ) is calculated
by using formula (13).
Sim(VGp , VGq ) =VGp · VGq
|VGp ||VGq |(13)

Here, VGp · VGq and |VGj | are calculated by using formula(14) and (15) respectively.
VGp · VGq = x′1,p · x′1,q + x′2,p · x′2,q + x′3,p · x′3,q(9)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

|VGj | =(14)√
x′1,j 2 + x′2,j 2 + x′3,j 2

(15)

45

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IIIN ORMALIZATION OF ATTRIBUTES IN TABLE II (x′i,j )

group
number
G1
G2
G3
G4
G5
G6
G7
G8
G9
G10
G11
G12
G13
G14
G15
G16
G17
G18
G19
G20

numberofcommittee
membersa1-0.434-0.434-0.434-0.4340.186
1.220-1.675-0.021-0.021
0.600
1.220
0.393
-0.434-0.021
1.427-2.088-0.021-1.6751.220
1.427

attributes
estimate
numberofparticipants
a2
0.936-1.422-0.479
0.804
0.955
0.129
-1.507
0.408
0.997
0.002
-0.809-0.672
1.611-1.092-0.880-0.8381.408
1.582-0.139-0.993

initialcapitala3
0.041-1.171
1.774
0.604
0.230
-1.2480.819-0.9241.230
1.292-1.2670.294-1.192-0.9100.175
0.123-0.8941.731
0.265-0.974cosinesimilarity
with
G20-0.7720.513-0.497-0.919-0.4550.811-0.3540.231-0.703-0.1430.979
0.568-0.2590.685
0.816-0.505-0.168-0.9830.648
1.000

C. Decision for group G20
We compare group G20 with other nineteen groups by using
formula (13). On the most right line in TABLE III, the cosine
similarities with group G20 are shown. Sim(VGp , VGq ) =1 means that vectors VGp and VGq turn in the same direction, that is, VGp is most similar to VGq . Conversely,
Sim(VGp , VGq ) = −1 means that vectors VGp and VGq turnin the opposite direction, that is, VGp is most different fromVGq .
What we have to do is to find the cosine similarity that is themost nearly 1 in TABLE III. We can find group G11 that has
the cosine similarity 0.979 with group G20 . We can also findgroup G18 that has the cosine similarity −0.983 with group
G20 . Then we verify whether group G20 is really similar togroup G11 , and whether group G20 is not really similar togroup G18 .
TABLE IV contains group G11 , G18 , and G20 which are
picked out from TABLE II. We can understand that group
G20 is surely similar to G11 and group G20 is not also similar
to group G18 .
From these verification, group G20 can decide whether theirreunion should be held by using the decision of G11 . GroupG11 had already decided that their reunion was not held. Then,in this simulation, group G20 is proposed that their reunion
should not be held.

TABLE IVC OMPARING G11 WITH G20

group
number
G11
G18
G20

attributes
estimate
numberofparticipants
a2
230
737
191

numberofcommittee
membersa1
18
4
19

initialcapitala3$245$1,975$414

final
decision
not held
heldundecided

A. Distributed Database
In recent year, distributed databases are developed [8][9].
Attributes and decisions of every group can be stored in these
databases. Final implied decisions can be varies by analysisof stored data. There are two analyses: one is comparison
between similar groups and another is observation of estrangement from average of all groups. When similar groups
are compared each other, the stored data are still distributed.
Wrong decisions can be implied because the groups that are
not similar to each other are ignored even if these ignoredgroups may have useful data. If all distributed data can begathered together, we can know the average of all data. Eachgroup can obtain suitable information by comparing own datawith average data.
B. Information Recommendation
In our design, some facts from every temporary voluntarygroup for similar purpose can be used. Not only the facts
are provided to each group, but also some proposals can berecommended by implying these facts [10][11]. In each group,
some proposals are available instead of deciding somethingfrom these facts. For example, it is happy that a suitable proposal is recommended after entering attributes to the systemthat realized the function mentioned above.
VII. C ONCLUSION
In this paper, we explained the design on information
providing service for temporary voluntary organizations. Until
now, such organizations could use only their own informationsuch as their attributes and decisions. Our design enables
each organization to use information from other organizations.
Actually, these information is not always received perfectly. In
future work, we will pay attention to the loss of information.
We will investigate how the sent information will be used by
the receivers. Moreover, we will observe how the decisions
will be changed by the abilities of the receivers.
ACKNOWLEDGMENT
This work was supported by CREST, JST.
R EFERENCES

VI. D ISCUSSION
In this section, we discuss how the service we designed in
Section III is realized.

[1] Tang Keow Ngang, Siti Huwaina Mohamed, and Somprach Kanokorn,
Soft Skills Of Leaders And School Improvement In High Performing
Schools, Procedia - Social and Behavioral Sciences, Vol. 191, pp.21272131, 2015

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

46

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[2] Angela Besana and Annamaria Esposito, Economics and Marketing of
USA Universities, Procedia Economics and Finance, Vol. 14, pp.68-77,
2014[3] J.L. Saupe, Functions of Insitutional Research, 2nd Edition, published by
Association for Institutional Research, 1990.
[4] Tetsuya Oishi, Masao Mori, and Eiichi Takata, An Environment to Codevelop Informational Systems for IR in Japan, Information Engineering
Express, Vol. 2, No. 1, pp.55-64, 2016[5] Tetsuya Oishi, Masao Mori, Toshie Tanaka, and Eisuke Ito, Cooperative
Development and Individual Operation for Information Systems of IR
in a Virtual Environment, 2nd International Conference on Institutional
Research and Institutional Management (IRIM 2013), 2013[6] Tetsuya Oishi, Masao Mori, and Eiichi Takata, A proposal of Human andMachine Reminding for High Response Rate in Use of a Web Questionnaire Survey System, International Journal of Computer & Information
Science, Vol.16, No. 1, pp.22-31, 2015[7] Satoshi Taoka and Toshimasa Watanabe, A Web-Based Questionnaire andIts Evaluation of Class-Assessment-by-Students, The Transactions of theInstitute of Electronics, Information and Communication Engineers D,
Vol J97-D, No.5, pp. 1024-1034, 2014.
[8] Bernstein, Philip A., and Goodman, Nathan, Concurrency Control in Distributed Database Systems, ACM Comput. Surv., Vol.13, No.2, pp.185221, 1981.
[9] Loesing, Simon and Pilman, Markus and Etter, Thomas, and Kossmann, Donald, On the Design and Scalability of Distributed SharedData Databases, Proceedings of the 2015 ACM SIGMOD International
Conference on Management of Data, SIGMOD ’15, pp.663-676, 2015.
[10] Kenta Oku, Shinsuke Nakajima, Jun Miyazaki, and Shunsuke
Uemura,Context-Aware SVM for Context-Dependent Information Recommendation, Proceedings of the 7th International Conference on Mobile
Data Management, MDM ’06, p.109, 2006.
[11] Ahmed I. Saleh, Ali I. El Desouky, and Shereen H.Ali, Promoting
the Performance of Vertical Recommendation Systems by Applying New
Classification Techniques, Know.-Based Syst., Vol.75, No.C, pp.192-223,
2015.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

47

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Design Tradeoffs for Cloud-Based Ambient
Assisted Living Systems
Yi Dong∗ , Yonggang Wen† , Han Hu† , Chunyan Miao∗† and Cyril Leung∗∗ Interdisciplinary Graduate School (IGS) and † School of Computer Science and Engineering (SCSE)

Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798
Email: {ydong004,ygwen,hhu,ascymiao,cleung}@ntu.edu.sg

Abstract—Ambient assisted living (AAL) has received considerable attention due to its ability to provide services to the elderly
by sensors and actuators. However, building such a system ischallenging on two fronts. First, the tradeoff between accuracy
and monetary cost should be understood. Accuracy of each sensor
can be empirically estimated from its sample rate. Typically,
higher rate indicates higher accuracy. As a result, higher raterequires more computation resources to process the sampled
data, incurring more monetary cost. Second, user needs change
frequently. Thus, we need a resource allocation scheme that is (a)able to strike a good balance between accuracy and monetary
cost and (b) adaptive enough to meet the frequently changing
needs. Unfortunately, several seemingly natural solutions fail on
one or more fronts (e.g., simple one shot optimizations). As
a result, the potential benefits promised by these prior efforts
remain unrealized. To fill the gap, we address these challenges
and present the design and analysis of a low-complexity online
algorithm to minimize the long-term accuracy-monetary cost on
a queue length based control. The design is driven by insightsthat queue-lengths can be viewed as Lagrangian dual variables
and the queue-length evolutions play the role of subgradient
updates. Therefore, the control decisions depend only on the
instantaneous information and can adapt to the changing needs.
Simulations demonstrate that the proposed algorithm can strikea good balance between accuracy and monetary costs. Moreover,
the asymptotic optimality of the proposed algorithm has beenshown by rigorous analysis and numerical results.

I. I NTRODUCTION
Caring for elderly people and the disabled is an importantbut challenging undertaking. According to United Nations,the number of people aged 60 years or over, is increasing
rapidly and will reach nearly 2.1 billion in 2050 [1]. Although
the elderly have wisdom and wealth gathered from their life
experience, they often require long-term assistance. Ambientassisted living (AAL) systems open up a new way to address theneeds of the aged and the disabled by utilizing the capabilities
of information and communications technologies [2]. Figure 1
shows the typical reference architecture of cloud based AAL
systems. To fulfill the various needs of the elderly, multiple
sensors and actuators coexist in ambient assisted living (AAL)
systems. The sensed data are processed in the cloud because
many applications are computation intensive. The applications
infer knowledge from the data, guiding the actuators to meetthe needs of the elderly.
For AAL systems, critical factors include the accuracy,
monetary cost and delay of the services. Each sensor samples
and transmits data at its own rate with a certain accuracy. The

accuracy of the sensed data can be estimated [3]. Experiments
show that data sensed at high rate usually means highaccuracy. However, it demands more computational resources
to process the sensed data, leading to higher monetary cost.
Moreover, users’ requirements change throughout a day andare highly time-varying. Therefore, it is important to design adynamic scheme to allocate the resources effectively so thatthe operational costs are reduced and the user experience issatisfactory.
There are several existing efforts on sensor accuracy estimation, AAL systems and mobile cloud. The accuracy estimation
for sensor systems has been done mainly through inference [4]and learning [5, 6] techniques. Synthesizing previous works, [3]proposed a framework to estimate the accuracy of certain sensor
measurements invoking KL divergence of the distributions.
For the AAL systems, most of the works focus on energy
efficiency. For example, Wu et al. [7] propose asynchronous
flow scheduling taking into account energy efficiency andimplementation simplicity. There have been extensive studies
on efficient operations of mobile cloud [8–12]. However,none of these publications consider accuracy as an important
performance metric and jointly solve this problem.
We investigate the tradeoff between accuracy and monetary
costs in a typical AAL scenario with multiple sensors andapplications, where the user pattern is not predictable. We
propose a low-complexity online algorithm by invoking the Lyapunov optimization and mathematically prove that it minimizes
accuracy-monetary cost for given delay constraints. Simulations
based on real measurements demonstrate the effectiveness ofthe proposed algorithm. Our investigation may provide insights
into the real world deployment of AAL systems.
Contributions and Roadmap:• Joint investigation of the accuracy, monetary cost andqueueing delay for cloud based AAL systems (§II).
• Proof of theoretical bounds and design of online algorithm
for optimal accuracy-monetary cost minimization (§III).
• Evaluations that demonstrate the effectiveness of the
proposed algorithm (§IV).
II. S YSTEM M ODEL AND P ROBLEM F ORMULATIONIn this section, we first introduce the typical cloud based
AAL system which is based on real world deployments. Then,
we present various models for the cloud based AAL system,
including the sensor accuracy model, the queueing model and© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

48

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Cloud Resources

Mobile Phone

Sensors & Cameras

App 1

App 2

Robot

Electrics

App 3

The data
visualization
Care Giver
The elderly
using themobiledeviceFigure 1. The reference architecure of a cloud based Ambient Assisted Living system. An AAL system consists of three parts: data sensing; data mining
and processing; control and feedback.

the cloud monetary model. Finally, we formulate the rate
selection as a constrained optimization problem. For futurereference, we summarize the models and key notations inFigure 2.
A. System Architecturedesired balance between accuracy, monetary cost and queueing
delay.
B. System ModelIn the AAL system, there are N sensors which monitor theenvironment. Sensor i samples and transmits at a rate ri (t) attime t. We consider a time-sloted system with time slots ofequal length indexed by t = 0, 1, . . . . The actual duration ofeach time slot is application dependent. We now describe sub
models as follows.
1) Sensor Data and Loss of Accuracy: The accuracy of
sensors can be easily estimated using the observed data. In thispaper, we adopt the framework proposed in [3]. The KullbackLeibler (KL) distance is used to measure the similarity of the
distribution of real value and the distribution of reported data.
Measurements on the datasets [14] show that there exists a
logarithmic relationship between the loss of accuracy and thesample rate in terms of KL distance, formally given by

We aim to design an AAL system which can fulfill thediverse needs of the elderly. For this purpose, various sensors
and actuators are deployed in the AAL environment, as shownin Figure 1. Based on this infrastructure, different applications
can provide personalized services to the elderly, meeting their
specific needs comprehensively.
The typical AAL system consists of three parts: data sensing
and transmission, data processing and mining, control andfeedback. There exists multiple data sources in the AAL. Eachsensor samples the elderly or the environment periodicallywith a certain rate. The loss of accuracy can be measured by
the Kullback-Leiber (KL) distance between the distance ofreported data and real value. Generally speaking, the accuracy
A(ri (t)) = − log(a1 r) + a0 ,(1)of the sampled data are proportional to the sample rate. We
employ cloud to process the sensed data because of the limited where A(ri (t)) denotes the loss of accuracy of sensor i at ratecomputation capacity of the on-chip processor. Moreover, the ri (t) in terms of KL distance. It follows the sensor dependent
cloud works as a central entity to fuse all the sensor data. distribution, which can be measured when the system has been
Due to monetary constraints, we allocate limited computing set up.
We aim to set the rate of sensors to meet users requirement
capacity for processing the incoming data. Therefore, someofaccuracy. In general, there is no closed form expressions
data are stored in a queue if the data arrival rate exceeds theabout
the relationship of the rate and loss of accuracy althoughdeparture rate. Each sensor maintains one queue. After theitisexpected
that the result tends to be more accurate if thedata are mined in the cloud, the results are sent back to mobiledataistransmitted
at a higher rate (with better quality) [15].
devices or actuators.
In
thispaper,
weassumethat the loss of final output accuracy
In this system, the main factors that impact user experience
A(r
(t))
to
be
a
decreasing
function of ri (t), which is similari
are accuracy, monetary cost and delay of the services. Gento(1).
erally speaking, accuracy will improve when the sample and2) Service and Data Incoming Types: Services in an AAL
transmission rate increases. However, the increased rate will
environment
can be divided into three types of services based
incur more monetary cost. How to strike a good balance is
on
its
user
pattern.
As shown in Figure 2, we categorize all
our main concern. Moreover, we investigate the relationshiptheservices
intothreetypes:
between the queueing delay and monetary costs. We jointly
analyze these factors using a Lyapunov framework. Two knobs• Permanent Service. These type of services operate continuously. Therefore we call them permanent services.
are provided for users to tune this AAL system to achieve the© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

49

𝑁
Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

𝑖=1
a. Accuracy Requirement
Loss of Accuracy(𝑟𝑖,min (𝑡), 𝒜min (𝑟𝑖,min (𝑡)))Scheduled

Spot

𝑟1

𝑄1 (𝑡)

𝑟2
Sample Rateb. Data Incoming Patterns
Permanentc. Queueing and Delay𝑟3

𝑄2 (𝑡)

𝑟5
𝑟6

𝑟𝑁

𝑤2

𝑤4 APP 4

∝

𝑤𝑖

× 𝛾𝑖

≤

𝑆

𝑖=1
𝑁

× 𝛾𝑖

𝑖=1

𝑤5 APP 5

𝑄5 (𝑡)𝑄𝑁 (𝑡)

𝑁

𝑀 𝒘 𝑡

𝑤𝑖

𝑄4 (𝑡)……

APP 2

𝑤3 APP 3

𝑄3 (𝑡)

𝑟4

𝑤1 APP 1

d. Cloud Monetary Costand Resource Constraint

𝑄6 (𝑡)

𝑤6 APP 6

𝑤𝑁 APP N

𝛾𝑖 Each APP is with processing density 𝛾𝑖Figure 2. Models in the cloud based AAL system. a, Illustration of the relationship between the loss of accuracy and sample rate. We assume a convex of
concave function in this paper. b, Three typical types of services exist in AAL systems. (i) Permanent service. The type of services should operate throughout
the day. (ii) Scheduled service. Some services have strong daily pattern and highly related to user’s biological clock. (iii) Spot service. Some services are
driven by users’ behaiviours, which is unpredictable. The duration of these services tend to be small compared to other services. c, The data transmitted by
sensor are stored in queues for the cloud to process and analyze. Each sensor maintains one queue. We can use r(t) = r1 (t), r2 (t), . . . , rN (t) to quantify the
input/arrival rate to the job queue and use the processing rate w(t) = w1 (t), w2 (t), . . . , wi (t) as the departure rate of the queue. Therefore, we introduce theγi to denote the processing density of sensor i. Let Qi (t) be the job queue length, which indicates the queueing delay. It evolves over time. d, We assume thereserved pricing model [13] since it fits the need of ambient assisted living environment. Users reserve a certain computation resource S and pay as they go.

For example, the surveillance camera should be running density means more CPU cycles required for every bit ofcontinuously in most cases.
sensed data. Table I summarizes the processing densities of• Scheduled Service. Some services have strong daily patternvarious tasks which are used in several studies [16–19]. At theand highly related to user’s biological clock. For example, tth slot, the required CPU cycles for sensor i per unit time cansleep related monitoring tend to be active only in the night be calculated as si (t) = wi (t) · γi .
time. These activities can be scheduled at a daily pattern.
We assume the reserved pricing model [13] since it fits
• Spot Service. Some services are driven by users’ be- the need of the AAL environment. Users reserve a certain
haviours, which is unpredictable. We call it spot services. computation resource S and pay as they go. Different sensors
For example, monitoring of skeletal movement when the and applications require si CPU cycles. Therefore, the total
user is doing rehabilitation exercises. The durations of CPU cycles neededPare limited by the reserved resources,
N
these services tend to be small compared to other services. formally given by i=1 si ≤ S. The total cost consists ofSince the spot services are unpredictable, we need to develop one-time payment, p0 and usage payment p for each CPU
an algorithm to dynamically select the rates and cloud resource cycle per unit time. We only consider the usage monetary costM (w(t)), which can be expressed asprovision to meet the user’s requirements.
3) Queue Dynamics: The data transmitted by a sensor
NNX
X
are stored in its individual queue for the cloud to process
M (w(t)) =si (t) · p =wi (t) · γi · p .
(3)
and analyze. Each sensor maintains one queue. We can use
i=1
i=1
r(t) = r1 (t), r2 (t), . . . , rN (t) to quantify the input/arrival C. Problem Formulation
rate to the job queue and use the processing rate w(t) =Our objective for a AAL system in Figure 1 is to dew1 (t), w2 (t), . . . , wN (t) as the departure rate of the queue. Let
velop a control algorithm. We make the control decision
Qi (t) be the job queue length. It evolves over time following
Θ(t) = (r(t), w(t)) at the beginning of each time slot
the dynamics specified byt. Specifically, the sensor i will sample and transmit atQi (t + 1) = max[Qi (t) − wi (t), 0] + ri (t) ,(2) ri (t) according to user’s accuracy demand. Then, the cloud
will process the transmitted data. Let the cloud processassuming that the initial queue is empty, i.e., Qi (0) = 0.
ing rate for queue i be wi (t). Therefore the control de4) Cloud Computing and Monetary Cost: The cloud process cision of the whole system is Θ(t) = (r(t), w(t)) =the data in each queue with the rate wi (t). The CPU cycles for (r1 (t), r2 (t), . . . , rN (t), w1 (t), w2 (t), . . . , wN (t)).
different data and applications are different. For example, facial
The accuracy-monetary cost can be expressed asrecognition would be more computation-intensive comparedN
X
with temperature monitoring. Therefore, we introduce γi tog(t) =A(ri (t)) + βM (w(t)) ,(4)denote the processing density of sensor i. Higher processing
i=1© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

50

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

where β ≥ 0 is referred to as the accuracy-cost parameter
indicating the relative importance of the monetary cost of thecloud. The long-term average cost can therefore be written as
t−1

1X
E[g(τ )] ,t→inf tτ =0ḡ , lim(5)

B. Derive an upper boundIt can be shown following the Lyapunov optimization
technique [20] that
Theorem 1 (Bound of Lyapunov Drift): Consider the quadratic
Lyapunov function (9), and assume E{L(Q(0))} < ∞. Then:N
X∆(Q(t)) ≤ B + Q(t)E[ri (t) − w(t)|Q(t)] ,(11)

where the expectation is taken with respect to therandom enviPt−1
i=1ronment. Similarly, we define r̄i , limt→inf 1t τ =0 E[ri (τ )],Pt−11w̄ , limt→inf t τ =0 E[w(τ )]. To minimize the long-term where B is a finite constant satisfying
 Naccuracy-monetary cost, we aim at developing a control policy

1 X
B≥ E
ri (t)2 + w2 (t)|Q(t)that solves the following problem
2
i=1
t−1
X
 X

N
1

min
ḡ , lim
E[g(τ )]
(6)−Eri (t) · min[Q(t), w(t)]|Q(t) ,t→inf tr(t),w(t),t=0,1,2,...
τ

s.t. r̄i ≤ w̄i ,NXsi ≤ S ,

(7)
(8)

i=1

(12)

i=1

for any t = 0, 1, . . . .
Proof: We first use (9) to derive a bound of the slot-to-slotchange in the Lyapunov function:

2
1X
where constraint (7) ensures the mean-rate queue stability in the
L(Q(t + 1)) − L(Q(t)) =[Qi (t + 1)2 − Qi (t)2 ]2 i=1long term, and constraint (8) limited the whole computational
resources is under the reserved value.
N
1X
There exists a stationary and randomized offline control pol=[(max[Qi (t) − wi (t), 0] + ri (t))2 − Qi (t)2 ]2
icy solving (6)-(8) based on Caratheodory’s theorem. However,i=1
N2
it requires full knowledge of the queue arrival statistics andX[ri (t)2 + wi (t)2 ] X
is rather challenging, if not possible, to obtain the optimal≤+
Qi (t)[ri (t) − wi (t)] , (13)2control policy. Hence, we develop a practical and onlinei=1
i=1
algorithm which optimize the objective slot by slot to achieve where in the final inequality we have used the fact that for any
the asymptotic optimality.
Q ≥ 0, w ≥ 0, r ≥ 0 , we have:(max[Q − w, 0] + r)2 ≤ Q2 + r2 + w2 + 2Q(r − w) .

III. DYNAMIC R ATE S ELECTION A LGORITHM
We adopt Lyapunov drift plus penalty technique [20–22]
to design the algorithm for sensor rate and processing rate
selection. Unlike the stationary offline policy, this algorithm
only needs the current queue lengths and throughput. Sinceuser patterns (workload arrivals) are not predictable, this fits
the use case well.

Now, plug in (10) and (13), we obtain:NX
ri (t)2 + wi (t)2∆(Q(t)) ≤ E[
|Q(t)]
2
i=1
NX
ri (t) − w(t)|Q(t)] .
+ Q(t)E[

(14)

i=1

A. Make Slot-by-slot Objective
We aim to minimize the loss of accuracy and monetary cost
as in (6) while stabilizing total task queues in (7). First, tomeasure the “size” of the vector Q(t), we define a quadratic
Lyapunov function L(Q(t)) as follows.
N

L(Q(t)) ,1X
Qn (t)2 .
2 n=1

(9)

i=1

This function can fairly stabilize all the queues, i.e., the queue
lengths should be analogously maintained. Then we define theone-slot conditional Lyapunov drift ∆(Q(t)) as follows.
∆(Q(t)) , E{L(Q(t + 1)) − L(Q(t))|Q(t)} .

Then, define B as a finite constant that bounds the first termon the right-hand-side of the above drift inequality. We cansee that
 N

1 X
B≥ E
ri (t)2 + w2 (t)|Q(t)
2
i=1
 X

N

−E
ri (t) · min[Q(t), w(t)]|Q(t) .

(10)

By adding V E[g(t)|Q(t)] on both side of (11), we obtain
the drift plus monetary cost upper bound as follows.
∆(Q(t)) + V E[g(t)|Q(t)] ≤B + V E[g(t)|Q(t)]We aim to minimize the Lyapunov drift ∆(Q(t)). However, it
requires knowledge of prior statistics. Instead, we derive andminimize its upper bound.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

NX+ Q(t)E[ri (t) − w(t)|Q(t)] ,
i=1

(15)

51

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

where V > 0 is a parameter indicates how close the long term
average achieved is to the optimal offline algorithm.
Remark 1: V is a cost-delay tradeoff parameter. Intuitively,
we cannot obtain both the minimal queueing length and theminimal cost. If more data are processed in the cloud, the queue
length will be shortened, which can meet the critical deadlineof delay sensitive applications. But the cloud has to provide
more computation resource which incurs more monetary cost.
Parameter V is introduced to indicate the importance of thecost.
C. Design the Algorithm
We design an online algorithm based on the upper bound (15)of Lyapunov drift plus monetary cost. The algorithm is formallypresented in Algorithm 1. Our algorithm only require the
Algorithm 1 Online Algorithm for Accuracy Monetary Tradeoff Operation
1: At the beginning of every time slot t, observe the queue
lengths Q(t) and the actual incoming packets.
2: The sensor chooses transmission rate ri (t) for the following
optimization problem.
V · A(ri (t)) + Qi (t) · ri (t)minri (t),t=0,1,2,...

(16)

ri (t) ∈ [0, ri,max ] .

s.t.

Table IP ROCESSING DENSITIES OF VARIOUS APPLICATIONS FOR THE
CLOUD - BASED AAL SYSTEMS .
Application
Face recognition [16], [17]Anomaly Detection [19]Video transcoding [18]Table IID EFAULT PARAMETER VALUES
Parameter
N
β
T
p
a0

value
10
1
200
1
15

min

Vβ

t→∞

w(t),t=0,1,2,...

M (wi (t)) −

i=1

N
X

Qi (t) · wi (t)

i=1

(17)
s.t.

N
X

w̄i ≤ S

i=1

wi ≥ 0 .
4: Update the job queue Q(t) as (2).

knowledge of current queueing states and workload arrivals. If
does not require any statistics about future states or workload
arrivals.

value
0.01
i2
1000000
100
100

t−1

lim supN
X

Parameter
V
γi
S
a1
rmax

Proof: Please see Appendix A.
Remark 2: This theorem shows that we can achieve near
optimal of the long term average accuracy-monetary cost by
setting V to a large value.
Theorem 3 (Queue Stability): All queues Qi (t) are mean
rate stable.
Proof: Please see Appendix B.
Theorem 4 (The Bound of Average Job Queue):

3: The cloud chooses processing rate wi (t) for the following

optimization problem.

Processing density (cycles/bit)
31680 [16], 2339 [17]240
200 - 1200

B+V ·D
1X
E[Q(τ )] ≤,t τ =0(19)

where D =Pg() − ḡ opt is an average accuracy-monetary costNsuch thati=1 r̄i ≤ w̄ −  for some  > 0 (i.e., Slater’s
condition).
Proof: The proof follows the main results in the proofsof Theorem 2 and Theorem 3. For brevity, we omit the details
here. We encourage the interested reader refer to [20].
The design of Algorithm 1 follows our derived analyticalresults. All the previous theorems can hold for this algorithm.
Numerical results further showed the effectiveness of our
proposed algorithm.
IV. P ERFORMANCE E VALUATION
A. Simulation Setup

Real accuracy measurement. To obtain realistic functionsofthe accuracy and rate in Section II, we measure the accuracy
Here we prove that we can develop a processing rate selection
parameters
for several types of sensors [14].
algorithm to achieve near optimal performance in terms of
Datasets(workloads and processing density). We create
average accuracy-monetary cost by appropriately tuning thetheworkload/traffic
arrivals by generating Bernoulli arrivals.
parameter V . The following theorems highlight our key finds.
Moreover,theprocessing
densities of various tasks are sumSpecifically, Theorem 2 shows the effectiveness of minimizing
marized
in
Table
I,
which
are
used in several studies [16–19].
the objective in (6). Theorem 3 further ensures the queue
Parameters.
The
default
parameters
are summarized instability in (7).
Table
II.
Theorem 2 (Bound of Long-term Average Accuracy-monetaryCost): The long-term average accuracy-monetary cost is B. Simulation Resultsbounded as follows.
In this section, we will show the effectiveness of the proposedt−1
1XBalgorithms and demonstrate the impact of various systemoptḡ = lim
E[g(τ )] ≤ ḡ +
(18)
t→∞ t
parameters. Please note the “average” number at time t inV
τ =0
D. Theoretical Analysis© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

52

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

the following simulations is calculated by summing up all the Then, by taking the expectations of (A4), summing up frompast values (include time t), then divided by t + 1.
0, 1, . . . , t and by dividing the sum by V , we obtain, for any1) Cost-delay Tradeoff Parameter V : As shown in Figure 3, t > 0,
parameter V can effectively strike the trade-off between delay
t−1
X
B
E[L(Q(0))] E[L(Q(t + 1))]and accuracy plus monetary cost functions. During this set 1E[g(t)] ≤ ḡ opt + +−t τ =0V
Vt
Vt
of simulation, we set β · V = 0.000001. The figures showthat as V increases, both the loss of accuracy and monetary
E[L(Q(0))]
B≤ ḡ opt + +.
(A5)
cost decrease, while the queue length increases, which means
VVt
average queueing delay increases.
Thus, Theorem 1 can be proved by taking t → ∞.
2) Accuracy-Monetary Cost Tradeoff Parameter β: The
results in Figure 4 show that β can provide a flexible tradeA PPENDIX B: P ROOF OF T HEOREM 1
off between the loss of accuracy and monetary cost. When β
By taking expectations on (A4) and summing up fromincreases, the monetary cost drops while the loss of accuracy 0, 1, . . . , t, we obtainwill increase. The observations confirm our objective function
E[L(Q(t+1))] ≤ E[L(Q(0))]+[B+V (ḡ opt −gmin )]·t , (B1)with β to indicate the relative importance of monetary cost.
3) Asymptotic optimality of the proposed algorithm:
where gmin ≤ E[g(t)]. Next, by the definition of the Lyapunov
Simulation results show that our algorithm can achieve nearfunction, we have
optimal average accuracy-monetary cost at the cost of larger
queue length.
E[Zi2 (t)] ≤ 2E[L(Q(0))] + [B + V (ḡ opt − gmin )] · t . (B2)
V. C ONCLUSION
We studied the three most cared metrics in AAL systems anddeveloped an algorithm to help users to achieve the balanceamong accuracy, monetary cost and delay. Specifically, user
can tune knob parameters V and β to operate the AAL system
in a desirable way. Moreover, since the proposed algorithm
only considers the current queue state and workload arrivals,
it can coordinate the system well in time varying environment,
which is common in practical AAL systems. The effectivenessof the proposed algorithm is proved by rigorous analysis andsimulation results.
A PPENDIX A: P ROOF OF T HEOREM 1
In Algorithm 1, we minimize the right hand side of theupper bound on the Lyapunov drift plus the accuracy-monetary
cost. Thus, for every time slot t, we have∆(Q(t)) + V E[g(t)|Q(t)]NX≤ B + V E[g ∗ (t)|Q(t)] + Q(t)E[ri∗ (t) − s∗ (t)|Q(t)] ,
i=1

(A1)
where ri∗ (t) and w∗ (t) are the arrival rate to the job queue andprocessing resource provisioning under any alternative control
policies, respectively. g ∗ (t) is the resulting cost. There exista stationary and randomized offline control policy Z, which
can minimize the accuracy-monetary cost. The following holds
under policy Z,
E[g ∗ (t)|Q(t)] = E[g ∗ (t)] = ḡ opt ,NXE[ri∗ (t) − w∗ (t)|Q(t)] ≤ 0.

(A2)
(A3)

i=1

Next, by plugging (A1)-(A3), we can get∆(Q(t)) + V E[g(t)|Q(t)] ≤ B + V ḡ opt .

(A4)

Because the variance of |Zi (t)| is always non-negative, wehave E[Zi2 (t)] − E[|Zi (t)|]2 ≥ 0. Hence, for any t > 0. By
dividing both sides by t and taking t → ∞, we prove
E[Zi (t)]≤ 0,twhich shows the mean-rate stability.
lim

(B3)

t→∞

ACKNOWLEDGMENT
This work is partly supported by the National Research
Foundation, Prime Ministers Office, Singapore under its
IDM Futures Funding Initiative, Singapore EIRP02 (Grant
NRF2012EWT-EIRP002-013).
R EFERENCES[1] D. o. E. Population Division and S. Affairs, “World population
aging 2015,” United Nations, Tech. Rep., 2015.
[2] P. Rashidi and A. Mihailidis, “A survey on ambient-assisted
living tools for older adults,” IEEE journal of biomedical andhealth informatics, vol. 17, no. 3, pp. 579–590, 2013.
[3] H. Wen, Z. Xiao, A. Markham, and N. Trigoni, “Accuracy estimation for sensor systems,” Mobile Computing, IEEE Transactions
on, vol. 14, no. 7, pp. 1330–1343, 2015.
[4] A. Thiagarajan, L. Ravindranath, K. LaCurts, S. Madden,
H. Balakrishnan, S. Toledo, and J. Eriksson, “Vtrack: accurate,energy-aware road traffic delay estimation using mobile phones,”
in Proceedings of the 7th ACM Conference on Embedded
Networked Sensor Systems. ACM, 2009, pp. 85–98.
[5] H. Wen, Z. Xiao, N. Trigoni, and P. Blunsom, “On assessing
the accuracy of positioning systems in indoor environments,” inEuropean Conference on Wireless Sensor Networks. Springer,
2013, pp. 1–17.
[6] D. Wang, T. Abdelzaher, L. Kaplan, and C. C. Aggarwal, “Recursive fact-finding: A streaming approach to truth estimation incrowdsourcing applications,” in Distributed Computing Systems(ICDCS), 2013 IEEE 33rd International Conference on. IEEE,
2013, pp. 530–539.
[7] D. Wu, Y. Cai, and M. Guizani, “Asynchronous flow scheduling for green ambient assisted living communications,” IEEE
Communications Magazine, vol. 53, no. 1, pp. 64–70, January2015.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

53

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

100
500

V = 0.01
V = 0.03
V = 0.05

4.5

#10 7

4
95

450

400

90

3

Monetary Cost

Average Loss of Accuracy

Average Queue Length

3.5

85

80

2.5
2
1.5

V = 0.01
V = 0.03
V = 0.05

75

1

V = 0.01
V =0.03
V = 0.05

0.5
350

70
0

20

40

60

80

100

120

140

160

180

0
0

200

20

40

60

80

100

120

140

160

180

200

0

20

40

60

80

Time Slot

Time Slot(a) Average Queue Length

100

120

140

160

180

200

Time Slot(b) Average Loss of Accuracy(c) Average Monetary Cost

Figure 3. Impact of V . We set β · V = 0.000001 in this set of simulations.

3

β = 0.00002
β = 0.00004
β = 0.00006

500
490

β = 0.00006
β = 0.00004
β = 0.00002

2.8
95

2.6

480
470
460
450
440

2.4

Monetary Cost

Loss of Accuracy

Average Queue Length

×10 7

100

510

90

85

80

420

2
1.8
1.6

β = 0.00002
β = 0.00004
β = 0.00006

430

2.2

1.4
1.2

410

1
75

400
0

20

40

60

80

100

120

140

160

Time Slot(a) Average Queue Length

180

200

0

20

40

60

80

100

120

140

160

180

200

0

20

Time Slot(b) Average Loss of Accuracy

40

60

80

100

120

140

160

180

200

Time Slot(c) Average Monetary Cost

Figure 4. Impact of β. We set β · V = 0.000001 in this set of simulations.

[8] Y. Dong, L. Zhou, J. Chen, B. Zheng, and J. Cui, “Energy
efficient virtual machine consolidation in mobile media cloud,”in Picture Coding Symposium (PCS), 2015, May 2015, pp. 248–
252.
[9] L. Zhou, “On data-driven delay estimation for media cloud,”
IEEE Transactions on Multimedia, vol. 18, no. 5, pp. 905–915,
May 2016.
[10] L. Zhou and H. Wang, “Toward blind scheduling in mobile
media cloud: Fairness, simplicity, and asymptotic optimality,”IEEE Transactions on Multimedia, vol. 15, no. 4, pp. 735–746,
June 2013.
[11] Y. Jin, Y. Wen, H. Hu, and M.-J. Montpetit, “Reducing
operational costs in cloud social tv: an opportunity for cloud
cloning,” IEEE Transactions on Multimedia, vol. 16, no. 6, pp.
1739–1751, 2014.
[12] Y. Dong, L. Zhou, Y. Jin, and Y. Wen, “Improving
energy efficiency for mobile media cloud via virtual
machine consolidation,” Mobile Networks and Applications,vol. 20, no. 3, pp. 370–379, 2015. [Online]. Available:
http://dx.doi.org/10.1007/s11036-015-0595-2[13] J. He, Y. Wen, J. Huang, and D. Wu, “On the cost–qoe tradeoff
for cloud-based video streaming under amazon ec2’s pricing
models,” Circuits and Systems for Video Technology, IEEE
Transactions on, vol. 24, no. 4, pp. 669–680, 2014.
[14] P. Bodik, W. Hong, C. Guestrin, S. Madden, M. Paskin, and
R. Thibaux, “Intel lab data,” Online dataset, 2004.
[15] X. Zhu and X. Wu, “Class noise vs. attribute noise: A quantitative
study,” Artificial Intelligence Review, vol. 22, no. 3, pp. 177–210.
[16] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman,

S. Saroiu, R. Chandra, and P. Bahl, “Maui: making smartphones
last longer with code offload,” in Proceedings of the 8thinternational conference on Mobile systems, applications, andservices. ACM, 2010, pp. 49–62.
[17] S. Kosta, A. Aucinas, P. Hui, R. Mortier, and X. Zhang,“Thinkair: Dynamic resource allocation and parallel execution
in the cloud for mobile code offloading,” in INFOCOM, 2012
Proceedings IEEE. IEEE, 2012, pp. 945–953.
[18] J. Kwak, O. Choi, S. Chong, and P. Mohapatra, “Processornetwork speed scaling for energy-delay tradeoff in smartphone
applications,” IEEE/ACM Transactions on Networking, vol. PP,
no. 99, pp. 1–14, 2015.
[19] K. H. Lee and N. Verma, “A low-power processor withconfigurable embedded machine-learning accelerators for highorder and adaptive analysis of medical-sensor signals,” SolidState Circuits, IEEE Journal of, vol. 48, no. 7, pp. 1625–1637,
2013.
[20] M. J. Neely, “Stochastic network optimization with application
to communication and queueing systems,” Synthesis Lectures
on Communication Networks, vol. 3, no. 1, pp. 1–211, 2010.
[21] H. Hu, Y. Wen, T.-S. Chua, J. Huang, W. Zhu, and X. Li,“Joint content replication and request routing for social video
distribution over cloud cdn: A community clustering method,”
Circuits and Systems for Video Technology, IEEE Transactions
on, 2016, in press.
[22] H. Hu, Y. Wen, T.-S. Chua, Z. Wang, J. Huang, W. Zhu, andD. Wu, “Community based effective social video contents placement in cloud centric cdn network,” in 2014 IEEE International
Conference on Multimedia and Expo (ICME). IEEE, 2014, pp.
1–6.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

54

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Distributed ACO Based on a Crowdingsourcing
Model for Multiobjective Problem
Jiaqi Lu

Li Pan*, Shijun Liu

School of Software Engineering
Shandong University
Jinan, China
453429978@qq.com

School of Computer Science and Technology
Shandong University
Jinan, China{panli, lsj}@sdu.edu.cn

Abstract—MOP(Multiobjective Optimization Problem) is aprevailing research field for its well-modeling on the decisionmaking dilemma in the real world. We present a distributed
ACO(Ant Colony Optimization) algorithm, as well as acompatible model, which enables to solve a continuous MOP on
network. The original MOP is expected to be decomposed into
multiple single-objective subtasks, which are then distributed tothe individuals on the network, and the non-dominated front for
the MOP is constructed by aggregating the solutions from thecrowd. Besides, a few strategies are suggested, in order to enable
the compatibility of ACO and continuous MOP. Finally, an
experiment on the KUR test problem illustrates that ourapproach is practical.

As for our problems, if the scale of MOP becomes larger, it
is a tough and time-costing job for solely one machine to tackle
with. The devisal of MOEA/D inspires us to deploy a job of
MOP on a network, where plenty of machines are working
simultaneously solving the subproblems divided from the MOP,
and thus solve the problem cooperatively.

Keywords—ACO; crowdsourcing model; MOEA;continuous
MOP.

I. INTRODUCTION
In decision-making, different from the ideal circumstance
where only one objective needs to be considered about,normally, there are several conflicting objectives involved.
Such cases can be modeled as MOPs(multiobjective
optimization problems). A Pareto front is a collection ofsolutions to a MOP, for which no single solution is better thanother solutions. For each pair of solutions on Pareto front, onemust be better than the other in at least one objective.
There exist numerous classic algorithms which can provide
good results for MOPs. One important research stream among
them is MOEAs (multiobjective evolutionary algorithms).
Previous work has proved the potential of MOEAs in tacklingwith MOPs [1-7]. Except for the above MOEAs based on
genetic algorithm, mACOs (multiobjective ant colony
optimization algorithms) also have been developed in recent
years [8-12].
In 2007, Q.Zhang, and H.Li proposed MOEA/D(multiobjective evolutionary algorithm based on decomposition)[13]. The proposition of MOEA/D can be easily associated withcrowdsourcing. Crowdsourcing is a promising model thatenables one to rely on the power of a crowd to get jobs done, bymeans of distributing the task to individuals in the crowd [14].
The idea of crowdsourcing has been widely applied in business,
medicine, sociology and other domains. A well-known
example is Wikipedia, where various netizens(Internet citizens)contribute their knowledge and efforts.

In this paper, borrowing the idea of crowdsourcing, we
propose a distributed and parallel ACO(Ant Colony
Optimization) algorithm for solving MOPs. The basic idea of
our algorithm is to decompose the original job of amultiobjective optimization problem into subtasks and assign
each task, which is used to solve a subproblem, to individualmachine on the network. The most significant contribution of
our work lies in that we introduce a compatible model, whichenables one to utilize our algorithm with the help of a crowd on
the network. In addition, there are several innovative strategies
involved during the execution, furthering the existing researchon combination of MOEA/D and ACO [15].
The rest of paper is organized as follows. Section 2
introduces the definition for MOPs. Section 3 presents ouralgorithm in detail, including the decomposition approach, thesubalgorithm for each decomposed subproblem, ourPersonalScope strategy, and finally proposes the framework of
the algorithm. Section 4 explicates how we deploy ouralgorithm on the network, in order to employ the power of thecrowd. Section 5 elaborates the details during the execution of
our algorithm and shows the experimental result.
II. PROBLEM DESCRIPTION
A. MOPA MOP can be defined as：
minimize F(x) = �f1 (x), … , f𝑚𝑚 (x)� (1)subject to x ∈ Ω
where Ω is the decision (variable) space, F = Ω → Rm
consists of m real-valued objective functions, and Rm is called
the objective space. The attainable objective set is defined as set{F(x)|x ∈ Ω}.

For each pair of u, v ∈ R𝑚𝑚 , u dominates v if and only if
𝑢𝑢𝑖𝑖 ≤ 𝑣𝑣𝑖𝑖 for every i ∈ {1, … , 𝑚𝑚} and 𝑢𝑢𝑖𝑖 < 𝑣𝑣𝑖𝑖 for at least one
objective j ∈ {1, … , 𝑚𝑚}. Point x ∈ Ω is a Pareto optimal if it isnon-dominated, and F(x) is called a Pareto-optimal vector. The© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

55

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)set of all the Pareto optimal is called the Pareto Set (PS), andthe set of all the Pareto-optimal vector is called the Pareto Front(PF)[16].
B. Continuous MOP versus Discrete MOP
MOPs can be generally categorized into two categories:
Continuous MOPs and Discrete MOPs. Continuous MOPs arecharacterized by continuous objective functions as well as
continuous domains, precisely, Ω = Rn or Ω is a continuous
subspace of Rn , while it is contrary to discrete MOPs .
A typical discrete MOP is the MOKP(Multi-Objective
Knapsack Problem)[8], which can be stated as:𝑛𝑛maximize f𝑖𝑖 (x) = � p𝑖𝑖,𝑗𝑗 x𝑗𝑗 ,
𝑗𝑗 =1

𝑛𝑛

𝑖𝑖 = 1, … , 𝑚𝑚

subject to � w𝑖𝑖,𝑗𝑗 x𝑗𝑗 ≤ c𝑖𝑖 , 𝑖𝑖 = 1, … , 𝑚𝑚
𝑗𝑗 =1

where p𝑖𝑖,𝑗𝑗 ≥ 0 is the profit of item j in knapsack i, w𝑖𝑖,𝑗𝑗 ≥ 0
is the weight of item j in knapsack i, and c𝑖𝑖 is the capacity ofknapsack i. 𝑥𝑥𝑖𝑖 = 1 means that item i is selected and put into all
knapsacks.
Apparently, with the constraints, Ω is no longer acontinuous space and neither the objective functions arecontinuous.
Normally, ACO is usually more suitable to be applied on
discrete problems due to its inherent characteristics. However,with a few alternations and adaptations, it is expected thatACO can perform well on continuous problems, either. Thus,
instead of dealing with MOKP or TSP problems, here we
employ KUR, which is a continuous MOP, as the testing
problem.
Our technique of transforming a continuous problem into a
discrete one is quite straightforward. By setting a given stride
length ε , we decompose the continuous domain into acollection of scalar points serving as the new domain, precisely,
𝑈𝑈−𝐿𝐿ℂ = {a1 , … , a𝑘𝑘 }, 𝑘𝑘 =+1.
εa𝑖𝑖 = 𝐿𝐿 + ε ∗ (𝑖𝑖 − 1) 𝑖𝑖 = 1, … , 𝑘𝑘
where U and L are the upper bound and lower bound of theprescribed variable bounds for the original problem.
Thus, our new problem space is as Ω = {(𝑥𝑥1 , … , 𝑥𝑥𝑛𝑛 )| 𝑥𝑥𝑖𝑖 ∈ℂ, 𝑖𝑖 = 1, … , n}.
III. ALGORITHM

A. Definitions and Annotations
Here are the definitions and annotations for the symbols
involved in the following contents:
N:The number of the subproblems considered.
m:The number of objectives.
n:The dimension of feasible solutions.
∗ T) , where z𝑘𝑘∗ is thez ∗ :The reference point, z ∗ = (z1∗ , … , z𝑚𝑚∗{f𝑘𝑘 (x) , x ∈ Ω}.
optimal value to objective k, i.e. z𝑘𝑘 = max⁡
ES:An external set, containing the non-dominated solutions
found.
T:The size of neighborhood.

B 𝑖𝑖 :The neighboring subproblems for subproblem i, B 𝑖𝑖 =(B1𝑖𝑖 , … , BT𝑖𝑖 )T ,B𝑗𝑗𝑖𝑖 is the global ID for the neighbor j.
x𝑖𝑖# :The recorded best solution for subproblem i.
λ:The collection of weight vectors. λ𝑖𝑖 = {λ1 , … , λN }. λ𝑖𝑖 is theweight vector for subproblem i, where λ𝑖𝑖 = (λ1𝑖𝑖 , … , λ𝑖𝑖𝑚𝑚 )T .
P:The number of ants in a colony.
H:A parameter controlling N and the generated weight
B. Decomposition and the Generation of Weight Vectors
There are various techniques for the decomposition of
MOP. Here we adopt a Tchebycheff Approach [16].
In this approach, the subproblem i, ℱ𝑖𝑖 (x) is in the form:
minimize 𝑔𝑔𝑡𝑡𝑡𝑡 (x|λ𝑖𝑖 , z ∗ ) = max {λ𝑖𝑖 (f𝑖𝑖 (x) − z𝑖𝑖∗ )} (2)1≤𝑖𝑖≤𝑚𝑚subject to x ∈ ΩFor each Pareto optimal solution, there exists a weightvector λ such that x is the optimal solution of (2) and each
optimal solution of (2) is a Pareto optimal solution of (1). Thus,by means of solving plenty of different subproblems andobtaining the optimal solution respectively, the Pareto Front
for MOP is constructed.
And as the definition of subproblem relies on weight vector,
a set of uniformly spread weight vectors is desired. A
parameter H is needed for the generation of weight vector. λincludes all possible permutations whose m elements are
0 1
H
selected from { , , … , }, with the constraint that the sum
H H
Hmust equal to 1.
Therefore, the size of λ, also as N, is:m−1
𝑁𝑁 = 𝐶𝐶H+m−1

C. PersonalScope Strategy
A defect of our alternation is that for continuous problem,the domain considered is massively wide even after
decomposing it into a finite set, and thus the construction of asolution would be time-costing compared with tackling with a
traditional discrete problem.
To limit the choice for each ant and stimulate the algorithm,
a PersonalScope strategy is adopted.
When an ant is initialized, a PersonalScope, which is asubset of the domain Ω , is randomly generated. Each ant
makes its choice within its PersonalScope respectively. Then
all the constructed solutions would be aggregated and sorted.
Each ant would be informed of its rank, which determines its
PersonalScope state then.
The size of the PersonalScope depends on a ratio ω. For
each dimension, the original domain is ℂ , a collection of
discrete scalar number. Hence the size of PersonalScope is S=ω ∗ |ℂ|.
With the concept of PersonalScope, the Ω for each ant istransformed into
Ω = {(x1 , … , x𝑛𝑛 )| x𝑖𝑖 ∈ ℂ𝑖𝑖 , ℂ𝑖𝑖 = �a1𝑖𝑖 , … , a𝑆𝑆𝑖𝑖 � , 𝑖𝑖 = 1, … , 𝑛𝑛}
D. Optimization Algorithm: ACO process
Each ant colony is responsible for a subproblem. With the
adopted optimization algorithm, mainly based on ACO, eachcolony searches for its respective optimal.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

56

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Here it is the description of the algorithm:
Algorithm 1 ACO Process

Input:·MOP·A stopping criterion·A pause criterion
·λi
·z*
·B𝑖𝑖
·𝑃𝑃
Output:x𝑖𝑖#
Step 1) Initialization:
1. For each ant t = 1,…,P, generate initial PersonalScope.
2. Initialize the public pheromone structure.
Step 2) Solution Construction: (continuing)
For each ant t = 1,…,P, ant t constructs a new feasible
solution x 𝑡𝑡 by using a probablilistic rule to choose solution
components.
Step 3) Ranking:
Rank the solutions constructed by P ants in ascending order.
Step 4) Update of x𝑖𝑖#
For the highest ranked solution xc , if ℱ(x𝑐𝑐 ) is less thanℱ(x𝑖𝑖# ), x𝑖𝑖# = x𝑐𝑐 .
Step 5) Termination
Cease and output x𝑖𝑖# , if reaching the stopping condition.
Step 6) Update of PersonalScope
Ants contributing a First Class solution can keep using its
PersonalScope; ants contributing a Second Class solution
have to randomly modify its original PersonalScope before
next solution construction; and as for ants contributing a
Third Class solution, a modified PersonalScope from FirstClass ants would be its new PersonalScope. The
classification accords to the rank calculated in Step 3), to beprecise, the top 30 percent would be classified into the FirstClass, while the bottom 30 percent would be categorized into
the Third Class, and the rest belongs to Second Class.
Step 7) Pause
Pause and output x𝑖𝑖# , if reaching the pause condition.
Continue after being invoked.
Step 8) Update of Pheromone Structure (continuing)
For each x𝑖𝑖# belonging to the First Class, for components in
x𝑖𝑖# , update pheromone. Go to Step 1).

E. Strategies Adopted in ACO Process in Detaila) Solution Construction
As we defined in Section II, a solution x ∈ R𝑛𝑛 , precisely,x = (𝑥𝑥1 , … , 𝑥𝑥𝑛𝑛 ). When constructing solution, each ant chooses
𝑥𝑥𝑖𝑖 , 𝑖𝑖 = 1, … , 𝑛𝑛 one at a time. And for each 𝑥𝑥𝑖𝑖 , valid candidates
are those in PersonalScope respectively.
𝑥𝑥1 is decided randomly, for the constraint of ourpheromone structure. The rest of 𝑥𝑥𝑖𝑖 are chosen according to aprobabilistic method. The possibility for choosing aj as x𝑖𝑖 is:p𝑗𝑗 =η𝑗𝑗 α ∙ τ𝑗𝑗 β∀𝑗𝑗 ∈ 1. . 𝑠𝑠, ∀𝑟𝑟 ∈ 1. . 𝑠𝑠
𝑠𝑠
∑𝑟𝑟=1 η𝑟𝑟 α ∙ τ𝑟𝑟 β(3)where η𝑗𝑗 and τ𝑗𝑗 respectively are the heuristic and pheromone
factors of a𝑗𝑗 , and α and β are two parameters that determine
the importance of two factors. Bigger parameter indicates thatthe corresponding factor is more highly valued.
b) Heuristic Factor
As we shown in Formula (3),η𝑗𝑗 is a heuristic factor. It
would be preferable if we could work out the exact ℱ(x) as ourheuristic information, but during the process of constructing,
all we have is an incomplete solution. To solve the dilemma,
we introduce a predictive method.
According to our algorithm, each ant maintains a current
optimal x # . And a predictive solution could be constructedwith the half-finished current solution and current optimal.
𝑝𝑝
𝑝𝑝
𝑝𝑝
For example, when choosing the value of x𝑖𝑖 , {𝑥𝑥1 , … , 𝑥𝑥𝑖𝑖−1 }is determined while {𝑥𝑥𝑖𝑖+1 , … , 𝑥𝑥𝑛𝑛 } is beyond consideration at#, … , 𝑥𝑥𝑛𝑛# } from the maintained
this moment. Thus, we slice {𝑥𝑥𝑖𝑖+1#
current optimal x as the predictive { 𝑥𝑥𝑖𝑖+1 , … , 𝑥𝑥𝑛𝑛 } for ourconstructing solution. Suppose the ant is computing thepossibility for choosing aj as 𝑥𝑥𝑖𝑖 . The predictive solution is𝑝𝑝
𝑝𝑝#
x 𝑝𝑝 = (𝑥𝑥1 , … , 𝑥𝑥𝑖𝑖−1 , 𝑎𝑎𝑗𝑗 , 𝑥𝑥𝑖𝑖+1, … , 𝑥𝑥𝑛𝑛# ) and we can get ℱ(x 𝑝𝑝 ) as
the heuristic factor η𝑗𝑗 for a𝑗𝑗 .
c) Pheromone Factor
There are various available pheromone structure,corresponding to different ways of laying pheromone trails[8].
Here, we adopt the strategy that laying pheromone on each𝑖𝑖
𝑖𝑖+1
couple �a𝑖𝑖𝑝𝑝 , a𝑖𝑖+1∈ ℂ𝑖𝑖+1 𝑖𝑖 = 1 … 𝑛𝑛 −𝑞𝑞 � , where a𝑝𝑝 ∈ ℂ𝑖𝑖 , a𝑞𝑞
1, revealing the idea that the desirability of choosing a𝑖𝑖+1
asq
𝑥𝑥 𝑖𝑖+1 would increase if the prior 𝑥𝑥 𝑖𝑖 is selected as a𝑖𝑖𝑝𝑝 .
In Formula (3), τj is the pheromone factor, representing
the preference for the colony towards this component.
𝑝𝑝
Continuing on our previous example, we have τj = τ�𝑥𝑥𝑖𝑖−1 , a𝑗𝑗 �.
Besides, other strategies are also feasible in our case, forexample, laying pheromone on each candidate for 𝑥𝑥 𝑖𝑖+1 , etc.
d) Phromone Update
First, pheromone trails are reduced in a given ratio ρ as asimulation to the process of evaporation in reality.
Second, for the solution x that we are going to update
pheromone on its components, add pheromone to thecomponents respectively. For each component (𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑖𝑖+1 ) in thesolution x, the quantity of pheromone added is1∆τ(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑖𝑖+1 ) =𝑖𝑖 = 1, … , 𝑛𝑛 − 1ℱ(x) − ℱ(x ∗ )∗where x is the recorded optimal for the specificsubproblem.
Hence, the formula for pheromone update is:τ(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑖𝑖+1 ) = (1 − ρ) ∗ τ(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑖𝑖+1 ) + ∆τ(𝑥𝑥𝑖𝑖 , 𝑥𝑥𝑖𝑖+1 ),𝑖𝑖 = 1, … , 𝑛𝑛 − 1© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

57

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

F. Algorithm Framework
Here we describe our main algorithm:
Algorithm Framework

INPUT:
·MOP
·H
·z*
·λ
·T
OUTPUT: ES
Step 1) Initialization:
1. Set ES = Φ
2. Compute the Euclidean distances between weight vectors
in λ in pair. For each λ𝑖𝑖 , i=1,…,N, add the T closest weightvectors into B𝑖𝑖 ,then pass the required parameters to the ACO
process for subproblem defined with λ𝑖𝑖 .
Step 2) ACO process
Proceed with the ACO process for each subproblem
respectively.
Step 3) Update with Neighboring Solutions:
For each subproblem i, i=1,…,N, for each neighboring
subproblem j in B𝑖𝑖 , if ℱ𝑖𝑖 (x𝑗𝑗 ) is less than ℱ𝑖𝑖 (x𝑖𝑖# ) , x𝑖𝑖# = x𝑗𝑗# .
Step 4) Termination
Cease and output ES, if all the ACO process finished. Else,go to Step 2).
Step 5) Update of ES:
For each subproblem i, i=1,…,N, remove all the x𝑟𝑟 in ES
dominated by x𝑖𝑖 ,and add x𝑖𝑖 into ES if no existing solution in
ES dominates x𝑖𝑖 .
IV. APPLICATION ON NETWORKThe algorithm above is designed based on the following
model, which guarantee our algorithm could well apply on the
network.

A. Process Model
We simplify the model proposed by A.Murturi, B.Kantarci
and S.F.Oktug in 2015[17], and outline the model we adopted
in Figure.1.
B. Structural Model
Figure.2 is the general view of model applied on our
decomposed MOPs.
C. Role Involveda) CrowdSourcer
Crowdsourcer is the one who seeks help from the crowd.
The crowdsourcer defines the task and has the responsibility to
decompose it into subtasks utilizing the task manager.
b) Task Manager
Task manager decomposes the task submitted bycrowdsourcer into multiple subtasks. For our problem, the task
manager would generate adequate weight vectors and find theneighbors for each subtask.
Then, task manager would keep waiting for individuals in
the crowd to request and then designate subtasks.
c) Crowd
Plenty of crowdsourcing providers constitute the crowd.
For each individual in the crowd, it owns a global ID
corresponding to its designated subtask, indicating that it is
responsible for the very subtask. The primary job for eachindividual is to work out the subtask, in the context of ourproblem, by means of running an ACO process.
It is worth noting at this point that individuals in thecrowd are capable of communicating with their neighbors.
d) Evaluatoin Engine
An evaluation engine collects the result from the crowd,
extracts those non-dominated solutions constituting the Pareto
Front for the destined problem, and returns the Pareto Front tocrowdsourcer.
Thread1-1

Crowdsourcer

Task Manager

1.Submit Task

PC1

Thread1-2

Subtasks on Thread1-4
and Thread 2-2 are
the neighbors ofsubtask on Thread2-1

Thread1-3

2.Request for Subtask

Thread1-4

Communicate
Thread2-1

PC2

Thread2-2

Crowd
3. Distribute Subtasks

Thread2-3
Thread2-4

Communicate

5.Reutrn PF
4.Submit Solutions Respectively

Thread3-1

PC3

Thread3-2
Thread3-3
Thread3-4

Evaluation
EngineFig. 1. Process Model© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

Fig. 2. Structural Model

58

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

V. EXPERIMENTS

2

A. Test Problem:KUR
The definition for KUR is:

0-25

Objective functions
n−1

Variable bounds

f1 (x) = �(−10e𝑖𝑖=1nf2 (x) = �(|x𝑖𝑖𝑖𝑖=1(−0.2�x 𝑖𝑖 2 +x 𝑖𝑖+1 2 )

|0.8

3

+ 5 sin x𝑖𝑖 )

)

-20

-15

n

-10

-5

-2 0
-4
-6

[−5,5]-8

3

B. Settings
T=20
P=100
H=149
ω=0.5
Stride Lengthε=0.05
Stopping Criterion: loop for 1000 iteration
Pause Criterion: loop for 200 iteration
We experimented on a local network constituted with 6
computers: one as crowdsourcer and five as a crowd. And
five threads were created for each individual in the crowd.
Specifically, the KUR was decomposed into 150
subtasks. Each individual is responsible for 30 subtasks.
More precisely, each thread takes charge of 6 subtasks.
For each thread, 6 subtasks were scheduled with timeslice strategy thus it is approximately parallel processing.
C. Result
Figure.3 shows the obtained non-dominated solutions
by our approach. The horizontal and vertical axis represent
f1 (x) and f2 (x) for a Pareto optimal x, respectively.

-10
-12
-14
Fig. 3. Result: the Obtained Nondominated Solutions

Total time ：321246ms
D. Analysis and Discussion
The result reveals that our approach is feasible toconstruct an approximation of PF for KUR problem.
However, we discovered that with the increase of iterationand more communication, while solutions for certain
subproblems are optimized by consulting with neighbors or
local search, the number of non-dominated solutions
collected from the crowd diminishes, as shown in
Figure.4.The diminution indicates that plenty subproblems
take the current optimal solution provided by neighbor as its
optimal solution, so they discard their current solutions,which are non-dominated in a global view but not better
solutions for their specific subproblems than their neighbors’.
Further work would experiment with more test problems andexplain the phenomenon. An expedient here is to commandindividuals in the crowd submit its the current solution toevaluation engine every time it pauses (before
communication), instead of solely submitting the solution at
the very end, thus the discarded solutions are also in thecontrol of evaluation engine and likely to contribute in the
final non-dominated front.

Fig. 4. Nondominated Solutions collected from the crowd at the end of 200 Iteration,400 Iteraton,600 Iteration,800 Iteration, and 1000 Iteration.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

59

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

VI. CONCLUSION
In this paper, we proposed a modified ACO algorithm.
With several innovative strategies and its underlying
crowdsourcing model, the algorithm could perform well on
the KUR test problem.
The work so far reveals the potential of ACO in tacklingwith continuous problem, and specifically, continuous MOP,
in our context. In addition, our research provides an
empirical case for the employment of distributed ACO. The
model and algorithm framework would be a practicalreference for various decomposable problems.
As our research is still at an early stage, we left a few
facts unexplained, such as, explanation for the diminution
and compatibility with other MOPs. In future work, we will
strive for these questions.
REFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

J.D.Schaffer, "Multiple objective optimization with vectorevaluatedgenetic algorithms",Proceeding of International Conference on
Genetic Algorithms, 1985, pp.93−100.
C.M.Fonseca ， P.J.Fleming,"Genetic
algorithm
formultiobjectiveoptimization: formulation，discussion and generation",
Proceeding of the 5th International Conference on Genetic
Algorithms, 1993, pp.416−423.
N.Srinivas ， K.Deb, "Multiobjective optimization using nondominated sorting in genetic algorithms", Evolutionary Computation，vol.2(3)，1994,pp.221−248.
K.Deb, A.Pratap, S.Agarwal, T.Megarivan, "A Fast and Elitist
Multiobjective Genetic Algorithm:NSGA-II", IEEE Transactions on
Evolutionary Computation,vol.6(2), 2002,pp.182-197
J. Knowles; D. Corne,"The Pareto archived evolution strategy: a new
baseline algorithm for Pareto multiobjective optimisation
Evolutionary Computation", 1999. CEC 99. Proceedings of the 1999
Congress on ,vol.1,1999.pp.98-105
E.Zitzler ， L.Thiele."Multiobjective Evolutionary Algorithms:A
Comparative Case Study and the Strength Pareto Approach", IEEE

Transactions on Evolutionary Computation，vol.3(4),1999，pp.257271.
[7] E.Zitzler E ， M.Laumanns ， L.Thiele ,"SPEA2: Improvingthe
strength Pareto evolutionary algorithm",Evolutionary Methods for
Design，Optimization and Control with Applications to Industrial
Problems，2002,pp.95-100.
[8] I.Alaya C.Solnon, K.Gh´edira, , "Ant Colony Optimization for Multiobjective Optimization Problems",19th IEEE International
Conference on Tools with Artificial Intelligence(ICTAI2007),vol.1,2007,pp.450-457[9] K. Doerner, W. J. Gutjahr, R. F. Hartl, C. Strauss, and C. Stummer,“ Pareto ant colony optimization: A metaheuristic approach tomultiobjectiveportfolio
selection,”
AnnalsofOperations
Research,vol.131(1),2004, pp.79-99[10] C. Garcia-Martinez, O. Cordon, and F. Herrera, “A taxonomy and an
empiricalanalysis of multiple objective ant colony optimization
algorithmsfor the bi-criteria TSP,” European Journal of Operational
Research, vol. 180(1), 2007, pp. 116–148.
[11] D. Angus and C. Woodward, “Multiple objective ant colony
optimization,”Swarm Intelligence, vol. 3, pp. 69–85, 2009.
[12] M. López-Ibáñez and T. Stützle, “The impact of design choices ofmultiobjectiveant colony optimization algorithms on performance: An
experimentalstudy on the biobjective TSP,” Conference on Genetic &Evolutionary Computation, 2010, pp. 71–78.
[13] Q.Zhang, H.Li, "MOEA/D: A Multiobjective Evolutionary Algorithm
Based on Decomposition",IEEE Transactions on Evolutionary
Computation, vol.11(6), 2009, pp.712-731.
[14] M. Hosseini, K. Phalp, J. Taylor, and R. Ali, “The four pillars ofcrowdsourcing: A reference model,” 2014 IEEE Eighth International
Conference on Research Challenges in Information Science (RCIS),
2014, pp. 1–12.
[15] L.Ke, Q.Zhang, R.Battiti, "MOEA/D-ACO: a multiobjective
evolutionary algorithm using decomposition and AntColony",IEEE
Transactions on Cybernetics, 2013, vol.43(6), pp.1845-1859.
[16] K. Miettinen, Nonlinear Multiobjective Optimization. Boston,
MA:Kluwer, 1999.
[17] A.Murturi, B.Kantarci, S.F.Oktug, "A Reference Model for
Crowdsourcing as a Service",IEEE International Conference on
Cloud Networking, 2015, pp.64-66.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

60

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Efficient Crowd-Powered Active Learning for
Reliable Review Evaluation
Xinping Min∗ , Yuliang Shi∗ , Lizhen Cui∗ , Han Yu†∗ School of Computer Science and Technology, Shandong University, China† Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), NTU, Singapore

minxinping0105@126.com, {clz, lqz}@sdu.edu.cn, han.yu@ntu.edu.sg
Abstract—To mitigate uncertainty in the quality of online purchases (e.g., e-commerce), many people rely on review commentsfrom others in their decision-making processes. The key challenge
in this situation is how to identify useful comments amonga large corpus of candidate review comments with potentiallyvarying usefulness. In this paper, we propose the Reliable Review
Evaluation Framework (RREF) which combines crowdsourcing
with machine learning to address this problem. To improvecrowdsourcing quality control, we propose a novel review query
crowdsourcing approach which jointly considers workers’ trackrecords in review provision and current workloads when allocating review comments for workers to rate. Using the ratings
crowdsourced from workers, RREF then enhances the adaptive
topic classification model selection and weighting functions of
AdaBoost with dynamic keyword list reconstruction. RREF has
been compared with state-of-the-art related frameworks usinga large-scale real-world dataset, and demonstrated over 50%
reduction in average classification errors.

I. I NTRODUCTION
Electronic services, such as e-commerce, are often provided
by people with diverse skills, resources or motives. There issignificant uncertainty, especially when a user consumes such a
service. There is a strong need from users for review commentsfrom previous customers in order to reduce the uncertainty intheir decision processes. This has given rise to many review
service providers such as TripAdvisor where users can postcomments about a wide variety of such services for others tobrowse.
The success of these review service providers has createdanother problem - the proliferation of review comments. For
example, in 2015, TripAdvisor users posted over 320 million
reviews.1 This makes it challenging for a user to manually lookthrough the typically large number of reviews associated withan entity to identify those reviews that are most useful. Thus,
review recommendation services powered by natural language
processing (NLP) techniques can be a useful tool to help users
solve this problem. Training effective NLP models typicallyrequires well-labelled data. However, employing professionalannotators to produce the labels is often costly and time
consuming. In recent years, crowdsourcing has become anincreasingly popular approach for acquiring labels for large
training datasets [1].
Crowdsourcing refers to the process of harnessing the contributions of a large number of people, typically through the
1 http://www.tripadvisor.com/PressCenter-c4-Fact Sheet.html

Internet, to complete tasks [2]. Successful examples of crowdsourcing include the reCAPTCHA system which provides online security while utilizing the crowd’s efforts to digitize precomputer era books [3], and fold.it which allows the crowd tocontribute towards scientific discoveries [4]. Existing literaturein Active Learning has realized the advantages of dynamicallyobtaining labels via crowdsourcing [5], [6], and accounting for
the workers’ abilities [7], [8]. Thus, it is important to provide
quality control mechanisms when crowdsourced labels are to
be used to train machine learning models.
In this paper, we propose the Reliable Review Evaluation
Framework (RREF) which infuses crowd intelligence into
active machine learning. It contains two novel approaches toenhance the accuracy of identifying useful review comments
to be recommended to users. Firstly, a situation-aware task allocation approach is proposed which performs crowdsourcing
quality control. It jointly considers each worker’s track records
in review rating (i.e., label) provision and current workload
to make optimal trade-offs between the overall label quality
and time elapsed. Secondly, RREF enhances the AdaBoost
framework by combining dynamic keyword list reconstruction
with adaptive selection and re-weighting of multiple topic classification models to improve overall classification accuracy.
We prove that the task allocation strategies computed by
RREF can achieve close to the optimal overall accuracy andthat the proximity to the optimal value can be controlledeasily through trade-offs in waiting time. Furthermore, RREF
has been extensively evaluated using a large-scale review
comments dataset crawled from Amazon China. Comparedto two state-of-the-art frameworks, RREF achieves more than
50% reduction in average classification error rates.
II. R ELATED W ORK
There is considerable work in the literature on NLP throughtraditional machine learning approaches [9], [10], [11], [12].
Techniques such as Latent Dirichlet Allocation (LDA) [13] areoften used.
As well-labelled training data are important for machinelearning, researchers are always looking for ways to gather
good quality labels at scale. In [1], [14], the authors haveshown that a multitude of workers can potentially produce
useful labelled data for NLP tasks as well as experts can. Following these successes in utilizing crowd intelligence, recently,
active learning frameworks which include human labellers© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

61

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

into the machine learning process through crowdsourcing arestarting to emerge. In [15], the authors proposed a framework
which adopts crowdsourced labels from workers and experts
to reduce the noise in the labels.
The framework proposed in [16] - JCF - is the mostclosely related to our proposed RREF in this paper. JCF
combines crowdsourced review comment ratings with active
learning SVM classifiers with the aim to improve classification
accuracy. Expert supervisors’ opinions are used to adjust the
Random Active Learning SVM and the Margin-based Active
Learning SVM, while opinions from common crowdsourcing
workers are used to adjust the Crowd Active Learning SVM
classifier.
Nevertheless, existing frameworks do not possess quality
control mechanism for the crowdsourcing step. Nor do they
adjust the keyword vectors during the active learning process
in response to the crowd’s feedback. In the following section,
we explain how the proposed RREF addresses these limitations.

comment ratings are optimized by the proposed HIT allocation
approach.
While workers are working on the HITs, the framework
calculates distributions of different topics and maps the comments into vectors. The comment classification model will thencompute the usefulness scores for the comments. When ratingsfrom the workers are received, they are used to update theweight values assigned to each classifier so that the workers’
opinions can be used to guide the adjustment of relative
importance given to each classifier. Once the weight values
converge, the training process terminates and a classification
model with accuracies improved by human intelligence can beobtained.
A. Crowdsourcing Quality Control

III. T HE P ROPOSED RREF
Our goal is to construct a crowdsourcing-powered machinelearning framework which can accurately identify useful review comments as shown in Figure 1. Based on the rating
information (i.e., the labels) provided by crowdsourcing workers, we aim to improve the classification model accuracy.
Fig. 2. Our rating request HIT template.
Convert a Review
Comment to an HITTask Allocation via
Crowdsourcing Quality
Control Mechanism

Update Workers’
Reliability Scores

Crowd Workers

Aggregate Ratings

Map Comments to
Vectors and Obtain Topic
Distributions

Topic 1 Classifier...

Update Classifier Weights

Topic i Classifier...

Topic n Classifier

Classified
Correctly?

Fig. 1. The system architecture of RREF.

When new review comments are received, the proposed
framework first maps these comments into human intelligencetasks (HITs) using our HIT template. Then, these HITs are
dynamically allocated to a multitude of workers to obtain theirratings. The quality and timeliness of the crowdsourced review

Review comments, especially those written in Chinese language, may contain subtleties which are difficult for today’snatural language processing techniques to recognize. However,they can be very clear to a speaker of the language. Thus, it
is advantageous to integrate ratings about the usefulness ofreview comments from humans through crowdsourcing into
the machine learning process. For this purpose, we designed
a crowdsourcing HIT template as shown in Figure 2 whichdisplays a review comment to a worker and allows him to
select how useful he feels the comment can be to his decisionmaking process.
Nevertheless, when obtaining such ratings, the heterogeneity
in workers’ experience and reliability will inevitability affect
the quality of their comments. Thus, it is not advantageous tosimply rely on the traditional mode of crowdsourcing where
the request for comments about a service is just advertised
to workers waiting for them to respond (similar to Amazon’s
Mechanical Turk (mTurk)2 ).
To address this problem, we propose a dynamic rating
crowdsourcing approach. Whenever new review comments
arrive, they are automatically composed into HITs by RREF
following our HIT template. With tracking tools such as
Turkalytics [17], workers’ behaviour related data in a given
crowdsourcing system can be monitored efficiently. Then, theHITs are allocated to a multitude of workers based on the
proposed crowdsourcing approach to obtain ratings.
2 https://www.mturk.com/mturk/welcome© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

62

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Let Qw (t) be the number of rating requests in a worker w’s
pending task queue at time slot t. The queueing dynamics canbe expressed as:

Qw (t + 1) = max[0, Qw (t) + aw (t) − cw (t)](1)

where aw (t) is the number of new rating requests sent toworker w at time slot t, and cw (t) is the number of rating
requests completed by worker w at time slot t.
A worker w’s reliability at time slot t based on his trackrecords is denoted by γw (t) ∈ [0, 1]. In this paper, we
adopt the model proposed in [18] to compute the value ofγw (t). The model computes not only worker’s probability of
producing a useful rating, but also the uncertainty based on theamount of available track records. Thus, γw (t) represents theprobability of a worker w’s rating for a comment being usefulfor improving our machine learning model discounted by theuncertainty involved. Nonetheless, other models for computing
a worker’s reliability [19] can also be used in conjunction withthe proposed crowdsourcing approach as long as the outputs
can be normalized to the range of [0, 1].
Let U (t) be the overall utility (i.e., the expected accuracy ofratings obtained) of allocating requests for ratings to a given
crowd of N workers at time slot t. We have:

U (t) =NX

γw (t)aw (t).

of the variations in workers’ workloads, we have:∆(Q(t)) = E{FL (t + 1) − FL (t)|Q(t)}T −1 N 
1 XX 1 2
1=
Qw (t + 1) − Q2w (t)T t=0 w=1 22
T −1 N
1 XX=
Qw (t)aw (t)T t=0 w=1

(4)

1
− cw (t)[Qw (t) + aw (t)] + [a2w (t) + c2w (t)]
2
T
−1
N
1 XX
6
[Qw (t)aw (t) + Ψ]
T t=0 w=1
where Ψ = 12 [a2max + c2max ] > 0. amax and cmax are thephysical upper limits on aw (t) and cw (t) for all w and t in agiven crowdsourcing system, and can be regarded as constantsfor our purpose.
Thus, we can formulate the utility-minus-surprise objectivefunction as:
σE{U (t)|Q(t)} − ∆(Q(t))
T −1 N
1 XX(σγw (t)aw (t) − Qw (t)aw (t) − Ψ)>T t=0 w=1

(5)

where σ > 0 is a weight factor determining the relative importance given to maximizing utility versus minimizing surprise.
By considering only the terms containing the solution variables
aw (t) for all workers, the optimization can be formulated as:
Maximize:T −1 N

(2)

w=1

1 XX
aw (t)[σγw (t) − Qw (t)]T t=0 w=1

(6)

γw (t) > γmin , ∀w, t(7)

aw (t) 6 cmax , ∀w, t(8)

Subject to:
Recent findings in social science suggests that human choice
behaviour can be accounted for by a mixture of utility
maximization and surprise minimization [20]. We adopt thisprinciple in our crowdsourcing approach in order to improvethe chance of the computed request allocation plan to be
accepted by the workers. To this end, we need a way to model
the concept of surprise for workers working on the requests.
The variation in workload for a worker can be regarded as aform of surprise, which shall be minimized.
The Lyapunov function [21], which measures the overall
congestion of demand on workers at time slot t, can be
expressed as:N

FL (t) =

1X 2
Q (t)
2 w=1 w

(3)

Let Q(t) denote a vector of all workers’ pending task queues
at time slot t. Using the Lyapunov drift ∆(Q(t)) as a measure

where γmin is the minimum reliability value a worker w mustachieve in order to be allowed to participate. The value can be
set by the system administrator. By setting the upper limit ofaw (t) to cmax in Constraint 8, we intend to keep the workers
busy. Nevertheless, other strategies can be easily implemented
by changing Constraint 8.
The solution variables for the above optimization areaw (t) ∈ {0, Z+ } for all workers whenever there are newreview comments to be rated by multiple workers. To maximize Eq. (6), at each time slot t, Algorithm 1 computes thevalues of the expression [σγw (t) − Qw (t)] (denoted as ηw (t)for simplicity) for every worker w. It then sorts all workers
in descending order of their ηw (t) values. For each worker w
who satisfies Constraint (7), set aw (t) based on Constraints(8). The rating request allocation terminates when there are
no more workers with ηw (t) > 0.
Once the ratings for a comment are received from all workers, they are aggregated through majority voting. Then, the© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

63

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Algorithm 1 Rating Request Allocation
Require: Number of review comments Nc (t) which need to
be rated at a given time t; σ; Qw (t), cmax and γw (t)values for all workers.
1: Compute ηw (t) for all w;
2: Rank all w in descending order of ηw (t);3: for each worker w do
4:
if ηw (t) > 0 and γw (t) > γmin then5:
if Nc (t) < cmax then6:aw (t) = Nc (t);
7:
else
8:
aw (t) = cmax ;9:end if10:else
11:
aw (t) = 0;12:
end if
13: end for
14: return {a1 (t), a2 (t), ...}, ∀w;

where ωm > 0 represents the weight of the m-th classifier.
Classifiers with higher error rates are given lower weights.
km (Ti ) represents the classification output of a classifier km
on a multidimensional input feature vector Ti (i.e., a keyword
list representing a review comment).
Suppose there are already M − 1 classifiers included in the
AdaBoost model, and we want to select one more classifier
from a pool of available classifiers to join the cascade.
The problem becomes how to select a classifier
from the classifier pool given a training set{{T1 , u1 }, {T2 , u2 }, ..., {Tn , un }} in which a comment Ti
has a usefulness score ui . To address this problem, the error
function  can be expressed as follows:

=

n
X

αim e−ωm f (ui ,km (Ti ))

(10)

i=1

where
workers’ performance for this task is evaluated by comparing
their individual ratings against the aggregated rating. Workers
whose ratings are the same as the aggregated rating will havetheir reliability scores increased, while those whose ratings
differ from the aggregated rating will have their reliability
score reduced following [18].

In order to make the proposed framework flexible in adapting what classification technique to employ with changing
situation, we adopt AdaBoost [22] as the basis for RREF.
AdaBoost provides an effective framework for combining
multiple classifiers to enhance the overall performance. It
consists of two steps. Firstly, suppose we have selected mclassifiers and need to select an additional classifier fromthe rest of available classifiers, the one with the minimum
misclassification cost among all remaining classifiers shall beselected. Secondly, the weight for the newly added classifier
is determined by minimizing the total misclassification cost.
With the help from human labellers through crowdsourcing,
the weight of each training sample can be obtained during
the training process. The training sample weights are updated
according to the following principle. Misclassified samples are
assigned higher weight values while the weight values forcorrectly classified samples are reduced. In this way, RREF
can focus on selecting a new classifier which can play amore significant role in improving the classification accuracy
for the misclassified samples, thereby improving the overall
performance.
The classification result of a model containing M classifiers
can be expressed as a linear combination:
MX

ωm km (Ti )m=1= C(M −1) (Ti ) + ωM kM (Ti )1e−ui C(m−1) (Ti )(9), if m = 1, otherwise.

(11)

The function f (ui , km (Ti )) is defined as:f (ui , km (Ti )) =B. Crowd-Powered Adaptive Classifier Selection

CM (Ti ) =αim =−1, if km (Ti ) 6= ui
1, otherwise.

(12)

Thus,  can be re-written as:X=

km (Ti )6=ui

αim eωm +X

αim e−ωm(13)

km (Ti )=ui

To determine the updated weight values ωm which minimize
the overall error rate , we find the first order derivative of with respect to ωm and equate it to 0. Solving the differential

Algorithm 2 keyword list reconstruction
Require: A keywords list K; a review comment Ti containing a set of keywords.
1: Vector ItemList = {∅};
2: if Predicted label for a comment Ti differs significantly
from the crowd annotated label for Ti then3:
Rank all keywords in K in ascending order of theircontributions to classification accuracy;
4:
Select the top k keywords from the re-ordered K toform a replacement list Ir ;5:
Randomly select keywords from (K − Ir ) to replace
the keywords in Ir ;6:
Combine the replaced Ir with (K − Ir ) to form Vector ItemList;
7: end if
8: return Vector ItemList;© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

64

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Since Qw (t) > 0 for all w and t, FL (·) > 0 and FL (0) = 0,
Eq. (17) can be re-written as:

equation yields:
P

1
= ln
2

Pn

=

1
ln
2

!

m
k (T )=u αi
P m i i m
km (Ti )6=ui αi1ωm = ln2mi=1 αi −



1−ε
ε

T −1

m
km (Ti )6=ui αiPmkm (Ti )6=ui αi

P

!

(14)

T −1

> U∗ +



km (Ti )6=ui
Pn
is the weighted error rate of awhere ε =m
i=1 αiclassifier m among all selected classifiers.
In practice, there can be cases in which one classifier
predicts a result which differs too much from the groundtruth. For instance, given a review comment Ti , a classifier
may label it as “over 90%” useful, while the aggregate label
from crowdsourcing workers is “10% to 30% useful”. To
deal with these situations, we further enhance AdaBoost by
proposing the comment vector reconstruction method as shownin Algorithm 2.

T −1

T −1

In this section, we analyze the performance bounds of the
proposed crowdsourcing quality control mechanism.
Assume there exist constants σ, ξ and ϕ at a given time slott such that:NX

Qw (t) − ϕ,

(15)

w =1

where U ∗ is the optimal rating accuracy produced by a mechanism which can perfectly predict the workers’ behaviour. We
have:
N
X

N

1 X X
E{Qw (t)}T t =1 w =1
6

σ

(18)

ϕ
1
ϕ
+
E{FL (Qw (T ))} > U ∗ − .
σ
σTσ
Therefore, we prove that the lower bound on the time averaged expected accuracy of the overall rating on review
comments from workers achieved by following the proposed
crowdsourcing quality control mechanism is within ϕσ of that
achieved by the theoretical optimal solution. If σ is increased
to significantly large, most rating requests will concentrate on
workers who have demonstrated good reliability in the past,
thereby resulting in high rating accuracy.
Since U ∗ > 0, re-arranging the terms in Eq. (18) yields:

IV. A NALYSIS

σU (t) − ∆(Q(t)) > σU ∗ + ξ

N

ξ X X
E{Qw (t)}σT t =1 w =1

−

αm
i

P

N

1 X X
E{γw (t)aw (t)}T t =1 w =1

N

σ X X
E{γw (t)aw (t)}ξT t =1 w =1

(19)

σ
ϕ
1
− U∗ + −E{FL (Qw (T ))ξξξT
T −1 Nϕσ XX
E{γw (t)aw (t)} + .
6
ξT t=1 w=1
ξ
Therefore, we prove that, by following the proposed crowdsourcing quality control mechanism, the upper bound on
the time-averaged pending task queue lengths for workers isdirectly proportional to σ. Overall, a larger σ value will resultin high accuracy of ratings for review comments, but longer
delays in obtaining these ratings from crowdsourcing workers.
V. E XPERIMENTAL E VALUATION

E{γw (t)aw (t)}w =1

− E{FL (Qw (t + 1)) − FL (Qw (t))} > σU ∗
N
X

+ξ

(16)

E{Qw (t)} − ϕw =1Fig. 3. An example review from Amazon.cn.

which holds for all time slots. By summing both sides of Eq.
(16) over all t ∈ {0, 1, ..., T − 1}, we have:

σ

T
−1 X
N
X

E{γw (t)aw (t)}t =1 w =1− E{FL (Qw (T )) − FL (Qw (0))} > σT U ∗
+ξ

T
−1 X
N
X
t =1 w =1

E{Qw (t)} − T ϕ.

(17)

Fig. 4. The keyword vector from our datasets.

In this section, we evaluate the performance of the proposed RREF using a dataset of real-world review comments© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

65

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

20%
Avg. Classification Error Rate

crawled from the Amazon China e-commerce website.3 The
product reviews are written in Chinese and are obtained using
a distributed web crawler over an 18 month period fromMarch 2014 to August 2015. In total, we have obtained 10
datasets each containing 2,000 to 3,000 review comments
under various sub-categories of mobile phone/communications
products. An example review is shown in Figure 3.4 . Using aword segmentation tool, we form a keyword vector from allcomments (Figure 4).
RREF is compared against two state-of-the-art approaches:1) the AdaBoost framework [22], and 2) JCF: the crowdsourcing enhanced classification framework proposed in [16]. We
adopt the average classification error rate, E ∈ [0%, 100%],
as the metric to compare the performance of RREF against
these frameworks. The lower the value of E, the better the
performance.

15%

10%

5%

0%

A. Results and Discussions

2

4
6
8
Test Dataset ID

10

20%
TCM 1
TCM 2
TCM 3
TCM 4
TCM 5
TCM 6

15%

10%

Avg. Classification Error Rate

Avg. Classification Error Rate0

Fig. 6. E values of different classifiers on TCMs.

20%

5%

0%

SVM
Random Forest
Bayes

0

2

4
6
8
Test Dataset ID

10

RREF
AdaBoost
15%

10%

5%

0%

0

2

Fig. 5. E values of topic classification models (TCMs).

In the experiments, we include six different topic classification models (TCMs) and the SVM classifier, Random Forestclassifier and the Naı̈ve Bayes classifier for the proposed
RREF to dynamically select from and adjust the weights.
The performance of the six TCMs on the 10 test datasets
we collected are shown in Figure 5. It can be observed that
the TCMs perform differently as they each have different
advantages and limitations. The same observations can bemade for Figure 6 which shows the performance of the SVM,
Random Forest and Naı̈ve Bayes classifiers for the 10 test
datasets.
Figure 7 compares the performance of RREF against thatof AdaBoost. The principles of dynamic TCM selection andweight adjustment are the same in both frameworks. However,4
6
8
Test Dataset ID

10Fig. 7. RREF v.s. AdaBoost.

RREF is equipped with the additional features of review comment rating crowdsourcing with quality control (Algorithm 1)
and keyword list reconstruction (Algorithm 2) based on TCM
performance. The reconstructed keyword vector produced by
RREF is shown in Figure 8. These two new features haveresulted in improved performance by RREF. Over all 10
training sets (with over 20,000 review comments), RREF
achieves an average classification error rate of 3.35%, which is
62% lower than the average classification error rate of 8.22%
achieved by AdaBoost.

3 http://www.amazon.cn/
4 The datasets used in our experiments can be downloaded from http://211.
87.227.218:8081/en/download/

Fig. 8. RREF reconstructed keyword vector.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

66

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Avg. Classification Error Rate

20%

61573212, 61272241; the Natural Science Foundation of Shandong Province of China under Grant No. ZR2014FM031,
ZR2013FQ014; the Shandong Province Independent Innovation Major Special Project No. 2015ZDXX0201B03; theShandong Province key research and development plan No.
2015GGX101015, 2015GGX101007; the Fundamental Research Funds of Shandong University No. 2015JC031; theNational Research Foundation Singapore under its Interactive
Digital Media (IDM) Strategic Research Programme; and the
Lee Kuan Yew Post-Doctoral Fellowship Grant.

RREF
JCF
15%

10%

5%

0%

R EFERENCES

0

2

4
6
8
Test Dataset ID

10Fig. 9. RREF v.s. JCF.

Figure 9 shows the performance of RREF against thatof JCF. JCF is closely related to the proposed RREF as it
combines crowdsourced review comment ratings with active
learning SVM classifiers with the aim to improve classification
accuracy. Nevertheless, JCF does not possess any quality
control mechanism for the crowdsourcing step, and does not
adjust the weight values assigned to variants of the SVM
classifiers. RREF achieves an average classification error rateof 3.5%, which is 58% lower than the average classification
error rate of 8.27% achieved by JCF.
VI. C ONCLUSIONS AND F UTURE W ORK
In this paper, we proposed a crowd intelligence powered
dynamic machine learning framework - RREF - to help ecommerce systems accurately identify useful review commentsfrom users. RREF contains a novel data-driven crowdsourcing
quality control mechanism based on the Lyapunov queueing system optimization technique, and a dynamic classifiercomposition and weight adjustment approach with keyword
list reconstruction. RREF offers an effective way to infuse
crowd intelligence into machine learning to improve theoverall review comment classification which is an importantresearch areas in the field of NLP. Theoretical analysis proves
the existence of performance bounds for the crowdsourcing
quality control mechanism. Extensive experimental evaluations
using a large scale real-world dataset demonstrated that RREF
significantly outperforms state-of-the-art related frameworks.
In future research, we will further improve the crowdsourcing quality control mechanism to include other considerations,such as workers’ social relationships [23], personal preferences[24] and miss-interpretation of outcomes [25], which may
impact the biases in crowdsourced review comments ratings.
ACKNOWLEDGMENTS
This research is supported, in part, by the National Natural
Science Foundation of China under Grant No. 61572295,[1] N. Zeichner, J. Berant, and I. Dagan, “Crowdsourcing inference-ruleevaluation,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL’12), 2012, pp. 156–160.
[2] P. Michelucci and J. L. Dickinson, “The power of crowds,” Science, vol.
351, no. 6268, pp. 32–33, 2016.
[3] L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum,“reCAPTCHA: Human-based character recognition via web security
measures,” Science, vol. 321, no. 5895, pp. 1465–1468, 2008.
[4] R. Silberzahn and E. L. Uhlmann, “Crowdsourced research: Many hands
make tight work,” Nature, vol. 526, pp. 189–191, 2015.
[5] Y. Yan, R. Rosales, G. Fung, and J. G. Dy, “Active learning fromcrowds,” in Proceedings of the 28 th International Conference on
Machine Learning (ICML’11), 2015.
[6] J. Zhong, K. Tang, and Z.-H. Zhou, “Active learning from crowds withunsure option,” in Proceedings of the 24th International Conference on
Artificial Intelligence (IJCAI’15), 2015, pp. 1061–1067.
[7] H. Yu, Z. Shen, C. Miao, and B. An, “Challenges and opportunities for trust management in crowdsourcing,” in Proceedings of theIEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’12), 2012, pp. 486–493.
[8] H. Yu, C. Miao, Z. Shen, C. Leung, Y. Chen, and Q. Yang, “Efficient
task sub-delegation for crowdsourcing,” in Proceedings of the 29th AAAI
Conference on Artificial Intelligence (AAAI-15), 2015, pp. 1305–1311.
[9] Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart, “Crowd
mining,” in Proceedings of the 2013 ACM SIGMOD International
Conference on Management of Data (SIGMOD’13), 2013, pp. 241–252.
[10] H. Su, K. Zheng, J. Huang, H. Jeung, L. Chen, and X. Zhou, “Crowdplanner: A crowd-based route recommendation system,” in Proceedings of the IEEE 30th International Conference on Data Engineering(ICDE’14), 2014, pp. 1144–1155.
[11] J. Su, D. Xiong, Y. Liu, X. Han, H. Lin, J. Yao, and M. Zhang,“A context-aware topic model for statistical machine translation,” inProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL’15), 2015, pp. 229–238.
[12] J. Derrac and S. Schockaert, “Inducing semantic relations from conceptual spaces: A data-driven approach to plausible reasoning,” Artificial
Intelligence, vol. 228, pp. 66–94, 2015.
[13] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,”
Journal of Machine Learning Research (JMLR), vol. 3, pp. 993–1022,
2003.
[14] D. Hovy, B. Plank, and A. Søgaard, “Experiments with crowdsourced reannotation of a pos tagging data set,” in Proceedings of the 52nd Annual
Meeting of the Association for Computational Linguistics (ACL’14),
2014, pp. 377–382.
[15] S. Hao, C. Miao, S. C. Hoi, and P. Zhao, “Active crowdsourcing forannotation,” in Proceedings of the 2015 IEEE/WIC/ACM International
Conference on Web Intelligence and Intelligent Agent Technology (WIIAT’15), 2015, pp. 1–8.
[16] J. Costa, C. Silva, M. Antunes, and B. Ribeiro, “On using crowdsourcing
and active learning to improve classification performance,” in Proceedings of the 11th International Conference on Intelligent Systems Design
and Applications (ISDA’11), 2011, pp. 469–474.
[17] P. Heymann and H. Garcia-Molina, “Turkalytics: analytics for human
computation,” in Proceedings of the 20th International Conference on
World Wide Web (WWW’11), 2011, pp. 477–486.
[18] Y. Wang and M. P. Singh, “Formal trust model for multiagent systems,”
in Proceedings of the 20th International Joint Conference on ArtificalIntelligence (IJCAI’07), 2007, pp. 1551–1556.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

67

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[19] H. Yu, Z. Shen, C. Leung, C. Miao, and V. R. Lesser, “A survey ofmulti-agent trust management systems,” IEEE Access, vol. 1, no. 1, pp.
35–50, 2013.
[20] P. Schwartenbeck, T. H. B. FitzGerald, C. Mathys, R. Dolan, M. Kronbichler, and K. Friston, “Evidence for surprise minimization over value
maximization in choice behavior,” Scientific Reports, vol. 5, no. 16575,
p. doi:10.1038/srep16575, 2015.
[21] M. J. Neely, Stochastic Network Optimization with Application to Communication and Queueing Systems. Morgan and Claypool Publishers,
2010.
[22] Y. Freund and R. E. Schapire, “A decision-theoretic generalization of
on-line learning and an application to boosting,” Journal of Computerand System Sciences, vol. 55, no. 1, pp. 119–139, 1997.
[23] J.-P. Mei, H. Yu, Y. Liu, Z. Shen, and C. Miao, “A social trustmodel considering trustees’ influence,” in Proceedings of the 17th
International Conference on Principles and Practice of Multi-Agent
Systems (PRIMA’14), 2014, pp. 357–364.
[24] H. Yu, Z. Shen, C. Miao, B. An, and C. Leung, “Filtering trust
opinions through reinforcement learning,” Decision Support Systems(DSS), vol. 66, pp. 102–113, 2014.
[25] Y. Liu, S. Liu, H. Fang, J. Zhang, H. Yu, and C. Miao, “Reprev:
Mitigating the negative effects of misreported ratings,” in Proceedingsof the 28th AAAI Conference on Artificial Intelligence (AAAI-14), 2014,
pp. 3124–3125.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

68

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

ETTF: A Trusted Trading Framework Using
Blockchain in E-commerce
Xinping Min1, Qingzhong Li1,2(), Lei Liu1, Lizhen Cui1,3
1School of Computer Science and Technology, Shandong University, Jinan, China
2
Dareway Software Co.,Ltd, Jinan, China
3
E-Commerce Research Center of Shandong University, Jinan, China
lqz@sdu.edu.cn

Abstract—In E-commerce, the security technology has become
major issue restricting the rapid development and popularizationof E-commerce. Existing solutions leverage blockchain protocols
to improve the credibility of transactions, but most of them havesome limitations, such as a lower throughput and higher
consensus latency. This paper presents a trusted framework(ETTF) using blockchain protocol in E-commerce to achieve ahigher credible trading. ETTF includes a peer blockchain
protocol (PBP) based on a peer blockchain architecture to
support the storage of massive transactions and instanttransactions. In PBP, the throughput scales nearly linearly
increase with the computation: the more computing power
available, the more blocks selected per unit time. In addition, thispaper introduces a strong consensus (ECA) in E-commerce toguarantee a higher credibility of transactions. ETTF is also
efficient because the number of messages it requires is nearlylinear in the network size. ETTF has been compared with
Bitcoin-derived blockchain in E-commerce, and demonstrates
better performance on throughput, latency, capacity, andnetwork bandwidth.
Keywords—Blockchain;transaction.

consensus

algorithm;

trustedI. INTRODUCTION
E-commerce plays an essential role in our life. Although
there are many major issues to be addressed in E-commerce.
First of all, the security problem of transaction should betreated seriously, such as information security. Second, theopenness of transactions should be taken into account carefully.
Because e-commerce platform is a centralized system,
provided services are massively centralized, resulting in bothstrict control by the central bodies and vulnerability if used bycriminals [1]. As a result of this, only trading center has
permission to provide related information or credibility. If a
trading center suffers from network attacks, it’s easy for a
attacker to tamper transactions through the treading center.
Moreover, Due to privacy information on current e-commerce
platform, it is inconvenient for the government to attainstatistics information. If it adopts one blockchain protocol [2]to process transactions in E-commerce, it can solve almost all
the problems.
The Bitcoin-derived blockchain holds great promises for
distributed ledgers. However, in E-commerce, each transaction
must be immediately processed, and massive transactions
occur simultaneously. In Bitcoin-derived blockchain, the block

size is currently limited to 1MB, resulting in that only 1 to 3.5transactions per second for Bitcoin [3]. If E-commerce adopts
Bitcoin-derived blockchain to solve those problems, it willcause a lower throughput and lower latency. So, we present a
Trusted Trading Framework Based on Blockchain in Ecommerce (ETTF), which can keep unlimited data and support
instant transactions, and performs much better to satisfy thedemands of E-commerce. The goal of ETTF is to construct ahigher security, public, trusted, autonomous E-commerce
ecosystem.
Most blockchain protocols can guarantee a higher securityof transactions by sending all transactions to other peers in theprice of consuming much computation resources.
The key idea in ETTF is to construct peer blockchain
protocol which divides all peers into different committee.
Unlike other blockchains, PBP doesn’t leverage proof-of-workmechanism used in Bitcoin. Some blockchain protocols use aset of validation peers to validate the legality of transactions,but all of those blockchains consider the validation peer valid
always. In PBP, we develop a validation peer selection
algorithm, which can dynamic changes membership in eachepoch. PBP is a collaborative protocol without wasting
computation resources. Because PBP will increase
communication costs. To reduce the communication costs, wedevelop a new propagation algorithm in E-commerce (EPA).
EPA introduces a validation peer set as the representation of allpeers to validate the legality of block. As PBP can’t achieve theopenness and a peer can still tamper the information, so wedevelop a new global consensus algorithm (ECA) that is run inall peers. ECA leverages proof-of-work mechanism to write thehash value of a block into a global block in every epoch.
The contributions of our research are as follows:
 A Trusted Trading Framework Based on Blockchain
in E-commerce. By constructing a three-layer
blockchain, we represents a Trusted Trading
Framework Based on Blockchain (ETTF), which has
better performance on E-commerce. ETTF canguarantee a higher credibility of transaction, support
instant transaction, and keep unlimited data.
Higher throughput and lower latency: By partitioning
network into several sub-committees, ETTF cansignificantly achieve higher throughput and lower
latency than Bitcoin when the same trust assumptions
as Bitcoin.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

69

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Peer Key BlockPeer Micro BlockPeer Key BlockPeer Micro Block
Figure 1: Peer Blockchain Architecture

Hash value: <sha3 hash>Timestamp: <block creation timestamp>
Owner: <public key of the node creating the block >Voter List: <list of federation nodes public keys>
signature : <ECDSA signature of vote block >Hash value: <sha3 hash>Timestamp: <block creation timestamp>
Owner: <public key of the node creating the block >Transaction List: <list of transactions >Peer micro block List: <list of Micro block >a: Peer Micro Block Architecture
b: Peer Key Block Architecture
Figure 2: Block Model in peer blockchain

The remainder of the paper is organized as follow: Section
II introduces the related work about blockchain scalability. In
Section III, we provides the overview of ETTF. Section IV
describes the ETTF in detail. Section V introduces theexperiment about ETTF. This paper is concluded in Section VI.
II. RELATED WORKThe core technology in Bitcoin-derived blockchianinnovation powering these systems is the Nakamoto consensus
protocol for maintaining a distributed ledger known as theblockchain. The blockchain technology provides adecentralized, Byzantine fault-tolerant transaction mechanism,
and promises to become the infrastructure for a new generationof Internet interaction, including anonymous online payments[4], remittance, and transaction of digital assets [5]. Ongoingwork explores smart digital contracts, enabling anonymous
parties to programmatically enforce complex agreements [6].
Despite its potential, Bitcoin-derived blockchain protocols also
face a significant scalability barrier. Bitcoin has scalability
issues in terms of throughput, latency, capacity, and network
bandwidth. For example, increasing block size can improvethroughput, but the causing bigger blocks take longer time topropagate in the network. Reducing the block interval reduces
latency, but leads to instability that the chain is forked into
branches.
Many consensus protocols use randomization mechanism,
especially in order to circumvent the well-known FLP
impossibility result for deterministic asynchronous networks[7]. Generally, nondeterministic restatement of the consensus
problem involves Las Vegas rules: the network must
eventually reach consensus, although the amount of time taken
may be unbounded. King and Saia [8] considered the Las
Vegas rules is too strict for large scale networks. Since
deterministic consensus requires at least ( n 2 ) message bits
[9], any randomized consensus protocol using fewer thanquadratic messages must be a Monte Carlo protocol with
nonzero probability of failure [10], [11].
In other aspects [12], Zerocash [13] and Zerocoin [14]provide anonymous payments through the use of a novel typeof cryptographic proofs. Unlike Bitcoin , [13] is an anonymous
payment system that can provide excellent blockchain privacy
and more higher throughout. However, the parties entrusted toanonymize transactions in [15] can still violate users’
anonymity, even if they are honest-but-curious. Ittay Eyal [16]presents Bitcoin-NG, a new Bitcoin-derived blockchain
protocol designed to scale. Bitcoin-NG achieve that Byzantine
fault tolerant, robust to extreme churn, and sharing the same
trust model obviating qualitative changes to the ecosystem.
Hong-Jie He [17] proposes a blockchain based fragile
watermarking scheme to solve the issue of security andaccuracy of tamper localization.
III. OVERVIEW OF THE TRUSTED TRADING FRAMEWORKUSING BLOCKCHAIN IN E-COMMERCE
We denote by p represents a peer. The identities of peers
are established using public-key cryptography as follows:
When a peer p joins the network for the first time, p generates
a public and private key pair. In E-commerce, each peer
represents a third party payment platform, logistics platform orsynthesis supervision platform. We introduce three types ofpeer: global blockchain generation peer ( gp ), global
blockchain validation peer ( vp ), supervision peer ( sp ) andordinary peer ( op ). Only gp has permission to generate global
block, only vp has permission to validate the ‘legality’ of global
block. Each peer can deal with transactions except sp .
Each sp just manger the whole network. All peers compose
Trusted Trading Network (TTEN) based on ETTF.
ETTF proceeds in epochs. In each epochs, first, each peer
writes the transactions into blocks. Meanwhile, according topeer’s honesty that indicates the probability of tamperingtransaction, available mining power that indicates peer's
computing power, ETTF generates gp and vp set from all peers.
Second, all gp collaborative generate E-commerce global block,and send global block to vp set. Thirdly, vp set integrates
multiple global block into a single one and sends it to sp set. If
this global block is valid, sp set sends this block to the© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

70

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

remained peers and vp set generates new gp , sp and vp set for
next epoch. In TTEN, each peer must keep all global block.
ETTF includes two parts: Peer Blockchain Protocol (PBP)
and a strong consensus algorithm in E-commerce (ECA). PBP
is a blockchain protocol to protect the security of transactions
in peer. ECA is just used to prevent the “dishonesty” peers totamper other peer’s transactions.
IV. THE TRUSTED TRADING FRAMEWORK USINGBLOCKCHAIN IN E-COMMERCE
A. Peer Blockchain Protocol
To protect the security and trusted of transactions in eachpeer, this paper presents a peer blockchain protocol (PBP).
Each peer has a peer blockchain, which just keeps transactions
involved in transfer of assets. Figure 1 indicates thearchitecture of peer blockchain. In PBP, status of each block
introduces three types: undecided, valid and invalid. In peer
blockchain, each peer micro block (PMB) contains a group oftransactions, a vote list by vp set and includes signature list as
shown in Figure 2a. Each peer key block (PKB) contains agroup of PMB as shown in Figure 2b. There is majority ofpositive or negative votes for a PMB in vp set, this PMB can gofrom undecided to valid or invalid. It’s a rule that only valid
PMB can be write into a PKB. If PMB is considered to be valid,those transactions PMB contained also are valid, andtransactions be in effect that will be executed.
PBP proceeds in epochs. Each transaction and PMB goesfrom undecided to valid or invalid. To guarantee the time fromoccurrence of transaction to transaction in effect less than one
second, multiple PMB will be generated in one time, but only
one PKB will be generated in one time. To protect the security
and trusted of transactions, each PMB must be signature
with vp set. As shown in algorithm 1: PBP can be divided into
three steps: firstly, PBP checks the ‘legality’ of transactions.
Second, PBP writes valid transactions into PMB, and sends tothe vp set. Thirdly, PBP writes valid PMB into PKB.
In general, if a vp receives a PMB, vp checks legality ofthis PMB, and send this PMB to any vp with peer itself
signature. If majority of vp voted that the block is valid, all
vp keep this PMB into itself peer blockchain. The traditional
p2p broadcast mechanism will consume much more network
bandwidth. If each peer sends the PBM with signature one by
one or in order, sometimes the first peer will never receive thevoting result, if any peer breaks down will stop the process ofvoting. So, this paper develops a new broadcast mechanism
EPA, which has lower communication costs. Our goal is to
reduce the communication costs from N N to K  N ,where N represents the number of peers in ETTN
and 1  K  N . The key idea in EPA is to assign each identity toa random group. First, we divides all vp into m groups g i .
Second, each peer generates the next sending list SendList byrandomly selecting one peer form each groups,
where SendList  { pi , pi  g i ,1  i  m} . Suppose the number ofpeers in one group is N / m , the maximum of propagation

frequency is m  N  m2 . Because m  N  m2  2 N 2 , and m N ,so m  N  m2  K  N . Moreover, if any vp finds majority of vp
voted PMB is invalid or valid, this vp returns this PMB to peer
who send this PBM at first and other vp . If the peer who sends
this PBM at first, receives enough PBM with signatures, thispeer will send a message to all vp to stop the voting process of
this PMB. So, EPA has better performance than traditional
mechanism.
In PBP, only if transactions were write into a PMB, anyone
can’t tamper the information in any ways. Because any PMB
must be agreed with majority of peers, this PMB can be valid.
Moreover, majority of vp must keep this PMB into theirblockchain. If one peer wants to tamper a exist transactions, it’s
impossible to tamper all information in other peers
simultaneously.
Algorithm 1: PBP
Input: All Transactions
Output: Peer Key Block
1. For each transaction T, whose status(T )  undecided
2.
Check legality of T and assigned invalid or valid
3.
If status(T )  invalid
4.
Remove T
5.
End If
6. End For
7. For each time epochs
8.
Select VTset that each T satisfy status(T )  valid
9.
If size(VTset)  1M
10.
Write VTset into a new PMB
11.
Else
12.
The size(VTset)==1M write into PMB
13.
The size(VTset)-1M write into anther PMB
14.
End If
15. End For
16. For each PMB
17.
If status( PMB)  undecided
18.
Send PMB to vp set
19.
Else if status( PMB)  invalid
20.
Delete the PMB
21.
Reassign each T to undecided
22.
Return to step 1
23.
Else if status( PMB)  valid
24.
Each T is in effect25.
Write PMB into a new PKB
26.
End If27. End For
B. E-commerce Consensus Algorithm
After each peer generate their PKB, it’s necessary toconstruct a global blockchain (E-Blockchain) to keep all PKB
in ETTN. In ETTN, the E-Blockchain provides an index to all
peer blockchain, so that each peer can validate PMB received
from other peers. Figure 3 shows the architecture of block in EBlockchain. Each global block (EGB) contains a group of PKB,
the constructor and voter list.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

71

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

In E-commerce, if we adopts the proof-of-work mechanism
to generate EGB, it’s will consume much resources that don’t
need to waste. In other words, it will increase cost of Ecommerce platform. Furthermore, the proof-of-workmechanism should be careful of the problem of crypto puzzle.
In Bitcoin, the problem of crypto puzzle, sets by this value, isdynamically adjusted such that blocks are generated at an
average rate of one every ten minutes. More problem difficultylager, more time needed to generate a block. In E-commerce, tosupport instant transactions, it’s necessary to use acollaborative mechanism without sacrificing security, which
can reduce the consensus latency. So, EGB adopts acollaborative mechanism as follows: before each EGB isgenerated, all gp should collaboratively generate the hashnumber. ECA uses the Shamir’s Secret Sharing Scheme (SSSS)
to generate the hash number. Each gp selects a random number,by Shamir’s Secret Sharing Scheme dividing random number
into multiple part with other gp signatures, and broadcasts
every part of random number to all gp . After receiving all partof any peer’s random number, each gp decodes the part as it
can and broadcasts those parts. Only each peer receives enoughunique part, such as K of N, this peer can get the initial random
number, and add all peer’s random number to a hash number.
Each peer generates a new EGB according to the hash number.
Hash value: <sha3 hash>Timestamp: <block creation timestamp>
Constructor: <public key of the node creating the block >Voter List: <list of federation nodes public keys>
signature : <ECDSA signature of vote block >PKB List: <list of PKB >Figure 3: Block Model in E-Blockchain.

As shown in Algorithm 2, BCA can be divided into threesteps: first, each gp gets the hash number, writes received
transactions into an EGB, and sends this EGB to vp set. Second,
all vp integrate received EGB into single one, and send thisEGB to sp set. Thirdly, all sp validate the “legality” of EGB,
and send EGB with votes list to remained peers.
The consensus latency of transaction depends on how busy
a peer is, network latency, and available computing power.
Nevertheless, we can still identify each step in the life of atransaction, to determine the main sources of latency. We canwrite the total latency as:
T total  tin  tint ernal  tout
(1)
Where tint ernal is the interval latency: the latency contributedby ETTF. tin and tout depends on network bandwidth. Suppose
having a well situation network bandwidth, tin ~ 0 and tout ~ 0 ,T total ~ t int erval . In ETTF, tint ernal depends on the time of PKB, not
EGB. According to our experiments, the time of PKB can besmall as peer can. It’s not important that PKB must validate thelegality of itself. All the groups can communicate internally inparallel and transaction don’t propagate across the Trusted
Trading network.

Algorithm 2: ECA
1. For each time epochs2.
For each gp
3.
Sort PKB by timestamp4.
By using SSSS to get hash number
5.
Check the legality of hash number
6.
Generate a new EGB
7.
Send EGB to all vp
8.
End For9.
For each vp
10.
Intergrade all EGB into a single EGB
11.
Signature and send to sp set
12.
End For13.
For sp set
14.
Using EPA to get a complete EGB
15.
If status(EGB)  valid
16.
Send the complete EGB to remained peers
17.
Generate the next vp and gp set
18.
End If19.
End For20.
For vp set
21.
Generate the next sp set
22.
End For
23. End For
V. EXPERIMENTS
In this section, we evaluate ETTF with 1000-nodeexperiments on an emulated network. We implemented all
ETTF elements that are significant for a performance analysis
in the absence of an adversary.
We evaluate ETTF and compare it with Bitcoin in threesets of experiments, varying the throughput of transaction,
block frequency and block size. ETTF qualitatively
outperforms Bitcoin, as it suffers no such deterioration, while
enjoying superior performance in almost all metrics across theentire measured range.
A. Throughput of transaction
First, we run experiments targeted at improving thethroughput of transactions. Due to Bitcoin adopts the proof ofwork consensus mechanism, the throughput of transactions is
so lower. In ETTF, it’s easy to get higher throughput without
sacrificing the security, by adopting collaborative mechanism.
We feed as many transactions as possible to the network andrun several Bitcoin blockchain of minutes. Figure 4 shows thatup to million transactions per millisecond in ETTN, rather thana few transactions in Bitcoin network. For example, the TX
rate are 3 and 7 TXs per second at epoch times in Bitcoin, andwe observe a significantly higher TX in ETTF.
B. Block Frequency
Second, we run experiments targeted at improving theconsensus delay. For Bitcoin, we vary the frequency of block
generation by reducing the proof-of-work difficulty. But if we
do this, it’s difficult to guarantee a higher security of© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

72

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

number of transactions in ETTN

transaction. For ETTF, we vary the frequency of micro block
generation. For each frequency, we choose the block size suchthat the payload throughput is identical to that of Bitcoin’s
operational system.
6

10

ETTF
Bitcoin

5

10

4

10

3

10

VI. CONCLUSION

2

10

1

10

0

10

0

50

100

150

200

250

time/minutes

Figure 4: the number of transactions per millisecond
3

construction time of block /second

10

2

10

ETTF
Bitcoin

1

10

0

20

40

60

80

100

120

140

160

180

EGB

Figure 5: the EGB constructed time5
4.5

EGB size/KB

miner to learn of a new block is longer, and so the chance forforks increases. But, in ETTF, it is not suitable that every peer
entity have the permission to generate key block. Every PMB
just contains a set of hash value of transactions. In ETTF, eachuser just can commit one transaction in one time. And, thecontent of every EGB just contains a set of hash value, and thesize of global block is very small, just a few KB as shown in
Figure 6.

4
3.5
3
2.5
2
1.5
1

50

100

150

200

250

time/minutes

Figure 6: the EGB size.

In ETTF, every peer generates their PMB and PKB. Wheneach peer finishes to generate one PKB, the peer will send the
PKB with itself signature to all gp and vp . The gp generates
EGB by making use of proof-of-work mechanism. So, in eachpeer, this is no inner consensus delay of the PMB, thefrequency of PKB as faster as peer entity can. General
speaking, in TTEN, all gp collaboratively generate each EGB,
and all vp integrate EGB by voting mechanism. In this situation,
after peer sends peer block, the peer can continue processing
itself business. We feed as many blocks as possible to the
network and run several Bitcoin blockchain of minutes. Figure
5 indicates that an EGB just need one minutes to construct
rather than ten minutes, while Bitcoin-derived blockchain needten minutes to generate a block.
C. Block SizeIn our ETTF, for each block, such as PMB, PKB and EGB,
block size dynamic change over time. In Bitcoin, a larger block
may take long time to verify and propagate. Therefore,
although block frequency is constant, the time it takes for a

In the past years, the problems of lack of credibility havebeen become urgent issues in E-commerce. Existing studies
focus on cryptography. With emergence of blockchain, it’s
possible to construct a higher credibility environment in Ecommerce. Although, Bitcoin-derived blockchains don’t havebetter performance on other areas, such as E-commerce, due tothe inherent scalability barriers. To facilitate instant andmassive transactions in E-commerce without sacrificing
credibility, this paper develops a trusted trading framework in
E-commerce (ETTF). Based on an E-commerce blockchainarchitecture for peers and a robust strong consensus algorithm,
ETTF can keep unlimited size of transaction data in one block,support instant transactions, achieve higher throughput andlower consensus delay, and have no forks in blockchain.
To be clear, ETTF shows that it is possible to improve thescalability of blockchain protocols to the point where the
network diameter limits consensus latency and the individual
node processing power is the bottleneck of throughput. And
ECA combines the rotation mechanism and voting mechanism,has better performance on E-commerce, which can construct acompletely decentralized autonomous system, achieve nobranches in blockchain, slow the communication costs inTrusted Trading network and can construct a trusted, public Ecommerce.
In the future, we will do more work about the consensus
algorithm, the storage mechanism of transactions and blocks,
the no-sql retrieval mechanism of transactions, and a newmechanism to guarantee the data consistent.
ACKNOWLEDGMENT
This work is supported by National Natural Science
Foundation of China under Grant No.61572295;Innovation
Method Fund of China No.2015IM010200; the Natural Science
Foundation of Shandong Province of China under Grant
No.ZR2014FM031; Science and Technology Development
Plan Project of Shandong Province No. 2014GGX101047,
No.2015GGX101015; Shandong Province Independent
Innovation Major Special Project No.2015ZDJQ01002,
NO.2015ZDXX0201B03.
REFERENCES[1]. M.Trent, M.Rodolphe, M.Andreas, D.Dimitri, M.Troy, M. Greg,
H.Ryan, B.S, Y.Alberto. "BigchainDB: A Scalable Blockchain
Database (DRAFT).",2016.
[2]. S.Melanie. "Blockchain Thinking: The Brain as a DAC(Decentralized Autonomous Organization." Texas Bitcoin
Conference, pp.27-29, 2015.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

73

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[3]. Blockchian.info, “Number of Transactions”, [online]. Available:
https://blockchain.info/charts/n-transactions.
[4]. A.Back, M.Corallo, L.Dashjr, M.Friedenbach, G.Maxwell,
A.Miller, A.Poelstra, J.Tim´on, P.Wuille. 2014. Enabling
blockchain innovations with pegged sidechains. Technical
Report. http://www.blockstream.com/sidechains.pdf.
[5]. Colored Coins Project. Colored Coins. http://coloredcoins.org/,
retrieved Sep. 2015.
[6]. Kendler, EH.Alison, A.Zohar, S.Goldberg. "Eclipse Attacks on
Bitcoin’s Peer-to-Peer Network." 24th USENIX Security
Symposium (USENIX Security 15). 2015.
[7]. A. Miller, J. LaViola. Anonymous byzantine consensus frommoderately-hard puzzles: A model for bitcoin. University ofCentral Florida. Tech Report, CS-TR-14-01, April 2014.
[8]. V. King, J. Saia, “Breaking the O(n¡sup¿2¡/sup¿) bit barrier:scalable byzantine agreement with an adaptive adversary,”
Journal of the ACM (JACM), vol. 58, no. 4, pp.18:1–18:24,
2011.
[9]. D. Dolev, R. Strong, “Authenticated algorithms for Byzantine
agreement,” SIAM Journal on Computing, vol. 12, no. 4, pp.
656–666, 1983.
[10]. H. Attiya, K. Censor-Hillel, “Lower bounds for randomized
consensus under a weak adversary,” SIAM Journal on
Computing, vol. 39, no. 8, pp. 3885–3904, 2010.

[11]. R. Tempo and H. Ishii, “Las Vegas randomized algorithms in
distributed consensus problems,” European Control Conference,
vol. 13, no. 2-3, pp. 189–203, Jun. 2007.
[12]. Heilman, Ethan, Foteini Baldimtsi, and Sharon Goldberg.
"Blindly Signed Contracts: Anonymous On-Blockchain and
Off-Blockchain Bitcoin Transactions." Fbim Transactions,pp37-48, 2015[13]. E.Sasson, A.Chiesa, C.Garman, M.Green, I.Miers, E.Tromer,
M.Virza. “Zerocash: Decentralized anonymous payments frombitcoin”. IEEE Security and Privacy (SP), pp.459–474, 2014.
[14]. I.Miers, C.Garman, M.Green, and A.Rubin. “Zerocoin:
Anonymous distributed e-cash from bitcoin”. IEEE Security
and Privacy (SP), pp.397–411, 2013.
[15]. A.Saxena, J.Misra, A.Dhar. “Increasing anonymity in bitcoin.”
Financial Cryptography and Data Security, pp.122–139.
Springer, 2014.
[16]. Duffield, E., Hagan: “Darkcoin: Peer-to-peer crypto currency
with anonymous blockchain transactions and an improved
proof-of-worksystem”.
Technical
report(March2014),http://www.darkcoin.io/downloads/DarkcoinWhitepaper.
pdf.
[17]. He, Hong-Jie, Jia-Shu Zhang, Heng-Ming Tai. "Block-chain
based fragile watermarking scheme with superior localization."
Information Hiding. Springer Berlin Heidelberg, pp.1–6, 2006.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

74

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Evaluating Complex Task through Crowdsourcing:Multiple Views Approach
Lingyu Lyu

Mehmed Kantardzic

Department of CECS
University of LouisvilleLouisville, Kentucky, USA
l0lv0002@louisville.edu

Department of CECS
University of LouisvilleLouisville, Kentucky, USA
mehmed.kantardzic@louisville.edu

Abstract—With the popularity of massive open online courses(MOOCs), grading through crowdsourcing has become aprevalent approach towards large scale classes. However, forgetting grades for complex tasks, which require specific skills andefforts for grading, crowdsourcing encounters a restriction of
insufficient knowledge of the workers from the crowd. Due toknowledge limitation of the crowd graders, grading based onpartial perspectives becomes a big challenge for evaluatingcomplex tasks through crowdsourcing. Especially for those tasks
which not only need specific knowledge for grading, but also
should be graded as a whole instead of being decomposed into
smaller and simpler sub-tasks. We propose a framework forgrading complex tasks via multiple views, which are different
grading perspectives defined by experts for the task, to provide
uniformity. Aggregation algorithm based on graders’ variances
are used to combine the grades for each view. We also detect bias
patterns of the graders, and de-bias them regarding each view of
the task. Bias pattern determines how the behavior is biased
among graders, which is detected by a statistical technique. The
proposed approach is analyzed on a synthetic data set. We showthat our model gives more accurate results compared to thegrading approaches without different views and de-biasingalgorithm.
Keywords—complex task; crowdsourcing; view; bias pattern; debias; Vancouver algorithm

I. INTRODUCTION
Grades and comments from instructors are very important toevaluate the level of the students’ understanding and provide
guidance for future studying process. Traditional ways ofgrading by instructors or teaching assistants are becoming a big
challenge for large classes such as massive open online classes(MOOCs), which are distributed on platforms such as Udacity,
Coursera and EdX [1], and the grading processes are really time
consuming and tedious. As an example, there were around
100,000 students signed up for the online machine learning
course offered on the Coursera platform. To evaluate students’work, a scalable way for grading is required for such large scaleclasses. Due to the restriction of un-scalability of the traditionalway, grading through crowdsourcing has been developed as oneof the prevalent ways to solve the problem.
Since Amazon launched Mechanical Turk in 2005,
crowdsourcing has become a powerful mechanism forcompleting large scale tasks which involve human intelligencecomputing. The tasks in MTurk range from labeling images withkeywords to writing product descriptions. However, the types oftasks accomplished through MTurk have typically been limited
to those that are low in complexity, independent, and require
little time and cognitive effort to complete [15]. In contrast to

the typical tasks posted on MTurk, there are many tasks that arecomplex and need specialized skills to deal with in the realworld. Students’ essays or research papers are examples ofcomplex tasks which are different from traditional MTurk’stasks.
To explore the use of crowdsourcing in complex task grading
which is not decomposable into sub-tasks, consider for example
the usual experience of giving course evaluations. The
evaluations are given by students to reflect the success of theteachers’ teaching skills for the whole semester. It actually isalso a task of grading. Usually, every student in one class is
asked to give course evaluation at the end of the semester. It
includes questions such as: did the instructor explain concepts
correctly; did the instructor provide clear constructive feedback;was the course effectively organized, etc. After each question,
the evaluation scale is provided for the student to pick from. Thisevaluation process of instructors’ success in teaching the course
is a grading task through crowdsourcing, where students are used
as a crowd. In order to give the evaluation for the course, it isnecessary to provide grades considering the full semester,instead of decomposing the task into weekly or monthly based
evaluation. Another issue arises during the evaluating process isthat different students would grade based on their own
perspectives and experiences without any guidance information.
To ensure perspective uniformity, those instructive questions areprovided, which are also called views, for the students grade.
After getting grades for each view, an overall grade could beobtained by aggregating the view grades. Derived from this idea,
we propose the approach of crowdsourcing grading process
through different views for complex tasks such as evaluating
programming projects or research papers.
The idea makes use of different views, which are diverse
grading perspectives worked as instructive information toprovide uniformity among graders. These different views are
specified by experts before distributing the tasks to the graders
for evaluation. We describe the graders used in the complex task
evaluation as experienced domain grader, which means they
have some knowledge about the domain of the assignment, but
at the same time are different from experts in the field.
Insufficient knowledge of the graders may lead to the existenceof bias in the grading process regarding each view. By detecting
the bias behavior of the graders, it is possible to de-bias themstatistically, and significantly improve the quality of the grading
process.
There are several concepts used in this work, and they aredefined as follows:© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

75

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Experts – Those who have enough knowledge and ability to havebest understanding of the complex task, e.g. instructors, who
specified and assigned the complex assignment for students, areconsidered as experts.
Experienced domain grader – Those who have some knowledge
about the domain of the assignment, but at the same time aredifferent from experts that they are not as knowledgeable andexperienced in the field. For example, the students, who havetaken and successfully finished the course, could be counted as
an experienced domain grader for corresponding course
assignment.
Complex task – The task which cannot be simply decomposed
into smaller sub-tasks, but instead is able to define different
views for it. As an example, grading essays or computer science
projects can be viewed as a complex task.
Observed view grade – Observed view grade is the grade given
by for every view by each grader without any processing.
Consensus view grade – The grade we get for each view after
applying aggregating algorithms or some other processing
algorithms, which used to combine observed view grades fromdifferent graders for the same submission.
Overall grade – The grade we get after combing all the view
grades according to the instructions of the expert. It is the gradefor every submission instead of each view.
True grade – The grade which could reflects the true quality ofthe submission of the complex assignment. True grades are thegrades given by experts.
We present in this paper the framework for crowdsourcing a
grading process for complex tasks. Two main contributions of
our approach are: extending the Vancouver algorithm [2],proposed by L. de Alfaro, and are applied for different views ofcomplex tasks defined by an expert. Applying de-biasing
process to discover biased graders, and the biases are removed
from the final grades.
The paper is organized as follows: Section II gives theliterature review of current methods used for grading bycrowdsourcing. Section III describes the proposed Vancouverbased method with de-biasing. Section IV shows our experimentresults. Section V presents comparison of the out-come of thiswork with other methodologies.
II. PREVIOUS WORK
2.1 Literature Review
To relieve the grading burden of the experts in the traditional
way [20-22], especially with the popularity of open online
classes such as MOOCs, peer reviewing has been proposed as a
new way to address the grading tasks [1-2][23-25]. As stated in[24], “students are trained to be competent reviewers and arethen given the responsibility of providing their classmates withpersonalized feedback on expository writing assignments”. Mostof the peer evaluation systems proposed nowadays are similar tothis concept. In [1], three statistical models are developed for
estimating true grades in peer grading. The models put priordistributions over latent variables such as true scores, grader’sreliability and bias. Gibbs sampling [13] and expectation

maximization (EM) [6-12] is then used as approximate inference
approach to estimate the true score. A system called
CrowdGrader is developed in [2] to explore the use of peer
feedback in grading assignments. It introduced a Vancouveralgorithm (derived from [14]) which relies on a reputation
system to aggregate the peer reviewing grades. [23] applied
Bayesian data analysis to model a computer supported peer
review process in a legal class. In [24], a web-based peer review
program called Calibrated Peer Review (CPR) system is
introduced to improve student learning. Peer reviewing
experiments have shown promise, but it may also cause
problems. As what Richard Smith proposed in his paper, “thepractice of peer review is based on faith in its effects, rather onfacts” [3]. Suppose all the students in one class try to give good
grades to each other, then with the absence of the true grades
provided by the instructor or teaching assistant used forcomparison, the consensus grades would be the submissions’grades. However, these grades actually cannot give correct
estimation of the true grades.
Many complex tasks, including complex grading tasks, aredifficult to be done by crowdsourcing through MTurk due to its
typical limitations. In [15] a framework was proposed to solve
the problem by breaking down the complex task into a sequenceof subtasks. Then subtasks are done through MTurk. This idea
inspired by MapReduce [16] systems consists of three steps:
partition, map and reduce. [17-19] presented the similar
approaches to address the problem encountered while
crowdsourcing the solutions to complex tasks. In [17],crowdsourcing is used as a novel approach to grade the maththrough Internet, by splitting the expert task to non-expert.
Divide-and-conquer algorithm, which depicts the ‘decompose,
solve and recompose’ structure, is proposed in [18] to solve
general problem via crowdsourcing. In [19], a system called
PlateMate is introduced to crowdsource nutritional analysis fromphotographs via MTurk. Foods in each image are identified andmeasured separately. Although crowdsourcing a complex task
by decomposing it into subtasks offers tradeoff between cost andperformance, the problem arises when the complex task is undecomposable. The framework is no longer available when thecomplex task should be solved as a whole.
Mostly, bias is unavoidable when collecting the data fromcrowdsourcing. Bias may be caused by personal preference,systematic misleading, and lack of interest [26]. Many
researches have included biases analysis in crowdsourcing
models to improve the approximation accuracy [26-28]. In [26],a Bayesian model, named as Bayesian Bias Mitigation for
Crowdsourcing (BBMC), is proposed to capture the sources ofbias. Authors in [27] introduced and evaluated probabilistic
models that can detect and correct task-dependent biases incrowdsourcing automatically. In [1], statistical models havebeen presented to infer the graders’ biases in peer reviewing
process.
2.2 Vancouver Algorithm
The Vancouver algorithm [2] measures each student’s
grading accuracy, by comparing the grades assigned by thestudent with the grades given to the same submission by otherstudents in the crowd. It gives more weight to the input ofstudents with higher measured accuracy.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

76Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)(a) Positive Bias Pattern(b) Negative Bias PatternFig. 1 Bias Patterns

The Vancouver algorithm is based on the fact of varianceminimization principle. This principle said that we could weighthe input provided by student i in proportion to 1/vi, where vi isgrader i’s variance. The algorithm proceeds in iterative fashion,using consensus grades to estimate the grading variance of eachuser, and using the information on user variance to compute
more precise consensus grades. The approach may be formalized
as: We denote by U the set of students, and by S the set of items
to be graded (the submissions). We let G = (T, E) be the graphencoding the review relation, where T = S ∪ U and S ∩ U = ∅,and where (i, j) ∈ E iff j reviewed i; for (i, j) ∈ E, we let gij be
the grade assigned by j to i. We denote by ∂t the 1-neighborhoodof a node t ∈ T.
The algorithm proceeds by updating estimates vj of thevariance of user j ∈ U, and estimates ci of the consensus gradeof item i ∈ S, and estimates vi of the variance with which ci is
known. To produce these estimates, the algorithm relies on
messages m = (l, x, v) consisting of a source l ∈ S ∪ U, of a value
x, and of a variance v. We denote by Mi, Mj the lists of messages
associated with item i ∈ T or user j ∈ U. Given a set M of
messages, we indicate by:∑(𝑙,𝑥,𝑣)∈𝑀 𝑥/𝑣

𝐸(𝑀) = ∑(1)(𝑙,𝑥,𝑣)∈𝑀 1/𝑣

1𝑣𝑎𝑟(𝑀) = (∑(𝑙,𝑥,𝑣)∈𝑀 )−1𝑣(2)

The best estimator 𝐸(M) we can obtain from M, and its variance𝑣𝑎𝑟(M).
III. CROWDSOURCING THE GRADING PROCESS OF COMPLEXTASKS THROUGH DIFFERENT VIEWSThe Vancouver algorithm is extended in order to aggregate
the observed view grades into consensus view grades. Section
3.1 describes the modified approach. Bias is analyzed for gradersregarding each view, for the complex task here. Section 3.2
depicts how the bias is evaluated through bias patterns, and theway to de-bias the graders. Section 3.3 gives the details of thecrowdsourcing process.
3.1 Modified Vancouver Algorithm
After defining multiple views for the grading tasks, eachview will be considered as one separate aggregation task, and themodified Vancouver algorithm is applied to iteratively estimate
the consensus grades for views. The extension of Vancouveralgorithm is still based on the same principle of Vancouveralgorithm, the main difference is that multiple views are taken
into account.
The details of the modified Vancouver approach can be seenin Algorithm 1. S is the set of submissions need to be graded,
and U is the set of graders. 𝑀𝑖 , 𝑀𝑗 are the lists of messages
associated with submission 𝑖 and grader 𝑗. The set of views andobserved view grades are used as the input for the algorithm, as
well as the review graph. Instead of propagating only one
variance as in basic Vancouver algorithm, different variances are
considered for multiple views of each grader in the extended
approach. After getting consensus view grades, it is required to
combine the view grades into overall grade for each submission.
The combining methods depends on the instructions of theexperts: One way is just simply add the view grades up to get theoverall consensus grade for the submission. Another way could
be given the weights of each view of the assignment from theexpert, aggregating the view grades in proportion to theirweights. The extended Vancouver algorithm incorporates theconsensus view grades combining steps as in lines 40-45 in
Algorithm 1. Here, we assume each view is given a weightassociated to it, thus the consensus grade for each submission
can be obtained by adding up view grades times their weight.
3.2 De-biasing
Bias patterns may be identified by analyzing the graders’fluctuation around true grade, for each view of the complex task.
Bias is then reduced from each view grade for the biased graderswith detected patterns.
The bias pattern is defined as: for all the submissions gradedby the grader g, if most of his/her grades for view v is
consistently higher (or lower) than the true grades of the view,
then we say the g has a bias pattern on v for the assignment.
There are two different types of bias pattern: a) most of thegrader’s scores of the view are higher than the true score, we callthis pattern as “positive bias pattern”, and b) graders’ scores arelower than the true score, which we call “negative bias pattern”.
Figure 1 gives an example for each of these two bias patterns.
X-axis represents different submissions graded by the grade (forone view) and y-axis is the score scale. For the specific grader,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

77

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)_______________________________________________________________
Algorithm 1 Modified Vancouver Algorithm
Input: A review graph G = ((S ∪ U), E) such that |∂t| > 1 for all t ∈ S ∪ U, alongwith {gij[view]}(i,j)∈E, and number of iterations K > 0, views set V and v𝑖𝑒𝑤 ∈𝑉.
Output: Estimates 𝑞̂𝑖 [𝑣] for i ∈ S.
1: {Initialization}
2: for all i ∈ S do
3: for all view ∈ V do
4:
Mi[𝑣𝑖𝑒𝑤] := {(j, 𝑔𝑖𝑗 [𝑣𝑖𝑒𝑤] , 1) | (i, j) ∈ E}.
5: end for
6: end for
7: for iteration k = 1, 2, . . . , K do
8: {Propagation from submissions}
9: for all j ∈ U do
10:
for all 𝑣𝑖𝑒𝑤 ∈ V do
11:
Mj[𝑣𝑖𝑒𝑤] := ∅12:
end for
13: end for
14: for all i ∈ S do
15:
for all j ∈ ∂i do
16:
for all 𝑣𝑖𝑒𝑤 ∈ V do
17:
Let M−j[𝑣𝑖𝑒𝑤] = {(𝑗 , , 𝑥, 𝑣) ∈ Mi[𝑣𝑖𝑒𝑤] | 𝑗 , ≠ 𝑗} in
Mj[𝑣𝑖𝑒𝑤] := Mj[𝑣𝑖𝑒𝑤] ∪ (i, E(M−j[𝑣𝑖𝑒𝑤] ), var(M−j[𝑣𝑖𝑒𝑤] ))18:
end for19:
end for20: {Propagation from graders}
21: for all i ∈ S do
22:
22:
for all 𝑣𝑖𝑒𝑤 ∈ V do
23:
Mi[𝑣𝑖𝑒𝑤] := ∅24:
end for
25: end for
26: for all j ∈ U do
27:
for all i ∈ ∂j do
28:
for all 𝑣𝑖𝑒𝑤 ∈ V do29:
Let M−i[𝑣𝑖𝑒𝑤] = {𝑖 , , (𝑥 − 𝑔𝑖 , 𝑗 )2 , 𝑣) | (𝑖 , , 𝑥, 𝑣) ∈ Mj, 𝑖 , ≠ 𝑖 }in Mi := Mi ∪ (j, 𝑔𝑖𝑗 , E(M−j ))30:
end for31:
end for32: end for
33: end for
34: {Final Aggregation}
35: for all i ∈ S do
36: for all 𝑣𝑖𝑒𝑤 ∈ V do37:𝑞̂𝑖 [𝑣𝑖𝑒𝑤] := E(Mi)38: end for
39: end for
40: for all 𝑖 ∈ 𝑆 do
41:𝑞̂𝑖 : = 042:
for all 𝑣𝑖𝑒𝑤 ∈ 𝑉 do43:𝑞̂𝑖 : = 𝑞̂𝑖 + 𝑞̂𝑖 [𝑣𝑖𝑒𝑤] × 𝑤𝑒𝑖𝑔ℎ𝑡[𝑣𝑖𝑒𝑤]44:
end for45: end for

his/her grades given by ‘+’ sign are consistently higher (orlower) than true grades represented by small red circles.
We propose the approach to detect and measure bias pattern
as the percentage of values that lie within a width of two standard
deviations around the mean of the differences of the true grades
and observed grades. Below we present the detailed way how werecognize the bias pattern.
Suppose there are n different submissions graded by graderg, for each of the views of the assignment v. True grades are
given as T, where ti ∈T is the true grade of submission i for thisview. O is the set of observed grades from g, where oi ∈ O is thegrade given by g to ith submission for v. Denote diff(ti, oi) as thedifference for each pair of ti and oi. Calculate:1

𝜇 = ∑𝑛𝑖=1 𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )(3)

𝜎 2 = 𝐸((𝑂 − 𝜇)2 )(4)(𝜇 − 2𝜎, 𝜇 + 2𝜎)(5)𝑛Interval (5) represents the band around the mean of thedifferences of the true grades and observed grades with a widthof two standard deviations. In statistics, we could make sureapproximately 95% difference values would fall into the range
as stated in (5). That means, if 𝜇 − 2𝜎 as calculated fromequation (3) and (4) is greater than 0, then we could say 95% of
the difference values (𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )) are positive. In other words,
95% of the grader’s grades for view v are higher than the true
grades. We state that this grader has a positive bias pattern with
95% confidence for view v. Similarly, if the result of 𝜇 + 2𝜎 isnegative, we have 95% confidence that the grader has a negative
bias pattern for view v. Grades from the graders with negative
bias pattern or positive bias pattern are then de-biased to give a
better performance before applying aggregating algorithm.
The way to de-bias the observed view grades provided by thebiased graders varies on different occasions. The approaches forde-biasing might be subtracting: a) min(𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )); b)max(𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )); c) median of (𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )) or d) average of(𝑑𝑖𝑓𝑓(𝑡𝑖 , 𝑜𝑖 )) from the observed view grades. The best method
would be determined based on experimental heuristics.
3.3 Grading Complex Task through Crowdsourcing
This research develops a crowdsourcing methodology forgrading complex tasks which will result in consensus grades as
close as possible to expert grades. By defining different views
for the complex task by expert, the approach provides uniformity
among the graders crowd. The framework of the approach ispresented in Fig. 2 and explained as follows:
1) Define different views for the complex task: Different
views are first defined for the complex task by experts. As an
example, while grading a research paper, the instructor may
define views for the assignment such as a) comprehensiveness;b) enough quotes; c) examples, and inferences; d) technicalstrength; e) data sets selection and f) presentation, etc. It is also
required that an expert gives the scale for each view of theassignment such as 0 to 5 for background information, 0 to 10
to technical strength, and 0 to 3 to data sets selection, etc.
2) Distribute the submissions to graders: each submission
should be reviewed and graded by several graders, and each
grader is able to review and grade several submissions. The
experienced domain graders are assumed in the proposed
approach. In addition, we assume that each grader must review
a minimum number of submissions to ensure that eachsubmission would have enough reviews. The problem mayoccur in real world applications: if there are graders who do not
do their reviews, which will lead to insufficient number ofreviews for some submissions. An online algorithm is proposedin [2] to solve the defect. The algorithm dynamically estimates
the probability that each review task will be completed, on thebasis of previous history, and assigns review tasks to achieve© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

78

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

algorithm in step 4 is applied again on the de-biased data set toget the consensus grades for each view of the submission.
6) Final overall grades for each submission: After applying
the aggregation algorithm, each view of every submission will
finally get its own consensus grade. The overall grade for eachsubmission is obtained through summation of the view grades.
IV. EXPERIMENTAL ANALYSIS
4.1 Data Set
The method is evaluated on synthetic data. In order tocompare the proposed approach with the methodology appliedin [2] called Vancouver, we used the same synthetic data set. 50
graders and 50 submissions were considered, where each grader
was reviewing 6 submissions. To give better understanding of
the process, we define 2 different views for the assignment. For
each view, we assumed: the true quality qi of each item i has
normal distribution with standard deviation 1. Each grader j hada characteristic variance vj, and we let the grade qij assigned by j
to i be equal to qi + ∆ij , where qi is the true quality, and ∆ij has
normal distribution with mean 0 and variance vj . We assumed
that the variances {vj}j∈U of the graders were distributed
according to a Gamma distribution with scale 0.4, and shapefactors k = 2, 3.
4.2 Models
Three different models are analyzed, and they are denoted
as: AVG, DM1 and DM2.
Fig. 2 Framework of Grading Complex Task through Crowdsourcing (Assuming
two graders are employed, and two views are defined for the task. Sub 1, Sub2
represent Submission1 and Submission 2. V1, V2 represent View1 and View2.
g11a is observed grade for view1, submission1 with grader a; g’11a is the de-biased
grade for view1, submission1 from grader a; ĝ11 is consensus grade for view1,
submission1; ĝ1 is the overall grade for submission 1. De-biasing process is done
based on the view grades collected from each grader. Ga and Gb represent Gradera and Grader b.)

uniform coverage.
3) Collect grades for all submissions: After reviewing eachsubmission, graders need to evaluate and give grades for eachview of the submission.
4) Aggregate the grades: The modified Vancouver algorithm
which implements a reputation system, which consists ofgraders’ variance and biases, for students is used in this researchto aggregate the grades for each view from step 3. This algorithm
ensures that higher accuracy leads to higher reputation, andtherefore to higher influence on the consensus grades [2].
Each view is considered as one separate aggregation task, andthe Vancouver algorithm is applied to iteratively estimate theconsensus grade for these views.
5) De-bias each graders’ grade for different views: Bias
patterns are first identified for each view of every grader using
the approach presented in section 3.1. True grades provided byexperts are used to compare with the grades given by thegraders. The bias patterns are detected from comparisons. Biasis reduced from the view grades afterwards by subtracting
min(diff(ti, oi)) from the observed grades for a given biased
grader. After de-biasing the view grades, the aggregation

AVG model represents the average model. It just simply
averages the grades for each submission received to get theconsensus grade. It acts as a baseline model.
DM1 is the model with different views defined by an expert,
but without the step of bias pattern recognition. Thus, there is no
de-biasing applied for the observed grades. DM1 is basicallyVancouver method in [2], but with extension to different views.
DM2 model uses both different views of assignments andde-biasing the graders. Different views are defined for thecomplex assignment before getting the grades from graders.
After getting the observed grades, bias patterns are recognized,
de-biasing is applied, and finally overall consensus grades are
obtained.
4.3 Evaluation Metrics
Our goal is to get consensus grades as close as possible to thetrue grades given by an expert. As a result, coefficientcorrelation (ρ), standard deviation (𝜎), and root mean squareerror (RMSE) are calculated between true grades and consensus
grades as the evaluation metrics. The metrics are calculated asfollows:
𝜌𝑌,𝑌̂ =𝐶𝑂𝑉(𝑌,𝑌̂ )

(6)

𝜎𝑌 𝜎𝑌
̂

𝐶𝑂𝑉(𝑌, 𝑌̂) = 𝐸((𝑌 − 𝜇𝑌 )(𝑌̂ − 𝜇𝑌̂ ))(7)

where 𝜌𝑌,𝑌̂ is the coefficient correlation of random variables
𝑌(true grades) and 𝑌̂ (consensus grades), 𝜎𝑌 , 𝜎𝑌̂ are the standard
deviation of 𝑌 and 𝑌̂, 𝜇𝑌 , 𝜇𝑌̂ are the mean of 𝑌 and 𝑌̂, 𝐶𝑂𝑉(𝑌, 𝑌̂)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

79Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 3 Standard Deviation, Coefficient Correlation and RMSE for view1 of Synthetic Data for All Three Models for 20 Runs (k=2)

Fig. 4 Standard Deviation, Coefficient Correlation and RMSE for view1 of Synthetic Data for DM1 and DM2 for 20 Runs (k=2)

is the covariance of 𝑌 and 𝑌̂.
∑𝑛 (𝑦̂𝑡 −𝑦)2

𝑅𝑀𝑆𝐸 = √ 𝑡=1𝑛(8)

where 𝑦̂𝑡 is the estimated value for true grade and 𝑦 is the true
value.
4.4 Experimental Results
The synthetic data is simulated through 100 runs and thecoefficient correlation, standard deviation and RMSE are thenreported as the average over these 100 runs. Figure 3 shows theevaluation metrics calculated for the view1 of first 20 simulation
runs, with shape factor k = 2 of the Gamma distribution, for thethree models. To present the difference between DM1 and DM2,
we show a more detailed figure in Fig. 4. The result of view2 isquite similar to view1. The results are summarized in Table 1.
Specifically, the overall outcome is calculated from aggregation
the 100 runs instead of averaging the results from the final
views’ outcome.
In order to better understand the influence of the bias patterns
to the results, we run the following experiment: Increasing thepercentage of biased graders (graders who have bias patterns)
for each of the view. We increase the number of biased graders
for view 1 from 9 to 24, and view 2 from 12 to 28. DM1 andDM2 model are applied to get the consensus grades. We thencalculate the metrics for evaluation. Table 2 shows the results.
4.5 Detailed Experimental Results Analysis and Discussion
As the results show in Table 1, all three metrics we selected
are improved for each view after applying DM1 compared tobaseline – AVG model. Thus, the overall results of DM1 model
improved 20.3% in contrast to AVG model for correlation whenk=2. After de-biasing, which means applying DM2, there isfurther improvement. As an example, if standards deviation has
70% improvement comparing DM1 with AVG, and 10%
improvement comparing DM2 with DM1, then 87%
improvement is given comparing DM2 with AVG model. In

conclusion, DM2 model gave the best performance for all threeevaluation metrics comparing with AVG baseline model and
DM1.
Figure 5 and 6 present the percentage of improvement of theresults comparing the three different models, which is calculated
from Table 1. Figure 5(a) and 6(a) show the enhancement
obtained for all three metrics from comparing DM1 model withbaseline AVG model, for shape factor equals to 2 and 3
respectively. Figure 5(b) and 6(b) give percentage ofimprovement of the accuracy after comparing DM2 with DM1(for k = 2, 3). It could be seen clearly in figure 5(a) and 6(a) that,no matter for each view or overall grades, significant
improvement can be obtained by applying DM1 in contrast with
AVG model. However, further improvement on the accuracy isable to achieve by using DM2 comparing with DM1. Thus, DM2
gives the best performance.
By defining different views and taking into account graders’bias pattern for each view, we could provide significant gains inaccuracy. To prove the importance of de-biasing the graders, we
increase the number of biased graders per view in our data set.
Table 2 shows the results of three selected metrics results andpercentage of improvement comparing DM2 (with de-biasing)with DM1 (without de-biasing). As results presented, the morebiased graders, the better improvement we could get by using
the DM2 model. The reason is that more graders with a bias
pattern means less of them make random error, and as a result,more accurate grades could be obtained after de-biasing thegraders.
We compared our method with the Vancouver algorithm in[2] (Alfaro and Shavlovsky, 2014). The Vancouver algorithm isessentially DM1 model, except for that no different views aredefined. Thus, the overall results in Table 1 is the approximationof the outcome of Vancouver algorithm. The improvement of
metrics of overall grades in Figure 5(b) and 6(b) presents theenhancement on accuracy comparing DM2 model with
Vancouver algorithm. Our results show that DM2 yields the© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

80

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

DM1 vs. AVG

DM2 vs. DM1

80

20

60

15

40

10

20

5

0

0
ρ(%)

σ(%)
View1

RMSE(%)

View2

ρ(%)

Overall

σ(%)
View1

View2

RMSE(%)OverallFig. 5(a) Improvement of Results Comparing DM1 with AVG Model (k=2)

Fig. 5(b) Improvement of Results Comparing DM2 with DM1 Model (k=2)

DM1 vs. AVG

DM2 vs. DM1

80

6

60

4

40
2

20
0

0
ρ(%)

σ(%)
View1

RMSE(%)

View2

ρ(%)

Overall

View1

Fig. 6(a) Improvement of Results Comparing DM1 with AVG Model (k=3)
Table 1 Results for all the models
ρ

AVG
DM1
DM2

Overall

RMSE

k=3

k=2

k=3

k=2

k=3

View1

0.812

0.619

0.690

1.255

0.697

1.269

View2

0.831

0.629

0.675

1.226

0.683

1.238

View1

0.977

0.913

0.201

0.425

0.207

0.431

View2

0.979

0.912

0.198

0.431

0.204

0.438

View1

0.982

0.920

0.173

0.413

0.179

0.410

View2

0.985

0.918

0.167

0.418

0.173

0.426

AVG

0.807

0.619

0.986

1.801

0.997

1.820

DM1

0.976

0.911

0.293

0.612

0.301

0.622

DM2

0.982

0.916

0.250

0.594

0.266

0.605

Table 2 Results for DM1 and DM2 after increasing the bias percentage
View1

ρ

σ

RMSE

View2

RMSE(%)OverallFig. 6(b) Improvement of Results Comparing DM2 with DM1 Model (k=3)

better results.

σk=2

Num. of biased graders

σ(%)

View2

9

24

12

28

DM1

0.984

0.968

0.977

0.971

DM2

0.988

0.99

0.982

0.995

Impr.(%)

0.40%

2.27%

0.30%

2.47%

DM1

0.177

0.223

0.225

0.246

DM2

0.149

0.137

0.193

0.0997

Impr.(%)

15.82%

38.57%

14.22%

59.47%

DM1

0.179

0.438

0.225

0.451

DM2

0.153

0.154

0.197

0.1

Impr.(%)

14.53%

64.84%

12.44%

77.83%In research [1], by developing the algorithm for estimating
the true grades through statistical models, the authors were able
to reduce the RMSE error on their prediction of ground truth by
31% to 33% comparing to baseline model. However, ourexperimental results show that we are able to reduce RMSE by
67% from 1.820 to 0.605 on overall grades by applying the
proposed framework.
V. CONCLUSIONS AND FUTURE WORKSIn this paper, we presented a novel framework for grading
un-decomposable, complex tasks through crowdsourcing. The
key innovations include: 1) Extending the Vancouver algorithm
to multiple views. 2) De-bias the graders for each view of thetask. By defining multiple views for the complex task,
uniformity is ensured among the graders. Bias pattern is thendetected for each view of the grader, and then we de-bias themin order to improve the grading accuracy. Experimental results
indicate that our model DM2 outperform the baseline AVG
model and DM1 which doesn’t include de-bias process.
To further justify and improve the proposed model, realworld complex problems will be formulated, collected, andgraded by experienced domain graders. Grading tasks in thiswork is performed in batch mode. However, a time domain
model might reveal the relevance and influences between
previous and future grading processes. For example, graders’bias pattern might be reused if some of the graders are persistentfor different grading tasks. In addition, the expert background of
the graders could also be taken into account while weighting theobserved grades for consensus grades. For instance, while© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

81Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

incorporating computer science students as graders for grading
Java programming assignments, graders who have experienceon Java coding are probably more reliable than those who do not.
Finally, instead of randomly choosing graders for tasks in thiswork, we plan to come up with algorithms to take into accountthe selection of graders with higher accuracy and reliability, as
well as optimization of the number of graders for specificcomplex tasks.

REFERENCES[1]

C. Piech, J. Huang, Z. Chen, C. Do, A. Ng, D. Koller. Tuned models ofpeer assessment in MOOCs. ArXiv preprint arXiv:1307-2579, 2013.
[2] L. Alfaro, M. Shavlovsky. CrowdGrader: A tool for crowd-sourcing theevaluation of homework assignments. Proceedings of the 45th ACM
technical symposium on Computer science education: 415-420, 2014.
[3] S. Smith. Peer review: A flawed process at the heart of the science andjournals. J R Soc Med. Apr; 99(4): 178–182, 2006.
[4] N. S. Foundation. Dear colleague letter: Information to principal
investigators (pis) planning to submit proposals to the sensors and sensing
systems (sss) program October 1, 2013.
[5] M. Merrifield and D. Saari. Telescope time without tears: A distributed
approach to peer review. Astronomy & Geophysics, 50(4):4–16, 2009.
[6] A. Dawid, A. Skene. Maximum likelihood estimation of observer errorrates using the EM algorithm. Applied Statistics: 20–28, 1979.
[7] A. Dempster, N. Laird, D. Rubin. Maximum likelihood from incomplete
data via the em algorithm. Journal of the Royal Statistical Society. Series
B (Methodological): 1–38, 1977.
[8] P. Smyth, U. Fayyad, M. Burl, P. Perona, P. Baldi. Inferring ground truth
from subjective labelling of venus images. In G. Tesauro, D. S. Touretzky,
and T. K. Leen, editors, NIPS: 1085–1092. MIT Press, 1994.
[9] R. Jin, Z. Ghahramani. Learning with multiple labels. In Advances inneural information processing systems: 897–904, 2002.
[10] J. Whitehill, T.-F. Wu, J. Bergsma, J. Movellan, P. Ruvolo. Whose voteshould count more: Optimal integration of labels from labelers of
unknown expertise. In Advances in neural information processing
systems: 2035–2043, 2009.
[11] V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, L. Moy.
Learning from crowds. J. Mach. Learn. Res., 11:1297–1322, Aug. 2010.
[12] P. Welinder, S. Branson, S. Belongie, P. Perona. The multidimensional
wisdom of crowds. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R.
Zemel, and A. Culotta, editors, Advances in Neural Information
Processing Systems 23: 2424–2432, 2010.

[13] S. Geman, D. Geman. Stochastic relaxation, gibbs distributions, and thebayesian restoration of images. IEEE PAMI, (6):721–741, 1984.
[14] D. Karger, S. Oh, D. Shah. Iterative learning for reliable crowdsourcing
systems. In Proc. of the 25th Annual Conference on Neural Information
Processing Systems (NIPS), 2011.
[15] A. Kittur, B. Smus, S. Khamkar, R. E. Kraut. CrowdForge:
Crowdsourcing complex work. 24th annual ACM symposium on Userinterface software and technology: 43-52, 2011.
[16] J. Dean, S. Ghemawat. Map Reduce: Simplified data processing on large
clusters. Communications of the ACM, 51, 1: 107-114, 2008.
[17] N. VanHoundnos. Can the Internet grade math? Crowdsourcing acomplex scoring task and picking the optimal crowd size. Dietrich Collegeof Humanities and Social Sciences at Research Showcase @ CMU.
[18] H. Zhang, E. Horvitz, R. C. Miller, D. C. Parkes. Crowdsourcing generalcomputation. Proceedings of ACM CHI Conference on Human Factors,
Vancouver, B.C., Canada, 2011.
[19] J. Noronha, E. Hysen, H. Zhang, K.Z. Gajos. Platemate: crowdsourcing
nutritional analysis from food photographs. Proc. UIST ’11, 2011.
[20] Hanover Research. Effective grading practices in the middle school andhigh school environments.

http://docplayer.net/5893105-Effective-grading-practices-in-themiddle-school-and-high-school-environments.html[21] J. Terwilliger. Some Thoughts on Grading Systems and Grading
Practices. Teacher Training in Measurement and Assessment Skills: 6,
1993.
[22] J. Ellen. Grades as valid measures of academic achievement of classroom
learning. Clearinghouse: A Journal of Educational Strategies. Issues andIdeas, 78, 218–223, 2005.
[23] K. Ashley, I. Goldin. Toward ai-enhanced computer-supported peer
review in legal education. In 24th International Conference on Legal
Knowledge and Information Systems (JURIX), volume 235, 2011.
[24] A. A. Russell. Calibrated peer review - a writing and critical-thinking
instructional tool. Teaching Tips: Innovations in Undergraduate Science
Instruction, page 54, 2004.
[25] P. M. Sadler, E. Good. The impact of self-and peer-grading on student
learning. Educational assessment, 11(1):1–31, 2006.
[26] F., Wauthier, M. Jordan. Bayesian bias mitigation for crowdsourcing. In
In Proc. of NIPS, 2012.
[27] E. Kamar, A. Kapoor, E. Horvitz. Identifying and accounting for taskdependent bias in crowdsourcing. Third AAAI Conference on Human
Computation and Crowdsourcing, 2015.
[28] H. Zhuang, J. Young, Leveraging in-batch annotation bias forcrowdsourced active learning. Proc. ACM WSDM: 243–252. Shanghai,
China, 2015.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

82

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Familiarity Design in Exergames for Elderly
Hao Zhang† , Chunyan Miao‡§ and Di Wang‡† Interdisciplinary Graduate School (IGS), ‡ Joint NTU-UBC Research Centre of Excellence inActive Living for the Elderly (LILY) and § School of Computer Science and Engineering (SCSE)
Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798
Email: I150001@e.ntu.edu.sg, ascymiao@ntu.edu.sg, wangdi@ntu.edu.sg

Abstract—Rehabilitation exergames aim to help the elderly
recover from their deteriorated capabilities due to aging. Research suggests that exergames may effectively engage the elderlyin physical activities. However, new technologies and devices
normally used to develop the exergames may be over complex
and difficult for the elderly to use, mainly because the hardware
and software components involved in modern exergames appear
unfamiliar or even alien to the elderly mentally. Moreover,the elderly consider the investment to acquire the unfamiliar
technology is too high. Therefore, the study of familiarity issuesof the elderly is important and necessary to guide the interaction
design of new technologies so that they can be built upon theelderly’s prior knowledge of real world interactions to help theelderly to apply their existing skills to a new domain. In thispaper, we propose three dimensions of familiarity design andthe approach to design familiar exergames for the elderly. By
following this approach, we hope to increase the usability ofthe exergames that the elderly can naturally interact with. For
case study, we introduce the design principles followed when
implementing the Virtual Escape Exergame to illustrate theusage of our proposed familiarity design approach to develop
an exergame in a familiar virtual environment for the elderly.

I. I NTRODUCTION
The number of older persons has increased substantially inrecent years in most countries and regions. The growth of
the aging population is projected to accelerate in the coming
decades [1]. Senior citizens will encounter various healthrelated issues as they age, which will cause the reduction in
their motor and cognitive capabilities.
Rehabilitation exercises aim to help the elderly to recoverfrom their deteriorated capabilities and to become as independent as possible to alleviate both the societal and individual home’s burden [2]. In order to recover efficiently andeffectively, the elderly are advised to frequently take exercises
according to their conditions. However, the incentive for theelderly to take exercises by themselves is often not enough.
Traditional rehabilitation treatments are usually administrated
on the one-to-one basis, which makes the treatments difficult
to operate due to the high cost and the dependency on humanresources. In addition, it is time consuming for elderly to travelto the clinics and they often feel isolated and bored during the
traditional rehabilitation exercises [3]. As a consequence, theadherence rate to traditional rehabilitation exercises is relatively low. A study [4] reports that only 31% elderly complete
the prescribed exercises, which reduce the effectiveness of
traditional treatments.
Literature has shown that the advances in technology canmake the rehabilitation treatment more efficient and effective[5]. New technologies such as video games, augmented reality,and virtual reality can decrease the monotony of repeatedexercises and provide instant feedback of elderly’s motions,which is beneficial to both the quantity and quality of the rehabilitation [6]. Video games with motion detection devices, suchas Microsoft Kinect, can stimulate the elderly to take exercisesby the attractive interfaces and interesting tasks. Hence, elderly
can complete the exercises in the game environment withenjoyment. Exergames are beneficial to senior citizens andprotect them from illness by providing appropriate exercises.
In addition, elderly can play the rehabilitation exergames
at home and other places, such as community centres, attheir convenient time. Family and community members canplay together with the elderly to enhance their relationshipbetween each other. However, the rehabilitation exergames
design seldom considers the incentives for the elderly toplay the exergames. It is not easy to motivate the elderly tovoluntarily play the exergames unless the games are attractive
enough. One of the key reasons that why exergames are not
popular in the aging population is the lack of engagement [7].
New technology, such as exergames, is normally perceived
as unfamiliar by elderly with the feeling of anxiety or evenhostility [7]. Moreover, despite the elderly may be willing tolearn how to interact with the computer, they often find theinvestment of personal resources is too high.
In this paper, we delineate our approach on applying thetheory of familiarity on rehabilitation exergames design toengage the elderly in playing the exergames. According to
the cognitive psychologists’ theory [8], familiarity is “therelationship between an individual and something that individual has had considerable experience with. The experienceis sufficient to advance to the development of an internal modelof that something.” The feeling of familiarity is derived froman individual’s prior experience that stored in one’s memory.
Repetition and frequency are the critical antecedents to feelfamiliar with something [9]. The increase of familiarity willlead to the strengthening of connections among less connectedfeatures. The elderly may feel more comfortable and willingto improve their social and functional capabilities when theyencounter familiar environments such as objects, places, andsounds [8]. However, current technical products for the elderly
are either specially targeted to the deficiencies of the elderly
or derived from the general products to meet the particularneeds of the elderly, which both fail in terms of familiarity[7]. Currently, the cohort of the elderly have little exposure© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

83Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

to the new technologies such as computers compared to
the younger generation [10]. Most of the designed softwareand hardware fail to consider the cultural background andexperiences of the elderly, which leads to the unfamiliarity
relationship between the technologies and the elderly. Our
research aims to guide the development of exergames forthe elderly with familiar interfaces and interactions that theelderly may have encountered in their real life. As such, we
can motivate the elderly’s engagement in exergames.
The rest of this paper is organized as follows. We review the
related work on exergame design for the elderly and theories
on the familiarity in Section II. We present the framework offamiliarity design and the methodology to design familiar exergames in Section III. We show our Virtual Escape Exergame
as a typical example of familiarity design in Section IV. In theend, we conclude this paper in Section V.
II. R ELATED W ORK
In recent years, technology advances in virtual reality, augmented reality, and motion detection lead to the emerging of
exergames for the purpose of rehabilitation. These exergames
aim to help the elderly take exercises in a relaxing and entertaining atmosphere without the feeling of monotony and isolated in traditional exercises. We review some well establishedexergames in Section II-A. Moreover, in Section II-B, wereview some research on familiarity that have been conducted
to increase the engagement of the elderly to new technologies,
which show the effectiveness of familiarity in motivating theelderly.
A. Exergames for Elderly
Elderly may suffer from various impairments as they age,particularly the reduction of the motor and cognitive capabilities. Research shows that repetitive exercises can provide a
sufficient amount of stimuli to remodel the elderly’s brain andhence regain a better control on one’s motor capability [11].
Other studies on exergames indicate that technologies, such asvideo games, can enhance the elderly’s motivation of repetitive
exercises and help them to recover from the decreased motorand cognitive capability more efficiently [5], [12]. Comparedwith the traditional rehabilitation exercises, programmablesystems, such as exergames, can provide attractive graphicalvirtual environments, which can be designed in varying ways
to encourage the users to become more engaged, immersed,
and motivated in the rehabilitation exercises [13].
Exergames are usually implemented on the wirelessly controllable platforms, such as Nintendo Wiimote and Microsoft
Kinect. Study shows that Wiimote is more acceptable by theelderly people compared with other types of game controllers[14]. Kinect is also popular due to its non-intrusive andcontroller-free features. These wireless devices can reduce
human intervention and enhance the motion of the elderly
in an unrestricted environment. As such, they offer a senseof self-determination, provide a feeling of independence, andimprove the quality of elderly’s life [15].

Because exergames are relatively new for majority of theelderly, some studies in the literature focus on enhancing
the incentives for the elderly to play the exergames. In
terms of exergames design, Rizzo and Kim emphasized thevisibility, feedback, and identification of the target users as
three important human factors [5]. Balaam et al. indicated thatentertainment-oriented exergames are effective to motivate theelderly to play the exergames [16]. Similarly, Alankus et al.
proposed that motion-based games, which combine motiondetection with fun in the exergames, can stimulate people toexercise with the games voluntarily [6]. Burke et al. identified
that meaningful play and challenge are the two principles ofgame design that have particular relevance to motivate theelderly. Moreover, they pointed out that positive feedback,such as high numerical scores, can be the gratifying incentives
to encourage the elderly to continue the exergames and reach aparticular goal [17]. On the other hand, the negative feedback,such as failure signs, should be used discretely in case of
discouragement to the elderly [17].
Besides the impairment in motor capability, the elderly may
also suffer from memory loss and other mental problems,
which can substantially affect their capability to interact withthe real world [18]. In order to exercise both the elderly’s
physical and mental capabilities, various exergames shouldincorporate physical exercises together with cognitive challenges. For example, the SilverBalance exergame designedby Gerling et al. aims to recover the both elderly’s bodily
and cognitive impairments [19]. The game’s sensitivity canalso be adjusted according to the degree of reduction on an
individual’s visual and cognitive capabilities.
From prior research on exergames, we can identify thatexergame is more acceptable to the elderly in the home environment, which can help them recover from the deteriorated
capabilities in a relaxed and enjoyable atmosphere. In order
to benefit more number of elderly, we need to improve themotivation for them to play exergames.
B. Literature on Familiarity
Tognazzini suggested that the operation of an interactive
system is best achieved by means of a metaphor or analogy,
which indicates the importance of familiarity [20]. Being
familiar with a system means we are ready to operate it in an
appropriate way based on our prior experiences [21]. Sufficientexperiences may lead to the development of an internal model,
or stereotype, about how one expects something to work [8].
Demirbilek and Demirkan [22] studied the impact of theenvironment on aging factors of the elderly. They foundthe elderly prefer to live in their familiar environments inthe “getting older” phase of their life, which is a prime
important aspect to design their homes. For elderly, familiarity
can increase the safety, usability, and attractiveness of theenvironment. Barry [23] also encouraged the incorporation offamiliarity into the home design to bring positive changes.
He suggested that home design should preserve the character
attributes of each specific elder individual. Boger et al. [24]assessed the impact of familiarity on the use of different© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

84Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

faucet designs by elderly with dementia symptoms. The results
showed that familiarity plays a substantial role in faucet
usability (effectiveness, efficiency, and satisfaction) for theelderly. Boger et al. indicated that these trends may well beapplicable to other products and activities.
Besides the home environment, the theory of familiarity
can be also applied in interactions between new technologies
and the elderly. Leonardi et al. [7] designed the acceptable
domestic technologies for elderly based on the concept offamiliarity. They found the interface functioning depended ona variety of secondary elements, such as menus and dialog
boxes, which are technical components that the elderly are
not familiar with. Therefore, Leonardi et al. designed theinteraction modality based on few rules of interaction andnatural gestures, such as replacing the “erase” command withscrubbing the finger. Hollinworth and Hwang [10] designed
the tmail application with familiar visual objects (e.g., email
messages shown in the inbox resemble paper envelopes) andbehaviors (e.g., messages and images can be moved by touching and sliding). All the elder participants found the visualobjects in this familiar interface easy to understand and theycan quickly master how to use tmail. The experimental results
indicated that familiar interfaces can help in orientating theelder users with an application.
Prior research showed the effectiveness of familiarity inhelping the elderly to understand and interact with the newtechnology, which encourages us to adopt the familiarity considerations in exergame design. Based on the literature review,
we can identify that familiarity can increase the effectiveness,
efficiency, and satisfaction when elderly interact with newtechnologies.
III. M ETHODOLOGY OF FAMILIARITY D ESIGN IN
E XERGAMES FOR E LDERLY
Familiarity design in exergames can help the elderly tointeract with the system naturally and increase the usabilityof the games. In this section, we propose the methodology offamiliarity design in exergames for the elderly.
A. Framework of Familiarity Design
Familiarity helps in encouraging the elderly to learn andunderstand how to interact with the new technology by using
their existing knowledge learned in their everyday life. Thismeans not only the interfaces of the systems should appear
familiar, but also the style of interactions in the whole systemshould fit the elderly’s past experiences and capabilities.
Moreover, good familiarity design can evoke the elderly’s
memories and corresponding emotions [25]. For instance, in
the interface design of a system, the familiar objects and scenesthat the elderly already experienced before may activate theircorresponding memories and similar emotions as before.
Based on the literature review of familiarity design on the
adoption of new technologies for the elderly (see Section II),
we propose the framework of familiarity design for new
technologies, which consists of the following three dimensions[26]:

Meaningful Design· Cultural Familiarity· Symbolic Familiarity · Actionable Familiarity

Representation

Manipulation· Symbolic Familiarity· Cultural Familiarity· Actionable Familiarity· Cultural Familiarity

Fig. 1. Illustration on the three aspects of familiar exergames design.

1) Symbolic Familiarity: Objects, scenes, and activities,
which frequently appear in elderly’s daily life, should be
infused into the system.
2) Cultural Familiarity: Traditions, patterns, concepts, andrituals, which exist in the elderly’s culture, should be
infused into the system.
3) Actionable Familiarity: Actions to interact with the system should be consistent with the actions used wheninteracting with the real world.
The identified three dimensions can help us design familiarinterfaces and styles of interactions in all systems for theelderly. During a holistic system design, each part of thefamiliarity design can be categorized into one of the three
dimensions. Moreover, each dimension can play a differentrole in improving the usability of the system for the elderly.
Depending on the degree of the fidelity of simulating the three
dimensions of familiarity to the real world, any system can beevaluated by whether they are familiar enough to the elderly.
B. Design Familiar Exergames
Based on the aforementioned three dimensions of familiarity
design, we propose our specific approach on familiar exergame
design. Other than the metaphorical and analogical connection of the exergames to the real world, familiarity design
also requires the exergames share the same background ofconcepts, meanings, and practices with the elderly. Therefore,
we indicate three specific aspects, namely representation,
manipulation, and meaningful design, which lead to goodfamiliarity design in exergames. The relationship among thethree aspects and the major familiarity dimensions of these
aspects are shown in Fig. 1.
1) Representation: Using the representation as similar as
possible to the real world in exergames is an effective method
to help the elderly understand the games. Representation provides the elderly the tools to interpret the system. Typical representations include the familiar environments such as objects,
pictures, music, etc., which can help the elderly maximize
their functional capability. Thus, symbolic familiarity is themost appropriate dimension to achieve familiar representation.
Meanwhile, some concepts of cultural familiarity can also

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

85

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

be used in the representation for the elderly with the same
cultural background. On the other hand, representation does
not necessarily mean duplicating the objects in real life to theexergames. Rather, we need to take advantage of the novelty
in the digital world and at the same time, follow elderly’s reallife knowledge and experiences during representation design.
2) Manipulation: Manipulation indicates how the elderly
manipulate the objects in exergames. One of the advantages
in exergames is that the elderly can act in the same manneras they do in real world rather than point and click using
various computer accessories. However, the physical motions
required in exergames should be consistent with motor patterns
the elderly use in real world. For example, wrist rotation can
open the door handle in the virtual environment, which shares
the same cognitive model learned in real world. Therefore,
actionable familiarity should be considered in the manipulation
design. Each manipulation design should refer to the real life
elements. However, even if the representation is familiar to
the elderly, the interactions and manipulation in exergames
may not be familiar to them. In particular, exergame designers
should pay more attention when there is no logical relationshipbetween the motions and the outcomes in exergames.
3) Meaningful Design: Familiar exergame design requires
not only the imitation of real world, but also the stimulationof the elderly’s memories and emotions. For instance, theinterfaces with specific concepts and meanings may activate elderly’s memories and practices experienced before andinvoke their past emotions. With the sense of anticipation
during exergame playing, guided by the past experiences, theelderly will find the interactions recognizable and enjoyable.
Therefore, meaningful design means to select the contexts anddomains in exergames which are not only objectively and socially suitable, but also related to the elderly’s past experiencesand emotions. Meaningful design requires to take all the three
dimensions of familiarity into account. However, we should
pay more attention to the cultural familiarity because the same
design may have different meaning for elderly from different
cultures. Meanwhile, due to the diverse personal experiencesof each individual, meaningful design may vary from person toperson. Thus, the personalization of meaningful design should
also be taken into consideration.
Kaplan and Kaplan [8] indicated that familiarity could
promote the cognitive processes of the elderly and increase
their confidence in interacting with new technologies. The
elderly will perform better and guide their own behaviors
when they face familiar objects and environments. Hence,by applying the aforementioned three aspects of familiarity
design in exergames, the elderly may not need to learn newtechnological language or adapt to unfamiliar environments.
Alternatively, they can utilize their prior experiences, which
are learned from non-technical areas, to complete the basic
exercise tasks embedded in the exergames. Another advantage
of our proposed approach is that the elderly can take less cognitive effort to accomplish the exercise requirements, which
is more practicable for the elderly with cognitive difficulties.
With all these advantages, we aim to follow this approach toFig. 2. A camera view of the virtual home in the Virtual Escape Exergame.

design the exergames that elderly can immediately understandand enjoy.
IV. V IRTUAL E SCAPE E XERGAME
The Joint NTU-UBC Research Centre of Excellence inActive Living for the Elderly (LILY), Nanyang Technological
University, Singapore, is one of the world’s first research centres focusing on interactive digital media empowered agelessaging. LILY has initiated a number of interactive exergames
for the elderly using Kinect as the non-intrusive motion detection device. We follow the approach of familiarity exergame
design presented in Section III when we developed the Virtual
Escape Exergame (VEE) to help the elderly exercise theirmotor and cognitive capabilities.
Conventional escape game requires players to escape from
imprisonment by exploring and exploiting the surrounding
environment. Due to the involvement of puzzles, players
need to solve the problem with repetitive attempts in mostcases while exercising their cognitive and physical capabilities,
which makes the escape game suitable and effective for theelderly’s exercises. VEE promotes various types of interactions
between the elderly and the virtual confined environment (aliving room, see Fig. 2). Another advantage of VEE is its
high flexibility, which enables the elderly to perform various
actions, such as pulling out drawers, pressing buttons, turningknobs, etc. This kind of extendable game flexibility makes
VEE more challenging and less boring. To solve the ingame puzzles, the elderly need to interact with various objects
in the living room with body movements in front of theKinect cameras. Therefore, VEE exercises not only the body
movements of the elderly but also their critical thinking skills.
Table I lists some selected familiarity designs implemented
in VEE and their related dimensions. The rest of this section
introduces each aspect of these familiarity designs in detail.
A. Representation Familiarity in VEEIn VEE, we develop a familiar virtual environment for theelderly to achieve representation familiarity. The escape roomis designed as a typical living room (see Fig. 2) that the elderly
encounter every day. The furnishings and objects in the virtualroom are displayed as similar as possible to the real world.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

86Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE IFAMILIARITY D ESIGN IN VEE
Representation

Manipulation
Meaningful
Design1) Furniture design (Symbolic & Cultural Familiarity)
2) View outside the windows (Symbolic Familiarity)
3) Room layout (Symbolic Familiarity)
1) Selecting chests in the TV (Actionable Familiarity)
2) Wiping the desk (Actionable Familiarity)
3) Pulling the drawers (Actionable Familiarity)
1) Background music: Xinyao (Cultural Familiarity)2) Encouragement characters (Symbolic Familiarity)

Hence, there will be no overwhelmed feeling when the elderly
first play this exergame. Due to the familiarity with the virtualenvironment, the elderly may find it is easy to know wherethe various objects placed when working on a given task.
Other than the familiar representation of the integral virtualroom design, we also pay attention to the familiarity designof each specific object in the room, especially those objects
that the elderly need to interact with. With the guidance ofsymbolic and cultural familiarity, we figure out how to achieve
the representation familiarity. For example, in stead of modern
furniture, we implement the old fashion furniture in the virtual
room. The elderly are familiar with these old fashion furnitureand easy to know how to interact with them. Moreover, someparts of the virtual room, such as the decorative painting on
the wall and the view outside the windows, are not design to
be interacted with. However, we still use familiar pictures andscenes to decorate the environment in order to increase theelderly’s home-like feel of the exergame.
B. Manipulation Familiarity in VEE
In exergame design, task design is the most importantpart regarding exercises for elderly. Manipulation familiarity
should be embedded in the tasks of all exergames. In termsof VEE, it requires the elderly to proceed with a series of
movements in the correct order to unlock the puzzles, which
is more difficult than the common exergames because players
may need to repeat similar movements. With the requirementof sequential tasks, VEE may appear difficult for the elderly.
Therefore, familiar manipulation is essential in VEE to help
the elderly to quickly understand and complete the tasks.
In order to incorporate the actionable familiarity into the
manipulation design, the tasks we designed in VEE mainly
come from the elderly’s daily life scenarios, which can encourage the elderly to transfer their daily life knowledge into
the virtual environment. For example, when the elder player
has to find the information from the television in the game(see Fig. 3), they just need to follow the steps of watching
television in their home: Open the television and then change
channels to find the correct chest. The difference lies in thatthey may not have a remote controller. All they need is to
use their arms to complete the tasks in front of the Kinect
cameras, which is beneficial to their physical exercises. In
order to design more familiar tasks for the elderly, we alsotransform some simple daily housework into the VEE tasks.
Because the elderly are familiar with these housework, theyshould be able to quickly understand the tasks based on theirFig. 3. An example of task design in Virtual Escape Exergame.

Fig. 4. Wiping task in Virtual Escape Exergame.

experiences. For example, we design a table coated in dust (seeFig. 4), where the required information to complete certain
tasks is hidden beneath. The elderly can wipe the table as
they normally do at home to obtain the information.
With the manipulation familiarity design, we hope theelderly would find it easy to act appropriately in order tocomplete tasks, which would improve their confidence incontinuing to play VEE. Hence, manipulation familiarity canincrease the adherence of exergame playing among the elderly.
C. Meaningful Design in VEE
Altman and Werner indicated that environment can serve as
a cue to remind people to connect present and past, and to linkexperience and memory [27]. Their findings bring us the ideathat familiarity may not mean to display ordinary objects, but
to present meaningful artifacts and environments to the elderly.
Meaningful design requires the exergame designers to investigate the commonly shared past experiences and culturesof a particular region where the exergame will be deployed.
The appropriate exergame design for one region may not bemeaningful for the elderly from another region. Therefore,
after investigating the past experiences of the elderly in
Singapore, we find the Singaporean pop songs (Xinyao, URL:
https://en.wikipedia.org/wiki/Xinyao) back in the 80s are thecommonly shared memory of Singapore’s elder generation.
Majority of the elderly are fond of Xinyao after experiencing
the charm of it. Thus, we select some popular Xinyao songs as© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

87Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

the background music of VEE to stimulate the elderly’s pastfeeling of passion or comfort.
Meaningful design may vary from individual to individual.
Therefore, personalization of the exergame is also important. Personalized familiarity can be defined as aligning or
customizing the attributes and features in order to enable
an individual to develop relevant knowledge or mastery. We
customize the encouragements used in VEE for each individual
elderly. Specifically, we choose the photos or videos of theelderly’s family members, such as their children and grandchildren, while displaying the supportive messages to encourage
the elderly to continually play VEE. The elderly will be more
attracted by VEE with these familiar faces appearing in the
exergame from time to time.
V. C ONCLUSION AND F UTURE W ORK
Exergames are proven to be effective and efficient to motivate the elderly to take exercises and stay healthy. Familiarity
is a highly important issue for the interactive system design,
such as exergames. We suggest that the metaphors of thedaily life objects and the cognitive models can help theelderly understand the interactivity of the exergames. In thispaper, we define the framework of familiarity design in threecomplementary dimensions. Based on our framework, we
further propose three specific aspects of familiarity design inexergames for the elderly. Moreover, we introduce the Virtual
Escape Exergame that we developed for the elderly. For eachfamiliarity aspect of this exergame, we give a typical example
in the context of VEE. By following our design approach,
VEE will be more explicit and attractive for the elderly by
exploring their prior experiences and knowledge.
For future work, we prepare to improve the viability andeffectiveness of our approach on familiarity exergame designby conducting case studies on VEE. We will follow the
feedback of the case studies to refine our design methodology
in order to make it more complete and simpler to use.
ACKNOWLEDGMENT
This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its IDM
Futures Funding Initiative.
R EFERENCES[1] Department of Economic and Social Affairs Population Division, World
Population Ageing. United Nations, 2015.
[2] D. Wang, B. Subagdja, Y. Kang, A.-H. Tan, and D. Zhang, “Towards
intelligent caring agents for aging-in-place: Issues and challenges,” inComputational Intelligence for Human-like Intelligence (CIHLI), 2014
IEEE Symposium on. IEEE, 2014, pp. 1–8.
[3] J. W. Burke, M. McNeill, D. Charles, P. J. Morrow, J. Crosbie, and
S. McDonough, “Serious games for upper limb rehabilitation following
stroke,” in Games and Virtual Worlds for Serious Applications, 2009. In
Proceedings of VS-GAMES’09. IEEE, 2009, pp. 103–110.
[4] M. Shaughnessy, B. M. Resnick, and R. F. Macko, “Testing a modelof post-stroke exercise behavior,” Rehabilitation Nursing, vol. 31, no. 1,
pp. 15–21, 2006.
[5] A. A. Rizzo and G. J. Kim, “A swot analysis of the field of virtual reality
rehabilitation and therapy,” Presence, vol. 14, no. 2, pp. 119–146, 2005.

[6] G. Alankus, A. Lazar, M. May, and C. Kelleher, “Towards customizable
games for stroke rehabilitation,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2010, pp.
2113–2122.
[7] C. Leonardi, C. Mennecozzi, E. Not, F. Pianesi, and M. Zancanaro,“Designing a familiar technology for elderly people,” Gerontechnology,
vol. 7, no. 2, pp. 151–156, 2008.
[8] S. Kaplan and R. Kaplan, Cognition and environment: functioning in an
uncertain world. Ulrichs Books, 1982.
[9] G.-R. Son, B. Therrien, and A. Whall, “Implicit memory and familiarity
among elders with dementia,” Journal of Nursing Scholarship, vol. 34,
no. 3, pp. 263–267, 2002.
[10] N. Hollinworth and F. Hwang, “Investigating familiar interactions to help
older adults learn computer applications more easily,” in Proceedings ofthe 25th BCS Conference on Human-Computer Interaction. British
Computer Society, 2011, pp. 473–478.
[11] J. A. Kleim, T. A. Jones, and T. Schallert, “Motor enrichment andthe induction of plasticity before or after brain injury,” Neurochemical
Research, vol. 28, no. 11, pp. 1757–1769, 2003.
[12] D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V.
Adamovich, M. Recce, and H. Poizner, “Virtual reality-enhanced stroke
rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation
Engineering,, vol. 9, no. 3, pp. 308–318, 2001.
[13] B. G. Witmer and M. J. Singer, “Measuring presence in virtual environments: A presence questionnaire,” Presence: Teleoperators and virtualenvironments, vol. 7, no. 3, pp. 225–240, 1998.
[14] S. Ganesan and L. Anthony, “Using the kinect to encourage older adults
to exercise: A prototype,” in Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems. ACM, 2012, pp. 2297–2302.
[15] D. Felce and J. Perry, “Quality of life: Its definition and measurement,”
Research in Developmental Disabilities, vol. 16, no. 1, pp. 51–74, 1995.
[16] M. Balaam, S. Rennick Egglestone, G. Fitzpatrick, T. Rodden, A.M. Hughes, A. Wilkinson, T. Nind, L. Axelrod, E. Harris, I. Ricketts
et al., “Motivating mobility: Designing for lived motivation in stroke
rehabilitation,” in Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, 2011, pp. 3073–3082.
[17] J. W. Burke, M. McNeill, D. K. Charles, P. J. Morrow, J. H. Crosbie, andS. M. McDonough, “Optimising engagement for stroke rehabilitation
using serious games,” The Visual Computer, vol. 25, no. 12, pp. 1085–
1099, 2009.
[18] K. D. Cicerone, D. M. Langenbahn, C. Braden, J. F. Malec, K. Kalmar,
M. Fraas, T. Felicetti, L. Laatsch, J. P. Harley, T. Bergquist et al.,“Evidence-based cognitive rehabilitation: Updated review of the literature from 2003 through 2008,” Archives of Physical Medicine andRehabilitation, vol. 92, no. 4, pp. 519–530, 2011.
[19] K. M. Gerling, J. Schild, and M. Masuch, “Exergame design for
elderly users: The case study of silverbalance,” in Proceedings of the
7th International Conference on Advances in Computer Entertainment
Technology. ACM, 2010, pp. 66–69.
[20] B. Tognazzini, The Art of Human-Computer Interface Design. Reading:
Addison-Wesley, 1991.
[21] J. Herstad and H. Holone, “Making sense of co-creative tangibles
through the concept of familiarity,” in Proceedings of the 7th Nordic
Conference on Human-Computer Interaction: Making Sense Through
Design. ACM, 2012, pp. 89–98.
[22] O. Demirbilek and H. Demirkan, “Universal product design involving elderly users: a participatory design model,” Applied Ergonomics, vol. 35,
no. 4, pp. 361–370, 2004.
[23] J. E. Barry, “Everyday habits and routines: Design strategies to individualize home modifications for older people,” Ph.D. dissertation,
Washington State University, 2008.
[24] J. Boger, T. Craig, and A. Mihailidis, “Examining the impact offamiliarity on faucet usability for older adults with dementia,” BMC
Geriatrics, vol. 13, no. 1, p. 1, 2013.
[25] T. Rohrer, “Metaphors we compute by: bringing magic into interface
design,” Center for the Cognitive Science of Metaphor, Philosophy
Department, University of Oregon, 1995.
[26] Z. Pan, C. Miao, H. Yu, C. Leung, and J. J. Chin, “The effects offamiliarity design on the adoption of wellness games by the elderly,”
in 2015 IEEE/WIC/ACM International Conference on Web Intelligenceand Intelligent Agent Technology (WI-IAT), vol. 2. IEEE, 2015, pp.
387–390.
[27] I. Altman and C. Werner, “Home environments: Human behavior andenvironment,” New York: Plenum, 1985.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

88Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Investigating the Influences of Gamification in
Citizen Science Systems:
A Motivational Affordances Perspective
Jian Tang

Yuxiang Zhao

School of Information
Central University of Finance and Economics
Beijing, China
jiantang@cufe.edu.cn

School of Economics and Management
Nanjing University of Science & Technology
Nanjing, China
yxzhao@vip.163.cn

Abstract—Gamification indicates the idea of providing
gameful experiences in non-game contexts to support overallvalue creation. Points-based gamification and story-based
gamification are two approaches that vary in game elements
implementation. This study aims to understand the influences ofgamification from the lens of motivational affordances. Drawing
upon stimulus-organism-response framework, we propose a
research model to delineate how different gamification
approaches trigger various perceived motivational affordances,which in turn lead to game engagement and user participation.
We aim to uncover the motivational affordances of two distinct
gamification approaches in citizen science systems and encourage
more research investigations in this direction.
Keywords—citizen
science;affordances; user participation

gamification;

motivational

I. INTRODUCTION
Crowdsourcing information systems is an umbrella termthat encompasses various information systems that rely on
open calls to recruit workers[1, 2]. The rapid growth of online
user population generates a big online labor market, where
requesters and organizations have access to a wide pool ofparticipants. Crowdsourcing research indicates ways ofleveraging the power of online labor to work on projects,
collecting ideas, and solving problems [3, 4]. Citizen sciencesystems are one type of crowdsourcing information systemdesigned to recruit general public to participate in scientificactivities [2]. Till now, well-known citizen science projects,
such as Galaxy Zoo, Citizen Sort, and Foldit, have gained
thousands of participants to work on scientific problems in
different areas, including astronomy, biology, archeology, etc.
It is notable that science tasks can often be repetitive andtedious, even some participants may not have any interest inscientific activities. Therefore, designing a system in a waythat can attract and sustain participants are critical for the
success of citizen science projects.
Gamification as an approach has gained increasing
attention from academia and practitioners, which is considereda potential approach to engage crowdsourced workforces inaddressing large-scale problems while playing games [5].
Gamification is considered as a significant motivating
technique that makes non-game tasks fun and enjoyable [6]and improves employees’ motivation and performance inorganizational activities [7]. In citizen science research,studies have identified the importance of gameful design instimulating positive psychological outcomes from participants[8-10].
There are two distinct views of gamification in theliterature. A systemic view defines gamification as the use ofgame elements in non-game contexts [11]. Yet, Huotari and
Hamari posits to denote gamification as affording gamefulexperiences in order to support users’ overall value creation[12]. The experiential view of gamification undertakes arelational perspective of game elements in non-game contexts.
Gamified information systems requires users’ capabilities to
identify the gameful experiences, and therefore, gamification
could be understood more broadly as a process that imbue thepossibilities for gameful experiences. Gamification does not
tie to particular game elements, and the use of a game service
may lead to a gameful experience for one user but not another[12]. This depends on players’ capabilities to perceive themotivational value afforded by gamification design.
Therefore, the notion of motivational affordances plays therole to explain the variety of players’ responses togamification approaches.
Our study contributes to the extant literature on
gamification in citizen science systems in several ways. First,
to examine the motivational and experiential aspects ofgamification design, we undertake the lens of motivational
affordances to investigate the relational influences ofgamification on participants. Our research attempts to anchor
gamification in the growing body of literature on citizen
science and aims to study the possibilities of gamification inaffording motivational experiences. Unlike prior explorativeand descriptive research [13, 14], our study is theoreticallygrounded in the Stimulus-Organism-Response model and we
proposes a research model that depicts how game elements© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

89

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

afford to motivate users and in turn influences their gameengagement and participation behavior.
II. LITERATURE REVIEW
A. Motivational Affordances
Affordances of objects in the environment are perceived
through featured information or salient clues picked byanimals or human beings [15]. In the realm of humancomputer interaction, affordances was introduced by Norman
to understand the interaction design of technology.
Affordances refer to potential actions that are possible,
requiring the match between technology and capabilities ofhuman subjects [16, 17]. Zhang [18] applies the notion ofaffordances to propose design principles of information andcommunication technologies so that the technologies can
support one’s psychological, cognitive, emotional, and social
needs, arguing that people tend to use and continue to use ICT
to fulfill various psychological, cognitive, social, andemotional needs. Therefore, features or functions of an IT
artifact that support these internal motivational needs caninfluence whether, how, and how much this artifact will be
used.
Motivational affordances covers more than objective
design, instead, it indicates the relational attributes between
the objective design and capabilities of subjects to act upon it.
Thus, one common understanding of affordances are action
possibilities that can be identified by users to support theiroverall well-being. From the affordance perspective,
gamification is not equivalent to add-on game elements,
instead, gamification indicates the relational attributes ofsystem design that can be perceived by participants. Fig.1
used a diagram to depict the associative role of motivational
affordances between game elements and subject needs.
Hamari et al. [8] summarized three main steps of
gamification: 1) the implemented motivational affordances, 2)
the resulting psychological outcomes, and 3) the further
behavioral outcomes. They argued that the implementation ofgame elements can enhance the motivational affordances andlead to better user experiences and performances. Yet, they
overlooked the fundamental characteristic of affordances,which is the relational attributes that reside between thetechnology and subject. Therefore, we argue that the
motivational affordances of gamification actually indicate how
players sense and perceive the motivational driving forces
afforded by game elements.

Fig. 1 Relational View of Motivational Affordances. The rectangles represent
the object entity and subjective entity; the diamond shape represents theassociate role (relational attributes) of affordances between object and subject.

B. Stimulus-Organism-Response Framework
Game elements are objective design characteristics that arepart of the environment where participants work on scientificassignments. S-O-R framework is widely applied to explaininterrelationships among three critical components of personenvironment interaction: stimulus, organism, and response[19-22]. Stimuli in the environment are design elements thattrigger internal processes inside a human’s (organism’s) mind,
which then affect behavioral responses or behavioral
tendencies. To address the question regarding the influencesof gamification on participants’ psychological states andbehavioral outcomes, stimulus-organism-response (S-O-R)
framework is considered appropriate to guide our research
investigation, as it delineates a picture of how environmentand human actor interplay with each other. The S-O-R
framework suggests a potential foundation to understand themechanisms of how gamificatoin can have impact on user
behaviors [23].
III. CONCEPUTAL DEVELOPMENT
Users are central part of citizen science projects.
Therefore, it is key to understand how these users are
motivated and driven to work on citizen science assignments.
Building on early mentioned notions of motivational
affordances and S-O-R framework, we examine how gameelements can be implemented to support motivational
affordances by instantiating Zhang’s [18, 24] design principles
in an attempt to promote user participation. Regarding thecharacteristics of citizen science systems, we focus on
cognitive motivational needs (i.e., competence andachievement) and emotional needs (i.e., affect and emotion) in
our research contexts. In particular, we are interested in theinfluences of two gamification approaches, points-based
gamification and story-based gamification, in support two
types of motivational affordances, perceived cognitive
affordances and perceived affective affordances. Fig.2 depicts
the proposed research model.
Points-based gamification is an approach that focuses on
quantified measurements of user performance, and threecommonly used game elements include points, badges, andleaderboards [25]. According to Zhang’s [18, 24] positive
design principle, providing positive feedback is a key
condition to satisfy competence need. Jung et al. [26]confirmed that providing feedback significantly increased
quantity and quality of user participation in activities of idea
generation. Perceived cognitive affordances refer to user’s
perception of action possibilities facilitated by game elments
to satisfy their cognitive needs, such as competence andachievement. These points-based game elements mostly
reflect player’s achievements and may have impact on
players’ perceived cognitive affordances. Therefore, wehypothesize:
Hypothesis 1. Citizen science system with points-based
gamification induces higher level of perceived cognitive
affordances than those systems without points-based
gamification.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

90

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 2 Proposed Research Model.

Story-based gamification, on the other hand, provides anenriched scenario with game narratives about “stories with abeginning, middle, and end that provide information about thecharacters and plot” [27]. Stories-based gamification includes
a storyline for players to follow, allowing more intensive
interaction between citizen science systems and participants.
Perceived affective affordances refer to a user’s perception ofaction possibilities that are facilitated by game elements toentertain themselves and satisfy emotional needs, such as
enjoyment. Interactions with story-based gaming environmentare beyond task completion and allow the possibilities forparticipants to seek for fun and satisfy their emotional needs.
Therefore, we hypothesize:
Hypothesis 2a. Citizen science system with story-based
gamification induces higher level of perceived affective
affordances than those systems without story-based
gamification.
Nevertheless, we should not overlook the possibilities ofstory-based gamification to support cognitive needs, such ascompetence and achievement. When the story unfolds, themissions and levels indicate the progress of participants, andthey are likely to be aware of their achievements. Therefore,
we hypothesize:
Hypothesis 2b. Citizen science system with story-based
gamification induces higher level of perceived cognitive
affordances than those systems without story-based
gamification.
Psychological outcomes denotes the states experienced byparticipants after working on citizen science projects. A few
related constructs also denote the psychological outcomes ofusing information systems, include flow [28], immersion, andengagement [8]. Game engagement refers to the player’s state
when interacting with a game. Game engagement is a part ofgame experiences, encompassing multiple dimensions of aplayer’s subjective feelings [29]. Brockmyer et al. [30]conceptualize game engagement as a generic indicator ofgame involvement, and other relevant concepts, such as
immersion, flow, psychological absorption, indicate theprogression of ever-deeper engagement. When participants
experienced motivational affordances, their motivational
needs are satisfied, they are more likely to feel engaged withthe citizen science projects. Therefore, we hypothesize:
Hypothesis 3. Perceived cognitive affordances positively
associate with game engagement.

In the citizen science game world, when player
experiences game engagement, it is very likely that theywould like to stay in the game environment and work on moreassignments. On the other hand, if they don’t feel engaged
with the game environment, they will probably choose toleave the game or work on very few assignments. Therefore,
we hypothesize:
Hypothesis 5. Game engagement positively associate withuser participation.
IV. RESEARCH METHODOLOGY
We will conduct a lab experiment and collect data to validate
our research model. The empirical study consists of threestages: experiment preparation, pilot tests, and the main
experiment. We plan to conduct a 2 (points-based
gamification: presence vs. absence) by 2 (story-based
gamification: presence vs. absence) factorial lab experiment tocompare the influences of these two main gamification
approaches (See Table I.).
TABLE I.

TREATMENTS OF EXPERIMENTAL GROUPS
Points- based gamification
Presence

Story-based
gamification

Presence
Absence

Points feedback,
Story scenario
Points feedback

Absence

Story scenario
None

The experiment preparation includes experimentenvironment development and instrument validation. Pilot
tests should be conducted with a small sample to verify theeffectiveness of experimental manipulation. Then, the main
experiment will be executed to validate the research model.
Participants will be randomly assigned to one of the four
experimental groups. The random assignment can control thepossible demographic differences. They will be asked toworking on a citizen science assignments, and then fill out aquestionnaire regarding their subjective experiences.
According to the power analysis calculation from Cohen [31],a group size of 44 is enough for a power of .80 to detect themedium effect size (f 2 = .15) at the significance level of .05 (α= .05) [31].
V. EXPECTED CONTRIBUTION
The nature of citizen science that requests for massive
public participation (the crowd) attracts increasing attention
from scientists and policy makers. Gamification as an
academic topic of study is relatively young [8], yet it is anapproach with great potential to promote citizen sciencesystems. Our research intend to anchor the development
gamification within citizen science systems literature andinvestigate the influences of gamification from the lens ofmotivational affordances. The findings may provide some
implications for citizen science design, and some pertinent
design principles and guidelines will be proposed to convert
the potential motivational needs to design practices.

Hypothesis 4. Perceived affective affordances positively
associate with game engagement.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

91

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

VI. ACKNOWLEDGMENT
This research is supported by the National Social Science
Fund of China (Grant No. 14AZD045).

[15]
[16]

VII. REFERENCES
[1]
[2]

[3]
[4]

[5]
[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

J. Howe. (2006). The rise of crowdsourcing
D. Geiger and M. Schader, "Personalized task recommendation incrowdsourcing information systems — Current state of the art,"
Decision Support Systems, vol. 65, pp. 3-16, 9// 2014.
J. Howe, Crowdsourcing: How the power of the crowd is driving
the future of business: Random House, 2008.
J. M. Leimeister, M. Huber, U. Bretschneider, and H. Krcmar,"Leveraging crowdsourcing: activation-supporting components for
IT-based ideas competition," Journal of management information
systems, vol. 26, pp. 197-224, 2009.
L. von Ahn, "Games with a purpose," Computer, vol. June, pp. 9698, 2006.
D. R. Flatla, C. Gutwin, L. E. Nacke, S. Bateman, and R. L.
Mandryk, "Calibration games: making calibration tasks enjoyableby adding motivating game elements," in Proceedings of the 24th
annual ACM symposium on User interface software andtechnology, 2011, pp. 403-412.
D. Liu, X. Li, and R. Santhanam, "DIGITAL GAMES AND
BEYOND: WHAT HAPPENS WHEN PLAYERS COMPETE,"MIS Quarterly, Forthcoming.
J. Hamari, J. Koivisto, and H. Sarsa, "Does gamification work?--aliterature review of empirical studies on gamification," in 47th
Hawaii International Conference on System Sciences (HICSS),
2014, pp. 3025-3034.
N. R. Prestopnik and J. Tang, "Points, stories, worlds, anddiegesis: Comparing player experiences in two citizen sciencegames," Computers in Human Behavior, vol. 52, pp. 492-506,
November 2015 2015.
A. Eveleigh, C. Jennett, S. Lynn, and A. L. Cox, "I want to be acaptain! I want to be a captain!: gamification in the old weather
citizen science project," in Proceedings of the First International
Conference on Gameful Design, Research, and Applications, 2013,
pp. 79-82.
S. Deterding, M. Sicart, L. Nacke, K. O'Hara, and D. Dixon,"Gamification. using game-design elements in non-gaming
contexts," presented at the CHI 2011, Extended Abstracts on
Human Factors in Computing Systems, Vancouver, BC, Canada,
2011.
K. Huotari and J. Hamari, "A definition for gamification:anchoring gamification in the service marketing literature,"
Electronic Markets, pp. 1-11, 2016.
A. Bowser, D. Hansen, Y. He, C. Boston, M. Reid, L. Gunnell, etal., "Using gamification to inspire new citizen science volunteers,"presented at the Proceedings of the First International Conference
on Gameful Design, Research, and Applications, Toronto, Ontario,
Canada, 2013.
D. Rotman, J. Preece, J. Hammock, K. Procita, D. Hansen, C. Parr,et al., "Dynamic changes in motivation in collaborative citizenscience projects," in Proceedings of the ACM 2012 conference on

[17]
[18]

[19]
[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]
[29]

[30]

[31]

Computer Supported Cooperative Work(CSCW), 2012, pp. 217226.
J. J. Gibson, The Ecological Approach to Visual Perception.
Boston: Houghton Mifflin, 1979.
D. A. Norman, The design of everyday things: Revised andexpanded edition: Basic books, 2013.
D. A. Norman, The Psychology of Everyday Things. New York:
Basic Books, 1988.
P. Zhang, "Motivational affordances: Fundamental reasons for ICT
design and use," Communications of the ACM vol. 51, pp. 145147, 2008.
A. Mehrabian and J. A. Russell, An approach to environmental
psychology. Cambridge, MA: MIT Press, 1974.
L. Deng and M. S. Poole, "Affect in web interfaces: a study of the
impacts of web page visual complexity and order," MIS Quarterly,
vol. 34, pp. 711-730, 2010.
V. A. Vieira, "Stimuli–organism-response framework: A metaanalytic review in the store environment," Journal of BusinessResearch, vol. 66, pp. 1420-1426, 2013.
S. A. Eroglu, K. A. Machleit, and L. M. Davis, "Empirical testingof a model of online store atmospherics and shopper responses,"
Psychology & Marketing, vol. 20, pp. 139-150, Feb 2003.
J. Tang and N. Prestopnik, "Toward an Understanding of The
Influences of Meaningful Framing on User Participation in A
Gamified Information System," presented at the European
Conference on Information Systems, Istanbul, Turkey, 2016.
P. Zhang, "Toward a positive design theory: Principles fordesigning
motivating
information
andcommunication
technology," in Designing Information and Organizations with a
Positive Lens, M. Avital, R. Bolland, and D. Cooperrider, Eds., ed,
2008.
C. Schlagenhaufer and M. Amberg, "A Descriptive Literature
Review and Classification Framework for Gamification inInformation Systems," in Twenty-Third European Conference on
Information Systems (ECIS), Münster, Germany, 2015.
J. H. Jung, C. Schneider, and J. Valacich, "Enhancing theMotivational Affordance of Information Systems: The Effects of
Real-Time Performance Feedback and Goal Setting in Group
Collaboration Environments," Management Science, vol. 56, pp.
724-742, Apr 2010.
A. S. Lu, "Narrative in Exergames: Thoughts on Procedure,
Mechanism, and Others," Games For Health Journal, vol. 4, pp.
19-24, 2015.
M. Csikszentmihalyi, Flow: The psychology of optimal experience.
New York, NY: Harper Perennial, 1991.
B. Gajadhar, Y. d. Kort, and W. IJsselsteijn, "Influence of social
setting on player experience of digital games," presented at the
CHI '08 Extended Abstracts on Human Factors in Computing
Systems, Florence, Italy, 2008.
J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M.
Burkhart, and J. N. Pidruzny, "The development of the GameEngagement Questionnaire: A measure of engagement in video
game-playing," Journal of Experimental Social Psychology, vol.
45, pp. 624-634, 2009.
J. Cohen, "A power primer," Psychological Bulletin, vol. 112, pp.
155-159, 1992.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

92

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Modeling of Crowd Science Network: An Approach Basedon Cyber-Anima
Jianping Shen

Yadong Huang

Yueting ChaiNational Engineering Laboratory for ECommerce Technologies
Department of Automation
Tsinghua University, Beijing, China
shenjianping@tsinghua.edu.cn

National Engineering Laboratory for ECommerce Technologies
Department of Automation
Tsinghua University, Beijing, China
huang-yd14@mails.tsinghua.edu.cn

National Engineering Laboratory for ECommerce Technologies
Department of Automation
Tsinghua University, Beijing, China
chaiyt@tsinghua.edu.cn

Abstract— Nowadays new models, technologies andapplications are jointly breeding a future networked system ofindustrial operation. To this end, a novel network call Crowd Science
Network (CSN) evolves from traditional internet to a large-scaled,
open-styled, self-organized and ecological intelligent network. Basedon CSN, a new generation of E-commerce platform can be built.
However modeling of CSN, we have to tackle with several challenges.
In this work, we presents a modeling approach based on CyberAnima, which is a semantic and generic model of cyber-physics
system proposed in our previous work. The main contributions of
this paper are threefold: (1) Design a generic and semantic model
for CSN participants based on Cyber-Anima; (2) Develop a model
learning technique to handle heterogeneous and non-structured
information resources; (3) Propose an ADI-based mechanism todrive the trading-styled connection and interaction between modeled
CSN participants.
Keywords— cyber-physics system; trade matching; semantic
model; E-commerce

I.

INTRODUCTION

In a new round of global revolution in technology andindustry, the fusion of internet and other domains, which
produces a strategical and holistic impact on the development of
economic society in various countries of the world, is quite
blooming and promising. New models, technologies andapplications are jointly breeding a future networked system ofindustrial operation. Compared to traditional systems, the future
networked system of industrial operation has four essential
characteristics: proactive personalized consumption, direct
centralized distribution, intelligent decentralized manufactureand ecological self-organized system. To facilitating suchcharacteristics, a novel network which evolves from traditional
internet to a large-scaled, open-styled, self-organized andecological intelligent network, called Crowd Science Network(CSN), is required by the aforementioned networked system.
In the perspective of composition, CSN composes of various
intelligent federated agents participating as persons, enterprises
and administrative departments which are related by supply anddemand. With a behavioral perspective, interactions between
any agent in CSN are behaviors or activities of trading in someextent. As a ternary system of information, physics and society,
CSN has essential logics of openness, federation, cooperationand sharing, and it pursuits the object of consistency, efficiency,
innovation and development.
In the light of definition, any individual or organization of therealistic world has a unique counterpart in CSN, which has thesame patterns of perceiving, thinking and behaving. Using
counterpart to reflect realistic individuals mostly emerges in the
research of cyber-physical system, e.g. Cyberself, CyberIndividual, etc. Cyberself[1] is proposed to be an embodiment of
individual as a means of identity signaling and as a medium of
online interaction. Whereas Cyber-Individual[2], with a shortterm ‘Cyber-I’, is the unique counterpart of real person on
cyberspace and also a digital counterpart going with one’s
whole life. However such researches on counterpart of realistic
individual remain in the phase of concept and is incapable tomodel CSN. Modeling of CSN is a twofold process. First, model
each participant of CSN in a generic and semantic way. Second,define the connections and interaction between models of CSN.
There are three basic challenges in modeling of CSN: (1) how
can we design a generic model for all kinds of CSN participants?
(2) how can we build a semantic model with various
heterogeneous and non-structured information resources? (3)how can we define a mechanism for trading-styled connection
and interaction between CSN participants?
In our previous work, we propose Cyber-Anima[3], a semantic
model to reflect individuals in the cyberspace and reason their
desires and intentions for recommender systems. Cyber-Animahas six dimensions: physiology, belief, character, knowledge,
experience and context, which correspond to intrinsic andextrinsic factors of humanity. On the one hand, Cyber-Anima
possesses the generalization which can develop to various
disciplines, e.g. biology, medical science, psychology,sociology, statistics, economics, management science, politics,
philosophy, anthropology and etc. On the other hand, a fourunitary semantic structure of dimension, concept, relation andaxiom empowers Cyber-Anima to reflect any realistic
participants to cyberspace. Based on Cyber-Anima, an AnimaDesire-Intention-based (ADI-based) reasoning approach is
proposed for recommendation interactions, which can also be
extended to the scenario of trading-styled interactions.
In this work, we presents an approach based on Cyber-Animato model CSN. Leveraging and refining the dimensions ofphysiology, belief, character, knowledge, experience and© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

93

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

context, we extend Cyber-Anima to fit all kinds of CSN
participant, e.g. persons, enterprises and departments ofgovernment. We developed a model learning technique basedon natural language processing for semantic model building andsynchronizing via heterogeneous and non-structured
information resources. We propose an ADI-based mechanism
for connection and interaction between modeled CSN
participants. We developed the prototype system of new
generation E-commerce platform to prove the effectiveness andsoundness of CSN modeling.
The main contributions of this paper are as follows: Design a generic and semantic model for CSN participants
based on Cyber-Anima; Develop a model learning technique to handle
heterogeneous and non-structured information resources;
 Propose an ADI-based mechanism to drive the tradingstyled connection and interaction between modeled CSN
participants.
The rest of the paper is organized as follows: Section 2
discusses existing related work and highlights the key
contributions from this study of work. Section 3 provides adetailed elucidation of the generic and semantic model for CSN
and its learning technique. Section 4 focuses on the mechanismof connection and interaction between models in CSN. Section
5 develops a prototype system based on the modeled CSN andevaluates the performance. Section 6 concludes the paper andprovides a summary of future work.
II. RELATED WORKThe researches on cyber-physical system have been quite
popular recently. Using counterpart to reflect realistic
individuals is the first step to modeling a cyber-physical system.
To this end, several concepts have been proposed. Robinson
proposed an embodiment of individual as a means of identity
signaling and as a medium of online interaction, called
Cyberself [1]. Besides Cyberself, Robinson also presents
Cyberpersona and Cyberbody as the counterparts of persona
and body in cyberspace. Wen et al proposes Cyber-Individual,with a short term ‘Cyber-I’, to create a unique, digital,
comprehensive description for every real person being in the
cyberspace including Human’s social context, mood, temper,
physical status and so on [2]. Cyber-I is a real individual’scounterpart in cyberspace. However such researches are still inconceptual phase and cannot be applied to modeling of cyberphysical system directly.
Eskins et al presents a framework of modeling Cyber-Human
Systems (CHS) which divides CHS elements into four types:
components, participants, processes and tasks [9]. Each type of
CHS elements is related with a set of properties. The
framework of modeling CHS provides a structured andquantitative means of analyzing cyber security problems whose
outcomes are influenced by human-system interactions. Hencesuch a framework can only be applied to a specific domain andits modeling of CHS lacks of generic and semantic.
In our previous work, we propose a semantic model, called
Cyber-Anima, to reflect individuals in the cyberspace andreason their desires and intentions for recommender systems [3].

Cyber-Anima composing of physiology, belief, character,knowledge, experience and context represents the intrinsic andextrinsic factors of user humanity. We perform a study on thesemantic modeling approach based on the Cyber-Anima andpropose a multi-phase, multi-task and multi-technology
mechanism for learning and populating the terms, concepts,
taxonomy relations, non-taxonomy relations and axioms in the
Cyber-Anima. We present the Cyber-Anima based reasoning
approaches to derive individual user desire and intention, whichleverage the existing or past individual desires and individual
intentions with a template to reason over with. It can produce
individual desire-oriented user profile and individual intentionoriented recommendation based on Cyber-Anima model. Albeit
Cyber-Anima is proposed for enhancing the performance ofrecommender system, its generic and semantic modelingapproach, model learning mechanism and ADI-based reasoning
framework can be extended to modeling of cyber-physicalsystem, especially for CSN.
III. MODEL DESIGNING AND LEARNING BASED ON CYBERANIMA
The first two basic challenges in modeling of CSN are: (1)how can we design a generic model for all kinds of CSN
participants? (2) how can we build a semantic model withvarious heterogeneous and non-structured information
resources? In this section, we introduce a generic and semantic
model for CSN participants based on Cyber-Anima and themodel learning technique to handle heterogeneous and nonstructured information resources.
A. Model Designing for CSN
CSN is a large-scaled, open-styled, self-organized andecological intelligent network. The participants of CSN arecounterparts of supply and demand sides in realistic trading
activities or interactions, which can be divided into three
categories: person, enterprise and administrative department.
Figure 1 illustrates the basic formation of CSN in which all
categories of participant connect and interact mutually in a
trading style, meanwhile service and information can beproduced and consumed by all the participants.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

94

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig 1. Basic Formation of CSN

In the light of aforementioned characteristics, a model which
can semantically and generically represent all three categories
of CSN participant should be designed with cross-disciplinary
perspective. In particular, each category of CSN participant
relates to both general and specific disciplines, shown in Table
1.
TABLE I.

Category
PersonCATEGORY OF CSN PARTICIPANT AND ITS RELATEDDISCIPLINES

GeneralDisciplinesociology,
philosophy,
anthropology

Specific Discipline

biology, medical science,
psychology
Enterpriseorganizational behavior,economics, management
science,
operations
research, statistics
Administrative
publicadministration,
Department
politics,
economics,
management
science,statistics
In our previous work[3], we propose a semantic model, called
Cyber-Anima, to reflect individuals in the cyberspace andreason their desires and intentions for recommender systems. In
light of the equivalence between ‘individual’ and ‘person’ in ourprevious work and this work respectively, CSN person can bemodeled by Cyber-Anima. Without the loss of generality, weformalize CSN participants based on Cyber-Anima as tuple 𝒜
with three attributes, as explained below.
Definition 1. (CSN participant Cyber-Anima model) A CSN
participant is modeled based on Cyber-Anima as tuple 𝒜 withthree attributes:
𝒜 ∷=< 𝐼𝐷, 𝐷𝐼 , 𝐷𝐸 >where:- 𝐼𝐷 is the identity of model;- 𝐷𝐼 is the set of intrinsic dimensions, which includes
physiology, belief, character, experience and
𝐼
𝐼
𝐼
𝐼
𝐼
knowledge, namely 𝐷𝐼 ∷= {𝐷𝑃𝐻
, 𝐷𝐵𝐸
, 𝐷𝐶𝐻
, 𝐷𝐸𝑋
, 𝐷𝐾𝑁
};
𝐸
- 𝐷 is the extrinsic dimension.
In accordance with definition 1, the intrinsic and extrinsic
dimensions of CSN person Cyber-Anima model are designedwith cross-disciplinary perspective shown in Table 2. CSN
person model has five intrinsic dimensions including
physiology, belief, character, experience and knowledge, and an
extrinsic dimension which is context.
TABLE II.

Dimension
Physiology
𝐼(𝐷𝑃𝐻)
Belief𝐼(𝐷𝐵𝐸)

Character
16PF, Big Five, psychology, sociology𝐼
psychogenic needs(𝐷𝐶𝐻 )
Experience life, work, study, psychology, sociology
𝐼
trade
(𝐷𝐸𝑋
)
Knowledge discipline, skill
sociology, psychology
𝐼(𝐷𝐾𝑁)
Context
heredity,group, biology,sociology,𝐸
cultureanthropology(𝐷 )Dimension is the maximum granularity composition of asemantic model. Formally, we define a dimension of CSN
participant Cyber-Anima model as tuple 𝐷 with two attributes,
as explained below.
Definition 2. (Dimension) A dimension of CSN participant
Cyber-Anima model is modeled as tuple 𝐷 with two attributes:
𝐷 ∷=< 𝐶, 𝑅 >where:- 𝐶 is the set of concepts, and 𝐶 ∷=< 𝑀𝐶, 𝐴𝑋𝐶 > ,where 𝑀𝐶 is the marker of concept, and 𝐴𝑋𝐶 is theset of concept axioms;- 𝑅 is the set of relations, and 𝑅 ∷=< 𝑀𝑅, 𝐶𝑃,
𝐴𝑋𝑅 >, where 𝑀𝑅 is the marker of relation, 𝐶𝑃 is thepair of concept, and 𝐴𝑋𝑅 is the set of relation axioms.
To clarify the representation of dimensions, we give eachdimension several classifications to distinguish their concepts
with different perspectives, e.g. physiology dimension of CSN
person Cyber-Anima model has two classifications of disease
and physique distinguished in perspectives of biology andmedical science respectively. Accordingly, we can extend CSN
person Cyber-Anima model to model the other two categories
of CSN participant with identical dimensions but different
classifications of concept. Table 3 and 4 show the dimensionsof CSN enterprise Cyber-Anima model and CSN administrative
department Cyber-Anima model with their classifications andperspectives.
TABLE III.

DIMENSIONS OF CSN ENTERPRISE CYBER-ANIMA MODEL

Dimension
Physiology
𝐼
(𝐷𝑃𝐻
)

Classification
finance, ownership,organizational
structure

Belief𝐼(𝐷𝐵𝐸)

values, corporate
identity, corporate
philosophy
brand,
corporate
cultureCharacter𝐼(𝐷𝐶𝐻)

DIMENSIONS OF CSN PERSON CYBER-ANIMA MODEL

Classificationdisease, physiqueworld view, values,
self-schema

Perspective
biology,sciencephilosophy,
psychology

medicalExperience
𝐼
(𝐷𝐸𝑋
)

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

R&D,
marketing,management,
production, trade

Perspective
organizational
behavior, economics,
management science,statistics
economics,
management science,sociology, philosophy
organizational
behavior, management
science,sociology,
philosophy
organizational
behavior, management
science,
operations
research, sociology

95Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Knowledge
𝐼(𝐷𝐾𝑁)

production
management,
operation

Context(𝐷 𝐸 )

administration,market,society,cultureTABLE IV.

Dimension
Physiology
𝐼
(𝐷𝑃𝐻
)

economics,
management science,
operations research,statistics
economics,
management science,statistics, sociology,
anthropology

DIMENSIONS OF CSN ADMINISTRAIVE DEPARTMENT CYBERANIMA MODEL

Classification
finance, system

Perspective
public administration,
politics, management
science, statistics
Beliefvalues,political public administration,𝐼
conviction,
self- politics,
sociology,(𝐷𝐵𝐸)
positioning
philosophy
Character
styleofwork, public administration,𝐼administrative
politics, management(𝐷𝐶𝐻)
culturescience, sociology
Experience administrative
public administration,𝐼affair , function politics,
economics,(𝐷𝐸𝑋)
adjustment, tradesociology
Knowledge administration,
public administration,
𝐼
service,control
politics,
economics,(𝐷𝐾𝑁 )
management science
Context
politics,
society, politics,
economics,
𝐸
culturestatistics,
sociology,(𝐷 )anthropology
Based on Cyber-Anima, all three categories of CSN
participant have been modeled generically and semantically. In
maximum granularity level, such models have uniform
dimensions: five intrinsic dimensions including physiology,
belief, character, experience and knowledge, and an extrinsic
dimension which is context. Albeit dimensions of CSN
participant Cyber-Anima model are identical in semantic
structure, they are distinguished by different classifications
generated in different perspectives. We use these classifications
as an explanatory tool, so these classifications of model
illustrated above are not exclusive and varied with weights ofblended perspectives.
B. Model Learning Technique
In light of Definition 2, each dimension of CSN participant
Cyber-Anima model is constituted by concepts, relations andcorresponding axioms. On account of perspective difference, we
confront heterogeneous and non-structured information
resources during model learning. For instances, when learning
character dimension of CSN person Cyber-Anima model,
corresponding information resources are usually blogs, tweets,
dialogue, speech and even pictures. Compared to homogeneousand structured, heterogeneous and non-structured information
lacks of artificial processing but approaches the original
information form. In CSN, heterogeneous and non-structured
information resources are dominant and required techniques
based on natural language processing to deal with.

In our previous work [3], we propose a semantic network,
called ConceptNet.Cyber-Anima, which is an extension of
famous MIT proposed semantic network, ConceptNet [4].
ConceptNet.Cyber-Anima inherits of node and edge elementsfrom ConceptNet graph structure to represent concept andrelation of CSN participant Cyber-Anima model, and extends
assertion element of ConceptNet by adding attributes including
restrict and object to represent axioms of concept and relation.
In this way, ConceptNet.Cyber-Anima can represent any
concept, relation and corresponding axiom of CSN participant
Cyber-Anima model. The maintenance of ConceptNet.CyberAnima is a job of crowdsourcing by researcher in various
disciplines including biology, medical science, psychology,sociology, statistics, economics, management science, politics,
philosophy, anthropology and etc. The more nodes, edges andassertions are acquired, the more ConceptNet.Cyber-Animafacilities jobs of natural language processing.
Rely on ConceptNet.Cyber-Anima, we propose a model
learning technique based on natural language processing which
has a threefold process: (1) Recognize concepts frominformation resources according to the node set ofConceptNet.Cyber-Anima; (2) Extract relations andcorresponding axioms by the edge set of ConceptNet.CyberAnima along with their assertions; (3) Populate the CSN
participant Cyber-Anima model with extracted concepts,
relations and corresponding axioms.
The difficulties of concept recognition are word segmentation,
NER (Named-Entity Recognition) and POS (Part Of Speech)
annotation. For this purpose, n-gram model based segmentation
method [5], CRF (Conditional Random Field)-based NER
method [6] and HMM (Hidden Markov Model)-based POS
annotation method [7] can be employed. In the phase of relation
extraction, we tackle with word-sense disambiguation by mutual
information based word-sense disambiguation method [8]. After
concepts, relations and corresponding axioms have been
extracted, population phase of Cyber-Anima model starts by
those population algorithms proposed in our previous work [3].
IV. CONNECTION AND INTERACTION OF CSN PARTICIPANTCYBER-ANIMA MODEL
The last basic challenge in modeling of CSN is: (3) how can
we define a mechanism for trading-styled connection andinteraction between CSN participants? After modeling CSN
participants based on Cyber-Anima, we explore the mechanismof connection and interaction generating between CSN
participants Cyber-Anima model in this section.
A. A CSN Based on Cyber-Anima Model
Figure 1 shows a basic formation of CSN in which all
categories of participant connect and interact mutually in a
trading style. In last section, three categories of CSN participant
are model based on Cyber-Anima generically and semantically.
Rely on CSN participant Cyber-Anima model, we formalize
CSN as tuple 𝑁 with three attributes, as explained below.
Definition 3. (CSN) A CSN based on Cyber-Anima model ismodeled as tuple 𝑁 with three attributes:
𝑁 ∷=< [𝒜1 , 𝒜2 , 𝒜3 … 𝒜𝑛 ], 𝐶𝑁, 𝐼𝑇 >© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

96

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

where:- [𝒜1 , 𝒜2 , 𝒜3 … 𝒜𝑛 ] is the set of Cyber-Anima models
in CSN;- 𝐶𝑁 is the set of connections between CSN participant
Cyber-Anima models;- 𝐼T is the set of interactions between CSN participant
Cyber-Anima models.
In light of Definition 3, CSN composes of Cyber-Animamodels which represent participants, connections andinteractions among Cyber-Anima models. In essence, theinteraction in CSN is a trading for producing and consuming
information or service. The connection has weight to represent
its intensity, and it is both prerequisite and consequence for theinteraction. There are four truths between connection andinteraction: (1) A connection must be generated before two
Cyber-Anima models interacts mutually; (2) Once generated,connection may have a longer lifecycle than corresponding
interaction; (3) Via each connection, more than one interaction
can be generated; (4) Only one connection exits between two
Cyber-Anima models. For instance, when a CSN person CyberAnima model intends to interact with a CSN enterprise CyberAnima model, a connection is generated between these two
Cyber-Anima models. In case of the interaction finishes, theconnection may preserve for subsequent interactions or remove
simultaneously.
B. ADI-based Connection and Interaction Generating
So far, generically and semantically modeling of CSN has
achieved based on Cyber-Anima. To build a large-scaled, openstyled, self-organized and ecological intelligent network, there
are two remaining problems required to be solve: (1) how to
generate and remove connection autonomously; (2) how to
generate interaction for optimal matching pair of producer andconsumer against information or service.
In our previous work [3], we propose an ADI-based reasoning
approach for recommender system. By converting
aforementioned two remaining problems to recommendation
problem of connection and interaction, we employ such areasoning approach to intelligently generate connection andinteraction of CSN. To this end, we formalize therecommendation problem of connection and interaction as
follows:
Definition 4. (Recommendation problem of connection andinteraction) For each CSN participant 𝑝 ∈ 𝑃 has a desire 𝒹𝑡𝑤𝑖 ∈𝒟𝑝 and an intention 𝒾𝑡𝑤𝑖 ∈ ℐ𝑝 on time point 𝑡𝑖 ∈ 𝑇 in possible
world 𝑤 ∈ 𝑊, we want to choose such connection 𝑐𝑛′ ∈ 𝐶𝑁, orsuch interaction 𝑖𝑡 ′ ∈ 𝐼𝑇 that 𝒾𝑡𝑤𝑖 = 𝛿(𝑚𝑎𝑥𝑒𝑥𝑝𝑣𝑎𝑙, 𝒹𝑡𝑤𝑖 ).
where:- 𝑊 is a set of possible worlds, and for each 𝑤 ∈ 𝑊, wehave 𝑤 ∷=< 𝑇𝑤 , 𝐴𝑤 , 𝐸𝑤 >. 𝑇𝑤 ⊆ 𝑇 is a set of time
points in possible world 𝑤; 𝐴𝑤 is a binary relation oftime points, namely ∀𝑖(𝑡𝑖 , 𝑡𝑖+1 ) ∈ 𝐴𝑤 ; 𝐸𝑤 is a set of
event type in possible world w;-

𝒟𝑝 is the possible desire world of CSN participant 𝑝 ∈𝑃, and 𝒟𝑝 ⊆ 𝑊 × 𝑇 × 𝑊;
ℐ𝑝 is the possible intention world of CSN participant𝑝 ∈ 𝑃, and ℐ𝑝 ⊆ 𝑊 × 𝑇 × 𝑊;𝛿 is the deliberation function;
𝑚𝑎𝑥𝑒𝑥𝑝𝑣𝑎𝑙 is the maximizing expected value.

In light of Definition 4, generating an optimum connection orinteraction for some CSN participant in a specific time point is
a twofold process shown in Figure 2. Firstly, we reason thedesire 𝒹𝑡𝑤𝑖 of the participant based on its Cyber-Anima model.
Secondly, we reason and find such a connection 𝑐𝑛′ orinteraction 𝑖𝑡 ′ that 𝒾𝑡𝑤𝑖 = 𝛿(𝑚𝑎𝑥𝑒𝑥𝑝𝑣𝑎𝑙, 𝒹𝑡𝑤𝑖 ).

Fig 2. Process of Generating Conncetion or Interaction for CSN participant

For the purpose of reasoning the desire of CSN participant,we acquire the CSN participant Cyber-Anima model 𝒜𝑝 on
current time point 𝑡𝑖 , then employ desire reasoning algorithmψ𝐷 to reason each event type 𝑒𝑡𝑤𝑖 of participant’s desire 𝒹𝑡𝑤𝑖
along with its payoff value 𝑃𝐴𝑌𝑂𝐹𝐹(𝑒𝑡𝑤𝑖 ) on current time point
𝑡𝑖 based on Cyber-Anima model 𝒜𝑝 . The desire reasoning
algorithm ψ𝐷 can be a hybrid of knowledge-based, contentbased and collaborated filtering algorithm for preference
extraction. The participant’s desire 𝒹𝑡𝑤𝑖 illustrates who it
expects to connect and what service or information it expects
to interact for.
Next, we employ intention reasoning algorithm ψ𝐼 to reason
the participant’s intention 𝒾𝑡𝑤𝑖 which illustrates the probability
𝑃𝑅𝑂𝐵(𝑒𝑡𝑤𝑖 ) and payoff value 𝑃𝐴𝑌𝑂𝐹𝐹(𝑒𝑡𝑤𝑖 ) towards eachcandidate to be connect and interact on current time point 𝑡𝑖
based on participant’s desire 𝒹𝑡𝑤𝑖 . The intention reasoning
algorithm ψ𝐼 can be a hybrid of knowledge-based, contentbased and collaborated filtering algorithm for utility calculation.
Finally, we sum up the utility value 𝑃𝐴𝑌𝑂𝐹𝐹(𝑒𝑡𝑤𝑖 ) ∙𝑃𝑅𝑂𝐵(𝑒𝑡𝑤𝑖 ) of each candidate in the participant’s intention 𝒾𝑡𝑤𝑖 ,and sort for the optimal connections or interactions.
In light of CSN’s essential logics of openness, federation,
cooperation and sharing, we can train the desire reasoning
algorithm ψ𝐷 and the intention reasoning algorithm ψ𝐼 with
CSN participant Cyber-Anima models and their history
connections and interactions, which ensures the performance ofgenerating connection and interaction.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

97

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

V. PROTOTYPE IMPLEMENTATION AND EVALUATIONCSN is proposed for the purpose of implementing the future
networked system of industrial operation. Compared to
traditional systems, the future networked system of industrial
operation has four essential characteristics: proactive
personalized consumption, direct centralized distribution,
intelligent decentralized manufacture and ecological selforganized system. The future networked system of industrial
operation has its own operation platform, called new generation
E-commerce platform, which is different from traditional Ecommerce platform in: (1) directly connection with consumer
end and producer end directly; (2) intelligent trade matching; (3)technical support by cloud computing, IOT (Internet Of Things),
big data; (4) personalized portal.
In last sections, we has generically and semanticallymodeled the CSN based on Cyber-Anima and present an ADIbased reasoning approach for generating trading-styled
connection and interaction between CSN participant CyberAnima models intelligently. In this section, we implement the
prototype system of new generation E-commerce platform relyon the modeling of CSN. The prototype system composes offour function modules: (1) Who am I; (2) My demand; (3) My
supply; (4) My space.
The new generation E-commerce platform is a personalized
portal for every unique CSN participant, which means any CSN
participant has his own independent portal. Intuitively, we calleach CSN participant the owner of his platform. CSN participant
Cyber-Anima model drives the platform for the realistic
counterpart of its owner in CSN autonomously, meanwhile theowner can operate the platform manually. The platform shows
the 3D virtual image of its owner and entrances of the fourfunction modules at a first glance shown in Figure 3.

Fig 4. ‘Who am I’ Function Module‘My demand’ function module tackles with all demands of its
owner rely on owner’s Cyber-Anima model and ADI-based
connection and interaction generation mechanism, coveringpull-styled demand and push-styled demand. Any connection orinteraction in CSN is trade-styled and associate with a demandpart and a supply part, so the owner is a demand part in ‘Mydemand’ function module. The owner’s demand can proactively
be launched by himself or passively be elicited by supply side,which are so-called pull-styled and push-styled respectively. No
matter who launches demand, an optimal matching of demandand supply sides can be generated by ADI-based reasoning.
Figure 5 illustrates the matching snapshot for a pull-styled
demand of automobile.

Fig 5. ‘My demand’ Function Module

Fig 3. New Generation E-commerce Platform at a First Glance‘Who am I’ function module is a snapshot of the owner’s
Cyber-Anima model. In this function module, we can overview
interested concepts, relations and axioms of owner’s CyberAnima model to know who he is. Figure 4 shows the ‘Who am
I’ function module of person category owner, including her 3D
virtual image, physique related concepts of physiology
dimension, values related concepts of belief dimension, traderelated concepts of experience dimension and etc.

Relatively, ‘My supply’ function module handles all supplies
of its owner rely on owner’s Cyber-Anima model and ADIbased connection and interaction generation mechanism,
covering pull-styled supply and push-styled supply. As the
supply part in ‘My supply’ function module, the owner’s supply
can proactively be launched by himself or passively be elicitedby demand side, which are so-called pull-styled and push-styled
respectively. No matter who launches supply, an optimal
matching of demand and supply sides can be generated by ADIbased reasoning. Figure 6 shows the process of launching a new
supply by owner.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

98Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

participants. We developed the prototype system of new
generation E-commerce platform to prove the effectiveness andsoundness of CSN modeling.
We intend to continue our research in the following two
directions: (1) improve the model learning techniques for its
robustness; (2) enhance the function of the prototype systemand deploy it to industrial application.
REFERENCES[1]
Fig 6. ‘My supply’ Function Module‘My space’ function module is a visualization of the CSN
from the owners’ perspective. Figure 7 presents a snapshot of a
‘My space’ function module. The node in the center represents
the owner of the platform while other nodes around represent
these directly and indirectly connected CSN participants. Driven
by owner’s Cyber-Anima model and ADI-based connection andinteraction generation mechanism, some CSN participants canbe added to the space of the owner and some ones can beremoved from it.

[2]

[3]
[4]
[5]

[6]

[7]

[8]

[9]

Robinson, L. (2007). The cyberself: The self-ing project goes online,
symbolic interaction in the digital age. New Media & Society, 9, 93-110.
Wen, J.; Ming, K.; Wang, F.; Huang, B. & Ma, J. (2009), Cyber-I: Visionof the Individual's Counterpart on Cyberspace., in 'DASC' , IEEE
Computer Society, , pp. 295-302 .
Shen, J. (2015). A Study on Cyber-Anima Model and Its Application inIntelligent Recommendation: [D]. Fudan Universtiy.
Speer,
R.
(2016):
ConceptNet
5.
Availableat:
http://conceptnet5.media.mit.edu/
Li, M.; Gao, J.; Huang C.; Li, J. (2003): Unsuperised Traning for
Overlapping Ambiguity Resolution in Chinese Word Segmentation. In:
Proceeding of the Second SIGHAN Workshop on Chinese Language
Processing. 11-12 July. Sapporo, Japan. pages 1-7
Leaman, R.; Gonzalez, G. (2008): BANNER: An Executable Survey ofAdvances in Biomedical Named Entity Recognition. Pacific Symposium
on Biocomputing, 13: 652-663
Marshall, I. (1983): Choice of Crammatical Word-class Without Global
Syntactic Analysis: Tagging Words in the LOB Corpus. Computers andthe Humanities, 17: 139-150
Brown, P.; Della Pietra, S.; Della Pietra, V.; Mercer, R. (1991): Wordsense Disambiguation Using Statistical Methods. In: Proceedings of ACL,
pages 264-270
Eskins, D.; Sanders, W. (2011): "The multiple-asymmetric-utility systemmodel: A framework for modeling cyber-human systems", EighthInternational Conference on Quantitative Evaluation of Systems, QEST,
pp. 233-242

Fig 7. ‘My space’ Function Module

After implementing the prototype system of new generation
E-commerce platform, we have invited experts from various
disciplines to review the system and received positive
evaluations to the system and the modeling of CSN behind it.
VI. CONCLUSION AND FUTURE WORKCSN is defined to be a large-scaled, open-styled, selforganized and ecological intelligent network and a ternary
system of information, physics and society, which evolves to thenext generation of internet. CSN composes of various intelligent
federated agents participating as persons, enterprises andadministrative departments which are related by supply anddemand. Interactions between any participant in CSN are
behaviors or activities of trading in some extent. To model CSN,
we have several basic challenges to tackle.
To this end, we presents an approach based on Cyber-Animato model CSN. We extend Cyber-Anima along with its five
intrinsic and one extrinsic dimensions to fit all kinds of CSN
participant. Towards heterogeneous and non-structured
information resources, we developed a model learningtechnique based on natural language processing for semantic
model building and synchronizing. We propose an ADI-based
connection and interaction generation mechanism for CSN© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

99

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Motivation Mechanism of Gamificationin Crowdsourcing Projects *Zhiyuan Zeng

Jian Tang

Tianmei Wang

School of Information
Central University of Finance andEconomics
Beijing, China
zengzy16@163.com

School of Information
Central University of Finance andEconomics
Beijing, China
jiantang@cufe.edu.cn

School of Information
Central University of Finance andEconomics
Beijing, China
wangtianmei@cufe.edu.cn

Abstract—Gamification can be applied in crowdsourcing
projects to encourage and engage participants, yet the
motivation mechanism of gamification is not well studied. This
research aims to investigate the motivational influences ofdifferent categories of game elements and the moderating
effect of task complexity in crowdsourcing projects. Based on
the theory of extrinsic and intrinsic motivation and selfdetermination theory, we proposed a model to depict the effectof four categories of game elements on three types ofmotivation, which may in turn influence user participation. We
also designed a series of lab experiments to collect data andvalidate our research model. The potential contribution andimplications of our research are also discussed.
Keywords—gamification, game element, categorization,
crowdsourcing, motivation mechanism, self-determination theory,
intrinsic motivation, extrinsic motivation, internalization, taskcomplexity

I.

INTRODUCTION

Gamification refers to the use of gameplay mechanism in
non-gaming contexts to encourage desired behavior[1].
Many scholars and practitioners have applied this approachin several contexts such as education, health, project
management, environment protection, outsourcing andsoftware development, etc.[2]. Gamification can benefit
corporations and companies by extending the market share,changing the attitude of users and enhancing the motivation
effect[3]. The application of game elements help managers ororganizers dismantle the obstacles like the disability to attract
users, inadequate participation and low engagement[4], andfinally guarantee that projects or activities hosted by those
corporations and companies could move forward as expected.
Crowdsourcing is one of the contexts where gamification
could be applied, which refers to a new outsourcing approachthat takes tasks as an open call to an undefined, large groupof people[5]. The cooperation and competition traits ofcrowdsourcing projects allow companies to make a full useof knowledge, skills and resources beyond their own staff[6,7], so they can reduce the cost and benefit a lot more[5].
Crowdsourcing relies on the number of participants who
are willing to devote and contribute their time and efforts.

Whether crowdsourcing projects could be operated smoothly
is highly relevant with the ability of those initiators to attract
and obtain adequate participants. Many crowdsourcing
projects have failed because they were unable to gain enoughattention from the public[4]. Gamification is considered as an
approach to attract participants and improve the engagement.
The application of gamification may avoid the situation
where a crowdsourcing project may fail owing to the lack ofparticipation. Hence, how to apply gamification
appropriately to increase participation and engagement has
become a major topic in our research.
Some prior research discussed the application of
gamification in the realm of crowdsourcing[8], yet the extant
literature lacks a systematic understanding of motivation
mechanism of gamification in crowdsourcing projects. Someof the scholars who are interested in the application of
gamification intended to abstract game elements to provide amore complete structure of gamification[4, 9, 10], but theprocess or mechanism via which these game elements canhave an impact on user participation are barely mentioned inthese frameworks; others intended to discuss the
motivational influence of some specific game elements indifferent contexts[11], but the categories of these gameelements having been discussed in these studies are limited.
As a result, which kinds of game elements are more effective
in increasing participation and engagement in crowdsourcing
projects, and whether motivation mechanism of these gameelements vary from one kind to another remains uncovered.
To fill these gaps, we intend to investigate themotivation mechanism of gamification in crowdsourcing
projects. First, we categorized game elements that have beenused widely in crowdsourcing projects. Then, we proposed a
research model that depicts the effect of these gameelements on human motivation, which may in turn influence
user participation. In addition, we are interested in identify
the moderating effects of task complexity on user
motivation. We designed a series of lab experiments tovalidate our research model. Finally, we discussed expected
contribution and implications of our research.

*This research is supported by the National Social Science Fund of China(Grant No. 14AZD045).

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

100Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

II.

LITERATURE REVIEW

A. GamificationGame has been considered as the inspiration for
designing captivating user interface since 1980[2, 12]. Until
now, many scholars and practitioners have applied game
dynamics and elements to different contexts to enhance theuser engagement. The most frequently mentioned and usedgame elements within these contexts are narratives, fantasy
world, point, level, status, badge, leaderboard, challenge,
countdown, deadline, etc.
A common definition of gamification embraces theconcept of using game elements and design techniques in a
non-game setting, often with the end-goal of shaping the user
behavior[13]. Thanks to the high adaptability of gamification,
many non-gaming contexts, such as education, health,project management, environment protection, crowdsourcing,
software development, have been used the idea ofgamification[2]. Many corporations and companies haveused the concept of gamification and game elements to help
them extend the market share, alter the attitude of users andenhance the motivation effect[3] to maintain a highengagement and obtain loyalty from their users. However,most initiators cannot meet their expectations because gameelements were not implemented appropriately. Hence, how
to select proper game elements under varied contexts has
gradually gained attention among scholars and practitioners,
so is the motivation mechanism of gamification.
Despite of the wide application of gamification, thediscussion of the inner mechanism of gamification is still
open. One of the most frequently used framework of
gamification is Hunicke, et al. [9]’s MDA (MechanicDynamic-Aesthetic) framework where game elements havebeen abstracted into three parts: game mechanics, game
dynamics and game aesthetics. Mechanics refer to thefundamental settings and rules including point, level, trophy,
badge, achievement, leaderboard and virtual gift, etc.;
dynamics refer to the motives which can lead to someparticular emotions, such as reward, status, achievement, self
expression, competition and altruism[14]; aesthetic, which issubstituted sometimes as emotion[10], refers to the feelingsand emotions evoked by game dynamics. MDA framework
have brought out a hierarchical picture of game elements.
However, MDA framework emphasized a lot on the abstractof the components of gamification while they left out the parton the motivation process of these components. Besides,
while this framework takes gamification as a whole toanalyze, the distinct ways in which those game elements may
function are not discussed.
Some scholars put their emphasis on the effect of somespecific game elements. Hanus and Fox [15] applied badge
and leaderboard to the context of education and discoveredthat these two kinds of game element may decrease themotives and final scores of students. Mekler., et al. [16] used
point, level and leaderboard in a tagging experiment andfound that these three game elements have no influence on
individuals’ intrinsic motivation, but will change users’
ultimate behavior by affecting extrinsic motivation. Thesestudies show that the impacts of game elements may vary

under different contexts. Hence, we should seek for a moredetailed explanation. In addition, game element categories
that have been mentioned in these researches are limited. The
effect and mechanism of game elements such as narratives,
progress bar are not well studied.
To address these issues, we discuss the motivation
mechanism and process of different categories of gameelements in crowdsourcing projects, and posit a theoreticalmodel that can uncover the relationships among varied gameelements, user motivations and user participation.
B. Crowdsourcing
Crowdsourcing is one of the contexts where the conceptof gamification has been frequently used, which refers to a
new outsourcing approach that takes tasks as an open call toan undefined, large group of people[5]. Knowledge seekers
can post their task requirements in some kind of a platform
and then problem solvers can participate in these tasks togain rewards provided by knowledge seekers after they
manage to meet all the requirements requested by knowledge
seekers[17]. By crowdsourcing projects, managers andorganizers are able to make a better use of knowledge, skills
and resources[6, 7], and reduce the cost[5].
However, as the cost on operating a crowdsourcing
project is relatively low, and there is an increasing number ofplatforms supporting the crowdsourcing projects, thecompetition among those crowdsourcing project initiators tocall up participants become fierce. Under this circumstances,many crowdsourcing projects failed as an end owing to theinitiators’ disability to absorb enough attention from thepublic[4]. As user engagement is of great necessity foraccomplishing a crowdsourcing project, the motives andincentives for those users to participate in crowdsourcing
projects is becoming a topic for relevant scholars.
C. Motivation Theory
Porter and Lawler [18]’s intrinsic and extrinsic
motivation, along with Ryan, et al. [19]’s self-determination
theory are the most applied motivation theories among allthese motivation researches. Zheng, et al. [20] developed a
research model to explain participation in crowdsourcing
contexts based on the theory of extrinsic and intrinsic
motivation; And by analyzing the case of Treadless,Kavaliova, et al. [13] used self-determination theory to
explore how companies can motivate contributions to a
crowdsourcing project.
Intrinsic motivation involves people doing an activity
because they find it interesting and derive spontaneous
satisfaction from the activity itself[21]. In contrast,
extrinsically motivated individuals perform an activity asmeans to achieve some separable objective or personal
benefits, so satisfaction comes not from the activity itself but
rather from the extrinsic consequences to which the activity
leads[22]. In Ryan, et al. [19]’s self-determination theory,
individual motivation are further categorized as intrinsic
motivation, internalized extrinsic motivation and extrinsic
motivation. Internalization is defined as people taking invalues, attitudes, or regulatory structures, such that theexternal regulation of a behavior is transformed into an© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

101

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

internal regulation and thus no longer requires the presenceof an external contingency. Besides, Zheng, et al. [20] put
forward an idea in his research on the motives of a
crowdsourcing project that job characteristic is of great
importance in influencing participants’ behavior, which may
act as regulatory variables to enhance or weaken the effect of
intrinsic motivation. This idea is also the kernel of Hackman
and Oldman [23]’s job design theory.
TABLE I.
Motivation
Type
Extrinsic
motivation

MOTIVES IN CROWDSOURCING PROJECTSFrequency

MotivesInternalized
extrinsic
motivation

Reference

Reward[13, 20, 24-27]

Recognition[13, 20, 25]High
New skills
acquirement
New job
opportunity
Middle[25, 27][27]

Relationship[13, 25]

Self-efficacy[13, 25, 26]

Achievement
Socialconsideration[13, 25][25, 27]

Fun
Intrinsic
motivation

Curiosity
Low[13, 25]
Exploratory
Addiction

TABLE II.

MOTIVES RELEATED TO GAME ELEMENTS IN
CROWDSOURCING PROJECTS

Motivation Type

Motives

ReferenceIII.

RESEARCH MODEL AND HYPOTHESIS DEVELOPMENT

A. Game element categorizationGame elements which have been frequently used incrowdsourcing are point/score, leaderboard/ranking,badge/achievement, level, reward, progress bar, mission,
feedback, storytelling and virtual territories [8]. Based on
the characteristics of these game elements and thesimilarities among them, we further categorized these gameelements into four categories: reward, recognition, progress
and fantasy. Reward refers to game elements that provide
rewards for some specific behavior or achievements ofparticipants, including tangible monetary and gifts as well as
virtual goods and game properties. Recognition refers togame elements that support mechanism where actions ofparticipants to accomplish the task are converted to somekind of countable values to be compared with, including
point (/score), leaderboard (/ranking), badge (/achievement),
and level. Progress refers to game elements that provide
information on the progression of an ongoing task on thepremise that the accomplishment of a specific project isdivided into several submissions or stages. Fantasy refers togame elements that create virtual circumstances, such as
storytelling or virtual territory.

Reward[13, 20, 24-27]

Recognition[13, 20, 25]

Self-efficacy[13, 25, 26]

The research model based on the categorization
mentioned above is shown as figure 1.

Achievement[13, 25]

Fig. 1. A proposal research model

Extrinsic motivation
Internalized
extrinsic motivation

which means these motives are difficult to be enhanced bygame elements. “Relationship building” is related with thefunction of crowdsourcing platform. If the platform is pretty
powerful in supporting social communication like providing
functions like liking others, commenting, sharing to otherSNS, creating a forum/group, this kind of motives will beenhanced. However, this can also be difficult for any gameelement to achieve. Hence, we exclude motives like “skills
development”, “professional development”, “social reason”
and “relationship building” when we discuss which motives
can be enhanced by those game elements (see table 2).

Fun
TASK
COMPLEXITY

Curiosity[13, 25]
REWARD

H1

RECOGNITION

EXTRINSIC
MOTIVATION
H8c
H8c

H2a

3a
H
H2
b

H3b

USER
PARTICIPATION

H5

INTERNALIZEDEXTRINSIC
MOTIVATION

H7

PROGRESS

H6

H8b

Besides, among all these motives listed above, “skills
development”, “professional development”, and “social
reason” (community building, social contribution, etc.) aredetermined by the context and purpose of the project itself,

H8 H8aa

Addiction

D. Motives in Crowdsourcing
Under the drive of these theories, there are a number ofmotives on crowdsourcing projects having been discovered
through various researches (see table 1). Among all thesemotives, “reward” is one of the most important one, which
could be found in all motive lists of distinct crowdsourcing
projects [13, 20, 24-27], “Recognition” appeared just as often
as “reward”. However, internalized extrinsic motivation andintrinsic motivation is relatively less found in crowdsourcing
projects [13, 20, 25].

aExploratory

H8

Intrinsic motivation

FANTASYH4

INTRINSIC
MOTIVATION

B. Motivation Mechanism
Reward refers not only to some tangible monetary or
gifts, but also to some virtual goods or properties. This kindof game element is unable to improve the enjoyment itself,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

102

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

which means it cannot enhance intrinsic motivation thatparticipants might feel. Once this kind of reward is cancelled,individuals won’t continue their behavior any longer just outof their habits[21].
However, no matter it is tangible money or gifts, orvirtual goods or properties, they are still of probability to
become the sake for participants to take part in a project.
Therefore, we hypothesize
Hypothesis 1. Reward-kind game elements are positively
associated with extrinsic motivations
Humans make ability judgments about the self and others
via comparison[28]. Festinger [29]’s social comparison
theory predicts that individuals compare themselves to others
in order to validate opinions, make judgments, and reduce
uncertainty. As an intuitive comparison result,leaderboard/ranking could help individuals ascertain thelevel of their competence. The rise of ranking is no doubt asymbol to recognize the improvement of someone’s ability.
Compared with ranking, game elements like points/score,level are not able to present individuals’ relative position in
the whole picture. However, as the basis of ranking andleaderboard, the value of points and levels can be a
representative of someone’s ability. Badge or achievement isinvolved when individuals’ score or level reached someextent, which can be used as a certificate for some kind of a“quality change” and recognition for someone. Hence,
recognition-kind game elements like ranking, scores, levels,
badges, and achievements can recognize the behavior or
improvement of individuals from different perspectives, meetthe psychological need of recognition. Topper ranking,higher scores and levels, more badges or achievements could
be considered as purpose or incentives for participants totake the activity enthusiastically. In another word, this kindof game elements can enhance participants’ extrinsic
motivation. In addition, the improvement of one’s ranking,the increase of scores and levels, the acquisition of new
badges, the fulfillment of a new achievement could provide
participants with a sense of achievement, make them realize
their accumulation and progress, which can help to turn sucha behavior stimulated by outer purpose into an internal
regulation. Therefore, by enhancing the sense of achievement,
game elements that are supposed to provide recognition
could internalize the extrinsic motivation. Based on thereasons above, we hypothesize
Hypothesis 2a. Recognition-kind game elements are
positively associated with extrinsic motivation
Hypothesis 2b. Recognition-kind game elements are
positively associated with internalized extrinsic motivation
Another kind of game elements that could provide
participants with a sense of achievement is the kind offering
the participants the information on his/her progress. The
common game elements of this kind are progress bar andsub-missions. Progress bar can take every efforts participants
put into account and turn it into a visible bar, so thatparticipants are able to get aware of how much have been
done as well as how much to be done in real-time, while thedesign of sub-missions or challenges enable participants toacquire a sense of satisfaction from every tiny fulfillment or
accomplishment until they finally meet all the acquirement
and finish their job.
Self-efficacy is defined as one’s belief in one’s ability toperform a task[30]. Vroom [31]’s expectancy theory has
indicated the positive relationship between self-efficacy andthe effort they are likely to put. In another word, higher selfefficacy participants feel leads to more efforts individuals
may expend. In spite of the fact that most crowdsourcing
projects are lack of challenges and difficulties, there is still a
possibility that participants feel so impatient andunmotivated. Many participants choose to give up owing to
the lack of information on how much time and efforts theymust put to reach the finishing line. However, game elements
designed to provide such kind of information can assist
participants in ascertaining where they are, how much effort
and time are still needed and give them the confidence of
their ability to accomplish the job, which in another word,
can enhance their self-efficacy.
On the one hand, progress-kind game elements arecapable of providing goals for participants to achieve andgiving constant recognition of their effort, by which canenhance extrinsic motivation of participants. On the other
hand, these game elements can provide information on one’s
progress that can lead to a sense of achievement as well as
enhance participants’ self-efficacy and eventually increaseinternalized extrinsic motivation. Hence we hypothesize
Hypothesis 3a. Progress-kind game elements are
positively associated with extrinsic motivation
Hypothesis 3b. Progress-kind game elements are
positively associated with internalized extrinsic motivation
Csikszentmihalyi [32] has brought up the theory of flow
to explain the acceptance intention and behavior ofindividuals toward some specific technology. Flow is defined
as a state in which people are so involved in something that
nothing else seems to matter. What’s more important is thatwhen individuals are immersed in the flow provided by some
object, they are more easily to feel a sense of enjoyment andsatisfaction. In another word, intrinsic motivation will beenhanced in this state.
Fantasy-kind game elements such as storytelling andvirtual territory are supposed to make participants inundatedwith the information flow. Such game elements could meet
all four dimensions brought up by Trevino and Webster [33]which are used to describe the characteristics when
individuals are in the state of flow. First, participants could
get a new clue or feedback when they have conducted any
specific activities, which makes the participants feel like they
are the one who are controlling the process of the wholeproject; The constant feedback is more likely to grab
participants’ attention; Participants’ curiosity may be aroused
considering they might want to know what will happen next;
And ultimately the constant information flow of the virtual
story or territory could provide participants with a sense ofenjoyment and satisfaction, meanwhile users’ intrinsic
motivation get enhanced. Hence, we hypothesize© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

103

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Hypothesis 4. Fantasy-kind game elements are positively
associated with intrinsic motivation
C. Intrinsic Motivation, Extrinsic Motivation andInternalized Extrinsic Motivation
Intrinsic motivation involves people doing an activity
because they find it interesting and derive spontaneous
satisfaction from the activity itself[34], which has an
influence on whether the application of gamification would
meet the expectation. Those companies that figure out how
to effectively use gamification to amplify the intrinsic
motivations of their employees, fans, and customers will
have a lasting competitive edge in their markets[2]. Intrinsic
motivation is one of the cardinal incentives for participants totake part in crowdsourcing projects, as participants mightchoose these tasks out of the enjoyment they might gain in
the process of overcoming the challenge and solve theproblem[20]. In a word, the effect of intrinsic motivation has
obtained a wide acceptance from scholars and practitioners.
Therefore, we hypothesize
Hypothesis 5. Intrinsic motivation is positively associatedwith user participation
Extrinsic motivation is the motivation to work forsomething apart from and external to the work itself, such as
reward or recognition from other people[20]. Extrinsic
motivation requires an instrumentality between the activity
and some separable consequences, so satisfaction comes notfrom the activity itself but rather from the extrinsic
consequences to which the activity leads[21]. Reward andrecognition are most common extrinsic motivates referring to
the context of crowdsourcing[13, 20, 24-26], which are both
capable of becoming a purpose for individuals to motivatethemselves to take some specific actions. Hence, wehypothesize
Hypothesis 6. Extrinsic motivation
associated with user participationispositively

The self-determination theory (SDT) and situational
relevance advocate that the motivational affordances of thegame mechanics are most effective when they work towards
personally meaningful goals and when users are empowered
to design or set their own goals[35]. Deterding, et al. [36]even have a view of point that a mechanic can motivate only
when there is a match between this mechanic and the user’s
profile (needs, values and interests). According to thedefinition of internalization [19], internalized extrinsic
motivations could make people “taking in values, attitudes,
or regulatory structures”, thus could align the personal goals
with the context of the project. Hence, internalized extrinsic
motivation could motivate the engagement behavior ofparticipants, even better than extrinsic motivation. Thus wehypothesize
Hypothesis 7. Internalized extrinsic motivation ispositively associated with user participation
D. Task Complexity as a Moderator
According to Wood [37], task complexity is a function of
the number of distinct acts that must be completed and thenumber of distinct information cues about the attributes of

the task-related stimulus object an individual has to process
when performing a task. Jiang and Benbasat [38] hold theopinion that high task complexity can increase information
processing requirements and demand more resources fromtask executors. The increase of task complexity equals theincrease of efforts and time participants must spend on thetasks, thus they may find the awards or recognition fromothers are not worthy of efforts of participating in the project.
As a result, they are less likely to be lured by extrinsic
motivation. Thus, we think
Hypothesis 8a. The effect of game elements that could
enhance extrinsic motivation will be less prominent whentasks become highly complex.
Besides, as tasks in crowdsourcing projects are relatively
simple, repetitive, and unchallenging, individuals’ attention
to tasks may get diluted when task complexity reaches acertain level [39, 40]. As participants’ attention are no longer
paid to the task itself, they may not be attracted by the
information flow those tasks provide and consequently getout of the state of flow and obtain no fun from the process ofaccomplishing the task. Therefore, we hypothesize
Hypothesis 8b. The effect of game elements that could
enhance intrinsic motivation will be less prominent when thetasks become highly complex.
Things may become different when referring to the gameelements that could enhance internalized extrinsic motivation.
The internalized extrinsic motives we have discussed so far
are self-efficacy and achievement. As simple tasks that could
be done in no time, the progress bar may not play asignificant role as a reminder. However, with the increase of
the task complexity, this kind of game elements is beginningto make a difference. These game elements can remind
participants of how much they have done and how much arestill ahead of them, thus can enhance the self-efficacy andweaken the anxiety led by task complexity. Besides, simple
tasks may not be able to provide participants with enoughsense of achievement, but complex tasks can make it happen.
Thus, we hypothesize
Hypothesis 8c. The effect of game elements that could
enhance internalized extrinsic motivation will be more
prominent when tasks become highly complex.
IV.

RESEARCH METHODOLOGYTo test the hypotheses, a 5 (no game element vs. reward
vs. progress bar vs. virtual territory vs. point and leaderboard)by 2 (different choice sizes of the kinds of clothes) betweensubjects lab experiments will be conducted. We plan todesign a crowdsourcing task where we will act as a clothingcompany who needs to collect opinions and preferences ofcustomers based on their choices of clothes.
In the control group, none of game elements will beapplied and a list of clothing will be shown to theparticipants who are supposed to choose the clothes they
prefer from each kind. In each of our experimental groups, a
specific game element will be embedded, so that the
motivation effect of game elements from different categories
will be examined seperately. The setting of our control group© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

104

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

and experimental groups are shown in table 3. While in thesecond dimension, the influence of task complexity acted asa moderator will be tested by using two different choice sizesof the kinds of clothes which participants are requested tochoose from.
Data will be collected from two sources, the performancedata from system log and the subjective data from a postsurvey. The time those participants have spent on choosing
clothes and the number of clothes they have chosen will be
retrieved from the system and used as the measure for user
participation. Measurement items for several major
constructs - intrinsic motivation, two kinds of extrinsic
motivation and two kinds of internalized intrinsic motivation- will be adapted from the Work Preference Inventory [41],which are used to assess intrinsic and extrinsic motivation.
Two different choice sizes of the kinds of clothes will betaken as the measure of task complexity.
TABLE III.

CONTROL GROUP AND EXPERIMENTAL GROUPS SETTINGSGame
Elements
AppliedControl
group

None

Experimentalgroup A

Experimentalgroup B

Experimentalgroup C

Experimentalgroup D

RewardProgressbar

Virtualterritory

Point andleaderboard

First, this research uncovers the motivation mechanism ofthese game elements, which enables us to clarify the latent
relationship between game elements and user behavior.
Second, we are more likely to find out which game elements
are more effective in enhancing engagement andparticipation in crowdsourcing projects through a series ofexperimental groups. Third, by categorizing game elements
and selecting representative game elements in each category
as our research subjects, we are able to discover motivation
processes of game elements. Forth, as we use taskcomplexity as a moderator, we may be able to identify
whether task complexity influence the application of gameelements, as well as which game elements should be selectedin varied extents of task complexity. Last but not the least, as
our model is proposed within the context of crowdsourcing,
the framework that indicates the interrelationship between
game elements, human motivation and user participation
could also be referred and adapted by other research
investigations that are not limited to crowdsourcing projects.
ACKNOWLEDGMENT

Settings
Clothes are categorized according to theirkinds. A list of clothing will be shown andparticipants are supposed to choose theirfavorite clothes from each kind.
Clothes are categorized according to theirkinds. A list of clothing will be shown andparticipants are supposed to choose theirfavorite clothes from each kind.
Participants will be informed that theymaybe gain a gift for themselves if they
did a good job, which will be assessed by
the time they spent, and the number ofclothes they choose.
Clothes are categorized according to theirkinds. A list of clothing will be shown andparticipants are supposed to choose theirfavorite clothes from each kind. A
progress bar that indicates the progress ofthe task is shown to remind participants of
how much work they have done and still
left.
Clothes are categorized according to theirkinds. Participants will first choose amodel as their virtual “boy/girl friend”.
Then they are requested to dress theirvirtual boy/girl friend in a given territory
by choosing their favorite clothes fromeach kind.
Clothes are categorized according to theirkinds. A list of clothing will be shown andparticipants are supposed to choose theirfavorite clothes from each kind. Once
participants choose one clothes from its
kind, they will obtain a certain amount ofscore that will be summed up to decide
their rankings. Both points of participants
and a leaderboard will be shown to theparticipants in real-time.

This research is supported by the National Social Science
Fund of China (Grant No. 14AZD045).
REFERENCES
[1]
[2]
[3]

[4]

[5]
[6]

[7]

[8]
[9]

[10]

[11]

[12]
[13]

[14]

V.

EXPECTED CONTRIBUTION

Gamification has attracted increasing attention fromresearchers and practitioners in varied contexts. Our research
can make contributions in several ways.

[15]

K. Werbach and D. Hunter, For the win. Philadelphia, PA:
Wharton Digital Press, 2012.
S. Deterding, "Gamification: designing for mitivation," 2012.
J. Hamari and J. Koivisto, "Social motivations to use
gamification: an empirical study of gamifying exercise," in the
21st European Conforence on Information Systems, Utrecht,
Netherlands, 2013.
J. McGonigal, Reality is broken: Why games make us better andhow they can change the world. New York, NY: Penguin Press,
2011.
J. Howe. (2006). The rise of crowdsourcing
A. K. Neyer, A. C. Bullinger, and K. m. Moeslein, "Integrating
inside and outside innovators: A socio-technical systems
perspective," R&D Management Journal, vol. 39, pp. 410-419,
2009.
P. Whitla, "Crowdsourcing and its application in marketing
activities," Journal of Contemporary Management Research,
vol. 5, pp. 15-28, 2009.
B. Morschheuser, J. Hamari, and J. Koivisto, "Gamification incrowdsourcing: a review," 2016.
R. Hunicke, M. LeBlanc, and R. Zubek, "MDA: A formal
approach to game design and game research," in AAAIWorkshop on Challenges in GAME AL, San Jose, CA, 2004.
K. Robson, K. Plangger, J. H. Keitzmann, a. Mccarthy, and L.
Pitt, "Is it all a game? Understanding the principles ofgamification," Business Horizons, vol. 58, pp. 411-420, 2015.
M. Sigala, "The application and impact of gamification funware
on trip planning and experiences: the case of TripAdvisor’s
funware " Electron Markets, vol. 25, pp. 189-209, 2015.
T. W. Malone, "Heuristics for Designing Enjoyable User
Interfaces: Lessons from Computer Games," 1981.
M. Kavaliova, F. Virjee, N. Maehel, and I. A. Kleppe,"Crowdsourcing innovation and product development:
Gamification as a motivational driver," Cogent Business &Management, vol. 3, p. 18, 2016.
J. Simoes, R. D. Redondo, and A. F. Vilas, "A social
gamification framework for a K-6 learning platform,"
Computers in Human Behavior, vol. 29, pp. 345-353, 2013.
M. D. Hanus and J. Fox, "Assessing the effects of gamificationin the classroom: A longitudinal study on intrinsic motivation,social comparison, satisfaction, effort, and academic© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

105

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

[16]

[17]

[18]
[19]

[20]

[21]

[22]

[23]
[24]

[25]
[26]

[27]

performance," Computers & Education, vol. 80, pp. 152-161,
2015.
E. D. Mekler., F. Bruhlmann, A. N. Tuch, and K. Opwis,"Towards understanding the effects of individual gamification
elements on intrinsic motivation and performance," Computersin Human Behavior, pp. 1-10, 2015.
T. X. Liu, J. Yang, L. A. Adamic, and Y. Chen, "Crowd
sourcing with all-pay auctions: A field experiment on Taskcn,"
Management Science, vol. 60, pp. 2020-2037, 2014.
L. W. Porter and E. E. I. Lawler, Managerial attitudes andperformance. Homewood, IL: Irwin-Dorsey, 1968.
R. M. Ryan, J. P. Connell, and E. L. Deci, "A motivational
analysis of self-determination and self-regulation in education,"in Research on motivation in education: The classrom milieu, C.
Ames and R. E. Ames(Eds), Eds., ed New York: Academic
Press, 1985, pp. 13-51.
H. Zheng, D. Li, and W. Hou, "Task design, motivation, andparticipation in crowdsourcing contests," International Journalof Electronic commerce, vol. 15, pp. 57-88, 2011.
M. Gagne and E. L. Deci, "Self-determinatin theory and work
motivation," Journal of Organizational Behavior, vol. 26, pp.
331-362, 2005.
B. Wong-On-Wing, L. Guo, and G. Lui, "Intrinsic and extrinsic
motivation and participation in budgeting: Antecedents andconsequences.," Behavioral Research in Accounting, vol. 22,
pp. 133-153, 2010.
J. R. Hackman and G. R. Oldman, Work redesign. Reading,
MA: Addison-Wesley, 1980.
D. C. Brabham, "Crowdsourcing as a model for problem
solving: An introduction and cases.," The International Journalof Research into New Media Technologies, vol. 14, pp. 75-90,
2008.
J. Fuller, "Refining virtual co-creation from a consumer
perspective," Californic Management Review, vol. 52, 2010.
Y. Sun, N. Wang, C. Yin, and J. X. Zhang, "Understanding therelationships between motivators and effort in crowdsourcing
marketplaces: a nonlinear analysis," International Journal of
Information Management, vol. 35, pp. 267-276, 2015.
D. C. Brabham, "Moving the crowd at Threadless: Motivations
for participation in a crowdsourcing application.," Information,
Communication and Society, vol. 13, pp. 1122-1145, 2010.

[28]

[29]
[30]

[31]
[32]
[33]

[34]

[35]
[36]

[37]
[38]

[39]

[40]
[41]

V. Hoorens and C. Van Damme, "What do people infer fromsocial comparisons? Bridges between social comparison andperson perception," Social & Personality Psychology Compass,
vol. 6, pp. 607-618, 2012.
L. Festinger, "A theory of social comparison processes," Human
Relations, vol. 7, pp. 117-140, 1954.
M. E. Gist, "Self-efficacy: A theoretical analysis of its
determinants and malleability," Academy of Management
Review, vol. 17, pp. 183-211, 1987.
V. H. Vroom, Work and motivation. New York: Wiley, 1964.
M. Csikszentmihalyi, Flow: The psychology of optimal
experience. New York: Harper and Row, 1990.
L. K. Trevino and J. Webster, "Flow in computer-mediatedcommunication: electronic mail and voice mail evaluation andimpacts," Communication Research, pp. 539-573, 1992.
J. Wu and X. Lu, "Effects of extrinsic and intrinsic motivators
on using utilitarian, hedonic, and dual-purposed information
systems: a meta-analysis," Journal of the association for
information systems, vol. 14, pp. 153-191, 2013.
M. Wu. (2011). Sustainable gamification: playing the game forthe long haul.
S. Deterding, D. Dixon, R. Khaled, and L. Nacke, "From gamedesign elements to gamefulness: defining gamification. ," in15th International Academic MindTrek Conference: Envisioning
Future Media Environments, 2011, pp. 9-15.
R. E. Wood, "Task complexity: Definition of a construct,"Organization Behavior and Human Decision Processes, 1986.
Z. Jiang and I. Benbasat, "The effects of presentation formats
and task complexity on online consumers' productunderstanding," MIS Quarterly, vol. 31, pp. 485-500, 2007.
P. Todd and I. Benbasat, "Evaluation the impact of DSS,
Cognitive Effort, and Incentives on Strategy Selection,"
Information Systems Research, pp. 356-375, 1999.
D. Kahneman, Ed., Attention and Effort. Prentice-Hall,Englewood Cliffs, NJ, 1973, p.^pp. Pages.
T. M. Amabile, K. G. Hill, B. A. Hennessey, and E. M. Tighe,"The work preference inventory: assessing intrinsic andextrinsic motivational orientations," Journal of Personality andSocial Psychology, vol. 66, pp. 950-967, 1994.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

106

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Research on the Characteristics and Operation
Mechanism of the Maker Space in the Perspective ofComplex Adaptive System *HOU Xiao, JIN Xin, WU Jing
School of Information
Central University of Finance and Economics
Bei Jing, China
houxiaocufe@163.com

Abstract—This paper discusses the new platform – makerspace under the background of "Internet plus innovation" in
China, and how to effectively promote the reform anddevelopment of innovation and entrepreneurship, which provides
a useful reference to enhance service level and improve theefficiency of maker space. With analysis of the relevant researchstatus and typical cases, this paper explores the connotation of
the maker space and the complexity of the characteristics of thespace system based on the CAS (Complex Adaptive System)
theory, and analyses the system operation mechanism. The
maker space is a complex adaptive system, and the optimization
strategy is put forward to improve the performance of the space
operation in this paper.
Keywords—CAS theory; Maker space; Innovation ecologicalsystem; Operation mechanism

I. INTRODUCTION
Nowadays, the innovation and entrepreneurship havebecome the consensus of the world. Throughout the world's
current economic development, many countries’ economic
growth is slowed down gradually into the deep structure
adjustment. To look for a new economic growth point as thebest breakthrough point, countries put forward the sustainable
development strategy based on innovation and supported by
the science and technology, such as the Internet industry
strategic planning, German industrial 4.0 plan, China's
strategy of "Internet +", etc. As the evolution trend of today's
latest innovative undertaking, "maker space "is not only the
implementation key of the strategy of" Internet + ", but also
spur new" public entrepreneurship, peoples innovation ".In
knowledge society innovation 2.0 times, the space is a patternin the category of entrepreneurial innovation. It is greatsignificance to actively develop the space with the
optimization of the integration resources, to promote theinnovation of entrepreneurship, to create new economic
development engine. This paper explores the connotation,
characteristics, the core idea of the space and the operation
mechanism combined with the CAS (complex adaptive system)
theory, through discussing the related research present
situation and the case analysis, so as to effectively promote
innovative entrepreneurial innovation and development, andimprove the efficiency of the service ability and provide

theoretical guidance.
This paper is divided into several sections. Section 2
presents a brief introduction of related works and main
purposes in our research. Section 3 discusses the connotationof the maker space in four aspects. Section 4 analyses the
maker space ecosystem with the entrepreneurial community as
an example. Section 5 and 7 discusses the characteristics ofspace ecological system complexity and the maker spacesystem operation mechanism. Lastly, conclusions andexpectation of the research are given in the fifth section.
II. RELATED RESEARCHIn 2007, "Mass Innovation" was expressed in the uniquestrategic vision that "we are all innovators" in the economist
magazine. In early 2015, the "Maker" called "Chuang Ke"
suddenly widely admired in China, which refers to having
independent thought, love design, innovative ideas, full of thespirit of creation and innovation action. The Maker Space
originated from USA, such as Hackerspace, TechShop, AccessSpace, Metalab, Fab Lab, etc. The maker space in Chinese
style has Chinese characteristics, which is for the mass to starta business to build new business platform, under the
background of " public entrepreneurship, peoples innovation ".
The space combined with online and offline network platform,
diversification of product innovation [1]. In 2015, Chinese
government published “On the development of the guidanceof the business and promote the innovation in the public
space”, defined the maker space as an adaptation of "Internet+" era requirements and characteristics of innovative
undertaking. Furthermore, it has low cost and convenience,
open characteristic of the new innovation service platform,
and its building path contains market-oriented mechanism,professional service and capitalization approach. That
document also pointed out the key operating subject, profit
model, the space change in the degree of participation [2].The
maker space has received wide attention of the government,
academic and industry. We took "maker space" as keywords toretrieve Hownet database, after eliminating obviously not
related articles, there were 45 articles in 2015, 18 articles in
2016, and the maker space already caught the attention of themainstream journals, such as CSSCI journals " Science andtechnology management", "Scientific and technologicalprogress and countermeasures", "Management of the world",* Supported by Beijing social science fund major project (15ZDA50)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

107

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)"Books intelligence work", other core journals such as"
China's soft science" in China. Since May 2013, at the centrallevel, more than 30 copies of relevant documents were
publised to support innovative undertaking. Under the supportof national policy, the space industry has developed rapidly.
According to the ministry of science and technology, before
the 2015, there are more than 70 spaces, but by the end of2015 there have been more than 2300 spaces.
In recently years, the domestic researchers began to focus
on maker space related research. Liu Zhi-ying (2015), put
forward the "C - D - M" conceptual model, and discussed theorigin, characteristics, classification and practice patterns andso on [3]. Lv Li (2015) discussed the relationship of the two
influence mechanism between innovation and entrepreneur,and pointed out different types of incubators countries should
adopt different policies [4]. Zhang Na (2015) further
excavated the characteristics of the maker space in theperspective of space contrast and combined with case studies,[1]. Wang You-mei (2015) put forward the functional featuresand space function model, operation mode and development
and service types through in-depth analysis maker spaces athome and abroad [5]. Chen Feng (2015) analyzed the coremechanism, the spatial characteristics of ecosystem, thestructure and mechanisms, and spatial performance
improvement path, from four dimensions view of ecologicalspace system, according to the maker spirit, makers ecosystem,
resources ecosystem and the foundation of platform andentrepreneurship policy [6]. Liu Chun-xiao (2015) discussed
the present situation and the management pattern of makerspaces, and how to create Entrepreneurial ecosystem in theage of innovation 2.0 [7]. Qiu Jin-you (2015) proposed tointegrate the core values of library into the construction andoperation of maker spaces [8]. Wang Zhan-ren (2016)
suggested that innovation entrepreneurship education based on
maker space is the future of the university education forward
field
and
important
supplement
basedon theinvestigation interview of the national 25 maker spaces [9].
CAS (complex adaptive system) theory was put forward by
Holland in 1994, which is a "complex macroscopic collection"of relatively "similar and partially connected micro-structures"
formed in order to adapt to the changing environment andincrease its survivability as a macro-structure according tocertain the implementation rules of the nonlinear interaction.
Ren Jin-luan (2002) identified innovation system as a complex
adaptive system (CAS). CAS theory can broaden theinnovation system research from a new view, which will help
us to know more features and elements of the maker space.
Chen Yu (2001) pointed out that CAS are systems that have alarge numbers of components, that interact and adapt or learn
with the external environment to understand and describe
complex systems [10]. Although CAS theory is at a start-up
and perfecting the summary stage, but the basic idea arewidely used in the economic and management, ecological, andother fields.
Through the above related research, We found that the
maker space integrated all kinds of factors of entrepreneurial
innovation service using the Internet thinking, whose
innovation pattern was implemented jointly by individuals,
government, Internet companies, universities, research
institutions, venture capital institutions, intermediaries ect.,under the background of "Internet +" and innovation
ecological system environment, and the space is good fit withthe characteristics of complex adaptive system. From theperspective of the complex adaptive system, the space is basedon the open spirit of the Internet with participation of multiagent, comprehensive business innovation ecosystem, not only
can provide entrepreneurs with innovation space, also canprovide a full support, and constantly improve thecomprehensive innovation service ability.
III. THE CONNOTATION OF THE MAKER SPACE
The maker space is different from the traditional incubators,
science and technology parks or Hackerspace, it is the new
innovation service mode driving by the Internet and thedevelopment of modern information technology with the
double spiral way. As the innovation subject, the public
especially for all public entrepreneurial innovation isaggregated and make fusion effect through the creative subject
for joint participation. The maker space contains physicalspace for work, network community, the virtual space of theInternet communication platform ect.. There are several
aspects of the maker space connotation which are following as:First of all, the space is a kind of public innovation which
the masses participate in, the subject is the public. The masses
can actively pursue the culture spirit of the open source,
sharing innovation and creation. Individual maker has equalityof opportunity and innovation organization always is strongly
dynamic, there are not highly close linked the interest
community among innovation subjects, but a weakrelationship based on creative collaboration or interests. Eric
Von Hippie (1988) put forward the democratization of
innovation opportunities, while the maker space model may
provide the possibility of creating this opportunity. The
essential characteristic of the space model is that everyone has
the chance to innovation, it brings the long tail effect withsignificant difference from the characteristics of the traditionalinnovation mode. The equality of innovation resources also
makes the public have a chance to obtain innovation resources.
The space to capture and transit creative inspiration, is acombination of knowledge and practice experience and its
derivatives, is a combination of innovation practice andinnovation idea.
Secondly, the space is an opening and sharing platform
based the Internet. The development of network technology
makes mass with different specialties from different areas
together to participate in the innovation of the interdisciplinary
activities. The space can eliminate the barriers of time andspace, different innovators in the field of knowledge exchange
and share ideas, promote cross-border thinking, create new
wisdom. The maker space depends on the combination of
online and offline to implement interactive shared social
platform to provide work space and resources information forthe purpose of the mass innovative entrepreneurs.
Thirdly, the space of the main advantage is that "the low
cost and more convenience". The spaces are opening to the
public, only charge a little bit of the service fee or take theform of membership, which greatly reduce opportunity costand risk of the past mode, and allow entrepreneurs to grow inthe environment of low cost. Sometimes certain cost isrequired at the beginning of the innovation stage, but it is© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

108

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

controllable at most time. All the spaces make full use ofexisting regional conditions to provide required equipment andvenues for innovation, and at the same time provide legal
justice, industrial and commercial registration, application ofsubsidies, such as one-stop services.
Finally, the maker spaces form a new innovation andentrepreneurship ecosystems [6]. The space is different fromtraditional science and technology park, it has more quantities
and more rich roles. Also be different from traditional
incubator, it is lower threshold, wider and more convenientservices. It is the ideal innovation field, also can provide
makers with a comprehensive innovation service ecologicalsystem. Characteristic value of the maker space is to provide
the auxiliary entrepreneurial innovation service. Different
forms of the spaces take a different approach, they provide
different levels of service value for entrepreneurs. Althoughdifferent spaces have the different nature and patterns, they
can actively integrate talents, technology and other elements,
to provide a full range of one-stop service for innovation andentrepreneurship, because they have the strong fusion abilityof all resources and elements.

and communication space make information transfer more
open between entrepreneurs and investors, reduce theoccurrence of malicious rent-seeking and other false behavior,
let makers fully focus on creative jobs, and efficiently pushtalents converging, resources agglomeration, service
agglomeration, information gathering. The biggest difference
between entrepreneurial community and the traditional
incubator lies in the "connection", such as interpersonalconnections, and space connection, connection with service,and the connection with the environment. The model is shownin figure 1 to present the innovation space ecological system
schematic diagram, to display the connection between theelements and the environment. Through mutual influence,
interaction between the elements and interdependence, createdcapital chain, employment chain, industry chain and value
chain, formed a variety of different state combination, jointly
safeguard the stability of the innovation and entrepreneurshipecosystem.

IV. THE MAKER SPACE ECOSYSTEM ANALYSIS-TAKING THE
ENTREPRENEURIAL COMMUNITY AS AN EXAMPLE

The maker space is a complex adaptive system according
it’s definition. John Holland (2001)，in his book “emerge:
from chaos to order”, mentioned peculiar mechanism of thecomplex adaptive system, which includes identification,
component and internal model. Identification can tell thedifference between the subjects and play role in the process ofthe generated aggregate and boundary. Each innovation
subject of the maker space ecosystem has its own unique
identifier, such as government support, enterprise financial
strength, research ability of scientific research institution, etc.,
and the spontaneous form of cooperation or interest alliances
which are in accordance with the common aim or interest. The
building blocks consist of soft elements, such as the space
capital, information and other hard elements of ecologicalsystem and talent, process, mechanism and technology.
Existing rules can form the original building block of the newrules and the solution of the problem. The different levelsubjects have complex internal mechanism, and the rules ofthe low-level building blocks are deduced high-level rules toadjust the structure to adapt to the environment development.
In the time of the modular management, the nonlinear
interaction of different modules mechanism is different levels
within the system to realize a key link in the process of
emerging phenomenon.
Zhong Guan Cun international center provides such atypical innovation platform for to integrate the "building
blocks" efficiently, which is called entrepreneurial community.
Venture fund companies use big data of enterprises andscientific research institutions to build jointly in "hatch + bank+ investment" mode of entrepreneurial platform, which
provides the social environment and the ecologicalenvironment of innovation service using the talents, resources
and the Internet GIS technology. With the aid of micro
mechanism, entrepreneurial community uses network spaceand resources sharing space to tie tightly capital chain andbusiness markets. In addition, fully transparent work space

Fig.1. The maker space ecological system schematic diagram

V. THE ANALYSIS OF CHARACTERISTICS OF SPACE ECOLOGICALSYSTEM COMPLEXITY

The maker space can be very good understanding of thestructure characteristics of the system from the perspective ofcomplex adaptive system. Zan Ting-quan (1991) in order tofacilitate in the research of economics, thought system iscomposed of the economic elements. there is mutual
connection and interaction between subsystems and economic
elements [11]. Based on this, the maker space can be formallydefined as:
S=(A,B) With A is hardware part of the system, that is the elements
set, including basis points each participation subject, flow
elements; B is for the system software part, that is theconnection between the elements set and function, including
gathering, emergence, nonlinear, adaptability of active
relations, can be used to depict the nature, conditions and rules.
A. Flow,Gather and Diversity"Flow" refers to the evolution process of material flow,
information flow and energy flow between the individual andthe environment that have been associated with system. The
flow of the ecological space system refers to exchange ofmaterial, information and energy of multiple subjects between
the internal system and external environment, and between
individual, government, enterprises, universities, research
institutions, venture capital institutions, such as intermediaries.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

109

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

The innovation-oriented maker space speeds up the evolutionof the system, in the process of its operation, the material flow,
cash flow, information flow, personnel flow and coordinateddegree directly affect competitive advantage in the innovation
ecosystem."Gather" refers to the aggregation of individual,
through interaction and the emergence of complex behavior.
In order to form micelles and rationally allocate resources,more subjects jointly cope with the change of external
environment. In the CAS system, when any subject in theecological system come into a new field, all of the othercollaborative andinteractive subjects have to adjust
themselves, result of which is diversity.
B. Nonlinear and Active adaptation"Nonlinear" refers to not a simple linear relationship when
the individual and their characteristic attributes change, andespecially during interaction between the system or
environment in the back and forth. Direction of subjects is"active", they have actively towards the reasonable state
involved, timely feedback on each other's actions, andpromptly improve their behavior. In the development process,those make all the maker spaces unceasingly rich and colorful.
Arthur (1999) argued that sometimes the complex adaptive
system between individuals was too small to be difficult toobserve, and the interaction between random by positive
feedback which produced dramatic amplification resultscannot be measured [12]. In order to observe the influence of
nonlinear effect on the maker space by the subject and theenvironment, we build following function:
H (S, E) ∝ G * f (a. Sa, b. Eb)
where, "S" is the "subject space, "E" is for internal andexternal environment, H (S, E) is a space parameter derived
from the coupled action of subject and the internal andexternal environment, G is a comprehensive parameter; α、βis coefficient of S、E respectively; a 、b is the number of
elements contained by S, E respectively. Here we introduce
the innovation vector Y to describe the evolution of the space
shown in Figure 2 . Overall, with the increase in H, the makerspace H presents a steady increase in long-term volatility. The
dotted line in the figure indicates that there may be a recession
in the process of evolution.

Fig.2. The maker space nonlinear evolution path diagram

C. Emergence and Environmental complexity
The macro economy, legal system, innovative culture,credit cooperation environment provide and support theinnovation platform and space, and change over time fordynamic innovation, but the random uncertain changes
increases the innovation ecosystem and space complexity. The
key characteristics of CAS theory -- "emergent", means thatthe subjects produce a qualitative leap through continuous
learning and adapt to environmental changes caused bymutations on macro system structure and performance.

[14].Mckelvey (1999) argued that complex adaptive system
under uncertainty environment had strong survival, growthand innovation ability, and lead to self-organization, emergent
behavior [15]. Each emergent property in all as shown in figure
3 is a space at different stages of development. There is a lackof emergent in the space initial stage, with its continuous
development, emergent is become stronger at stable stage, andgrowth tends is to be the biggest. In addition, the space is anunique innovation service platform contained more
technologies, reflected in its stage of development bound to
experience circuitous state, only down freely to promote theinnovation of victory. At the same time, the world entered into
the era of "building block innovation", it covers two aspects ofmeaning, the one refers to the innovation activities of theelements of the transverse combination, the other refers to thevertical combination of each stage of development, and willcause a new wave of innovation.

Fig.3.The maker space relationship between the emergence and stage ofdevelopment

VI. THE MAKER SPACE SYSTEM OPERATION MECHANISMANALYSIS

The mechanisms indicate relationships among all theorganisms, the function and connect with each other. On thewhole ecological system space, the innovation mechanism is
the innovation subject coordinated the relationship between
the elements in the complex external environment, in order tobetter play a role of specific operation mode. Because of thespace complexity of subjectivity, active adaptation andenvironment, we need certain innovation mechanism toprovide safeguard to realize the utility maximization of thespace. The basic elements of space operation are the public,
goods, capital, information, etc.. Based on the foregoing
ecological system, we describe the operation mechanism ofvalue chain of ecological space system, as shown in figure 4.
It can be found in various stages of the value chain, the makerspace provide different innovation service system, promote
innovation entrepreneurship by efficient mechanism toachieve system function. There are four core mechanism:
credit guarantee mechanism, organization and coordination
mechanism, the mechanism of value creation, interactive
feedback mechanism:© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

110Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

incubator and tracking service and make the space system
keeping in the steady state.
In addition, there are some other mechanism in maker space,such as the metabolic mechanism to ensure maker spacevitality, network nested mechanism to maintain efficiency of
resources integration of the space, dynamic promoting
mechanism to boost the running of space life cycle, etc. Thosemechanisms together play the key role in innovation andentrepreneurship in space ecological system.

Fig.4.Maker space ecosystem operation mechanism Based on the value chain

The credit guarantee mechanism. The maker spaces cover a
multitude of subject with the help of the Internet trading
platform and space online mode. There is no doubt that it’sbased on the credit guarantee of the subjects as the premises.
Based on big data technologies, social network, the makerspaces pay more attention to reflect the credit situation of"software information", which ensure real information and thedevelopment of space online activities, such as for industrial
and commercial registration. Dig up the credit risk and the riskof fraud information, thus promote innovation activities
orderly.
The organization and coordination mechanism. Relationshipof the effective coordination between different subjects is thebasic conditions which the spaces carry out successfully. Theinteraction among individuals, government, enterprises, andfinancial institutions can produce better products and services.
Intrinsic motivation of each participation that subject produces
mutual cooperation can strengthen the innovation supply
ability of complementary advantages, and show significant
scale effects, namely system involved in the subject. The more
immediately, the more the content of interactive feedback can
produce nonlinear expansion effect, and promote each kind offactor to spurt to create greater value.
The mechanism of value creation. Under the "invisible
hand", all the innovation subjects take realization of the value
maximization as the behavior guidance. Based on the various
types and properties of different subjects, the target of thevalue is different. The maker space model has two escorts, lowcost and high efficiency, to produce high quality innovation,
thus progressive transformation, to promote the participation
subjects to obtain the corresponding value. The maker space isa kind of open innovation model, the whole integration flow is
widely used inside and outside with the organization
innovation resources, and all kinds of subjects ofentrepreneurial innovation which have can create multiple
benefits.
The interactive feedback mechanism. The participation
subjects in maker space carry out service activities ofinnovation and entrepreneurship in the macroeconomic
environment, legal system and credit cooperation environment.
Then, a perfect operation system is gradually formed alongwith exchange of material and energy, with the help of theflow of talent and technology resources, and adaption to theenvironment. Positive and negative feedback control system isthe intrinsic mechanism which can undertake secondaryVII. THE CONCLUSION
Based on CAS theory and related research, this paper
clarified the meaning of the maker space and its core idea,
elaborated the space which had diversity, complexity andnonlinear characteristics, active adaptation and emerging. This
paper used entrepreneurial community in Zhong Guan Cuninternational center as an example to analyze the spaceecological system, put forward the system based on value
chain operation mechanism. Through the above research, we
present some basic optimization strategies as following:Firstly, we need condense the maker spirit, strengthen themutual learning between the subjects, improve the legal
system, and build a good environment for innovation
entrepreneurship. Secondly, we need increase accumulation ofentrepreneurship resources, raise the overall strength of the
hardware and software, seek innovation consistently, andmake rich creativity of ecosystem. Thirdly, we need build core
carrier, drive product innovation and collaborative innovation,broaden the government support, set up multi-level financial
guarantee forms. Finally, the most important is to perfect theoperation mechanism based on powerful information network
integration service resources, thus get through thecollaboration of industry value chain, overcome thedownstream bottlenecks, promote the construction of highlevel innovation space, optimize the ecological environment
and improve the ecosystem performance.
In the future, we will continue to explore a new generation
maker space with Chinese characteristics. Based on our
research, it will provide a useful reference for the reform of
the field of innovation and entrepreneurship, service
capabilities and efficiency of platform, and deeply build theresearch foundation for the quantitative research of spaceecosystem in the next research step.

References[1] Zhang Na. The localization of the maker space – “Internet +” era andhackerspace[J]. Science and technology association BBS, 2015:10-25[2]The general office of the state council on the development of the businessand promote the innovation in the public space of guidance [EB/OL]. (201503-11) [2015-07-25]. HTTP: / / http://www.gov.cn/zhengce/content/201503/11/content_9519.htm.
[3] Liu Zhi-ying, Chen qing-xiang, Xu yi. The concept of the maker spacemodel and its theoretical analysis [J]. Journal of management science, andscience and technology, 2015, 11:52-61[4] Li Lv, Li qian, Fang zhu-qing, Qiao hui. The maker space, creates thespace and the entrepreneurial process [J]. Science and technology
entrepreneurship monthly, 2015, 10- 15.
[5] Wang You-mei, Ye ai-min. From a guest space to the maker space: 2.0
based on the innovation function model and service path [J]. Journal ofelectrochemical education research, 2015, 11:5-12.
[6] Chen Feng, Xiang li-yao, Yu rong-jian. The entrepreneurial ecosystem andspace: the characteristics, structure, mechanism and strategy – Hang Zhou

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

111

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
dream town, for example [J]. Journal of business economics and management,
2015, 11:35-43.
[7] Liu Chun-xiao. Innovation 2.0: the present situation, type, and the spacemodel [J]. Journal of Internet economy, the 2015 preceding: 38 and 43.
[8] Qiu Jin-you. The maker space: the library setting the direction of the future
space [J]. Journal of library and intelligence, 2015, 12:78-80.
[9] Wang Zhan-ren, Liu hai-bin, Li zhong-yuan. All the space in the role ofinnovation in colleges and universities entrepreneurship education
research - based on six cities in China all 25 and space field visiting
survey [J]. Journal of ideological and theoretical education, 2016, 11:8591.
[10] Chen Yu. Complex adaptive system (CAS) theory and its application,
origin, content and enlightenment [J]. Journal of system dialecticalstudies, 2001:35-39.

[11] Zan Ting-quan. System economics to explore: concepts and principles [J].
Journal of exploring nature, 1991, 11:38-42.
[12] Arthur WB, the Complexity and the pa, Science, 1999 (2), pp. 107-109.
[13] Zhang Yu-ming, Liu de-sheng. From linear to nonlinear, firm growththeory review, the present situation and prospect [J]. Journal of fujian
normal university (philosophy and social sciences edition), 2010,01:2328.
[14] Wang Shu-xian. Enterprise integration based on CAS theory innovation
research [J]. Science and technology progress and countermeasures,
2008, 11:128-131.
[15] B.M ckelvey, the Complexity and found in organization science: seizing
the promise or becoming a fad?"The Emergence, 1999 (1), pp. 5-25.
[16] Zhou Xin, Huang wei, Li Ji. Network platform for the growth ability of
information niches suitability evaluation [J]. Journal of library
intelligence, 2016 01:1 - 10.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

112

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Role Fulfilment Model for Motivating Participationin Mundane Crowdsourcing Tasks
Xinjia Yu∗ , Chunyan Miao∗† , Cyril Leung†‡ and Charles T. Salmon§∗ School of Computer Science and Engineering (SCSE), Nanyang Technological University (NTU), Singapore† Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), NTU, Singapore‡ Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada§ Wee Kim Wee School of Communication and Information, NTU, Singapore{xyu009, ascymiao}@ntu.edu.sg, cleung@ece.ubc.ca, salmon@ntu.edu.sg
Abstract—Some of the major problems lead to the failure ofcrowdsourcing, like low response rate and unqualified submission, are related directly to the participants’ lack of motivation.
However, it is not well understood how incentives influence
crowdsourcing participation. This paper proposes a role fulfilment model from a sociological perspective to understand motivation in crowdsourcing. According to the requesters’ expectation,
the participants should take the roles of workers. It means thatthey should respond effectively in a satisfied quality with serious
attitude towards the tasks. However, especially for mundane
crowdsourcing tasks, the participants are less likely to takeownership of the tasks. Based on the related role requirementsof fun, reward and killing time, they will be less likely to treattasks seriously. To resolve the role conflict, the proposed rolefulfillment model helps participants feel more involved in a closeknit task oriented group. The experiment results support thatsuch an approach can enhance the participants’ motivation incrowdsourcing.
Index Terms—Crowdsourcing; role theory; role fulfilment
model

I. I NTRODUCTION
Crowdsourcing refers to an online, distributed problemsolving and production model involving open-source practice[1], [2], [3], [4]. Crowdsourcing outperforms the traditional
industries with improved efficiency and reduced cost. It has
several obvious advantages: (1) aggregating vast of crowd
wisdom with an acceptable cost [5], (2) involving diverse participants and opinions [6], (3) offering a facilitate to exchange
ideas [7], and (4) avoiding the waste of time on unnecessarycommunication and compromise [5].
Some famous crowdsourcing platforms, such as Amazon
Mechanical Turk (mTurk), offer a marketplace for online
work. The requesters post tasks and offer compensations while
the participants (a.k.a. workers) can contribute to the tasks.
Some of the tasks, such as logo design, require specializedskills. Others are open to all without special requirements.
For example, a website may invite diverse people to assess
its human computer interaction design via crowdsourcing. We
refer to the latter category of crowdsourcing tasks mundane
tasks. The participants who respond to this kind of tasks
do not need to possess specific skills. Corresponding to thisfact, these participants are not motivated by factors such as
to develop their abilities. In order to motivate participationin mundane crowdsourcing tasks, existing approaches mostly

focus on providing workers with incentives according tovarious considerations [8], [9], [10], [11], [12], [13], [14], [15].
In this paper, we explore the impact of organizing workers into
teams to play different roles on their motivation to participate.
Brabham [1] emphasized in his understanding about theagenda for crowdsourcing research that we need to understandhow members of the crowd feel about their roles in thecrowdsourcing process. This paper tries to understand theparticipants’ motivation in crowdsourcing from a sociologicalperspective. We introduce the concept of role to understandcrowdsourcing workers’ motivation. The absence of traditionalroles, such as peers, around a member of the crowd canlead to conflict between role requirements of the individual
and role expectations from the requester. This conflict will
negatively impact the worker’s motivation and participation incrowdsourcing.
In this paper, we propose the Role Fulfilment Model(RFM) to solve the role conflict while try to understand therelationship between the fulfilment of absent roles aroundworkers and their motivations in crowdsourcing. We focus on
mundane crowdsourcing tasks where diverse participants work
towards a single goal (the task) with little compensation. These
participants can be seen as a task organization group.
The rest of this paper is organized as follows. Section
II introduces related studies of motivation in crowdsourcing.
Section III describes the RFM and how it can help resolve
the role conflict and enhance the participants’ motivation.
Section IV describes the experiment design, and analyzes
the results. The paper concludes with a summary of thepotential applications of the RFM in crowdsourcing and gives
an overview of future research directions.
II. R ELATED W ORK
Motivation research aims to explain the factors driving
people to take a certain course of action [16]. It is a populararea of research in crowdsourcing due to the fact that the lackof high quality responses is a major cause of inefficiency
in crowdsourcing [17]. A crowdsourcing service provider
who builds the task marketplace must strive to build good
relationships with the crowd. The organization must study the
daily activities of the crowd in the platform. This can be an© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

113

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

effective way to make the participants respond to the tasks
proactively with serious attitude.
Current motivation research in crowdsourcing categorized
motivation into two types: 1) intrinsic motivation and 2) extrinsic motivation. The former is generated by the participants’ internal decision-making processes. Intrinsic motivations include
fun, sense of accomplishment, and killing time [1]. Anotherform of intrinsic motivation is community based motivationsuch as social interaction, charity and customer loyalty to therequesters or the crowdsourcing platform [18]. Extrinsic motivation rooted from people’s desire to achieve something in theenvironment [17]. The most common extrinsic motivation is
payoff. This payoff can be monetary (e.g., money or vouchers)[1], [18], [19] or non-monetary (e.g., skill improvement) [1],[18].
Specific to mundane task crowdsourcing, since it is always
unrelated to any specialized skill, there are few participants
motivated by skill development, work opportunity, or keeping
in touch with the industry trends. Therefore, non-monetary
extrinsic motivation can hardly motivate the workers. Besides
common intrinsic motivation and money, we focus on the senseof accomplishment and the social communication motivationin this paper. Furthermore, we propose more detailed metrics
based on the role theory.
III. ROLE F ULFILMENT IN C ROWDSOURCING
A. Preliminaries
Role theory uses the item of role to describe and analyze
individuals’ behaviors in social interactions. Role is a basis forindividuals in a society to communicate with each other. Roles
reflect norms, attitudes, contextual demands, negotiation, andthe evolving consideration of situation which are commonly
understood by all members of a society [20], [21], [22]. The
understanding of his/her role can affect an individual’s goal,
and behavior pattern [23]. This effect happens through theinteraction and compromise among the role requirement andself-concept of an individual and the role expectation from thesociety or organization to which the individual belongs [24].
McClelland [25], [26] proposed the theory of achievement
motivation within a small group of people who work together
to complete a specific task. This theory emphasized the close
tie aiming to achieve a common goal within the group of people. This kind of groups can be an entrepreneurial organizationor other types of groups with similar institutional architecture.
Group role motivation is generated based on McClelland’s
theory. It defined several role requirements in a healthy andeffective group [27]. The following three can be applied to
crowdsourcing:1) Affiliation Requirement: As social animals, human beings are used to work together. They have the desire
to interact socially and cooperate with others, especially their peers. Psychological and management studies
offered evidence that the lack of interaction may jeopardize both group work and group members’ emotion[27].

2) Acceptance Requirement: Group members have the desire to identify their roles in the society. A stable and acceptable role can make the individuals feel an increased
sense of belonging [28]. Furthermore, individuals need
the sense of being reaffirmed by someone else from thesame group to feel safe [29]. This support is mutual.
While the group satisfies the members’ role requirements, the group members can find and take their roles
more smoothly [30]. This positive relationship between
the group and its members can enhance motivation andenhance effectiveness [31].
3) Feedback Requirement: Group members need to get
feedback from peers and the environment to makeunderstand whether they are doing well. This can be
manifested as the establishment or termination of trust
relationships [32], [33], [34]. Otherwise, they may feel
uncertain about their work process and quality. Besides
this, they also desire favorable attitudes from their peers.
These interactions can stimulate positive emotions andenhance group members’ motivation [35].
B. The Role Fulfilment Theory
The majority of motivation studies in crowdsourcing focus
on the relationship between the participants and the work or
the crowdsourcing platform. They ignore the social aspectof working in crowdsourcing which can be explained by the
motivation studies in role theory.
The integration of role requirements of an individual in an
organization can be abstracted as the role he/she would like toplay in the organization. For example, if someone desires tocontrol the process of team work and organize the coworkers,
that means he would like to take the role of the “leader” inthe group. The organization will also have expectations on
the individuals’ behaviors and attitudes. Those expectations
coalesce into the role which the organization would like amember to take up.
The coherence between these two roles on one individualis the key toward building a positive relationship between
the individual and the organization. According to the role
theory, the common beliefs and understanding of the members’roles can lead to both individuals’ feeling of comfort andthe effectiveness of the group. Otherwise, when the roles
which the group would like the members to take conflictwith the members’ role expectations, the organization will facedisruptions.
In crowdsourcing, the unqualified work and low response
rate are products of a worker’s role conflict. To the requesters, the role of each participant is “worker”. The worker
is expected to perform effectively with high quality, takeresponsibility for their work, and complete the tasks in a timely
manner. However, because of the online and individualizedworking environment, lack of understanding about the task
and the requester, and unfamiliarity with the task content,
the participants often not perceive themselves as workers in
the traditional sense. This is especially the case for mundane
tasks due to the boring task content and the lack of technical© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

114

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Requester
Expectations

Effective
Response

Responsibility

Work Quality

WorkerRoles

Unrelated
Passers

WorkerAffiliation

Feedback

Killing Time

Acceptance

Fun
RewardRequirements

Requirements

Participants
Fig. 1. The logical framework of the Role Fulfilment Theory (RFM) for crowdsourcing mundane tasks.

requirement. In this case, the participants may see themselves
as unrelated “passers” who does the tasks just to kill time andearn some payoffs.
When the participants consider themselves as “workers” justas the requesters intended, he/she will hold a different set ofrole requirements. For the majority of the individuals, theirunderstanding and requirements of the role of worker comesfrom their working experience in the traditional industries.
Based on this origin, the work will require affiliation andfeedback from the peers, and acceptance from the group. In
other words, the “worker” needs to feel that they are working
in a group with some others. The logical framework of the
RFM is summarized by Figure 1. By matching workers’ rolerequirements with the requesters’ role expectations, workers
could be better motivated to perform mundane tasks. With theRFM, we conduct an empirical study to investigate if the above
hypothesis holds.
IV. E MPIRICAL E VALUATIONS
According to the proposed RFM, we designed a crowdsourcing open call to ask the participants to label pictures. We
released the task on one of the most popular social networks– Weibo1 – which can be regarded as the Chinese equivalentof Facebook. Notice that not all the role requirements areconscious [27]. Individuals are either not willing to talk aboutit, or are not aware of the different types of motives. Therefore,studies in this field need to be able to extract these motivations
indirectly instead of asking direct questions.
A. Experiment Design and Evaluation Metrics
Our empirical study consists of the following steps:
1) Step 1: Design a mundane crowdsourcing task (Task1). Participants are presented with pictures and asked
to label the emotions expressed by the given pictures.
The task is simple and requires no specialized skill.
The emotional strength shall be expressed on a scale
between 0 (none) and 5 (extremely strong). The options
for the emotions are Ekman’s six basic emotions (i.e.
happy, sad, angry, bored, surprised, and excited) [36].
An example of the task is shown in Figure 2. At the endof the task, the participants are also required to reporttheir opinions about the task.
1 http://www.weibo.com/© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

115

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

A and Group B.
2) Responsibility: At the end of each task, the participants
need to answer a multiple choice question “What does
this work mean to you?”. The choices are: “I regard thework as a game”; “I regard the work as a serious task”;“I regard the work as a burden”; “Nothing, I was justkilling time”. This answer is used to assess their attitude
toward the tasks.Furthermore, we embed a same mistakein both Task 2A and Task 2B. The scale under one of thepictures is 2–7 instead of the normal version of 0–5. If
the worker pays enough attention to the questions, they
can find this problem easily. We can use the report rateof this error to measure their sense of responsibility.
The better the performance as reflected by these metrics, thestronger the evidence of role fulfilment and motivation among
crowdsourcing participants.
B. Results
The experiment was run on Weibo for one week. At theend of Task 1, 40 participants returned valid responses. They
are all Chinese. Thirty two (80%) of them are female whileeight (20%) are male. The age and gender distributions of theparticipants are shown in Figure 3.

Gender2) Step 2: Design the second task (Task 2). The nature of
Task 2A is very similar to Task 1 but with different
pictures. To support the further comparisons, we designed another task (Task 2B) with the same content as
Task 2A. However the latter one contains a statement
at the beginning stating that “This is teamwork andsome other participants will proceed based on youranswers.” Furthermore, we insert several questions, suchas “Would you like to be a leader in your team?”, into thelabelling work to remind them that they are collaboratingwith others.
3) Step 3: Release Task 1 on Weibo.
4) Step 4: We separated the participants in Task 1 equallyand randomly into two Groups A and B. Then, we sent
Task 2A to Group A and Task 2B to Group B.
5) Step 5: We spelt out the requirements of “worker”
to Group B with the purpose of getting them to feelthat they are working in a target oriented group. We
used a controlled Weibo ID to play the role of theirpeers. This ID communicated with the participants inGroup B with the information about their process, thecomplaint about the task, and other feedbacks. During
this communication, we attempt to provide a way tosatisfy the requirements of affiliation and feedback ofpeers, and acceptance of the group for participants inGroup B.
6) Step 6: Collect the responses to Task 2.

Male

Female<21 21~40 41~60 >60
Age Group

Fig. 3. The demographics of the study participants.

Fig. 2. An example task. The task asks a participant to label the presence andstrength of various emotions in the picture. Options for the emotions include“happiness”, “sadness”, “anger”, “boredom”, “surprise”, and “excitement”.

To understand whether the participants can satisfy the role
expectation of the requester, we need to analyze their response
effectiveness, and responsibility. We use a combination ofself-report and performance observation to measure these two
factors:
1) Response Effectiveness: We measure the response effectiveness by comparing the response rates between GroupWe then divided the 40 participants randomly into twogroups of 20 each, and sent Group A Task 2A and GroupB Task 2B. At the end of Task 2, 8 out of the 20 participantsin Group A submitted their work. The response rate is 40%.
Meanwhile, 16 out of the 20 participants in Group B submitted
their work. The response rate from Group B is 80%, which
doubles that of Group A. The distribution of the number of
responses to Task 2 across different age groups is shown in
Figure 4.
To understand the participants’ sense of responsibility, we
analyse their answers to the survey question “What does
this work mean to you?”. The distribution of perceptions byparticipants from different age groups for Task 1, Task 2A© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

116

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)100
Task 2A
Task 2B

8
6
4
2
0

<21

21~40
41~60
Age Group

40

0

>60

[Game][Burden] [Work] [Killing Time]
Perception of the Task

Fig. 6. The distribution of perceptions by participants (Task 2A).

100

100

60
40

<21
21~40
41~60
>60

80
% of Responses

<21
21~40
41~60
>60

80
% of Responses

60

20

Fig. 4. The distribution of responses to Task 2A and 2B by participants.

60
40
20

20
0

<21
21~40
41~60
>60

80
% of Responses

No. of Responses

10[Game][Burden] [Work] [Killing Time]
Perception of the Task

0

[Game][Burden] [Work] [Killing Time]
Perception of the Task

Fig. 5. The distribution of perceptions by participants (Task 1).

Fig. 7. The distribution of perceptions by participants (Task 2B).

and Task 2B are shown in Figure 5, 6 and 7, respectively.
The participants who regard the task as a serious work can be
regarded as a statement of responsibility to the group. It canbe observed that the percentage of participants with serious
attitude in Task 2A is 12.5%. This number is close to theone in Task 1 (15%). At the same time, more than 56.25%
of participants regard Task 2B as serious work. This shows
significant evidence that, when the people believe they areplaying a certain role in a team, they are more likely to takethe mundane crowdsourcing task seriously.
Furthermore, 3 participants in Task 2B reported the mistake
purposely embedded in the test question while no one in Task
2A reported it. This result can also support our hypothesis thatGroup B who believed that they were playing a role in a teamworking on Task 2 show more sense of responsibility.

21 to 40 regards the task as serious work. A possible reason
behind this observation may be that this age group contains
working age people who are most familiar with and keen to
teamwork. The environment and mental model from their daily
life influence their behaviors in crowdsourcing.
We also found that the RFM did not work well within
the younger group under 21 years old. It maybe becausethat people in this age group tend to be students who are
more accustomed to working individually instead of playing
significant roles in teams.

C. Discussions
By cross-analysis, we found that the RFM reaches is themost effective for people in the age group 21–40. Within thisage group, only 13%–15% of participants regard Task 1 or
2A as serious work . For Task 2B, 60% of participants aged

V. C ONCLUSIONS AND F UTURE W ORK
Fulfilling the participants’ role requirements as workers canaddress their role conflict in crowdsourcing. These requirements include affiliation to a group, acceptance by a group, andfeedback from the peers and the environment. The harmony
between role expectations from the requesters and the rolerequirements from the participants can motivate and lead to amore productive crowdsourcing environment. Through a realworld empirical study, we demonstrate that by infusing the
proposed RFM as a motivation mechanism in crowdsourcing,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

117

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

participants’ attitude towards the mundane tasks become more
conducive for achieving good quality work.
In subsequent research, we proposed to include more variables in the RFM into the empirical study. We need to investigate the influence of personal factors such as age, gender,
personality, and working status on the effective of RFM.
Findings from such studies may eventual yield computationalalgorithms to help the crowdsourcing platform and requesters
to form productive teams of participants automatically.
RFM could impact the technological foundation of crowdsourcing platforms as well. In the future, artificial agents could
be infused into a crowdsourcing platform to take up the roleas the coworkers around the isolate workers to fulfill theirrole requirements. The agent can be either anthropomorphousor just an intelligent messaging system. We will study theeffectiveness of these types of artificial intelligent agents infulfilling the absent roles in mundane crowdsourcing tasks.
The role fulfilling system may also affect the long term
relationship between the participants and the crowdsourcing
platform. To understand this aspect, we plan to conduct longterm observation experiments in the future.
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its IDM
Futures Funding Initiative; and the Interdisciplinary Graduate
School, Nanyang Technological University, Singapore.
R EFERENCES[1] D. C. Brabham, “Crowdsourcing as a model for problem solving: An
introduction and cases,” The International Journal of Research into NewMedia Technologies, vol. 14, no. 1, pp. 75–90, 2008.
[2] H. Yu, Z. Shen, C. Miao, and B. An, “Challenges and opportunities for trust management in crowdsourcing,” in Proceedings of theIEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’12), 2012, pp. 486–493.
[3] R. Silberzahn and E. L. Uhlmann, “Crowdsourced research: Many hands
make tight work,” Nature, vol. 526, pp. 189–191, 2015.
[4] P. Michelucci and J. L. Dickinson, “The power of crowds,” Science, vol.
351, no. 6268, pp. 32–33, 2016.
[5] J. Surowiecki, The Wisdom of Crowds. Anchor, 2005.
[6] H. Jenkins, Convergence Culture: Where Old and New Media Collide.
NYU Press, 2006.
[7] T. Terranova, Network Culture: Politics for the Information Age. Pluto
Press, 2004.
[8] H. Yu, C. Miao, B. An, C. Leung, and V. R. Lesser, “A reputation
management model for resource constrained trustee agents,” in Proceedings of the 23rd International Joint Conference on Artificial Intelligence(IJCAI’13), 2013, pp. 418–424.
[9] H. Yu, C. Miao, B. An, Z. Shen, and C. Leung, “Reputation-aware
task allocation for human trustees,” in Proceedings of the 13th International Conference on Autonomous Agents and Multi-Agent Systems(AAMAS’14), 2014, pp. 357–364.
[10] L. Tran-Thanh, T. D. Huynh, A. Rosenfeld, S. D. Ramchurn, and N. R.
Jennings, “Budgetfix: Budget limited crowdsourcing for interdependenttask allocation with quality guarantees,” in Proceedings of the 13th International Conference on Autonomous Agents and Multi-agent Systems(AAMAS’14), 2014, pp. 477–484.
[11] H. Yu, C. Miao, Z. Shen, C. Leung, Y. Chen, and Q. Yang, “Efficient
task sub-delegation for crowdsourcing,” in Proceedings of the 29th AAAI
Conference on Artificial Intelligence (AAAI-15), 2015, pp. 1305–1311.
[12] H. Yu, C. Miao, Z. Shen, and C. Leung, “Quality and budget aware task
allocation for spatial crowdsourcing,” in Proceedings of the 14th International Conference on Autonomous Agents and Multi-Agent Systems(AAMAS’15), 2015, pp. 1689–1690.

[13] S. Liu, C. Miao, Y. Liu, H. Yu, J. Zhang, and C. Leung, “An incentive
mechanism to elicit truthful opinions for crowdsourced multiple choiceconsensus tasks,” in Proceedings of the 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology(WI-IAT’15), 2015, pp. 96–103.
[14] H. Yu, C. Miao, Z. Shen, J. Lin, C. Leung, and Q. Yang, “Infusing
human factors into algorithmic crowdsourcing,” in Proceedings of the
28th Conference on Innovative Applications of Artificial Intelligence(IAAI-16), 2016, pp. 4062–4063.
[15] Z. Pan, H. Yu, C. Miao, and C. Leung, “Efficient collaborative crowdsourcing,” in Proceedings of the 30th AAAI Conference on Artificial
Intelligence (AAAI-16), 2016, pp. 4248–4249.
[16] E. L. Deci and R. M. Ryan, Intrinsic Motivation and Self-Determinationin Human Behavior. Springer US, 1985.
[17] N. Kaufmann, T. Schulze, and D. Veit, “More than fun and money.
worker motivation in crowdsourcing - a study on mechanical turk,” inProceedings of the 17th Americas Conference on Information Systems(AMCIS’11), 2011.
[18] D. C. Brabham, “Moving the crowd at threadless: Motivations for participation in a crowdsourcing application,” Information, Communication& Society, vol. 13, no. 8, pp. 1122–1145, 2010.
[19] J. M. Leimeister, M. Huber, U. Bretschneider, and H. Krcmar,“Leveraging crowdsourcing: Activation-supporting components for ITbased ideas competition,” Journal of Management Information Systems,
vol. 26, no. 1, pp. 197–224, 2009.
[20] B. J. Biddle, “Recent developments in role theory,” Annual Review ofSociology, vol. 12, pp. 67–92, 1986.
[21] J. Heiss, “Social roles,” in Social Psychology: Sociological Perspectives,
M. Rosenberg and R. H. Turner, Eds. Transaction Publishers, 1981,
pp. 95–129.
[22] S. Stryker and A. Statham, “Symbolic interaction and role theory,”in Handbook of Social Psychology, C. Linzey and E. Aronson, Eds.
Transaction Publishers, 1985, ch. 1, pp. 311–378.
[23] C. Gordon and P. Gordon, “Changing roles, goals, and self-conceptions:
Process and results in a program for women’s employment,” in Personality, Roles, and Social Behavior, W. Ickes and E. S. Knowles, Eds.
Springer New York, 1982, pp. 243–283.
[24] J. Piaget, The Language and Thought of the Child. Psychology Press,
1959.
[25] D. C. McClelland, The Achieving Society. Princeton NJ, 1961.
[26] ——, “Business drive and national achievement,” Journal of Management Information Systems, vol. 40, pp. 99–112, 1962.
[27] J. B. Miner, Role Motivation Theories. Routledge, 2008.
[28] A. Erez and T. A. Judge, “Relationship of core self-evaluations to goal
setting, motivation, and performance,” Journal of Applied Psychology,
vol. 86, no. 6, pp. 1270–1279, 2001.
[29] P. H. Blaney and R. J. Ganellen, “Hardiness and social support,” inSocial support: An interactional view, B. R. Sarason, I. G. Sarason, and
G. R. Pierce, Eds. Wiley New York, 1990, pp. 297–318.
[30] T. N. Bauer, E. W. Morrison, and R. R. Callister, “Organizationalsocialization: A review and directions for future research,” in Research
in Personnel and Human Resource Management, G. R. Ferris and K. M.
Rowland, Eds. JAI Press Inc., 1998, ch. 16, pp. 149–214.
[31] T. Judge, C. Thoreson, J. Bono, and G. Patton, “The job satisfaction–
job performance relationship: A qualitative and quantitative review,”
Psychological Bulletin, vol. 127, pp. 376–407, 2001.
[32] H. Yu, Z. Shen, and B. An, “An adaptive witness selection method for
reputation-based trust models,” in Proceedings of the 15th International
Conference on Principles and Practice of Multi-Agent Systems (PRIMA’12), 2012, pp. 184–198.
[33] J.-P. Mei, H. Yu, Y. Liu, Z. Shen, and C. Miao, “A social trustmodel considering trustees’ influence,” in Proceedings of the 17th
International Conference on Principles and Practice of Multi-Agent
Systems (PRIMA’14), 2014, pp. 357–364.
[34] Y. Liu, S. Liu, H. Fang, J. Zhang, H. Yu, and C. Miao, “RepRev:
Mitigating the negative effects of misreported ratings,” in Proceedingsof the 28th AAAI Conference on Artificial Intelligence (AAAI-14), 2014,
pp. 3124–3125.
[35] A. C. Crouter, “Participative work as an influence on human development,” Journal of Applied Developmental Psychology, vol. 5, pp. 71–90,
1984.
[36] P. Ekman, “Basic emotions,” in Handbook of Cognition and Emotion,
T. Dalgleish and M. Power, Eds. Sussex, UK: John Wiley & Sons,
1999, ch. 3, pp. 45–60.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

118

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Role of Human Emotions in Social Influence(for Special Track on Crowd Decision Making)Pankaj Mishra

Rafik Hadfi

Takayuki Ito

Department of Computer Science
Nagoya Institute of Technology
Gokiso, Nagoya, 466-8555
Email: pankaj.mishra@itolab.nitech.ac.jp

Department of Computer Science
Nagoya Institute of Technology
Gokiso, Nagoya, 466-8555
Email: rafik@itolab.nitech.ac.jp

Department of Computer Science
Nagoya Institute of Technology
Gokiso, Nagoya, 466-8555
Email: ito.takayuki@nitech.ac.jp

Abstract—Diffusion of information can be found in any social network, either it be a small network of co-workers orlarge as social networking applications like Facebook, Twitter,
LinkedIn, etc. In this paper, we propose a system to analyse
the correlation amongst the group of people and build theinfluential relationship between them, based on the diffusionof emotions within the network. Basically we track the facial emotions of all the participants in the network, basedon the reaction and action taking place in the network weconstruct the existing influential graph in the network; which
is a weighted graph and weights represents the influence
between each pair of nodes. Considering that the numberof participants in the network varies, so to maintain therobustness and scalability of the proposed algorithm we haveincorporated multi-agent paradigm in our model. Further, ourresults were validated by analysing a scripted discussion donein our laboratory. Such knowledge of the influences among
the participants of the network, could be used to find theinfluential individual; further can find many applications suchin consensus, negotiation, viral marketing, etc.

1. Introduction
A social influence is the behavioural change of a person
because of perceived relationship with other person. In the
recent years, many works were proposed to analyse thesocial correlation in the social network comprising of small
group of people or large social networking applications
like Twitter, Facebook, etc. Social correlation in the larger
networks like Facebook, Twitter were analysed as discussed
in the works [3], [4]; which tracks the online actions to build
the correlation. Similarly, there are many works to analyse
the social correlation based on the multimedia data of theworkgroup, social gathering, etc., by tracking their bodypose, interaction frequency,speaking and writing frequency
etc., as discussed in [1], [2]. Briefly, most of the existing
works to analyse the interaction in the small group, thesocial interactions focuses on the pose detection, namely
head pose, shoulder pose, hand pose or their interaction
interaction, etc. However, few attempts were taken to anal-

yse the social influential correlation based on the facial
emotions.
In this paper, we shall discuss a facial emotion basedsocial correlation analysis among small group of people,
comprising of co-workers in discussion, members in thebusiness meeting, etc. In any such groups, people havethe tendency to come together and form virtual groups. In
these virtual groups, the actions of the influential person
is propagated to others, mainly because of three factors;
cofounding, homophile and social influence [5]. According
to homophile and cofounding; two individuals are most
likely to follow each other because of factors like common
interest, friend, colleagues, etc. Such factors are mainly
decided on their background information, such as theirprofession, knowledge of the topic of discussion, age, family
ties,etc. Because, based on these parameters, we can find theexistence of some kind of social ties two person; such as
boss and employee relation, siblings, collogues, scholar ofsame field, etc. However, the analysis of influence is purely
based on tracking the action and the corresponding reaction
between any pair of persons; where, the action can be a pose,talk (spoken sentences), emotion, interactions, etc. In ourwork, we are focused on tracking the emotion and talking
sequence of all the participants to analyse the influenceamongst them. Also, facial emotions can be considered as
one of the important action for human communication andunderstanding the social relationship between them, because
when an influential person has a emotion people aroundhim/her can read it and react according irrespective of thewords spoken or any body pose. So in this work, we focus
on human emotions to understand their correlation, also
emotions are voluntary reaction on any action, so could
depict the appropriate feeling of the person for an particularperson in the network. In order to maintain the scalability[6] of the proposed algorithm, we have adopted multi-agent
paradigm, where each participants in the network is treatedas homogenous autonomous agent.
In this paper, we have made two main contributions:(i) social influential correlation building algorithm basedon emotion diffusion analysis in the network, (ii) Profilingalgorithm to label all participants based on the available dataof each nodes. Apart from this, the testing video dataset,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

119

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

which was a scripted discussion, could be used to build thesimilar emotion based systems.

1.1. Architecture
The architecture of the system is depicted in the figure 1,
which consist of two main modules, facial emotion analysis
and social correlation analysis. A captured video data of
the social network is first pre-processed and later passed
through these two modules. As previously mentioned, eachindividual is handled by homogenous autonomous agents,
these agent interact with the two modules; facial emotion
analysis and social correlation analysis, independently. Finally a weighted graph representing the correlation among
these agents, is obtained as the output of the system. Briefly,
our whole work is divided into two main steps; facial
emotion analysis and social correlation analysis, using themachine learning techniques and social dynamics concepts
respectively. Later, implementation of the whole system isrealised in multi-agent paradigm.
The rest of the paper is organised as following. Section
2 explains the details about facial emotion analysis, which
is divided into two main steps; feature extraction and emotion recognition by comparing different classifiers. Section
3 explains the social correlation analysis along with theall the node centrality algorithm and correlation building
algorithms. Section 4 presents the experimental results thatvalidate our proposed algorithm. Finally, we conclude andprovide the future work.

2. Facial Emotion Analysis
In the proposed work, the system expects a frontal
video data of a group of participants discussing or in abusiness meeting. Wherein, change in one’s emotion being
induced on the other’s emotions can be observed. This video
data is first pre-processed; which includes : framing of thevideo and cropping the frontal image in all the frames.
Locating and cropping is done using Voila Jones algorithm[7]. These, processed frontal frames for all the participants
along with the talking sequence 1 of all the participants ineach frame is recorded; where the actions include, emotion,
talk sequence or both based on the metadata. After the preprocessing, we detect the facial emotion in all the fames in
two stages, namely (i) Features extraction and (ii) Emotion
classification, as discussed below.

2.1. Facial features extraction
Feature extraction is done in two stages: tracking thefacial feature points; namely eyes, eye brows, nose, lips,
etc., and then extracting the shape and appearance feature
vectors around these detected feature points. The shapefeature vectors basically resembles co-ordinates (x and y) of
all the facial features; whereas appearance features can be aGabor descriptor or Local Binary Pattern (LBP) descriptor,
1. Tracking when the participant is talking or not

etc as discussed in [8]. However, most of the past works
like, [9], [10], [11], are based on the deformable model
named Active appearance model (AAM). The AAM is acomputer vision algorithm to track the facial feature points
on human face, which provides a compact statistical representation of the shape and the appearance variation of it.
Therefore, we adopted a variation of the AAM, that is apart based AAM [12] to track 68 facial-landmarks points
in the frontal frames. Finally we extract four different typesof the feature vectors: (i) shape vector resembling the x&ycoordinates of all the facial features, called as similarity
normalised shapes (s-pts), (ii) holistic appearance feature
vector; called as similarity normalised appearance (s-app)and canonical appearance (c-app) [9], and (iii) A 32 × 32
patched vector called as scale-invariant feature transform(SIFT) [13] descriptors. Later, classifiers are trained withthese feature vectors to classify the frames based on the
carrying emotion.

2.2. Emotion Classification
The four feature vectors associated with all the frontal
frames are used to detect the emotion in the each frames.
Basically, emotions are recognised by presence of one or
more, particular Facial Action Units (FACS) [14]; further the
combination of these FACS defines one of the 7 basic emotions; namely, happy, sad, disgust, anger, contempt, surpriseand fear. The classifiers are trained with the feature vectors
extracted from the frames, against the labels of FACS or
combinations of FACS (emotion). These classifiers classify
the frames based on the presence of FACS or emotions;
therefore the accuracy of emotion recognition is mainly
based on the accuracy of the emotion detection module. So,
we compare different classifiers, and classifier that would
give the best classification accuracy will be adopted. For thatmatter, we are comparing three classifiers, namely, Scalar
Vector Machine (SVM) [15], Scalar Vector Machine with
AdaBoost (SVM+Adb) [16] and the Deep Belief Network(DBN) [17]. Whereas, we will evaluate the classifiers basedon cross validation and maximum area under the curve;
discussed further in the experimental section. Briefly, facial
emotion analysis module comprises of a trained AAM modeland the best classifier amongst the three. Once all the frames
are labeled with the carrying emotion, then these emotion
labels are usef for the social correlation analysis.

3. Social correlation Analysis
In the previous section, we obtain the emotion labels
associated with all the frames. However, for the experimental purpose we considered only the emotion of the person
occurring per second, by summarising all the emotions
occurred in all the frames per second.These emotion data ofall the participants is used for social influential correlation
analysis. The overall analysis of social correlation amongst
the participants is based on the detected emotion per seconds(E), the tracked action per second (A) and the backgroundknowledge (K). Wherein, K consist of whatever information© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

120

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Figure 1: System Architecture
about the participants in the network is available; such asage, gender, profession, seniority, family ties etc. Further,based on this background knowledge, we label all theindividuals on the basis of their importance and affinity.
Although, there are many different methods to calculateaffinity (γ ) or the node centrality in a network, as discussed
in [18], in the considered domain, the mentioned methods
do not suffice our purpose to label each participants as per
the affinity between them. In such social network of groupof people in social gathering, it is more appropriate to decide
the affinity and importance of the node on various factors;
such as, age, experience of work, knowledge about the work,
position in the company, etc., that is the details present inK.
Algorithm 1 Node Affinity
1: procedure G AMMA
2:
Input: L is number of labels
3:
X → {{x1 }, {x2 }, . . . , {xn }} is set of n factor sets.
4:
Each factor set is of size i; all i node5:
Output: γ for all the i node
6:
Start
7:
for all X j → 1 to n do
8:{cn } ← Cluster({xn })
9:
10:
11:
12:

Sorting the labels{ln i } =PSort({cn })n nlγi → 1n i
End

To rank the each individuals, we propose a simple
clustering algorithm to find the affinity label (γ ) for all theindividuals; based on the background knowledge (K). Theγ calculation algorithm is shown in algorithm 1; whereinclustering is based on K-means clustering algorithm. In thealgorithm 1, n is the number of factors considered for affinity
analysis. The Cluster is basically the K-means clustering
algorithm. The Sort is the algorithm which sorts the affinity
labels cn for individual considered factors in ascending orderof the data points in the cluster. Now, these sorted labelsof all the factors are used to calculate the γi for all theindividuals i, using the equation on line 11 of algorithm 1.
At the end of the algorithm, we get γ for all the individuals
in the network. Now, in any given social graph, emotion
propagates among the nodes of the graph. This knowledge ofemotion propagation helps to find the influential correlation
amongst all the nodes. In our approach, this is done by analysis of change in one’s emotion or action causing change inthe emotions of others. The intuition behind the γ is, if oneof the participant has some social ties with the other, thenthey tend to form the virtual groups and in this virtual groupchances of induction is more as compared to other pairs
without any social ties. However, mere γ does not contributeto the influential characteristics of a person. Because, aninfluential character depends on many factors such as talking
manner, expressions, way of conveying, credibility of theperson, etc. So we use γ to guide the algorithm to locate thevirtual groups and later analyse influential node in the virtualgroup first and then the rest of un related nodes. However,it can be said that, higher the γ , higher is the chances of
the node to be an influential node in the particular virtualgroup. The whole process of social influence detection isdivided into two basic steps (i) Emotion diffusion trackingand (ii) Nodal Social influence calculation, as discussed in
the next subsections.

3.1. Emotion Diffusion Tracking
In any given social graph, action and reaction interactions are propagated among the nodes of the graph, in our
approach the interaction are in the form of facial emotion.
This knowledge of emotion propagation helps to find theinfluential correlation amongst all the nodes in the network.
Basically, our algorithm analyse the change in one’s emotions because of change in the emotions of others. However,in order to consider the case where there is induction ofemotion in absence of emotion of other nodes. Therefore
we also considered the induction of emotion on a particular
node by the action of the other node. We tracked the talking
sequence of all the nodes; that is tracking the whether thenode is talking or not talking, if the node is talking thannode is called as active node or else inactive node. So,
induction of emotion is analysed in two different scenarios:
induction by action (a → e) and induction emotion (e → e).
In a → e scenario, emotions of active node induces emotion
on the other nodes; whereas in e → e emotions of a node(irrespective of its being active of inactive) induces emotion
on other nodes. The whole idea in considering the two
scenarios was to refrain from considering wrong orientationof influence between the pair of nodes. Because, may be thechange in emotion of a particular node is because of actionsof particular node and building the correlation merely basedon the emotions may lead to wrong interpretation. Now, thediffusion parameter (∆E ) for both the scenario is calculatedusing the equation 1.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

∆E =τe × ωe
T

(1)

121

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

In the equation 1, τe is the time interval of a particularemotion e, and ωe is a participant’s emotion coefficient for aparticular emotion e. The T is the total frame considered foremotion diffusion calculation ∆E . Whereas, ωe is calculatedusing the equation 2, which is pre-calculated for all theparticipants for every emotion. te is the total time instance,
for which participant has an emotion e and Ne is the total
number of instances, when all the participants has emotione.
ωe =γ × teNe(2)

Other than this, we define the orientation of emotion
diffusion by δ , which can have value 1 or -1, based on
the orientation of the induced emotion. The δ for all theconsidered emotion is listed in the table 3, which is basedon the discussion in [19]. The emotion diffusion in boththe scenarios is calculated using the algorithms 2 and 3.
Let us first discuss the algorithm 2 for a → e scenario,
which accepts the list of inactive (participant with action:
emotion or no emotion) agents for every active (agent withaction: speech or both) agents, and gives ∆E 1 and δ 1 for
all combination of passive and active participants as output.
In the equation on line 8, we calculate the ∆E 1e values
for each emotion e carried by the participant p for time τ e ,during the interval Ta , for which the participant a was active.
Then, the maximum value in set ∆E 1e is chosen as the final∆E 1 . Thus, from the above we deduce that emotion e was
induced on agent p by agent a. Later final δ 1 is calculated
by multiplying the δ of a and e, which gives the directionof the emotion induction. Similarly, in algorithm 3, emotion
diffusion ∆E 2 and δ 2 are calculated for all the pairs of
nodes, irrespective of the active participant, where value of
T is 1. At the end of this step, we get four values, ∆E 1
and ∆E 2 denote the emotion diffusion parameters and δ 1
and δ 2 denote the direction of emotion diffusion values for
all the pairs of participants. Then, we build the influential
correlation amongst the participants, as discussed in the next
subsection.
Algorithm 2 Emotion Propagation (a → e)
1: procedure E MOTION D IFFUSION
2:
Input: a Active participant
3:
P List of passive participant4:
Output: ∆E 1 and δ 1 for all passive participant
5:(where δ is −1 or 1 )6:
for all p enumerate(P) do
7:
for all e Emotion Ee do e×ωτ
8:{∆E 1e ap } → apT p

Algorithm 3 Emotion Propagation (e → e)
1: procedure E MOTION D IFFUSION
2:
Input: A List of all participant
3:
B List of all participant4:
Output: ∆E 2 and δ 2 for all pair of participants
5:(where δ is −1 or 1 )6:
for all a enumerate(A) do
7:
for all b enumerate(B) do
8:for all e Emotion Ee do e
9:{∆E 2e ab } → τ abT×ω b
10:(where T is 1 )
11:
12:
13:
14:
15:

(Maximum of ∆E 2e is final ∆E 2 )(δ e is emotion having maximum for ∆E 2e )∆E 2 ab → max({∆E 2e ab })δ 2 ab → δa × δ e b
End

3.2. Nodal Social Influence
From the above algorithms 2 and 3, we get the emotion
diffusion ∆E 1 ,∆E 2 , δ 1 and δ 2 for both the scenarios. From
this we calculate the influence of each agent on the otheragents, based on the algorithm 4. The objective is to findthe social influence amongst all the agents in the network,in terms of weight of the weighted graph. The input to thisalgorithm is list of ∆E 1 , ∆E 2 , δ 1 and δ 2 for all the pairsof agents.
Based on these values, the weight Wij between agent iand j is calculated using the equation on line 7. Where,summation of ∆E 1 and ∆E 2 for i and j is divided bysummation of ∆E 1 for all the neighbouring nodes of i.
Finally, the orientation of the influence is associated with
the Wij , by multiplying δij . Thus, as an output we get aweighted graph, where weight represents the influence andsign ’-’ and ’+’ represents the orientation of the influence
being induced.
Algorithm 4 Influence Calculation
1: procedure I NFLUENCE
2:
Input: ∆E 1 and ∆E 2 for each pair of nodes3:δ 1 and δ 2 for each pair of nodes
4:
Output: W weight for each pair of nodes
5:
for each pair of node i and j do
6:
δij → δ 1 ij × δ 2 ij
7:
8:
9:

∆E 1 +∆E 2

Wij = ( Pk ij ∆E 1 ij ) × δijik
N (i)(N(i) neighbours of node i)
End

a

9:
10:
11:
12:
13:

(Maximum of ∆E 1e is final ∆E 1 )(δ e is emotion having maximum for ∆E 1e )∆E 1 ap → max({∆E 1e ap })δ 1 ap → δa × δ e p
End

4. Experimental Results and Discussion
This section reports experimental results and evaluating
the obtained results by comparing it with the ground truth of
the testing data. Our system analyses the social influence inthe network, based on change in facial expression amongst© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

122

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

the participants of the network. Our system is implemented
in a multi-agent paradigm, where all the participants are represented by homogenous agents; these autonomous agents
interact with the different modules and finally correlationis build amongst them. The facial emotion analysis module
is build up of AAM based facial feature extraction and aclassifier. The AAM is implemented in Menpo [21] python
libraries, and trained with two publicly available databases;
FERA database [22] and LFPW database [23]. Now, different feature vectors are extracted by tracking the featurelandmarks using the trained AAM is used for training theclassifier. The tracked frontal frame using the Viola Jones
algorithm is depicted in figure 2a, and the figure 2b depicts
the tracked features points (landmarks) and the x&y coordinates are the shape features a.k.a. similarity normalised
shapes. Whereas, figures 2c and 2d depicts the appearance
features (s-aap) and patched based features respectively; thec-aap appearance features are similar to s-aap without any
facial feature movement.

(a) Frontal Face(b) Similarity normalised shapes(s-pts)(c) Similarity normalised appearance (s-aap)(d) Patched SIFT descriptor

Figure 2: Feature Extraction
Later, these feature vectors are feed in to the trainedclassifier, to classify the frames on the basis of carrying set
of FACS. As mentioned before, we compared 3 classifier‘s,
and adopt thecomparatively accurate classifier. The classifier SVM and
SVM+Adb is implemented with python API; whereas, DBN
implementation is based on the DBN for classifying the
MINST handwritten digital database [17], because problem
of classifying handwritten images is similar to the emotion
classification.

In our experiment, we have only considered 5 emotions;
namely, neutral, happy , sad, surprise and angry. The training
and testing of the classifiers are done using the publicly
available CK+ database [11]. The table 1 enlist the scores ofdifferent classifier for all the different emotions; we adopted
SVM+Adb for emotion classification as it can be seen thatthe score of SVM+Adb classifier is more for most of theemotions. The table 3 enlist the combination of FACS foreach considered emotions along with their orientations δ .
In order to test our proposed algorithm, we need a input
frontal video data of small network to detect the emotions
and later build the correlation. However, such dataset is notavailable, therefore a small experiment was conducted in
our laboratory, we record a discussion of five participants
on a video conference; let us say them as participants A,
B, C, D and E. Additionally, the video should be able
to showcase the influence caused by the emotion and anaction. Therefore, we structured the script, such that eachparticipants does the action (talks) for approximately 90
seconds one by one, yielding a total of 450 seconds video.
Further, the script also shows that the person A, B and C
belongs the same organisation and D and E belongs to otherorganisation. Also, A and D are seniors amongst their group
and B has the higher knowledge on topic of discussion.
Based on these assumptions of seniority, knowledge about
the topic, etc, and the algorithm 1 we get 3, 5, 4, 5 and 2
as γA , γB , γC , γD and γE respectively.
Once video is pre-processed, each individual in the
network is associated with an autonomous and homogenous
agent. These agents first interact with the emotion analysis
and the four types of feature vectors are extracted using theAAM tracker.
Later, these vectors are used to recognise the emotions
carried by each agent (frame by frame), by classifying itwith SVM+Adb classifier. Basically, we used binary classifiers; 5 classifiers were used, one for each emotion. Whereas,
the approximate dimensions of s-pts, s-aap, c-aap andpatched based features are 136 (x&y coordinates of 68 landmark points), 56000, 56000 and 132000 (approximately)
respectively.The table 2 represents the emotions detected
per second for all the agents at the end of classificationof all the 450 frames. Now, the social correlation analysis
for both the scenarios is done using the algorithm 2 and 3.
The table 4 and 5 enlist the obtained ∆E 1 and ∆E 2 forboth the scenarios. Further based on the algorithm 4 anddiffusion parameters, we calculate the influence between thepair of nodes, in terms of weights of weighted graph; listed
in the table 6. The evaluation and validation of the results
is done on the basis of the ground truth of the input data,
as discussed in the next subsection.
FACS

SVM

Adb+SVM

DBN

NeutralHappy
Surprise
Sad
Angry

80
81
73
79
70

93
92
83
85
82

75
69
75
70
65

TABLE 1: Emotion detection on CK+ dataset© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

123Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Agent

NeutralHappy

Surprise

Angry

Sad

A
B
C
D
E

100
45
30
27
58

20
190
140
190
140

150
75
80
50
80

35
100
125
153
112

145
40
75
30
60

TABLE 2: Detected Emotion per seconds
Label

Emotion

δ

FACS

1
2
3
4
5

NeutralHappy
Surprise
Sad
Angry

+1
+1
+1
-1
-1

null
6+12
1+2+4+25
1+4+15
4+7+23+25

TABLE 3: Emotion Orientation (δ ) and FACS
Agent

A

B

C

D

E

A
B
C
D
E

0
2.5
0.8
0.6
0.8

1.8
0
1.5
1.2
0.8

0.9
2.2
0
0.4
0.2

0.4
1.5
0.7
0
1.8

0.8
1.2
0.4
3.2
0

TABLE 4: ∆E 1
Agent

A

B

C

D

E

A
B
C
D
E

0
2.8
1.2
0.8
0.4

2.5
0
1.5
1
0.6

1.2
2.5
0
0.7
0.2

0.8
1.2
0.2
0
1.8

0.5
0.8
1.2
2.8
0

TABLE 5: ∆E 2
Agent

A

B

C

D

E

A
B
C
D
E

0
-2.4
0.512
0.34
0.3

-0.81
0
0.53
-0.372
0.22

0.75
3.13
0
0.33
0.114

0.3
-0.93
0.24
0
1.38

0.27
0.45
0.3
2.5
0

TABLE 6: Final Weights

4.1. Discussion
The input data to our system is the scripted group discussion of 5 persons (nodes); wherein the roles and responsesof each node are known. This information gives idea about
the social ties between the nodes in the network. Further
in to build the ground truth of the fed data, we analyse thechange of emotions of each node with respect to other for
all time instances. In order to do that, we plot the changeof emotion with time for all nodes in both the scenarios:a → e and e → e. The emotion propagation curves 3
and 4 represents scenario a → e and e → e respectively;
where 3 represents the emotion change curves for all the 5nodes with respect to one node is active and 4 represents
the emotion change curve for throughout the discussion

irrespective of any node is active. The y-axis denotes the 5emotion labels normalized along the axis and x-axis denotes
time in second. If we analyse curves for both the scenario
point by point and emotion labels listed in table 2, we
could build the ground truth of existing correlation in theconsidered network. Firstly, from the emotion propagation
curve 3 in a → e scenario, It can be said that emotion
diffusion of B induces an positive emotion change on C
ofently. Also, B induces negative emotion change on nodeA, i.e., when A is happy then most likely the node B is sad.
Similarly, it can be said that emotions of E is more often
influenced by emotions of D. Whereas from the emotion
propagation curve in the figure 4 for e → e scenario, It canbe said that, B induces emotion on C and A; also inductionof emotions in reverse sense is not reciprocated from thecurves. Such correlations can also be observed between
agent D and agent E. Now next step is, based on these key
observed characteristic in the graph, we build the groundtruth. Further, in order to validate the predicted influential
correlation among the individuals, we simply compare theground truth conclusion of correlation with the obtainedcorrelation.
Lets analyse the predicted weights (influence) listed in
the table 6. From the weights, it can be said that node B
induces node A in the opposite orientation with weight -2.4,
whereas agent B induces C to a greater extent; that is 3.13.
Similar correlation is observed between node D and node
E. Node D induces node E with weight 2.5; although, viceversa correlation is not observed in both the cases. Other
than this, node B induces node D in negative sense. It canalso be said that, if the value of weight is less, that means
it has lower induction; then that edge or correlation can beignored or removed form the network. Therefore, we set a
threshold value for weights to be considered, the weights
below that threshold is avoided in the final influence graph.
As depicted in the figure 5, influence on nodes E → C, C →D, A → D and E → B are ignored because the influence isless then set threshold value, that is value less than 0.3. Thus,
it will reduce the complexity of other searching algorithms(if any) in the network. Finally, by analysing the obtained
results, it can be said that the emotion propagation in thescripted discussion is very well represented in terms ofweights of the weighted graph. Therefore, it can also be
said that our algorithm proves to be efficient to measure theinfluence in the network.
Such system can find many application such as negotiation or consensus building among the group of people, alsofinding the influential person can help the organization totarget the appropriate person, who can persuade large groupwith its influencing capability.

5. Conclusion
We introduced an algorithm to analyse the influential
correlation in the small social networks formed by group ofpeople in discussion or business meetings in a multi-agent
paradigm. Wherein, we were more interested in understanding the role of human facial emotion on social influential© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

124

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)(a) Active Node: A

Figure 5: Influence Grpah(b) Active Node: B(c) Active Node: C(d) Active Node: D(e) Active Node: E

Figure 3: Emotion Propagation

based on emotion diffusion parameter and node centrality
calculation as discussed in the above sections. From thesealgorithms, we should be able able to determine one influential person in the whole group or multiple influencer foreach virtual groups in the network. Finally, we evaluated andvalidated our system based on the ground truth of the video
data of a scripted discussion, fed in to the system. Concluding that the final output of our system was able to reflect
the emotion propagation in the scripted discussion. As afuture work, it would be interesting to extend our method
to larger real life networks. Moreover, we are thinking ofincorporating emotion from speech data along with facial
emotion, so that the resultant emotion can depict humans
emotion more closely and be used for consensus building.
We also plan on investigating how social association can bebuild on any anonymous network by different social network
mining methods.

Acknowledgments
This work has been partially supported by the project
Large-scale Consensus Support System based on Agents
Technology in the research area Intelligent Information Processing Systems Creating Co-Experience Knowledge andWisdom with Human- Machine Harmonious Collaborationof JST CREST projects.
Figure 4: No Active Nodescorrelation and how effectively a person can induce his/heremotion on other person. So, If any particular participant in
the network brings a high degree of emotion change on theother participant/s in the network, then it can be said thatthe participant is an influential node in the network.
In our system, humans emotions were considered foranalysing the influential analysis; whereas emotion recognition was based on the AAM feature tracking and the
SVM+Adb classifier for emotion classification. Also, we
compared three classifiers; namely, SVM+Adb, SVM and
DBN and choose the comparatively good classifier for oursystem. Our algorithm for influential correlation building is

References[1]

C.-W. Chen, R. C. Ugarte, C. Wu, and H. Aghajan, “Discoveringsocial interactions in real work environments,” in Automatic Face &Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on. IEEE, 2011, pp. 933–938.

[2]

D. Zhang, “Probabilistic graphical models for human interaction
analysis,” IDIAP, Tech. Rep., 2006.

[3]

J. Tang, J. Sun, C. Wang, and Z. Yang, “Social influence analysis
in large-scale networks,” in Proceedings of the 15th ACM SIGKDD
international conference on Knowledge discovery and data mining.
ACM, 2009, pp. 807–816.

[4]

A. Anagnostopoulos, R. Kumar, and M. Mahdian, “Influence andcorrelation in social networks,” in Proceedings of the 14th ACM
SIGKDD international conference on Knowledge discovery and data
mining. ACM, 2008, pp. 7–15.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

125Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[5]

M. McPherson, L. Smith-Lovin, and J. M. Cook, “Birds of a feather:
Homophily in social networks,” Annual review of sociology, pp. 415–
444, 2001.

[6]

P. Stone and M. Veloso, “Multiagent systems: A survey from amachine learning perspective,” Autonomous Robots, vol. 8, no. 3, pp.
345–383, 2000.

[7]

O. H. Jensen, “Implementing the viola-jones face detection algorithm,” Ph.D. dissertation, Technical University of Denmark, DTU,
DK-2800 Kgs. Lyngby, Denmark, 2008.

[8]

S. Happy and A. Routray, “Robust facial expression classification
using shape and appearance features,” in Advances in Pattern Recognition (ICAPR), 2015 Eighth International Conference on. IEEE,
2015, pp. 1–5.

[9]

A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M.
Prkachin, and P. E. Solomon, “The painful face–pain expression
recognition using active appearance models,” Image and vision computing, vol. 27, no. 12, pp. 1788–1796, 2009.

[10] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance
models,” IEEE Transactions on Pattern Analysis & Machine Intelligence, no. 6, pp. 681–685, 2001.

[23] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300
faces in-the-wild challenge: The first facial landmark localization
challenge,” in Computer Vision Workshops (ICCVW), 2013 IEEE
International Conference on. IEEE, 2013, pp. 397–403.
[24] T. Simon, M. H. Nguyen, F. De La Torre, and J. F. Cohn, “Action unit
detection with segment-based svms,” in Computer Vision and Pattern
Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp.
2737–2744.
[25] T. Finin, Y. Labrou, and J. Mayfield, “Kqml as an agent communication language,” Software Agents. MIT Press, Cambridge, vol. 239,
1995.
[26] A. Ruiz, J. Van de Weijer, and X. Binefa, “From emotions to action
units with hidden and semi-hidden-task learning,” in Proceedings ofthe IEEE International Conference on Computer Vision, 2015, pp.
3703–3711.
[27] Y. Zhu, F. De la Torre, J. F. Cohn, and Y.-J. Zhang, “Dynamic
cascades with bidirectional bootstrapping for action unit detection in
spontaneous facial behavior,” Affective Computing, IEEE Transactions
on, vol. 2, no. 2, pp. 79–91, 2011.

[11] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, andI. Matthews, “The extended cohn-kanade dataset (ck+): A complete
dataset for action unit and emotion-specified expression,” in Computer
Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE
Computer Society Conference on. IEEE, 2010, pp. 94–101.
[12] W.-S. Chu, F. Torre, and J. Cohn, “Selective transfer machine for
personalized facial action unit detection,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2013, pp.
3515–3522.
[13] D. G. Lowe, “Object recognition from local scale-invariant features,”in Computer vision, 1999. The proceedings of the seventh IEEE
international conference on, vol. 2. Ieee, 1999, pp. 1150–1157.
[14] P. Ekman and E. L. Rosenberg, What the face reveals: Basic andapplied studies of spontaneous expression using the Facial Action
Coding System (FACS). Oxford University Press, 1997.
[15] C.-W. Hsu, C.-C. Chang, C.-J. Lin et al., “A practical guide to support
vector classification,” 2003.
[16] X. Li, L. Wang, and E. Sung, “Adaboost with svm-based component classifiers,” Engineering Applications of Artificial Intelligence,
vol. 21, no. 5, pp. 785–795, 2008.
[17] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm
for deep belief nets,” Neural computation, vol. 18, no. 7, pp. 1527–
1554, 2006.
[18] J. Sun and J. Tang, “A survey of models and algorithms for social
influence analysis,” in Social network data analytics. Springer, 2011,
pp. 177–214.
[19] K. Ghamen and A. Caplier, “Positive and negative expressions classification using the belief theory,” International Journal of Tomography& Statistics, vol. 17, no. S11, pp. 72–87, 2011.
[20] F. Bellifemine, F. Bergenti, G. Caire, and A. Poggi, “Jadea java agent
development framework,” in Multi-Agent Programming. Springer,
2005, pp. 125–147.
[21] J. Alabort-i Medina, E. Antonakos, J. Booth, P. Snape, andS. Zafeiriou, “Menpo: A comprehensive platform for parametricimage alignment and visual deformable models,” in Proceedings of
the ACM International Conference on Multimedia, ser. MM ’14.
New York, NY, USA: ACM, 2014, pp. 679–682. [Online]. Available:
http://doi.acm.org/10.1145/2647868.2654890[22] M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, “The first
facial expression recognition and analysis challenge,” in Automatic
Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE
International Conference on. IEEE, 2011, pp. 921–926.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

126

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Sharing Economy: Sustainable Development Driven
by Internet
Zhu Xinyue

Guo Rong

Institute of Science of Technology
Investment
Chinese Academy of Science andTechnology for Development
Beijing, China
zxykjb@163.com

Management committee
Chinese Academy of Science andTechnology for Development
Beijing, China
grkjb2004@163.com

Abstract—The Sharing Economy as a brand new businessmodel is expanding rapidly in the world because of its evident
advantages in energy efficiency, effectiveness improvement andcost reduction. In particular, after the latest international financial
crisis, countries in the world are actively sharing new economic
fields, making full use of connected infrastructure and researchinnovations to improve and develop the Sharing Economy. At
present, the Chinese economy is in the stage of its new normal. The
research innovations, mainly represented by Sharing Economy,
are developing rapidly. In particular, the "Internet+" car field in
China is catching up quickly with other countries in the arena of
Sharing Economy. However, China still faces some problems thatcannot be neglected, e.g., friction between the old and new
economic models and yet to improve regulatory system. In light of
this, this thesis works to make general conclusions and offer
relevant policy suggestions after studying the current developmentof Sharing Economy in the world.
Keywords—sharing e conomy ； internet
research & innovation；venture capital

Zhao Xuankai
Central University of Finance andEconomics
Beijing, China
xuankaizhao@126.com

of output created by the global Sharing Economy will
increase by 26 times from the current USD13.5 billion to
USD348.9 billion by 2025.
I.

SHARING ECONOMIES ARE SURING FORWARD IN THE
WORLD

Sharing Economies can be classified into the following
categories (Fg.1.).

technology ；Sharing Economy, also known as "Collaborative
Economy", is a new cooperative production andconsumption mode that enables the sharing of various goods
and services by virtue of the Internet; and it has such
advantages as cost reduction, making the utilization of
resources more effective, and narrowing the gap between therich and the poor. Therefore, these types of economies haverecently become one of the important forms of achieving
sustainable development by use of scientific andtechnological innovations. That is, in a Sharing Economy,
the Internet (mobile Internet) will be made use of to promote
the combination of technology and the economy (traditional
industries) and achieve the transformation from labor
intensive industries to knowledge, information, andtechnology intensive industries by use of scientific andtechnical innovations so as to transform the previous patternof economic development simply based on being driven bycost factors, dedicating large amounts of resources andenvironment consumption. It is predicted that the total value•

The first category is space sharing, which includes
the releasing the information and prices of idle
properties through various platforms on the Internet for
tourists, people who are taking business trips, and otherpeople with a need for, or those who are looking for
housing, to select.

•

The second category is the sharing of transportation
tools of many types and in a number of forms,
including "private car services", "public car rentals",and "Rideshare".

•

The third category is the sharing of skills and time,
which for the most part means to match services by useof intangible assets owned by community members,
such as professional skills and time. Included in thiscategory are professional service platforms, P2P labor
employment, and scientific research crowdsourcing.

•

The fourth category is known as merchandise
sharing, which mainly includes the leasing of finishedproducts, commodity customization, and a trading
platform for second hand goods.

•

The fifth one is capital sharing, which includes online
P2P capital borrowing and lending platforms, as well as
crowdfunding, a way to raise money through thecollection of contributions of various sizes by way of
the internet, events, or even mail subscriptions.

•

The sixth category is the sharing of applications inother fields, including online classes with Coursea as arepresentative and the model of sharing food and1© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

127

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

beverages, namely "meeting friends at the dining table".

Economy originated, the United States continuously
strengthens and supports fair competition and its protection
over the rights and interests of consumers and sharers; theBritish government puts forward the creation of a global
center for Sharing Economies on the due to the fact that
London is one of the world’s largest financial and tradecenters. The British government also takes measures
including establishing an economic technology platform (thefirst TravelTech Lab), carrying out pilot city programs,
holding creative contests, formulating evaluation standards,
implementing special tax policies, and opening data
information (such as criminal records) to encourage
economic and social sectors to take part in the Sharing
Economy. The German government not only supports theparticipation of Daimler Group, BMW, Volkswagen,Mercedes-Benz, and other automobile manufacturers insharing, but also encourages community residents to use newenergy technology to set up "electricity production
cooperatives" while vigorously promoting shared "street
public welfare" projects, such as street public bookcases andwardrobes.

Fig.1. Classification of Sharing Economy
Source: Sharing New Buying Collaborative Economy

II.
The following points should be noted and paid
attention to in the development process of Sharing
Economies:

ANALYSIS ON REASONS FOR RISE OF THE SHARINGECONOMYAt present, the global economy is still faced with many
challenges; the environment and the income gap between
the rich and the poor are still the main obstacles restricting
the development of all of the countries in the world. While
reducing production and living costs, Sharing Economy alsoenhances production efficiency, weakens the ownership ofassets by means of scientific and technological innovation inorder to achieve the improvement of efficiency, cost
reduction, and shrinkage of the gap between the rich and thepoor. The rapid development of Sharing Economies is most
likely attributed to the following two points:

First, large multinational companies have beenplaying an active role in exploring the Sharing Economy
after the international financial crisis. Daimler haslaunched the Car2go car rental project and put about
125,000 cars in 29 cities in a number of countries on theroad; IKEA and Patagonia have successively provided theircustomers with relevant platforms on which goods can be
resold; Marriott Hotel Group has cooperated with LiquidSpace, an office space leasing platform, and rented out hotel
conference facilities by hour or by day in 432 hotels ownedby the Group throughout the world in order to meet theneeds of entrepreneurs for office space.

On the one hand, a new generation of information
technology propels the development of Sharing
Economies. In the less developed period of information
technology, "sharing" exists only on a small scale or amongacquaintances. "Collaboration" was first used in the 1950s
and its usage rate began to soar after the invention of thecomputer and internet since the 1960s. 2014 marked therapid development of internet around the world and thedevelopment of communication technology weakens theownership of articles, breaks through the limitation of spaceon intangible assets and urges people to pay more attention
to energy saving, efficiency improvement, and protection of
the environment as well.

Second, Sharing Economy has the alluring featuresof low economic costs and high social benefits. Multiple
studies have shown that the price of a house subject to shortterm rental is 20% to 70% lower than that of a hotel at thesame level and location; a "shared car", or a special type of
car used for carpooling which works effectively can replaceabout 4 to 10 private cars. Compared to year 2000, thenumber of German people using car sharing services has
increased by 375% while the number of private cars owned
by every 1,000 young people with an age range between 18
to 29 years old has decreased by 34%; while in the city ofBremen, the carbon dioxide emissions have reduced in theform of 1,600 tons each year by using "shared cars". In the
future, it’s estimated that other cities in various countries
around the world will also be able to decrease carbon
dioxide emissions in a similar manner.

On the other hand, the low cost accelerates thedevelopment of Sharing Economies. Compared with thetraditional hotel industry, shared houses have moreadvantages and are priced lower. According to the survey
and research of PRICEONOMICS, in most major cities in
the United States, the prices of apartments on Airbnb are
21% cheaper than that of hotels on an average basis. In
addition, the survey and statistics of iRESEARCH show thatthe prices of houses which are used for short term rentals in
China are significantly lower than that of hotels at the same

Third, the governments in major countries andregions attach great importance to the promotion of
Sharing Economies. As the place where the Sharing2© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

128

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

level and location, and the discount rate of houses used forshort term rentals can often reach as high as 70%.
III.

In addition, people at different age levels have shown
respective levels of willingness to participate in network
sharing. People aged from 10 to 29 have the strongest
willingness to share. In the coming years, as this group
gradually becomes the backbone of society, the role of theInternet in driving mutual benefit, sharing, cooperation, andinnovation will be more obvious and will be shown in
different ways through various manifestations.

THE SHARING ECONOMY IN CHINA HAS A HUGE
POTENTIAL FOR DEVELOPMENT

The Chinese culture has long had the spirit of "sharinghappiness with others is better than enjoying happinessalone". After the Sharing Economy was imported into China,
its total revenue and the domains it covered in China grew
rapidly to catch up with and even surpass the advanced
world level. For example, the total revenue of the online
short term rental industry was only RMB 7 million when it
started in 2011 but reached RMB 4.05 billion in 2014, and it
is expected to exceed RMB 10 billion in 2015, indicating
the growth of more than ten thousand times over five years.
The fields and representative enterprises (platforms) of
Sharing Economy in China are shown in Table 1 below:
TABLE 1

MAIN APPLICATION FIELDS OF CHINA'S SHARING ECONOMY

Short Term House Rental
Short Term Office Rental
Means of Transportation

YOUTX.COM, MAYI.COM, TUJIA.COM,
XIAOZHU.COM
Fig.3. Proportion of Netizens at Different Ages Willing to Share on theInternet

SOHO 3Q, YOU + International Youth
Apartments

Second, the amount of trust people have in theInternet has gone up gradually. Compared to in 2007,
trust internet users in the Internet had increased significantly
from 35.1% to 54.5% in 2014. The increase in the
proportion of trust in the Internet provides a stronger
guarantee for the future development of Sharing Economy
in China and has made up for the lack of the traditionalsocial trust to some extent.

KUAIDI & DIDI GROUP, eHi Car Services,
PPZUCHE.COM

Capital

RENRENDAI.COM, DREAMORE.COM

Goods

BUY42.COM

Time and Skills

ORDER APP, SKILLBANK.CN

Education

OTONE EDUCATIONFirst, the willingness to share of market players has
increased. Thanks to the rapid development of the Internet,
the living costs of consumers have been cut down and alarger space has been provided for the development of the
Sharing Economy. According to China Internet Network
Development State Statistic Report (Hereinafter referred toas Statistical Report), 60% of internet users who were polled
hold a positive attitude in regards to network sharing, 13%of which are extremely willing to conduct network sharing
activities and 47% are willing to do so.

Fig. 4.

Proportion of Trust of Netizens in the Internet

Third, keen attention and recognition have beensuccessively received from venture capitals. Whether acompany engaged in Sharing Economy as a new business
form and business model can get access to financial supportin time is the key for its rapid expansion and development.
Chinese companies involved in the Sharing Economy havejust ushered in a financing boom: on April 10, 2015,
TTYONGCHE.COM completed its Round C of financing
and its value has been closer to USD200 million; early in
July 2015, XIAOZHU.COM finished its Round C offinancing of USD60 million; in the middle of July 2015,
MUNIAO.COM obtained its Round A of financing of USD
60 million; early in August 2015, TUJIA.COM achieved its
Round D of financing of USD 300 million, and its valuation

Fig. 2. Survey on Sharing Willingness of Chinese Netizens

3© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

129

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)has exceeded USD 1 billion. In the venture capital industry,the valuation of enterprises related to short term renting
industry is increasingly improved, which reflects strong
market development potential of Sharing Economy in China
to a certain extent.

A. Didi and Kuaidi occupy most of the market shar
With the rapid expansion of Uber all over the world,
China's shared car renting industry represented by Kuaidi
and Didi are the leading trend. After China's first taxi
hailing software came out in 2012, Didi and Kuaidi had
been grabbing off market shares with strong market
occupation rate and completed the final merger in February
2015. The existing data shows that as of May 2015, 70% ofdrivers within the national coverage area use Didi and
Kuaidi platforms every day, which has become the world's
largest online travel platform with about 6 million orders
every day and the trade scale of which is 6 times that of theworld's second travel platform. According to statistics on
the market share of professional cars, from May 2015, thecoverage rate of Didi and Kuaidi's active users in the special
car market of China reached 86.2% (see Figure 5).

Fourth, "Internet+" has become the major stimulusthat shifts the growth pattern of the Chinese economy.
At the 2015 Summer Davos, the Chinese Premier Li
Keqiang pointed out that the world’s Sharing Economies aredeveloping rapidly, thus becoming a new path of
stimulating economic growth. This was the first time for the
Chinese leader to voice China’s acceptance of the Sharing
Economy. At present, China is moving from rapid economic
growth to the moderately fast growth. The Chinese
economic structure and development methods are facing
major changes. In addition, it is imperative and urgent toabandon the previous pattern of economic growth which
included the mammoth consumption of resources. As one of
the major strategies adopted by the Chinese economic
development at present, “Internet+” would, when the global
economy is under great downward pressure, encourage
people to make technological innovations, upgrade
institutional innovations, management and marketing
innovations, and research & innovation capacity. It is also
the important carrier of mass innovation andentrepreneurship. As the supply side and structure reform is
gradually getting attention from the Chinese government
and national leaders of China, the Sharing Economy willdefinitely become the new stimulus for the future shift ofeconomic growth patterns.
IV.

Fig. 5. Top Five with the Highest Coverage of Special Car Active Users in
China in 2015
Source: Enfodesk

B. The modes of operation of Didi and Kuaidi
Considering different population densities, town
distribution, and development processes of first and second
tier cities in China, Didi and Kuaidi Group adopted thestrategy of "multiple business and line coordination",
namely, operation with various forms including taxis,
professional cars, and shared payment carpooling. In termsof taxi business, Didi and Kuaidi adopt the mode of project
management POWERED BY DIDI, and establishes unified
standards, that is, all orders will be uploaded the network,
and if no taxi driver gives a response within 60 seconds, thesystem will automatically judge the transport capacity ofnear special cars, express cars and carpools, so as to provide
consumers with more choices and realize real "zero empty
running".

CHINA'S "INTERNET+" CAR FIELD HAS BEEN IN THE"LEADING" STATE

A "shared car" refers to the car shared by many people,
that is, the driver only has the right to use rather own it, anda certain company coordinates the cars and is responsiblefor issues concerning the cars such as insurance and parking.
This mode of car sharing that rises quickly based on thedevelopment of internet technology not only lowers the costof living, but also helps to ease traffic congestion andreduces highway wear, air pollution, dependence on andconsumption of energy resources at the same time, thus
ultimately achieving sustainable development. These aresome of the many benefits of shared cars which arebecoming more and more popular as time goes on.

In the special car market, Didi and Kuaidi pursues agoal of "dignity", namely enabling users to travel more
decently and with more dignity; therefore, Didi and Kuaidi
puts forward the "ACE Certification" by reference to thestandards of the motorcade for state guests, selects the best(and even the most handsome) drivers for training andarranges the most complete in-car services reaching thestandard of five star; the drivers meeting the standards of"ACE Certification" will also enjoy 30% of the negotiatedprice.

According to the report, there are more than 1 billion
cars running on the road around the world, but these cars arein idle state in about 96% of the time. On the other hand,more than 15% of the space in our cities is occupied bythose cars which are restricted and idle. Therefore, in 2010,
Uber began to launch its taxi hailing service based on its
mobile terminal in San Francisco, achieving the rapid
integration of resources which were not in use, and at thesame time effectively reducing the travel cost of passengers,
and providing drivers with the possibility of taking a parttime job and getting additional, supplemental income.

C. "Internet+" car field has extended to the freighttransport field
At the same time when Didi and Kuaidi develop
quickly, the field of road freight transport has also

4

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

130

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

undergone a wave of innovations and starting businessunder the mode of the Sharing Economy. At present, somecompanies engaged in freight transport with the mode ofonline APP, such as 56qq.com, are developing and begin totake shape. As one of the earliest explorers of Sharing
Economy modes in the world, Transfar Company of Chinabegan to put the freight logistics port mode into practice in
2003, helping drivers and freight forwarding companies
implement highly efficient matching of logistics informationin the form of offline aggregation and information exchange.
As the market environment for the Mobile Internet becomesincreasingly mature, Transfar Road Logistics Port graduallyuses the Internet in its work to achieve rapid expansion andthe network effect. In the long term, the shared logistics
mode represented by Transfar Road Logistics Port is
expected to surpass traditional large scale logistics giants interms of the overall operating efficiency and lead the global
industry renovation.

carbon emissions from taxis. According to the data of theUnited Nations Environment Program, a common tree canabsorb 0.012 ton of carbon dioxide each year, so 5.7 million
tons of carbon emissions are equivalent to the ecologicalcompensation quantity of 475 million trees throughout ayear.
In addition, the development of the "carpooling"business also has a great influence on the reduction ofcarbon emissions. If the carpooling implemented refers to
two people taking one taxi, the utilization rate of taxies is
74.3%, and it is expected that carbon emissions will
decrease by 14.649 million tons each year. Also found to betrue according to the above data, 14.649 million tons ofcarbon emissions are equal to the ecological compensation
quantity of 1.22 billion trees throughout a year, and areapproximately equal to every single person in China, thecountry with the largest population in the world, planting atree. This form of business can not only reduce the pressure
put on the environment by cars, but can also help to restore
and ease some of the damage that has already been done.

D. The field of shared cars helps boost technologicalinnovations
According to the data set forth in the Statistics Bulletinon the Communication Operation Industry in 2013 of China,
the penetration of mobile phone users has reached the large
number of 90.8 phones per one hundred people, whichmarks that fact that the mobile phone market of China hasentered the era focusing on updating its demands. However,the update cycle of mobile phones is usually two to three
years, while the update cycle of smart phones is less than
two years. In order to improve the efficiency and thepossibility of grabbing many orders as fast as possible,
many Didi and Kuaidi drivers who never used smart phones
before have begun to use smart phones and replace theirown SIM cards with 3G/4G ones. This, on the one hand, cancontinue to boost smart phone hardware and software andspeed up the innovation in a bid to meet more and moreconsumer demands, and on the other hand, will promote theupgrade of consumption structure in the informationindustry, so as to meet the demand of internet users for
network speed.

V.

MAIN PROBLEMS IN THE DEVELOPMENT OF SHARINGECONOMIES

Since it is something that is relatively new in the world,
Sharing Economies have strong development potentials but
also face many underlying problems which can be seen in
the examples below:
First, the friction between the old and new
economic models increases. The rapid development of the
Sharing Economy brings a great challenge to the existing
law and system of regulations and may get the interests of
the original large service groups affected; there are no set
rules on certification and insurance, etc. in such fields as
sharing of accommodation and transportation means;practitioners of the Sharing Economy may consequently
obtain higher benefits while traditional industries thus haveto pay more costs. There have been incidents where thelegality of professional cars was debated over from time totime in various countries in the world. In the first half of2015, the incidents where professional cars hailed throughthe internet were debated under law happened in 15 cities
within China; similar cases have also occurred in many
countries, including the United States, Britain, France, andGermany.

E. Shared cars may effectively lower the empty run rate
and reduce carbon emissions
Shared cars not only can drive the boost oftechnological innovation, but also make outstanding
contributions to energy saving and efficiency improvement
at the same time. According to the Research Report on
Economic and Social Impacts of Taxi Hailing (2014) by
Tsinghua University, 55% of drivers believe that the empty
run rate in each month decreases by 10%, while 41.2%
drivers think that the empty run rate in each month falls by
10%-30%. If it is assumed that, before the decrease of theempty run rate, the number of taxi passengers and thenumber of taxis are 1 respectively, on the premise that theempty run rate falls by 10%, 1+10% passengers can besatisfied under the condition of constant current transportcapacity as estimated in the ideal state. However, before theempty run rate decreases, in order to meet the demands ofpassengers, there must be 1+10% taxies. The 10% oftransport capacity saved can reduce 5.7 million tons ofSecond, the lack of supervision has resulted in theoccasional occurrence of security problems. Even though
the platforms engaged in the Sharing Economy strictly
examine the qualifications of both the suppliers anddemanders, the absence of supervision and supporting laws
still occasionally leads to repeated occurrences of security
problems in countries around the world. For example,
drivers of professional cars are notorious for their misdeeds
in the United States, India, and other countries. It is also
common that the safety of life or property of owners andtenants of houses used in online short term rentals is under
threat. In addition, no credit system covering most countries
in the world has been formed, which also challenges thesupervision over multinational sharing.

5© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

131

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Third, regional IT technical resources are
distributed unevenly. The gap between the rich and thepoor in various countries in the world continues to increase
and the development level of science and technology varies
from country to country. The gap in terms of Internet
penetration is rather clear: the Internet penetration in Europe
is generally very high while that in African countries isnotably lower. According to the Measuring the Information
Society Report (hereinafter referred to as the "Report")
promulgated by the Intentional Telecommunition Union(ITU), a professional organization of the United Nations(UN), in November 2015, various regions still differ greatly
in the development index of information andcommunication technology (hereinafter referred to as the"Index"); 29 out of 37 African countries are part of thequarter of countries with the lowest Index, highlighting theimportance of solving the gap between Africa and otherregions of the world. Moreover, it is also pointed out in theReport that the average growth rate in the Index of highincome economies is significantly more than that of low
income economies, reflecting the close connection between
the development of information and communication
technology and also the national income level. The higher
the income level of a country is, the more advanced theircommunication technology will be.
VI.

venture capitalists with its strong advantages in energy
saving, efficiency improvement, cost reduction, shrinkageof the gap between the rich and the poor, and other aspects,
and it develops rapidly across various countries all over theworld. However, it still cannot be ignored that thedevelopment of the Sharing Economy is confronted withmany obstacles arising from the frictions between the oldand new economic models, the lack of legal supervisionsystem, and uneven distribution of communication
technology. As such, there are some general suggestions
below which may help to balance out the disadvantages
found in the rise of Sharing Economies. First, comparativeadvantages shall be utilized to urge various countries to
explore new models of the Sharing Economy. Second, it isnecessary to accelerate the construction of a fair, inclusive,
national, and highly unified subtle supervision mechanism.
The last but not the least, the Internet coverage shall beenhanced in various countries, especially those in Africa.

REFERENCES[1] Jeremy Rifkin, The Zero Marginal Cost Society, 2014, London, UK,
Pal grave MacMillan Press.
[2] Research Department of CITIC Securities, “Uber: Sharing Economy,
Reconstructing Transportation and Others”, 2015, internal report.
[3] Mu Nan, “Data Published by Didi and Kuaidi Group after Merger for
the First Time”, unpublished.

BASIC CONCLUSION

In short, the Sharing Economy no longer simply
emphasizes the ownership of articles but achieves the"Access but not Ownership" of assets by the use of theInternet as a medium; it has attracted broad attention from.

6

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

132

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

The Existence of Herding in P2P Lending and the
Rationality of It
An empirical study from China
Xiangyu Wang

Yan TuSchool of Information
Central University of Finance and Economics
Beijing, China
tcxdd@163.com

School of Information
Central University of Finance and Economics
Beijing, China
tuyancufe@163.com

Abstract—Online peer-to-peer(P2P) lending is an innovative
way of lending, which enables borrowers fetch money fromindividual lenders online rather than through traditional
financial institutions. Using data from Prosper, which is thelargest P2P loan platform in the United S tates, many researchers
have found that there is a herding behavior in P2P lending and as
opinions vary, the rationality of it is uncertain. We study herding
behavior in P2P lending in China. Data is based on PPDai, whichis China’s largest P2P lending platform. To elaborate the effect ofherding, we define a herding index which describes the extent ofherding rather than simple bidding numbers as many
researchers do. Based on some limitation of current research, we
extend our model and empirically test the existence of herding
and the rationality of it. The research enriches the study ofherding in P2P lending. We divide herding behavior into two
aspects, which is beneficial to deep understand the herding andelaborately express the extent of herding, which benefit theexamination of the rationality of it.

of loan. The website will show the personal information ofborrowers including gender, occupation and credit rate.
Lenders who intend to lend then bid on the listing. Other
choices of bidders will be shown in the page of listing, which isa potential cause of herding.

Keywords—P2P lending; herding; PPDai; herding index;
rationality

We intend to solve two problems: first, whether there is aherding behavior in P2P lending platform and second, whether
such a herding behavior is rational. Different fro m current
research, we analyze the representation of extent of herding
and incorporate an index into the model e xamin ing therationality of herding. We believe the index can perfectly
represent the extent of herding.
The rest of the paper is organized as follows. First, we
review the literature about P2P lending and herding behavior.
Second, we elaborate the limitation of current research andpropose our method to improve current work. Last, we describe
our data, analyze results and draw a conclusion.
II. LIT ERAT URE REVIEW

I. INT RODUCT ION
The past decade has witnessed a rapid growth of P2P
lending, which is accelerated by Web 2.0. In P2P lending,borrowers and lenders transact on online platforms, such as
Prosper, Lending Club, PPDai, and Zopa. According to thereport of Price water house Coopers in 2015, the US P2P
lending platforms issued approximately $5.5 billion in loans in2014. In China, the volume of P2P loan is even larger andreached $41.3 billion in 2014(Wang et al. 2015b). P2P lending
makes it possible that both borrowers and lenders can transact
online without intermedia ries such as banks (Lee and Lee2012). P2P lending can not only finance SMEs but also financeindividuals. P2P lending is beneficial to both borrowers andlenders. Due to financial decentralization, borrowers can getloans directly fro m lenders and pay low interest rates (Light
2012). In addition, P2P lending can mitigate financial
exclusion, i.e., P2P lending can finance borrowers with low
credit grades (Komarova Loureiro Y, Gon zalez L 2015). For
lenders, they can obtain more opportunities of investment an dget high returns(Magee 2011).

A. P2P lending
P2P lending has attracted both practical and theoreticalattention due to its popularity. Research concerning P2P
lending can be divided into three parts. First, early research
focuses on the reasons for the emergence of online P2P lending.
Second, current research examines the determinants of funding
success and default, which considered as the basis of decisionmaking of borrowers and lenders. Third, some research
investigates the performance of online P2P loan taking risk into
consideration. Above them, the second part is very importantand there is extant research about it. Borrowers wish to getloans in low interest rates while lenders make efforts to discern
whether borrowers pay loans in time. Research based on theperspective of borrowers is scarce. Borrowers face a tradeoff
between loan amount and interest rate, i.e., borrowers may
have to set high interest rates high to ensure the funding
success (Puro L et al. 2010). Research based on perspective ofborrowers usually use auction theory and provide advice ofloan amount and interest rate setting (Puro L et al. 2010, Li S etal. 2011).

A typical process of P2P lending is as follows. First,
borrowers apply for a loan request called listing, containing theinformation of loan amount, interest rate, duration and purpose

Research based on perspective of lenders usually focus on
the factors of decision-making. One of problems in P2P© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

133

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

lending is informat ion asymmetry, i.e., lenders don’t know thecredibility of borrowers for certain as borrowers do (Emekter R2015). Information asymmetry leads to high risk for lenders. In
traditional banking systems, information asymmetry can belessened due to financial intermediaries, the absence of whichmake it acute in P2P lending. Further, the inherent anonymityof online environment intensifies the problem. Since
information asymmetry between borrowers and investors leadto moral hazard (St iglitz and Weiss 1981) and adverse
selection (Akerlof, 1970), lenders must be cautious when
making decisions.
In P2P lending, lenders use both hard information and soft
information about borrowers to make investment decisions(Iyer et al. 2009; Freed man and Jin 2014; Lin et al. 2013). Hard
information refers to quantitative information that can be
accurately expressed, e.g., demographic information, debt toincome ratio(BC Collier, R Hampshire, 2010), FICO score,credit grade(Duarte et al. 2012;6). Hard informat ion is often
difficult to obtain, insufficient, or unreliable, so lenders may
tend to soft information, which is available in P2P lending
platforms and can be diagnostic (Michels 2011). Soft
information refers to non-standard qualitative information (Iyeret al. 2009), e.g., narrat ive, appearance, social networking(e.g.,
virtual community and friendship). Lin et al. (2013) discoverthat strong social ties and friendship can increase theprobability of funding success and decrease the probability ofdefault. The result is consistent with the research of Freedman
and Jin (2014). Liu D et al. (2013) further distinguish different
effects of three types of friendship, i.e., the pipe effect, theprism effect and relational herding effect.
B. Herding behavior
We firstly review the research about herding not confinedto the context of P2P lending. Herding is a phenomenon
usually happens in animal kingdom and the definition ofherding behavior is that individuals are strongly influenced by
the decisions of others (Asch 1956) and many researchers findthe existence of herding behavior in various contexts, e.g.,
investment (Graham 1999), online auctions (Simonsohn andAriely 2008), IT adoption (Kauffman and Li 2003).
The most prevalent reason why we behave like herd is“information cascades” (Bikhchandani et al. 1992), which isbased on the interpretation of conformity preference(Jones
1984). “Information Cascades” means private information is
too weak to resist the information of mainstream. It occurs
especially when individuals can’t obtain the full informat ion,
which is the context of P2P lending (Bikhchandani et al., 1992).
Many researchers study P2P lending by incorporating
herding model into the context of P2P lending. Herd behavior
in P2P lending refers to the tendency of lenders to gravitate
toward and bid on auction listings with existing bids
(Simonsohn and Ariely 2008; Herzenstein M et al. 2011). The
main question can be divided into two dimensions, i.e.,
whether there exists herding behavior and whether it is rational.
1)

The existence of herding in P2P lending

Many researches use data from different P2P lending
platforms and testify that there exists herding behavior. For

examp le, Herzenstein M et al.(2011) used data from Prosper
and conducted an elaborately study. They found that before
loans receive enough bids, a 1% increase in the number o f bids
increases the likelihood of an additional b id by 15% but after
loans receive enough bids and comes to phase of auction, a
1% increase in bids increases the likelihood of an additionalbid by only 5%. This diminishing effect of herding is also
found by Lee E, Lee B(Lee E, Lee B.,2012), who used data
fro m Popfunding and found that as the level of participation
increase, the newer biddings increase tardily.
Apart fro m empirical study based on various P2P lending
platforms, some researchers theoretically testify the existenceof herding in P2P lending. Using decision tree model, Luo B,
Lin Z.(2013) exp lained the formation of herding in P2P
lending and further found emp irical evidence of herd ing byusing data from Prosper.
2)

The rationality of herding

When it comes to the research about rationality of herding,conclusions are varied. Herzenstein M et al.(2011) found thatthough there is adverse effect of herding for buyers in eBay,
herding is advantageous for lenders in P2P lending.
Dramatically, Luo B, Lin Z.(2013) used data fro m the sameP2P lending platform, i.e., Prosper, and believed that herding
will impair benefits of lenders. Other P2P lending platforms arealso concerned as for the rationality of herding. Chen D, Lin
Z.(2014) used PPDai and found that herding behavior lowers
the final interest rate and even worse, increases default rates.
They believed that cultural and economic factors play a role inthe difference of their results from others.
III. LIMIT ATION OF CURRENT RESEARCH AND MODEL BUILDING
We believe that there are some limitations in the current
research which interests us.
A. The expression of extent of herding
To examine the rationality of herding, the expression ofextent of herding is essential. Most researchers use numbers ofbidders or biddings to express the extent of herding, following
the study of online auction websites, e.g., eBay. We don’t
believe it’s appropriate for the context of P2P lending.
First, in online auction websites, as long as there issomeone bidding, the auction continues while in P2P lending
platforms once the total bidding amount reaches the request
amount, the bidding ceases. Such restrict of amount leads to thebias: the number of bidders or biddings can’t precisely denotethe extent of herding and is relevant to the former amount ofbidding. For example, the amount of total bidding has reached99%, comp leted by 90 bidders. There are 100 bidders aspiring
to bid the left because of herding. Due to restrict of amount,
only 1 bidder bid while 99 b idders left can’t. The real numberof bidders denoting the herding is (90+100) wh ile the observed
value is (90+1), 99 bidders are overlooked. There is no bias foronline auction websites because we assume that those 100
bidders bid because of herding and bid whatever the final priceis and they can all bid successfully.
Second, it’s worth discussing the mechanism of end of alisting. In some P2P lending platforms, the listing is closed© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

134

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

once it receives enough bids. We call such platforms type A
platform. In other platforms e.g., Prosper, the listing is not
closed immediately when it receives enough bids. They
provide an auction mode and continue the listing( we call it“auction phase”). Additional bids can lower interest rates,
which benefit borrowers. We call such platforms type B
platform. Apparently those who believe the numbers of bidders
or biddings can represent the extent of herding can only
conduct the study on type B platforms, which impair thegeneralization of their results.
Third, even in type B platforms, the number of bidders or
biddings can’t precisely represent the extent of herding. The
number of bidders or biddings before the auction phase andafter it holds different effect to the representation of herding.
For example, there are 2 listings named listing1 and listing2.
The total numbers of bidders are both 100 but in the listing1
there are 90 b idders bid during auction phase(it means that
before the auction phase 10 bidders bid the request amount)and 10 in the listing2. Apparently the extent of herding inlisting1 is higher than listing2, in which everyone is reluctant tobid more and latently increase the number of bidders.
In sum, the different mechanism between auction andlending means the method used in auction is not quietly
appropriate for lending.
B. The definition of rationality
We believe in the current research the definition ofrationality is ambiguous and not intact. Some researches
simp ly use the variance between final interest rates and initialinterest rates to evaluate the rationality. However, suchvariance is obvious and other factors must be detected. Others
consider the default rates but they overlook that bidding on thelisting that many others already bid can ensure the success of
the listing, which is important because failure o f listing impair
benefit of b idders for that their illiquidity asset is locked in theplatform and lose time value of money. In short, synthetic
factors need to be evaluated.
In this paper, we intend to solve two questions: theexistence of herding behavior and the rationality of it. The
reason for herding in P2P lending can be explained in two
aspects. First, the problem of info rmation asymmet ry is acutein P2P lending co mpared to tradit ional banks and lenders can’t
deduce the credit status of borrowers alone. They use
informat ion of other lenders to form their judge towards th eborrowers. Second, the mechanism of P2P lending causes
herding because when the process of a listing nearly reaches tothe request amount, lending to the listing can assure the listingis successful, which also benefits lenders because their money
can be effective as soon as possible. Thus we have hypothesis
H1.
H1: There exists herding behavior in P2P lending
We believe that herding behavior of lenders in P2P lending
can be reflected in two aspects: the decision of choosing
borrowers and the decision of choosing listings.

When lenders choose borrowers, they not only exert theirown information towards borrowers but also refer to judgmentsof others, which can practically be realized by the function ofviewing history borrowing records of borrowers in PPDai
platform. Successful funding records in the past indicate therecognition of other lenders towards borrowers and a potential
lender might leverage such information and make decision.
Thus we assume that when times of successful funding records
in the past increase, the probability of successful funding
increases, which indicate the herding behavior of lenders.
H1a. When times of successful funding records ofborrowers in the past increase, the probability of successful
funding increases.
When lenders choose listings, they can obtain bidding
information fro m other borrowers in the webpage of the listing.
To elaborately prove the existence of herding, we intend tofollow Lee E, Lee B. (2012)’s research and use one day as atime span. The dependent variable is the ratio of bidding
amount to the request amount in the day for certain listing andthe main independent variable is the ratio of accu mulated
bidding amount to the request amount before the day. Other
control variables, e.g., credit rate, interest rate, duration areconcerned. For examp le, the request amount of listing i is$10000. Before the day j there have been $5000 bided in thelisting and in the day j there are $3000 bided in it. Thus thevalue of the independent variable and the dependent variable is
50% and 30% respectively in the observation. If there exists
herding behavior, the bidding amount in a day increases when
the amount before the day increases. Based on this, we build
hypothesis H1b.
H1b. The bidding amount in a day increases when theamount before the day increases
As aforesaid the conclusion of rationality is not the same indifferent study. Even using the same dataset - Prosper,
Herzenstein M et al.(2011) and Luo B, Lin Z.(2013) draw
different conclusions. Thus the rationality of herding is
debatable and the study of it is necessary. We follow theconclusion of Chen D, Lin Z.(2014) because they use data fromthe same P2P lending platform and we believe that due tocultural and economic factors herding is irrational.
H2: The herding behavior in P2P lending is irrational
Following the examination of existence of herding, weanalyze the rationality of herding in two aspects: herding
behavior towards choosing borrowers and herding behavior
towards choosing listings.
To examine the rationality of herding we need to measure
the extent of herding and examine the relation of default rateand it. Thus we define an herding index which we think isappropriate for representation of the extent of herding. Figure 2
shows the source of the index of certain listing. The y axis
means the ratio of accumu lated bidding amount to the request
amount and the x axis means the ratio of elapsed time fro m© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

135

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)beginning of the listing to the duration of bidding. The path
fro m O to B represents the process the listing ends. Straightline OB is the most common situation in which the speed ofbidding is average while arc OAB represent the herding
because in the beginning of the listing the speed is low butwhen it comes to the end of listing the speed is high. The
herding index is the value of the area of the black part, i.e., thearea of the part surrounded by Straight line OB and arc OA B.
If the extent of herding is acute, the degree of crook of OA B islarge and thus the area of black part is large.
Some researchers use average time interval of b idding or
relative time elapsed (the proportion of time that had elapsed in
the auction when the loan received full funding) (Herzenstein
M et al.,2011), which we believe is inappropriate. The reason
can be illustrated in Figure 2 and Figure 3. Suppose figure 2
shows the bidding process of listing A and figure 3 shows thebidding process of listing B. We suppose the average speed ofbidding is equal between A and B and the only difference isthat the speed is first low and then fast in A while first fast thenlow in B. In the listing B, the bidding speed at first is high but
when it co mes to the end, the speed is low, which means
bidders at first are interested in the listing because the listing
itself is attractive and they are afraid not to bid before it ends.
The speed of subsequent bidders’ biding is lower than normal,
which means they are not as in need as former b idders and theformer bidding is not related to theirs. In other words, they are
not imitating and there is totally no herding behavior. The
average time interval of bidding is equal between listing A andlisting B but apparently the extent of herding is not. Yet theherding index can distinguish the variance between the listing
A and B because the herding index of B, the listing in wh ich
herding totally not exist, is negative.

There may not be matching perfect arc OAB in the realtransaction context. To calculate the herding index, we can use
calculus method, i.e., we divide the black part in figure2 intosome echelons and calculate the sum of the area of echelons.
Figure 3-5 show three examples of bidding process in the realtransaction, indicating low herding, normal herding and highherding respectively. The ids of listing are 404404,404406 and428701. The information of listings can be viewed in website.
URL can be inferred based on id of the listing (e.g., theinformation of listing 404404 can be viewed on
http://www.ppdai.co m/list/404404). After calculation, we findthat the corresponding herding index is -0.2995, -0.0074 and0.3323 respectively.
Fig. 1.T he bidding process of listing A

Fig. 2.T he bidding process of listing B

100%

Such graphic and method of calculation is similar to Gini
coefficient. Gini coefficient measures the extent of inequalityof income, the calculation of which is fro m Loren z curve. In a
Lorenz curve, the x axis denotes the percentage of population
fro m the poorest to the richest and the y axis denotes thepercentage of welfare. Suppose figure 2 represent a Lorenz
curve and straight line OB means absolute equality of income
and arc OAB means actual distribution of income. The Gini
coefficient is the result of area of black part divide 0.5.
We incorporate Gini coefficient and construct a herding
index because we believe the essence of herding is theinequality of bidding speed while Gini coefficient exactly
measures the extent of inequality of d istribution. However
there is misuse concern, i.e., in Loren z curve the x axis denotethe percentage of population that is already ranked from thepoorest to richest and arc OAB will always be under straig htline OB wh ile in herding index arc OAB can surpass straightline OB. We believe that there is some tiny difference between
Gini coefficient and herding index. Gini coefficient focuses on
the inequality of distribution but herding not only means theinequality of bidding speed but also the concept that thebidding speed is slow in the beginning but fast in the end. If the
speed is fast at first but slow in the end herding is not
significant while the inequality of bidding speed is. W hen arc
OAB surpass straight line OB there is punishment for
representation of herding.

B

A

O

100%

Fig. 3.Listing of low herding in the real transaction

Fig. 4.Listing of normal herding in the real transaction© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

136

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

TABLE II.

QUALITATIVE VARIABLES

Type

NumberVariable

listing

102430T ype

Borrower

79709

Credit rate

Fig. 5.Listing of high herding in the real transaction

genderoccupation

IV. DAT A
Data is fro m PPDai, the largest P2P lending platform. We
captured the data from 2014.1.1 to 2014.3.31 and alltransaction records during this time interval are recorded in the
database. The mechanism of URL in PPDai brings out great
convenience because all listing IDs are parts of URL of thelisting. We first find out the range of loan ID because loan ID
is determined according to time order. Then we use python torecord all the web text fro m those pages. To analyze these texts
and derive the data needed, we write program to handle it. The
main data item includes type, title, the percent of progress, loan
amount, interest rate, duration, bidding list, purpose. The web
page of listing does not consist of result of repayment, i.e.,
whether there is default or not, thus we captured the data aboutborrowers concerned. The same procedure is conducted. The
main data item includes age, gender, occupation, credit rate,authentication status. There are 102430 listings and 79709
borrowers incorporated in the dataset. The number of bidding
record is 565539. A mong all listings there are 15906 (15.5%)
listings successfully funded. The variables and simple
descriptive statistic results are shown in Table 1 and Table 2.
TABLE I.
Type
listing

Borrower

QUANTITATIVE VARIABLES

VariableloanAmount(RMB)
Annual interest rate
Duration(month)# of Bidders# of Successful
borrowing#offailedborrowing# of bidding
Weighted lending
rate

Max
400000
24
12
1516
728

Min
1000
8
1
0
0

Average
3956
15.7
9
5
11

STD
9426
3.9
2.8
23
66

31

0

2.45

2.36

40918
25

0
0

23
1.41

473
4.66

Dummy
VariableT ype AT ype BT ype CT ype D
AAA
AA
A
B
C
D
E
F
Male
Femalestudent
wageearners
PrivatebusinessownersOnlineshop
ownersOthersNumber
15643
5229
5564
751
690
13523
57
488
1146
5115
47441
11249
68855
10854
2684
45532
17527

3836
10130

Table 1 shows the quantitative variables while Table 2
shows the qualitative variables. Some informat ion can be
discovered. First, the largest request amount of listing is
400000 RMB, which is a considerable amount but the averageamount is 3956, wh ich means the request amount is not large inmost of listings. Second the highest interest rate is 24%, andthe average of it is 15.7%. The rate in P2P lending is muchhigher than the rate of checking accounts, which is only nearly
0.35% at that time. Such variance of rates attracts many lenders
to P2P lending platforms. Th ird, we discuss the listing type in
PPDai. In PPdai, there are 4 types of listing. Type A means
platforms compensate for lenders when the listing is overdue.
Type B means the money borro wing in the listing can’t be
withdrawn and can only be used within the platform, e.g.,
bidding or repay ment. Type C means the listing that the
payback source is guaranteed, e.g., accounts receivable ormoney saved in the platform. Type D means listing withguarantee. Forth, we can tell that the main borrowers of P2P
lending are male. The reason may be that males have muchburden in life than females and need more money. Last we
notice that more than half borrowers are wage-earners because
the wage they earn may not support their extra need while theoccupation is stable for them and make it credib le for them topay back.
V. RESULT
A. The existence of herding
We analyze the existence of herding by using logistic
model. The main variables are shown in Table 3.
1) The existence of herding when lenders choose borrowers
We create logistic regression model in which dependent
variable is whether listing is successfully funded and the main© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

137

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)independent variable is times of successful borrowing records
in the past. Table 4 shows the result of the model.
TABLE III.
Variable
GABC

MAIN VARIABLES

RAT E
AGE
OCCUPATION
CER

Meaning
If the credit grade of borrower is A,B or C and the value is
1, else 0. We set this variable because borrowers withcredit grade D,E and F is very few.
T he amout of successful borrowing records in the past
T he amout of failed borrowing records in the past
Four types of listing
The log of amount. We use logarithm process because theamount is very large
Interest rate
T he age of borrowers
T he occupation of borrowers
T he certification of borrowers

TABLE IV.

T HE EXISTENCE OF HERDING WHEN LENDERS CHOOSE

SUCCESSFAILURET YPE
LNAMOUNTBORROWERS

Variable

Coefficient Std. Error z -Statistic Prob.

C-18.59984
GABC
5.149296
SUCCESS
0.018199
FAILURE-0.204898
T YPEB
0.756571
T YPEC
0.312124
T YPED
2.765786
LNAMOUNT
1.134480
RAT E
0.395779
AGE
0.001056
OCCUPAT IN_ST UDENT 0.175779
OCCUPATION_WAGE_EA
RNERS
0.112231
OCCUPATION_OSOWNER
S
0.268957
OCCUPATION_PRIVATEO
WNERS
0.307766
CER_IDENTITY
1.071651
CER_VIDEO
0.483560
CER_EDUCATION
0.248571
CER_MOBILE
0.668838
CER_ONLINEBANKING 1.062544

0.628439
0.053042
0.001996
0.004889
0.097528
0.089595
0.108327
0.050280
0.003127
0.000476
0.059845-29.59690 0.0000
97.07980 0.0000
9.115842 0.0000-41.90597 0.0000
7.757447 0.0000
3.483729 0.0005
25.53182 0.0000
22.56341 0.0000
126.5839 0.0000
2.218445 0.0265
2.937235 0.0033

0.033785

3.321912 0.0009

0.057165

4.704937 0.0000

0.038843
0.475227
0.031295
0.036469
0.025835
0.027964

7.923381 0.0000
2.255028 0.0241
15.45163 0.0000
6.815975 0.0000
25.88836 0.0000
37.99642 0.0000

The result shows that the coefficient of successful fundingtimes in the past is significantly positive, which confirms
hypothesis 1a.
Other conclusions can be found. First, when the credit
grade is high (GABC), the probability of successfully fundingis high and the coefficient value is relatively large, wh ichmeans the credit grade is main concern of lenders whenchoosing borrowers. Second, when listing is speciallycertificated, the probability of successfully funding increases.
The coefficient of type D is relatively h igh, because type Dmeans the listing is guaranteed, which is a direct signal of low
risk while other types are indirect for lenders. Third, thecoefficient of rate is positive, which means lenders pursue highprofit. When it co mes to age of borrowers, we find that lenders
prefer old borrowers and one possible exp lanation is that oldborrowers are more credible. The finding is the same asprevious research (Gonzalez and Ko marova Loureiro, 2014).
Last, the private business owners are more likely to get loans
because the coefficient is relatively high. When borrowers areauthenticated by PPDai, the probability of successfully fundingis high especially when identity of borrowers are authenticated.
2) The existence of herding when lenders choose listings
We draw the figure 6 using our dataset in which x axis
means the percent of complement of listing, i.e., the ratio ofaccumulated bidding to the request amount and y axis means
the time interval between two successive bidding, which is amean value of all data fro m our dataset. As shown in the figure,when the current percent of complement of listing is less than
10%, the speed of listing is very low and average time interval
between two successive bidding is nearly 1 hour. When thebidding proceeds and the percent of complement of listing is90%, the speed of bidding is very fast and average time interval
between two successive bidding is nearly 1 minute. The timeinterval between two successive bidding decreases whenaccumulated amount of bidding increases, proving theexistence of herding.
In addition, when current percent is low, the decrease oftime interval is not acute while when current percent is high thedecrease of time interval is inconspicuous, proving themarginal decrease effect of herding. The result is the same as
Herzenstein M et al. (2011).
Fig. 6.Evidence of herding

McFadden R-squared
S.D. dependent var
Akaike info criterion

0.550145
0.492929
0.611309

Schwarz criterion

0.613096

Hannan-Quinn criter.

0.611851

LR statistic
Prob(LR statistic)

75698.10
0.000000

Obs with Dep=0
Obs with Dep=1

59150
42168

Mean dependent var 0.416195
S.E. of regression
0.306796
Sum squared resid9534.632
Log likelihood
30949.31
Restr. log likelihood 68798.36
Avg. log likelihood 0.305467

T otal obs

101318© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

138

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

B. The rationality of herding
We conduct logistic regression model and analyze two
aspects both. The dependent variable is default. The result isshown in Table 5.
1)
Rationality of herding when choosing borrowers
The coefficient of SUCCESS is significantly positive while
FAILURE is significantly negative, which is contrary tocommon sense. We believe one possible exp lanation is thatborrowers with mo re successful borrowing records can obtainloans easily (see Table 4) and they may use such advantage andmake a fortune before leaving the market. Borrowers withmore failed borrowing records should have little chance to getloans successfully but once they get loans, they cherish suchchance and try to refund to improve their credit status. Thus webelieve that the herding behavior of lenders when choosing
borrowers is irrational.
Other conclusions can be found. First, when credit grade islow, borrowers tend to default. Second, the coefficient of
TYPE A and TYPE D is positive. One possible explanation is
as follows. For type A listings, platforms compensate forlenders when the listing is overdue. When borrowers default,
platforms will replace borrowers to compensate, which
alleviate the responsibility and willing to pay back ofborrowers. The exp lanation of type D listings is the same.
Second, when rate and duration is large, the possibility ofdefault is large because borrowers are not willing to pay back.
Third, males are more likely to default. The result is the sameas previous research (Chen D et al., 2013). Forth, wage owners
are more likely to default because they have more liv ing
pressure than other occupations and the average financial
situation is not good.
2)
Rationality of herding when choosing listings
The herding index is not present in Table 5 because the pvalue is very high. To validate such result, we incorporate
independent variable: the number of bidders, which is thesame method of precious research, and the p value is still high.
Thus we believe when lenders choose listings, the rationalityof herding behavior is not certain.
Though herding in the listing is irrelevant to default,
herding in the borrowers is significantly relevant to default.
Thus considering the two aspects of herding behavior both, we
conclude that herding behavior is irrat ional. The result is thesame as Chen D et al. (2014)
TABLE V.

VariableC
GABC
SUCCESSFAILURET YPEAT YPEBT YPEDRAT E
DURAT ION

T HE RATIONALITY OF HERDING

Coefficient Std. Error

t-StatisticProb.

0.302860-0.137964
0.000343-0.010680
0.134429-0.102351
0.051061
0.002478
0.011603

10.31501-9.986190
14.16693-7.033386
11.88790-7.384078
2.445383
2.015790
10.87373

0.00000.00000.00000.00000.00000.00000.01450.04380.00000.029361
0.013815
2.42E-05
0.001519
0.011308
0.013861
0.020881
0.001229
0.001067

MALE
0.039104
OCCUPATIN_ST UDE
NT-0.206123
OCCUPATION_WAG
E_EARNERS
0.043478
CER_EDUCATION -0.080725
CER_MOBILE-0.031996
TOTALLY_PAYBAC
K-0.012228
DEFAULT BUTPAYB
ACK-0.011569
R-squared
Adjusted R-squared
S.E. of regression
Sum squared resid
Log likelihood
F-statistic
Prob(F-statistic)

0.275382
0.274590
0.328665
1482.369-4199.191
347.6843
0.000000

0.007969

4.9071020.00000.018751-10.99244

0.00000.006031
0.007109
0.006513

7.208884-11.35561-4.912334

0.00000.00000.00000.000292-41.91884

0.00000.001515-7.635030

0.0000Mean dependent var
S.D. dependent var
Akaike info criterion
Schwarz criterion
Hannan-Quinn criter.
Durbin-Watson stat0.182037
0.385889
0.613610
0.622376
0.616531
1.977418

VI. CONCLUSION
We study herding behavior in P2P lending in this paper.
We try to solve two problems: whether there exists a herding
behavior and whether it is rational. We div ide herding
behavior into two aspects: herding when lenders choose
borrowers and herding when lenders choose listings. We
assume there exists herding behavior and such herding
behavior is irrational. Data is fro m PPDai, the largest P2P
lending platform in China. Empirical study prove that there
exists herding behavior in P2P lending and when we examine
the rationality of herding, we define herding index to exp ress
the extent of herding. Consistent with previous research, wefind herding in P2P lending in Ch ina is irrat ional. We believe
the paper contribute study of herding in so me aspects: first we
deep discuss the herding and divide it into two aspects, second
we define herd ing index based on Gin i coefficient to exp ress
the extent of herd ing, third we use new dataset from China andenrich the empirically study of herding.
A CKNOWLEDGMENT
This paper is supported by professor Tu giving a lot of
specific instructions and help. This paper is sponsored by
Human ities and Social Sciences of Ministry of Education
Planning Fund “Research on the Decision-making Behavior of
Investors and Borrowers and Risk Control Strategy in P2P
online lending” (Project No. 15YJAZH066), Social Sciences
Association of "Youth Talents of Social Sciences" Fund
"Research on the E-co mmerce Pro motion Strategy for
Industrial Structure Optimizat ion in Beijing" (Pro ject No.
2015SKL009). and scientific research innovation fund for
graduate students of Central University of Finance andEconomics (Pro ject No. 201526). Thank all of them very much.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

REFERENCES

139

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[1]

Akerlof B G A. The Market for "Lemons": Quality Uncertainty and The
Market Mechanism[J]. Quarterly Journal of Economics, 2013,
84(3):488-500.
[2] Asch S E. Studies of independence and conformity: I. A minority of one
against a unanimous majority.[J]. Psychological Monographs, 1956,
70(9):1-70.
[3] Bikhchandani S, Welch I. A theory of Fads, Fashion, Custom andcultural change as informational Cascades[J]. Ivo Welch, 1992,
100(5):992-1026.
[4] Chen D, Lai F, Lin Z. A trust model for online peer-to-peer lending: a
lender’s perspective[J]. Information Technology & Management, 2014,15(4):239-254.
[5] Duarte J, Siegel S, Young L. Trust and Credit: The Role of Appearance
in Peer-to-peer Lending[J]. Review of Financial Studies, 2012,25(8):2455-2484.
[6] Emekter R, T u Y. Evaluating credit risk and loan performance in online
Peer-to-Peer (P2P) lending[J]. Applied Economics, 2015, 47(1):54-70.
[7] Freedman S, Jin G Z. The Signaling Value of Online Social Networks:
Lessons from Peer-to-Peer Lending[J]. Nber Working Papers, 2014.
[8] Iyer R, Khwaja A I, Luttmer E F P, et al. Screening in new credit
markets: Can individual lenders infer borrower creditworthiness in peerto-peer lending?[C]//AFA 2011 Denver Meetings Paper. 2009.
[9] Kauffman R J, Li X, Au Y, et al. Payoff externalities, informationalcascades and managerial incentives: a theoretical framework for IT
adoption herding[C]// 2003.
[10] Komarova Y, Gonzalez L. Competition against common sense: insights
on peer-to-peer lending as a tool to allay financial exclusion[J].
International Journal of Bank Marketing, 2015.
[11] Lee E, Lee B. Herding behavior in online P2P lending: An empiricalinvestigation[J]. Electronic Commerce Research and Applications, 2012,11(5): 495-503.

[12] Li S, Qiu J, Lin Z, et al. Do borrowers make homogeneous decisions in
online P2P lending market? An empirical study of PPDai in China[J].
2011:1-6.
[13] Light J. Is' Peer-to-peer'lending worth the risk[J]. Wall Street Journal,
2012.
[14] Lin M, Prabhala N R, Viswanathan S. Judging borrowers by thecompany they keep: friendship networks and information asymmetry in
online peer-to-peer lending[J]. Management Science, 2013, 59(1): 17-35.
[15] Lin M, Viswanathan S. Home bias in online investments: An empirical
study of an online crowdfunding market[J]. Management Science, 2015.
[16] Liu D, Brass D, Lu Y, et al. Friendships in online peer-to-peer lending:Pipes, prisms, and relational herding[J]. Mis Quarterly, 2015, 39(3):729-742.
[17] Luo B, Lin Z. A decision tree model for herd behavior and empiricalevidence from the online P2P lending market[J]. Information Systems
and e-Business Management, 2013, 11(1):141-160.
[18] Magee J R. Peer-to-peer lending in the United States: surviving afterDodd-Frank[J]. NC Banking Inst., 2011, 15: 139.
[19] Michels J. Do unverifiable disclosures matter? Evidence from peer-topeer lending[J]. The Accounting Review, 2012, 87(4): 1385-1413.
[20] Puro L, T eich J E, Wallenius H, et al. Borrower Decision Aid forpeople-to-people lending[J]. Decision Support Systems, 2010, 49(1):5260.
[21] Simonsohn U, Dan A. When Rational Sellers Face Nonrational Buyers:
Evidence from Herding on eBay.[J]. Management Science, 2008,
54(9):1624-1637.
[22] Stiglitz J E, Weiss A. Credit rationing in markets with imperfect
information[J]. The American economic review, 1981, 71(3): 393-410.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

140

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Toward a Multiagent Reinforcemnet Learning
Approach for Cooperative Game Playing with Users
on a Sugoroku-like Board Game
Kihaya Sugiura

Naoki Fukuta

Graduate School of Integrated Science and Technology
Shizuoka University
Johoku, Naka-ku, Hamamatsu-shi, Shizuoka, Japan
Email: gs16021@s.inf.shizuoka.ac.jp

Graduate School of Integrated Science and Technology
Shizuoka University
Johoku, Naka-ku, Hamamatsu-shi, Shizuoka, Japan
Email: fukuta@inf.shizuoka.ac.jp

Abstract—In this paper, we show an analysis on a reinforcement learning based approaches to allow a behavior that iscooperative and also competitive for the human players in oursugoroku-like board game playing environment. We model thegame as a multiagent environment and we develop the leaning
agents which learnes their behaviors by using the seven reward
functions to cooperatively play the game with other players. We
conduct a set of experiments to examine the behaviors on playing
the game with imitated human player agents on various levels ofskills.

I.

I NTRODUCTION

In recent years, entertainment in computer games is coming
to be more diverse, and the relationship between human and the
AI is much more complicated because of the development of
the AI that intelligently plays a computer game. Game playing
AI is not only as strong as humans (e.g., AlphaGo[1]) but also
capable of playing very complex action games[2] instead ofhuman players. However, to make a game enjoyable for diverselevel of users, realizing a cooperative behavior of (computer)
players to entertain the human players to give them a feel ofcompeting to other (computer) players as well as overcoming
them at the end of the playing. We argue that, realizing apeaceful playing of a competitive game by multiple players is
an important goal to make the platform of gaming viable andlong living. It could be viewed as a similar one on discussing
a shared issue with multiple users[3][4] without breaking their
human relations. Furthermore, even on a bilateral negotiation
among just two entities is very complex to be analyzed[5], thecomplexity of three or more entities for a repeated game could
be far difficult to be analyzed theoretically.
In this paper, we show an analysis on our reinforcementbased learning approaches to allow a behavior that is cooperative and also competitive for the human players in oursugoroku-like board game playing environment. We model thegame as a multiagent environment and we develop the learning
agents which learn their behaviors by using the seven reward
functions to cooperatively play the game with other players.
In the approach, each player agent is initially constructed withdifferent types of reward functions and formed its learning
process on a single-agent reinforcement learning with randomly behaving agents. To evaluate our approach, we prepare
different skill-levels of imitated human player agents in thegame environment and also we prepared several kinds of thelearning agents based on the different reward functions for thereinforcement learning. At the same time, we examine whether
the learning agents can induce the entertainable (cooperativeand competitive) situation for the user player when multiple
agents are set to play with the imitated human player. The
purpose of the work is to realize a cooperative behavior ofagents for the user players in our multiplayer simulation gameenvironment. We use some techniques used in the multiagent
systems[6] and agent learning techniques[7] to realize cooperative player agents on our simulation game environment todesign an user entertainable and competitive playing in thegame play.
II.

BACKGROUND

A. Difficulties on Analyzing Repeated Multiplayer Games
Board game is usually played by multiple players andprofits of those players sometimes conflict each other in the
result of the decision of players. In a simple prisoner’s dilemma
model that two players to select the cooperation or noncooperation, it is known that there is a dominant strategy.
A dominant strategy does not depend on the decision of theother player. Therefore, the players can make their decisions
whether they know the other player’s decision or not. N
repeated prisoner’s dilemma model is a prisoner’s dilemma
model that has multiple players and players make decisions
over times, the process of learning appropriate behaviors canbe seen as an evolutional model[8]. Although the N repeatedprisoner’s dilemma model is similar to the real world model, ona real world, players can not repeat making decisions infinitely
and the repeats can be finite times by the reason of time
limitation. A gaming on a real life has a time limitation and atermination condition to finish playing the game. They makethe analysis of multi player limited repeated games difficult
without experimental analysis.
B. Railway Management Simulation GameAs a platform to analyze multi player repeated games, we
designed a game with a motif of railway management. Figure
1 shows the model of railway management game in this paper.
This game can be played on a Web Browser. Players go aroundthe stations by throwing the dice when their turn comes. An© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

141

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

important rule in this game is that a player can aggressively
attack other players after stepping into a yellow station insteadof throwing a dice. On this game, obtaining a profit for a player
does not always mean the loss of other players and therefore
this game is not zero-sum game.
This game has a map as a non-directed graph that may
have branches, dead end nodes, join nodes and loops. Since themap of this game may include a loop structure with branches,players throw the dice to move and then the the players haveto decide the directions to move.

cooperative behaviors. One is the situation that several agents
share their purpose or task and they try to communicate aeachother to share their intensions or the obtained information toachieve the goal. In particular, approaches based on the gametheory[9] is one of the possible way. The other situation isthat each agent has its own goals with a shared mechanismthat naturally enforce the agents to cooperate.[6]. Since there
are noises on the agent’s learning environment because of theagents’ exploration behaviors, noise reduction method is often
necessary for such multiagent learning environment[7].

Following are the details of our game.
Player
A player is allowed to choose one action fromtwo basic actions: throwing a dice to go aroundthe station, and aggressively attacking to anotherplayer when a player is in a yellow station. The
aim of each player is to maximize its owned
money at the end of the turn of the game.
Station
There are ”blue”, ”red” and ”yellow” stations in
the game. When a player stopped at a blue station,
the player gets the specified amount of money.
When a player stopped at a red station, the player
loses the specified amount of money. When aplayer stopped at a yellow station, the player getsa chance to aggressively attack another player.
This attack can be done at once in any turn after
stopped at a yellow station.
AttackIII.

I MPLEMENTATION

We have implemented our prototype platform to makethe game playable on a web browser as well as making
some experiments with the learned agents. Figure 2 shows anexample of the game platform with a simple map and log view,
and Figure 3 shows another example with more complicatedmaps with a playable user interface. Figure 4 shows the structure of the implemented platform. Here, we used Node.js[10]and WebSocket protocol to implement communications among
the Web browsers and our platform. We implemented the
WebSocket game server that runs on the Node.js to storeand hold the game state object. We also implemented thelearning player agents as WebSocket clients to communicate
with the game server to play the game on behalf of humanplayers. Because of the implementation on the WebSocket, ourplatform does not deeply depend on its running environment.
Unlike ordinary WebSocket communications, since the server
can intentionally push some data to the clients, it is easy toreflect the game state to all connected clients.

If a player has a right to attack another player, theplayer can select to attack a specified player instead of throwing a dice. In such a case, appointed
player loses the specified amount of money.


 




  
Fig. 1.

Basic model of the railway management simulation game

C. Applying Multiagent Learning
Wicke[6] suggested that a reinforcement learning-based
cooperative behavior could be obtained with a very simple reward function without having complex communications
among agents. On an environment that has multiple learning
agents, there are two different situations for acquiring theirFig. 2.

The railway management game (five stations)

A player agent in this paper is implemented as a client
program on the client-server model. The player agents connect
to the game server and send their decisions to the server when
their own turn come. Player agents have attributes that express
the states of the game. For example, a player agent can selectto throw the dice or to attack other players when its attack flagis on. If the player agent selected to throw the dice, the agent
also calculates where the agent are able to go depending on
the points of the dice. After calculating where they can go, theagent makes a decision by using ϵ-Greedy selection.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

142

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 3. The railway management game (with complex map and playableinterface)

Following are some important keywords of the game.
Position
Position of the agent denotes the current station
the player agent stopped at.
Money
Stopping at a blue station and getting more moneyis the most promised way to win the game. At
the final turn, comparing the mount of moneythat each agent has and the player that has largest
amount of money is the winner of this game.
Attack Flag
When a player stopped at a yellow station, theattack flag is on. When the attack flag of the agent
is on, the player can attack to other players andthis makes decrease of the money of the attacked
player.
IV.

D ESIGN OF R EWARD F UNCTIONSIn this section, we focused on the acquisition of rightbehaviors on a Q-Learning agent[11] on the experimental gameenvironment.
TABLE I.
num of blue stations
2

E XPERIMENTAL GAME CONDITION

num of red stations
2

num of yellow stations
1

num of players
3

A leaning agent receives the ”reward” from the gameenvironment on the learning steps. We design some different
rewards for learning agents to acquire their behaviors.
Following are the details of the prepared reward functions.
functionA
This reward function refers to the actions that thelearning agent takes. Going to a blue station ora yellow station and then aggressively attacking

to other players will be positive value rewards.
Going to a red station will be a negative value
reward.
functionB
This reward function refers to the game rank
after the player agent’s action. This gives its max
reward for being at the rank 1 and gives its
minimum reward for being at rank 3 when there
are three players.
functionC
This reward function refers to the particular
player’s money after each action of the learning
agent. If another player’s money is equal to thelearning agent’s money, it will be a positive reward, otherwise it will be a negative reward.
functionD
This reward function gives a negative reward
for the learning agent’s aggressive attack after
stopping at yellow station. Otherwise this will give
positive value reward.
functionE
This reward function refers to the difference ofthe money between agents. If difference of themoney between the learning agent and the closest
agent and the value is less than or equals to 2,
this will give a positive value reward. Otherwise
it will give a negative value reward.
functionF
This reward function refers to the change of therank in the game. If the rank of a learning agent
moves to upward after the learning agent’s action,
it will give a positive reward. If the rank of alearning agent has been down after the learning
agent’s action, it will give a negative reward.
functionG
This reward function gives a negative reward
when selecting an aggressive attack to other players. If the learning agent selects an action of
attacking to other players, it will give a negativevalue reward. Otherwise it will give a positive
reward.
Table I shows the experimental conditions to acquire thebasic behaviors of learning agents. To make the experiment
simple, we set the number of players to three. In this phase, we
prepared an agent as a Q-Learning agent and others are random
agents that select their action at random. We put five stationsin the experimental game environment. We set the maximum
turns in one game to 99 turns and conducted 7000 games. As
we mentioned above, when a player stopped at a blue or a
red station, we increase or decrease the player’s money with
1, respectively. If a player decides to aggressively attack otherplayers, we decrease the selected player’s money with 2. Whena learning agent’s turn comes, if it is possible to select anaction from throwing the dice or aggressively attacking to otherplayers, the action will be chosen by using ϵ-Greedy (ϵ=0.1). If
the learning agent decides to throw the dice, the learning agent
lists up where the agent is able to go depending on the points
of dice throw and selects the direction to go. If the learning
agent decides to aggressively attack other players, the learning
agent has to select the player to attack. Table II shows theperformance of learning agents with the shown seven reward© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

143

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)"#$%&&'    "    !!" !

#$%&'(  #  
# 
Fig. 4.

The architecture of implemented platformfunctions when playing with two random players.
V.

We examined the possible combinations of those agents inthe experiments.

E VALUATING C OOPERATIVE B EHAVIORS

A. Experimental Conditions

B. Result

In this section, we discuss what kind of player agents cancooperate with human player. Based on the experiment on
the acquisition of basic behaviors, the agents that are using
evaluation function A or evaluation function B performed with
better average ranks on playing the game. Here, we prepared
the basis to implement an imitated human player by using
the agent’s stochastic action selection methods for evaluating
cooperative behaviors of learning agents with actual humanplayers who may have different skills to play the game. We
have chosen that as the learning agent learned by the evaluation
functionB (reward refers to the rank) and the agent chooses
actions based on the learned Q table. This agent is appeared
to be able to act like keeping high game rank in the game. At
the time that the imitated human players plays the game, it has
a mechanism to imitate human player’s un-optimal behaviors
in the game environment. To realize the behavior, we applieda mechanism that adjusting the ϵ value in that condition.
This shows that, the presented approach could not sufficiently
work when the human player is too strong for those learned
agents. This is one of the limitations of the presented work. By
adjusting the agent’s ϵ values used in the process of choosing
probabilistic actions, the agent could imitate different levelsof the human players. The imitated human player does notalways make decisions that will make benefits for the agent
and such random behaviors have been chosen in probabilityof ϵ. We conducted the game playing experiment using threeplayer agents. Instead of using a human player, we prepare theimitated human player agent as we mentioned above. For theϵ value for the variety of imitated human player agents, we
set 0.1, 0.3 and 0.5 for ϵ. For the opponent players of imitateduser player, we use the best agents, learned with functionA,functionB and functionE, which have been performed in the
preliminary experiments with the seven different evaluation
functions that we mentioned above.

Figure 5, Figure 6, and Figure 7 show the results of theexperiment at game playing that were conducted by threeplayers. In those experimental playing records, one imitatedhuman player and two other learning agents have been used.
Here, an important observation is that, in most cases (excluding
e=0.1), having different kind of player agents with a imitatedhuman player can be seen that they cooperatively acting toentertain the user player. In other cases, the human player has
been the leader of the game on their early stages or a computer
player leaded the game but the imitated human player had little
chance to win. In all cases, the player agents did not have any
pre-learning for such cooperative behaviors. This suggests thathaving a good combination of different kind of players will
make the agents cooperative in some conditions. However, at
this moment, there is still a question whether this could beviewed as a factor of collective intelligence which can be seen
on a human-group[12]. The work could be extended to havean ability to find a better combination of computer players fora specific user to entertain on the game playing.
VI.

D ISCUSSION

In the case that the imitated human player with ϵ=0.1
played with the combination of the agents that is learned by
two different evaluation functions, the imitated human player
begins to dominate and get the advantage of money among thethree players at the turn six. In the case the imitated humanplayer with ϵ=0.3 or ϵ=0.5, player’s moneys are not so different
among three players from the beginning of the game and it
continues until the middle to later turns of the game. Finally,
the imitated user player wins. in that condition. This showsthat, the presented approach could not sufficiently work when
the human player is too strong for those learned agents. Thisis one of the limitations of the presented work.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

144

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)
TABLE II.
function
A
B
C
D
E
F
G

P ERCENTAGE OF THE ACTIONS OF EACH EVALUATION FUNCTIONS
blue station
48.2%
49.3%
50.5%
24.8%
39.9%
44.84%
25.68%

red station
7.25%
3.9%
43.0%
46.5%
42.1%
22.60%
47.11%

yellow station
43.8%
24.0%
6.1%
27.9 %
14.0 %
20.89%
26.56%

attack
0.69%
22.6%
0.4%
0.6%
3.8%
11.67%
0.65%

average rank
1.1
1.0
1.9
2.6
1.9
1.29
2.68

&!"

!"#$%&"'&()$#*+&
&

%#"
%!"

,-!.&"$#"/012314501"06"78947:;"$!"73687731<"=0"=>8":3?8781/8"#"!"$" '" (" $!" $&" $)" $*" %%" %#" %+" &$",-.#+

&!"!"#$%&"'&()$#*+

%#"
%!"

)*!+#"$#",-./0.12-."-3"4561478"$!"
4535440.9":-":;5"70<545.,5"#"!"$" &" #" '" ("$$"$&"$#"$'"$("%$"%&"%#"%'"%("&$"&&",-.#+

Fig. 5.

Each agent’s transition of money in each turns with imitated human player (ϵ=0.3:upper,ϵ=0.5:lower), functionA and functionE (one game)VII.

C ONCLUSION

In this paper, we showed an analysis on a reinforcementbased learning approaches to allow a behavior that is cooperative and also competitive for the human players in oursugoroku-like board game playing environment. We prepareda simplified sugoroku-like board game design which refers thecompetitive railway management simulation game that is amulti player board game played on a web browser. In theapproach, each player agent was initially constructed withdifferent types of reward functions and formed its learning process on a single-agent reinforcement learning with randomly
behaving agents. We prepared different skill-levels of imitatedhuman player agents in the game environment and also we
prepared several kinds of the learning agents based on the
different reward functions for the reinforcement learning. We
examined whether the heterogeneity of learning agents can
induce the entertainable (cooperative and competitive) situation
for the user player when multiple agents are set to play theuser. As a result, when the human player is on the middle tolower skills at the playing of the game, those heterogeneously
constructed team of agents well entertained the playing of thegame for the imitated human players. Further discussions could
be done on how this kind of good mixture of diverse types ofplayers could be done for the case that has two or more human© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

145

&!"

&!"

%#"

%#"

%!"
)*!+&"$#",-.-,"/0"/1-"234-,-56-"$!",-.-,"/0"/1-"234-,-56-"!"#$%&"'&()$#*+

!"#$%&"'&()$#*+

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)#"$" &" #" '" ("$$"$&"$#"$'"$("%$"%&"%#"%'"%("&$"&&",-.#+

,-.#+&

&!"

&!"

%#"

%#"

%!"
)*!+&"$#",-.-,"/0"/1-"234-,-56-"$!",-.-,"/0"/1-"234-,-56-"!"#$%&"'&()$#*+

!"#$%&"'&()$#*+

Each agent’s transition of money in each turns with imitated human player (ϵ=0.3:left,ϵ=0.5:right), functionE and functionE (one game)

[4]

[5]

[6]

)*!+#"$#",-.-,"/0"/1-"234-,-56-"$!",-.-,"/0"/1-"234-,-56-"!"$" &" #" '" ("$$"$&"$#"$'"$("%$"%&"%#"%'"%("&$"&&"$" &" #" '" ("$$"$&"$#"$'"$("%$"%&"%#"%'"%("&$"&&",-.#+

,-.#+&

Each agent’s transition of money in each turns with imitated human player(ϵ=0.3:left,ϵ=0.5:right), functionA and functionA (one game)

R EFERENCES

[3]

%!"

#"

players.

[2],-.-,"/0"/1-"234-,-56-"

!"

!"

[1]

,-.-,"/0"/1-"234-,-56-"$!"$" &" #" '" ("$$"$&"$#"$'"$("%$"%&"%#"%'"%("&$"&&"#"

Fig. 7.

)*!+#"

$#"

#"

!"

Fig. 6.

%!"

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski,
S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran,
D. Wierstra, S. Legg, and D. Hassabis, “Human-level control throughdeep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533,
02 2015. [Online]. Available: http://dx.doi.org/10.1038/nature14236
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou,
D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement
learning,” in NIPS Deep Learning Workshop, 2013.
T. Ito, Y. Imi, M. Sato, T. Ito, and E. Hideshima, “Incentive mechanism for managing large-scale internet-based discussions on collagree,”Collective Intelligence 2016, 2015.
A. Sengoku, T. Ito, K. Takahashi, S. Shiramatsu, T. Ito, E. Hideshima,
and K. Fujita, “Discussion tree for managing large-scale internet-based
discussions,” June 1-3 2016.
T. Baarslag, A. Reyhan, K. V. Hindriks, K. Fuijita, T. Ito, and M. J.
Catholijn, “Automated negotiating agents competition 2010-2015,” AI
Magazine, 2016.
D. Wicke, D. Freelan, and S. Luke, “Bounty hunters and multiagent
task allocation,” in Proceedings of the 2015 International Conference
on Autonomous Agents and Multiagent Systems, ser. AAMAS ’15.
Richland, SC: International Foundation for Autonomous Agents

[7]

[8]

[9]

[10]
[11]
[12]

and Multiagent Systems, 2015, pp. 387–394. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2772879.2772930
M. K. Colby, S. Kharaghani, C. HolmesParker, and K. Tumer,“Counterfactual exploration for improving multiagent learning,”
in Proceedings of the 2015 International Conference on
Autonomous Agents and Multiagent Systems, ser. AAMAS ’15.
Richland, SC: International Foundation for Autonomous Agentsand Multiagent Systems, 2015, pp. 171–179. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2772879.2772904
R. Suzuki and T. Arita, “Interactions between learning andevolution:: The outstanding strategy generated by the baldwin effect,”
Biosystems, vol. 77, no. 13, pp. 57 – 71, 2004. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0303264704000565
Y. Hu, Y. Gao, and B. An, “Learning in multi-agent systems
with sparse interactions by knowledge transfer and game
abstraction,” in Proceedings of the 2015 International Conference
on Autonomous Agents and Multiagent Systems, ser. AAMAS ’15.
Richland, SC: International Foundation for Autonomous Agentsand Multiagent Systems, 2015, pp. 753–761. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2772879.2773250“Node.js,” https://nodejs.org/en/.
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning,
1st ed. Cambridge, MA, USA: MIT Press, 1998.
A. Woolley, C. Chabris, S. Pentland, N. Hashmi, and T. W. Malone,“Evidence for a collective intelligence factor in the performance ofhuman groups,” Science, October 2010.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

146

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Towards Age-friendly E-commerce ThroughCrowd-improved Speech Recognition, Multimodal
Search, and Personalized Speech Feedback
Lei Meng∗ , Quy Hy Nguyen∗ , Xiaohai Tian∗ , Zhiqi Shen∗ , Eng Siong Chng† ,Frank Yunqing Guan∗ , Chunyan Miao∗† , and Cyril Leung∗‡∗ Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY),

Nanyang Technological University, Singapore† School of Computer Science and Engineering, Nanyang Technological University, Singapore‡ Department of Electrical and Computer Engineering, the University of British Columbia, Vancouver, BC, Canada

Email: {lmeng, Nguyen.QH, XHTian, ZQShen, aseschng, yunqing.guan, ASCYMiao, CLeung}@ntu.edu.sg

Abstract—This paper presents an age-friendly system for
improving the elderly’s online shopping experience. Different
from most related studies focusing on website design and content
organization, we propose to integrate three assistive techniques
to facilitate the elderly’s browsing of products in E-commerce
platforms, including the crowd-improved speech recognition,
the multimodal search, and the personalized speech feedback.
The first two techniques, namely, the crowd-improved speech
recognition and the multimodal search, work together to allow
the elderly search for desired products flexibly using either
speech, an image, text, or any combination of them whichever
are convenient for the elderly. The personalized speech feedback
provides a speech summary of search result in a personalizedvoice. That is, the elderly are allowed to choose or even create
their desired voices, and also can customize the voices in termsof pitch, speaking speed, and loudness. As a whole, the proposedsystem is expected to help and engage the elderly’s E-commerce
adoption. Testing on real-world E-commerce product datasets
demonstrated the usability of the proposed system.
Index Terms—Age-friendly E-commerce, enhanced user browsing, Crowd-improved speech recognition, Multimodal search,
Personalized speech feedback

I. I NTRODUCTION
In the past one decade, a dramatic growth in online retailingand electronic commerce (E-commerce) has been observed
across the world. E-commerce, defined as the purchase andsale of information, products, and services using any one of thethousands of computer networks that make up the Internet [1],has brought about a revolution in business practices. ThroughE-commerce, many businesses have been able to increase
profitability, gain market share, and improve customer service.
Older people have a great deal to gain from E-commerce,
leading to a huge market and business potential with Ecommerce for the elderly. According to the World Population
aging 2013 report [2], the population aging is taking placein nearly all countries in the world. Globally, the number ofolder persons aged 60 years and over is expected to morethan double by 2050, from 841 million in 2013 to more than
2 billion in 2050. Beyond using the Internet to communicate
with family and friends, acquire information, and evaluate

services, the senior population has an increasing trend to adopt
E-commerce transactions, utilize online access to financial
services such as banks and brokerage firms, obtain travel
information, and research health-related services.
There are, however, open challenges and issues for theelderly people exploring the E-commerce websites. Some
research [3] found that the elderly users aged over 65 years old
were over 40% slower than those of the younger generationin using the internet, and were more likely to give up theirtrials. One explanation is the decline of physical and cognitive
functions of the elderly [4]; while another reason is thatthose aged people are not computer technologies savvy incomparison with the young generation. As such, developing
age-friendly user interface and functions to improve theusability of an E-commerce website for the elder users canbe a predictor of success for modern E-commerce [5].
Existing research efforts for the age-friendly E-commerce
mainly focus on the discovery of reasons for age-related
digital divide [6] and various design principles for agefriendly websites to facilitate the elderly users browsing the
webpages [7], [8], [9]. The challenges and opportunities with
E-commerce for the elderly have also attracted the attention
from industry. China’s Baidu launched a portal that caters to
the elderly [10]. Besides, Amazon has launched a specificonline store targeting at elderly users and their caregivers,
namely, the Amazons 50+ Active & Healthy Living1 . However,these efforts typically try to improve the elderly’s browsing
experience on the Web in terms of simplifying the webpages
to facilitate their understanding and gathering and organizing
information that are expected to be interesting to them; while
limited attention has been given to assistive techniques that
help the elderly during online shopping, such as flexible ways
for the elderly to search and navigate products, convenient
approaches to facilitate the elderly finding desired products,
and personalized intelligent assistants that provide online help
and suggest products of the elderly’s interests.
1 http://www.amazon.com/b?node=5856180011© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

147

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

To address the aforementioned problems, we propose
an age-friendly system for improving the elderly’s online
shopping experience, empowered with multimodal searchand personalized speech feedback capabilities. Specifically,
this system includes three functional modules, namely, thecrowd-improved speech recognition, the multimodal productsearch, and the personalized speech feedback. The proposed
crowd-improved speech recognition engine (CISR) is a twostep speech recognition model leveraging the Microsoft
Speech Platform and the Bing Voice Recognition API, whichenables accurate speech recognition in real-time through apersonalized and crowd-improved product list; the multimodal
product search engine (MM-PSE) is built upon a newly
proposed multimodal search algorithm, namely, the online
multimodal co-indexing adaptive resonance theory (OMCART) [11], which works together with CISR to allow theelderly to search for desired products flexibly using either
speech, an image, text, or any combination of them; and thepersonalized speech feedback (PSF) model makes use of voice
conversion techniques [12] to present the summary of search
results to the elderly with personalized voices. The elderlyis also allowed to create new personalized voices or choose
the voice from a listed of predefined voices and customize
the voices in terms of pitch, speaking speed, and loudness.
Prototype systems for these functional modules have beendeveloped and tested with real-world E-commerce transaction
data.
The reminder of paper is organized as follows. Section II
reviews related work for age-friendly E-commerce. Section
III formulates the studied problem. Technical details andimplementation of the proposed system are presented in
Sections IV and V, respectively. The last section summarizes
and highlights future directions.

Another line of research lies on theories and techniques that
help web browsing of the elderly, which can be categorized
into three key directions. Studies of the first direction aim to
explore principles for web page design [17], [7], [8], [9], [21].
In [17], Thirty-six websites designed for older adults were
evaluated as to how well they complied to 25 “senior-friendly”
guidelines recommended by the National Institute of Aging2 .
“Non Browser” [7] is proposed as a new web design, in whichfunctionality is both restricted and presented differently. The
book [9] introduces general design principles of applications
for the elderly in terms of easy-to-use interface and easy-tounderstand functions. A web search and navigation system
for the elderly [8] is presented with clean web page designand clear guidance, which includes three layers of information
content, a reliable and hierarchical catalogue structure withpredictable results, and a layering of functionality. A recent
study [21] evaluates the usability of E-commerce website forthe elderly and highlights the importance of clear and smart
navigation design for senior customers. The second direction
for age-friendly web browsing [22], [23] explores software
enhancements to Web browser to give users access to control
their browsing environment. The last direction focuses on the
exploration of novel assistive functions. Google investigatedand launched the function of voice search [24], [25], [26] anddeveloped the search engine enabled visual search “Google
images”3 . The study [27] presents a user study on using voice
demands for the elderly when browsing websites. Assistive
virtual agent is proposed in [28] for ecommerce applications
to enhance accessibility for older users, in which the elderly
consumers’ preferences for the characteristics of the virtual
agents are evaluated, in terms of modality, interaction style,animation, realism, embodiment, and gender.

II. R ELATED W ORKIII. P ROBLEM S TATEMENTWith a target of improving the elderly’s online shopping
experience, the study of age-friendly E-commerce is related
to those on theories and technologies that aim to facilitate theelderly to use E-commerce applications, navigate E-commerce
websites, and conduct E-commerce related activities.
A large body of related studies is on Internet adoption by
the elderly with a focus on factors resulting in age-related
digital divide [13], [14], [6], wherein E-commerce is often
treated as a case study. Through user study based approaches,
observations and factors ranging from physical behavior toliving styles are explored [15], [16], [17], [14], [18], [19],[20], [6]. For example, an early study [15] explores theeffect of web page design for the elderly by a user study
on two types of websites, in terms of fonts, multimedia
content, and diversity of inputs. An eye-tracking based study[16] explores how well the elderly use a search engine. The
studies [14], [18] point out that usefulness, ease-of-use, andInternet safety are key indicators for use intention; resourcesavings and self-actualization as motivators determine seniorcitizens’ usefulness perception, whereas computer anxiety andcomputing support are antecedents of ease-of-use.

In view that most related studies focus on theories anddesign principles of websites for improving the elderly’s Ecommerce/web utilization and navigation, while few attempts
have been done in terms of innovative assistive techniques,
this study aims to develop assistive technologies to enhancethe elderly’s browsing experience in E-commerce platforms.
Specifically, we propose an age-friendly system with flexible
multimodal search and personalized speech feedback forenhanced browsing experience for the elderly. With theelderly’s query in terms of photos, text, speech, and arbitrary
combination of them, the proposed system is able to retrieve
all similar products in the E-commerce platform at a highaccuracy, and a personalized speech feedback is subsequently
generated to summarize the search result for the elderly. To fit
the speech to different requirements of the elderly, the elderly
could be able to tune the styles of the speech, such as speaking
speed and volume, and even to create their desired voices.
2 https://www.nia.nih.gov/
3 https://www.google.com.sg/imghp?hl=en&ei=DIn VoOfAo27uASPw7 4D
A&ved=0EKouCAIoAQ© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

148

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Multimodal input
By photo

By text

User voice

By voice

T-shirtMan
White
CottonUser
Interface

Crowd-improved
speech recognition

Visual & textual features

User options tochoose/create voice

Personalized feedback generation

Featuresfromrecognizedtext

Search resultsummarization

Personalized voice
feedback

Multimodal
product search

Product list

Fig. 1: Architecture of the proposed system for age-friendly E-commerce.

IV. E NHANCED U SER B ROWSING FOR AGE - FRIENDLY
E- COMMERCE

Web Browser

B. Crowd-improved Speech Recognition
The crowd-improved speech recognition (CISR) system enables the voice search function of the system by recognizing
users’ voice queries for product search and then feeding therecognized text to the search engine. As illustrated in Figure 2,
CISR employs a two-stage model for speech recognition:•

Stage 1: with users’ voice queries captured by Web
Audio [29] capability of modern web browsers, Microsoft
Speech Platform [30] is used to recognize the speech inreal-time by a specific grammar built from a personalized
and crowd-improved product list.

Product list

Microsoft
Speech Platform

A. Overview
Looking through E-commerce websites to search for desired
products is a core part in E-commerce; while typing for
keyword-based query and the dramatic number of products interms of brands and categories often bring difficulties for theelderly in browsing products on E-commerce platforms [21].
Towards age-friendly E-commerce, we propose a system withflexible multimodal search and personalized speech feedback
to improve the browsing experience of the elderly in Ecommerce platforms. As shown in Figure 1, this system
includes mainly three functional modules, namely, the crowdimproved speech recognition, the multimodal product search,and the personalized speech feedback. The integrated systemenables the elderly to flexibly search for products using
a combination of photos, text, and voice as inputs, andemploys a personalized speech feedback function that provides
a summary of the product search result, such as the numberof products and the number and key information of different
types of products involved, in a personalized voice, which canbe tuned or created by the elderly. In the following sections,
technical details of the three modules are presented.

ServerSpeech Query
Recognized?

Recognized
text

No

Text Query

Search engine

Yes

Bing service

Fig. 2: Architecture of crowd-improved Speech Recognition•

Stage 2: if the recognition from the first stage yields lowconfidence score, i.e. the voice query does not match any
grammar in stage 1, Bing Voice Recognition API [31],as a fallback mechanism, is activated to perform speechto-text recognition.

The Microsoft Speech SDK is chosen from other speech
recognition platforms, including Bing Voice Recognition API,
Google Speech API, Nuance, and Wit.ai, due to its extensibility to incorporate external grammars and its convenience
for implementation on server; while Bing Voice Recognition
API is employed mainly because of its reasonable performanceand engineering flexibility as a general purpose recognitionengine, which can be easily substituted by many others forfurther development when necessary, such as Google Speech
API and Nuance etc.
It is worth mentioning that the grammar-based speech
recognition in Stage 1 provides a real-time process of users’
voice inputs, and may accurately recognize keywords, suchas product names, from users’ speech with the product list.
To optimize the recognition performance gain, the product list
could be constructed in two manners:•

Personalized: the product list is constructed using current© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

149

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)users’ purchase and browsing history.
Crowd-sourced: the product list is constructed using therecent most popular products in an E-commerce platform.
The personalized product list enables the proposed speech
recognition model to react fast and accurately for different
users’ speech by learning from individual user’s browsing
behavior; while the crowd-sourced product list takes advantage
of collective intelligence to make the proposed model contextaware when applied to different E-commerce platforms.

Product Search•Multimodal input
By photo
User
Interface

Product Indexing

By text
T-shirtMan
White
CottonFeatureVectors

Ranking
Algorithm(OMC-ART)dkeran
top ctsof rodu
DIp

Prod
Inform uct
ation

Product
Database

C. Multimodal Product Search
The multimodal product search engine (MM-PSE) aims at
providing users flexible ways to search for their desired
products in the E-commerce platform. MM-PSE is built upon a
newly proposed algorithm for the multimodal co-indexing andretrieval of weakly labeled web images, namely, the online
multimodal co-indexing adaptive resonance theory (OMCART) [11]. As shown in Figure 3, MM-PSE includes a productsearch module for processing user queries and retrieving
relevant products for the user, and a product indexing module
for product index base creation and update. We observe thatwith the indexing and ranking algorithms of OMC-ART,
MM-PSE enables users to search for products flexibly using
multimodal queries, such as an image, several keywords,
or a combination of both, by creating a multimodal index
base of E-commerce products using product photos and theassociative textual description. In the following two parts, theproduct indexing and search modules are briefed, of which thetechnical details can be referred in [11].
1) Product Indexing: The utilized co-indexing algorithmof OMC-ART for building the product index base is basedon an heterogenous data co-clustering algorithm, i.e. thegeneralized heterogeneous fusion adaptive resonance theory(GHF-ART) [32]. Given a raw product dataset with productphotos and textual description, without additional resources,
this algorithm automatically extracts the visual and textual
features of products and builds a two-layer index hierarchy, of
which the bottom layer contains clusters of similar products
while the upper layer contains generalized characteristics ofproducts in the same cluster, represented by cluster weights.
Thus, products are co-indexed by the key visual and textual
features in this hierarchy, which serves as the product index
base and enables image- and/or text-based search. As statedin [11], the proposed co-indexing algorithm is much fasterthan related state-of-the-arts with light parameter tuning of aratio value. Besides, the discovered common characteristics
of products in the upper layer serves as the basis for the
fast and accurate ranking algorithm for product search. More
importantly, the built product index base can be online
updated, which meets the requirements of E-commerce tohandle live big data streams.
2) Product Search: The ranking algorithm of OMC-ART
takes advantages of the two-layer index hierarchy generated
by the co-indexing algorithm for product ranking. Given aquery in terms of a photo, textual description, or a combinationof both, the ranking algorithm first compares the extracted

FeatureExtraction
Algorithm

Raw Product Data
Product Image
Product Description

FeatureExtraction
Algorithm

Indexing
Algorithm(OMC-ART)

Indexed Products
Using Key
FeaturesFig. 3: The architecture of the proposed multimodal searchengine (MM-PSE). The arrowlines indicate the online processof product search; the dashed ones indicate the offline processof product indexing.
visual/textual features of the query and key features of thegeneralized characteristics of clusters in the upper layer ofthe index hierarchy; products in the cluster more similar tothe query are subsequently compared to the query. Comparedwith commonly-used ranking algorithms, the proposed ranking
algorithm of OMC-ART has been demonstrated more effective
in search accuracy and more efficient by using key features
as intermedium layer for ranking [11]. In addition, with thebatch-mode ranking manner, products in clusters less similar
to the query are less likely to be the desired ones for users.
As such, with a limited number of search result, the ranking
algorithm may stop without traversing the entire index base if
the ranking list keeps unchanged for a certain period of time,in view that the images most similar to the query are presented
prior to those of lower similarity.
D. Personalized Speech Feedback
The personalized speech feedback (PSF) engine aims topresent the summary of search results with personalizedvoices, which can either be created by the elderly or chosenfrom a list of pre-defined voices and can be customized by theelderly.
As shown in Figure 4, once receiving a textual descriptionof the search results, PSF first employs the IVONA text-tospeech (TTS) engine4 to generate the speech with IVONA’svoice, which subsequently will be transformed to the voice
designated by the elderly using voice conversion techniques.
This voice can either be chosen from a list of pre-defined
voices or created by the elderly through voice recording.
Furthermore, the elderly are allowed to make the personalizedvoices more comfortable by customizing the characteristics ofthe voices, in terms of pitch, speaking speed, and loudness.
To ensure the high quality of the generated personalizedvoices, PSF utilizes a newly proposed voice conversion
4 https://www.ivona.com/© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

150

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

User Interface
voice recording

voice selectionuser options

Personalized
speechfeedback

Search results
IVONA TTS
engineVolume indicator

Predefined voice

Record button

Voicecustomisation

Voice conversion

Training
data

Personalised feedback engine

Fig. 4: Overview of the personalized speech feedback engine.

technique [12] to transform the IVONA’s voice into thedesignated voice. Compared with other voice conversion
methods which have been shown to have own advantages anddrawbacks [33], [34], the adopted technique uses a systemfusion framework to leverage the merits of different conversion
methods [35], [36]. Please refer technical details in [12].
It is notable that, besides the voice conversion, there areother methods to generate the voice of a specific person, suchas TTS. However, building a personalized TTS system requires
the long-time recording of the target speaker and technicalskills to manually generate the training data. Therefore, PSF isdeveloped with the cost-effective voice conversion techniques.
V. S YSTEM I MPLEMENTATIONIn this section, we illustrate the functionality and user interfaceof the developed system. At current stage, the multimodal
product search engine has been implemented in a Linux
environment and tested using real-world ecommerce data,
namely, the REC-TMALL dataset5 ; while the crowd-improved
speech recognition and personalized speech feedback functions are implemented in a Windows environment and tested
using a real-world E-commerce product dataset. Note that the
integrated system of the three functional modules above has
been in development and the following demonstration is basedon our tested version.
A. Crowd-improved Speech Recognition
Figure 5 illustrates the user interface of the crowd-improved
speech recognition system. The interface provides users witha microphone button to record speech search query directlyin the web browser. When the recording function is active, avolume indicator below the search box shows up to help theusers make recordings with appropriate loudness.
As described in Section IV-B, our proposed CISR system
uses a pre-defined grammar in its first stage to improve therecognition. Table I gives examples on the grammar used.
In those examples, product titles in the selected product list
are used to extract certain keywords such as brand names

Fig. 5: User interface of CISR integrated with the searchengine of a prototype E-commerce website
and product categories. After that, the system automaticallycreates recognition grammar using certain combinations of
those keywords. Note that “*” is the wild card that matches
any words so that keywords in grammars appearing in a user’s
speech will be accurately detected in Stage 1 of CISR in
real-time. In our current implementation, the extraction of
keywords from product titles requires human intervention withcertain filtering techniques.
TABLE I: Examples of generated grammar from product listin CISR. Each line in the grammar column is a template usedin Stage 1 of CISR. The notation “*” is the wild card.
Product title

Keywords

GrammarSennheiser RS120 OnEar 926MHz WirelessRF headphones with
Charging Cradle

Sennheiser,
headphone* Sennheiser ** headphone ** Sennheiser headphone *Transcend 8 GB Class
10 SDHC flash memory
card (TS8GSDHC10E)

Transcend,memory card

* Transcend ** memory card ** Transcend memory card *Table II illustrates two examples on the comparison of
speech recognition results from the Bing voice recognitionengine, the Stage 1 of CISR, and the output of CISR.
The first example demonstrates the behaviors of different
systems with an input speech when the requested product(i.e. Sennheiser headphone) is inside the product list for
CISR, while the second example demonstrates those when
TABLE II: Examples of speech recognition results from theBing voice recognition engine, the Stage 1 of CISR, and theoutput of CISR.
Input speech
Do you haveany
Sennheiser
headphone?
I want tobuy aSamsung
Chromebook.

5 https://tianchi.aliyun.com/datalab/index.htm?spm=5176.100075.2368.2.5c

Recognition engine

ResultBing voice recognition engine
Stage 1 of CISR

Do you have anything either
headphone?
Sennheiser headphone

CISR

Sennheiser headphone

Bing voice recognition engine
Stage 1 of CISR

I want to buy a Samsung
chromebook.
N/A

CISR

I want to buy a Samsung
chromebook.

zf7X© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

151

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

(a)

(b)

(c)

Fig. 6: An example of search results of the multimodal search engine on the REC-TMALL dataset, with queries of (a) an
image; (b) keywords; and (c) both image and keywords.

the requested product (i.e. Samsung Chromebook) is outside
the product list. We observe that, as a general purpose
recognition engine without the help of product list, the Bing
voice recognition engine may not recognize complicated brandname (in this case, the Sennheiser) correctly. In contrast, thestage 1 of CISR is able to successfully recognize the productkeywords. In this case, the Stage 2 of CISR is skipped due
to high confidence of the result from Stage 1 so that the
final result of CISR is the same as the result of Stage 1.
When the requested product is outside the product list, asshown in the second example, the Stage 1 of CISR cannot
produce any results with high confidence. However, CISR stillobtains high-quality result by activating the Stage 2, whichuses a general purpose recognition engine, currently the Bing
voice recognition engine, without constraint of the product list
grammar. In this way, CISR ensures a robust performance inspeech recognition through the two-stage model.
B. Multimodal Product Search Engine
Figure 6 illustrates the user interface and the functionalityof the developed multimodal product search engine using an
example of search results on the REC-TMALL dataset. As
observed, the designed user interface follows that adopted
by traditional ecommerce platforms with a “camera” icon to
upload images, which will be displayed on the left side of thesearch bar.
As shown in Figure 6a, by uploading an image without text,
the search engine will return a list of products with similar
product photos in a top-down left-right order in terms ofsimilarity. This similarity is, as suggested in [11], measured bythree types of visual features, including the grid color moment,
the edge direction histogram, and the wavelet texture. We
observe that three products in the top four returned ones are of
the same brand to the uploaded image. Note that the uploaded

image is exactly that of the top one returned product. The clothof the top third product is not of the same style to the querythough, it is visually similar in terms of the whole. Because
all related products are listed in the top four products, the rest
products are not similar to the query. However, these products
are also related to fashion cloth. These findings demonstrated
the effectiveness of the search engine.
As a flexible multimodal search engine, besides visual
search, users are also allowed to query by text or a combinationof both image and text, as shown in Figures 6b and 6c.
Note that the text is in Chinese because the utilized RECTMALL dataset is from a Chinese E-commerce platform.
From Figure 6c, we observe that, with additional textual
description in the query, the search result has a significant
improvement, compared with those in Figure 6a; besides the
products of the same brand to the query, the rest retrieved
products are all about fashion lady cloth of Korean style,which meet all requirements in the textual description. We
also observe that the search result using solely keywords in
Figure 6b is quite different from that in Figures 6a and 6c.
This is because the multimodal search engine, namely, the
OMC-ART [11], is able to make a tradeoff for measuring thesimilarity between the query and products in the index base,which is an automatically leaned parameter during the productindexing process and can be fine-tuned manually.
C. Personalized Speech Feedback
As described in Section IV-D, PSF allows the elderly to selectpreferred voices, create personalized voices, and customize
voices. Figure 7 shows the user interface of PSF. As observed,
the pre-defined and personalized voices are listed in different
sectors, and the elderly may play the audio sample of eachvoice by clicking the flag on the right side of the speaker’s
name, say “amy”, and select their favourite ones for speech© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

152

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Fig. 7: The user interface of the prototype system with thepersonalized speech feedback engine.

Please read thts sentence.

Many farms could be bankrupted as a result.

t Start Recording~ PlayhUploadSentences
You can review or re·record any senter.ce by dlcking on one betow.
farms eould ~ bankru~d as a ~sutt.

",. matn ehal"nG' for cutturat tndusm.s Is obvlousty information *hnology.

Slaws: Recorded

(a)

(b)
Fig. 8: The user interface of the prototype webpage for voicerecording.

Fig. 9: The pop-up panel for voice customization.

“Start Voice Conversion Training” button at the bottom of the
webpage (Figure 8b) to train a personalized transformation
function. When completed, the trained voice will be added
to the user interface for voice selection, in our case shownin Figure 7, the option “Simon” under the “Personalizedvoices” list. It is worth mentioning that to achieve a robusttransformation function, the recording should be done in aquiet environment, and the speaker is preferred to use a neutral
speaking style.
Beyond the creation of personalized voices, the elderly canclick the “Tune voice” button to modify any voices in thelists of predefined voices and personalized voices through a
pop-up panel. As shown in Figure 9, the elderly may tune thecharacteristics of the selected voice to customize the voices tobe more comfortable to them, in terms of the speaking speed,
the pitch, and the loudness. Besides, a preview function isprovided for checking the customized voices. After that, user
can save the changes by clicking the “Save Configuration”
button.
VI. C ONCLUSION AND F UTURE W ORKS

feedback. Besides, the elderly are able to create personalizedvoices or customize the selected voice by clicking the buttons“Create new voice” and “Tune voice”, respectively.
Clicking the “Create new voice” button will lead the elderly
to the webpage for voice recording, as shown in Figure 8. The
elderly may create the voice of any person by inviting him/her
to read at least 20 prepared sentences for recording. Notethat we have a corpus of 100 sentences and the recording of
more sentences will result in a higher quality of the converted
voice. We also developed well-designed functions to help the
speaker finish the recording process. Specifically, as shownin Figure 8a, a speaker can start to record by following
sentences showing one by one on the top of the webpage,check the recording by clicking the “Play” button, and upload
the recordings of good quality to server. Once uploaded, thespeaker can check the recording of each sentence. Note that
Figure 8a captures the screenshot of two example sentences.
Once the recording is finished, the speaker may click theThis paper presents a system for age-friendly E-commerce,targeting at enhancing the elderly’s browsing experience in
E-commerce platforms. Different from most existing work
focusing on age-friendly website design, the proposed system takes advantage of recent advanced search and speechtechnologies, and provides assistive functions for the elderly,
including crowd-improved speech recognition, flexible multimodal search, and personalized speech feedback. The crowdimproved speech recognition function makes use of a crowdsourced and personalized product list to provide fast andaccurate recognition of the elderly’s voice query. The flexible
multimodal search engine, working together with the crowdimproved speech recognition function, support speech, visual,and text queries for product search, in terms of individual or
any combination of them whichever are convenient for theelderly. The personalized speech feedback function provides a
speech summary of search result in a personalized voice for theelderly, which is aimed to facilitate the elderly to understandthe large amount of returned products relevant to their query.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

153

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Note that through the personalized speech feedback function,
the elderly are able to customize the synthetic voice, such asvolume and speed, and even create their preferred voice.
Despite the current achievement, our future work mainly
includes three aspects. First, the three testified functions of
the proposed system will be implemented into one platform.
Second, a series of user study will be conducted to evaluate
the system and guide further research on system improvement.
Third, other assistive techniques will be explored to enrich thesystem, such as age-friendly online assistant and personalized
product recommendation.
ACKNOWLEGEMENT
This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its IDM Futures
Funding Initiative.
R EFERENCES[1] E. Lawrence, S. Newton, B. Corbitt, J. Lawrence, S. Dann, andT. Thanasankit, Internet commerce: digital models for business. John
Wiley & Sons, 2003.
[2] “World population ageing 2013,” Population Division, United Nations
Department of Economic and Social Affairs, 2013. [Online].
Available: http://www.un.org/en/development/desa/population/publicat\\ions/pdf/ageing/WorldPopulationAgeing2013.pdf[3] J. Nielsen, “Seniors as web users,” Nielsen Norman Group. Accessed
August, 2013. [Online]. Available: https://www.nngroup.com/articles/
usability-for-senior-citizens/[4] S. A. Becker, “A study of web usability for older adults seeking online
health resources,” ACM Transactions on Computer-Human Interaction(TOCHI), vol. 11, no. 4, pp. 387–406, 2004.
[5] L. Chen, M. L. Gillenson, and D. L. Sherrell, “Consumer acceptance of
virtual stores: a theoretical model and critical success factors for virtual
stores,” ACM Sigmis Database, vol. 35, no. 2, pp. 8–31, 2004.
[6] B. Niehaves and R. Plattfaut, “Internet adoption by the elderly:
employing is technology acceptance theories for understanding the agerelated digital divide,” European Journal of Information Systems, vol. 23,
no. 6, pp. 708–726, 2014.
[7] A. Dickinson, P. Gregor, L. McIver, R. Hill, and S. Milne, “The nonbrowser: helping older novice computer users to access the web,” inProceedings of the 2005 international conference on Accessible Designin the Digital World. British Computer Society, 2005, pp. 18–18.
[8] A. Dickinson, M. J. Smith, J. L. Arnott, A. F. Newell, and R. L.
Hill, “Approaches to web search and navigation for older computernovices,” in Proceedings of the SIGCHI conference on Human factors
in computing systems. ACM, 2007, pp. 281–290.
[9] A. Holzinger, G. Searle, T. Kleinberger, A. Seffah, and H. Javahery,
Investigating usability metrics for the design and development ofapplications for the elderly. Springer, 2008.
[10] “China’s baidu launches portal for the elderly,” PCWorld, 2009.
[Online]. Available: http://www.pcworld.com/article/162914/article.html[11] L. Meng, A.-H. Tan, C. Leung, L. Nie, T.-S. Chua, and C. Miao,“Online multimodal co-indexing and retrieval of weakly labeled web
image collections,” in Proceedings of the ACM International Conference
on Multimedia Retrieval. ACM, 2015, pp. 219–226.
[12] X. Tian, Z. Wu, S. W. Lee, N. Q. Hy, M. Dong, and E. S. Chng, “Systemfusion for high-performance voice conversion,” in INTERSPEECH,
2015.
[13] A. Tatnall and J. Lepa, “The internet, e-commerce and older people: anactor-network approach to researching reasons for adoption and use,”
Logistics Information Management, vol. 16, no. 1, pp. 56–63, 2003.
[14] C. W. Phang, J. Sutanto, A. Kankanhalli, Y. Li, B. C. Tan, and H.-H.
Teo, “Senior citizens’ acceptance of information systems: A study in thecontext of e-government services,” IEEE Transactions on Engineering
Management, vol. 53, no. 4, pp. 555–569, 2006.
[15] L. Groff, C. Liao, B. Chaparro, and A. Chaparro, “Exploring how theelderly use the web,” Usability News, vol. 1, no. 2, pp. 1–2, 1999.

[16] R. Fukuda and H. Bubb, “Eye tracking study on web-use: Comparison
between younger and elderly users in case of search task with electronic
timetable service.” PsychNology Journal, vol. 1, no. 3, pp. 202–228,
2003.
[17] T. A. Hart and B. Chaparro, “Evaluation of websites for older adults:
How” senior friendly” are they,” Usability News, vol. 6, no. 1, p. 12,
2004.
[18] T. J. Smith, “Senior citizens and e-commerce websites: The role ofperceived usefulness, perceived ease of use, and web site usability,” Informing Science: International Journal of an Emerging Transdiscipline,
vol. 11, pp. 59–83, 2008.
[19] C. Hudson, C. Scialfa, R. Diaz-Marino, J. Laberge, and S. MacKillop,“Effects of navigation aids on web performance in younger and older
adults,” Gerontechnology, vol. 7, no. 1, pp. 3–21, 2008.
[20] E. Helsper, “The ageing internet: digital choice and exclusion among theelderly,” Working with Older People, vol. 13, no. 4, pp. 28–33, 2009.
[21] L. Kang and H. Dong, “B2c websites usability for chinese seniorcitizens,” in Human-Computer Interaction. Applications and Services.
Springer, 2014, pp. 13–20.
[22] V. L. Hanson, J. P. Brezin, S. Crayne, S. Keates, R. Kjeldsen, J. T.
Richards, C. Swart, and S. Trewin, “Improving web accessibility throughan enhanced open-source browser,” IBM Systems Journal, vol. 44, no. 3,
pp. 573–588, 2005.
[23] V. L. Hanson and S. Crayne, “Personalization of web browsing:
adaptations to meet the needs of older adults,” Universal Access in the
Information Society, vol. 4, no. 1, pp. 46–58, 2005.
[24] A. M. Franz, M. H. Henzinger, S. Brin, and B. C. Milch, “Voice interface
for a search engine,” Apr. 11 2006, uS Patent 7,027,987.
[25] J. Schalkwyk, D. Beeferman, F. Beaufays, B. Byrne, C. Chelba,
M. Cohen, M. Kamvar, and B. Strope, ““your word is my command:
Google search by voice: A case study,” in Advances in Speech
Recognition. Springer, 2010, pp. 61–90.
[26] F. Biadsy, P. J. Moreno, and M. Jansche, “Google’s cross-dialect arabic
voice search,” in Acoustics, Speech and Signal Processing (ICASSP),
2012 IEEE International Conference on. IEEE, 2012, pp. 4441–4444.
[27] V. L. Hanson, J. T. Richards, and C. C. Lee, “Web access for older adults:
voice browsing?” in Universal Acess in Human Computer Interaction.
Coping with Diversity. Springer, 2007, pp. 904–913.
[28] V. Chattaraman, W.-S. Kwon, J. E. Gilbert, and S. In Shim, “Virtual
agents in e-commerce: representational characteristics for seniors,”
Journal of Research in Interactive Marketing, vol. 5, no. 4, pp. 276–297,
2011.
[29] “HTML5 Web Audio specification.” [Online]. Available: http://www.
w3.org/TR/webaudio[30] “Microsoft Speech Platform.” [Online]. Available: https://msdn.
microsoft.com/en-us/library/jj127858.aspx[31] “Bing Voice Recognition (beta).” [Online]. Available: https://datamarket.
azure.com/dataset/bing/speechrecognition[32] L. Meng, A.-H. Tan, and D. Xu, “Semi-supervised heterogeneous fusion
for multimedia data co-clustering,” IEEE Transactions on Knowledge
and Data Engineering, vol. 26, no. 9, pp. 2293–2306, 2014.
[33] T. Toda, H. Saruwatari, and K. Shikano, “Voice conversion algorithm
based on Gaussian mixture model with dynamic frequency warpingof straight spectrum,” in Acoustics, Speech, and Signal Processing,
2001. Proceedings.(ICASSP’01). 2001 IEEE International Conferenceon, vol. 2. IEEE, 2001, pp. 841–844.
[34] D. Erro and A. Moreno, Asunciónand Bonafonte, “Voice conversion
based on weighted frequency warping,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 18, no. 5, pp. 922–931, 2010.
[35] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum-likelihood estimation of spectral parameter trajectory,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8,
pp. 2222–2235, 2007.
[36] X. Tian, Z. Wu, S. W. Lee, N. Q. Hy, E. S. Chng, and M. Dong,“Sparse representation for frequency warping based voice conversion,”in in IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP). IEEE, 2015, pp. 4235–4239.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

154

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Towards Data-driven Software Engineering Skills
Assessment
Jun Lin† , Han Yu† , Zhengxiang Pan† , Zhiqi Shen∗† and Lizhen Cui‡
∗ School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore† Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), NTU, Singapore‡ School of Computer Science and Technology, Shandong University, China{jlin7, han.yu, panz0012, zqshen}@ntu.edu.sg, clz@sdu.edu.cn
Abstract—Today’s software engineers often work in teams todevelop complex software systems. Therefore, successful software
engineering in practice not only require team members to possess
sound programming skills such as analysis, design, coding, andtesting, but also soft skills such as communication, collaboration,and self-management, etc. However, existing examination based
assessments are often inadequate for quantifying students’ soft
skill development. In this paper, we explore alternative ways forassessing software engineering students’ skills through a datadriven approach. Leveraging our proposed HASE online agile
project management (APM) tool, we conducted a study involving
21 Scrum teams consisting of over 100 undergraduate software
engineering students in multi-week coursework projects in 2014.
During this study, students performed close to 170,000 software
engineering activities logged by HASE. By analyzing the collected
activity trajectory dataset, we demonstrate the potential for thisnew research direction to enable software engineering educators
to have a quantifiable way of understanding their students’ skill
development, and take a proactive approach in helping them
improve their programming and soft skills.

I. I NTRODUCTION
Most hiring managers in software engineering companies
understand that a successful member of software engineer
needs to be strong in both programming skills (e.g., software
design, coding, and testing skills) and soft skills (e.g., communication, collaboration, and self-management skills) [1]. Programming skills can be gauged, at least in part, from students’performance in examinations and programming contests. Softskills are much harder to assess, especially during the limited
time given in job interviews. Although the concept of theseskills can be taught, the ability to apply them consistently inpractice can only be acquired through one’s own experience.
In tertiary education institutions, software engineering students are often assessed by a combination of examinations
and coursework projects. Many educators have realized thelimitations of examinations in assessing students’ practicalskills. Thus, coursework projects often serve as an opportunity
for students to both practice and demonstrate their skills.
However, as an instructor has to face tens or even hundredsof students in a semester, it is not practical for him/her toknow the weaknesses and strength in each student’s skills in
detail through observation. Technologies that can subjectively
quantify students’ skill development are needed to enable
instructors to proactively and effectively help each student.

With the emergence of systems capable of collecting personal behavior trajectory big data [2], data-driven analysis ofpeople’s characteristics over time is changing how students’performance can be measured. Some funding agencies arestarting to support research in data-driven student assessment
technologies to complement traditional examination scores.
For example, the Ministry of Education in Singapore has
started an initiative to build technological solutions capableof holistically assessing students’ 21st Century Competencies(e.g., critical thinking, self-directed learning skills)1 .
Following a similar line of thinking, in this paper, we
explore how software development behaviour data can beused to assess students’ programming and soft skills. As
agile software development (ASD) involves many humanfactors reflecting developers personal characteristics comparedto other plan-driven methodologies [3], we focus on tracking
students’ activities in the ASD process. For this purpose,
we conducted a 12 week study involving 125 undergraduate
software engineering students from the Beihang University,
Beijing, China. The students self-organized into 21 ASD teamsof 5 to 7 persons. Each team developed one software systemof significant complexity following the Scrum ASD method
as part of their course requirements. Some examples of thecoursework projects include “A Personal Healthy Living App”,“A Social Network App for Senior Citizens”, and “An Activity
Tracking App for the Elderly”, etc.
Students in this study carried out software engineering
activities at various stages of the Scrum methodology in our
online agile project management (APM) tool - the Humancentred Agile Software Engineering (HASE) platform (http://www.linjun.net.cn/hase/) [4]. HASE mainly supports activities during the sprint planning and sprint review/retrospective
phases. Such activities include proposing tasks, estimating thepriority, difficulty and time required for each task, deciding
how to allocate tasks, collaboration information, reviewing thetimeliness and quality of completed tasks, and providing feedback on individual team member’s mood at different points
in time during a sprint. During the study, students logged
169,137 ASD activities in the HASE platform. By analyzing
the collected dataset to reflect students’ programming skills,
1 The plan for the initiative can be accessed from: http://www.moe.gov.sg/
media/press/2010/03/moe-to-enhance-learning-of-21s.php© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

155

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

collaboration, and mood stability, we demonstrate the potentialof this research direction and discuss its implications forsoftware engineering education.
II. R ELATED W ORK
To the best of our knowledge, there has yet to be publishedprevious studies using software engineering activity data toassess software engineers’ skills. Nevertheless, as the skills
assessment has always been an important problem, othermethods have been applied in an attempt to address it.
Author in [5] advocated evidence-based software engineering (EBSE) similar to what is happening with evidence-based
medicine. A technological platform for tracking and analyzing important factors in software engineering such as skills
factors and lifecycle factors were called for, and the benefitsof which were analyzed. Nevertheless, the work intends to
produce methods to support the development of high qualityof software through objective analysis of performance related
indicators. Although similar in principle to our work, [5] does
not specify how EBSE can assess the skills demonstrated by
the software engineers or how such insight can be used to
improve software engineering education.
In [6], the authors presented the results of a systematic
literature review concerning agile pair programming effectiveness. The paper analyzed compatibility factors, such as the feel
good, personality, and skill level factors, and their effect onpair programming effectiveness. Four metrics were used in theanalysis: 1) academic performance, 2) technical productivity,
3) program/design quality and 4) learning satisfaction. As thestudy was not focused on assessment, the general findings
are not useful for skills assessment. Nevertheless, it did point
towards the importance of soft skills in software engineering.
The study reported in [4] started to track personal performance data with agile project management tools to study
task allocation related decision-making under Scrum. It employed the same research techniques as reported in this paper.
However, the study in [4] focused on analyzing students’programming skills and did not consider their soft skills suchas collaboration and mood.
III. S TUDY D ESIGNIn this section, we present our research approach, and themetrics adopted in our analysis.
A. Research Approach
We use the HASE APM platform to unobtrusively track
the student participants’ activities in the Scrum ASD process,including their decision-making, collaboration, task assignment and mood, etc. The platform provides five main features
to support agile project management which cover the sprint
planning and sprint review/retrospective phases:
1) Registration: In order to build user profiles, HASE
requires registrants to specify their self-assessed competence levels in different areas of expertise such as
familiarity with specific programming languages, systemdesign methodologies, and user interface (UI) design

tools, etc. This information will only be used to compute
an initial assessment for a user in the absence of peer
ratings or performance data. Once data from these
relatively more objective sources become available, theuser’s self-assessment will be excluded from the assessment result.
2) Team and Role Management: HASE supports the creation of teams, the selection of product owners andstakeholders into the teams, and the assignment ofdifferent roles within a team (e.g., programmers and UIdesigners).
3) Task Management: Task information including task description, skills required for the task, and the person who
proposed each task is displayed for all team members
to view. The difficulty value of each task τ , is recorded
using an 11-point Likert scale [7] (with 0 denoting“extremely easy” and 10 denoting “extremely hard”).
Each team member can input his/her estimated difficulty
value for each task into the HASE platform. The HASE
platform then uses the average difficulty value for thetask (Dτ ). The students were asked to take into accountthe technical challenge as well as the amount of effort
required when judging the difficulty of a task. The
priority value of each task τ , is also recorded using
an 11-point Likert scale (with 0 denoting “extremely
low priority” and 10 denoting “extremely high priority”).
Each team member can input his/her estimated priority
value for each task into the HASE platform. The HASE
platform then uses the average priority value for the task.
4) Sprint Planning: HASE records the teams’ decisions
on which tasks are assigned to which team member
during each sprint. Once assigned, the status of thetask becomes “Assigned”. The assignee i inputs his/her
confidence value (Confτi ) for each task τ on an 11-point
Likert scale (with 0 denoting “not confident at all” and10 denoting “extremely confident”). Each team member
also inputs the estimated required time to complete eachtask (in number of days). The HASE platform uses theaverage estimated time required to generate the deadline
for the task (Tτest ). Apart from a primary assignee,
multiple students can collaboratively work on a task. The
collaborator information for each task is also recorded
by HASE.
5) Sprint Review/Retrospective: Once a task is completed,the assignee changes its status in the HASE platform to“Completed”. This action will trigger HASE to record
the actual number of days (Tτact ) used to complete thistask. HASE also provides functions for team members
to peer review the quality (Qualτ ) of each completed
task τ . The quality of a completed task is recorded in
the platform using a 11-point Likert scale with 0 representing (“extremely low quality”) and 10 representing(“extremely high quality”). The average quality rating
for each task is used by HASE as the final quality rating
for that task.
6) Team Morale Monitoring: During the sprint planning© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

156

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

4

100

18

90

3

16

80

2

14

70

12

Participant ID

Technical Productivity

20

10
8

60

0

50
−1
40
−2

6

30

4

20

2

10

0

1

0

0.1

0.2

0.3

0.4

0.5
0.6
Competence

0.7

0.8

0.9

−3
−4

2

1

4

Fig. 1. Students’ competence v.s. productivity.
120
5

4

Participant ID

80

3

60

40

2

20

1

2

4

6
Week

8

10

12

0

Fig. 2. Average number of collaborators per task.

meeting, team members can report their current moodvalues into the HASE platform. A person i’s mood atthe beginning of a sprint t (mbegin(t)) is representedion a 5-point Likert scale with 1 representing “verylow” and 5 representing “very high”. During the sprint
review/retrospective meeting, each task assignee i canreport his/her mood after completing a task at the endof sprint t (mend(t)) using the same 5-point Likert scale.
i
The input data to the HASE platform required from ASD
teams are as a result of students’ activities following the Scrum
methodology. In this way, users of HASE can behave as if they
are using any APM tool without expending additional effort
to help with data collection. Thus, the data collection process
remains unobtrusive to the participants. Over the 12 week
period of the coursework project, the HASE platform collected
169,137 behaviour trajectory records related to software engineering activities from the 125 students who participated inthis study. The anonymized dataset is made available to theresearch community to support further studies and analysis2 .
2 Link to the dataset.

8

10

−5

12

Fig. 3. Intra-week mood variation.
6

100

6
Week

B. Metrics
In this paper, we adopt the exploratory data analysis (EDA)
approach [8] to analyze the data collected. EDA is an approach
for analyzing data sets to summarize their main characteristics,
often with visual methods. It is primarily for understanding
what can be learnt from the data beyond the formal modelingor hypothesis testing task. We use the following metrics tofacilitate our analysis:
1) Technical Productivity (µi ): it refers to the averageamount of workload a student i can complete duringa sprint. In this study, we use the task difficulty value
as an indicator of the workload of a task as the taskdifficulty values reported by students denote both thetechnical challenge and the amount of effort required tocomplete the task.
2) Competence (Compi ): it refers to the probability astudent i can complete a task assigned to him/her withsatisfactory quality before the stipulated deadline. In thispaper, the outcome of a task needs to achieve an average
quality rating higher than 5 out of 10 in order to beconsidered as having satisfactory quality. This metric issimilar to a student’s reputation. Thus, we adopt a reputation computation model - the Beta Reputation model[9] - which is widely used in the fields of online services,
artificial intelligence and network communications [10],[11], [12], [13], [14]. It is calculated as follows:
Compi =

αi + 1∈ (0, 1)(αi + 1) + (βi + 1)

where αi and βi are calculated as:Xαi =1[T act −T est ≤0 and Qualτ >5] Dτ
τ
τ

(1)

(2)

τ ∈φ(i)βi =X

1[Tτact −Tτest >0 or Qualτ ≤5] Dτ .

(3)

τ ∈φ(i)

The function 1[condition] in Eq. (2) and Eq. (3) equals to
1 if “condition” is true. Otherwise, 1[condition] equals to
0. φ(i) denotes the set of tasks i has previously worked© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

157

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

100
90
80
70
Skills Score

on until the current point in time. The “+1” terms in
the numerator and denominator of Eq. (1) are Laplacesmoothing terms [15] which ensure that if i has noprevious track record, Compi evaluates to 0.5 indicating
maximum uncertainty about i’s performance.
3) Team Morale (Begin) (Mjbegin (t)): it refers to the average of the mood values reported by members of team jduring the sprint planning meeting of sprint t.
4) Team Morale (End) (Mjend (t)): it refers to the average ofthe mood values reported by members of team j during
the sprint review/retrospective meeting of sprint t.

60
50
40
30
20

IV. R ESULTS AND A NALYSIS

0

0

10

20

30

40
50
60
Examination Score

70

80

90

100

Fig. 4. Participants’ skills score v.s. their examination score.

Technical Productivity

An exploratory data analysis has identified certain personal
characteristics which may become useful markers for assessing
students’ skills in the future. Figure 1 shows the participants’competence scores versus their productivity scores at the endof the study. It can be observed that the participants’ performances in terms of these metrics are quite distinguishable. In
general, participants who demonstrated high competence tend
to also be able to handle high workloads allocated to them(r = 0.7443, p < 0.01). One participant achieved significantly
higher competence and productivity scores than the rest of theparticipants.
Collaboration is generally regarded as a useful way to
improve the effectiveness and efficiency of a software team.
Figure 2 shows a heat map of the number of collaborators pertask each participant had for each of the 12 weeks. The lighter
the color of a point on the figure, the more collaborators pertask that participant had for that particular week. The color
scale mapping different color gradients to the actual numberof collaborators per task is shown on the right-hand side of
the figure. Participants are ranked according to their average
number of collaborators per task per week. Those who areshown at the bottom of the figure ranked the highest among
their peers. It can be observed that this metric can distinguishthe behaviors among different participants clearly.
Stability of mood is a sign showing one’s maturity andself-management skills. Figure 3 shows a heat map of theintra-week mood change (which is computed as ∆mi (t) =mend(t) − mbegin(t) ∈ (−5, 5) for each week) over theii
12 weeks. 102 out of the 125 participants provided valid
reports on their mbegin(t) and mend(t) values. The coloriiscale mapping different color gradients to the intra-weekmood change is shown on the right-hand side of the figure.
Participants are ranked according to their average intra-weekmood change values per week over 12 weeks. Those who areshown at the bottom of the figure ranked the highest among
their peers. It can be observed that this metric can distinguishthe behaviors among different participants quite clearly. The
mood of those who ranked high on this metric tends to increaseat the end of a week after a sprint of development. And as
their mood at the beginning of the week also tends to be high,the increments are generally small. Thus, their mood remain
relatively stable throughout a sprint. Those who ranked low on

10

20

5

18

4.5

16

4

14

3.5

12

3

10

2.5

8

2

6

1.5

4

1

2

0.5

0

0

0.1

0.2

0.3

0.4
0.5 0.6
Competence

0.7

0.8

0.9

0

1

Fig. 5. Students’ average morale before a Sprint.

this metric (top part of the figure) tend to have big negative
mood swings, especially towards the end of the study.
In order to explore if the assessment of participants’ skills
may help us identify students who are good at hands-on
software engineering but did not stand out in examinations, weconstruct a skills score to aggregate the effect of competence,
productivity, collaboration, and mood stability into one scalarmeasurement. In this study, the skills score, Sskills (i), for aparticipant i is computed as:
Sskills =Sµi + SCompi + Scoli
100,
×1 − S∆mi
3(4)

where Sµi ∈ [0, 1], SCompi ∈ [0, 1], Scoli ∈ [0, 1] andS∆mi ∈ (−1, 1) are the normalized scores for i in terms of
productivity, competence, collaboration, and mood stability,
respectively (Sskills (i) ∈ [0, 100]).
Figure 4 plots the participants’ skills scores against their
examination scores for the subject of software engineeringin the same semester. The examination paper used was thestandard software engineering end-of-semester examination
paper from the Beihang University, which has been designedby the professors in charge of the course and reviewed by
the university examination board. It can be observed that,© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

158

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

3.50.01), indicating a statistically significant positive correlation.
Therefore, based on these analysis, team members with hightechnical productivity tend to have high morale, especially atthe end of a sprint after completing the tasks allocated to them.

3

V. I MPLICATIONS

20

4.5

18

4

16

Technical Productivity

14
12
2.5
10
2
8
1.5

6
4

1

2

0.5

0

0

0.1

0.2

0.3

0.4
0.5 0.6
Competence

0.7

0.8

0.9

1

0

Fig. 6. Students’ average morale after a Sprint.

according to their exam scores, their performance clustered in
the range of 80 to 100 marks, enabling almost all of them toachieve a grade of A or A+. However, their skills scores spreadfrom as low as 10 marks to as high as 80 marks, making theirperformance more distinguishable compared to their exam
scores. The skills scores have only a weak positive correlation
with the exam scores (r = 0.2129, p < 0.05). Furthermore,
the top three best performing participants in terms of skills
scores achieved only average exam scores among their peers,and many participants with high exam scores achieved low
skills scores.
We acknowledge that there may be other ways to computethe skills score and we refrain from claiming that our current
formulation for the skills score is the most effective. Nevertheless, the results show that the data-driven skills score canpotentially help us distinguish the performance of software engineering students better than examination-based assessments.
Figure 5 shows the distribution of students’ average selfreported mood values during the sprint planning meeting at thestart of each sprint. The colour scale represents the average
self-reported mood values. The average mood value is 3.86
out of 5. The correlation between students’ mood during thesprint planning meetings and their competence values is (r =−0.0025, p = 0.9394), indicating no statistically significant
correlation. The correlation between students’ mood during
the sprint planning meetings and their technical productivity
values is (r = 0.1505, p < 0.01), indicating a statisticallysignificant albeit weak positive correlation.
Figure 6 shows the distribution of students’ average selfreported mood values during the sprint review/retrospective
meeting at the end of each sprint. The colour scale represents
the average self-reported mood values. The average moodvalue is 3.80 out of 5 which is slightly lower than at the
beginning of the sprint. The correlation between students’mood during the sprint review/retrospective meetings and theircompetence values is (r = −0.0148, p = 0.5946), indicating
no statistically significant correlation. The correlation between
students’ mood during the sprint review/retrospective meetings
and their technical productivity values is (r = 0.4207, p <By providing a technological platform for the longitudinal
tracking of software engineers’ behaviour trajectory data related to software development, we open up new possibilities fordifferent parties involved in software engineering researchers,
educators, and practitioners.
Implications for software engineering researchers: Theavailability of large software engineering behaviour datasets
will present new challenges to researchers to develop new
analytics techniques. With detailed information on each user’s demographics, skills indicator values over time, detailed
interactions with the software engineering tools provided, interactions with other team members, and decisions made, etc.,
the high dimensionality of the datasets makes it a challenge to
identify which feature, or combination of features, can form
accurate predictors for certain behaviours of interest. Machinelearning [16] can be leveraged to develop useful techniques
for this purpose.
However, before this step can happen, additional efforts areneeded to complement the behaviour datasets with labelled
meta-data on what the observed behaviour patterns mean. Thiscan potentially be achieved by conducting follow-up interviewbased studies with the participants through carefully designed
questionnaires once unique behaviour patterns have been identified. This also further opens up the research question on how
to present the behaviour pattern data in a human interpretableformat to facilitate the interviews.
Implications for software engineering educators: Software engineer educators may be a viable source of knowledge
in the effort of building up a repository of meta-data for the
behaviour patterns obtained by the tracking platform. As theyfrequently interact with students who may be using the proposed tracking platform, they can potentially provide insights
into the meanings of the behaviour patterns. The challenge
here is for software engineering researchers to provide tools
to enable educators who are willing to contribute meta-data forthe behaviour patterns to do so with ease. Techniques from thefield of crowdsourcing [17] may offer a starting point for suchan effort.
Once new techniques for automatically assessing the astudent’s skill development based on his/her behaviour patterns
are developed, new forms of real-time personalized inventions
may become available to educators. The simplest possibility
is for the system to send out alerts on students who may
require help in specific areas to course instructors. Throughmining the behaviour patterns of many students and crosschecking with their academic performance, or even employment prospects if such data are available, the system may
be able to suggest behaviour trajectories that are the mostbeneficial for students from different backgrounds, thereby
making data-driven personalized software engineering training© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

159

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

possible. The envisioned behaviour data tracking platform
can potentially convert software engineering education into atest bench for open science and enable a more adaptive andindividualized learning experience.
Implications for software engineering practitioners: Thebehaviour data are tracked in an unobtrusive manner by the
APM tool automatically. The peer review related functions
represent activities that an ASD team member already having
to perform when following the ASD practice. Overall, theproposed APM tool based behaviour data tracking approach
does not require software engineers to incur additional overhead. However, the data analytics functions provide ASD
teams with insights into detailed team dynamics and performance information which can be useful for decision-making.
Furthermore, with the behaviour data as input, automatic
context-aware software engineering task allocation decision
support mechanisms [18] become a distinct possibility. Thesemechanisms can be based on similar mechanisms available in
the field of crowdsourcing [19], [20], [21].
VI. D ISCUSSIONS AND F UTURE W ORK
In this paper, we explore a novel data-driven approach toassess software engineering students’ skills. Different fromtraditional interview/internship-based methods, our study isbased on participants ASD activity trajectory data collected unobtrusively during normal ASD processes through our
HASE APM platform. This type of data objectively reflects
developers ASD activities and performance at fine granularity.
As the data collection and analytics technologies further develop, software engineering students may eventually perform
all coursework activities in a technology platform capable ofunobtrusively collecting their behavior data and continuously
assessing a wide range of their skills over time. In this way, thestudents’ practical skill development can be monitored by theirinstructors so that pedagogical methods can be personalized tohelp individual students in specific areas. Such a tool will enable software engineering educators to have a quantifiable wayof understanding their students’ skill development and take aproactive approach in helping them develop programming andsoft skills. The skills scores may, one day, be part of a student’s
academic profile and be taken into consideration by industry
recruiters to help companies identify well rounded software
engineering talents suitable for their teams.
From this study, we see the start of a series of research andapplications in data-driven software engineering skills assessment. In future research, we plan to conduct surveys/interviews
to understand more in-depth how students collaborate. We willcontinue using the HASE platform to collect agile programming activity data over subsequent semesters, and expand ourdata collection effort to include more universities so as toinvestigate the possible effects of socio-cultural factors. Morefinely grained data such as the time each student spent on atask and the breakdown of the usage of the time will also
be collected in future versions of the HASE platform. The
resulting datasets will be published to support the discoveryof new insights.

ACKNOWLEDGMENT
This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its IDM
Futures Funding Initiative; Interdisciplinary Graduate School,
NTU; and the Lee Kuan Yew Post-Doctoral Fellowship Grant.
R EFERENCES[1] P. Freeman, A. I. Wasserman, and R. E. Fairley, “Essential elements ofsoftware engineering education,” in Proceedings of the 2Nd International
Conference on Software Engineering (ICSE’76), 1976, pp. 116–122.
[2] P. Heymann and H. Garcia-Molina, “Turkalytics: analytics for human
computation,” in Proceedings of the 20th International Conference on
World Wide Web (WWW’11), 2011, pp. 477–486.
[3] A. Cockburn and J. Highsmith, “Agile software development, the people
factor,” Computer, vol. 34, no. 11, pp. 131–133, 2001.
[4] J. Lin, H. Yu, Z. Shen, and C. Miao, “Studying task allocation decisionsof novice agile teams with data from agile project management tools,”
in Proceedings of the 29th IEEE/ACM International Conference on
Automated Software Engineering (ASE’14), 2014, pp. 689–694.
[5] B. A. Kitchenham, T. Dyba, and M. Jorgensen, “Evidence-based software engineering,” in Proceedings of the 26th International Conference
on Software Engineering (ICSE’04), 2004, pp. 273–281.
[6] N. Salleh, E. Mendes, and J. Grundy, “Empirical studies of pair programming for cs/se teaching in higher education: A systematic literature
review,” IEEE Transactions on Software Engineering (TSE), vol. 37,
no. 4, pp. 509–525, 2011.
[7] R. Likert, “A technique for the measurement of attitudes,” Archives of
Psychology, vol. 22, no. 140, 1932.
[8] J. W. Tukey, Exploratory Data Analysis. Addison-Wesley, 1977.
[9] A. Jøsang, R. Ismail, and C. Boyd, “A survey of trust and reputation
systems for online service provision,” Decision Support Systems (DSS),
vol. 43, no. 2, pp. 618–644, 2007.
[10] L. Pan, X. Meng, Z. Shen, and H. Yu, “A reputation pattern for service
oriented computing,” in Proceedings of the 7th International Conferenceon Information, Communications and Signal Processing (ICICS’09),
2009.
[11] H. Yu, Z. Shen, C. Miao, C. Leung, and D. Niyato, “A survey oftrust and reputation management systems in wireless communications,”
Proceedings of the IEEE, vol. 98, no. 10, pp. 1755–1772, 2010.
[12] H. Yu, S. Liu, A. C. Kot, C. Miao, and C. Leung, “Dynamic witness selection for trustworthy distributed cooperative sensing in cognitive radio
networks,” in Proceedings of the 13th IEEE International Conference
on Communication Technology (ICCT’11), 2011, pp. 1–6.
[13] S. Liu, H. Yu, C. Miao, and A. C. Kot, “A fuzzy logic based reputation
model against unfair ratings,” in Proceedings of the 12th International Conference on Autonomous Agents and Multi-agent Systems(AAMAS’13), 2013, pp. 821–828.
[14] H. Yu, Z. Shen, C. Leung, C. Miao, and V. R. Lesser, “A survey ofmulti-agent trust management systems,” IEEE Access, vol. 1, no. 1, pp.
35–50, 2013.
[15] Y. Wang and M. P. Singh, “Formal trust model for multiagent systems,”
in Proceedings of the 20th International Joint Conference on ArtificalIntelligence (IJCAI’07), 2007, pp. 1551–1556.
[16] Y. Anzai, Pattern Recognition and Machine Learning. Academic Press,
Inc., 1992.
[17] A. Doan, R. Ramakrishnan, and A. Y. Halevy, “Crowdsourcing systems
on the world-wide web,” Communications of the ACM, vol. 54, no. 4,
pp. 86–96, 2011.
[18] J. Lin, “Context-aware task allocation for distributed agile team,” in Proceedings of the 28th IEEE/ACM International Conference on AutomatedSoftware Engineering (ASE’13), 2013, pp. 758–761.
[19] H. Yu, Z. Shen, C. Miao, and B. An, “A reputation-aware decisionmaking approach for improving the efficiency of crowdsourcing systems,” in Proceedings of the 12th International Conference on Autonomous
Agents and Multi-agent Systems (AAMAS’13), 2013, pp. 1315–1316.
[20] H. Yu, C. Miao, B. An, C. Leung, and V. R. Lesser, “A reputation management approach for resource constrained trustee agents,” inProceedings of the 23rd International Joint Conference on Artificial
Intelligence (IJCAI’13), 2013, pp. 418–424.
[21] H. Yu, C. Miao, Z. Shen, C. Leung, Y. Chen, and Q. Yang, “Efficient
task sub-delegation for crowdsourcing,” in Proceedings of the 29th AAAI
Conference on Artificial Intelligence (AAAI-15), 2015, pp. 1305–1311.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

160

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

161

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

162

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

163

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

164

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

165

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

166

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

167

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

168

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

Understanding the Factors that Influence the
Implemented New Product Ideas in Organization’s
Crowdsourcing Communities
Qian Liu

Bao Wen Sun

China Center for Internet Economy ResearchCentral University of Finance and Economic
Beijing, China
liuqian00_china@163.com

China Center for Internet Economy ResearchCentral University of Finance and Economic
Beijing, China
sunbaowen@263.com

Abstract—In organization’s crowdsourcing communities, users
share new product ideas as co-creators of value. This has becomea new driving force of organizations in R&D and innovation, andthe key for organizations to win competition. However, the priorresearches have limited knowledge about users’ individual
ideation efforts and performance as ideators. From theperspective of ideators, this paper introduces Elaboration
likelihood model and explores the influence of individual
experience and textual characteristics of ideas on theimplementation of new product ideas. According to the study,
there is an inverted U-shaped relationship between the past
successful experience and the implemented ideas; past submitting
experience negatively influences the implementation of ideas; thelength of ideas negatively influences the implemented ideas; therichness of ideas positively influences the implementation of ideas.
In the end, the theory and practical application of the researchconclusions are discussed.

At present, researches on generating new product ideas
through organization’s crowdsourcing communities are still at
initial stage and most of them are only exploratory studies.
Despite the benefits of the relevant research results fororganizations and academic fields to better understand theimportance of organization’s crowdsourcing communities ingenerating new product ideas, these studies are limited in
exploring the most basic but the most important individual’s
ideation efforts and performance since they are conducted
only from the macroscopic angle of organizations.

Keywords—Crowdsourcing; Ideation; Past Success; Richnessof Ideas1

I.

INTRODUCTIONEnterprises use Web2.0 technology to establish
collaborative crowdsourcing-based open innovation platform
in order to obtain user-generated ideas by encouraging
interaction between enterprises and users, and among users [1,2]. As value creators, users submit ideas which promote thedevelopment and innovation of companies’ products [3, 4]. As
enterprises integrate users’ ideas to achieve open innovation of
product development, the quality of users ’ ideas directly
affects the innovation of product research and development. If
the user is unable to continue to provide high-quality ideas,
open innovation platform established by enterprises will exist
in name only. Therefore, company managers have faced apractical problem that is how to take effective measures toimprove the value of users’ ideas to apply in the process of
research and development of new products.

1. National Natural Science Foundation of China (Grant No. 71272234;
71402136)

This paper, focusing on the influential factors of
implementing new product ideas, explores the effects ofideators’ individual experience and characteristics of ideas on
the implementation of ideas. According to Elaboration
Likelihood Model (ELM), we integrate the influence ofindividual experience and textual features of ideas on theimplementation of ideas. First, we choose past success as theindex of depth of individual experience and choose past
submitting experience as the index of breadth to explore theinverted U-shaped relationship between the past success andthe implementation of ideas and the negative influence of past
submitting experience on implementing the ideas. Next, wechoose the length of ideas as the structural feature of ideas andthe richness of ideas as the content feature to explore thenegative influence of the length of ideas and the positiveeffects of ideas’ richness on the implementation of ideas.
The theoretical contributions of this paper are
demonstrated from the following three aspects. First, we haveproposed an inverted U-shaped non-linear relationshipbetween past success and the implementation of new productideas. This offers a comprehensive theoretical explanation for
the relationship between the ideators’ past success and theimplementation of new product ideas. Second, we haveintroduced basic characteristics of the structure and content ofideas’ text, and revealed the negative influence of the length ofideas and positive effects of the richness of ideas on theimplementation of ideas. In this way, we have improved theprevious studies which lack of discussion about the influenceof textual features and offer reference for users’ generatingideas. Third, we have integrated the influence of individual
experience and textual characteristics of ideas on theimplementation of ideas. Thus, we have improved the prior© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

169

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

researches which only focus on characteristics of individuals
or features of ideas.
II.

LITERATURE REVIEWNew product ideas refer to proposed new ideas or solutions
by users to meet a company’s need for innovation of new
products. On the crowdsourcing platforms, users respond on
the voluntary basis to the company’s call for new productideas generation and propose a large amount of ideas which
are related to functions and types of products and containusers’ indications of interest and defects of products. However,
due to ideators’ inadequate professional knowledge, theproposed ideas may not be detailed or innovative enough or
perfect in certain aspects. Besides, the submitted ideas may
have been already known by internal staff or be identical withthose generated by other ideators. For a company, it wants not
only innovative (relatively more novel than the existing ones)[5, 6], but also of potential commercial value to the company
with reasonable implementing costs from the short- and longterm perspectives [5-8]. Such ideas will be accepted by thecompany and then applied to its research and development of
new products. On the contrary, ideas featuring poor novelty,
little commercial value and high costs will not be accepted or
implemented by the company. Thus, the implementation of
new product ideas, as an objective outcome indicator, canevaluate the innovativeness, commercial value andimplementing costs of the ideas.
In the situation of organizations’ crowdsourcing
communities, empirical researches related to theimplementation of new product ideas are still at the starting
stage. Some researches have studied the implementation ofideas from the perspective of individual experience and thecharacteristics of ideas. Though with certain achievements,
those studies still have their limitations. Bayus (2013) [9] fromthe view of ideators’ individual experience, has proposed thatpast success poses negative effect to the implementation of
new product ideas under the guideline of cognitive fixation
theory. However, numerous past successes should provide
ideators with adequate information and knowledge so as toimprove their individual innovative capacity [10]. But why dothey have impact on enhancing their performance ininnovation? Thus we think there remain controversial points
on the issue of possible negative effects of past success on theimplementation of ideas. Therefore, further exploration isneeded.
Based on characteristics of ideas, there have already beenstudies discussing the effects of characteristics of ideas on theimplementation of the ideas. For instance, Shin et al.(2012)[11]
focused on the issue “ What kind of new product ideas will
be accepted or implemented by organizations orenterprises?” According to the diffusion of innovation theory,
they proposed that the relative advantage, appropriability,
market value, return on investment, development of a newcapability, and compatibility of the ideas have a positive effecton organizations’ decision to accept the ideas. Hossain andIslam(2015) [12] through analyzing the data from Starbucks’
My Starbusk Idea, divided new product ideas into Linked
ideas and Sole ideas in view of whether the ideas are linked toother ones. In their research, they proved that Sole ideas are
more quickly accepted and implemented by organizations than
Linked ideas only when the former cost more time, gain morecomments and accumulated points than the latter.
In conclusion, the prior studies only start from a single
aspect such as individual experience or idea features to discuss
the idea implementation; There remain controversial points inthe negative relationship between ideators’ early success andthe implementation of ideas. The relevant researches on idea
characteristics have overlooked the basic role of the structuraland content characteristics of the objective idea text
implementing the new product ideas. Thus the future studies
can improve the existing studies from the mentioned three
aspects.
III.

THEORETICAL FRAMEWORKThe ELM is introduced to explore the influence factors of
the implementation of new product ideas. ELM, a dualprocess theory on the formation and changes of individual
attitude, was proposed by social psychologist Petty andCacioppo in 1986 through concluding multiple relevant
theories on the formation and changes of individual attitudeand cognition [13]. ELM explains the strategies of influencing
an individual to alter his attitude towards specific objectives,
events or behaviors, and the relative effects they havecompared with each other [14]. In ELM, central route andperipheral route together shape an individual’s attitude
towards specific objectives, events, or behaviors. During
information processing, the central route and peripheral route
require different individual efforts in thinking and varying
degrees of the elaboration of his cognition. More specifically,
central route requires the individual to actively think about theopinions and contents of the information and comprehensively
understand the quality and relevant characteristics of theargument before he has an judgment to the specific objective.
However, when the specific objective doesn’t interest theindividual, he will not think deeply about the opinions andcontents of the information, thus making relatively lesscognition efforts. In this case, the individual will rely on
peripheral route to make a judgment, which means theindividual subjectively depends on peripheral characteristics
such as the features of the information promulgator to form his
attitude towards a specific objective. In the researches
conducted by Bhattacherjee (2006) [15] and Tam and Ho(2005) [16], the central route and peripheral route are studied
respectively to attain the argument quality and the reliabilityof the information resources which jointly influence theindividual’s attitude towards a specific objective.
Under the framework of source credibility and argument
quality of ELM, this paper integrates the characteristics ofindividual experience with innovative texts to prove their joint
influential effects on the implementation of new product ideas[15, 16]. In this paper, the source credibility is regarded as
individual experience and ideators’ early success and past
submitting experience are counted respectively as the depth
and breadth of individual experience. The argument quality is
operationalized as the feature of ideas and we choose thelength of ideas as the structural features and the richness ofideas as the content features of ideas’ texts. Jonassen (1993)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

170

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)[17], an educational psychologist believes that information or
knowledge generally has two major characteristics: thestructural and the content feature. The experts in organizations
read, evaluate and comment on a new product idea which is aprocess of dealing with text information, thus the structuraland content characteristics of the new product idea influence
experts’ attitudes towards and decisions to implement the idea.
A. Individual Experience and the Implementation of New
Product Ideas
Past success is defined as the cumulative number of ideas
proposed by an individual and implemented by companies [9,18], thus it is an indicator of the depth of individual
experience. Prior researches have different opinions andconclusions on the relation between the past success andindividual innovation performance. On one hand, researches
on individual innovation indicate that past success promotes
the individual innovation performance mainly because past
success contributes to enhancing individual creative selfefficacy [19], stimulating an individual’ s intrinsic motivation[20], and accumulating rich knowledge or experience so as topositively influence an individual’ s innovation performance.
On the other hand, some researches suggest that past successdecreases individual innovation performance due to cognitive
fixation.
Experimental researches confirm cognitive fixation andfind that an individual tends to fixate on the principles andfeatures of prior examples, leading to ideas that are lessoriginal. Cognitive fixation is individuals’ unconscious
plagiarism since they are often unaware that they are fixatingon the characteristics of past examples [21-23]. A series ofstudies [24-26] demonstrate that individuals have great
difficulty in deviating from previously successful problem
solving strategies even when a problem requires a new
solution approach. Therefore, due to cognitive fixation, past
success limits individuals’ knowledge and heuristics appliedin ideation process, causing their lower innovation
performance.
Experimental studies on new product ideas confirm thatcognitive fixation is detrimental to future ideation efforts. For
example, Jansson and Smith (1991) [21] have demonstrated
the fixation effects in the product design process. They foundthat due to incomplete exploration of knowledge by
individuals in the fixation group who received an example of a
possible design, their design solutions were much more likely
to include features of the example designs, leading to lesscreativity or commercial value of their design solutions. Audia
and Goncalo (2007) [27] have proved that individuals withrich successful experience are likely to generate ideas withknowledge or models they are familiar with, thus they put
forward new product ideas with lack of diversity. Bayue (2013)[9] has confirmed that ideators’ past success has negative
effects on the rate of ideas implemented by companies throughstudying the data from Dell’s IdeaStorm. Bayus (2010) [18]further applied the data of Dell’s IdeaStorm and proved thatpast success reduces ideators’ likelihood of generating andsubmitting ideas. The above experimental studies ignored thetheoretic logic of the positive effects from past success on
generating implemented ideas. Therefore, we believe that therelation between past success and the implementation of ideas
is debatable and requires further analysis and argumentation.
This paper proposes an inverted U-shaped non-linear
relationship between past success and the implementation of
new product ideas. Ideators are major information source of
new product ideas thus their past success demonstrates thereliability of the information source. According to specificanalysis, for ideators at the early stage of accumulating
successful experience, the whole process of generating,submitting, discussing, revising, and improving ideas to be
implemented by companies not only helps them gain more
knowledge or experience necessary for new productinnovation [10], but also enhances ideators’ innovative selfefficacy [19] and stimulates their intrinsic motivations [20] sothat they are more likely to spend more time and energy on
new product ideas, making ideas valuable enough to be
implemented by companies. However, for ideators at the later
period of accumulating successful experience, their knowledge
and experience from prior success are gradually fixated. Dueto cognitive fixation, ideators tend to refer to previous
successful mode [28, 29], leading to lower creativity andcommercial value of their ideas which thus are less likely to be
implemented by companies [22, 30]. Based on ELM, we
believe that past success as information credibility influencescompanies’ evaluation on and implementation of ideas. Morespecifically, with the accumulation of prior success, there is aninverted U-shaped relationship between ideas submitted byideators and those implemented by companies. In other words,
the rate of implemented ideas firstly rises and then drops
gradually as past success increases.
Hypothesis 1: An inverted U-shaped relationship between
past success and the implementation of new product ideas
Past submitting experience, as a breadth indicator ofindividual experience, refers to the number of new productideas previously submitted by ideators [18]. As a characteristicof information source, past submission is negatively related togenerating implemented ideas. With a growing number of
their submitted ideas, ideators are increasingly familiar withthe features and demands of generating new product ideas,
gradually forming a habitual thinking mode. The fixed
structural and similar content features of a text reduces
ideators’ likelihood of proposing ideas with novelty andcommercial value. Ford (1996) [31] put forward that
constantly cumulative working experience allows employees
to be excessively familiar with work tasks so that they tend to
propose solutions with less novelty and originality by applying
fixed thinking mode or working methods. Based on ELM, we
believe that past submitting experience, as informationcredibility, is associated with experts’ value judgment ofideas and decision-making of implementing them. Therefore,
in this paper, we propose that ideators’ past submitting
experience is negatively related to the implementation of ideas
by companies.
Hypothesis 2: Past submitting experience is negatively
related to the implementation of new product ideas.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

171

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

B. Characteristics and Implementation of New Product Ideas
The length of ideas, as the structural characteristic of anidea’s text, refers to the sum of words in a text of a new
product idea. As argument quality, length of ideas determines
the value of an idea, negatively influencing experts’ decision
to implement the idea. Prior studies confirm that the length ofideas is in direct proportion to the amount of information. Thatis, the longer text contains more information values. Under the
background of knowledge sharing community, Wang et al.
(2015) [32] put forward that the number of words in a thread
is an important feature of argument quality and a longer threadis the information of greater value. Ransbotham et al. (2012)[33], against the background of wikipedia knowledge forum,
confirm that the value of informative articles might lie in morerather than better information they contain. Therefore, priorresearch shows that the length of knowledge information is the
key feature of argument quality with more knowledge
information indicating higher corresponding value. However,different from the integrity of knowledge information, mostInternet information features fragmentization, large amount,timeliness [34]. Thus redundant information goes against
ideators’ fast and accurate reading and comprehending, also
impedes the extensive spread of information. Susarla et al.
(2012) [35] demonstrate a negative correlation between thelength and the spreading breadth of a video based on YouTube.
In crowdsourcing communities of organizations, a large
number of ideators continuously submit lots of ideas.
Confronted with the complex selecting task, experts arehindered to understand the content of the submitted ideas
quickly and correctly by too long textual information,
resulting in lower rate of accepted ideas. Therefore, we thinkthe length of ideas is negatively associated with the possibilityof ideas to be implemented by organizations.
Hypothesis 3: The Length of ideas is negatively related tothe implementation of new product ideas
The richness of an idea, as the content characteristic, refers
to the sum of multimedia argument materials in the new
product idea. Multimedia materials, including pictures, videos,inter-linkages, and attachments, are argument or referencematerials added by ideators for content demonstration of new
product ideas. The richness of an idea, as the quality of anargument, determines the value of a new product idea andpositively influences organizations’ decision-making in
implementing the idea. In other words, the richness of pictures,videos, inter-linkages, and attachments in argument materials
determines the value of an idea to a large degree. Pictures andvideos demonstrate what a new product idea contains indiversified forms, enhancing the readability and quality of theidea. Besides, inter-linkages and attachments provides
adequate argument materials for a new product idea,
increasing the reliability and quality of the idea. Wang et al.
(2012) [32] based on knowledge sharing community,
demonstrate that the interlinkage of a thread is a key
characteristic of argument quality, and a thread containing aninterlinkage is more valuable information. Ransbotham (2012)[33] against the background of wikipedia knowledge forum,
prove that texts with more external references and internalinter-linkages are of greater value. Susarla et al. (2012) [35]verifies that videos with more inter-linkages are more widely

spread. When experts of an organization process theinformation of new product ideas, pictures and videos can help
them actively think about opinions and content expressed by
the idea while inter-linkages and attachments facilitate themfully understand the quality and relevant features of thearguments, thus ideas with diverse pictures, videos, interlinkages, and attachments are of higher quality and greater
possibility of being accepted by organizations. In this case, we
propose that the richness of ideas is positively related to thelikelihood of a new product idea to be implemented byorganizations.
Hypothesis 4: Richness of new product idea is positively
correlated with implementation of new product idea
IV.

RESEARCH METHODOLOGY

A. Data Source
To validate the research model and hypothesis, we collectdata from MIUI forum (http://www.miui.com/forum.php)
established and operated by Beijing Xiaomi Technology Inc.
(hereinafter referred to as Xiaomi Company). MIUI forum
introduces “enthusiastic fans”(users) into the R&D mode of
products to gain ideas of new functions and suggestions for
improvement from users. A section for discussing suggestions
on new functions and one for bug feedback are set on MIUI
forum to collect, screen, and integrate users’ ideas. The
discussing section on new functions suggests users to releaseideas on optimizing or adding functions of the operating
system of MIUI phones （ http://www.miui.com/forum-381.html）. The bug feedback section advises users to submit
any defect and problem in using and testing MIUI’s operating
system of various versions. As of February, 2016, the users
have submitted 127，085 ideas on new functions and 264184
bug feedback opinions [36].
B. Data Collection
Two rounds of data collection are operated in this research.
The first round is to select sample users who meet therequirements of this study while the second is to obtain therelevant data of new product ideas submitted by sample users.
The first round of data collection aims at selecting sample
users who belong to “MIUI Honorary Moderator”, startingfrom April 15th to 29th in 2015. Two means are jointly
applied in this study to trace the members of MIUI Honorary
Moderator: one is by looking through the users as well as theiridentities present in the new threads and the responds on thetreads; the other is by looking for other members of the MIUI
Honorary Moderator in the friend circle of the members found.
In the above two ways, researchers have collected 244
members of MIUI Honorary Moderator. In collecting data, wehave also recorded the names, linkage of their personal zone,
registering time and other information of these member
ideators of MIUI Honorary Moderator.
The second round of data collection aims at collecting thedata information of threads on new product ideas submitted byusers of MIUI Honorary Moderator, starting from May 4th toJuly 30th in 2015. Firstly, researchers select threads on new© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

172

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

product ideas (suggestions on new functions or feedback ofbugs) from the topic lists in the personal space of sample users.
Among those users, ten set protection for their personal zone
so that their data information of submitted new product ideas
was unavailable, thus these ten users are removed in this round.
Secondly, we collect the data information of new productideas, including the releasing time, linkages, titles, number ofviews, number of replies, grades, text content, length of ideas,
photos, videos, attachments and so on. Note that the number ofreviews and applies, grades and all the information of replies
change over time. The deadline of submitting new productideas was set on March 30th in 2015 by the researchers, andall the data information of new product ideas are collected amonth after those ideas are released. The data informationsuch as views varies little, for threads submitted a month ago
will become ‘Outdated posts’ in the vast number of updata.
These two rounds of data collecting were done by two
trained researchers who coordinately examine each other’s
missing or error information to make sure the objectivity andaccuracy to the large degree. Artificial screening and computer
technology are applied in the two rounds of data collecting toensure the accuracy and objectivity of the information.
C. Descriptive Analysis
The sample of this research is adopted from the MIUI
Honorary Moderator. MIUI Honorary Moderator, whose
nickname is ‘Rong Zuer’, is the core experienced users groupwhich is established as the earliest in MIUI Forum. The maintasks of MIUI Honorary Moderator are submitting feedbackon and testing recurrent problems and offering suggestions of
new products functions.
TABLE 1. STATISTICAL DESCRIPTIONName of Variables

Mean
Value

Standard
Deviation

Minimum
Value

Maximum
Value

0.43

0.49

0

2

25.13

31.09

0

205

60.01

72.22

0

371

87.61
0.71

157.05
1.3

0
0

7331
29

Implemented New
Product Ideas
Past successful
experience
Past Submitting
Experience
Length of Ideas
Richness of Ideas

TABLE 2. CORRELATIONS OF VARIABLES
Name of Variables
Implemented New
Product Ideas
Past successful
experience
Past Submitting
Experience
Length of Ideas
Richness of Ideas

1

2

3

4

5

1
0.177*

1

1-0.001

1-0.057*0.418*

0.004
0.119**-0.0850.005

1-0.07*
0.001

The samples in this study are the 234 users from MIUI
Honorary Moderator whose registration time is ranged fromthe year 2010 to 2014. There are 13,661 new product ideas

sampled in this study. The submitting time of these ideas
range from the year 2010 to 2015 .
The descriptive statistics of the variables used in ourempirical model are given in Table1. Table 2 shows thecorrelations between model variables.
V. EMPIRICAL STUDY
A.

Variables and Operationalization
This study involves one dependent variable, four
independent variables, and five control variables, as shown intable 4. Among them, the innovation implementation of new
product ideas obtains data through content analysis.
1) Dependent Variable
The dependent variable of this study is theimplementation of the new product ideas, which is theobjective result of the comprehensive measurement of thecreative value [37]. We use content analysis method todetermine whether the new product ideas are adopted byenterprises. Content analysis can translate the meaning of thetext into objective data to ensure the objectivity,
reproducibility and reliability of the data analysis [38-41].
This paper uses the method of content analysis based on
encoding content and step analysis proposed by Krippendorff(1989) [42], Landis and Koch (1977) [43] to conduct
operation and measurement of variables. To ensure that every
content-analysis encoded of by two coders at the same time,
this study employs 4 graduates engaging in content encodingwith a group of two people and a total of two groups involved.
In this paper, the author doesn’t participate in the contentcoding work. And the author does not allow the four coding
workers to communicate with each other in the coding process.
In addition, we give each coder enough time to complete thework, and limit their work time to no more than eight hours,
maximum reducing coding errors caused by fatigue.
Prior to the formal large-scale coding, this study firstly
uses partial data for precoding analysis. The goal of theprecoding analysis is "to revise the coding versions before
formal coding by repeated coding, to reduce or eliminate themisunderstanding errors between the coders. If the precoding
results do not reach the critical value of 0.70 Kappa, the author
will organize four coders to discuss coding results that don’treach agreement. Continue discussions between the author andthe encoders by revising disagreement in coding rules, in orderto reach an agreement in coding understandings informal
encoding. Inter coder reliability (inter-coder armis) is
calculated by the percentage of consistent variables. The
results show that the value is between 0.70 and 0.97,
indicating that all the coding results of variables have highcredibility [42, 43].
This study adopts the two objective criteria judging
whether the ideas of a new product are implemented byenterprises: one is the official sealant the top right of the poston new product ideas; the second is content significance of theratings and replies by the enterprise experts (MIUI
Development Group). The meaning of the official seal, or thesignificance of content of corporate experts’ reviews, can© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

173

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

demonstrate the new product ideas have been implemented by
the enterprises, assigned to the 1". Otherwise, if the twostandards are not met, it is believed that the new product ideas
are not implemented by the enterprise, assigned to be 0".
2) Independent Variable
Past successful experience is defined as the number of
adopted and implemented new product ideas submitted by theideator in earlier stage. Operationalization of past successful
experience is defined as number of adopted and implemented
new product ideas submitted by the ideator before day t. Past
submission experience is defined as number of new productideas submitted by the ideator in earlier stage.
Operationalization of past submissions defined as number of
new product ideas submitted by the ideator before day t.
Length of ideas is defined as the length of the text of the new
product ideas, which can calculated through the use ofstatistical software.Richness of ideas is defined as the richnessof multimedia documents in text of the new product ideas, orthe sum of the number of pictures, videos, attachments andlinks in practical operation.
B. Results
Because the implementation of new product ideas
involving dependent variables is binary variable, we construct
the Logit Model to verify the research hypothesis about the
impact of individual experience and creative text
characteristics on the implementation of ideas.
The research model formula of the influence ofindividual experience and creative text feature on theimplementation of ideas:
Logit (Implementaton of the Idea) =β 0 Past successful experience+β 1 Past successful experience2 +β 3 Past
submission experience+β 4 Length of ideas +β 5 Richness of ideas +β 6
Control variables + ε
Notes: (1) the control variables include community tenure experience,
popularity of ideas (views), attention of ideas (reviews and scores), dummy
variables of years, dummy variables of category.
(2) The effect of the corresponding research variables on theimplementation of new product ideas(3) The error term, used to control the non-observable heterogeneity

Table 3 shows significant likelihood ratio test of theresearch model on the influence of individual experience andcreative text feature on the implementation of ideas, and themodel is well set up. Pseudo R2 value of model 1 is 0.1184.
There is a significant inverted U-shaped relationship between
the past successful experience and the implementation of ideas(the past successful experience β=1.21 P<0.001 ， past
successful experience*past successful experience β= -0.09
P<0.001), so the hypothesis 1 is verified. Past submission(β= 0.79 P<0.001） is negatively correlated with implementationof ideas, so the hypothesis 2 is validated. Length of ideas （β=-0.05 P<0.05）is negatively correlated with implementation,
so the hypothesis 3 is validated. Richness of ideas（β=0.43
P<0.001）is positively correlated with the implementation ofideas, so the hypothesis 4 is validated.

TABLE 3. THE INFLUENCE OF INDIVIDUAL EXPERIENCE AND CREATIVE TEXTFEATURE ON THE IMPLEMENTATION OF IDEAS

Variables
Model
Past successful experience
1.21***
Past successful experience2-0.09***
Past submission experience-0.79***
Length of ideas-0.05*
Richness of ideas
0.43***
Control variables
Community tenure experience
0.1**-0.07**
Popularity of ideas：views
0.34***
Attention to ideas：reviews
0.24***
Attention to idea：ratings
Dummy variables of years
included
Dummy variables of category
includedLog-likelihood-8035.607
X2(df)
2159.3***（85）Pseudo R2
0.1184N13350
notes：* stands for P < 0.05,， ** for P < 0.01，*** for P < 0.001

VI. DISCUSSION
Based on the detailed and possible model framework, thispaper integrates the individual experience and creative
features to explore the effect of the two factors on theimplementation of the new product ideas. The main
conclusions are as follows. There is an inverted U-shaped
relationship between the ideator's past successful experienceand quality of new product ideas, namely when the ideator isin early success, its innovative ability and innovative
performance continues to rise, and subsequent submission of
new product ideas are of much novelty and high commercial
value; however, when the ideator is in late success, they aretrapped in the existing successful model, so the individual
innovative ability and innovative performance decrease
gradually and subsequent submission of new product ideas areof less novelty and commercial value. The ideator's past
submission experience negatively affect the quality of new
product ideas, that is to say, frequent submission of new
product ideas makes the ideator less creative and sensitive,
thus weakening the novelty and diversity of quality of new
product ideas. This paper verifies that new product ideas withhigh quality are concise and comprehensive, diverse in forms
and well-supported. Lengthy new product ideas are not helpful
for corporate experts and ideators to quickly and accurately
read and comprehend; pictures and videos vividly and directly
display the content of new product ideas; and if the idea for a
new product contains sufficient information to demonstratethat (attachments and links) that the content of new productideas are true and reliable, so the idea has a high quality.
A.

Implications for Theory
The following theoretical meanings are embodied in thispaper:First, we have proposed in this paper an inverted Ushaped non-linear relationship between past success and theimplementation of new product ideas, and analyzed if there is© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

174

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)a negative or positive relationship between ideators’ past
success and the implementation of new product ideas
concluded by prior studies. The previous studies demonstratethat ideators’ past success negatively influences theimplementation of their new product ideas due to cognitive
fixation effects. However, individual innovation studies
proposes that ideators’ prior success helps produce individual
innovative self-efficacy and enrich their knowledge andexperience, thus contributing to their innovative performance.
To explain these controversial results, we put forward an
inverted U-shaped non-linear relationship between ideators’
past success and the implementation of new product ideas.
More specifically, ideators’ past success produces innovative
self-efficacy for them in the early stage so that the quality ofideas are increasing; however, when the past success isaccumulated to a certain degree, ideators’ are likely to rely on
the mode of previous success so that the quality of ideas are
decreasing. The inverted U-shaped relationship developed theprior studies and offers comprehensive theoretical explanation
for the relationship between ideators’ past success and theimplementation of their ideas.
Second, we have introduced basic structural and content
characteristics of ideas’ text, indicating that the length andrichness of ideas respectively have negative and positive
influences on the implementation of ideas, which may be
applied by ideators for reference. Prior researches only notices
influence of the content features subjectively assessed byexperts (e.g., relative advantages and compatibility) andobjective classifying features of ideas on the implementationof ideas, ignoring the basic textual features of ideas. To offset
the previous studies, this paper has introduced the structuraland content features of the ideas’ text as the argument quality,and explored their influence on the implementation of ideas.
Third, based on ELM, we have integrated the influence ofindividual experience and textual characteristics of ideas on
the implementation of ideas, which makes up the one-sided
prior researches which only focus on individual features orfeatures of ideas. The literature review demonstrates that priorstudies have explored influential factors to the implementationof new product ideas from the perspective of individualfeatures or characteristics of ideas. However, as the co-creatorof value, ideators generate and share new product ideas,
therefore, ideators’ characteristics coexist with thecharacteristics of ideas and they jointly influence the value ofideas generated by ideators. In other words, “thecharacteristics of ideators and their generated ideas directly
influence whether the ideas are valuable enough to be
implemented by companies.” Therefore, based on ELM, thispaper constructs a theoretical model that individual experienceand textual features of ideas have influence on theimplementation of ideas.
B.

Implications for Practice
The conclusion of this study demonstrates an inverted Ushaped relationship between ideator’ prior success and theimplementation of new product ideas, and proved that with theaccumulation of successful experience, individual innovative
performance increases in the first period and decreases in thelate period. Thus, organizations should attach great importanceto the ideators whose prior success is increasing and in peak,
and take encouraging measures to fully unleash these ideators’potential in innovation. For instance, organizations may allow
them to have access to professional materials related to
products, invite them to participate in internal R&D activities
on ideas, open the privilege to test the latest version of
products, etc. Although the innovative performance of ideatorswith rich successful experience is gradually decreasing, theideators have become professional in the product because of
their rich product information and professional knowledge
accumulated from their past success. Organizations may
encourage these ideators to play a role in contributing
knowledge, answering questions, training, etc. in generating
new product ideas, facilitating other ideators to improve their
new product ideas. For example, organizations may award
them honorary titles (“senior member”, “supervisor”, etc.),
and invite them to take tasks of some internal experts
including training new ideators, answering questions,commenting and selecting ideas. Experts and managers in acompany may take different measures to fully discover thecreative and knowledge advantages of ideators who are indifferent phases of successful experience.
According to the conclusion, the past submitting
experience negatively influences the implementation of new
product ideas, demonstrating a gradually declining trend of
individual innovative performance with the accumulation ofworkload experience. Thus, organizations should realize that
excessive working experience will consume ideators’ passion
and capacity in innovation, so they shouldn’t excessively
focus on the quantity of new product ideas submitted byideators but take measures to encourage ideators to value thequality of each idea. For example, organizations are suggested
to promote the accepting rate of new product ideas ( thequantity of accepted new product ideas/ the total of new
product ideas) as the evaluating requirements for ideators’
upgrading, enhancing ideators’ awareness of the relationshipbetween the quantity and the quality of new product ideas.
The results demonstrate that the length of ideas
negatively influences the implementation of ideas; however,
the richness of ideas positively influences the implementationof ideas. Thus it is indicated that concise and brief new
product ideas with images and adequate arguments havehigher values and higher likelihood of being implemented byorganizations. Therefore, organizations should release
instructive opinions on generating new product ideas andencourage ideators to illustrate their ideas with certain words,
present the ideas with images, videos, etc., and back the ideas
through fully applying the system running materials andexternal materials.
C. Limitations
There exist some inevitable limitations in this research.
Firstly, new product ideas in other field are sampled to verify
this research conclusion. Secondly, the universality of users
adopted in this research is limited. Thirdly, the effects ofunobservable factors on the implementation of new productideas are not considered in this paper.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

175

Proceedings of the 2016 International Conference on Crowd Science and Engineering (ICCSE 2016)

REFERENCES[1] T. O'Reilly, "What is Web 2.0? Design patterns and business models forthe next generation of software," Communication & Strategies, vol. 1, pp. 1737, 2007.
[2] R. Hoegg, R. Martignoni, M. Meckel, and Katarina, "Overview ofbusiness models for web 2.0 communities," Proceedings of GeNeMe, pp. 2337, 2006.
[3] S. L. Vargo and R. F. Lusch, "Evolving to a new dominant logic formarketing," Journal of Marketing, vol. 68, pp. 1-17, 2004.
[4] D. Zwick, S. K. Bonsu and A. Darmody, "Putting consumers to work:"Co-creation" and new marketing govern-mentality," Journal of Consumer
Culture, vol. 8, pp. 163-196, 2008-07-01 2008.
[5] T. M. Amabile, Creativity in context. Boulder: Westview Press, 1996.
[6] J. M. George, J. H. Jones and J. Zhou, "Dual tuning in a supportive
context: Joint contributions of positive mood, negative mood, and supervisory
behaviors to employee creativity," Academy of Management Journal, vol. 50,
pp. 605-622, 2007.
[7] J. E. Burroughs, C. P. Moreau and D. G. Mick, "Toward a psychology of
consumer creativity," Handbook of Consumer Psychology, pp. 1011-1038,
2008.
[8] C. E. Shalley, J. Zhou and G. R. Oldham, "The effects of personal andcontextual characteristics on creativity: Where should we go from here?"
Journal of Management, vol. 30, pp. 933-958, 2004.
[9] B. L. Bayus, "Crowdsourcing new product ideas over time: An analysis of
the Dell IdeaStorm community," Management Science, vol. 59, pp. 226-244,
2013.
[10] S. J. Shin, T. Y. Kim, J. Y. Lee, and L. Bian, "Cognitive team diversity
and individual team member creativity: A cross-level interaction," Academy ofManagement Journal, vol. 55, pp. 197-212, 2012-02-01 2012.
[11] P. M. Di Gangi and M. Wasko, "Steal my idea! Organizational adoptionof user innovations from a user innovation community: A case study of Dell
IdeaStorm," Decision Support Systems, vol. 48, pp. 303-312, 2009-12-01 2009.
[12] M. Hossain and K. M. Z. Islam, "Ideation through online open innovation
platform: Dell IdeaStorm," Journal of the Knowledge Economy, vol. 6, pp.
611-624, 2015.
[13] R. E. Petty and J. T. Cacioppo, Communication and persuasion. central
and peripheral routes to attitude change. New York: Springer-Verlag, 1986.
[14] A. Bhattacherjee, Social science research: Principles, methods, andpractices: Global Text Project, 2012.
[15] A. Bhattacherjee and C. Sanford, "Influence processes for information
technology acceptance: An elaboration likelihood model," MIS Quarterly, vol.
30, pp. 805-825, 2006.
[16] K. Y. Tam and S. Y. Ho, "Web personalization as a persuasion strategy:
An elaboration likelihood model perspective," Information Systems Research,
vol. 16, pp. 271-291, 2005.
[17] D. H. Jonassen, K. Beissner and M. Yacci, "Structural knowledge :Techniques for representing, conveying, and acquiring structural knowledge,"
Erlbaum Associates Publishers Hove, pp. 125-131, 1993.
[18] B. L. Bayus, "Crowdsourcing and individual creativity over time: The
detrimental effects of past success," Ssm Electronic Journal, 2010.
[19] P. Tikrney and S. M. Farmer, "Creative self-efficacy: Its potential
antecedents and relationship to creative performance," Academy ofManagement Journal, vol. 45, pp. 1137-1148, 2002.
[20] A. M. Grant and J. W. Berry, "The necessity of others is the mother of
invention: Intrinsic and prosocial motivations, perspective taking, andcreativity," Academy of Management Journal, vol. 54, pp. 73-96, 2011.
[21] D. G. Jansson and S. M. Smith, "Design fixation," Design Studies, vol.
12, pp. 3-11, 1991.
[22] S. M. Smith, T. B. Ward and J. S. Schumacher, "Constraining effects ofexamples in a creative generation task," Memory & Cognition, vol. 21, pp.

837-845, 1993.
[23] C. Cardoso and P. Badke-Schaub, "Fixation or inspiration: Creative
problem solving in design," The Journal of Creative Behavior, vol. 45, pp. 7782, 2011.
[24] N. R. F. Maier, "Reasoning in humans," Journal of Comparative
Psychology, vol. 12, pp. 181-194, 1931.
[25] K. Duncker and L. S. Lees, "On problem solving," Psychological
Monographs, vol. 58, pp. 1-110, 1945.
[26] H. G. Birch and H. S. Rabinowitz, "The negative effect of previous
experience on productive thinking," Journal of Experimental Psychology, vol.
41, pp. 121-125, 1951.
[27] P. G. Audia and J. A. Goncalo, "Past success and creativity over time: A
study of inventors in the hard disk drive industry," Management Science, vol.
53, pp. 1-15, 2007.
[28] R. L. Marsh, T. B. Ward and J. D. Landau, "The inadvertent use of priorknowledge in a generative cognitive task," Memory & Cognition, vol. 27, pp.
94-105, 1999-01-01 1999.
[29] R. L. Marsh and J. D. Landau, " Item availability in cryptomnesia:
Assessing its role in two paradigms of unconscious plagiarism," Journal of
Experimental Psychology: Learning, Memory, and Cognition, vol. 21, pp.
1568-1582, 1995.
[30] R. L. Marsh, J. D. Landau and J. L. Hicks, "How examples may (and maynot) constrain creativity," Memory and Cognition, vol. 24, pp. 669-680, 1996.
[31] C. M. Ford, "Blood pressure and end-stage renal disease in men," New
England Journal of Medicine, vol. 334, pp. 13-18, 1996.
[32] G. A. Wang, H. J. Wang, J. Li, A. S. Abrahams, and W. Fan, "Ananalytical framework for understanding knowledge-sharing processes inonline Q&A communities," ACM Transactions on Management Information
Systems (TMIS), vol. 5, p. 18, 2015.
[33] S. Ransbotham, G. C. Kane and N. H. Lurie, "Network characteristics
and the value of collaborative user-generated content," Marketing Science, vol.
31, pp. 387-405, 2012.
[34] Y. Sun, "The characteristics and classification of information resourceson the Internet," Information and Documentation Services, vol. 02, pp. 24-25,
2002.
[35] A. Susarla, J. Oh and Y. Tan, "Social networks and the diffusion of usergenerated content: Evidence from YouTube," Information Systems Research,
vol. 23, pp. 23-41, 2012.
[36]MIUI, "Ideas and Buglist", 2016.
[37] E. E. Levitt, "Psychotherapy with children: A further evaluation,"
Behaviour Research and Therapy, vol. 1, pp. 45-51, 1963.
[38] P. A. Pavlou and A. Dimoka, "The nature and role of feedback text
comments in online marketplaces: Implications for trust building, pricepremiums, and seller differentiation," Information Systems Research, vol. 17,
pp. 392-414, 2006.
[39] A. S. Abrahams, J. Jiao, W. Fan, G. A. Wang, and Z. Zhang, "What's
buzzing in the blizzard of buzz? Automotive component isolation in social
media postings," Decision Support Systems, vol. 55, pp. 871-882, 2013.
[40] A. S. Abrahams, J. Jiao, G. A. Wang, and W. Fan, "Vehicle defectdiscovery from social media," Decision Support Systems, vol. 54, pp. 87-97,
2012.
[41] W. Fan, L. Wallace, S. Rich, and Z. Zhang, "Tapping the power of text
mining," Communications of the ACM, vol. 49, pp. 76-82, 2006.
[42] K. Krippendorff, "Content analysis," International Encyclopedia ofCommunication, vol. 1, pp. 403-407, 1989.
[43] J. R. Landis and G. G. Koch, "The measurement of observer agreement
for categorical data," Biometrics, pp. 159-174, 1977.

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved.

176

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Demo/Posters
Poster Session I
Date and Time: Wednesday, July 27, 2016, 12:00 PM-13:00 PM
Location: #1013, Earth Sciences Building, UBC
A Cost-Optimal Service Selection Approach for Collaborative Workflow Execution in Clouds
Yi Wei, Li Pan, Dong Yuan, Shijun Liu, Lei Wu and Xiangxu Meng
A Distributed Game-theoretic Approach for IaaS Service Trading in an Auction-based Cloud Market
YiWei, Li Pan, Dong Yuan, Shijun Liu, LeiWu and Xiangxu Meng
A Dynamic Priority based Profit Driven Scheduling Approach in Mobile Clouds
Shuo Zhang, Li Pan, Shijun Liu, Lei Wu and Xiangxu Meng
A Fast Shapelet Transform for Time Series Classification Based on Important Data Points
Cun Ji, Shijun Liu, Qingyang Wang, Yuliang Shi, Chenglei Yang, Lei Wu and Li Pan
A Fast Shapelet Transform for Time Series Classification with Limited Important Data Points
Cun Ji, Shijun Liu, Chenglei Yang, Lei Wu and Li Pan
A Fraud Resilient MedicalInsuranceClaimSystem
Yuliang Shi, Chenfei Sun, Qingzhong Li, Lizhen Cui and Han Yu
A Genetic Algorithm Based Data Replica Placement Strategy for Scientific Applications in Clouds
Lizhen Cui, Junhua Zhang, Lingxi Yue, Yuliang Shi, Hui Li and Dong Yuan
A neural network for Learning Spatiotemporal Models of Human Daily Activities
Shan Gao and Ah-Hwee Tan
A game based assessment tool for Parkinson’s Disease
Siyuan Liu, Zhiqi Shen, Martin J. McKeown, Cyril Leung and Chunyan Miao
A Social Curiosity Inspired Recommendation Model
Qiong Wu, Siyuan Liu, Chunyan Miao, Yuan Liu and Cyril Leung
Accuracy-Cost Tradeoff for Cloud-Based Ambient Assisted Living - Poster
Yi Dong, Yonggang Wen, Han Hu, Chunyan Miao and Cyril Leung
Activity Recognition using Kernel Embedding
Hangwei Qian, Sinno Jialin Pan and Chunyan Miao
AdaptScale: An adaptive data scaling controller for multiple performance requirements in cloud
Lei Liu, Wenbin Zhang, Yuliang Shi, Yongqing Zheng and Lizhen Cui
Airline Revenue Maximization
Haipeng Chen
© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved

196

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Demo/Posters
An approach to Performance Prediction for Parallel and Distributed Applications
Qingshi Shao, Li Pan, Shijun Liu and Lei Wu
An Automated Wandering Pattern Detection Algorithm for Person with Dementia (PwD)
Ashish Kumar, Chiew Tong Lau and Syin Chan
An Effective Hybrid Fraud Detection Method
Chenfei Sun, Qingzhong Li, Lizhen Cui, Zhongmin Yan and Hui Li
An Optimal and Iterative Pricing Model forMulticlass IaaS Cloud Services
Shuo Zhang, Li Pan, Dong Yuan, Shijun Liu and Lei Wu

Poster Session II
Date and Time: Thursday, July 28, 2016, 12:00 PM-13:00 PM
Location: #1013, Earth Sciences Building, UBC
An Incentive Mechanism for Consensus Tasks
Siyuan Liu, Xiuyi Fan and Chunyan Miao
An Overview of Crowdsourcing Approaches for MOOCs
Simon Fauvel, Han Yu and Chunyan Miao
BASKETBALL GENIUS FOR STROKE REHABILITATION
Yongmei Yuan, Chunyan Miao, Cyril Leung, Zhiqi Shen and Liang Zhang
Big datasets for research: A survey on flagship conferences
Yi Wei, Shijun Liu, Jiao Sun, Lizhen Cui, Li Pan, Lei Wu
Combining FineGrained Analysis and Scheduling to Smooth Response Time Fluctuations in Multi-tier Services
Shuangshuang Guo, Lizhen Cui, Shijun Liu and Calton Pu
Computational Assessment on Adaptive Level of Rehabilitation Exergames for Elderly Stroke PatientsHao Zhang, Di Wang and Zhiqi Shen
Computerizing More Repeatable Trail Making Test for Longitudinal Cognitive Assessment
Zhiwei Zeng, Cyril Leung and Zhiqi Shen
Constrained Texture Mapping via Voronoi Diagram Base DomainPeng Cheng, Jiaye Wang and Thomas W. Calvert
Crowd Wisdom Drives Intelligent Manufacturing
Jiaqi Lu, Shijun Liu, Lizhen Cui, Li Pan and Lei Wu© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved

197

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Demo/Posters
CTP：a Scheduling Strategy to Smooth Response Time Fluctuations in Multi-tier Website System
Wenbin Zhang, Yuliang Shi, Shidong Zhang, Zheng Yongqing, Lizhen Cui and Han Yu
Data Organization Patterns for Cloud Enterprise Applications
Yi Wei, Lei Wu, Shijun Liu, Li Pan and Xiangxu Meng
Device Data Ingestion for Industrial Big Data Platforms with a Case Study
Cun Ji, Qingshi Shao, Jiao Sun, Shijun Liu, Li Pan, Lei Wu and Chenglei Yang
Distributed ACO Based on a Crowdingsourcing Model for Multiobjective Problem
Jiaqi Lu, Li Pan and Shijun Liu
Efficient Crowd-Powered Active Learning for Reliable Review Evaluation
Xinping Min, Lizhen Cui, Qingzhong Li and Han Yu
Empathy Map Game: A Social Cognitive Game for Elderly
Feng Xiao, Han Lin and Chunyan Miao
ETTF: A Trusted Trading Framework Using Blockchain in E-commerce
Xinping Min, Qingzhong Li, Lei Liu and Lizhen Cui
Explained Activity Recognition with Computational Assumption-based Argumentation
Xiuyi Fan and Siyuan Liu
Familiarity Design in Exergames for the Elderly
Hao Zhang, Di Wang and Zhiqi Shen

Poster Session III
Date and Time: Friday, July 29, 2016, 12:00 PM-13:00 PM
Location: #1013, Earth Sciences Building, UBC
Follow Me: A Personal Robotic Companion System for the Elderly
Rong Wang, Huiguo Zhang, Yuan Miao, Cyril Leung and Zhiqi Shen
Forming Effective Teams for Sustainable Living Using Familiarity Design
Zhengxiang Pan, Jun Lin, Han Yu, Chunyan Miao and Cyril Leung
Game Theoretical Approach to Prevent Illegal Network Flow
Qingyu Guo, Bo An and Yair Zick
IBDP: An Industrial Big Data Ingestion and Analysis Platform and Case Studies
Cun Ji, Shijun Liu, Chenglei Yang, Lei Wu and Li Pan
Integrating Theoretical Modeling and Experimental Measurement for Soft Resource Allocation in Multi-Tier Web Systems
Yuliang Shi, Jiwei Huang, Xudong Zhao, Lei Liu, Shijun Liu and Lizhen Cui© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved

198

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Demo/Posters
Learning the Influence Probabilities Based on Multipolar Factors in Social Network
Weibo Wang, Zhaohui Peng, Ziyan Liu, Tianchen Zhu, Xiaoguang Hong and Lizhen Cui
Long term magnitude and phase information for synthetic speech detection
Xiaohai Tian, Steven Du and Eng Siong Chng
Making Use of Virtual and Augmented Reality in an Elderly Setting
Benny Tan, Frank Guan and Cyril Leung
MapReduce short jobs optimization based on resource reuse
Yuliang Shi, Kaihui Zhang, Lizhen Cui, Lei Liu, Zheng Yongqing, Shidong Zhang and Han Yu
MGCAD: An Transient Anomaly Detection Approach for Industrial Equipment Based on Multiple Group Correlation
Probability Models
Xiangzhen Xu, Yuliang Shi, Shanqing Guo, Shijun Liu, Lizhen Cui and Hui Li
Minimizing the Cost to Win Competition in Social Network
Ziyan Liu, Hong Xiaoguang, Zhaohui Peng, Zhiyong Chen, Weibo Wang, Tianhang Song and Lizhen Cui
MyLife: A Digital Autobiographical Memory System
Di Wang and Ah-Hwee Tan
Online Multimodal Co-indexing and Retrieval of Weakly Labeled Web Image Collections
Lei Meng, Ah-Hwee Tan, Cyril Leung, Liqiang Nie, Tat-Seng Chua and Chunyan Miao
Optimal Electric Vehicle Charging Station Placemen
Yanhai Xiong, Jiarui Gan, Bo An and Ana L. C. Bazzan
Optimizing Replica Exchange Strategy for Load Balancing in Multitenant Databases
Teng Liu, Qingzhong Li, Lanju Kong, Lei Liu and Lizhen Cui
Performance Analysis of Cloud Computing Centers Serving Parallelizable Jobs Using M/M/c/c+r Queuing Systems
Xiulin Li
Performance and Resource Prediction at High Utilization for N-Tier Service Systems in Cloud An Experiment driven
ApproachWenbin Zhang, Yuliang Shi, Lei Liu, Lizhen Cui and Zheng Yongqing
Personality Prediction from Chinese Social Media Profile
Shuji Hao, Chunyan Miao and Steven Hoi

Poster Session IV
Date and Time: Saturday, July 30, 2016, 12:00 PM-13:00 PM
Location: #1013, Earth Sciences Building, UBC© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved

199

Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)

Demo/Posters
Preliminary analysis of the evolution of social networks based on MSM triple feature
Ziyu Guo
Profit Based Two-Step Job Scheduling in Clouds
Shuo Zhang, Li Pan, Shijun Liu, Lei Wu and Xiangxu Meng
Role Fulfilment Model for Motivating Participation in Mundane Crowdsourcing Tasks
Xinjia Yu, Chunyan Miao, Cyril Leung and Charles T. Salmon
Second-price Auction-based Dynamic Allocation Model for Cloud Computing Resource
Bingbing Zheng
Self-Directed Learning Skill Assessment Using Behavior Data in an Crowdsourcing Platform
Ailiya Borjigin, Chunyan Miao, Zhiqi Shen, Han Yu and Zhiwei Zeng
Selective Attention – Can it Be Improved?
Siyuan Liu, Zhiqi Shen, Han Lin, Zhengjin Guo, Chunyan Miao, Cyril Leung and Chin Jing Jih
Silver Assistants for Aging-In-Place
Di Wang, Budhitama Subagdja and Ah-Hwee Tan
SRConfig: An Empirical Method of Interdependent Soft Configurations for Improving Performance in n-Tier Application
Yuliang Shi, Xudong Zhao, Shanqing Guo, Shijun Liu and Lizhen Cui
Teachable Agents Framework for Developing Heterogeneous Virtual Assistants
Budhitama Subagdja and Ah-Hwee Tan
Temporal Recommendation via Modeling Dynamic Interests with Inverted-U-Curves
Yang Xu, Hong Xiaoguang, Zhaohui Peng, Guang Yang, Philip S Yu and Lizhen Cui
Towards Best Region Search for Data Exploration
Kaiyu Feng, Gao Cong, Sourav S. Bhowmick, Wen-Chih Peng and Jane Wang
Towards Data-driven Software Engineering Skills Assessment
Jun Lin, Han Yu, Zhiqi Shen, Hao Zhang and Yuan Miao
A Piecewise Linear Representation Method Based on Importance Data Points for Time Series Data
Cun Ji, Shijun Liu, Chenglei Yang, Lei Wu, Li Pan and Xiangxu Meng
A Self-Evolving Method of Data Model for Cloud-based Machine Data Ingestion
Cun Ji, Shijun Liu, Yingbo Liu, Chenglei Yang, Lizhen Cui, Li Pan and Lei Wu
A framework for recognizing behavior of the elderly from sensor data
Wei Wang, Sinno Jialin Pan and Chunyan Miao

© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved200Proceedings of the 2016 International Conference on
Crowd Science and Engineering (ICCSE 2016)© 2016, Association for Crowd Science and Engineering (ACE). All rights reserved