International Journal of Service and Knowledge Management
International Institute of Applied Informatics
2020, Vol. 4, No. 2, 19 - 33

Comparative Verification of Design Ordering Methods
using Reader's Feeling for Document Production Outsourcing Services
Kazuo Shibuta *†, Kunihiko Higa *Abstract
We showed that there are three levels of production purpose, reader's feeling and design
specifically as an ordering method when requesting design work. It was statisticallyconfirmed that the appropriateness and the easiness of understanding improve when theorder content includes reader’s feeling. In particular, it was recognized that appropriatenesswould be improved if the reader's feeling instruction was given without the production
purpose instruction. Furthermore, appropriateness decreased when instructing production
purpose and design at the same time. As a result, it is considered that the inclusion ofreader’s feeling in the content of the order has improved the evalu-ation of the worker forthe ordering method.

Keywords: design, outsourcing, crowdsourcing, ordering method

1 Introduction
1.1 Document production outsourcing services
In recent years, as the simple copier business falls into price competition, the industry of
office automation equipment has started to provide strategic outsourcing services foroffices as a new business to improve profitability. For example, at Fuji Xerox [1], themainstream is the service that manages the optimal output environment in the office, whichis called managed print service. Furthermore, they launched a new service: an outsourcing
service for document production, not for managing output environment. With this service,
they undertake production of documents used in corporate activities, such as educations
and proposals, for example: catalogs, instruction manuals (user's manuals), service
manuals, flyers, provisions, etc. for products or services, with optimizing quality, cost, anddelivery date. In this service, they undertake not only simple outputs, but also the
production of documents including design. It is important to develop operation
technologies and systems in general production including design, and to have know-how to
product documents.
*†

Tokyo Institute of Technology, Tokyo, Japan
Kwansei Gakuin University, Hyogo, Japan20

K. Shibuta, K. Higa

1.2 Emergence of crowdsourcing
On the other hand, along with the development of ICT, the use of crowdsourcing which
utilizes ICT and is a new system of human resource procurement has been increasing.
"Crowdsourcing" is a coined word that combines "crowd" as a group of people and"sourcing" as procurement, which is said to be used for the first time in a blog article [2]posted on U.S. "WIRED" by Jeff Howe: the contributor to the magazine. It is a system thataccesses an unspecified number of individuals or companies using ICT to procure humanresources in need. Ordering, application, decision of worker, delivery and payment aredone via ICT. Herewith, it became possible to place an order to general individuals such as
students, housewives, retired seniors, etc. The contents of jobs are wide-ranging such as
development, design, writing, office work, business support, etc. According to the survey
by Y's STAFF [3], the jobs ordered in Japan are mostly related to design, web design, web
development, etc., and it seems that many clients have needs for special skills.
The jobs will be generally classified into project type, competition type and task type
depending on the appointment method of workers and procedure. The project type is a
project-based job with a fixed production period and deliverables, and the fee may be
decided by negotiation after the worker is appointed. The competition type is a job inwhich only an outstanding result is adopted, and the fee is often predetermined. The task
type is a finely divided task that anyone can do, and the fee is often predetermined. In thecases related to design, it is usual to order in the competition type, such as production oflogos, flyers, posters, etc., but there are cases that workers who have been checked theresult in the prior competition type may be asked more complex and advanced tasks in the
project type.

2 Purpose of Research
2.1 Problems in ordering jobs related to design
When considering the commercialization of outsourcing services, the current problems of
crowdsourcing could be helpful. Although it is not a face-to-face process, an ordering incrowdsourcing is a process of contract, and therefore communication skills of both clients
and workers are also required. There are also a few cases where troubles occur over thedeliverables. The cause is, for example, that the delivery date is not suitable, but thefollowings can be considered especially in jobs related to design.

1) Client's instructions on design are complicated
2) The image for the deliverable is different between the client and the worker
3) The fee for deliverable is underestimated by the client
The ordering process in crowdsourcing includes client ordering, worker's application,
worker selection by client, worker's work, and client evaluation of deliverables. Regardingordering by clients, Kittur et al. [4] explained about a complex job, if the task of creating along article is divided into small units and requested, the quality of the deliverable was

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

effected by the human capability in the middle. In task design, Chandler and Kapelner [5]examined how to manage worker motivation, and Eickhoff and Vries [6] examined how toprevent fraud. Regarding the selection of workers by clients, Kittur et al. [7] showed thatincorporating a mechanism to confirm that the instructions were read to the end whenplacing an order was effective in eliminating workers who answer incorrectly. Regarding theevaluation of deliverables by clients, Heer and Bostock [8] visualized evaluations using"Amazon Mechanical Turk".
In jobs related to design, client's instructions are given in different levels: sometimes in view ofthe production purpose such as concept, targeted persons, etc., sometimes in specific design
such as finished image, coloring, font size, etc. The instructions at these different levels may not
be consistent at clients. In addition, because of the difficulty to express everything in words if
there is no reference or base, clients may look at the unfinished product to add some
instructions. This is the reason that situations like 1) are considered to occur. Furthermore,when the client cannot express the image specifically and gives instructions only for the simple
purpose of the product, such as improvement of the purchase rate, the worker cannot have theexact image of the client, and therefore cause troubles. This is the reason that situations like2) are considered to occur. In Lancers [9], a comprehensive crowdsourcing website in Japan,
and CrowdWorks [10], another comprehensive crowdsourcing site, the following ideas havebeen made for the jobs related to design. At the time of order, they are preparing a template todescribe the purpose and outline of production, a user interface to instruct images with a scale
such as monotone-colorful, simple-complex, casual-classic, reasonable-high class, traditional-modern, feminine-masculine, or a user interface to select the desired color with a radio
buttons such as red, yellow, green, blue, pink, orange, black, etc. Herewith, it is consideredthat they are trying to solve the problem 2). Furthermore, a contradictory instruction may be
given by simultaneously designating a desired image and a desired color. In fact, in theindications of desired color, words related to the image such as bluish (cool, refreshing,masculine), pink (cute, young, feminine), black (dainty, luxury, elegance), etc. are alsopresented. At last, the value of the product seems to be how much it can achieve its purpose,but it is difficult to predict in advance. That is because it is determined by the reader who is
the targeted persons of the product, and not by the client or the worker. Therefore, the worker
cannot claim its value objectively to the client. This is the reason that situations like 3) areconsidered to occur. This also results in the question of whether or not a production should bedone faithfully to the client's image regarding the problem 2). These situations can beconsidered even in the case of regular outsourcing but expected to be particularly noticeablein crowdsourcing because the orders often proceed via ICT.

2.2

Consideration of an effective ordering method of design

With regard to such jobs related to design, an effective ordering method using crowdsourcing
will be proposed and conduct an experiment to verify the effect. The expected effect will be thatthe complexity of instructions is eliminated, the image for the product is consistent between theclients and the workers, and the clients don't underestimate the fee for the deliverables.
However, it is not considered whether the images of the deliverables match and whether the fee
for the deliverables are appropriate at this time, because it is necessary to evaluate the design ofthe deliverable itself. The evaluation of the worker, in other words the designer, is used as analternative indicator of the effect.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

21

22

K. Shibuta, K. Higa

3 Extraction of Problems in Design Ordering
3.1 Contents
In “Promotion program of utilizing crowdsourcing at The Small and Medium Enterprise
Agency 2016” [11], with the purpose of promoting crowdsourcing in small and medium
enterprises, a crowdsourcing producer offered a program to let participant’s experience
actual ordering and to raise awareness of crowdsourcing. The problems upon its use werealso extracted. The crowdsourcing producer is a company that is certified by thecrowdsourcing association under the certification system established in the previous year to
support small and medium enterprises with the use of crowdsourcing. This project aims togain the competitiveness of small and medium enterprises by utilizing crowdsourcing, and it
is conducted by the crowdsourcing utilization consortium consists of 3 members, Fuji
Xerox Co., Ltd., Crowdsourcing Association in Japan, and Japan Telework Association by
the commission of The Small and Medium Enterprise Agency in Japan. In the orderexperience program, two courses were held. The overview of crowdsourcing was explainedin the first round and they registered on the crowdsourcing website and order a flyer design(A4 size) in the competition type. The second round was held about 1 or 2 weeks later, inwhich they selected from the flyers gathered in the actual competition. The order experience
program was held at about 150 locations nationwide in Japan from February to June 2016,
with more than 800 small and medium enterprises participated.

3.2 Research method
Survey questionnaires about satisfaction levels and problems of small and medium enterprises will be conducted for in the order experience program participants. The satisfaction level is measured by a Likert scale (5: very good, 4: good, 3: neutral, 2: bad, 2: verybad) regarding the entire program, the fee and quality of the deliverables, the operation andregistration on the website, etc. Other than that, the definition of requirement (1: well
accepted without review, 2: well accepted after reviewed once, 3: well accepted after
reviewed for multiple times, 4: not accepted), the number of applicants or proposals, etc.

3.3 Result
The number of valid answers was 733. First, we looked at the relationship between the
average satisfaction level of the entire order experience program and whether the requirements were defined successfully. As a result, the satisfaction level decreased when thedefinition was "not accepted" than else. Through one-way analysis of variance, statisticallysignificant difference (F (3,729) = 19.14, p < 0.01) was indicated (see Figure 1). The error
bars in the figure indicates standard errors.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

Definition of requirements

Satisfaction level

Well accepted without review

Well accepted after reviewed
once
Well accepted after reviewed
for multiple times
Not accepted

Figure 1: Relationship between
satisfaction level and definition of requirements
Next, we look at the relationship between the average number of applicants and proposals,
and whether the requirements were defined successfully. As a result, the number of applicants
and proposals tend to increase when the definition is well accepted (see Figure 2). The error
bars in the figure indicates standard errors.
Definition of requirements

The number of applicants/proposals

Well accepted without review
Well accepted after reviewed
once

Applicants
Proposals

Well accepted after reviewed
for multiple times
Not accepted

Figure 2: Relationship between
The number of applicants/proposals and definition of requirements
From the analysis results, it was found that both the number of applicants and proposals
increase when the definition of requirement is well done at the time of ordering. This is likely toincrease satisfaction with the order experience program, as it offers more options for
proposals and allows more desirable designs to be adopted.

4 Consideration of Design Orderings
4.1 Design process
Considering the general design process, this time there is a third party, the reader, in
addition to the client and worker. In Axiomatic Design [12], the design process is divided into a customer domain, a functional domain, an actual domain, and a process

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

23

24

K. Shibuta, K. Higa

domain, and the customer who is the reader is considered. Further, it is shown that the designer
starts with the function requirement of the functional domain. In this case, we will examine what
the function requirement is. Since the document design influences the affection, the demandvaries greatly from person to person. Matsuoka et al. [13] and Yanagisawa et al. [14] haveproposed design models that deal with affection. According to them, customer’s psychologicalfactors are defined in the function requirement.

4.2 Relationship between production purpose and designFirst of all, the final evaluation of the product can be determined by how much it
achieved the production purpose. For example: in the case of a catalog, the purchase rate shouldbe improved; in the case of a manual, the understanding level should be improved; in thecase of a form, entry errors should be decreased; Furthermore, the satisfaction level of thetargeted persons should be increased. However, it is very difficult to objectively show them inadvance. The reason may be the fact that how design can be achieved by designing the layout
and coloring largely differs depending on the type of the target product and persons. For thisreason, in the production work including design, regardless of how the production purpose canbe achieved, the fee is usually calculated by multiplying man-hour and the unit price per hour.
However, in this case, the reduction in man-hour and the unit price per hour becomes acompetitive advantage. On the contrary, there is also a problem that it becomes difficult todesign effectively for the production purpose.
On the other hand, in the field of usability evaluation, the following three are often
measured: production purpose as whole evaluation, feelings as psychological factor, and design
as physical factor (see Figure 3). The psychological factors are, in particular,
psychophysical quantities that quantitatively express the reader's feelings such as "easy tounderstand", "easy to fill in", "reliable", "neat", "fun", etc.

Figure 3: Relationships between production purpose –feeling –designin the usability evaluation
Tanaka et al. [15] classified these psychological factors into several evaluation scales related
to journal articles, and then performed multiple regression analysis between the evaluation
scales and the final rating "good" and finally indicated that it leads to the regression
equation. However, this regression equation is limited to the one objected to journal articles.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

Since the relationship between the psychological factors and the design is related to humanperception and cognition as a living body, it is considered that the relation is not greatly
influenced by the type of the product and the targeted persons. On the other hand, therelationship between the production purpose and the psychological factors is related toadvanced information processing, because it is about what kind of effect do those perceived
from the product give human behavior. Therefore, it is considered that this is greatly
influenced by the type of product and the targeted persons. In the elaboration likelihood
model (ELM) [16], the information processing route for causing attitude change differs
depending on the information processing ability and motivation of the consumer; the central
route is reported as it uses more direct and logical information, and the peripheral routes arereported as they use rather indirect and sensory information. For example, in the case of a
product catalog, some people may be motivated to purchase because of "easy-tounderstand" catalog that explains the details of the product, and some people may preferan "interesting" catalog that gives an image of the scene using the product. This also
varies depending on the product type. The psychological factors required for productcatalogs used in business and those used for an individual are expected to be different. This
relationship is also influenced by the product type. For example, in the case of a service
manual, it is considered that accurate and "reliable" information is written in an "easyto-find" manner to achieve the production purpose. Usually, design patterns are infinite, and it
is impossible to directly obtain the relationship between the production purpose and designfor each type of product or targeted persons. However, in case of a psychological factor, it
is possible to condense it to a statistically finite number using factor analysis, etc. In otherwords, in the case of relationship between production purpose and psychological factors, it
is possible to organize by type of product or targeted persons. Furthermore, if
connecting correspondence of psychological factors and design together, it is possible toexplain the whole relationship between production purpose and design for each type of
product or targeted persons.

4.3 Design management
By using psychological factors as mediators, design management using this is considered tobe effective for the process of ordering design. So far, when ordering a production, clients had
directly included specific design instructions such as layout and coloring in their requests. On
the other hand, it may be sufficient to indicate by the psychological factors which are required
for the product. In fact, clarifying the relationship between the production purpose and thepsychological factors is conducted as market research. Therefore, it is feasible in theprocess on the client's side to determine what psychological factors will be focused on.
On the other hand, designers who receive production orders often know the relationshipbetween psychological factors and design as know-how. For example, in the field of color
coordination, Nippon Color & Design research institute [17], which has spread the scale of"soft-hard "and "cool-warm," has a practical training program for educating designers inwhich learning the words representing image expressions corresponding to psychologicalfactors and the corresponding colors. It may not be so difficult for designers to express
something instructed by the psychological factors in a specific design. It is expected thatan efficient production request process can be realized by instructing with psychologicalfactors in the design work.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

25

26

K. Shibuta, K. Higa

5 Experiment of Design Order Evaluation
5.1 Ordering method in crowdsourcing
Up to this point, it was shown that instructing with psychological factors in design work
could lead to the realization of an efficient process of ordering, but in actual
crowdsourcing, we confirmed that there is no problem with this method. Using
CrowdWorks [10], which is a comprehensive crowdsourcing website in Japan, ordered
the actual design job in a competition type and asked the applicants to answer the questionnaire. As a result, we got 5 applicants. Also, the following answers are obtained fromthe questionnaire.

1) Production purpose is essential and should be more specific.
2) Production purpose is not sufficient.
3) There are two ways: leave everything to the designer or focus the client's image as muchas possible.

As for 1), we got opinions such as “It's easy to create if you present objective data such as thetarget audience, display place, and appeal points”. There is an opinion that “There are awide range of design options, and it is expected that the concrete production purpose will be thecriterion when making a concrete design. Because there are so many design options, it isexpected that the specific production purpose becomes the criterion when dropping into an
actual design.” As for 2), we got an opinion: “There is an instruction saying to extract freely
from HP and past works, but because it is a work of planners or copywriters, it will be moredifficult and leads to decrease of the number of proposals". While production purpose isessential, it seems difficult for normal designers to drop into an actual design without other
instructions. Furthermore, we got an interesting opinion as for 3): “The method of leaving it to
the designer is an effective way that emphasizes designer's ability and let them present
unexpected expressions that the clients never imagine”. We also got opinions such as "Whenleaving it to the designer, the necessary information may be just words," or "When focusing theimage of the client, they need to specify the order (importance) of the words to be used, add
pictures, colors, other posters to refer to, words to describe the poster image, and give more
specific instructions." When leaving to the designer, abstract instructions are required, and when
focusing the client's image, more specific instructions are required. In the former case, there is arisk that the designers cannot handle the job depending on their skill, as described in thecomment, and in the latter case, the job may be an operating task and reduce designer's
creativity.

5.2 Classification of design ordering method
Here, we review the ordering method in the job related to design. As a method forordering a design work, we propose three levels: production purpose, reader's feelings,
and concrete design. It will be concrete in this order, but the amount of instructions will
also increase. Also, if you give all the instructions at the same time, there is a possibility ofconflict. The reader's feelings correspond to psychological factors in the former chapters.
In order to simplify the instruction to the minimum required, it may be important at what

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

degree of instruction is included in the order. The targeted persons and concept of the productare included in the production purpose. The desired image and the desired color which areprepared on the current crowdsourcing website, are included in each reader's feeling and design.

5.3. Outline of design order experiment
We ordered the job related to design to the designers employed by Fuji Xerox groupcompanies in a competition type. Because the designers were distributed in Tokyo,
Yokohama, Osaka and Fukushima, they used email and file servers. We used a file server
to provide and deliver materials and exchanged the order request and questions via email.
[Competition for production of a volunteer recruitment flyer]
Event: “Fuji Xerox Super Cup Volunteer of inviting children with intellectual disabilities”
Application due date: 15 December (Mon) - 26 December (Fri) 2014
Deliverable: A4 size (PDF)＊A gift voucher will be awarded upon adoption.
This volunteer event is called "Hasu (fraction) Club" and asks participants for the fraction(less than 100 JPY) of their salaries. An independent volunteer group that conducts
activities with the contribution was holding the event. Therefore, it was suitable for thisevaluation experiment as it was not subjected to design restrictions by corporate identity(CI). In fact, we called for approximately 30 designers working in the following 8 departments in 7 companies. As a result of recruitment, we got 10 proposals from 7 designers in 5 companies, and the deliverable obtained was satisfactory to a person actuallyplanning events.

5.4 Details of the experiment method
In the evaluation experiment, the designers employed in the Fuji Xerox Group companies
were ordered with different instructions, and evaluations of the designers about the
instructions was compared. This time, the target was not the general crowdsourcing site,
but the in-house designers of each group company in order to verify the same product bychanging the ordering method. Because different ordering conditions will also be disclosed on a general crowdsourcing site, so it is considered that there is a risk that thecontractor will apply only for the competition of the desired conditions and statisticalcomparison cannot be done. Ordered with different contents that combines three levels of
instructions at random; production purpose, reader's feeling, and design. we also
considered making 3 groups per each instruction, but because there was also a risk that the
number of applicants could not be sufficient, we changed to verify multiple groups withsmall samples by combining the instructions. The specific instructions of three levels are as
follows.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

27

28

K. Shibuta, K. Higa

<Production purpose>•

To let many people apply for the event who had not participated in volunteer
experience<Reader's feeling>•

Prominent•Interesting•

Feeling something possible done by own•

Feeling contribution to society<Design>•

Please make the word "volunteer" bigger.

•

Please use many photos.

•

Please make the color noticeable.

The designer’s evaluation of the ordering was measured by a Likert scale(appropriateness, easiness of understanding, specificity, and objectivity). The specific scales
are as follows.
Appropriateness(5: appropriate. 4: somewhat appropriate, 3: neither is, 2: somewhat inappropriate, 1:
inappropriate)
Easiness of understanding(5: easy to understand. 4: somewhat easy to understand, 3: neither is, 2: somwhat not easy tounderstand, 1: not easy to understand,)Specificity(5: specific. 4: somewhat specific, 3: not neither, 2: somewhat abstract, 1: abstract)
Objectivity(5: objective. 4: somewhat objective, 3: not neither, 2: somewhat subjective, 1: subjective)

5.5 Result
12 designers answered the evaluation of the ordering method. An analysis of variance ofappropriateness, easiness of understanding, specificity and objectivity was performed
depending on whether the order content includes production purpose, reader'sfeeling and design. As a result, a statistically significant difference was found about
appropriateness and easiness of understanding. Firstly, appropriateness was improved(F (1, 8) = 6.40, p <0.05) when instructing reader's feeling (see Figure 4).

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

29

AppropriatenessAppropriatenessInstruction: least mean square
Current effect: F (1, 8) = 6.400, p = 0.03527
The vertical bar presents 0.95CI

Instruction of reader’s feeling

Instruction of design with production purpose

averageaverage ± SEaverage ± 1.96 * SE

Figure 4: Difference in Appropriatenesswith/without instruction of reader’s feeling
Next, easiness of understanding was improved (F (1, 8) = 5.88, p <0.05) when
instructing reader's feeling, in particular, in the absence of an instruction for production purpose(F (1, 8) = 10.05, p <0.05) (see Figure 5). Herewith, it is considered that reader's feeling cancomplement production purpose in terms of easiness of understanding. Appropriatenessdecreased (t = 2.64, f = 6, p <0.05) when instructing production purpose and design at
the same time (see Figure 4). The cause of this is imagined at the moment, but it is thoughtthat the design instruction may contradict the production purpose. The design instruction
approaches the mere operation instruction, and thus may reduce the motivation to create. It
may be noticeable when the contradiction is felt between the instruction of design andproduction purpose.
Instruction: least mean square
Current effect: F (1, 8) = 10.052, p = 0.01318
The vertical bar presents 0.95CI

Easiness of understanding

Easiness of understanding

Instruction: least mean square
Current effect: F (1, 8) = 5.8783, p = 0.04156
The vertical bar presents 0.95CI

Instruction of
production
purpose◯×

Instruction of reader’s feeling

Instruction of reader’s feeling

Figure 5: Difference in Easiness of understanding
with/without instruction of reader’s feelingWith regard to specificity and objectivity, no statistically significant difference was foundat this time.
The designer's questions for confirmation that were exchanged from ordering to application
were classified into three types: concept image, material usage, and output form. A groupwhose order contents did not include reader's feeling asked a question about concept image,

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

30

K. Shibuta, K. Higa

but no other group asked about this. The following comment was given from the designer who
asked the question about concept image.
“I would like to know the details of the presentation of the planning concept, themes andrequests, and the target group (who should I appeal to?).”“Isn't this method difficult for design proposals as a solution to the vague needs andrequirements of the client?”“I would like you to describe in more detail the part of the person who has no experience as a
production purpose volunteer.”On the other hand, questions regarding material usage and output form were the same fromall groups. In this experiment, because it was a competition type, there was no precontractual interaction like the project type, and the absolute number of questions was small,so statistical verification could not be performed. However, if the contents of the order include
reader's feeling, it may be possible to reduce the number of questions asked by the designer for
confirmation, especially regarding the concept image. In addition, this time, 2 designers havemade multiple proposals. Both were designers of groups whose order contents did notinclude reader's feeling. Even if the ordering content includes the production purpose but does
not include the reader's feeling, it is possible to assume that multiple images are produced, so it
can be imagined that multiple proposals have been made in advance. Since this time it was acompetition type, it is not a bad thing that many proposals gather, but in the case of a projecttype, it is effective to include reader's feeling in the order contents in order to reduce theeffort of the contractor. From the results, it was not possible to statistically verify that the
number of questions of the designer for confirmation decreases when the order is placed in
the reader's feeling. However, there is such a tendency, and it cannot be denied.

6 Conclusion
6.1 Results and discussion
We presented that there are three levels; production purpose, reader's feeling and specificdesign concerning the ordering method of the jobs related to design. Among these, byordering including reader's feeling, it was expected that the difference in the image of thedeliverables of the client and the worker would be small, and the worker's evaluation on theorder method would be high. In this regard, it was statistically confirmed that theappropriateness and the easiness of understanding improve when the order content includes
reader's feeling. In particular, in the absence of an instruction for production purpose, theinstruction of reader's feeling helps to enhance easiness of understanding. Herewith, it isconsidered that the inclusion of reader's feeling in the order has enhanced the worker's
evaluation on the ordering method.
On the other hand, it is not yet known how the client's evaluation differs depending on the
difference in the ordering method. We consider that the survey results of aforementioned"Promotion program of utilizing crowdsourcing at The Small and Medium Enterprise
Agency 2016" [11] will become a good reference. In this ordering experience program,

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services

the crowdsourcing producer was supposed to support the definition of requirements at the timeof ordering from the following viewpoints.

•

Clarify and specifically describe the purpose and outline of the production of theflyer with using templates.

•

Specify the desired image, etc. with using U/I to tell the worker the image of thedeliverable.

The former corresponds to production purpose and the latter corresponds to reader'sfeeling. Herewith, it is considered that if the crowdsourcing producer supports thedefinition of requirements from these viewpoints, the number of applicants and proposals
might increase as the worker accepted it well. Further, 77% of the respondents claimed "placing
order / defining requirement" as the part that they asked crowdsourcing producers for help. It
was the highest ratio among other particulars such as "member registration (41%)","correction request (32%)", "selection of worker (40%)", and "inspection of deliverable(16%)".

6.2 Limitations
The verified jobs were limited to the jobs related to design at this time. The characteristic is thatthe readers decide whether the production purpose can be achieved or not, and it is not by theclient or the worker. For this reason, this survey does not mention the quality of thedeliverables. The results may not be applicable to regular operations such as development
or desk work that are different from design work. However, we consider that there is apossibility applied to the business of producing products that depend on customer's mind suchas marketing.

6.3 Future perspective
With regard to the worker's evaluation, we examined what causes the satisfaction level of theclient, based on the survey result of "Promotion program of utilizing crowdsourcing
at The Small and Medium Enterprise Agency 2016" [11]. We asked people with actual
ordering experiences whether the definition of requirements was successful or not. The
result showed that the number of applicants and proposals increase and the satisfactionlevel of the client is enhanced as the definition is written successfully. From the contents ofseminar of the ordering experience program, it is expected that the requirements havedefined more successfully as the order includes more production purpose or reader'sfeeling. In order to clarify this, it seems to be necessary to individually confirm what kindof instructions the client has given. We intend to present the possibility of simplifying the
instruction and enhancing worker's evaluation through ordering with reader's feeling. Also,
the number of applicants and proposals will increase and the orders will be more effective
ordering for the clients. In addition, we have not directly mentioned the improvement of theorder acceptance ratio and avoidance of problems which are the current issues of
crowdsourcing, but we hope that the results in this study will contribute to these issues.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

31

32

K. Shibuta, K. Higa

References[1] Fuji Xerox Co., Ltd., Dec. 2014; http://www.fujixerox.co.jp/solution/gs/.
[2] J. Howe, “The Rise of Crowdsourcing,” Dec. 2014;
http://archive.wired.com/wired/archive/14.06/crowds.html?pg=4&topic=crowds.
[3] “A Survey on Usage of Crowdsourcing in Japan,” Y’s Staff Co., Ltd. report, Smalland Medium Enterprise Agency, 2013 (in Japanese).
[4] A. Kittur, B. Smus, and S. Khamkar, R. E. Kraut, “Crowdforge: Crowdsourcing
complex work,” Proceedings of the 24th annual ACM symposium on User interface
software and technology, 2011, pp.43-52.
[5] D. Chandler, and A. Kapelner, “Breaking monotony with meaning: Motivation incrowdsourcing markets,” Journal of Economic Behavior & Organization 90, 2013,
pp.123-133.
[6] C. Eickhoff, and A. P. d. Vries, “Increasing cheat robustness of crowdsourcing
tasks,” Information retrieval 16(2), 2013, pp.121-137.
[7] A. Kittur, E. H. Chi, and B. Suh, “Crowdsourcing user studies with MechanicalTurk,” Proceedings of the SIGCHI conference on human factors in computing
systems, 2008, pp.453-456.
[8] H. Jeffrey, and M. Bostock, “Crowdsourcing graphical perception: using mechanical turk to assess visualization design,” Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, 2010, pp.203-212.
[9] Lancers Inc., May. 2020; http://www.lancers.jp/.
[10] CrowdWorks Inc., May. 2020; http://crowdworks.co.jp/.
[11] Announcement of "Cloud Sourcing Ordering Experience" for SMEs developed by
the Consortium for Promoting Crowdsourcing Utilization, " May. 2020;
https://www.japan-telework.or.jp/oshirase/75.html.
[12] N.P. Suh, “Axiomatic Design of Mechanical Systems,” Journal of Mechanical Design and the journal of Vibration and Acoustics, Transactions of the ASME, Volume
117, June 1995, pp.1-10.
[13] Yoshiyuki Matsuoka, “Multispace Design Model: Toward the Construction of the
Framework for Design Science,” The 55th Spring Seminar of Japanese Design
Society Conference summary collection, 2008 (in Japanese).
[14] H. Yanagisawa, T. Murakami, K. Ohtomi, and R. Hosaka, “Quantification methodof product emotional quality considering its diversity (application to quantificationof emotional quality in product sound design),” Transactions of the Japan Society of
Mechanical Engineers, Series C, Vol.74, No.746, 2008, pp.2607-2616 (in Japanese).

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Comparative Verification of Design Ordering Methods using Reader's Feeling for Document Production Out-sourcing Services[15] Takeshi Tanaka, Masakazu Fujimoto, Tomomi Urushihara, “On the evaluation
scale for document expression quality,” Proceedings of the 42nd National Conference of Information Processing Society of Japan, the first half of 1991 (3), 1991, pp.
331-332 (in Japanese).
[16] R. E. Petty, J. T. Cacioppo, and D. Schumann, “Central and peripheral routes toadvertising effectiveness: The moderating role of involvement,” Journal of consumer research, 10, 1983, pp. 135-146.
[17] “What is the image scale | NCD-WEB | Japan Color Design Research Institute forColor Psychology, Consumer Research, Trend Analysis,” Japan Color Design Research Institute Inc., May. 2020;
http://www.ncd-ri.co.jp/about/image_system/imagescale.html.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

33