言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Insights from a large scale web survey for
Acceptability Rating Data for Japanese (ARDJ) project
Kow Kuroda1 , Hikaru Yokono2 , Keiga Abe3 , Tomoyuki Tsuchiya4 , Yoshihiko Asao5 ,Yuichiro Kobayashi6 , Toshiyuki Kanamaru7 , and Takumi Tagawa8
1 Kyorin University, 2 Fujitsu Laboratories Ltd., 3 Gifu Shotoku College, 4 Kyushu University, 5 NICT,
6 Nihon University, 7 Kyoto University, 8 Tsukuba University

1.

Introduction

Acceptability plays a crucial role in linguistic theorizing. But it is far from fully understood how “ordinary”
people react to sentences with varied degrees of devianceor anomaly, simply because there is no such data. ARDJ
is a project that aims at exploring true nature of acceptability judgment/rating based on a large-scale survey with least
theoretically biased stimuli.
ARDJ has done two experiments for this purpose. The
first one, called “survey 1,” was carried out in 2017. It wasintended to be a pilot study with only a limited variety ofresponders (roughly 200 college students only) on 200 sentences. The results we obtained are reported in [3].
The second experiment, called “survey 2,” was carried in
2018. It was the main study of the ARDJ project, with theexpansion of stimulus set to 300. It has two phases: phrase
1 is a small scale experiment in which roughly 300 college
students participated, which is comparable to the pilot study
done in 2017. Phase 2 is a large scale web survey in which
responses were obtained from over 1,600 participants withsignificantly more variations in their attributes. This paper
reports on phase 2 of ARDJ survey 2.

2.

ARDJ survey 2, Phrase 2

2.1

Stimuli construction

candidates for deviant sentences were constructed by randomly replacing lexical items on either nominal, verbal orpositional sites, or by randomly swapping of any pair ofphrases in them. Mutation types and their ratios in survey 2
are given in Table 1.
Survey 2 exploited the 27 Japanese verbs in Table 2.
Table 2: Verbs used in survey2
v.indexv18v22v25v26
v40
v44v111
v116
v131
v145
v155
v183
v210
v326
v338
v345
v377
v447
v450v470v713v807v829v831v958v1147v1197
sum

Table 1: Mutation types and ratios in survey 2 stimuli
edit.typeo(riginal)s(wapping)p(ostposition)v(erb)n(ominal)
sum

count
36
70
5865
71
300

ratio
0.12
0.23
0.19
0.22
0.24
1.00

Effective exploration into the possibility space of acceptability requires careful manipulation of stimuli. It is evident, however, that humans are not capable of producing alarge number of potentially and actually deviant sentences
systematically, and without theoretical biases. This necessitates automatic method. The procedure we followed was
the semi-automatic method described in [3]. In summary, Contact person is the first author, who can be reached at
kow.k@ks.kyorin-u.ac.jp.

v.form
聞いた
行った
入れた
話しかけた
教えた
感じた
伝えた
答えた
探した
聞こえた
繰り返した
届いた
遊んだ
黙った
負けた
助けた
表れた
つないだ
載った
襲った
間違った
直した
助かった
届けた
習った
知り合った
感染した

count
23
1
16
1
3
10
17
1
1
17
9
8
17
2
1
17
1
8
18
29
923
9
17
29
12
1
300

ratio
0.077
0.003
0.053
0.003
0.010
0.033
0.057
0.003
0.003
0.057
0.030
0.027
0.057
0.0070.003
0.057
0.003
0.027
0.060
0.097
0.030
0.077
0.030
0.057
0.097
0.040
0.003
1.000

To start with, candidates were selected semi-randomly
from the verb list of NINJAL-LWP for BCCWJ1) . They
were manual coded with 6 attributes, F1: [effect is physical], F2: [effect is mental], F3: [effect is social], F4: [eventis interactive], F5: [event is interactional], F6: [effect is intended (if subject is agent)]. Kuroda [5] gives a detailed report of the data and analysis. Then, we obtained the Formal
Concept Analysis [1] presented in Figure 1, where verbs
with red frames correspond to the candidates for the supplement to the 9 verbs used in survey 1.
Candidate sentences were constructed by manually fill1) http://nlb.ninjal.ac.jp/search/― 253 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

within the framework of social survey. Thus, we collected
the 11 social attributes specified in (3). Layered analysis
using those social attributes is planned, but not carried outyet.
(3) Q1. [Age] How old are you now?; Q2. [Gender] Which gender is yours? (Male/Female/Unsure); Q3. [Native language]
Is Japanese your mother tongue? (Yes/No/I don’t know); Q4.
[Lived places] Which area have you lived in the past? Answer as many prefectures as necessary); Q5. [Experienceof living abroad] Have you lived for more than one year inan area where people don’t speak Japanese?); Q6. [Numberof known languages] How many languages have you everlearned? Answer the number of them irrespective of theirduration; Q7. [Length of foreign language learning] Howlong did you spend learning foreign languages? Answer innumber of years); Q8. [Daily contact with foreigners] Do
you have regular contacts with those who speak foreign languages?); Q9. How many books do you read in a month,roughly?; Q10. How many years did you spend learning
after elementary school? Answer in the number of years;
Q11. Which intellectual orientation describes you better?
(Science-oriented, relatively science-oriented, neutral, relatively humanity-oriented, humanity-oriented)

Figure 1: FCA of sampled verbs
ing in the four lexical gaps in the constructions in (1).
(1) P1: ____ が ____ で ____ に ____ と V した．
P2: ____ が ____ で ____ に ____ を V した．
P3: ____ が ____ で ____ を ____ に V した．
P4: ____ が ____ で ____ から ____ を V した．
P5: ____ が ____ で ____ と ____ を V した．

Each stimulus group contains 30 sentences, of whichs281 and s282 were carried over from survey 1, and wereused as common ground. 280 sentences were randomly
Some facts. The ratios of male and female responders aresampled from the candidates to fill in the other 28 sentences.
nearlyequal. The ages of responders range from “under 13Table 3 presents samples from gr0.
yo” to “over 70 yo,” as illustrated in Figure 2. Age ranges
are far from evenly distributed, with oversampling at 20s
2.2 Rating task
Questionnaire was implemented by Google Forms. Each and 30s. The rank distribution of lived places (in terms ofgroup consists of 11 questions for personal and/or social prefecture) is as illustrated in Figure 3. It is safe to say that
attributes specified in (3), followed by 30 questions for ac- places are well diversified, as far as we take Zipf’s law into
ceptability rating. On rating the stimuli, participants, whom account.
we call “responders,” are asked to choose one of the fourchoices on a scale in (2).2)(2) 0. natural and easy to understand [違和感がなく自然に理解できる文]; 1. more or less deviant but comprehensible [違和感を感じるが理解可能な文]; 2. deviant and difficult to understand [違和感を感じて理解困難な文]; 3. quite unnaturaland incomprehensible [不自然な理解不能な文]

Figure 2: Age distribution over 10-year ranges
Prefixed indices roughly encode the degrees of deviance.
Note that the questionnaire does not directly ask if suchand such sentence are acceptable (or not). We avoided it for
two reasons. First, the very notion of acceptability is not as
simple as ordinary people can fully appreciate. Second, weaimed at factoring out of acceptability, by explicitly trying
to dissociate semantic anomaly from grammatical (morphological and/or syntactic) anomaly.
2.3 Social attributes collected
ARDJ is primarily a project of data collection targeting
acceptability ratings. Aware of the social nature of grammar, we found it necessary to embed our data collection2) One of the responders pointed out that “違和感を感じる” was a wrong
expression, and claimed that this error debunked the authenticity of thesurvey.

Figure 3: Rank distribution of lived places

2.4 Outlier removal: data cleaning
Outliers in survey are not rare because responders always
have incentive to play “satisficing” [2]. This means that― 254 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Table 3: Sample stimuli in gr0
s.indexs10s30s50s70s100s140s170s210s250
s281.0s282.0v.idexv25v831v831v345v470v958v145v345v958v1147
v44

pattern
P4
P3
P3
P1P4P5
P3
P1
P1
P1
P4

author
3
1
1
3
2
3
1
3
1
1
1

typen
nv
sov
v
nspn

gr
gr0
gr0
gr0
gr0
gr0
gr0
gr0
gr0
gr0
gr0
gr0

ver
A
A
A
A
AA
A
AA
AA

gr.index
13
5
7
1014
17
21252930

sentence
担当者が携帯で出張もさから電話を入れた。
伝書鳩が戦地で進攻を司令官に届けた。
伝書鳩が戦地で戦況を司令官に送り届けた。
続編で宿敵がピンチに主人公と助けた。
暴漢が鋭利な刃物で背後から人を襲った。
弟が家で妹と料理を習わせた。
ランナーが路上で悲鳴を夕暮れ時に聞き取れた。
宿敵が続編で苦境に主人公と助けた。
医学生が解剖実習で看護師と医師に習った。
夫が職場で真夜中に妻へ知り合った。
学生が合格発表の場で足下から幸福を感じた。

survey data always comes with outliers, at least potentially.
Analysis goes awry when their effect is ignorable.
In survey, there are predictably two typical ways of playing satisfice, and fortunately, the two kinds of satisficing
can be easily detected by statistical assessment of data. Insurvey, case 1) a responder always returns the same value
to all questions; or case 2) a responder chooses values fullyor nearly randomly. The first case can be detected if someone’s responses do not have enough standard deviation. The
second case can be detected if someone’s responses shows
too much randomness, which can be measured using suchmeasures as Mahalanobis distance.
Figure 4: Selected response patterns (last 6 stimuli for gr0,Table 4: Distribution of outliers (M-dist discard rate = 0.05) gr2, g5, gr7)
ID
gr0
gr1
gr2
gr3
gr4
gr5
gr6
gr7
gr8
gr9
sum

#sd
658
57
103
57460

#M-dist
8
98888
9888
82

#shared
1
0
0
0
0
0
0
0
00
1

#unified
13
14
16
13
15
18
12
13
15
12
141

#effectives
153
160
153
155
151
145
166
155
147
152
1,538

roughly the same responses, as are the case with s281 ands282, which are common to all groups; ii) despite this, responses are rather varied: different sentences are likely tocause different responses; iii) still, responses are far form
random. These findings, if combined, suggest a hypothesisthat there seem to be only a limited number of classes of responses, thereby predicting a typology of response patterns.

We used double filters to remove the outliers in the data 3. Analysis of responses
at hand: responses in each group are discarded if either not
The suggested existence of response typology naturally
0.5 < sd < 1.6 or out of the 95% range3) of Mahalanobis begs the question, “How to recognize response classes, anddistance. Table 4 gives a brief description of the statistics.
associate each to other?”
Hierarchical clustering is a popular method for grouping
2.5 Overview of the obtained data
data. Principal Component Analysis (PCA) is a popular
We have 300 sentences to examine and it is clearly unmethod for revealing a simple geometry in the data. An
realistic to give all the plots here. So, we present only 24
R package FactoMineR [4] provides the combination of thecases nearly randomly selected, given in Figure 4.
two. We decided to use it to reveal the hidden structures inThe selection here is nearly random, because, first, the 10
the data.
groups were randomly constructed and second, the presentation order of the 30 stimuli was randomized. What is not
3.1 Standardizing responses
random about this selection is inclusion of s281 and s282,
We needed one intermediate step, however. Note that4)
which are common to all stimulus sets.
gr0, gr1, . . . , gr9 are different data sets, and they cannot
Inspection of the 300 responses, sampled in Figure 4,
be directly compared. Comparison of them requires stanleads to the following insights: i) the same stimuli produce
dardization. It was carried out in the following way: sen3) This discard rate was determined through trial and error. It is possible
tences in each set were transformed into binned density arto increase the number of common outliers, but discard rate needs to go up
ray, pŒ0; 1/; pŒ1; 2/; pŒ2; 3/; pŒ3; 4/, over four rating ranges
to 0.3 if we want to each group has at least one common between sd-based
rŒ0; 1/; rŒ1; 2/; rŒ2; 3/; rŒ3; 4/. All sentences in our dataset
and Mahalanobis-based detections.
4) s281.i and s281.i encode the stimuli in group i .
correspond to quadruples .pŒ0; 1/; pŒ1; 2/; pŒ2; 3/; pŒ3; 4//.

― 255 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

These are commensurable even if the sets of responders are to mind, but the most convincing one would be that Dim
different for groups.
2 encodes semantic and/or syntactic complexity that often
blurs the judgment. At least, this seems to provide a natural
3.2 PCA and hierarchical clustering
account for the parabolic shape of the area of plotted points.

4. Discussion
We have seen that 1) unsupervised grouping (i.e., hierarchical clustering) of responses resulted in three clusters rather than two clusters; 2) dimension reduction of responses revealed that acceptability is encoded by Dimension 1, with correlation with the polar opposition between
rŒ0; 1/ and rŒ2; 3/, but Dimension 2 cannot be ignored. The
first point confirms that acceptability cannot be properly
modeled as a categorical judgment. The second point, coupled with the first point, strongly suggests that acceptability
Figure 5: HCxPCA of combined responses in gr0-gr9
is not a monolithic notion and is likely to be affected by anumber of factors.
Building on standardized responses, a multivariate analThe analysis of phrase 1 of survey 2, which is not deysis was conducted where PCA was combined with hierar- scribed in this paper, gave us different profiles in terms ofchical clustering, resulting in visualization in Figure 5. In PCA/Factor Map. The shape of data points is not parabolic.
this, we recognize three major classes of stimuli: clusters The two results from phases 1 and 2 need to be reconciled.
1 (in black, of acceptable stimuli), cluster 2 (in red, of undecidable stimuli) and cluster 3 (in green, of unacceptable 5. Conclusionstimuli). Clusters 2 and 3 form a larger cluster.
Analysis of responses from a large scale survey, throughits unsupervised clustering and dimension reduction, revealed some true properties of acceptability judgment. Our
results are robust, based on the unbiased nature of stimuli,
scale of survey and controlled varieties of responders.
Let us close with future plans. First and foremost, wemake the obtained data public. Second, we try out layered
analyses. Does the result involve gender difference? Wesuspect so. Does the result involve location/dialect difference? We suspect so. They are definitely worth trying.
Figure 6: Factor Map/PCA of combined responses

Acknowledgement
The first dimension (Dim 1) of PCA roughly corresponds
to the polar opposition of rŒ0; 1/ and rŒ2; 3/. The seconddimension (Dim 2) is mildly encoded by rŒ1; 2/, and weaklyby rŒ3; 4/.
We should point out here that response structures local togr0, . . . , gr9 are similar to the one in Figure 6, thereby predicting that the geometry is an invariant self-similar image.

This research was supported by JSPS through Grant-inAid (16K13223).

References[1] Bernhard Ganter and Rudolf Wille. Formal Concept Analysis:
Mathematical Foundations. Springer-Verlag, Berlin, 1999.
[2] Jon A. Krosnick. Response strategies for coping wiht the cognitive demands of attitude ameasure in surveys. Applied Cognitive Psychology, 5(3):213–236, 1991.
[3] Kow Kuroda, Hikaru Yokono, Keiga Abe, Tomoyuki
Tsuchiya, Yoshihiko Asao, Yuichiro Kobayashi, Toshiyuki
Kanamaru, and Takumi Tagawa. Development of Acceptability Rating Data of Japanese (ARDJ): An initial report. InProc. of the 24th Annual Meeting of the Association for NLP,
pp. 65–68, 2018.

Figure 7: PCA/Factor Map of responses in gr2 and g6[4] Sebastien Le, Julie Josse, and Francois Husson. FactoMineR:
An R package for multivariate analysis. J. of Statistical Software, 25(1):1–18, 2008.

The interpretation of Dim 1 is straightforward. It encodes
the degree of deviance from right to left, or of acceptability, [5] 黒田 航. 意味の社会性を意識した動詞の分類とその理論的含意. In 認知科学会第 35 回大会発表論文集, pp. 65–68,
from left to right. In contrast, the interpretation of Dim 2 is2018.
not as simple as Dim 1. A few likely interpretations come― 256 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.