言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Reassessing PCA of Acceptability Rating Data for Japanese (ARDJ) datausing kernel Multivariate Analysis
Kow Kuroda
kow.k@ks.kyorin-u.ac.jp
Medical School, Kyorin University

1.

Introduction
Acceptability Rating Database of Japanese (ARDJ) [4]is a research project that aims at providing normative datathat prepares for “evidence-based linguistics” [12]. The
project released a dataset called s2u (for “survey 2 unified”) that consists of 1,880 responses to 300 stimuli sentences by 877 individuals, as well as answers to 11 questions for social attributes like age, gender, lived places.1)
Recent work [5] presented a preliminary analysis of thedata thus acquired, but it was likely that it suffered froma flaw. Four kinds of data were prepared: 1) raw countdata, 2) normalized data2) , 3) scaled by sentence without centering, and 4) scaled by sentence with centering.
Application of PCA to these data yields the results in Figure 1, where colors are added to identify clusters membership3) and contours are added to indicate density.

Figure 1: PCA of s2u data, raw count, normalized, ands-wise scaled (uncentered and centered)

Another possibility worth exploration was unsupervised metric learning such as Isometric Mapping (Isomap) [11] and Locally Linear Embedding(LLE) [9], which serves as kernel-based versions ofMDS. RDRToolbox [1] package for R provides them.
In what follows, I present results of kernel based PCA
in 2. (ones using Gaussian RBF in §2.1 and ones using
Laplacian RBF in §2.3), ones using Isomap in §3.1, andones using LLE in §3.2. Due to space limitation, presentation of results is selective.
2. Applying Kernel PCA
2.1 RBF kernels
Gaussian kernel uses exp(−σ ||x −x ′||2 ), and Laplace kernel uses exp(−σ ||x − x ′||) for radial base functions (RBFs),
as plotted with varying σ s in Figure 2.

Figure 2: Gaussian and Laplacian RBFs with varying σ ’s

The horseshoe-like shape of data points, common toall plots here, is called the “horseshoe” effect, or Guttman 2.2 Gaussian kernel PCA
effect [2, 8]. It is not peculiar to PCA. Results of Corre- 2.2.1 Gaussian PCA on raw count data
Gaussian PCA applied to the raw count data yieldedspondence Analysis (CA) and Multi-dimensional Scaling
theresults in Figure 3, where isolation seems successful.
(MDS) of the same data take the same shape. In otherwords, normal methods for MVA suffer from this.
This effect is considered an artifact arising from theresponse design (e.g., Likert or Guttman scales). Quite anumber of techniques [7, 8] were proposed to remove
or at least attenuate this effect, but most of them require reimplementation of basic functions used to calculate variance. I wondered if there was an easier way tomake a shortcut and came up with the idea of applying
kernel-based multivariate analysis (kMVA) to this end.
The most obvious possibility to explore is to use kernel PCA [10]. For example, kernlab package [3] for R
Figure 3: Gaussian PCA of s2u raw count data
provides Gaussian and Laplacian kernels among others.4)
1) Thisdataset,

along

with

someothers,

isavailable

athttps://kow-k.github.io/Acceptability-Rating-Data-of-Japanese/
2) Normalization means conversion of range-wise raw counts intodiscretized density distribution.
3) Recognition of clusters inside the dimensionally reduced data is always done with X-means provided by clusternor package [6].
4) Only results from Gaussian and Laplacian kernels are discussed in

2.2.2 Gaussian PCA on s-wise scaled, uncentered data
Gaussian PCA applied to the the sentence-wise scaled,
uncentered data yielded the results in Figure 4.
this paper, though other options like polynomial, ANOVA RBF, hyperbolic, Bessel kernels were tested. R version 3.5.3 was used for all runs.

― 1523 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Figure 7: Laplacian PCA of s2u raw count data

Figure 4: Gaussian PCA of s2u s-scaled, uncentered data

2.3.2 Laplacian PCA on s-wise scaled, uncentered data
Laplacian PCA applied to the scaled, uncentered data
yielded the results in Figure 8.

2.2.3 Gaussian PCA on s-wise scaled, centered data
Gaussian PCA applied to the sentence-wise scaled,
centered data yielded the results in Figure 5.

Figure 8: Laplacian PCA of s2u s-scaled, uncentered data

2.3.3 Laplacian PCA on s-wise scaled, centered data
Laplacian PCA applied to the scaled, centered data
yielded the results in Figure 9.

Figure 5: Gaussian PCA of s2u s-scaled, centered data
2.2.4 Gaussian PCA on normalized data
Gaussian PCA applied to the normalized data yielded
the results in Figure 6. Effective values for σ are verylarge, compared to other cases, suggesting that normalization is harmful.
Figure 9: Laplacian PCA of s2u s-scaled, centered data

2.3.4 Laplacian PCA on normalized data
Laplacian PCA applied to the normalized data yielded
the results in Figure 10.

Figure 6: Gaussian PCA of s2u normalized data

2.2.5 Summary of Gaussian PCA results
In terms of σ optimization, raw count data and normalized turned out to be harder to handle with Gaussian
RBF kernel: raw data requires extremely small values,
and normalized data required rather large values. Scaleddata, with or without centering, seem to give us better
results.

2.3.5 Summary of Laplacian RBF
In terms of σ optimization, Laplacian kernel turned outto be easier to handle than Gaussian kernel. But it can be
more selective in terms of data: the normalized data looks
incompatible with Laplacian kernel.

2.4 Gaussian vs Laplacian kernels
On comparing the results from Gaussian and Lapla2.3 Laplacian kernel PCA
cian kernels, it would be safe to say that Laplacian kernel
2.3.1 Laplacian PCA on raw count data
performs better. In addition, selection of effective values
Laplacian PCA applied to the raw count data yielded for σ is harder with Gaussian kernel, probably because it
the results in Figure 7.
responds to a squared term ||x − x ′||2 .

― 1524 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Figure 13: Isomap of s2u s-scaled, centered data

Figure 10: Laplacian PCA of s2u normalized data
3.

Applying Isomap and LLE
Figure 14: Isomap of s2u normalized data
Isometric Mapping (Isomap) [11] and Locally Linear
Embedding (LLE) [9] form another class of kernel-based
3.2 Locally Linear Embedding (LLE)
MVA, serving as kernel-based versions of MDS.
Like Isomap, LLE [9] has a parameter k that encodes
thenumber of nearest neighbors to consider. With our
3.1 Isometric Mapping (Isomap)data,
k cannot exceed 3 on whichever data.
3.1.1 Isomap on raw count data
Isomap applied to the raw count data yields the results
3.2.1 LLE on raw count data(k = 5 was accepted) in Figure 11.
LLE applied to the raw count data gave the results (k =4 was not accepted) in Figure 15.

Figure 11: Isomap of s2u raw count data
Figure 15: LLE of s2u raw count data
3.1.2 Isomap on s-wise scaled, uncentered data
Isomap applied to the sentence-wise scaled, uncentereddata gave us the results in Figure 12.
3.2.2 LLE on s-wise scaled, uncentered data
LLE applied to the sentence-wise scaled, uncentereddata gave us the results (k = 4 was accepted, but k = 5was not) in Figure 16.

Figure 12: Isomap of s2u s-scaled, uncentered data

3.2.3 LLE on s-wise scaled, centered data
LLE applied to the sentence-wise scaled, centered data
gave us the results in Figure 17.
3.2.4 LLE on normal data
LLE applied to the normalized data gave the results (k =4 was not accepted) in Figure 18.

3.1.3 Isomap on s-wise scaled, centered data
Isomap applied to the sentence-wise scaled, centereddata gave us the results in Figure 13.
3.2.5 Summary of LLE results
LLE produces less stable and coherent results than
3.1.4 Isomap on normalized data
Isomap in that it gives different results to raw count, sIsomap applied to the normalized data gave us the re- wise scaled with and without centering, and only withsults (k = 5 was accepted) in Figure 14.
smaller values like as 2, 3.
3.1.5 Summary of Isomap results
3.3 Isomap vs LLE
Based on comparing results from Isomap and LLE, it
Isomap brings stable and coherent analysis in that it
produces roughly the same results on raw count, s-wise would be safe to say that Isomap performs significantlybetter. Isomap is easier to use and provides better analyses.
scaled with and without centering, with larger k.

― 1525 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

5. Conclusion
The Guttman effect shown by PCA, CA and MDSof ARDJ s2u data was effectively removed by Isometric
Mapping (Isomap). Kernel PCA was shown to be also
effective, but to a limited extent, due to messiness in optimization for hyper-parameters such as σ .
Figure 16: LLE of s2u s-scaled, uncentered data

Acknowledgement
This research was supported by JSPS through Grantin-Aid (16K13223).
References[1] Ch. Bartenhagen. RDRToolBox: A package for nonlinear dimension reduction with Isomap and LLE, 2014. R package
version 1.30.0.

Figure 17: LLE of s2u s-scaled, centered data
4.

Discussion[2] D. W. Goodall. Objective methods in the classificationof vegetation. III. An essay on the use of factor analysis.
Australian Journal of Botany, 2:304–324, 1954.
[3] A. Karatzoglou, A. Smola, K. Hornik, and A. Zeileis.
kernlab — An S4 package for kernel methods in R. J.
of Statistical Software, 11(9):1–20, 2004.

My motivation for the investigation reported here wasto check if any of kernel MVA’s can remove the Guttman [4] K. Kuroda, H. Yokono, K. Abe, T. Tsuchiya, Y. Asao, Y.
effect [2, 8]. I tested kernel PCA [10] with various kernels
Kobayashi, T. Kanamaru, and T. Tagawa. Development(Gaussian, Laplacian, ANOVA RBF, polynomial, Bessel),of Acceptability Rating Data of Japanese (ARDJ): An iniand two methods for metric learning, i.e., Isomap [11]tial report. In Proc. of the 24th Ann. Meeting of the Assoc. forNLP, pp. 65–68, 2018.
and LLE [9]. Based on the results reported and some others omitted here, I conclude that the attempt was success- [5] K. Kuroda, H. Yokono, K. Abe, T. Tsuchiya, Y. Asao, Y.
ful if proper methods are chosen and hyper-parameters
Kobayashi, T. Kanamaru, and T. Tagawa. Insights from(e.g,. σ , k) are properly set.
a large scale web survey for Acceptability Rating Data forJapanese (ARDJ) project. In Proc. of the 25th Ann. Meeting
Which ones are successful, however? By comparing
for the Assoc. of NLP, pp. 253–256, 2019.
the results, the most promising remedy for the Guttman
effect seems to be achieved by Isomap for two reasons. [6] D. Mhembere, D. Zheng, C. E. Priebe, J. T. Vogelstein,and R. Burns. knor: A NUMA-optimized in-memory,First, Isomap seems to yield results similar enough to thedistributed and semi-external-memory k-means library.
results of normal PCA: no drastic transformations are deHigh-Performance Parallel and Distributed Computing, 26,tected. This is a good property. Second, Isomap gives2017.
us more robust results in comparison with results fromkernel PCA, which tend to be “fragile” in that they are [7] J. Podani and I. Miklós. Resemblance coefficients and thehorseshoe effect in Principal Coordinates Analysis. Ecolhighly sensitive to different values for hyper-parametersogy, 83(12):3331–3343, 2002.
(especially σ ): a lot of tweaking was required to obtainreasonable results in kernel PCA. Metric-based methods [8] J. Rijckesvorsel. The Application of Fuzzy Coding andHorseshoes in Multiple Correspondence Analysis. DSWO
like Isomap and LLE are free from this sort of annoyance,Press, 1987.
though LLE turned out to have less descriptive power.
The performance of kernel PCA would be best char- [9] S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323–2326,
acterized as a mixture of wheat and chaff. If Laplacian
2000.
and Gaussian kernels are compared, the former seems towork better with my data, especially in exploration for [10] B. Schölkopf, A. Smola, and K.-R. Müller. Nonlinear
component analysis as a kernel eigenvalue problem. Neueffective values for σ .
ral Computation, 10, 1998.
[11] J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global
framework for nonlinear dimensionality reduction. Science, 290:2319–2323, 2000.
[12] 黒田 航. 証拠に基づく医療 (EBM) との比較を通じて理論言語学の方法論を見直す. In 第 16 回日本認知言語学会発表論文集, pp. 580–585, 2016.

Figure 18: LLE of s2u normalized data

― 1526 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.