Discovering Universal Geometry in Embeddings with ICA
Authors: Hiroaki Yamagiwa
Momose Oyama
Hidetoshi Shimodaira
Venue: EMNLP 2023 Main (2023)
Keywords: Embeddings
Independent Component Analysis
Principal Component Analysis
Cross-lingual
Interpretability
Isotropy
Whitening
TL;DR: We studied the independent semantic components that are universal to embeddings of various languages and image models.

Abstract: This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.

Discovering Universal Geometry in Embeddings with ICA
Hiroaki Yamagiwa*
Momose Oyama* 1,2
Hidetoshi Shimodaira" 1.2
Kyoto University
RRIKNN
hiroaki yamagiwaesys.....o.o..... wttp:/esss.....ytoooo.. www..ikooo.........
oyama www.se@sss.eesmommmmomm w...:/eess....k.oooouuuu w.o.o-y.occc..ooo jp, shimo@i kyoto-u. ac. jp
Abstract
meat
cars
pofatces
grilld
Gerari
This study utilizes Independent Component Analysis (ICA) to unveil a consistent seman- tic structure within embeddings of words or images. Our approach extracts independent mantic components from the embeddings of pre-trained model by leveraging anisotropic in- formation that remains after the whitening pro- cess in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure the geometric patterns of embeddings enhances under- standing of the representations in embeddings.
rice
Frorda
tuyota
tomata
0.8
28: (cors)
lambborghini
cheeses
pizza
SHZURI
frooderm
terran
cans
0.6
Sanue
Vantce
honda
nissn
quattro
34: (ital am)
lambörghini
tim
-..4
fim
Heatmap words
scorsese
Deher
comedies
camerr
felini
-02
scorsesa
Jurosawa
d
comoddis
modinn
anime
mmlyazaki
japen
japanese-
scra
Japan'
Suzuk!
sushi
18
[apanese]
Figure 1: (Left) Heatmap of normalized ICA- transformed word embeddings shown a selected of five out of 300 dimensions. Each axis has its own meaning, and the meaning of word is repre- sented as combination of a few axes. For example, ferrari [cars] [italian] and kurosawa [film| [japanese]. (Right) Scatterplots of normalized ICA- transformed word embeddings for the (/italian), [cars)) axes and ((japanese], [film]) axes. The word embed- dings in the heatmap were plotted as black dots. The words are highlighted with colors corresponding their respective axes. For more details, refer to Section 3 and Appendix
1 Introduction
Embeddings play a fundamental role in represent- ing meaning. However, there are still many aspects of embeddings that are not fully understood. For instance, issues such as the dimensionality of em- beddings, their interpretability, and the universal properties shared by embeddings trained with dif- ferent algorithms or in different modalities pose important challenges in practical applications.
Discussions and research have explored the top- ics of low-dimensionality and interpretability of embeddings (Goldberg, 2017). Proposals have been made for learning and post-processing meth- ods that incorporate constraints. aiming achieve sparse embeddings or acquire semantic axes. Addi- tionally, research has focused on aligning embed- dings trained in different languages through various transformations. However, in contrast to this on- going trend, our specific focus lies on the intrinsic independence present within embeddings.
In this research, we post-process embeddings us- ing Independent Component Analysis (ICA), pro- viding a new perspective on these issues (Hyväri- nen and Oja, 2000). There are limited studies that have applied ICA to a set of word embeddings, with only few exceptions (Lev et al., 2015; Al- bahli et al., 2022: Musil and Mareček, 2022). There has also been a study that applied ICA to word- context matrices instead of distributed represen- tations (Honkela et al., 2010). Although it has received less attention in the past, using ICA al-
The first two authors contributed equally to this work. Our code and data are available at https: //gi thub. com/shimo- htm/shimo-labbuue hne//nniisrall Geometry wiw-witthie

---

English
Spanish
Russian
Arabic
Hindi
Chinese
Japanese
0.8
0.8
0.2
0.0
(a) Normalized ICA-transformed word embeddings for seven languages.
English
Spanish
Russian
Arabic
Hindi
Chinese
Japanese
1.0
0.6
20 40 50 B0
40 90 80
20 40 60 B0
20 42 16 80
20 40 60
20 40 00 80
20 40 60 0U
0.4
(b) Normalized PCA-transformed word embeddings for seven languages.
The rows represent fastText embeddings of words for seven languages transformed by (a) ICA and (b) PCA. Each embedding is normalized to have norm of for visualization purposes. The components from the first 100 axes of embeddings are displayed in the upper panels, lower panels. For each axis of English embeddings, the top 5 words were chosen based on their component values, and their translated words were used for other languages. Correlation coefficients between transformed axes were utilized to establish to align with the other languages when matching the axes, and the aligned axes were rearranged in descending order according to the average correlation coefficient, as detailed in Section and Appendix
Figure 2:
first axes are magnified in the
correspondences and carry out axis permutations; English served as the reference language
lows us to extract independent semantic compo- nents from set of word embeddings. By leverag- ing these components the embedding axes, anticipate that each word can be represented a composition of intrinsic (inherent the original embeddings) and interpretable (sparse and consis- tent) axes. Our experiment suggests that the num- ber of dimensions needed to represent each word is considerably less than the actual dimensions of the embeddings, enhancing the interpretability.
when the embedding algorithms modalities (i.e.. word image) different.
Principal Component Analysis (PCA) has tra- ditionally been used to identify significant axes in terms of variance, but it falls short in compar- ison ICA; the patterns less clear for PCA in Figs. 2b and 3b. Embeddings are known be anisotropic (Ethayarajh,, 2019), and their isotropy can be greatly improved by post-processes such as centering the mean, removing the top princi- pal components (Mu and Viswanath, 2018), stan- dardization (Timkey and van Schijndel, 2021), whitening (Su et al., 2021), which can also lead to improved performance in downstream tasks. While the whitening obtained by PCA provides isotropic embeddings regarding the mean and covariance of components, notably, ICA has succeeded in discov- cring distinctive axes by focusing on the anisotropic information left in the third and higher-order mo-
Fig. shows an example of independent seman- tic components that are extracted by ICA. Each axis has its own meaning, and word is repre- sented combination of few axes. Further- more, Figs. 2a and 3a show that the semantic found by ICA are almost common across different languages when we applied ICA individually to the embeddings of each language. This result is not limited to language differences but also applies

---

1.0
2012: Luo al., 2015), or é-reguularzzaton (Sun et at.. 2016). Additionally, the sense polar approach designs objective functions to make each axis of BERT embeddings interpretable at both ends (En-
6.5
650
6d5
600
-1.5
-550
dn 66 (0
gler et al., 2022; Mathew et al., 2020).
ICA
However, our study takes a distinct approach. We do not rely on explicit constraints utilized in the aforementioned methods. Instead, we leverage transformations based on the inherent information within the embeddings. Our motivation aligns with that of Park al. (2017) and Musil and Mareček (2022), aiming to incorporate interpretability into each axis of word vectors. Similar to previous studies (Honkela et al., 2010; Musil and Mareček, 2022), we have confirmed that interpretable axes are found by applying ICA to set of embeddings.
40
40
20
Privaded OuP- (01444
Iemuudd F.A.
(h) PCA
Figure 3: Cross-coreelation coefficients for ICA and PCA-transformed embeddings. The clear diagonal lines for the ICA-transformed embeddings indicate good alignment. (Left) English fastText and Spanish fastText shown in Fig. 2. (Middle) Tast and BERT shown in Fig. (Right) Image models of ViT-Base and ResMLP 12 shown in Fig. 8.
2.2 Cross-lingual embeddings
Cross-lingual mapping. To address the task of cross-lingual alignment, numerous methodologics have been introduced to derive cross-lingual map- pings. Supervised techniques that leverage transla- tion pairs as training data have been proposed, such as the linear transformation approach (Mikolov et al., 2013b). Studies by Xing et al. (2015) and Artetxe et al. (2016) demonstrated enhanced per- formance when constraining the transformation matrices to be orthogonal. Furthermore, Artetxe et al. (2017) proposed method for learning trans- formations from a minimal data As for un- supervised methods that do not leverage transla- tion pairs for training, Lample et al. (2018) pro- posed an approach incorporating adversarial Iearn- ing, while Artetxe et al. (2018) 8) introduced a ro- bust self-learning method. Additionally, unsuper- vised methods employing optimal transportation have been presented: Alvarez-Melis and Jaakkola (2018) introduced a method utilizing the Gromov- Wasserstein distance. while studies by Grave et al. (2019) and Aboagye et al. (2022) suggested meth- ods that employ the Wasserstein distance.
bost
octino
trece flo
plamas
plants
flored arbbiies
alimentta
food
English
Spanish
Rotation and reflection
Figure 4: Illustration of the spiky shape of the word em- bedding distributions in nigh-dimensional space. This spiky shape is actually observed in the scatterplots of the normalized CCA-trannformed embeddings for seven languages, as shown in Fig, 15 in Appendix C.
ments of whitened embeddings.
2 Background
Interpretability in word embeddings The interpretability of individual dimensions in word embedding is challenging and has been the subject of various studics.
2.1
Multilingual language models. Studies have demonstrated that a single BERT model, pre- trained with a multilingual corpus, acquires cross- lingual grammatical knowledge (Pires et al., 2019; Chi et al., 2020). Further research has also been conducted to illustrate how such multilingual mod- els express cross-lingual knowledge through em- beddings (Chang et al., 2022).
A variety of studies have adopted explicit con- straints during re-training or post- post-processing of em- beddings to improve sparsity and interpretability. This includes the design of loss functions (Arora al., 2018), the introduction of constraint con- ditions by sparse overcomplete vectors (Faruqui et al., 2015), and re-training using k-sparse au- toencoders (Makhzami and Frey, 2013; Subrama- nian et al., 2018), sparse coding (Murphy et al.,
Our approach. These cross- coss--lnguall studies, even the 'unsupervised" mapping, utilize embeddings

---

from multiple languages for training. Unlike these. we apply ICA each language individually and identify the inherent semantic structure each without referencing other languages. Thus ICA, as well as PCA, is an unsupervised transformation in stronger sense. In our study, the embeddings from multiple languages and the translation pairs are used solely to verify that the identified semantic structure is shared across these languages. While it is understood from previous research that there exists a shared structure in embeddings among mul- tiple languages (i.e., their shapes are similar), the discovery in our study goes beyond that by reveal- ing the universal geometric patterns of embeddings with intrinsic interpretable axes (i.e., clarifying the shapes of embedding distributions; see Fig. 4).
3.2 ICA-transformed embeddings In Independent Component Analysis (ICA), the goal is to find a transformation matrix B с RRdd such that the columns of the resulting matrix S E Raxd are as independent as possible. This transfor- mation is given by:
S
The columns of S are called independent compo- nents. The independence of random variables is stronger condition than uncorrelateddesss and when random variables are independent, it implies that they are uncorrelated with each other. While both PCA and ICA produce whitened embeddings, their scatterplots appear significantly different, as observed in Fig. 5; refer to Appendix B for more details. While PCA only takes into account the first and second moments of random variables (the mean vector and the variance-covariance matrix), ICA aims to achieve independence by incorporat- ing the third moment (skewness), the fourth mo- ment (kurtosis) and higher-order moments through non-linear contrast functions (Fig. 6).
3
ICA: Revealing the semantic structure in the geometric patterns of embeddings
We analyzed word embeddings using ICA. It was discovered that they possess inherent interpretabil- ity and sparsity. ICA can unveil these properties of embeddings.
In the implementation of FastICA', PCA is used as a preprocessing step for computing Z, and
3.1 PCA-transformi embeddings
S ZRica
Before explaining ICA, we briefly explain PCA, widely used for dimensionality reduction and whitening, sphering, of feature vectors.
is actually computed?, and we seek an orthogonal matrix Rica that makes the columns of S as in- dependent as possible (Hyvärinen and Oja, 2000). The linear transformation with an orthogonal ma- trix involves only rotation and reflection of the z; vectors, ensuring that the resulting matrix S is also whitened, meaning that the embeddings of ICA, well as those of PCA, are isotropic with respect to the variance-covariance matrix (Appendix A). According to the central limit theorem, when multiple variables are added and mixed together, they tend to approach a normal distribution. There- fore, in ICA, an orthogonal matrix Rica is com- puted to maximize measure of non non-Gaussianicg for each column in S, aiming recover indepen- dent variables (Hyvärinen and Oja, 2000). This idea is rooted in the notion of 'projection pur- suit' (Huber, 1985), a long-standing idea in the field. Since the normal distribution maximizes en- tropy among probability distributions with fixed mean and variance, measures of non non-Gausee Non-Gaussianity are interpreted as approximations of neg-entropy.
The pre-trained embedding matrix represented as X E Randd where X is assumed to be cen- tered; it is preprocessed so that the mean of each column is zero. Here, n represents the number of embeddings, and d is the number of embedding di- mensions. The i-th row of X. denoted as x; E Rd corresponds to the word vector of the i-th word, or the embedding computed by an image model from the i-th image.
In PCA, the transformed embedding matrix Z E Raxd is computed using algorithms such as Singu- lar Value Decomposition (SVD) of X. This pro- cess identifies the directions that explain the most variance in the data. The transformation can be ex- pressed using a transformation matrix E IRdxd as follows:
Z XA.
The columns of Z called principal components. The matrix Z is whitened, meaning that each col- umn has variance of and all the columns are uncorrelated. In matrix notation, Z Z/n = La- where Ia E dd represents the identity matrix.
'ee used FastCA in Scikit-learn (Pedregosa al., 2011). "Since we can express S = XA Rica, and thus specifying Rica is equivalent to specifying B = A.R.ss

---

15
ICA
20
10
10
axis 100
20
-10
axiis
axis 99
Figure 5: Scatterplots word embeddings along specific axes: (0. 1). (2, 3), (49, 50), and (99, 100). The axes for ICA and PCCA-transforrt embeddings arranged in descending order of skewness and variance, respectively.
skewness
kurtosis
lagcosh
Gaussian
0.025
200
PCA
150
0.020
0.015
random
0.015
100
0.010
50-
0. 0.005
0.005
0.000
0.000
1000
200
100
1.00
200
300
1:00
200
300
2x05
Figure 6: Measures Gaussianity for each axis of ICA and PCA- ccA-transformed word embeddings. Additionally, the measures for each component of the raw word embedding and Gaussian random variable plotted baselines. Larger values indicate deviation from the normal distribution. The axes found by ICA are far more non-Gaussian than those found by PCA. For more details, refer to Appendix
3.3 Interpretability and low-dimensionality of ICA-transformed embeddings
is approximately represented by these two axes. Quantitative evaluation is provided in Section 6.
Fig. 1 illustrates that the key components of word embedding are formed by specific axes that capture the meanings associated with each word. Specifi- cally, this figure showcases five axes selected from the ICA tr---rannformedd word embeddings, that is, five columns of S. The word embeddings X have a dimensionality of d 300 and were trained on the text8 corpus using Skip-gram with negative sampling. For details of the experiment, refer to Appendix B.
Universality across languages
This section examines the results of conducting ICA word embeddings, each trained individually from different language corpora. Interestingly, the meanings of the axes discovered by ICA appear be the same across all languages. For detailed description of the experiment, refer to Appendix C.
Setting. We utilized the fast'Text embeddings by Grave et al. (2018), each trained individually on separate corpora for 157 languages. In this exper- iment, we used seven languages: English (EN), Spanish (ES), Russian (RU), Arabic (AR), Hindi (HI), Chinese (ZH), and Japanese (JA). The dimen- sionality of each embedding d 300. For each language, we selected = 50,000 words, and com- puted the PCA-transformed embeddings Ziang and the ICA-transformed embeddings Slang for each of the seven centered embedding matrices X (lang E (EN, ES, RU, AR, HI, JAJ)).
Interpretability. Each of these axes represents a distinct meaning and can be interpreted by ex- amining the top words based their normalized component values. For example, words like dishes, meat, noodles have high values on axis 16, while words like cars, car, ferrari have high values on axis 26. We labeled each axis with the word having the highest component value, enclosed in brackets like [dishes] for axis 16, and [cars/ for axis 26.
Low-dimensionallty. The meaning of a word is approximately represented by the combination of few axes. For example, the ferrai has large values on (cars) (axis 26) and [italian] (axis 34). This indicates that the meaning of the wordfferrat
We then performed the permutation of axes find the best alignment of axes between languages. The matching measured by the cross-correlation coefficients between languages.

---

The upper panels of Fig. 3 display the cross-correlation coefficients between the first 100 axes of English and those Spanish embeddings. The significant diagonal elements and negligible off-diagonal elements observed in Fig. 3a suggest a strong alignment of axes, indicating good cor- respondence between the ICA-transformed embed- dings. Conversely, the less pronounced diagonal elements and non-negligible off-diagonal elements observed in Fig. 3b indicate less favorable align- ment between the CCA-transformed embeddings.
Results.
fastText
BERT
TastText
BERT
0 30-106000
0.30066366
0:209960800
0.4
The semantic axes identified by ICA, referred to 'independent semantic axes' or 'independent se- mantic components', appear to be nearly universal, regardless of the language. In Fig. 2a, heatmaps visualize the ICA-transformed embeddings. The upper panels display the top words for each of the first 100 axes in English and their corresponding translations in the other languages. It is evident that words sharing the same meaning in different languages represented axes. The lower panels show the first 5 axes with their corresponding words. ICA identified axes in each language related to first names, ships-and-sea. country names, plants, and meals as independent semantic components. Overall, the heatmaps for all languages exhibit very similar patterns, indicating shared set of independent semantic axes in the ICA Cc-trranffomee embeddings across languages.
0.2
02
0.0
0.0
01234
01234 01234
01237
(a) ICA
(b) PCA
Figure 7: The rows represent normalized word embed- dings for English fastText and English BERT trans- formed by (a) ICA and (b) PCA. The components from the first 100 axes 500 fastText and 1,500 BERT em- beddings are displayed in the upper panels, and the first five axes are magnified in the lower panels. For each axis fastText embeddings, the top words were cho- based their component values. For each of these words, 3 corresponding tokens from BERT were chosen randomly. Correlation coefficients between transformed axes utilized to establish axis correspondences and carry out axis permutations, as detailed in Section 5.1 and Appendix D.
the corresponding
5 Universality in algorithm and modality
Results. In Fig. 7a, the heatmaps for fastText and BERT exhibit strikingly similar patterns, indicating shared set of independent semantic axes in the ICA-transformed embeddings for both fastText and BERT. The lower heatmaps show the first five axes with meanings first names, community, ships-and- sea, verb, and number. Furthermore, the middle panel in Fig. 3a demonstrates a good alignment of axes between fastText and BERT embeddings. On the other hand, PCA gives a less favorable align- ment, as seen in Figs. 3b and Fig. 7b.
We expand the analysis from the previous section to two additional settings. The first setting involves comparing fastText and BERT, while the second setting involves comparing multiple image models and fastText simultaneously.
5.1 Contextualized word embeddings
Setting. Sentences included in the One Billion Word Benchmark (Chelba et al., 2014) were pro- cessed using a BERT-base model to generate con- textualized embeddings for n = 100,000 tokens, each with a dimensionality of d = 768. We com- puted PCA and (CA-tranffrrmed embeddings for both the BERT embeddings and the English fast- Text embeddings. As with the cross-lingual case of Section 4, the axes are aligned between BERT and fastTexx embeddings by permuting the axes based on the cross correlcotnl coefficients. Further details can found in Appendix D.I.
5.2 Image embeddings
Setting. We randomly sampled images from the ImageNet dataset (Russakovsky et al., 2015), which consists of 1000 classes. For each class. collected 100 images, resulting in total of 100,000 images. These images were passed through the five pre-trained image models listed in Table 1, and we obtained embeddings from the layer just before the final layer of each model.

---

Base
RcsMLP-12
Swin-S
ResNet-
RegNetY-200MF
fastText
1.0
0.8
20 40 00
20:40 60
20 40 60
80
20:40 00
20 40 00
20 00 90
rnde
0.6
0.4
ser bett
bwc
bes pap
bar
103
wy
0.2
0.0
01234
01234
(a) Normalized ICA-transformed embeddings of images and words for five image models and English fast
VIT-Base
ResMLP-12
Swin-S
ResNet-18
RegNetY-200MF
fastText
0.8
20 42
20 40 8000660000 40
20 40
B0
615
107
0.6
0.4
14th
0.2
sxt
dch
poy
Patie
0.0
01236
2
(b) Normalized PCA-transformed embeddings images and words for five image models and English fastText.
Figure 8:
The rows represent (a) CCA-transformed embeddings and (b) PCA- ICA-transformed embeddings of images or words. The lower panels magnify the first five axes. The components from the first 100
displayed for the 1500 image embeddings and 500 word embeddings. For each axis of ViT-Base embeddings, the top 5 ImageNet classes were chosen based on mean of their component values, and images were sampled randomly from each class. These images were also used for the other image models. For fastText, words that best describe the ImageNet class name were selected. Correlation coefficients between transformed axes were utilized to establish correspondences and carry out axis permutations; ViT-Base was used as the reference model among the image models to match the axes, and then the axes were aligned between ViT-Base and English fastText to ensure the correlation coefficients are in descending order, as detailed in Section 5.2 and Appendix D.2.
model
weight
d
beddings (Radford al., 2021). We computed PCA and CcA-trranformed embeddings for the five im- age models. As with the cross-lingual case of Sec- tion 4, the axes are aligned between ViT-Base and each of the other four image models by permuting the axes based on the cross-correlation coefficients. Additionally, to align the axes berween ViT-Base and English fastText, we permuted the axes based on the cross-correlation coefficients that were com- puted using ImageNet class names and fastText vocabulary means of linking the modali- ties. Further details can be found in Appendix D.2.
VIT-Base
vit_.base_pa
tct.t_ch2224420 22t.c222___pppeeeeee 222.ch22222...li.eeelonn 1elonn
768
ResMLP-12
resmlp. 12,
Swin-S
awin_small
www.ccn_.wiiiieaa.e ndow7_2244
768
RogNerY-2XXIMF
regnety_002
368
Table 1: pre-trained image models.
Among these models, we selected a specific model ViT-Base (Dosovitskiy al., 2021) as our refer- ence image model. This particular ViT-Base model was trained with focus on aligning with text em-

---

Results. In Fig. Sa, the heatmaps of the five im- age models and fastText exhibit similar patterns, indicating shared set of independent semantic axes in the ICA-transformed embeddings for both images and words. While we expected a good align- ment between ViT-Base and fastText, it is note- worthy that we also observe a good alignment of ResMLP (Touvron et al., 2023) and Swin Trans- former (Liu al., 2021) with fastTTxx. Further- more, Fig. 3a demonstrates very good alignment of axes between ViT-Base and ResMLP. This sug-
ZCA
PCA
Varimax
ICA
DistRatio
1.04
1.13
1.26
1.57
Table 2: A large value of DistRatio indicates consis- tency of word meaning in the word intrusion task. We set l = 5, and the reported score is the average of 10 runs with randomly selected intruder words.
Analogy
Word Similarity
Top acc.
gests that the independent semantic components captured by ICA are not specific to particular image model but are shared across multiple mod- els. On the other hand, PCA gives a less favorable alignment, as seen in Figs. 3b and 8b.
0.3
6:25
0.23
ZCA
0:13
PCA
0.23
Varimax
00
20.
201
10"
221
207
Number of non-zaro axes
Number of non-zero 2005
Figure 9: The performance of several whitened word embeddings when reducing the non-zero components. The values are averaged over 14 analogy tasks or six word similarity tasks. The performance of a specific case in analogy and word similarity tasks is shown in Fig. 17 in Appendix E.4.
6 Quantitative evaluation
We quantitatively evaluated the interpretability (Section 6.1) and low-dimensionality (Section 6.2) -re-frranforrmed embeddings comparing with other whitening methods (PCA, ZCA) as well as a well-known rotation method (varimax). These baseline methods are described in Appendix 1. used 300-
6.2 Low-dimensionality: analogy task & word similarity task
In the monolingual experiments, dimensional word embeddings trained using the SGNS model on the text8 corpus, as outlined in Appendix B. Furthermore, we assessed the cross- lingual performance (Section 6.3) of PCA and ICA- transformed embeddings, along with two other su- pervised baseline methods.
We conducted analogy tasks and word similarity tasks using a reduced number of components from the transformed embeddings. Specifically, we eval- uated how well the transformed embedding retains semantic information even when reducing the non- zero components from the least significant ones. For each whitened embedding, we retained the k most significant components unchanged while set- ting the remaining components to zero. The spe- cific axes we used depend on each embedding. The performance was evaluated for the number of axes k ranging from to 300.
6.1 Interpretability; word intrusion task
We conducted the word intrusion task (Sun et al., 2016; Park et al., 2017) in order to quantitatively evaluate the interpretability of axes. In this task, we first choose the top h words from each axis, and then evaluate the consistency of their word meaning based the identifiability of the intruder word. For instance, consider a word group of k namely, (windows, microsoft, linux, unix, os) with the consistent theme of operating systems. Then, hamster should be casily identified as an intruder. Details are presented in Appendix E.3.
Results. Fig. 9 demonstrates that the ICA- transformed embedding has the highest average per- formance throughout the entire dataset. Detailed settings and results, including those for unwhiten- ing cases, are presented in Appendix E.4. These ex- perimental results show that the ICA-transformed embedding effectively represents word meanings using only a few axes.
The experimental results presented in Table 2 show that the top words along the axes of (CA-transformed embeddings exhibit more consis- tent meanings compared to those of other methods. This confirms the superior interpretability of axes in edA-rranformee embeddings.
Results.
6.3 Universality: cross-lingual alignment In Fig. 2a. visually inspected the cross- lingual alignment obtained by permutating ICA- transformed embeddings, and we observed remark- ably good alignment across languages. In this sec-

---

Dataset
Method
Original
Rand.
optimal mapping from the source language the 55.28
target language by leveraging both the embeddings as well as translation pairs. Therefore, we con- sider LS as the reference method in this experiment. Proc performed similarly to LS in the original em- beddings, but its performance deteriorated with the random transformation. The original word em- beddings had very similar geometric arrangements across languages, but the random transformation distorted the arrangements so that the orthogonal matrix in Proc was not able to recover the original arrangements.
LS
84.72
Proc
84.95
18.72
1a7langs-fas(Tett
ICA
19.48
18.60
PCA
0.92
0.64
LS
78.91
51.44
Proc
78.41
20.19
MUSE-fasText
ICA
43.27
43.31
PCA
2.13
0.40
Table 3: The average top-I accuracy of the cross-lingual aligmment task from English other languages. Two datasets of fastText embeddings (157langs and MUSE) were evaluated with the two types embeddings (Orig- inal and handomm-ransformatiin) LS and Proc supervised transformations using both the source and target embedddings, while ICA and PCA are unsuper- vised transformations. For the complete results, refer Table 15 in Appendix E.5.
ICA generally performed well, despite being an unsupervised method. In particular, ICA was not af- fected by random transformation and performed as well as or better than Proc. The higher performance of ICA for MUSE than for 157langs is likely due to the fact that MUSE pre-aligned. On the other hand, PCA performed extremely poorly in all the settings. This demonstrates the challenge of cross- lingual alignment for unsupervised transformations and highlights the superiority of ICA.
tion, we go beyond that by thoroughly examining the performance of the alignment task. Details are presented in Appendix E.5.
The observations from this experiment can be summarized as follows. ICA was able to identify independent semantic axes across languages. Fur- thermore, ICA demonstrated robust performance even when the geometric arrangements of embed- dings were distorted. Despite being an unsuper- vised transformation method, ICA achieved im- pressive results and performed comparably to the supervised baselines.
Datasets. In addition to 157laagssfastTees used in Section 4 (Grave et al., 2018), we used 'MUSE- fastText', pre-aligned embeddings across lan- guages (Lample et al., 2018). The source language is English (EN), and the target languages are Span- ish (ES), French (FR), German (DE), Italian (IT), and Russian (RU). For each language, we selected 50,000 words.
Supervised baselines. We used two supervised baselines that learn transformation matrix W e Rdxd by leveraging both the source embedding X and the target embedding Y. Each row of X and Y corresponds to translation pair. We minimized JXW - Y by the least squares (LS) method (Mikolov et al., 2013b) or the Procrustes (Proc) method with the constraint that W is an orthogonal matrix (Xing et al., 2015; Artetxe al., 2016).
7 Conclusion
We have clarified the universal semantic structure in the geometric patterns of embeddings using ICA by leveraging anisotropic distributions remaining in the whitened embeddings. We have verified that the axes defined by ICA are interpretable and that embeddings can be effectively represented in low- dimensionality using few of these components. Furthermore, we have discovered that the meanings of these axes are nearly universal across different languages, algorithms, and modalities. Our find- ings are supported not only by visual inspection of the embeddings but also by quantitative eval- uation, which confirms the interpretability, low- dimensionality, and universality of the semantic structure. The results of this study provide new insights for pursuing the interpretability of models. Specifically, it can lead to the creation of inter- pretable models and the compression of models.
Random transformation. In addition to the orig- inal fastTexx embeddings, we considered embed- dings transformed by a random matrix Q E Rdddd that involves random rotation and random scaling. Specifically, for each X, we independently gener- ated Q to compute XQ.
Results. Table 3 shows the t accuracy of the cross-lingual alignment task. The values are aver- aged over the five target languages.
LS consistently performed the best, or nearly the best, across all the settings because it finds the

---

Limitations
References
Due to the nature of methods that identify se- mantic axes by linearly transforming embed- dings, the number of independent semantic components is limited by the dimensionality of the original embeddings.
Prince Osei Aboagye, Yan Zheng, Chin-Chia Michael Yeh, Junpeng Wang. Zhongfang Zhuang. Huiyuan Chen, Liang Wang, Wei Zhang, and Jeff M. Phillips. 2022. Quantized wasserstein procrustes aligument of word embedding spaces. In Proceedinggs of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), AMTA 2022, Orlando, USA, September 12- 16, 2022, pages 200-214, Association for Machine Translation in the Americas.
ICA transforms the data in such a way that the distribution of each axis deviates from the normal distribution. Therefore, ICA is not applicable if the original embeddings follow multivariate normal distribution. However, no such issues were observed for the embeddings considered in this paper.
Salch Albahli, Awais Awan, Tahira Nazir, Aun Irtaza, Ali Alkhalifah, and Waleed Albattah. 2022. A deep learning method dewr with hanet for stock market prediction using news articles. Complex & Intelligent Systems, 8(3):2471 1427.
In Section 4, we utilized translation pairs to confirm the shared semantic structure across languages. Consequently, without access to such translation pairs, it becomes infeasible to calculate correlation coefficients and achieve successful axis matching. Therefore, in the future, we intend to investigate whether it is possible to perform matching by comparing distributions between axes using optimal trans- port or other methods without relying trans- lation pairs.
David Alvarez Melis and Tommi S. Jaakkola, 2018. Gromov- Noomo--aaseesteinn alignment of word embedding spaces. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. October 31 November 4, 2018, pages 1881-1890. Association for Computational Linguistics.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016 Learning principled bilingual mappings of word em- beddings while preserving monolingual invariance. In Proceedings of the 2016 Conference Empirical Methods Natural Language Processing, EMNLP 2016. Austin, Texas, USA, November 1-4, 2016, pages 2289-2294.. The Association for Computational Lin- guistics.
When comparing heatmaps of embeddings across languages in Fig. 2, we looked at five words from each axis. Thus only small frac- tion of vocabulary words were actually ex- amined for verifying the shared structure of gcometric patterns of embeddings. Although this issue is already compensated by the plot of cross-correlations in Fig. 3, where a sub- stantial fraction of vocabulary words were ex- amined, we seek a better way to verify the shared structure in future work. For example, the scatter plots in Fig. 15 may help us under- stand the entire structure of word embedding distributions.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 451-462. Association for Computational Linguisties.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross- lingsllingual mappings of word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 789-798. Association for Com- putational Linguistics.
Ethics Statement
This study complies with the ACL Ethics Policy.
Acknowledgements
We would like to thank Sho Yokoi for the discus- sion and anonymous reviewers for their helpful advice. This study was partially supported by JSPS KAKENHI 22H05106, 23H03355, JST CREST JPMJCR2IN3.
Anthony Bell and Terrence Sejnowski. 1997. The "independent components" of natural scenes are edge filters. Vision research, 37(23)3327-3333
Peter J. Bickel, Gil Kur, and Boaz Nadler. 2018. Projec- tion pursuit in high dimensions. Proceedings of the National Academy of Sciences, 115(3759151--1556

---

Michael W Browne. 2001. An overview of analytic rotation in exploratory factor analysis. Multivariate behavioral research, 36( 1): 1-150.
of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1491-1500, Beijing, China. Association for Computational Linguistics.
Elia Brumi, Nam-Khanh Tran, and Marco Baroni. 2014, Multimodal distributional semanties. Journal of Arti- ficial Intelligence Research, 49:1-47.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Ey- tan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on information systems, 20( ( 16-131.
Tyler Chang. Zhuowen Tu, and Benjamin Bergen. 2022. The geometry of multilingual language model repre- sentations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 1 13,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Sim Verb-350:: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173-2182.
Ciprian Chelba, Tomás Mikolov. Mike Schuster, Qi Ge Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTER- SPEECH 2014, 15th Annual Conference of the In- ternational Speech Communication Association, Sin- gapore, September 14-18, 2014, pages 2635-2639. ISCA.
Yoav Goldberg. 2017. Neural network methods for natural language processing. Synthexis lectures on human language technologies, 10(1):1-309.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar- mand Joulin, and Tomás Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Re- sources Association (ELRA).
Ethan A. Chi, John Hewitt, and Christopher D. Man- ning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564-5577, Online, Association for Computational I Linguistics.
Edouard Grave, Armand Joulin, and Quentin Berthet. 2019. Unsupervised alignment of embeddings with wasserstein procrustes. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learn- ing Research, pages 1880-1890. PMLR.
Charles B Crawford and George A Ferguson. 1970. A general rotation criterion and its use in orthogonal rotation. Psychometrika, 35(3):321-332.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Lei Han, Menggi Ji, Lu Fang, and Matthias Nießner. 2018. Regnet: Learning the optimization of direct image-to-image pose registration. CoRR.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Jan Engler, Sandipan Sikdar, Marlene Lutz, and Markus Strohmaier. 2022. SensePOLAR: Word sense aware interpretability for pre-trained contextual word em- beddings. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 4607- 4619, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlcx-999 Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- rics, 41 (44:666556
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joiní Conference Natural Lan- guage Processing (EMNLP- IJCNLP), pages 55-65, Hong Kong, China. Association for Computational Linguistics.
Timo Honkela, Aapo Hyvärinen, and Jaakko J Väyry- nen. 2010. Wordica-emmrgenee emergence of linguistic rep- resentations for words by independent component analysis. Natural Language Engineering, 16(3):277- 308.
Peter J Huber. 1985, Projection pursuit. The annals of Statisties, pages 435-475.
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. 2015. Sparse overcom- plete word vector representations. In Proceedings
A. Hyvarinen. 1999. Fast and robust fixed-point algo- rithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3):226664..

---

Aapo Hyvärinen and Erkki Oja. 2000. Independent component analysis: Algorithms and applications. Neural networks, 13(4-5):41144 1-430..
Tomás Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean. 2013c. Distributed represen- tations of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems, pages 3111-3119.
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. 2018. Optimal whitening and decorrelation. The American Statistician, 72(4):309-33) -144.
Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-oop: Simple and effective postprocessing for word repre- sentations. In International Conference on Learning Representations.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ran- zato, Ludovie Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Truck Proceedings. OpenRe- view.net.
Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012 Learning effective and interpretable semantic models using non- non-negative sparse embedding. In Proceed- ings of COLING 2012, pages 1933-1950. Mumbai, India. The COLING 2012 Organizing Committee.
Guy Lev, Benjamin Klein, and Lior Wolf. 2015. In defense of word embedding for generic text repre- sentation. In Natural Language Processing and In- formation Systems: 20th International Conference on Applications of Natural Language to Information Systems, NLDB 2015, Passau, Germany, June 17-19. 2015, Proceedings 20, pages 35-50. Springer.
Tomáš Musil and David Mareček. 2022. Independent components of word embeddings represent semantic features. arXiv:221 2t:::22/2.000 1200950.
Sungjoon Park, Jin Yeong Bak, and Alice Oh. 2017. Rotated word vector representations and their inter- pretability. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, pages 401-411, Copenhagen, Denmark, Associ- ation for Computational Linguistics.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEF/CVF Interna- tional Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992-10002. IEFF
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Seikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Hongyin Luo, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2015. Online learning of interpretable word embeddings. In Proceedings of the 2015 Con- ference Empirical Methods in Natural Language Processing, pages 1687-1692, Lisbon, Portugal. As- sociation for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual multilingual BERT? In Proceed- ings of the 57th Anmual Meeting the Association for Computational Linguistics, pages 4996-5001, Flo- rence, Italy. Association for Computational Linguis- tics.
Thang Luong, Richard Socher, and Christopher Man- ning. 2013. Better word representations with recur- sive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Nat- ural Language Learning, pages 104-113.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Matt Mahoney. 2011. About the test data. http:// mat nattmahoney. http//////eextttttaaaeee htmm..
Alireza Makhzani and Brendan Frey. 2013. K-sparse autoencoders. arXiv preprint arXiv: 1312.5663.
Binny Mathew, Sandipan Sikdar, Florian Lemmerich, and Markus Strohmaier. 2020. The polar framework: Polar opposites enable interpretability of pre-trained word embeddings. Proceedings of The Web Confer- ence 2020.
Kira Radinsky. Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Computing word relatedness using temporal seman- tic analysis. In Proceedings of the 20th International Conference on World Wide Web, page 337-346.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representa- tions in vector space. 1st International Conference on Learning Representations, ICLR 2013, Scotisdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla. Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):11-252.
Tomás Mikolov, Quoc V. Le, and Nlya Sutskever. 2013b. Exploiting similarities among languages for machine translation. CoRR, abs/ 1309.4168.
Robyn Speer. 2022. rspeer/wordfreq: v3.0. https: //doi. org/10 /00 5281//28///// 0085522280da /2enodo..79993...

---

Jianlin Su. Jiarun Cao. Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. CoRR, abs/2103.15316.
can be represented in the form of linear transforma- tions using an orthogonal matrix R:
ZR.
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor BerggKirrppatrick and Eduard H. Hovy. 2018. SPINE: sparse interpretable neural embeddings. In Proceedings of the l'hirty-Second AAAI Conference on Artificial Intelligence, (AAAI-1S) 18), the 30th inno- Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4921 4928. AAAI Press.
These transformed embeddings Y with any R. are again whitened', meaning that the embeddings are isotropic with respect to the variance-covariance matrix. Also, the centering step of whitening makes the embeddings centered, and the transformed em- beddings Y with any R are again centered', mean- ing that the embeddings are isotropic with respect to the mean vector. However, the linear transfor- mation cannot make the embeddings isotropic with respect to the third and higher-order moments.
vative Applications
Fei Sun, J. Guo, Yanyan Lan. Jun Xu, and Xueqi Cheng. 2016. Sparse word embeddings using I1 regularized online learning. In International Joint Conference on Artificial Intelligence.
The row vectors y; = (Ma1..... Wid) and z; (zi1...., 2ia) of Y and Z, respectively, satisfy the equation::
William Timkey and Marten van Schijndel. 2021. All bark and bite: Rogue dimensions in transformer language models obscure representational quality. In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages 4527-4546, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
(y:, = (Zi, Zz):
where (a, b) Ei= ogbb, represents the inner product. Therefore, the inner products of embed- dings preserved under this transformation, indi- cating that the performance of tasks based on inner products, such as those using cosine similarity, re- mains unchanged.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve. Jakob Verbeek, and Hervé Jégou. 2023. Resmlp: Teedforward networks for image classification with data-efficient training. IEEE Trans. Pattern Anal. Mach. Intell,, 4544)55314-5311
B Details of experiment in Section 3
We summarize the details of the embeddings used in Figure I and the monolingual quantitative evalu- ations in Sections 6.1 and 6.2.
Ross Wightman. 2019.
Pytorch image mod- https: /gi thub. wwmub.com/eeee wtwccoc//////eeeee com/rwightman/ pytorch-iman iamag--modell-
els.
Corpus. We used the text8 (Mahoney, 2011) 1), which is an English corpus data with the size N = 17.0x 10 tokens and V = 25 10 vocab- ulary words. We used all the tokens separated by spaces. The frequency of word w E V in the cor- pus is denoted as p(w), where Ewev p(w) = 1.
Chao Xing. Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal trans- form for bilingual word translation. In NAACL HILT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Denver, Colorado, USA, May 31 June 5, 2015, pages 1006 1011. The Association for Computational Linguistics.
Training of the SGNS model. Word embeddings were trained by optimizing the same objective
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE Interna- tional Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society.
'In general, for an orthogonal matrix R. i... R R L. consider transformed matrix Y = ZR. Then Y is whitened. because Y/n (ZR) ZR/n R Z ZR/n R L..
R R'R *For centered embeddings Z. the mean vector is (1./n)) 2 = 02 where 1, = (1,..., E R" and
0₂ (0. .. 0) E Rd. Then, for any orthogonal matrix R, (1n Y = (1./n) (l./n)TZR ZR 02 R O
A Whitening and isotropic embeddings
Since = R,R, we have (y), = y.y.
In addition to the whitened embeddings Z ob- tained through PCA, we also consider various other whitened embeddings (Kessy et al., 2018). They
R(z,R) z,RR 2JJ = z:z (Z.,
"We used AMD EPYC 7702 64-Core Processor (64 cores x 2). In this setting, the CPU time is estimated at about 12 hours.

---

function used in Mikolov al. (2013c). Param- eters used train SGNS are summarized in Ta- ble 4. The learning rate shown is the initial value, which we decreased linearly to the minimum value of 1.0 x 10 4 during the learning process. The negative sampling distribution was proportional to the 3/1-th power of the word frequency, p(w)3/4
observed that either a 3/4 or a is the best, and we decided that using 1 appropriate for ease of implementation in general.
Analogy
Word Similarity
0.5
2.40
30 acc
044
Sprarman's
The elements of were initialized by the uniform distribution over [- 0.5, 0.5] divided by the dimen- sionality of the embedding, and the elements of x were initialized by zero.
0.3
2.0
unweighted
0.25
1/2
02
ct 3/4
102 201 15
157
151
107
Dimensionality
300
Number of non-zero axes
Number of non-zero axes
Epochs
100
Figure 10: Performance of ICA-transformed embed- dings using reduced number of components with vary- ing resampling weights specified by Larger values indicate better performance. We measured the top-10 accuracy for the analogy task and the Spearman rank correlation for the word similarity task. These values represent averages across 14 analogy tasks and six word similarity tasks.
Window size h
10
Negative samples
Learning rate
0.025
Min count
Table 4; SGNS parameters.
Discarding low-frequency words. After com- puting the embeddings, we discarded words that appeared less than 10 times in the text8 corpus. The new vocabulary V' consists of the 47,134 words that appeared 10 times or more in the text8 corpus.
Analogy
Word Similarity
Resampling word embeddings. We resampled the word w with probability ((w) when preparing the data matrix X E IRadd As an example, we will explain the procedure when employing g(w) x p(w) for E V'. First, we randomly sampled 100,000 words from V with replacement using the weight q(w). This resulted in the selection of 14,942 unique words. Subsequently, we added 15,058 words from the remaining unselected words, ordered by descending frequency, to reach a total of 30,000 unique words. Each row of X represents the embeddings of the n 115,05s words selected through the aforementioned process.
Top acc:
0.4
1100 iiers.)
ICA 1200 iters))
ICA 11000 iters)
1100m0 neek.)
0.1
8.15
107
30t-
12'
127
10"
10t
Number of non-zero axes
Number non-zero axes
Figure 1I: Performance ICA-transformed embed- dings with varying iterations. The settings the same as those in Fig. 10.
Selection of the number of FastICA iterations. We demonstrate that our analysis remains unaf- fected by changes in the number of iterations in Fas- tICA. Specifically,
evaluated the embeddings obtained by varying the number of iterations (100, 200, 1000, 10000) in FastCAA. The evaluation was performed on the word intrusion task, analogy task, and word similarity task. The results are presented in Table and Fig. 11. In both experiments, we observed that reducing the number of FastICA it- erations slightly diminished task performance, al- though the difference was very small. Therefore, conclude that changing the number of FastICA iterations did not significantly impact the results.
Selection of resampling weight. We considered resampling weights in the form of q(w) & p(w)", where a takes values from the candidate set a E (1/2,3/4, 1). We conducted an experiment to de- termine the optimal value of a. For each g(w), we prepared the data matrix X using the resampling method explained above. Additionally, we created unweighted X with |V' We then com- puted ICA -rattraafforeed embeddings and evaluated the performance on the analogy and word similar- ity tasks using a reduced number of components, which are explained in detail in Section 6.2 and Ap- pendix E.4. The results are shown in Fig. 10. We
ICA-transformed embeddings. We utilized the implementation of FastICA, with the default setting

---

Trmest
pollstoRs
16: (dishes)
japanese
felling terrari modden
kurosawa pkiira
lamborghin
miyazaki
anime
56: [japanese]
34: [italian]
Cers
film film
car
Terran
mazda
toyota-
honda
ducati
26: [cars]
ambborghini
film
kurcsawa
scorsese
SUZUKI
comedies
fellini
Fakiaa
mivazaki
akira
Japan
miyazika
venice
jaaanese
Lamborghini
ferral
modenn
56: [ispanese]
34: [italian]
4il
films
mazda
ferrart
toyota honda
26: Caars
-lamaorggiin
[rlm
scorsese
eurphaaw
suzuki
Tellini
akire
meat
miyyzakki
tomako
dishes
sushi
meat
dishes
susmi
16: [dishes]
16: [dishes]
cars
tallia
mazda
toyota
honda
venice
26: (cars)
hissan
amborghini
[italian]
moderials
fellini
kurosawa scorsese
films film
scorsese
anima
Japan
Canimation careere
japanese
Tellini
nissan
[japanese]
Figure 12: Scatter plots of the normalized ICA ----ranfforredd embedding for eight combinations of the five axes. The plots for the other two combinations are shown Fig. 1. The words are highlighted with colors corresponding to their respective axes. We observe that each axis be interpreted by words that have large values along that axis. Some words are represented by combination of axes, such as sushi [japanese] [dishes] or fellini = fitalian] + [film]. In some pairs of two axes, there may no words represented by their combination. For example, in the plot of ([dishes), (cars)) axes, words found where both of these components are large.

---

Varimax (baseline)
ICA (100 iterations)
ICA (200 iterations)
ICA (1000 iterations)
ICA (10000 iterations)
DistRatio
1.26
1.42
1.47
1.52
1.55
Table 5: Performance of (CA-transformed embeddings with varying iterations, Large values of DistRatio indicate better interpretability. The settings are the same as those in Table 2.
10c
15
15
10a
10
15
10
axis 50
axis 100
10a
10a
-5
10
20
10
(a) ICA-transformed word embeddings. The axis numbers are sorted by skewness.
105
7.5
10a
2.5
aMis 100
10c
axis
0.0
0.0
10a
255
-5.0
-5.0
10a
-7.5
-7.5
10a
(b) -cA-traasformed word embeddings. The axis numbers are sorted by variance.
Figure 13: Scatter plots transformed word embeddings along specific (0, I), (2, 3), (49, 50), and (99, 100).
Colors indicate word frequency in the corpus, with warmer colors being more frequent.
except for the number of iterations set 10,000 and a tolerance of 10 10. The contrast function used was G(z) = log cosh(z). It should noted that the embeddings obtained through ICA have ar- bitrariness in the sign of each axis and the order of the axes. We calculated the skewness of each axis and flipped the sign of axes as necessary to ensure positive sign of skewness. We then sorted the axes in descending order of skewness. When visualizing embeddings or selecting word sets, we normalized each embedding to have a unit norm for facilitating interpretation unless otherwise specified.
Next, we created scatterplots of the normalized ICA-transformed embeddings for all the ten combi- nations of two axes selected from {[dishes]. [cars], [films] [italian], [japanese]]. Two these scatter- plots are shown in Fig. 1, and the remaining eight are presented in Fig. 12.
Scatter plots of ICA and PCA-transformed word embeddings. To visualize the transformed embedding, scatterplots are displayed with selected pairs of axes. Fig. 13a shows the ICA-transformed embeddings given in Section 3.2, plotting specified columns of S, with the axis numbers sorted in de- scending order of skewness. Fig. 13b shows the PCA-transformed embeddings given in Section 3.1, plotting specified columns of Z, with the axis num- bers sorted in descending order of variance. Colors indicate word frequency in the corpus, with warmer colors being more frequent. Unlike other visualiza- tions, each embedding is not normalized have unit norm. The plots in Fig. superpositions of the plots in Figs. 13a and 13b, but the frequency information was omitted.
Details of Fig. in Section To illustrate the ad- ditive compositionality of embeddings, the words in the heatmap were selected as follows. For each of the six combinations of axes [[dishes], [cars), [films]] {[italian], [japanese]], we selected 20 words with the largest sum of the two component values. From these 20 words, we chose the top five words based on the second-largest component value among the five axes. As a result, a total of 6 x 5 30 words were selected this process.
Sectoin 3 are

---

The distributions of these two types of word em- bedding have exactly the same shape in R because they can be transformed into each other by rotation as S Z However, there are significant differences in the word distributions on each axis. The distribution of ICA-transformed embeddings exhibits anisotropy with a heavy-tailed shape along each axis, thereby characterizing the meaning of each axis. The distribution of PCA-transformed embeddings is more isotropic and lacks specific characterization of each axis, except for the top few axes that encode word frequency as pointed out by Mu and Viswanath (2018). Interestingly, pronounced frequency bias is not observed in the axes of CCA-trranformed embeddings.
The fourth measure labeled Gaussian is (B(G(X)) B(G(Z))) with the con- trast function G(x) - -exp(- x2/2) and
E(G(Z)) = 1/
Properties of the last two measures are well stud- ied in Hyvarinen (1999).
C Details of experiment in Section 4
The procedure described for this cross- mingsll ex- periment serves as a template for the other experi- ments in the following sections.
Dataset. We employed the fastTexx embeddings trained for 157 languages by Grave et al. (2018), referred to as '157langs-fasTTex" in this study. We also made use of the dictionaries provided by MUSE (Lample et al., 2018) to aid the map- ping of English words to their counterparts in other languages'. Given that the vocabulary of 157langs- fast'Text is substantial, containing 2,000,000 words, our initial step was to lowercase and select non- duplicate words included in both the English-to- other language and other language-to-Engliin dic- tionaries provided by MUSE. These dictionaries contain 6,500 unique source words from the com- bined train and test sets. Subsequently, from the 2,000,000 words, those not yet chosen were se- lected based on their frequency. We determined the frequency using the Python package word- freq (Speer, 2022), which provides word frequen- cies across various languages. The vocabulary then capped 50,000 words for each language.
The spiky shape of the distribution of word em- beddings observed in Fig. 13a is illustrated in Fig. 4. As Figs. 2a and 3a suggest, the distribution of word embeddings across multiple languages has a nearly common shape, so they can be mapped by orthogo- nal transformations.
Measures of non-Gaussianity. Fig. 6 in Sec- toin 3 displays four non- non-Gaussianity measures. Let denote a random variable representing the com- ponent on a specified axis, and let Z denote a Gaussian random variable; here the symbols X and Z are not related to X and Z. Since both PCA and ICA- -ranfranfooree embeddings are whitened, we assume that X and Z have mcan of zero and a variance of one; E(X) = BZ()) = 0 and B(X²) E(Z 1, where E() denotes the ex- pectation.
The first measure is the skewness B(X*). If it is negative, we flip the sign of the axis so that E) O. Since EE(33) 0, a large value of skewness indicates deviation from Gaussianity.
PCA and ICa-transformed embeddings. We then implemented PCA and ICA transformations on the fastText embeddings, each containing 50,000 words per language. We used PCA and FastICA in Scikit-learn (Pedregosa et al., 2011) I) with ICA performed for maximum of 10,000 it- erations. Finally, we computed the skewness for each axis and flipped the sign of the axis if nec- essary to ensure positive skewness. These PCA and ICA transformations were applied to each lan- guage individually to identify the inherent semantic structure within each language without referencing other languages. The following steps are devoted to verifying that the identified semantic structure in
The second measure is the kurtosis E(X*) EZZ4), where E(Z*)- 3. We observed that is nomnegative for most of the components of embeddings, indicating that their distribu- tions are more spread out with heavy tails than the normal distribution.
The third measure labeled logcosh is de- fined as (E))(X)) E(G(Z))) with the contrast function G(x) log cosh(z) and (Z))))) 0.3775672074914333 This mea- sure is used as the objective function in the implementation of FastICA.
Heeee, we solely utilized the dictionaries provided by MUSE and did not make use of the embeddings included in MUSE. The multi-lingual embeddings in MUSE are pre- aligned across languages, which makes them unsuitable for this particular experiment.

---

the axes of ICA-transformed embeddings is shared across these languages.
Lnnglish word embeddings and those of other lan- guages by appropriately permuting the indices of axes. This involves permuting the columns of the transformed embedding matrix. For illustrative purposes, we will explain the procedure using En- glish and Spanish word embeddings. Since both the English and Spanish word embeddings have di- mensions of 300, we computed total of 300 x 300 cross-correlation coefficients using the translation pairs of English words and their corresponding Spanish words. From these cross-correlation coef- ficients, we identified matched pairs of axes with high correlations in greedy fashion, starting from the highest correlation. The columns of Spanish word embeddings were then permuted to match those of English word embeddings. This procedure was applied to all other languages, ensuring that their axes align with those of English.
Dataset
ES
RU
AR
HI
ZH
JA
Pains
11965
10883
11559 8689 8645 6992
157langs Train Source
4991
4996
4988
4983
4965
4821
Target 10154
9512
9650
7149 6521 5715
Table 6: The number of translation pairs, unique source English words, and unique target words for each target language.
Pearson's
Pearson's
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
100
200
300
100
200
380
Reordering axes. Subsequently, we computed the average correlation coefficients of the aligned axes to determine the reordering of the axes based on their degree of similarity. For each language. we calculated 300 correlation coefficients between the axes and the corresponding axes of English, These correlation coefficients were then averaged across Spanish, Russian, Arabic, Hindi, Chinese, and Japanese. Finally, we reordered the axes in descending order based on the average correla- tion coefficient. The cross-correlations between the aligned axes as well as the average correlation coefficients are shown in Fig. 14.
akis
axis
(a) ICA
(b) PCA
Figure 14: Correlation coefficients between the matched components of English and each of the other languages for ICA and PCA- AcC-trrasformed embeddings. The axes were sorted by the correlation coefficients averaged across the six languages.
Word translation pairs. As the first step of the verification, we established the correspondence be- tween the embeddings of English and the embed- dings of other languages in the 157langs-fastTex dataset. This linking information between embed- dings is necessary to compute cross-correlation coefficients. For the purpose of illustration, we will explain the procedure using the language pair of English and Spanish. First, we gathered all pairs of English and Spanish words from the train set of the MUSE dictionaries, including both the English-to- Spanish and Spanish-to-English translation pairs. Next, we filtered out any pairs where either the En glish word or the Spanish word was not included in the vocabulary set of 50,000 words prepared for each language. The total number of pairs collected was 11,965, where a word may be included in mul- tiple translation pairs. The number of unique En- glish words obtained from this process was 4,991. The results of applying this procedure to the six target languages are presented in Table 6, where the source language is English. Alignment of axes via permutation. Next, we established a correspondence between the axes of
Diagnosing the axis alignment. For both ICA- transformed and PCA-transformed embeddings, we demonstrated the 100 x 100 cross-correlation coefficients between the first 100 axes of reordered English and Spanish word embeddings in Fig. 3. The diagonal elements represent the correlation efficients between the aligned pairs of axes, while the off-diagonal elements represent the correlation coefficients between unaligned axes. the case of ICA -CA-trrasformed embeddings, as depicted in Fig. 3a, it is clear that the diagonal elements ex- hibit significantly positive values, while the major- ity of the diagonal elements are close to zero. Thus, ICA gives strong alignment of the axes. In contrast, for PCA-transformed embeddings, shown in Fig. 3b, the diagonal elements are smaller compared to ICA, and a considerable number of off-diagonal elements deviate significantly from zero. Thus, PCA gives a less favorable alignment.

---

Word selection for visualization. After reorder- ing the axes, we normalized all the embeddings to ensure that their norm equal to 1. We focused on the first 100 axes for further analysis. We limited the selection to words that have translation pairs in the complete set of MUSE in all languages, in- cluding English. Spanish, Russian, Arabic, Hindi, Chinese, and Japanese. From this restricted set, we selected the top 5 words for each axis of the English word embeddings based on their largest component values. For the remaining languages, we selected words from the translation pairs. To ensure diversity, we excluded duplicates that occur in both singular and plural forms of nouns.
along the axis. When viewed as whole, the distri- bution of embeddings for each language exhibits a distinctive shape with spikes along axes. This spiky shape is universally observed across all languages.
We should discuss whether this spiky shape is real or not. ICA seeks axes that maximize non- Gaussianity (Hyvärinen and Oja, 2000). More gen- erally, projection pursuit aims to find 'interesting' projections of high- dimensional data (Huber, 1985). However, these methods may detect apparent struc- tures that are not statistically significant, particu- larly when 7 - d/n is large (Bickel et al., 2018). In the case of cross-lingual word embeddings, where =00 and 50,000, 7 0.006 < 1 is very small. Therefore, it can be said that the chances of detecting apparent Gaussann structures are quite low. Taking into account the discovery of numerous common axes across all languages, it can be argued that the universal shape in the cross- lingual embedding distributions is real.
Analyzing the heatmaps. In Fig. 2, we presented heatmaps that illustrate the components of the nor- malized 500-word embeddings selected through this procedure for both CcA-trranformed and PCA- transformed embeddings.
In Fig. 2a, which represents the ICA-transformed embeddings, we observe that the top 5 words for each axis have significant values along that axis. This leads to a diagonal pattern of large values due to the sparsity in other components. This consis- tent pattern is observed in all seven languages. The lower panels of the figure magnify the first five axes, allowing us to identify the top 5 words cho- sen for each axis. It is evident that each axis has specific semantic relevance. For example, the first axis corresponds to first names, and we ob- serve that such semantically meaningful axes can be effectively aligned across languages.
The signature of ICA-transformed embed- dings. To investigate the characteristics of ICA- transformed word embeddings for each language, we plotted two measures of non-Gaussianity. namely skewness and kurtosis, along each axis in Fig. 16. Summary statisties for the skewness and kurtosis of the embeddings are given in Table 7. These results allow us to discern deviations from the isotropy of the distributions of word embed- dings. All the kurtosis values for all axes in all languages were positive, indicating that the distri- butions are spreading more than the normal distri- bution. Although general trend is observed in the plots, there are differences depending on the language. In particular, English shows the high- est non-Gaussianity, indicating a most spiky shape. In contrast, Chinese and Hindi have the lowest non- non-Gaussianity and smoother shapes. It remains unclear whether these differences are language- specific or induced by the embedding training pro- cess.
Conversely, in Fig. 2b representing the PCA- transformed embeddings, it can be observed that the top 500 words do not exhibit significant values along the diagonal. Additionally, in the magnified heatmaps depicting the 25 words, when compared to Fig. 2a, the semantics of the words for each axis appear to be more ambiguous.
Visualization via scatterplots along the five axes. In the heatmaps, we visualized embeddings for 25 words in each language. To overcome the limita- tion of the heatmap, which only allows us to view a small subset of words, we utilized scatterplots to visualize all the words in the vocabulary. In Fig. 15, we projected the normalized ICA- -e-ttransformed em- beddings into two dimensions using the same five axes as those used in the heatmaps.
D Details of experiment in Section 5
D.1 Contextualized word embeddings
Dataset. We used bert-base-uncaal a pre- trained BERT model from the huggingface trans- formers library. This model was pre-trained on the BookCorpus (Zhu et al., 2015) and English Wikipedia. Sentences from the One Billion Word Benchmark (Chelba et al., 2014) were sequentially
Similarly to the heatmaps, the meaning of each axis can be interpreted based on the words arranged

---

boat
7 Ship ocean
2 10 russia
sea
33japan
11 countr
14 iran
jeff
jim
mike
1 dave
flower
ttere
16 plants
17 garden
24 dinner
19 seeds
mcat
22 restaurant
20 food
4
(a) English
(b) Spanish
(c) Russian
(d) Arabie
(e) Hindi
(f) Chinese
gg) Japanese
Figure 15: Normalized ecA-trranformed word embeddings for seven languages. Scatterplots along the five axes presented in Fig. 2a were drawn by projecting the embeddings into two dimensions. For each language, all words in the vocabulary were plotted in respective colors. The 25 words the heatmap were labeled and indicated by black dots.

---

l1g(skewness+1)
lurttsrtss +1)
significant correlation coefficients, thereby select- ing 300 pairs of axes.
Englion
spanti
8
Sussian
3
Reordering axes. Subsequently, we reordered the axes in descending order of the correlation co- efficients between the aligned axes of fastText and BERT. The cross-correlation coefficients between the first 100 axes are presented in the middle panels of Fig. 3. ICA gives a strong alignment of the axes, while PCA gives a less favorable alignment.
+indi
Chinese
2
japanese
0
0
100
200
300
0
100
200
300
axis
axis
Figure 16: Skewness and kurtosis calculated for each axis of ICA-transformed word embeddings for seven languages. For each language, the axes are sorted in descending order of skewness.
Word selection for visualization. We normal- ized each embedding have a norm of 1. We limited BERT tokens to those included the fast- Text vocabulary. We selected the top 5 words from the fastText axis based on their component values, To examine the diversity of words, duplicates in sin- gular and plural forms of nouns were disregarded. To verify the variations in dynamic word embed- dings, randomly selected 3 BERT embeddings for each word, instead of selecting those with the three largest component values. In the heatmaps. these three BERT embeddings were placed in de- scending order based on the component values. We performed this process for the initial 100 axes of both the ICA-transformed and PCA-transformed embeddings, and presented the heatmaps of the components of the embeddings when selecting 500 words from fastText and 1,500 tokens from BERT.
skewness
kurtosis
lang
mean
median
mean
median
EN
3.25
1.84
129.63
30.12
ES
2.50
1.84
54.43
24.18
RU
2.49
1.75
48.65
22.63
AR
2.06
1.50
57.03
23.58
HI
1.53
1.01
35.70
20.08
ZII
1.38
1.34
15.62
11.84
JA
2.13
1.88
58.38
21.31
Table Summary statistics of the skewness and kurto- sis calculated for each axis of ICA (CA-trrasformed word embeddings shown in Fig. 16.
inputted into BERT, generating 100,000 tokens, including [CLS] and [SEP]. Unlike static word em- beddings like those from fastText, BERT yields dy- namic word embeddings. Consequently, the same word can have different embeddings, labeled se- quentially in their order of appearance, for instance, sea_0. O, sea__, and on.
A Analyzing the heatmaps. In Fig. 7a, which il- lustrates the ICA-transformed embeddings, we ob- serve similar patterns to the cross-lingual experi- ment. In the upper panels, representing 500 words and 1,500 tokens, the components of the top 5 words and sampled 15 tokens for each axis ex- hibit large values. This is due to the sparsity of the remaining components, resulting in significant values along the diagonal. Moreover, in the lower panels, we can observe consistent component pat- terns across most pairs of axes for the 25 words and 75 tokens.
PCA and ICA-transformed embeddings. Simi- lar to the cross-lingual experiments, we computed PCA- PCA-transformed and ICA-trande tranforasfo embed- dings for the obtained BERT embeddings. All the axes of the transformed embeddings were adjusted to have positive skewness.
Alignment of axes via permutation. We utilized the same English fastTexx embeddings used in Ap- pendix C along with the BERT embeddings. We paired the fastText words with the BERT-labeled tokens. If the same word appeared & times in the BERT tokens, we created l pairs with that word. To adjust for this effect, the correlation coefficients were computed using the inverse frequency weight, 1/k. Given that the dimensionality of fastTexx em- beddings 300 while that of BERT is 768, we greedily matched pairs of axes based on the most
On axis- of fastText embeddings, words such as people and those are relatively ambiguous and context-dependent. Considering that even in static Text,ext, their component values are smaller com- pared to other axes, it might be more challenging for a single axis to exhibit more significant compo- nents in dynamic embeddings of BERT.
On axis-2 of BERT embeddings, the compo- nent value for shore_0 is nearly zero, while the component values for shore_ and shore_2 are large. Upon reviewing the sentences containing

---

these words, shore_ 0_ appeared in the verb phrase shore up. while shore_ I and shore 2 were used to refer to the land along the edge of a sea. Therefore, axis-2 effectively represents a consistent meaning related to ships-and-sea.
ViT-Base model that matched all the other mod- cls. As result, there were 292 axes for the PCA- transformed embeddings and 276 axes for the ICA- transformed embeddings that were common among all the models.
Conversely, in Fig. 7b, which illustrates PCA- transformed embeddings, characteristics seen Fig. 7a, mirroring the find- ings from the cross- longull scenario.
Diagnosing the axis aligument. As an exam- ple, we presented the cross-correlation coefficients between the first 100 axes of the ViT-Base and ResMLP-12 models for both the PCA-transformed and ICA-transformed embeddings in the bottom panels of Fig. 3. As observed in previous ex- periments, the alignment between the ViT-Base and ResMLP-12 models evident for the ICA- transformed embeddings. However, for the PCA- transformed embeddings, the alignment appears to be less clear and more ambiguous.
did not observe the
D.2 Image embeddings
Dataset. Beyond examining the universality across different languages and distinct word em- bedding models, we also investigated the univer- sality across different image models. For the 1000 classes in ImageNet (Russakovsky et al., 2015), we assembled a dataset comprising 100,000 images, randomly selecting 100 images per class. As for the image models. we chose ViT-Base (Dosovitskiy et al., 2021) which is the backbone of CLIP (Rad- ford et al., 2021), ResMLP (Touvron et al., 2023), Swin Transformer (Liu et al., 2021), ResNet (He al., 2016), and RegNet (Han et al., 2018). The feature embeddings of these image models were extracted from their penultimate layer. We utilized the pre-trained weights from the huggingface Py- Torch Image Models (Wightman, 2019) for these investigations, and Table outlines the model types, weight types, and the dimensionalities of the em- beddings.
A Alignment with fastText. Next, we considered the correspondence between the axes of ViT-Base embeddings and English fastText embeddings. It important note that the class names in Ima- geNet are not individual words but sentences, such as 'king snake, kingsmake'. We parsed the class names into separate words and searched for those words in the vocabulary of English fastText, such as 'king' and 'snake". For each ImageNet class, randomly sampled 100 images, resulting in 100 pairs for 'king' and 100 pairs for 'snake" in the case of 'king snake, kingsnake' If a class name did not contain any of the words present in the vocabulary, that particular class was excluded from further anal- ysis. When calculating the cross-correlation coeffi- cients between the axes of ViT-Base embeddings and English fastText embeddings, each image-word pair was weighted inversely proportional to its fre- quency. Utilizing these cross-correlatio coeffi- cients, we employed the greedy matching approach to align the axes of VIT embeddings with the axes of fastText embeddings.
PCA and ICA-transformed embeddings. Just as we did for fastText and BERT word embed- dings, we computed PCA-transformed and ICA- transformed embeddings for image embeddings. All the axes of the transformed embeddings were adjusted to have positive skewness.
Alignment of axes via permutation. Similar to the cross-lingual experiment, where English fast- Text served as the reference model, we used ViT- Base as the reference model among the image models. In the process of computing the corre- lation coefficients between ViT-Base and another image model, the embeddings for the same im- age were treated as a pair. To align the axes of the ViT-Base embeddings with those of the other four image models, we employed the greedy match- ing approach based on the cross-correlation coef- ficients. Unlike the cross-lingual scenario, the im- age model embeddings have different dimensions. Therefore, we extracted only the axes from the
Reordering axes. Lastly, we rearranged the aligned axes of ViT-Base and fastText embeddings to ensure the correlation coefficients of the aligned axes are in descending order. The axes of other image models, previously matched with ViT-Base axes, were also rearranged according to the order of the ViT-Base axes.
Analyzing the heatmaps. We normalized each embedding to have a norm of 1. For each axis of ViT-Base, we selected the top 5 classes that have the highest average component values. For

---

each selected class, we randomly chose 3 images from the 100 images and sorted them in descend- ing order based on the component value. The class name was parsed to find words in the English fast- Text vocabulary, and we selected the word with the largest component value on the corresponding axis. This process was applied to the first 100 axes for both PCA-transformed and ICA-transformed embeddings. The resulting heatmaps of the embed- dings are displayed in Fig. 8.
on tasks based on inner products, they do differ in terms of interpretability, low-dimensionality, and cross-lingual performance. This study aims to iden- tify the inherently interpretable subspace within pre-trained embeddings and analyze their sparsity and interpretability. Consequently, the baselines were chosen with this perspective in mind. In other words, embeddings learned directly from a cor- pus, which incorporate explicit sparsity and inter- pretability objectives in their optimization, are be- yond the scope of this research.
From Fig. 8a for ICA-transformed embeddings, in the case of the 1,500 images and 500 words pre- sented in the upper panels, the 15 images and words on each axis exhibit significant values on that particular axis, much like in the cross-lingual and BERT experiments. This is because other com- ponents sparse, causing large values appear on the diagonal. The lower panels illustrate the heatmaps of the embeddings of 75 images and 25 words corresponding to the first five axes, For instance, axis-2 of ViT-Base is oriented toward alcohol-related concepts, the component values are large for images of bottles and beers. Models such as ResMLP-12, Swin-S, and RegNerY-200MF also manifest larger component values axis-2 for images related to alcoholic beverages. When class names are partitioned into words, words se- lected on axis-2 of fastText include beer, bottle, and wine. The significant component values on the corresponding axes suggest a successful alignment between the axes of the image models and fastText, Conversely, in Fig. 8b for PCA-transformed em- beddings, the correspondence of axes is not evident. Furthermore, the interpretation of the top 5 classes associated with ViT-Base axes remains unclear. It important to note that ViT-Base, extracted from the pre-trained CLIP, aligns well with other models such as ResMLP-12 and Swin-S. This ob- servation suggests that the decent alignment of ViT- Base with fastTTex is not merely result of CLIP learning from both image and text.
PCA. Let Σ = X X/n n be the covariance ma- trix of the row vectors of X. In one implementa- tion of PCA, eigendecomposition is performed Σ = UD²UT, where U is an orthogonal matrix consisting of the eigenvectors and D² is a diagonal matrix consisting of the eigenvalues. Using these, the transformed matrix Z is computed as
Z - -UDD-
Another implementation directly computes from the singular value decomposition X = ZDUT.
ICA. As mentioned in Section 3.2, ICA is repre- sented as S ZRi with the orthogonal matrix Rica. Thus S is whitened.
ZCA-Mahalanoba whitening. The whitening transformation that minimizes the total squared distance between X and Y is computed as:
Y 2ea XI-1/2
where 1-1 = UD UD (Bell and Sejnowski, 1997: Kessy et al., 2018). This can be expressed as Yzca = ZRRza where Reca = UT. Since the columns of U represent the directions of princi- pal components, Yzca simply rescales the original X along these directions without introducing any rotation.
Crawford-Ferguson rotation family. A fam- ily of measures for the parsimony of matrix Y is proposed by Crawford and Ferguson (1970) as fx (Y) -1- k) En Σ
E
Details of experiment in Section 6
E.1 Whitened embeddings
2i + 1= E EA where 0 < < 1 is
We introduce several whitening methods below. In Appendix A, we mentioned that they can be repre- sented as
parameter.
Although initially proposed for post-processing the factor-loading matrix in factor analysis (Craw-
Y = ZR
from the embeddings Z obtained through PCA and an orthogonal matrix R. Although these whitened embeddings do not differ in terms of performance
* XA with UD in Section 3.1.
By noting X = ZDUU we have Y... = XX -/2
= ZDD -U" = 2UT

---

ford and Ferguson, 1970; Browne, 2001), this mea- sure can be used to find an optimal R by minimiz- ing Ja (ZR), and use ZR. Different values of correspond to different rotation methods, such as quartimax (K- = 0), varimax (K = 1/nn, parsimax (n = - () + d 2)), or factor parsimony (K 1).
Quartimax
Varimax
Parsimax
Parsimony
DistRatio
1.26
1.26
1.26
1.26
Table 8: Consistency of word meaning in the word intrusion task with Crawford- rotation family.
E.3 Interpretability: word intrusion task
Z is a whitened matrix, the resulting matrix
Selection of the intruder word. Our objective is to assess the interpretability of the word embed- dings Y R where each row vector yr E IRd word w;. order select the Wintruder (a) for the set of top & words of each axis E {I...., d). denoted as (a), we randomly chose a word from a pool of words that satisfy both of the following criteria simultaneously: (i) the word ranks in the lower 50% in terms of the component value on the axis a, and (ii) it ranks in the top 10% in terms of the component value some axis other than a.
Y = ZR. is almost the same regardless of the choice of k. This is because the second term of (Y) satisfies Σx=1 Σi=1 Eif Wi = dn² Ek=1 Ex= Uik = dn²(1 + Op(n !)) as the rotated matrix Y is also whitened, i.c.. E /ig/n = 1. Therefore, the second term is almost constant with respect to Y, and the result of the minimization is not significantly influenced by the value of K.
corresponds
E.2 Unwhitened embeddings
We introduced some embeddings obtained by rotat- ing the centered embeddings X without rescaling.
Evaluation metric. We adopted the following metric proposed by Sun et al. (2016).
PCA. The diagonal elements of D represent the standard deviations of X in the directions of the principal components. Simply rescaling Z by these standard deviations results in the same scaling as X, yielding
InterDist(a) IntraDist
DistRatio
dist(w;; (w;, w³)
IntraDist(a) =
Σ
k(k
Xpea :- ZD = XU.
wi,wyytopy.a]
dist(w;, Wintruder (a))
ICA.
Since
ZRica, we define
InterDist(o)
Σ
k
mn (t)
Xica = X Xpea = X(URica).
In this formula, we defined dis wy) = lly: yill. Here, IntraDist(a)) denotes the average dis- tance between the top k words, and InterDist(a)) represents the average distance between the top words and the intruder word. The score is higher when the intruder word is further away from the set top (a). Therefore, this score serves quantita- tive measure of the ability to identify the intruder word, thus it is used as measure of the consis- tency of the meaning of the top h words and the interpretability of axes.
It should be noted that columns of S are intended to be independent random variables, while this is not the case for columns of Xica-
ZCA. Given that Y ZUT, we define
X X pea U X,
which brings us back to the original X. This ex- plains that ZCA involves only scaling without rota- tion.
Results for Crawford- FFrouu-ne rotation family. Table 8 shows the DistRatio for whitened embed- dings with the four different choices of value. As have discussed in Appendix E.1, there is no significant difference between the four rotation methods. So, we presented the result for the well- known varimax rotation in Table of Section 6.1.
Crawford-Fergusse rotation family. We sim- ply apply the optimization procedure X. Specif- ically, find an optimal R by minimizing fa (XR),, and use XR. For unwhitened matrix X, the minimization of (XR) depends on the value of k.

---

E=1
A=
10
100
500
Tasks
ZCA
PCA
Vari.
ICA
ZCA
PCA
Vari.
ICA
ZCA
PCA
Vari.
ICA
ZCA
PCA
Vari.
ICA
rapital-commoon-cenniiss
0.00
0.03
0.17
0.57
0.26
0.57
0.44
0.94
0.98
0.94
0.97
0.97
0.97
0.97
0.97
capital-worid
0.00
0.01
0.08
0.33
0.13
0.20
0.34
0.74
0.87
0.87
0.89
0.92
0.92
0.92
0.92
0.92
currency
0.00
0.05
0.17
0.24
0.27
0.23
0.23
city-in-state
0.00
0.01
0.00
0.14
0,14
0.15
0.08
0.33
0.63
0.67
0.62
0.66
0.72
0.72
0,72
0.72
family
0.00
0.06
0.19
0.21
0.16
0.23
0.37
0.51
0.73
0.70
0.75
0.75
0.81
0.81
0.81
0.81
grami adjsttiee-ooddeee
0.01
0.00
0.11
0,13
0,13
gram2 opposite
0.00
0.00
0.00
0,00
0.00
0.00
0.00
0.01
0.09
0.12
0.11
0.13
0.14
0.14
0.14
0.14
Analogy
grum3 comparative
0.00
0.06
0.33
0.34
0.38
0.42
0.42
0.42
grams-ouperlative
0.00
0.00
0.00
0.12
0,02
0,02
0.01
0.14
0.25
0.25
0.25
0.29
0.32
0.32
0.32
0.32
gramS present participle
0.00
0.02
0.00
0.03
0.04
0.04
0.05
0.19
0.32
0.30
0.35
0.38
0.40
0.40
0.40
0.40
0.01
0.05
0.98
0.99
gram? past tense
0.00
0.01
0.00
0.06
0.02
0.03
0.09
0.17
0.31
0.31
0.34
0.34
0.40
0.40
0.40
0.40
grami-plural gram9 plural verbs
0.00
0.00
0.09
0.24
0.19
0.15
0.30
0.54
0.68
0.72
0.09
0.74
0.788
0.78
0.78
0.78
0.00
0.03
0.00
0.05
0.01
0.03
0.04
0.15
0.28
0.30
0.31
0.35
0.39
0.39
0.39
0.39
Average
0.00
0.02
0.07
0.18
0,10
0.13
0.18
0.37
0.48
0.48
0.49
0.52
0.54
0.54
0.54
0.54
0.08
0.00
0.30
0.65
0.67
0.68
0.70
0.70
0.30
0.70
WS353
0.14
0.21
0.31
0.33
0.31
0.32
0.46
0.66
0.64
0.67
0.68
0.71
0.71
0.71
0.71
0.71
MTark
0.10
0.07
0.26
0.37
0.25
0.33
0.47
0.57
0.54
0.52
0.54
0.58
0.55
0.55
0.55
0.55
Similarity
-0.05
0.02
0.08
0.07
0.35
0:29
0.35
0.35
0.35
SimLex999
0.05
0.00
-0.01
0.03
0.08
0.13
0.10
0.14
0.25
0.25
0.22
0.23
0.26
0.26
0.26
0.26
Vor33303000
-0.03
-0.02
0.01
0.03
0.04
0.12
0.11
0.12
0.15
0.15
0.15
0.15
Avenge
0.05
0.06 0.14
0.20
0.17
0.20
0.27
0.36
0.4
0.42
0:42
0.44
0.45
0.45
0.45
0.45
Table 9: The performance of whitened embeddings (Appendix E.1) with components of top k absolute value was evaluated. The values represent the top-10 10 accuracy for analogy tasks and the Spearman rank correlation for word similarity tasks.
A
10
100
300
Tasks
Orig:
Parsi
Orig.
PCA
Parsi.
ICA
Orig.
PCA
Quaeti.
ICA
Orig:
PCA
Vari.
ICA
copitul--ommon common countries
0.01
0.02
0.32
0.56
0.32
0.49
0.84
0.94
0.98
0.98
0.97
0.99
0.98
0.98
0.98
0.98
capital-world
0.90
0.01
0:21
0.32
0.30
0.36
0.64
0.82
0.92
0.92
0.93
0.95
0.95
0.95
0.95
0.95
currency
0.30
0.00
0.27
0.22
0.04
0.09
0,28
0.28
0.23
0.25
0.24
0,26
0.27
0.27
0,27
0.27
city in state family
0,00
0.00
0.08
0.13
0.16
0.19
0.21
0.37
0.67
0.74
0.69
0.73
0.76
0.76
0.76
0.76
0.00
0.05
0.26
0.21
0.25
0.41
0.51
0.53
0.78
0.50
0.84
0.83
0.86
0.86
0.86
0.86
graml -xdjective-o-ddveeb
0.81
0.00
0.00
0.00
0.01
0.04
0.03
0.05
0.19
0.20
0.19
0.15
0.22
0.22
0.22
0.22
gram2-opposite
0.02
0.03
0.16
0.14
0.18
0.20
0.20
0.20
Analogy
gram3 comparative
0.01
0.00
0.02
0.23
0.09
0.07
0.12
0.40
0.44
0.47
0.48
0.53
0.56
0.56
0.56
0.56
gram4 superlative
0.03
0.03
0.05
0.17
0.37
0.36
0.36
0.43
0.43
0.43
grams present- purticiple
0.00
0.04
0.03
0.01
0.06
0.10
0.11
0.23
0.44
0:43
0.44
0:48
0.52
0.52
0.52
0.52
gramb aallionnityy-adeeeit
0,91
0.01
0.21
0.37
0.37
0.49
0.71
0.89
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
gram'- pere--ppt--enee past-tense
0.00
0.07
0.04
0.06
0.08
0.11
0.12
0.26
0.44
0.44
042
0.44
0.52
0.52
0.52
gram8- planal
0.02
0.00
0.13
0.23
0.21
0.19
0.32
0.55
0.74
0.80
0.78
0.80
0.84
0.84
0.84
0.84
gram9 plural-verbs
0.03
0.23
0.48
0.46
0.44
0.52
0.52
Average
0.90
0.02
0.11
0.18
0.13
0.19
0.29
0.41
0.54
0.57
0.57
0.58
0.62
0.62
0.62
0.62
Table 10: The performance of unwhitened embeddings (Appendix E.2) with components of top & absolute value was evaluated. The values represent the top- accuracy for analogy tasks.
A
10
A
100
k
300
Tasks
Orig.
PCA
Vari.
ICA
Orig.
PCA
Quarti.
ICA
Orig.
PCA
Vari.
ICA
Orig.
PCA
Vari.
ICA
MEN
0.12
0.13
0.34
0,36
0.35
0,34
0.53
0.60
0.68
0.68
0.69
0.70
0.72
0.72
0.72
0.72
WS353
0.13
0.11
0.40
0.32
0.33
0.40
0.58
0.67
0.68
0.70
0.73
0.73
0.73
0.73
0.73
0.73
MTurk
0.21
0.38
0.28
0.46
0.62
0.62
0.61
0.64
0.62
0.64
0.64
0.64
Similarity
RW
0.06
0.13
0.13
0.19
0.14
0.22
0.37
0.34
0.34
0.37
0.38
0.38
0.38
Simlex999
0.05
0.06
-0.00
0.06
0.10
0.17
0.12
0.16
0.25
0,26
0,25
0.25
0.27
0.27
0.27
0.27
Sim Verb3300
-0.03
0.03
-0.06
0.02
0.03
0.07
0.06
0.07
0.14
0.14
0.14
0.14
0.16
0.16
0.16
0.16
Average
0.08
0.10
0.19
0.21
0.20
0.27
0.34
0.39
0.46
0.46
0.46
0.47
0.48
0.48
0.48
0.48
Table 11: The performance of unwhitened embeddings (Appendix E.2) with components of top i absolute value was evaluated. The values represent the Spearman rank correlation for word similarity tasks.
E,4 Low-dimensionality: analogy task & word similarity task
types of word relations for the analogy task. Each task is composed of four words that follow the re- lation wi ww = w3 wcs Using w1, we and wy. we calculated W3 + w2 w1 and identified the top
Analogy task. We used the Google analogy dataset (Mikolov et al., 2013a), which includes 14

---

A
10
A:
100
- 300
Quarti
Vari.
Parsi.
FacPunai
Quarti.
Vari.
Parsi.
FacPursi.
Quarti.
Vari.
Parsi.
FacPuns.
Quarti.
Vari
Parsi.
FacPansi
Analogy
Average
0.07
0.07
0.07
0.07
0.18
0.18
0.18
0.18
0.49
0.49
0.49
0.69
0.54
0.54
0.54
0.54
Similarity
0.14
0.14
0.14
0.14
0.27
0.27
0.27
0.27
0.42
0.42
0:42
0.42
0.45
0.45
0.45
0.45
Table 12: The average performance of the four rotation methods for whitened embeddings. The values are averaged over 14 analogy tasks and six word similarity tasks.
A =
=
10
k 100
It = 300
Quarti
Vari.
Parsi.
FaaParsii
Parsi.
FacParsi.
Quami.
Vari.
Parsi.
FacParsi.
Quarti
Vari.
Parsi.
FacParsi.
Anology
Average
0.04
0.03
0.11
0.03
0.24
0.22
0.29
0.20
0.57
0.56
0.56
0.55
0.62
0.62
0.62
0.62
Similarity
Average
0.18
0.19
0.12
0.32
0.46
0.48
0148
0.48
Table 13: The average performance of the four rotation methods for unwhitened embeddings. The values are averaged over analogy tasks and word similarity tasks.
Analogy teppiall.commonn-conniiees Word Similarity (WS353)
whitened embeddings and Tables 10, 11 for un- whitened embeddings. These results are derived from varying the number of non-zero components k, to k 1, 10, 100, and 300 for each em- bedding. The performance of a specific case in analogy and word similarity tasks is also illustrated in Fig. 17.
0.
acc
3.6
Spearman's
3.5
3.4
0.4
ZCA
Verimax
ICA
122 Number nan-zoro axos
Numbor of nan-zoro
For 300, all the components the 300- dimensional word vectors were used as they are. Note that the performance for = 300 is identical for all the whitened (or unwhitened) embeddings, because the difference is only their rotations, and both analogy tasks and similarity tasks are based on the inner product of embeddings.
Figure 17: The performance of several whitened word embeddings when reducing the non-zero components. The top-10 accuracy for the analogy task (capital- common-countries and the Spearman rank correlation for the word similarity task (WS353).
10 words with the highest cosine similarity to see if w4 included in them.
A Although there is tendency for accuracy decrease when reducing the number of non-zero components, it can be confirmed that the degree of decrease is smaller when using ICA compared to the other methods. The specific tasks depicted in Fig. 17 are those that achieved the highest per- formance at k = 300 in Table 9; capital- -ooimal-common- countries has the highest top-10 10 accuracy 0.97 in the analogy tasks, and WS353 has the highest Spearman correlation 0.71 in the word similarity tasks.
Word similarity task. We used MEN (Bruni al., 2014), WS353 (FFinkelstein et al., 2002), MTurk (Radinsky et al., 201 1), RW (Luong et al., 2013), SimLcx999 (Hill et al., 2015), and Sim Verb- 3500 (Gerz et al., 2016). They provide word pairs along with human-rated similarity scores. As the evaluation metric, we used the Spearman rank cor- relation coefficient between the human ratings and the cosine similarity of the word embeddings.
Results with embeddings that are transformed by the Seawford-Ferguuse rotation family are shown in Table 12 for whitened embeddings and Table 13 for unwhitened embeddings. For whitened embed- dings, there is no significant difference between the four rotation methods as discussed in Appendix 1. So, we presented the results for the well-known varimax rotation in Section 6.2 and Table 9. For unwhitened embeddings, quartimax, varimax, and parsimax were similarly good. This result con- sistent with the findings of Park et al. (2017). The best rotation was identified as boldface Table 13
Reducing the non-zero components. For each transformed embedding y (y1,..., ya) and specified value of k. retained only the & most significant components of y. For example, if the components are lyi 2 1821 2 2 Iyal. then we used (y1, ./...... V/. O, .,) E Rdd where d - k least significant components are replaced by zero.
Results. The detailed results of the experiments presented in Section 6.2 are shown in Table 9 for

---

for the analogy task and word similarity task at each k, and the selected rolation method was used in Tables 10, 11.
(0, 1/d), the normal distribution with mean 0 and variance 1/d, and 4 Exp(1), the exponen- tial distribution with mean 1. The random matrices M and N primarily induce rotation because they are roughly orthogonal matrices, while L induces random scaling.
E.5
Cross-lingual alignment
Dataset
ES
FR
DE
IT
RU
Word translation pairs. We established the cor- respondence between the embeddings of English and the embeddings of other languages in the -aatteg and MUSE-fastText datasets. To accomplish this, we followed the procedure out- lined in Appendix C, but now we applied it to both the train set and the test set of MUSE dictionaries. The results of applying this procedure the five target languages are presented in Table 14, where 2447
Pains
11965
10861
14669
9648
10883
157langs Train Source
4991
4994
4995
4993
4996
Target
10154
9194
11221
8622
9512
Pairs
2975
2943
3660
2585
2447
157langs Test
Source
1500
1500
1500
1500
1500
Target
2869
2855
3429
2532
2374
Pairs
11977
10872
14677
9657
10887
MUSE Train
Source
5000
5000
5000
5000
5000
Target
10166
9205
11229
8631
9516
Pairs
2975
2943
3660
2585
the source language is English. The train pairs were used for training supervised baselines and also for computing cross-correlation coefficients. The test pairs were used for computing the top-1 accuracy.
MUSE Test
Source
1500
1500
1500
1500
1500
Target
2869
2855
3429
2532
2374
Table 14: The number of translation pairs, unique source English words, and unique target words for each target language.
Supervised baselines. Two supervised baselines were considered to learn a linear transformation from the source embedding X to the target em- bedding Y. We rearranged the word embeddings so that each row of X and Y corresponds to a translation pair, i.e., the meaning of the i-th row X; corresponds to that of yı. We then computed the optimal transformation matrix W c IRddd that solves the least squares (LS) problem (Mikolov et al., 2013b):
Datasets and visual inspection. In addition to the fastText by Grave et al. (2018) utilized in Sec- tion 4, we employed fastText by MUSE (Lam- ple et al.. 2018). We refer to these word embed- dings as fasttngs-fastText and MusE-fastText, and chose some of the languages common to these two datasets. For the cross-lingust alignment task, English (EN) was designated as the source lan- guage, while Spanish (ES), French (FR), German (DE), Italian (IT), and Russian (RU) were specified as the target languages. Following the same proce- dure as in Appendix C, we limited the vocabulary size to 50,000 in each language. The embeddings for the six languages are visualized in Fig. 18, by applying the same procedure in Fig. 2.
min XW YI
WeR
In the optimization of the Procrustes (Proc) prob- lem, the transformation matrix W was restricted an orthogonal matrix. Although LSS more flexi- ble, the performance of cross-lingual alignment can possibly be improved by Proc (Xing et al., 2015; Artetxe et al., 2016). In these supervised methods, the two embeddings X and Y underwent centering and normalization as preprocessing steps.
Applying a random transformation. Note that fusE-fastText already has pre-aligned word em- beddings across languages. To resolve any such relationship across languages, we applied a random transformation to embeddings. For each embed- ding matrix X E paxd we generated random matrix Q E Rdxd to compute XQ. The random matrix was generated independently as
Cross-lingual alignment methods. We consid- ered both supervised and unsupervised transforma- tions for cross-lingual alignment from the source language to the target languages. In the super- vised transformation methods, LS and Proc, we first trained the linear transformation using both the source and target embeddings.
Q = MLN,
where all the elements of M, N E Rdxd and L diag(11....,44 E ddxd are distributed indepen- dently. Specifically, the elements are Maj, Nij- ~
In the unsupervised transformation methods, PCA and ICA, we first applicd the transformation individually to each language and then permuted

---

English
Spanish
French
German
Italian
Russian
1.0
0.8
0.6
20 40
20 40
60
20 40
60 80
0 20 40 60 80
20 40
60
20 40
60
80
0.0
3
(a) Normalized ICA-trunsformed embeddings from multiple languages.
English
Spanish
French
German
Italian
Russian
1.0
20 90
20 40
80
40 60 80
20 60 80
20 49
00
20 40 00
80
0.4
0.2
0.0
(b) Normalized PCA-transformed embeddings from multiple languages.
Figure 18: The embeddings for the six languages are visualized. They were obtained by applying the same procedure
Fig. 2. Details of the datasets presented in Appendix E.5.
Original
Random transformation
Dataset
Method
ES
FR
DE
IT
RU
Avg.
ES
FR
DE
IT
RU
Avg.
LS
89.33
87.00
85.73
87.20
74.33
84,72
66.93
62.73
58.53
44.33
43.87
55.28
Proc
88.40
86.93
86.80
86.80
75.80
84,95
26.00
20.93
20.00
14.73
11.93
18.72
157langs-fastText
ICA
27.20
22.20
18.93
17.73
11.33
19.48
22.73
21.40
19.87
17.00
12.00
18.60
PCA
1.53
0.87
0.80
0.80
0.60
0.92
0.80
0.67
0.67
0.60
0.47
0.64
84.47
80.53
80.53
65.13
78,91
60.60
59.73
44.33
39.00
51.44
Proc
84.40
83.13
80.87
79.87
63.80
78.41
25.87
23.13
21.40
17.40
13.13
20.19
ICA
53.00
54.27
45.80
43.60
19.67
43.27
52.93
52.80
45.93
43.33
21.53
43.31
PCA
2.73
3.00
1.67
1.93
1.33
2.13
0.67
0.47
0.53
0.07
0.27
0.40
Table 15: The 1 accuracy the cross -ingull alignment task from English to other languages. Two datasets of fastText embeddings (1557angs and MUSE) were evaluated with the two types of embeddings (Original and maandommtransformattion LS and Proc are supervised transformations using both the source and target embeddings. while ICA and PCA are unsupervised transformations.
the axes based on the cross-correlation (see Ap- pendix C). Although PCA and ICA are unsuper- vised transformations, the axis permutation is su- pervised because cross-correlation coefficients are computed from the embeddings of both languages.
Evaluation metric. For each word in the source language, we computed the transformed embed- ding and found the closest embedding from the target language in terms of cosine similarity. To mitigate the hubness problem, used the CSLS

---

method (Lample et al., 2018) instead of the stan- dard k-NN method. The top-1 accuracy was com- puted as the frequency of finding the correct trans- lation word.
The top-1 accuracy for all the five tar- get languages is shown in Table 15, and only the average value is shown in Table 3. We used two datasets: 157langs-fastTes and MUSE-fastText. Two types of embeddings were considered: one us- ing the original word embeddings for all languages, and the other applying a random transformation to all the embeddings. The conclusions obtained in Section 6.3 regarding the average results of cross- lingual aligmment hold true when considering each of the target languages.
Results.


Reviews:
[{"review_id": "ywnueuvz1e", "reviewer": "Reviewer_PoMJ", "summary": "", "questions": "1. It's unclear to me how the five axes in Fig.1 (Left) are associated with the five words, dishes, cars, film, Italian, and Japanese.\n2. It's not so clear what does it mean by \"intrinsic interpretable axes\" in the abstract.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\nFor the reasons to reject, below are our responses:\n\n---\n\n# Question about Clarity of the Paper (1/2)\n\n> It's unclear to me how the five axes in Fig.1 (Left) are associated with the five words, dishes, cars, film, Italian, and Japanese.\n\nThank you for your feedback. For clarity, we will add the following explanation.\n\nWords like dishes, meat, noodles, etc. have high values on the \"axis 0\", while words like cars, car, ferrari, etc. have high values on the \"axis 1\". In this way, each axis can be interpreted by a set of semantically related words. For the sake of interpretation of the axis, we have labeled the axes in the figure with the word with the highest value on each axis (e.g., 0:dishes, 1:cars, etc.). We are working on a clearer explanation and may update the figure and the caption in the camera-ready version. Thanks for your constructive feedback!\n\n---\n\n# Question about Clarity of the Paper (2/2)\n\n> It's not so clear what does it mean by \"intrinsic interpretable axes\" in the abstract.\n\nWe apologize for any confusion caused by our explanation. You're right; we didn't clarify this in the main text. Thank you for pointing this out. The axes of the ICA-transformed embeddings can be interpreted based on the set of words with high component values (interpretable axes). Since these axes are obtained by linearly transforming the original embeddings, they exist intrinsically in the original embeddings (intrinsic axes). We will explain this in the camera-ready version.\n\n---\n\n# Question about the observation on the composition of semantic axes\n\n> The observation on the composition of semantic axes works the best for English fasttext but is not so clear for other languages and BERT.\n\nWe apologize for the lack of clarity in our paper, and it appears that there may be a misunderstanding. Thank you for bringing this to our attention. We recognize the need to provide clearer explanations in our paper. In the section you point out, we illustrate how axis matching works well for English fastText as base embedding compared to other embeddings.\n\nEnglish fastText is used as **the base embedding** (or a reference embedding) to be compared with other embeddings. In Fig. 2, we show the alignment of other languages with English, such as English-Spanish, ...,  English-Japanese. Similarly, in Fig. 4, we use English fastText as the base embedding to examine its alignment with BERT embedding.\n\nTo further explain, for the \"Normalized ICA-transformed word embeddings\" languages closely related to English, such as Spanish, show significant component values for the top-ranked translated pairs (darker colors), while languages more distant to English, such as Chinese and Japanese, show smaller component values (lighter colors). Although fastText is a static embedding and BERT is dynamic, we observed good alignment between their axes and similar component values for the same words.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The finding is interesting and strengthens the understanding of embeddings.\n2. The result in Fig. 6 is encouraging that embeddings can be projected to a lower dimension using ICA while preserving the performance.", "reasons_to_reject": "1. Clarity of the paper can be improved (see questions below).\n2. The observation on the composition of semantic axes works the best for English fasttext but is not so clear for other languages and BERT.\n\n===== After Rebuttal =====\n\nThanks to the author for the clear explanations in the response. All my questions are addressed. Thus, I decided to raise the Soundness to 4 (strong).", "questions_to_authors": "1. It's unclear to me how the five axes in Fig.1 (Left) are associated with the five words, dishes, cars, film, Italian, and Japanese.\n2. It's not so clear what does it mean by \"intrinsic interpretable axes\" in the abstract.", "paper_topic_and_main_contributions": "The paper tries to understand the universal geometry in embeddings using independent principal analysis (ICA) and principle component analysis (PCA). The study shows that each embedding can be represented in low dimensions using ICA-transformed axes. In addition, different languages (English v.s. Spanish), algorithms (fasttext v.s. BERT), and modalities (words v.s. images) are considered in the study.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1visDKeRAP", "reviewer": "Reviewer_fVPb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\nFor the reasons to reject, below are our responses:\n\n---\n\n# Question about the details of FastICA\n\n> Given how much of the analysis hinges on ICA, it would have been helpful to provide more details in the main text about the FastICA approach. For example, does it make any assumptions? Is it exact and if not what sort of approximation error is incurred?\n\nThank you for pointing out the importance of providing more details about the FastICA approach, given its importance in our analysis. Changing the parameter settings of FastICA does not significantly affect the results. In the paper, we set the number of iterations for FastICA to 10,000. However, even when reduced to 200 iterations, important axes could still be extracted using ICA. In the revision, we will try to include some experimental results for other settings of ICA parameters.\n\n---\n\n# Question about sentence embeddings \n\n> Although ViT-base is extracted from CLIP which learns from both images and text (Appendix C), I think this paper may benefit from more focus on text. For example, what about popular contrastively trained text embeddings like sentence-bert (SBERT)? It would be surprising and interesting if the findings applied both to generatively trained and discriminatively trained representations. What about the impact of model size/capacity?\n\nWe appreciate your suggestion that our work could benefit from a more focused investigation of text embeddings. Your suggestion to consider embeddings such as Sentence-BERT (SBERT) is well taken. The potential impact of model size and capacity is indeed intriguing, and we look forward to exploring this aspect in future work.\n\nAs a first step into sentence embeddings, we examined the ICA-transformed CLS token of BERT. This is because models such as Sentence-BERT and SimCSE, which are based on BERT, often use the CLS token as a representation for sentence embeddings that are further refined by fine-tuning. We observed several axes in the ICA-transformed BERT embeddings where CLS and SEP tokens were highly ranked. When we examined the original sentences associated with these special tokens, we found that they had consistent semantic meaning along certain axes. For example, we noticed different axes representing \"headline\" and \"baseball\" topics. Examples include\n\nThe axis interpreted as \"headline\":\n- LOS ANGELES , California ( CNN ) -- A 24-year-old gang member was arrested Thursday in connection with...\n- PARIS ( Reuters ) - The body of the pilot of an Air France plane that crashed into the Atlantic on June...\n- NEW YORK ( AP ) - An undercover cop chased a Times Square scam artist through sidewalks crowded with...\n\nThe axis interpreted as \"baseball\":\n- Hernandez escaped trouble in the third and fourth innings , then led off the fifth by hitting Byrd .\n- Victorino staked Philadelphia to a 1-0 lead in the first inning with a home run to right field His bloop...\n- The Angels took a lead in the third when rookie shortstop Jed Lowrie went to field a grounder with two...\n\nThese results suggest that BERT's pre-training successfully encodes the semantics of a sentence into its special tokens. As Sentence-BERT and SimCSE undergo additional fine-tuning, it's possible that such trends in these models will become even more apparent. As we considered the axis matching of multiple image model embeddings from a single image in Fig. 5, we believe that an experiment could evaluate the axis matching of sentence embeddings derived from multiple models for a single sentence.\n\n---\n\n# Quesiton about the embedding dimensionality\n\n> It is somewhat well known that, across modalities, higher dimensionality embeddings often result in significantly better downstream performance. This is somewhat at odds with the conclusions of this paper that suggest that lower dimensional space is sufficient to capture the important information. What's lost in the ICA project? Are the embeddings less robust, for example?\n\nAs you pointed out, higher-dimensional embeddings tend to perform better in downstream tasks. ICA linearly transforms the embeddings to make them sparse, thus extracting a few essential axes for each word. Even when components with absolute values below a certain threshold are set to zero, the performance degradation is not significant (as shown in Fig. 6). However, these small-value components also encode some information, so setting them to zero results in a loss of information at some degree.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Although there have been other studies that have applied ICA to word embeddings, this is the first to show that it yields embeddings that align to a significant degree across languages without explicit alignment between them (the \"universal\" claim).\n\n* The study comprises an analysis of multiple text and image embedding models, including BERT, fastText, SGNS, ViT, ResMLT, Swin-S, ResNet, and RegNet.\n\n* The findings regarding the degree of alignment of various independently trained embeddings under ICA, without explicit alignment steps, are somewhat surprising, and validate the \"universal\" claim.", "reasons_to_reject": "* Given how much of the analysis hinges on ICA, it would have been helpful to provide more details in the *main text* about the FastICA approach. For example, does it make any assumptions? Is it exact and if not what sort of approximation error is incurred?\n\n* Although ViT-base is extracted from CLIP which learns from both images and text (Appendix C), I think this paper may benefit from more focus on text. For example, what about popular contrastively trained text embeddings like sentence-bert (SBERT)? It would be surprising and interesting if the findings applied both to generatively trained and discriminatively trained representations. What about the impact of model size/capacity?\n\n* It is somewhat well known that, across modalities, higher dimensionality embeddings often result in significantly better downstream performance. This is somewhat at odds with the conclusions of this paper that suggest that lower dimensional space is sufficient to capture the important information. What's lost in the ICA project? Are the embeddings less robust, for example?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper contributes an analysis of the geometry of embeddings for two domains: text and vision. The paper uses independent component analysis (ICA) to find a transformation of the embedding space in which the columns are independent components using the FastICA implementation in scikit-learn. The main claim put forth is that examples (e.g., words) can be represented as a set of intrinsic interpretable axes under ICA, and that the number of dimensions needed to represent each word is considerably less than the actual dimensions of the embeddings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L3olLuWWwI", "reviewer": "Reviewer_QYwU", "summary": "", "questions": "1. The pretrained word embeddings are generally coming from complex nonlinear neural network. However, this work mainly use some linear analysis to get the results, like PCA and ICA. It is not well understood if those linear model are enough to analyze the nonlinear relationships among pretrained word embeddings.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\n\n---\n# Question regarding the Linearity of Analytical Method\n\n> The pretrained word embeddings are generally coming from complex nonlinear neural network. However, this work mainly use some linear analysis to get the results, like PCA and ICA. It is not well understood if those linear model are enough to analyze the nonlinear relationships among pretrained word embeddings.\n\nThank you for your insight. \n\nWhile we recognize the potential of nonlinear analyses, we think that even even linear analyses such as PCA and ICA can provide good analyses for understanding the nonlinear relationships between pretrained word embeddings. This is because  it's well known that linear relationships can provide good analyses for understanding the nonlinear relationships between pretrained word embeddings, e.g. **\"king - man + woman = queen\"**\n\n---\n\n# Discussion on Limitation of Technical Contribution\n\n> The technique contribution is limited as it mainly use PCA/ICA to do post-hot analysis.\n\nThank you for your insight. Indeed, the technical aspect of our research is limited to mainly using ICA to do post-hoc analysis. We agree that embeddings are so complicated that it requires more advanced technical contributions to deepen our understanding. \n\nOn the other hand, despite the simplicity of our analysis, we have gained new insights into the geometry and interpretability of different types of embeddings. These results may be useful for future technical analyses of embeddings. \n\nGiven the importance of the points you raise, we intend to include this perspective in the discussion of our camera-ready version.\n\n---\n\n# Source Code Submission\n\n> It is better to provide the source code for better reproducibility.\n\nThank you for your feedback. We recognize the importance of reproducibility. We plan to release the source code after making sure it's well organized. When we do, we will include a link to it in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper has some good findings that pretrained words embedding can be expressed as a composition of a few intrinsic interpretable axes and that remain consistent across different languages, algorithms, and modalities.\n\n2. The paper is well written and provide many experimental results to validate their assumption. This findings may be useful for NLP model design.", "reasons_to_reject": "1. The pretrained word embeddings are generally coming from complex nonlinear neural network. However, this work mainly use some linear analysis to get the results, like PCA and ICA. It is not well understood if those linear model are enough to analyze the nonlinear relationships among pretrained word embeddings.\n\n2. It is better to provide the source code for  better reproducibility.\n\n3. The technique contribution is limited as it mainly use PCA/ICA to do post-hot analysis.", "questions_to_authors": "1. The pretrained word embeddings are generally coming from complex nonlinear neural network. However, this work mainly use some linear analysis to get the results, like PCA and ICA. It is not well understood if those linear model are enough to analyze the nonlinear relationships among pretrained word embeddings.", "paper_topic_and_main_contributions": "this paper found that words embedding can be expressed as a composition of a few intrinsic interpretable axes and that remain consistent across different languages, algorithms, and modalities.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper undertakes an exploration of the universal geometry within embeddings through independent principal analysis (ICA). The study demonstrates that embeddings can be effectively represented in lower dimensions using ICA-transformed axes, encompassing different languages, algorithms, and modalities.

Overall, the reviewers generally find the paper sound. All reviewers agree that the paper is well-written and extensively supported by experimental results, enhancing its credibility and potential usefulness for NLP model design. In light of these merits, the consensus among reviewers is to accept the paper for publication.
Decision: Accept-Main