<figure><img src="image_1.png" /></figure>

HAL
open science

Comparison of the Best Parameter Settings in the
Creation and Comparison of Feature Vectors in
Distributional Semantic Models Across Multiple
Languages

András Dobó, János Csirik

<figure><img src="image_2.png" /></figure>

► To cite this version:

András Dobó, János Csirik. Comparison of the Best Parameter Settings in the Creation and Comparison of Feature Vectors in Distributional Semantic Models Across Multiple Languages. 15th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), May 2019, Hersonissos, Greece. pp.487-499, 10.1007/978-3-030-19823-7_41. hal-02331340

HAL Id: hal-02331340

https://inria.hal.science/hal-02331340

Submitted on 24 Oct 2019

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Comparison of the best parameter settings in the
creation and comparison of feature vectors in
distributional semantic models across multiple
languages

András Dobó¹[0000-0002-8640-6073] and János Csirik¹[0000-0003-3870-0363]

University of Szeged, Szeged, Hungary
{dobo,csirik}@inf.u-szeged.hu

**Abstract.** Measuring the semantic similarity and relatedness of words is impor-
tant for many natural language processing tasks. Although distributional semantic
models designed for this task have many different parameters, such as vector sim-
ilarity measures, weighting schemes and dimensionality reduction techniques,
there is no truly comprehensive study simultaneously evaluating these parame-
ters while also analysing the differences in the findings for multiple languages.
We would like to address this gap with our systematic study by searching for the
best combination of parameter settings in the creation and comparison of feature
vectors in distributional semantic models for English, Spanish and Hungarian
separately, and then comparing our findings across these languages.

During our extensive analysis we test a large number of possible settings for all
parameters, with more than a thousand novel variants in case of some of them. As
a result of this we were able to find such combinations of parameter settings that
significantly outperform conventional settings combinations and achieve state-of-
the-art results.

**Keywords:** distributional semantic models · semantic similarity and relatedness · best combination of parameter settings · comparison of findings across languages · English, Spanish and Hungarian.

# 1 Introduction

In many NLP problems, including information retrieval [16], spelling correction [3] and
noun compound interpretation [14] among many others, knowing the semantic similar-
ity or semantic relatedness of words can be very useful. Although distributional se-
mantic models (DSMs) [1] calculating these measures have many possible parameters,
most research focuses only on one or two aspects of these models, while using some
conventional settings for the rest of the parameters (e.g. cosine as vector similarity and
(positive) pointwise mutual information as weighting). Therefore a truly comprehensive
study evaluating the numerous parameters of DSMs for any language is still missing,
and would be needed, as also suggested by [21]. Moreover, despite the fact that the
best parameter settings for the parameters can differ for different languages, the vast
majority of papers consider DSMs for only one language (mostly English), or consider
multiple languages but without a real comparison of findings across languages. In this
article we would like to address these gaps.

There are two distinct phases of DSMs in general: the extraction of statistical in-
formation from raw text, and the creation and comparison of feature vectors for words
based on the extracted information. Within this study we focus on the parameters of the
second phase, as the two phases are relatively distinct from each other, and the number
of possible combinations of parameter settings (CPS) for the second phase is already
well in the trillions. So we are searching for the best CPS of those 10 parameters during
the creation and comparison of feature vectors in DSMs that we considered important,
for English, Spanish and Hungarian separately, and then we compare our findings for
the different languages.

A detailed description of our analysis can be found in [13], where tests were done
only for English, without any comparison of findings across languages, and with far
less settings tested for several parameters.

## 2 Background

Although there are a vast number of studies dealing with DSMs, most of them only
consider one or two parameters of these models, and take the others granted with
some standard setting. Most commonly, these models use cosine as vector similar-
ity [35,2,1,27,21,17,29,8,32,30,6,34] and (positive) pointwise mutual information as
weighting [18,35,2,21,17,29,30]. Further, they also usually do not care for the interac-
tion of these parameters, and experiment with the considered parameters one by one,
and not simultaneously. Of course there are some studies that experiment with several
parameters with multiple possible settings [7,20,19], but even these are far from being
truly comprehensive.

Moreover, most models were only tested for English and neglect any other lan-
guages despite the fact that DSMs might work differently across multiple languages.
Of course, there are several studies in which results were presented for languages other
than English, including Spanish [15,23,4] and Hungarian [11,24]. However, even those
that include multiple languages usually only present some test results for the differ-
ent languages separately, without any real analysis of the differences in the findings
between the languages.

## 3 Data and evaluation methods

For input we used information extracted from the British National Corpus (BNC),
the Spanish Wikicorpus (EsWiki) [28], and the 23.01.2012 dump of the Hungarian
Wikipedia (HuWiki) for English, Spanish and Hungarian, respectively, with the help
of the information extraction methods of [11], [12] and [29].

Tests were done on parts of the MEN [2] dataset for English, the Spanish WordSimi-
larity-353 [15], the Moldovan [23] and the Spanish Rubenstein-Goodenough [5] datasets
for Spanish, and parts of the Hungarian version of the TOEFL [11] and Rubenstein-
Goodenough datasets for Hungarian. The last was constructed the same way as the
Hungarian TOEFL and Miller-Charles datasets in [11].
Out of these datasets only the MEN dataset is truly reliable, as the others are rather small and except for the Moldovan dataset just translated from English datasets, during which they can be distorted. The Hungarian datasets are especially small, and the type of the TOEFL dataset also makes the results on it even less reliable compared to the other datasets. However, due to the lack of truly suitable resources, we had to settle for these.

In case of the TOEFL dataset, the accuracy (A) of the models on the questions were calculated, while in case of all the other datasets, the Pearson's (P) and Spearman's (S) correlations with the gold standard scores and their modified harmonic mean (H) were computed, as follows:

$$H(P, S) = \frac{2 \times P \times S}{|P| + |S|} \qquad (1)$$

For more information, please refer to [13].

# 4 Our heuristic analysis

As the number of possible CPSs are in the magnitude of trillions, we had to use a heuristic approach to find the best one in case of each language. First, each parameter was tested separately on a development dataset, where a candidate list of settings were selected for each parameter. Then all combinations of the selected settings of all parameters were tested on a different dataset, to find the best CPS. These were done for the three languages separately.

The 10 parameters tested, together with the number of settings tried for them, are listed in Table 1. For several parameters, a large number of novel settings were tested. These were either brand new settings, modified versions of conventionally used settings, or combinations of multiple settings. A detailed description of the tested parameters and their settings can be found in [13].

We have to note that when using singular value decomposition (SVD) for dimensionality reduction or the various smoothing options, we usually did a smaller number of runs than in other cases due to our limited resources. Further, in case of Spanish we had to set MWFFreq to 3 instead of NoLimit when using SVD due to the too many features otherwise, which would have made running SVD unmanageable.

First we have done this two-step heuristic analysis for English and evaluated the results extensively in [13]. Then we have repeated the same analysis, with a greatly increased number of settings for several parameters, for English, Spanish and Hungarian, and compared the findings across the different languages in this article.

# 5 Results

## 5.1 Results of the first phase

During the first phase of our analysis multiple runs were done for each setting of every parameter, and the most promising ones in case of each parameter were selected to be included in the second phase. In case of English, we used half of the development part
<table><caption>**Table 1.** The tested parameters, with the number of settings tested for each</caption><thead><tr><td>Parameter</td><td>Abbreviation</td><td>Count</td></tr></thead><tbody><tr><td>Vector similarity</td><td>VecSim</td><td>1221</td></tr><tr><td>Weighting scheme</td><td>Weight</td><td>2907</td></tr><tr><td>Feature transformation</td><td>FeatTranf</td><td>22</td></tr><tr><td>Dimensionality reduction</td><td>DimRed</td><td>21</td></tr><tr><td>Smoothing</td><td>Smooth</td><td>5</td></tr><tr><td>Vector normalization</td><td>VNorm</td><td>3</td></tr><tr><td>Stop-word filtering</td><td>StopW</td><td>2</td></tr><tr><td>Minimum limits on word-feature frequencies</td><td>MWFFreq</td><td>6</td></tr><tr><td>Minimum limits on word-feature weights</td><td>MWFWeight</td><td>26</td></tr><tr><td>Minimum limits on feature frequencies</td><td>MFFreq</td><td>14</td></tr></tbody></table>

of the MEN dataset for evaluation, while for Spanish the Spanish WordSimilarity-353 dataset and for Hungarian half of the Hungarian TOEFL dataset was employed. The top 5 performing settings for each parameter are listed in Table 2 in case of each language.

Although presenting the definition of all settings for every parameter would be impossible within this article due to their large number, below we briefly define a couple of them to help interpreting our most important results.

In case of vector similarity measures, we have defined many new variants based on one or more conventional measures. For example, the best one for English is a combination of the Pearson, MarylandBridge [9] and AdjCos [31] measures, with some additional transformations:

$$ PearsMbAdjCosMod-3.Lb(u, v) = \begin{cases} 1, & d \ge 0.1 \\ \frac{d}{0.1}, & d < 0.1 \end{cases} \\ d = 0.5 \times \left( \frac{\sum_{i=1}^{n} \operatorname{sgn}(u_i - \bar{u}) \times \operatorname{lb}(|u_i - \bar{u}| + 1) \times \operatorname{sgn}(v_i - \bar{v}) \times \operatorname{lb}(|v_i - \bar{v}| + 1)}{\operatorname{lbinv}\left(\sum_{i=1}^{n} (\operatorname{lb}(|u_i - \bar{u}| + 1))^2\right)} + \right. \\ \left. \frac{\sum_{i=1}^{n} \operatorname{sgn}(u_i - \bar{u}) \times \operatorname{lb}(|u_i - \bar{u}| + 1) \times \operatorname{sgn}(v_i - \bar{v}) \times \operatorname{lb}(|v_i - \bar{v}| + 1)}{\operatorname{lbinv}\left(\sum_{i=1}^{n} (\operatorname{lb}(|v_i - \bar{v}| + 1))^2\right)} \right) \\ \operatorname{lbinv}(x) = \min(\max(\operatorname{sgn}(x) \times (2^{|x|} - 1), -2^{100}), 2^{100}) $$

On the other hand, the best vector similarity measure for Spanish is rather different. It is a modified and transformed version of the Hindle_r measure [22]:

$$ \begin{array}{l} \text{LinHindleRMod-7.1.2.Cu}(u, v) = \frac{\sqrt[3]{\sum_{i=1}^{n} \text{lhr}-1.Cu(u_i, v_i)}}{\sqrt{\sum_{i=1}^{n} u_i^2} \times \sqrt{\sum_{i=1}^{n} v_i^2}} \\ \text{lhr}-1.Cu(x, y) = \begin{cases} \min(x^3, y^3), & x \neq 0 \land y \neq 0 \\ 0, & \text{otherwise} \end{cases} \end{array} \tag{3} $$
Weighting schemes were constructed similarly as vector similarity measures, and here too there were also numerous new variants. For example, the best one for English is a combination of PMIApha [21], PMI with Laplace smoothing [33], Unisubtuples [26] and PMI with discounting factor [25]:

$$ 
\begin{align*}
PmiAl-Tc3Tw0S2P4(x, y) &= \frac{f'_{xy}}{f'_{xy} + 1} \times \frac{\min(f'_{x}, f'_{y})}{\min(f'_{x}, f'_{y}) + 1} \times \\
&\quad \left( \text{lb} \left( \frac{n'_{\alpha} \times f'_{xy}}{f'_{x} \times f'_{y}^{0.75}} \right) - 3.29 \times \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}} \right) \\
a &= f'_{xy}, \quad b = f'_{x} - f'_{xy}, \quad c = f'_{y} - f'_{xy}, \quad d = n' - f'_{x} - f'_{y} + f'_{xy} \tag{4} \\
f'_{x} &= f_{x} + 1, \quad f'_{y} = f_{y} + 1, \quad f'_{xy} = f_{xy} + 1, \quad n' = n + 1, \quad n'_{\alpha} = \left( \sum_{i=1}^{|V|} f_{i}^{0.75} \right) + 1 \\
f_{x}, f_{y}: \text{ word frequencies, } &f_{x,y}: \text{ xy tuple frequency} \\
n: \text{ total number of words in the corpus, } &|V|: \text{ size of the vocabulary}
\end{align*}
$$

In case of feature transformation, we have experimented with transforming either raw frequencies or weights, both before and after normalization, and 7 different transformation functions were tried in all cases. For smoothing, we tried different versions of the Kneser-Ney smoothing [10]. In case of dimensionality reduction, we tried a couple of different techniques, including SVD and the method of [18]. And for minimum limits on word-feature weights we have tried the following two novel variants with multiple limit values:

$$ \text{limit}(w, \minValue) = \begin{cases} w & \text{if } w \ge \minValue \\ \minValue & \text{otherwise} \end{cases} \tag{5} $$

$$ \text{zero}(w, \minValue) = \begin{cases} w & \text{if } w \ge \minValue \\ 0 & \text{otherwise} \end{cases} \tag{6} $$

The other parameters are much less complex and more commonly used in NLP, thus one should be able to have enough understanding of them from Table 1.

A more detailed description of the different parameters and settings can be found in [13] (although with far less settings for several parameters).

## 5.2 Results of the second phase

In the second phase all possible combinations of the settings of each parameter were tested in case of all three languages, in order to find the best CPS for all languages. The second half of the development part of the MEN dataset was used for testing in case of English, while the Moldovan dataset and the second part of the Hungarian TOEFL dataset were used for Spanish and Hungarian, respectively. The top 5 performing CPSs for each language are presented in Table 3.
<table><caption>**Table 2.** The top 5 performing setting for each parameter in case of all 3 languages, in descending order of maximum H scores</caption><thead><tr><th rowspan="2">Parameter</th><th colspan="2">English</th><th colspan="2">Spanish</th><th colspan="2">Hungarian</th></tr><tr><th>Setting</th><th>H</th><th>Setting</th><th>H</th><th>Setting</th><th>H</th></tr></thead><tbody><tr><td rowspan="5">VecSim</td><td>PearsMbAdjCosMod-3.Lb</td><td>0.71</td><td>LinHindleRMod-7.1.2.Cu</td><td>0.37</td><td>PearsMbMod-1.Lb</td><td>0.80</td></tr><tr><td>PearsMbAdjCosMod-4.Lb</td><td>0.71</td><td>LinHindleRMod-6.1.2.Cu</td><td>0.36</td><td>PearsMbMod-4.Lb</td><td>0.80</td></tr><tr><td>PearsMbAdjCosMod-2.Lb</td><td>0.71</td><td>LinHindleRMod-1.1.2.Cu</td><td>0.36</td><td>MbMod-6.Lb</td><td>0.80</td></tr><tr><td>PearsMbAdjCosMod-6.Lb</td><td>0.71</td><td>LinHindleRMod-7.1.2.Sq</td><td>0.36</td><td>PearsMbMod-5.Lb</td><td>0.80</td></tr><tr><td>PearsMbAdjCosMod-6.Sigm</td><td>0.71</td><td>LinHindleRMod-3.1.2.Sq</td><td>0.36</td><td>PearsMbMod-2.Lb</td><td>0.80</td></tr><tr><td rowspan="5">Weight</td><td>PmiAl-Tc3Tw0S2P4</td><td>0.70</td><td>PmiAlUnis-Tc4Tw3S2P2</td><td>0.38</td><td>PmiAl-Tc4Tw2S1P1</td><td>0.85</td></tr><tr><td>PmiAl-Tc3Tw0S2P0</td><td>0.70</td><td>PmiAlUnis-Tc4Tw3S2P1</td><td>0.38</td><td>PmiAlUnisAm-Tc0Tw3S2P1</td><td>0.85</td></tr><tr><td>PmiAlUnis-Tc3Tw0S0P4</td><td>0.70</td><td>PmiAlUnis-Tc4Tw2S1P1</td><td>0.38</td><td>PmiAlUnisAm-Tc0Tw2S2P2</td><td>0.85</td></tr><tr><td>PmiAlUnis-Tc3Tw0S0P0</td><td>0.70</td><td>PmiAlUnisAm-Tc4Tw3S2P5</td><td>0.38</td><td>PmiAl-Tc4Tw3S0P2</td><td>0.85</td></tr><tr><td>PmiAl-Tc4Tw0S2P5</td><td>0.70</td><td>PmiAlUnis-Tc4Tw2S1P2</td><td>0.38</td><td>PmiAlUnisAm-Tc0Tw2S2P1</td><td>0.85</td></tr><tr><td rowspan="5">FeatTransf</td><td>Weight AftNorm Lb</td><td>0.67</td><td>Weight AftNorm Lb</td><td>0.34</td><td>Weight AftNorm Sqrt</td><td>0.85</td></tr><tr><td>Freq Sq</td><td>0.67</td><td>Weight AftNorm Sigm</td><td>0.34</td><td>Weight BefNorm Sqrt</td><td>0.85</td></tr><tr><td>Weight BefNorm Sigm</td><td>0.67</td><td>NoTransf</td><td>0.34</td><td>Weight AftNorm Sigm</td><td>0.80</td></tr><tr><td>Weight AftNorm Sigm</td><td>0.67</td><td>Weight BefNorm Lb</td><td>0.34</td><td>NoTransf</td><td>0.80</td></tr><tr><td>NoTransf</td><td>0.67</td><td>Weight BefNorm Sigm</td><td>0.34</td><td>Weight AftNorm Lb</td><td>0.80</td></tr><tr><td rowspan="5">DimRed</td><td>SVD 200</td><td>0.70</td><td>SVD 100</td><td>0.37</td><td>IslamInkpen 0.025</td><td>0.80</td></tr><tr><td>SVD 100</td><td>0.70</td><td>SVD 200</td><td>0.36</td><td>IslamInkpen 0.25</td><td>0.80</td></tr><tr><td>SVD 300</td><td>0.69</td><td>SVD 500</td><td>0.35</td><td>SVD 200</td><td>0.80</td></tr><tr><td>SVD 500</td><td>0.68</td><td>SVD 300</td><td>0.34</td><td>IslamInkpen 0.005</td><td>0.78</td></tr><tr><td>IslamInkpen 0.05</td><td>0.67</td><td>IslamInkpen 0.01</td><td>0.34</td><td>IslamInkpen 0.01</td><td>0.78</td></tr><tr><td rowspan="5">Smooth</td><td>NoSmooth</td><td>0.67</td><td>Freq KNS</td><td>0.34</td><td>Freq KNS</td><td>0.83</td></tr><tr><td>Weight KNS</td><td>0.65</td><td>Freq MDKNSPOMD</td><td>0.34</td><td>Freq MDKNSPOMD</td><td>0.80</td></tr><tr><td>Freq KNS</td><td>0.62</td><td>NoSmooth</td><td>0.34</td><td>NoSmooth</td><td>0.78</td></tr><tr><td>Freq MDKNSPOMD</td><td>0.62</td><td>Freq MKNS</td><td>0.33</td><td>Weight KNS</td><td>0.75</td></tr><tr><td>Freq MKNS</td><td>0.57</td><td>Weight KNS</td><td>0.31</td><td>Freq MKNS</td><td>0.73</td></tr><tr><td rowspan="3">VNorm</td><td>L<sub>2</sub></td><td>0.67</td><td>L<sub>2</sub></td><td>0.34</td><td>L<sub>2</sub></td><td>0.80</td></tr><tr><td>L<sub>1</sub></td><td>0.67</td><td>L<sub>1</sub></td><td>0.34</td><td>NN</td><td>0.78</td></tr><tr><td>NN</td><td>0.67</td><td>NN</td><td>0.34</td><td>L<sub>1</sub></td><td>0.75</td></tr><tr><td rowspan="2">StopW</td><td>false</td><td>0.67</td><td>true</td><td>0.34</td><td>false</td><td>0.80</td></tr><tr><td>true</td><td>0.67</td><td>false</td><td>0.34</td><td>true</td><td>0.75</td></tr><tr><td rowspan="5">MWFFreq</td><td>NoLimit</td><td>0.67</td><td>NoLimit</td><td>0.34</td><td>NoLimit</td><td>0.80</td></tr><tr><td>2</td><td>0.60</td><td>2</td><td>0.30</td><td>3</td><td>0.68</td></tr><tr><td>3</td><td>0.57</td><td>3</td><td>0.29</td><td>2</td><td>0.63</td></tr><tr><td>5</td><td>0.54</td><td>7</td><td>0.28</td><td>5</td><td>0.58</td></tr><tr><td>7</td><td>0.49</td><td>5</td><td>0.27</td><td>7</td><td>0.55</td></tr><tr><td rowspan="5">MWFWeight</td><td>Zero 0.05</td><td>0.68</td><td>Zero -0.2</td><td>0.34</td><td>Zero 0</td><td>0.80</td></tr><tr><td>Zero 0.1</td><td>0.68</td><td>Limit -0.1</td><td>0.34</td><td>Limit -0.02</td><td>0.80</td></tr><tr><td>Zero -0.05</td><td>0.68</td><td>Limit -0.2</td><td>0.34</td><td>Zero -0.05</td><td>0.80</td></tr><tr><td>Limit -0.01</td><td>0.67</td><td>Limit -0.5</td><td>0.34</td><td>Limit -0.01</td><td>0.80</td></tr><tr><td>Zero 0.02</td><td>0.67</td><td>NoLimit</td><td>0.34</td><td>Zero -0.01</td><td>0.80</td></tr><tr><td rowspan="5">MFFreq</td><td>NoLimit</td><td>0.67</td><td>100</td><td>0.36</td><td>2</td><td>0.80</td></tr><tr><td>2</td><td>0.67</td><td>50</td><td>0.36</td><td>NoLimit</td><td>0.80</td></tr><tr><td>3</td><td>0.67</td><td>30</td><td>0.35</td><td>3</td><td>0.80</td></tr><tr><td>5</td><td>0.67</td><td>20</td><td>0.35</td><td>20</td><td>0.80</td></tr><tr><td>7</td><td>0.67</td><td>15</td><td>0.35</td><td>15</td><td>0.80</td></tr></tbody></table>

## 5.3 Results on the test datasets

The best CPS for English was tested on the test part of the MEN dataset (MT), and the best CPSs for Spanish and Hungarian were tested on the respective version of the Rubenstein-Goodenough dataset (RG) to give us the final results. The best CPS of each language was also evaluated on the datasets of the other languages, to provide us a way of comparison. The results of these test can be found in Table 4.

## 6 Evaluation and discussion

In this section we evaluate our results presented in the previous sections. Please note that the scores are not fully comparable across languages, even when considering the
<table><caption>**Table 3.** The top 5 performing CPSs for each language with their achieved scores, in descending order of maximum H scores</caption><thead><tr><th>Lang</th><th>#</th><th>Parameter settings</th><th>P</th><th>S</th><th>H</th></tr></thead><tbody><tr><td rowspan="10">En</td><td rowspan="2">1</td><td>VecSim<br>PearsMbAdjCosMod-3.Lb</td><td>Weight<br>PmiAl-Tc3Tw0S2P0</td><td>FeatTransf<br>NoTransf</td><td rowspan="2">0.72</td></tr><tr><td>DimRed<br>IslamInkpen 0.05</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>Zero 0</td><td>MFFreq<br>NoLimit</td></tr><tr><td rowspan="2">2</td><td>VecSim<br>PearsMbAdjCosMod-3.Lb</td><td>Weight<br>PmiAl-Tc3Tw0S2P0</td><td>FeatTransf<br>NoTransf</td><td rowspan="2">0.72</td></tr><tr><td>DimRed<br>NoDimRed</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>Zero 0</td><td>MFFreq<br>NoLimit</td></tr><tr><td rowspan="2">3</td><td>VecSim<br>PearsMbAdjCosMod-3.Lb</td><td>Weight<br>PmiAl-Tc3Tw0S2P0</td><td>FeatTransf<br>NoTransf</td><td rowspan="2">0.72</td></tr><tr><td>DimRed<br>NoDimRed</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>Zero -0.05</td><td>MFFreq<br>NoLimit</td></tr><tr><td rowspan="2">4</td><td>VecSim<br>PearsMbAdjCosMod-3.Lb</td><td>Weight<br>PmiAl-Tc3Tw0S2P0</td><td>FeatTransf<br>NoTransf</td><td rowspan="2">0.72</td></tr><tr><td>DimRed<br>IslamInkpen 0.05</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>Zero -0.05</td><td>MFFreq<br>NoLimit</td></tr><tr><td rowspan="2">5</td><td>VecSim<br>PearsMbAdjCosMod-4.Lb</td><td>Weight<br>PmiAl-Tc3Tw0S2P0</td><td>FeatTransf<br>NoTransf</td><td rowspan="2">0.72</td></tr><tr><td>DimRed<br>NoDimRed</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>Zero -0.05</td><td>MFFreq<br>NoLimit</td></tr><tr><td rowspan="10">Es</td><td rowspan="2">1</td><td>VecSim<br>Cos</td><td>Weight<br>Pmi-Tc1Tw3S2P0</td><td>FeatTransf<br>Weight AftNorm Lb</td><td rowspan="2">0.43</td></tr><tr><td>DimRed<br>SVD 100</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>true</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>3</td></tr><tr><td rowspan="2">2</td><td>VecSim<br>Cos</td><td>Weight<br>PmiAl-Tc3Tw3S2P0</td><td>FeatTransf<br>Weight AftNorm Lb</td><td rowspan="2">0.43</td></tr><tr><td>DimRed<br>SVD 100</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>true</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>3</td></tr><tr><td rowspan="2">3</td><td>VecSim<br>Cos</td><td>Weight<br>Pmi-Tc1Tw3S2P0</td><td>FeatTransf<br>Weight AftNorm Lb</td><td rowspan="2">0.43</td></tr><tr><td>DimRed<br>SVD 100</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>true</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>100</td></tr><tr><td rowspan="2">4</td><td>VecSim<br>Cos</td><td>Weight<br>PmiAl-Tc3Tw3S2P0</td><td>FeatTransf<br>Weight AftNorm Lb</td><td rowspan="2">0.43</td></tr><tr><td>DimRed<br>SVD 100</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>3</td></tr><tr><td rowspan="2">5</td><td>VecSim<br>Cos</td><td>Weight<br>Pmi-Tc1Tw3S2P0</td><td>FeatTransf<br>Weight AftNorm Lb</td><td rowspan="2">0.43</td></tr><tr><td>DimRed<br>SVD 100</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>true</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>3</td></tr><tr><td>Lang</td><td>#</td><td>Parameter settings</td><td>A</td><td></td><td></td></tr><tr><td rowspan="10">Hu</td><td rowspan="2">1</td><td>VecSim<br>MbCosAm</td><td>Weight<br>NPmiAlpha-Tc4Tw4S0P4</td><td>FeatTransf<br>Weight AftNorm Sqrt</td><td rowspan="2">0.65</td></tr><tr><td>DimRed<br>SVD 200</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>2</td></tr><tr><td rowspan="2">2</td><td>VecSim<br>MbCosAm</td><td>Weight<br>NPmiAlpha-Tc4Tw4S0P4</td><td>FeatTransf<br>Weight AftNorm Sqrt</td><td rowspan="2">0.65</td></tr><tr><td>DimRed<br>SVD 200</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>2</td></tr><tr><td rowspan="2">3</td><td>VecSim<br>Cos</td><td>Weight<br>NPmiAlpha-Tc4Tw4S0P4</td><td>FeatTransf<br>Weight AftNorm Sqrt</td><td rowspan="2">0.65</td></tr><tr><td>DimRed<br>SVD 200</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>2</td></tr><tr><td rowspan="2">4</td><td>VecSim<br>Cos</td><td>Weight<br>NPmiAlpha-Tc4Tw4S0P4</td><td>FeatTransf<br>Weight AftNorm Sqrt</td><td rowspan="2">0.65</td></tr><tr><td>DimRed<br>SVD 200</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>1</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>2</td></tr><tr><td rowspan="2">5</td><td>VecSim<br>PearsMbMod-1.Lb</td><td>Weight<br>NPmiAlpha-Tc4Tw4S0P4</td><td>FeatTransf<br>Weight AftNorm Sqrt</td><td rowspan="2">0.63</td></tr><tr><td>DimRed<br>SVD 200</td><td>Smooth<br>NoSmooth</td><td>VNorm<br>L<sub>2</sub></td><td>StopW<br>false</td><td>MWFFreq<br>NoLimit</td><td>MWFWeight<br>NoLimit</td><td>MFFreq<br>2</td></tr></tbody></table>
<table><caption>**Table 4.** Results on the test datasets, in descending order of maximum H scores</caption><thead><tr><th>Lang</th><th>Test set</th><th>CPS</th><th>P</th><th>S</th><th>H</th></tr></thead><tbody><tr><td rowspan="3">En</td><td rowspan="3">MT</td><td>BestEn</td><td>0.71</td><td>0.71</td><td>0.71</td></tr><tr><td>BestHu</td><td>0.67</td><td>0.68</td><td>0.67</td></tr><tr><td>BestEs</td><td>0.63</td><td>0.63</td><td>0.63</td></tr><tr><td rowspan="3">Es</td><td rowspan="3">RG</td><td>BestHu</td><td>0.83</td><td>0.83</td><td>0.83</td></tr><tr><td>BestEn</td><td>0.82</td><td>0.80</td><td>0.81</td></tr><tr><td>BestEs</td><td>0.80</td><td>0.79</td><td>0.80</td></tr><tr><td rowspan="3">Hu</td><td rowspan="3">RG</td><td>BestEn</td><td>0.73</td><td>0.72</td><td>0.72</td></tr><tr><td>BestEs</td><td>0.65</td><td>0.61</td><td>0.63</td></tr><tr><td>BestHu</td><td>0.58</td><td>0.68</td><td>0.62</td></tr></tbody></table>

same datasets on different languages, as except for the Moldovan dataset all of the used Spanish and Hungarian datasets were constructed by translating the English versions, and thus the results on them can be distorted and less reliable than on their English counterparts. Furthermore, the Spanish and Hungarian datasets, especially the latter ones, are rather small, which also makes them less reliable than the English ones.

As there are many differences in the syntax and morphology of the different languages, we anticipated from the beginning that there will be at least some small differences in our findings for the different languages. However, our intuition was that our findings for the different languages will be subtle, and we will be able to find good and rather language-independent CPSs. As English and Spanish belong to the family of Indo-European languages, while Hungarian does not, we expected that the results for English and Spanish will be similar due to this. Further, as both Spanish and Hungarian have very rich morphology, we expected that there will also be a higher similarity between our results for Spanish and Hungarian because of this. We anticipated that the least similarities will be between English and Hungarian, as these languages are the least similar to each other.

In the first phase of our analysis we could observe that some of the parameters worked exactly the same way or very similarly across languages. These parameters were the weighting scheme, feature transformation, vector normalization and minimum limits on word-feature frequencies. These finding are in line with our initial intuitions. Dimensionality reduction seemed to be similar for English and Spanish, while a bit different for Hungarian. Smoothing seemed to perform similarly for Spanish and Hungarian, while differently for English. Minimum limits on word-feature weights seem to behave a bit differently for all three languages. However, it was interesting to see that the results for vector similarity measures, stop-word filtering and minimum limits on feature frequencies were rather similar for English and Hungarian, but different for Spanish, which is contrary to what we anticipated.

In the second phase, although there were similarities in the found best CPSs across the different languages, one could also observe many differences. Here too, the weighting schemes, feature transformation and minimum limits on word-feature frequencies were mostly similar. Compared to the first phase vector similarity, smoothing and min-
imum limits on feature frequencies were also alike for all languages. The other parameters showed a different behaviour for at least one language compared to the others.

We have to note here that there were actually two distinct CPSs with the same best score for English, and they were only different in their DimRed parameter setting. We have chosen the one with the "IslamInkpen 0.05" setting as BestEn, as that setting achieved better performance in the first phase than the "NoDimRed" setting in the other CPS. Furthermore, for Hungarian there were even more CPSs with the same best score. We have used a similar approach in selecting the BestHu version, as we have done in case of the BestEn version. However, as these different CPSs with the same best results have different settings in case of some parameters, one has to be careful drawing conclusions from the best CPSs of the different languages, and thus any conclusions drawn from them should be taken with some reservations.

The final conclusions for the parameters are the following:

- VecSim: for all languages measures based on cosine similarity achieve the best results

- Weight: measures based on PMI dominate the top of the table by far in case of all languages

- FeatTransf: no transformation and transforming the word-feature weights after normalization preforms best for all languages

- DimRed: dimensionality reduction seems to help in most situations: while in case of English the IslamInkpen version performed the best alongside no dimensionality reduction, for Spanish and Hungarian SVD is superior to these options

- Smooth: the no smoothing option clearly outperforms all others for all languages

- VNorm: for English the L₁ option clearly seems to be the best, while for Spanish and Hungarian the best CPSs use either L₁ or L₂ normalization, and most CPSs achieve the same or very similar results with either

- StopW: stop-word filtering seems to improve the results to some extent in case of Spanish, while it does not in case of English and Hungarian

- MWFFreq: no limit is by far superior to the other options for all languages

- MWFWeight: no limit seems to be the best option in case of Spanish and Hungarian, while the Zero option with different parameters seems to excel in case of English

- MFFreq: a low limit or no limit seems to be best in case of all languages (as noted before, in case of SVD for Spanish we had to use a limit of 3 instead of no limit for computational reasons)

As we anticipated, there were parameters where the results for Spanish and Hungarian were similar, but different for English. However, it was interesting that we did not find any parameters that were alike for English and Spanish, but different for Hungarian. Further, to our surprise we found such a parameter, where the results were similar for English and Hungarian, but different for Spanish. These latter findings were in contrast to our initial intuition.

Although all Spanish scores in the second phase are much lower than the English and Hungarian ones, these are almost completely due to the dataset used, and do not mean that the found Spanish CPSs are worse than their English and Hungarian counterparts, as it was noted in the beginning of this section and can be seen from our results on
<table><caption>**Table 5.** Comparison of our best English CPS with a conventional and a state-of-the-art CPS, using the information extracted from the BNC with the method of [29] and the MT dataset for all tests</caption><thead><tr><th>CPS</th><th>P</th><th>S</th><th>H</th></tr></thead><tbody><tr><td>BestEn</td><td>0.67</td><td>0.67</td><td>0.67</td></tr><tr><td>OSC of [29]</td><td>0.59</td><td>0.58</td><td>0.58</td></tr><tr><td>CosPPmi</td><td>0.56</td><td>0.58</td><td>0.57</td></tr></tbody></table>

the test datasets (see Table 4) too. It simply suggests that the dataset used for Spanish in this phase is considerably tougher than the ones used for English and Hungarian.

It was interesting to see that in the cross-language experiments on the test datasets the order of the Best CPSs of the different languages with respect to their performance is different in case of the datasets of the three languages. The best English CPS was always superior to its Spanish counterpart, but it has no absolute superiority over the best Hungarian CPS. Further, there is also no clear ranking between the best Spanish and Hungarian CPSs. It was also interesting to see that in case of the Spanish dataset, although the best Spanish CPS achieved rather good results, actually it achieved the lowest score out of the three best CPSs tested. It was the same for the best Hungarian CPS on the Hungarian dataset too.

All in all, there seems to be no clear ranking between the best CPSs of the different languages, and all of them achieved good results on the datasets of all languages. So, although we got different best CPSs for the different languages, all of them seem to be rather language-independent. These findings give us a strong intuition that our heuristic approach was good, and that our found best CPSs for all languages and their results are robust and reliable.

To further prove that our heuristic approach was successful, that our results are robust and reliable, and that our found best CPSs perform much better than conventional CPSs, we compared our best English CPS with the conventional cosine with positive pointwise mutual information setting (CosPPmi) and the state-of-the-art original settings combination (OSC) of [29], using the information extracted from the BNC with the method of [29] and the MT dataset for all tests. The results of these tests, presented in Table 5, clearly show that our found best CPS is robust, and is not just superior to conventional settings, but to a current state-of-the-art CPS too.

The fact that the best CPSs found in the second phase are not simply made up of the best parameter settings in the first phase proves that our intuition was correct, and the parameters of DSMs need to be tested simultaneously, rather than separately.

# 7 Conclusions

Within this article we have presented a systematic analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for English, Spanish and Hungarian, including novel parameters and novel parameter settings. To
our best knowledge, we are the first to do such a detailed analysis for these parameters,
and also to do such an extensive comparison of them across multiple languages.

With our heuristic approach we were searching for the best combination of parame-
ter settings for all three languages. In accordance with our intuition, there were several
parameters that worked very similarly in case of all three languages. We also found such
parameters that were alike for Spanish and Hungarian, and different for English, which
we also anticipated. However, it was interesting to see that there was such a parameter
that worked similarly for English and Hungarian, but not for Spanish, and we did not
find any parameters that worked similarly for the two Indo-European languages, but
differently for Hungarian.

Although we have found that the very best results are produced by different set-
tings combinations for the different languages, our cross-language tests showed that all
of them work rather well for all languages. Based on this we think that our heuristic
approach was successful, and we could find such combinations of parameter settings
that are rather language-independent, and give robust and reliable results. Further, our
best English CPS, incorporating multiple novel parameter settings, significantly outper-
formed both conventional and state-of-the-art parameter combinations.

Although our results seem rather robust and reliable for Spanish and Hungarian
too, it would be interesting to redo our analysis on larger and more reliable Spanish
and Hungarian datasets to check whether we could find even better CPSs for these
languages, when such datasets will become available in the future.

References

1. Baroni, M., Dinu, G., Kruszewski, G.: Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In: 52nd ACL. pp. 238–247 (2014)

2. Bruni, E., Tran, N.K., Baroni, M.: Multimodal distributional semantics. Journal of Artificial Intelligence Research **48**, 1–47 (2013)

3. Budanitsky, A., Hirst, G.: Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. In: Workshop on WordNet and other lexical resources. p. 2 (2001)

4. Camacho-Collados, J., Pilehvar, M.T., Collier, N., Navigli, R.: Semeval-2017 task 2: Multi-lingual and cross-lingual semantic word similarity. In: SemEval-2017. pp. 15–26 (2017)

5. Camacho-Collados, J., Pilehvar, M.T., Navigli, R.: A Framework for the Construction of Monolingual and Cross-lingual Word Similarity Datasets. In: 53rd ACL. pp. 1–7 (2015)

6. Christopoulou, F., Briakou, E., Iosif, E., Potamianos, A.: Mixture of Topic-Based Distribu-
tional Semantic and Affective Models. In: 12th IEEE ICSC. pp. 203–210 (2018)

7. Curran, J.R.: From Distributional to Semantic Similarity p. 177 (2004)

8. De Deyne, S., Perfors, A., Navarro, D.J.: Predicting human similarity judgments with distri-
butional models: The value of word associations. In: 26th IJCAI. pp. 4806–4810 (2017)

9. Deza, M.M., Deza, E.: Encyclopedia of distances. Springer-Verlag Berlin Heidelberg (2016)

10. Dobó, A.: Multi-D Kneser-Ney Smoothing Preserving the Original Marginal Distributions. Research in Computing Science **147**(6) (2018)

11. Dobó, A., Csirik, J.: Magyar és angol szavak szemantikai hasonlóságának automatikus kiszámítása. In: IX. Magyar Számítógépes Nyelvészeti Konferencia. pp. 213–224 (2012)

12. Dobó, A., Csirik, J.: Computing semantic similarity using large static corpora. In: SOFSEM 2013: Theory and Practice of Computer Science. LNCS 7741. pp. 491–502 (2013)
13. Dobó, A., Csirik, J.: A comprehensive study of the parameters in the creation and comparison of feature vectors in distributional semantic models. Journal of Quantitative Linguistics (2019)

14. Dobó, A., Pulman, S.G.: Interpreting noun compounds using paraphrases. Procesamiento del Lenguaje Natural **46**, 59–66 (2011)

15. Hassan, S., Mihalcea, R.: Cross-lingual semantic relatedness using encyclopedic knowledge. In: 2009 CoNLL. pp. 1192–1201 (2009)

16. Hliaoutakis, A., Varelas, G., Voutsakis, E., Petrakis, E.G.M., Milios, E.: Information retrieval by semantic similarity. International journal on semantic Web and information systems **2**(3), 55–73 (2006)

17. Iosif, E., Georgiladakis, S., Potamianos, A.: Cognitively Motivated Distributional Representations of Meaning. In: 10th Language Resources and Evaluation Conference (2016)

18. Islam, A., Inkpen, D.: Semantic text similarity using corpus-based word similarity and string similarity. ACM Transactions on Knowledge Discovery from Data **2**(2), 1–25 (2008)

19. Kiela, D., Clark, S.: A systematic study of semantic vector space model parameters. In: 2nd CVSC at EACL. pp. 21–30 (2014)

20. Lapesa, G., Evert, S.: A Large Scale Evaluation of Distributional Semantic Models: Parameters, Interactions and Model Selection. Transactions of the Association for Computational Linguistics **2**, 531–545 (2014)

21. Levy, O., Goldberg, Y., Dagan, I.: Improving Distributional Similarity with Lessons Learned from Word Embeddings. Transactions of the ACL **3**, 211–225 (2015)

22. Lin, D.: Automatic retrieval and clustering of similar words. 36th ACL pp. 768–774 (1998)

23. Moldovan, C.D., Ferré, P., Demestre, J., Sánchez-Casas, R.: Semantic similarity: normative ratings for 185 Spanish noun triplets. Behavior research methods **47**(3), 788–799 (2015)

24. Novák, A., Novák, B.: Magyar szóbeágyazási modellek kézi kiértékelése. In: XIV. Magyar Számítógépes Nyelvészeti Konferencia. pp. 66–77 (2018)

25. Pantel, P., Lin, D.: Discovering word senses from text. 8th ACM SIGKDD **41**, 613 (2002)

26. Pecina, P.: Lexical association measures and collocation extraction. Language Resources and Evaluation **44**(1-2), 137–158 (2010)

27. Pennington, J., Socher, R., Manning, C.D.: GloVe: Global Vectors for Word Representation. In: EMNLP 2014. pp. 1532–1543 (2014)

28. Reese, S., Boleda, G., Cuadros, M., Padró, L., Rigau, G.: Wikicorpus: A Word-Sense Disambiguated Multilingual Wikipedia Corpus. In: 7th LREC. pp. 1418–1421 (2010)

29. Salle, A., Idiart, M., Villavicencio, A.: Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations. In: 54th ACL. p. 419 (2016)

30. Salle, A., Idiart, M., Villavicencio, A.: LexVec. URL: https://github.com/alexandres/lexvec/blob/master/README.md (2018), https://github.com/alexandres/lexvec/blob/master/README.md, [Accessed 04.07.2018]

31. Shalaby, W., Zadrozny, W.: Measuring Semantic Relatedness using Mined Semantic Analysis. arXiv preprint arXiv:1512.03465 (2016)

32. Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In: 31st AAAI. pp. 4444–4451 (2017)

33. Turney, P.D., Pantel, P.: From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research **37**, 141–188 (2010)

34. Vakulenko, M.: Calculation of Semantic Distances Between Words: From Synonymy to Antonymy. Journal of Quantitative Linguistics pp. 1–13 (2018)

35. Yih, W.t., Arbor, A.: Measuring Word Relatedness Using Heterogeneous Vector Space Models University of Michigan. 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2010), 616–620 (2012)