# Solving morphological analogies: from retrieval to generation

Esteban Marquer, Miguel Couceiro

► To cite this version:

Esteban Marquer, Miguel Couceiro. Solving morphological analogies: from retrieval to generation. Annals of Mathematics and Artificial Intelligence, 2025, 93 (2), pp.263-298. 10.1007/s10472-024-09945-7. hal-04056908

HAL Id: hal-04056908

https://hal.science/hal-04056908v1

Submitted on 31 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Solving morphological analogies: from retrieval to generation

Esteban Marquer¹* and Miguel Couceiro¹*

¹Université de Lorraine, CNRS, LORIA, Nancy, F-54000, France.

*Corresponding author(s). E-mail(s): esteban.marquer@loria.fr; miguel.couceiro@loria.fr;

## Abstract

Analogical inference is a remarkable capability of human reasoning, and has been used to solve hard reasoning tasks. Analogy based reasoning (AR) has gained increasing interest from the artificial intelligence community and has shown its potential in multiple machine learning tasks such as classification, decision making and recommendation with competitive results. We propose a deep learning (DL) framework to address and tackle two key tasks in AR: analogy detection and solving. The framework is thoroughly tested on the Siganalogies dataset of morphological analogical proportions (APs) between words, and shown to outperform symbolic approaches in many languages. Previous work have explored the behavior of the Analogy Neural Network for classification (ANNc) on analogy detection and of the Analogy Neural Network for retrieval (ANNr) on analogy solving by retrieval, as well as the potential of an autoencoder (AE) for analogy solving by generating the solution word. In this article we summarize these findings and we extend them by combining ANNr and the AE embedding model, and checking the performance of ANNc as an retrieval method. The combination of ANNr and AE outperforms the other approaches in almost all cases, and ANNc as a retrieval method achieves competitive or better performance than 3CosMul. We conclude with general guidelines on using our framework to tackle APs with DL.

**Keywords:** Morphological analogy, Analogical proportions, Analogy solving, Retrieval, Word generation
# 1 Introduction

Abstraction and analogical inference are remarkable capability of human reasoning [1, 2]. To some extent, analogical inference can be thought of as transferring knowledge from a source domain to a different, but somewhat similar, target domain by leveraging simultaneously on similarities and dissimilarities. Analogical inference has been used to solve hard reasoning tasks, and analogy based reasoning has gained increasing interest from the artificial intelligence community and has shown its potential with competitive results in multiple machine learning tasks such as classification, decision making and recommendation [3–6]. Furthermore, analogical inference can support data augmentation through analogical extension and extrapolation for model learning, especially in environments with few labeled examples [7]. Also, it has been successfully applied to several classical natural language processing (NLP) tasks such as machine translation [8], several semantic [9–11] and morphological tasks [12–18], (visual) question answering [19], solving puzzles and scholastic aptitude tests [20], and target sense verification (TSV) [21] which tackles disambiguation.

Many analogy based reasoning approaches are built upon the notion of *analogical proportions* (APs), statements of the form “A is to B as C is to D” denoted *A : B :: C : D*. Significant effort has been done in the past three decades to formally define and describe the properties of APs, and we cover key approaches related to our work in Section 2. In particular, two tasks are usually considered for APs: analogy detection and analogy solving. *Analogy detection* refers to deciding whether a quadruple *A*, *B*, *C*, *D* forms a valid analogy *A : B :: C : D*, and *analogy solving* is the task of finding possible values *x* for which *A : B :: C : x* constitutes a valid analogy. Expressions *A : B :: C : x* for an indeterminate *x* are referred to as *analogical equations* or AP equations. We further detail the two tasks and related approaches to tackle them in Subsections 2.3 and 2.4.

Morphology has been used to develop analogy-based approaches [12, 22–24] using various formal frameworks, further described in Section 2. Indeed, morphological APs can be seen as character-string APs [12, 23], which can be generalized to APs between sequences of symbols and, in turn, to APs between structures. In that way, morphological APs relate strongly to many application settings of analogy. Additionally, morphological APs covers many regular phenomena of morphology and can be used to generate new but plausible words which are easily understandable by humans. This makes morphology an ideal empirical framework to develop and analyze approaches to tackle APs.

Morphological APs are also interesting as a standalone task. Being able to successfully model morphological APs is useful to integrate morphological innovation in automatic systems and to generate linguistic resources, in particular for languages with complex morphology or that rely heavily on morphological innovation. For instance, [24] proposes a toolkit to extract analogical clusters of word morphology, which summarize words with similar morphological behavior by relying only on text data. This kind of result can
serve as a basis for empirical analysis of morphology or as teaching material. In that trend, our previous work [18] details results on the transfer of analogy detection models between languages, and showcases the use of morphological analogy to compare languages empirically.

All these aspects led us to develop a complete framework for AP manip-
ulation using deep learning that we extensively tested on the morphology of
more than 16 languages in the present article as well as other languages in our
previous works [13–18]. This paper serves as a summary for previous work on
analogy detection and analogy solving [13–18], and extends on papers pub-
lished in the ATA@ICCBR2022 and IARML@IJCAI2022 workshops [16, 17].
It also serves as a standalone reference for future used of the analogy detec-
tion and solving models. In particular, we improve the results of the retrieval
approach [17] using word generation from [16]. We compare the performance
of all our models on analogy solving, including new model variants.

The paper is organized into two main parts.

* In Part I we recall the basic background on APs and we introduce the main components of our framework. More precisely:

- We introduce the basic formalism of APs in Subsection 2.1 with a more detailed description of morphological APs in Subsection 2.2. We also present a brief summary of recent approaches in tackling APs with a focus on morphological APs in Subsections 2.3 and 2.4.

- We describe the framework components in Section 3, with the embed-
  ding models, analogy processing models, and training strategy described
  respectively in Sections 4 to 6.

* In Part II, we report experimental results on the morphological tasks consid-
ered in this paper, namely, state of the art results on multilanguage analogy
detection and solving dealing with morphology. Specifically:

- We present new experiments performed on 16 languages of the Siganalogies [25] dataset described in Sections 7 and 8, including languages not studied in our previous approaches [13–18]. The list of all models used is detailed in Section 9, and Section 10 covers the model performance.

- We also provide an overview of the results and conclusions from our previous and current experiments and provide guidelines for future use of the ANNc / ANNr framework (Analogy Neural Network for classification / for retrieval and generation) in Section 11. Key aspects of the framework are discussed in details, such as the benefit of using ANNc and ANNr over other methods, the data augmentation process and its link with the axiomatic setting, the use of the framework in a multilingual context, as well as several limitations of the approaches used.

We conclude the paper with an overview on the main contributions together
with overall analysis of the framework in Section 12.
# Part I

# Analogy modeling and solving

## 2 Related approaches

Analogy has been in a variety of settings and refers to similar but distinct notions. Several efforts have been made to provide a unifying formal framework of analogy using different axiomatic, logical, model theoretic, and functional approaches [23, 26–28]. For instance, [29] separates analogies into: analogical proportions (APs), relational proportions, simile, and metaphor. While there is not general consensus on the different notions of analogies, it is generally accepted that they can be reformulated as APs. This is true for instance for relational proportions from [29] that follow the same writing as APs, while simile and metaphor can be seen as APs where some elements are not explicitly expressed.

In this paper, we focus on morphological APs, i.e., APs $A : B :: C : D$ that capture morphological transformations of the four words A, B, C, D (e.g., conjugation or declension). In Subsection 2.1 we introduce the formal framework we use for our approach, which follows the seminal work of [30] by exploiting the postulates of analogical proportions. We discuss the limitations of the formal framework in Section 11. To give the reader an overview of possible formulations of the problem of analogy, we also provide a brief introduction to other key formal frameworks. In Subsection 2.2 we expand on the notion of morphological transformation and the resulting morphological analogical proportions. In Subsections 2.3 and 2.4 we present key approaches to detecting and solving morphological analogical proportions. As deep learning approaches to morphological analogies are strongly related to approaches on semantic word analogies, the latter will also be discussed here.

### 2.1 Analogical proportions and other formal frameworks of analogy

In this work we focus on the notion of analogical proportions (APs) [27, 31], and in particular we follow the axiomatic setting introduced by Lepage [32] consisting in 4 axioms in the linguistic context for analogical proportions: *symmetry* (if $A : B :: C : D$, then $C : D :: A : B$), *central permutation* (if $A : B :: C : D$, then $A : C :: B : D$), *strong inner reflexivity* (if $A : B :: C : D$, then $D = C$), and *strong reflexivity* (if $A : B :: A : D$, then $D = B$). These postulates imply several other properties, for instance *identity* ($A : A :: B : B$ is always true), *inside pair reversing* (if $A : B :: C : D$, then $B : A :: D : C$) and *extreme permutation* (if $A : B :: C : D$, then $D : B :: C : A$). This axiomatic setting is related to the common view of APs as geometric (Equation (1)) or arithmetic proportions (Equation (2)) or, in
geometric terms, as parallelograms in a vector space (Equation (3)):

$$ \frac{A}{B} = \frac{C}{D}, \qquad (1) $$

$$ A - B = C - D, \qquad (2) $$

$$ \vec{A} - \vec{B} = \vec{C} - \vec{D}. \qquad (3) $$

Most of the works on analogy relying on such a formalization manipulate APs between symbols and strings of symbols (abc : abd :: efg : efh), which can be directly related to morphology (cat : cats :: dog : dogs), but it is also possible to apply the axiomatic setting to other kinds of analogy for instance Boolean data as was done in [33, 34]. While these axioms seem reasonable in the word domain, they can be criticized in other application domains [28].

In [18], we explore some limitations of the axiomatic setting for training models of morphological APs, and in particular how accepting or refusing central permutation impacts the performance of our ANNc models.

## 2.2 Morphological analogical proportions

In this paper, we focus on the study of morphological APs, that is, APs between words in which the underlying relation is a morphological transformation. A morphological transformation is a transformation of the structure of a word that follows a set of morphological rules, defined by the language.

Morphology is usually separated into inflectional morphology and derivational morphology. On the one hand, derivational morphology refers to morphological transformations that allow to create new words in a systematic manner, for instance in English the prefix un- allows to create unaware from aware in the same manner as untold from told. On the other hand, inflectional morphology describes morphological transformations that express a change in the grammatical nature of a word without altering the core meaning of the word. For example, looked is the result of a change of tense in the English verb to look by adding the suffix -ed. The data we use in our approach is extracted from inflectional morphology datasets as described in Section 7, so our experiments are limited to APs on inflectional morphology. Given that the morphological mechanisms of derivational and inflectional morphology are very close, this limitation in the scope of our work could easily be solved by using data containing derivational morphology, without modifying our approach. Additionally, examples presented in this paper contain both derivational and inflexional morphological transformations, to better illustrate the mechanisms of morphology.

## 2.3 Analogy detection

Analogy detection can be seen as a classification task: given a quadruple A, B, C, D, we classify the AP A : B :: C : D as valid or invalid according to our notion of AP. It can be applied to extract analogies from data, for instance
the tools in [24] are designed to generate analogical grids, i.e., matrices of transformations of various words, similar to paradigm tables in linguistics [9]. Each row of an analogical grid contain words using the same root, and going from one column to another corresponds to a single morphological transformation no matter the row. By taking two rows and two columns of an analogical grid, we obtain an AP. This approach [24] detects morphological analogies using manually designed features such as the number of character occurrences and the length of the longest common subword.

Analogy detection on a single quadruple is less trivial than it may appear, as in many cases the boundary between valid and invalid analogies is not clearly defined. A solution is to use machine learning to learn the boundary from data, as Lim et al. implemented [10] for semantic word analogies and further explored in [11]. Using a dataset of semantic APs, they learn an artificial neural network to classify quadruples *A*, *B*, *C*, *D* into valid or invalid analogies, using pretrained word embeddings *e*<sub>A</sub>, *e*<sub>B</sub>, *e*<sub>C</sub>, and *e*<sub>D</sub>. We follow a similar approach to morphological analogies in [13] by replacing the GloVe [35] semantic embeddings used in [10, 11] with a morphology-oriented word embedding model. The classifier is detailed in Subsection 5.1 and Figure 4 under the name Analogy Neural Network for classification (ANNc).

## 2.4 Analogy solving by retrieval and by generation

Analogy solving is the process of completing an incomplete analogy, in particular AP equations (or analogical equations) like *A* : *B* :: *C* : *x* where *x* is unknown. Analogy solving strongly relates to the concept of analogical innovation, an important mechanism of creativity [2, 6]. It can also be seen as a transfer operation, where *A* is a source situation with a solution *B*, and *C* is a target situation for which we want the solution. The notion of analogy is particularly useful in this setting as it leverages simultaneously the similarities and differences between *A*, *B*, *C* to adapt *B* to the target situation *C* and obtain the solution.

The *Alea* algorithm [8] relies on the number of character occurrence in *A*, *B*, *C* to determine the characters occurring in *D* the solution to the analogical, and proposes a Monte-Carlo estimation of the permutation of these characters to solve the AP equation. More specifically, [8] follows the results of [36] about closed form solutions and proposes a Monte-Carlo estimation of the solutions of an analogical equation by sampling among multiple sub-transformations. With *bag*(*A*) being the set of all characters in *A*, *Alea* considers *bag*(*D*) = (*bag*(*B*) − *bag*(*A*)) + *bag*(*C*) and thus *D* is a permutation of the characters of *bag*(*D*). For example, the solution of *cat* : *cats* :: *animal* : *x*, *x* = *animals* is a permutations of the characters {*a*, *a*, *i*, *l*, *m*, *n*, *s*}, which can be obtained from the characters in *cats* but not in *cat* (i.e., {*s*}) together with the characters of *animal* (i.e., {*a*, *a*, *i*, *l*, *m*, *n*}). This process cannot handle mechanisms like reduplication (repeating part of a word) that requires more complex comparisons between character occurrences.
A similar approach was later proposed in [37], using the postulates of [30] to address multiple characteristics of words, such as their length, the occurrence of characters and of patterns. Based on these features, the author proposes an algorithm to solve analogies between character strings by first extracting relevant features, then estimating the position of each character using an arithmetic on character positions, and finally generating the corresponding word. The method accounts for multiple potential solutions by comparing the predicted word with permutations of its characters.

A more empirical approach which does not rely on the axioms of APs was proposed by [12], and considers some transformation *f* such that $B = f(A)$ and $f(C)$ is computable. Following the observation that humans tend to use the simplest thinkable analogy to solve analogical equations [12, 23], the simplest transformation *f* is found by minimizing its Kolmogorov complexity. This is done by generating *f* by combining simple operations (insertion, deletion, etc.) and computing the length of the resulting program. With this process, not only obtain the solution of the AP equation but also the transformation *f* is obtained. Additionally, *Kolmogorov complexity* is able to handle more complex mechanisms than *Alea* including reduplication.

In [10, 11], a retrieval approach was proposed for semantic APs and later adapted to morphological APs in [15, 17]. Similarly to the analogy detection approach proposed by the authors (see Subsection 2.3), the analogy solving approach relies on pre-trained embeddings. An artificial neural network is used to predict the embedding of a solution, and the corresponding word is retrieved from a list of possible words based on its embedding. A similar neural network is detailed in Subsection 5.2 and Figure 5 under the name Analogy Neural Network for retrieval/generation (ANNr).

Recently, [38] proposed a generation framework to solving sentence analogies. They use an autoencoder model (named ConRNN) trained to reconstruct sentences, and perform simple arithmetic operations on the embedding space to solve analogies. Once the analogy between embeddings is solved, the decoder part of ConRNN is used to generate the solution from the predicted embedding. Their model is a sequence-to-sequence model composed of 2 elements. First, a sentence (as a sequence of words) is used as input to an encoder RNN, and the last hidden state of the RNN is used as the sentence embedding. The latter is then fed to a decoder RNN that tries to predict the words of the input sentence. The use of a generative model achieves significantly better results than the nearest neighbor algorithm on the same embedding space. With a similar sequence-to-sequence autoencoder (AE) model as [38], we proposed in [16] an approach to generate words at the character level, and using simple vector arithmetic we achieved significant performance improvement over baselines on solving morphological AP equations. One of the aims of this paper is to further improve the performance by integrating the ANNr approach developed in [10, 11, 17] with the AE model.
# 3 Overview of the framework

Our framework can be split into two groups of models, namely the embedding models and the analogy models, summarized in Figure 1.

To obtain suitable representations of words, we use two different kinds of embedding models described in Section 4: an embedding model designed for morphological applications and inspired from [39] and an autoencoder (AE) model following a traditional sequence to sequence approach.

To detect and solve analogies, we rely on the two neural network models described in Section 5: the *Analogy Neural Network for classification* (ANNc) and *Analogy Neural Network for retrieval/generation* (ANNr), which were initially introduced in [10] and refined along our experiments. The architecture of ANNc and ANNr is based on intuitions driven by the axiomatic setting as described in Subsection 2.1. In addition to these, we experiment with multiple non-parametric approaches such as the *parallelogram rule* and *3CosMul* [40] that we will also describe in Section 5.

We separate retrieval approaches from generation approaches, as the open world assumption is used for the generation model and the closed world assumption for the various retrieval models. The closed world assumption means that we know all the solution candidates, and no solution outside of these is accepted. Conversely, in the open world assumption the space of solution is not known beforehand, and we can obtain solutions beyond what is present in the data.

The data augmentation described in Subsections 6.3 and 6.4 enables training models to fit the formal postulates of AP (see Subsection 2.1) by becoming invariant to them.

<figure><img src="image_2.png" /><figcaption>**Fig. 1** Overview of the framework: morphological embedding models, data augmentation, analogy classification (ANNc) and analogy solving (ANNr) models.</figcaption></figure>
# 4 Embedding models

An important part of many machine learning systems is finding features relevant to the task to accomplish and using the features to represent the data. In deep learning, this task is usually handled by a family of approaches called *embeddings*, real-valued vector representations of the data obtained by learning an *embedding model* from data. It is commonly accepted that the quality of such representations, which corresponds to the amount and nature of the information contained in the embedding as well as the properties of the embedding space, is a key factor in the performance of deep learning approaches. Learning an embedding model of high quality often requires high amounts of data and significant training time which can be challenging, so for many applications on text pre-trained large-scale embedding models are made available, for instance, Bert [41]. However, in our early experiments with pretrained word embeddings models (in particular, GloVe [35], word2vec [42], and fasttext [43]) we obtained poor performance<fnref n="1" />, which can be easily understood as these models contain information of a semantic nature<fnref n="2" /> while we need morphological information to deal with morphological APs.

To solve this issue, we used with two different embedding models: a model inspired from [39], using a *convolutional neural network (CNN)* [44], and a model based on the autoencoder (AE) technique [45] and *Long- and Short-Term Memory network (LSTM)* [46].

## 4.1 CNN-based embedding model

The CNN-based model learns to detect key morphological patterns from the characters forming a word. To do so, the input of the model are the characters of a word. First, the characters are embedded into vectors of size *m* learned together with the rest of the word embedding model. As schematized in Figure 2, multiple CNN filters are used, and each filter goes over the character embeddings by spanning over the full embeddings of 2 to 6 characters, resulting in filter sizes between 2 by *m* and 6 by *m*. For each filter, the model computes the maximum output to serve as a component of the word embedding. We use 16 filters of each size between 2 to 6, resulting in embedding of 80 components. This last operation keeps only salient patterns detected by each of the filters, and forces each CNN filter to specialize in identifying a specific pattern of characters. By using character embeddings to encode character features, the model is able to capture patterns based on character such as “-ing” but also based on features of characters like “vowel-vowel-consonant”. This flexibility is useful to deal with phenomena like *euphony* which is, in very simple terms, a change of sound to make a word easier to pronounce (e.g., *far* becomes *further* when adding the suffix *-ther*). These character patterns correspond to *morphemes*, which are the minimal units of morphology. As the

<footnote n="1">¹We reproduced a similar experiment on the English data in Subsection A.2.</footnote>

<footnote n="2">²To be exact, these models relate to distributional semantics, which is based on the co-occurrence of words in text.</footnote>
CNN-based embedding model

<figure><img src="image_3.png" /><figcaption>**Fig. 2** Morphological word embedding model based on character-level CNNs. The special characters <BOW> and <EOW> allow the CNN filters to identify characters at the beginning and end of the word.</figcaption></figure>

main components of this embedding model are CNN filters, we coin it the
CNN-based model.

When used for analogy solving, the CNN-based model does not allow us
to use an arbitrary embedding to generate the solution. Instead, we need to
compute the embeddings of a list of candidate words and select the most
relevant word accordingly, making the space of solution closed. This property
is not an issue when working in a closed world assumption (i.e., we know
all the solution candidates, and no solution outside of these is accepted), but
in an open world assumption, we would theoretically have to compute the
embeddings of all possible character strings, which is not feasible.

## 4.2 AE embedding model

Our AE model stems from the above-mentioned limitation of the CNN-based
model for analogy solving. To tackle this issue, we use an AE model that learns
the embedding of a word from its characters and learns to generate word from
embeddings.

An AE can be seen as a lossy compression algorithm, composed of
an *encoder* and a *decoder*. Taking an example in our setting, the encoder
compresses the information of a word into an embedding and the decoder
decompresses the embedding back into a word. In order to properly decode
the embedding (and to minimize the compression loss in our compression algo-
rithm analogy), the model is trained to encode words and then decode the
resulting embedding back into the original word. The difference between the
Autoencoder embedding model

<figure><img src="image_4.png" /><figcaption>**Fig. 3** Morphological word embedding model based on character-level CNNs. The special characters <BOW> and <EOW> mark the beginning and end of the word, and are used in the generation process.</figcaption></figure>

original and the decoded word is used as the training objective in the autoen-coding task. Because the model is able to recreate the original word from the embedding, the latter contains the key information to represent the former. In more details, the decoder will learn to reproduce systematic or redundant parts of the training data without relying on the embedding, while key information to differentiate the training are encoded in the embedding by the encoder.

The architecture for our model is a character-level sequence-to-sequence autoencoder model, based on the model described in [47]. We use Long- and Short-Term Memory network (LSTM) [46], which are neural networks designed to handle sequences, and which allow us to encode and decode embed-dings of a constant size no matter the length of the word, which we see as a sequence of characters. Indeed, in our setting we need morphological informa-tion, which requires knowing the characters composing a word. Each character of a word is encoded into a one-hot vector<fnref n="3" /> and fed into the encoder, which is a Bidirectional LSTM (BiLSTM)[48], a variant of the LSTM that reads the elements of the input sequence in the forward and backward order simul-taneously. This layer outputs four vectors: the last hidden state $h_f$ and cell state $c_f$ in the forward direction, and similarly $h_b$ and $c_b$ for the backward direction. The concatenation of these vectors $e_w = \text{concat}(h_f, h_b, c_f, c_b)$ is the embedding of the word. The decoder is an LSTM followed by a fully connected layer (or perceptron or dense layer) with softmax activation. The input for the first step of the decoder is the above-mentioned embedding, split into two

<footnote n="3">³A one-hot encoding vector of an element $x_i$ for a set $\{x_1, ..., x_n\}$ is a vector of size $n$ containing 0 for all components except component $i$ which is 1.</footnote>
states $h = \text{concat}(h_f, h_b)$ and $c = \text{concat}(c_f, c_b)$. The output of the decoder is a sequence of character predictions, with each prediction being a probability distribution over all the characters seen in training obtained with the softmax activation.

The process of decoding is usually a step by step process where each character of the output is predicted one after the other, with the previous character used as input to predict the next one. The first character is primed with *beginning of word* (BOW) character that marks the start of the word, and prediction is stopped when we encounter an *end of word* (EOW) character. During training, we accelerate training by using *teacher forcing*, a variant of this decoding process where the characters of word to predict are used in place of the predicted characters as input for the next steps as illustrated in Figure 3.

# 5 Models for analogy processing

To manipulate APs, we propose multiple neural network models, building upon the embedding models. Here we describe the structure and inner workings of these models, as well as how to use them for analogy detection and analogy solving. We also describe three approaches that do not rely on neural networks to solve AP equations, namely, 3CosAdd [42] and 3CosMul [40], and the parallelogram rule.

## 5.1 Analogy detection using ANNc

<figure><img src="image_5.png" /><figcaption>**Fig. 4** Analogy Neural Network for classification (ANNc). The embedding of the four input elements A, B, C, D are shown vertically, each in a different color.</figcaption></figure>

The Analogy Neural Network for classification (ANNc) follows the idea that a quadruple A, B, C, D constitutes a valid analogy A : B :: C : D if A and B differ in the same way as C and D. If this is true for all the features of A, B, C, D, i.e., for all dimensions of the embeddings $e_A, e_B, e_C, e_D$, then the analogy is true.
The model we use is based on multiple CNN layers taking as input $e_A, e_B, e_C, e_D$, each of size $n$, stacked into a $n \times 4$ matrix. Then, a first set $F_1$ of CNN filters of size $1 \times 2$ is applied on the embeddings, such that for each component $e_i$ of the embedding vector, each filter spans across $A_i$ and $B_i$ simultaneously, and across $C_i$ and $D_i$ simultaneously, with no overlap. A second set $F_2$ of CNN filters of size $2 \times 2$ is applied on the resulting $|F_1| \times n \times 2$ tensor. Each filter in the second set of filters moves along the embedding dimension one component at a time, as illustrated in Figure 4, thus resulting in a $|F_2| \times (n-1) \times 1$ tensor. For all CNN filters, the activation function is ReLU. Finally, the output of the second group of filters is fed to a fully connected layer with a single output that we bind between 1 and 0 using a sigmoid activation. This final output serves as a classification score, with 0 for “not an AP” and 1 for valid APs.

Intuitively, the first set of CNN filters extracts the relations $A: B$ and $C: D$ and the second set compares $A: B$ and $C: D$, which can be seen as the predicate “as” in “A is to B as C is to D”. Additionally, while it is possible to assume that embedding dimensions are independent from each other, it is rarely the case in practice. To handle dependent dimensions, the second set of filters have a size of 2 which results in overlaps between adjacent embedding dimensions, and the fully connected layer mixes the results of all filters from $F_2$ over all dimensions.

## 5.2 Analogy solving using ANNr

<figure><img src="image_6.png" /><figcaption>**Fig. 5** Analogy Neural Network for retrieval/generation (ANNr). The embedding of the three input elements $A, B, C$ are shown horizontally, each in a different color – notice that $A$ appears twice.</figcaption></figure>

An approach to solving $A: B :: C: x$ is to find how $e_B$ differs from $e_A$ and to generate an $e_x$ that differs from $e_C$ in the same way. Central permutation (see Subsection 2.1) allows us to apply the same operations on $A: C :: B: x$, to obtain $e_x$ from $e_B$ using the difference between $e_A$ and $e_C$. The Analogy Neural Network for retrieval/generation (ANNr) follows this intuition by using a two step process illustrated in Figure 5. First, two separate fully connected layers $f_1, f_2$ with ReLU activation are applied respectively on the concatenation of
A and B and the concatenation of A and C. Intuitively, these layer determine the relation between $e_A$ and $e_B$ on the one side while keeping the key content of $e_B$, and similarly for $e_A, e_C$ the other side. Then, the concatenation of the outputs of $f_1$ and $f_2$ is fed into a last fully connected layer $f_3$ without activation, which generates $e_x$ the embedding of the predicted $x$.

In preliminary experiments, we observed that using different weights for $f_1, f_2$ achieved better performance than sharing the weights (which would result in a siamese architecture). This indicates a need for asymmetry between the relations between A and B and between A and C, and goes in the direction of relaxing the central permutation axiom.

## 5.3 Analogy solving with the parallelogram rule

As mentioned in Subsection 2.1, the parallelogram is one of the inspirations of the axiomatic setting of APs and has been used to solve APs since early works on word embeddings [42, 49]. To compute the solution of an AP equation $A: B :: C: x$, the embeddings $e_A, e_B$, and $e_C$ are computed by the embedding model and used to compute $e_x = e_B - e_A + e_C$. Then, $e_X$ is used to find the word X solution to the analogy. With the AE embedding model, $x$ is obtained using the decoder and with the CNN-based model retrieval is used instead.

## 5.4 Analogy solving with 3CosAdd or 3CosMul

3CosAdd [42] and 3CosMul [40] are retrieval approaches to solve AP equations within an embedding space and are often used even if they have known limitations. As retrieval approaches, both methods rely on the embeddings of candidate solutions to solve the equation. In [17], we report results of 3CosAdd and 3CosMul applied on the embeddings trained with ANNc to measure the improvement brought by ANNr. 3CosAdd (4) can be seen as the parallelogram rule combined a cosine similarity to recover the closest solution. 3CosMul (5) maximizes a similarity built upon the cosine. Intuitively, the formula the angle between B and D on one side and the angle between C and D on the other side, and normalizes the result by the angle between A and D. In this reading of the formula, 3CosMul shares a similar intuition as ANNr. To avoid a 0 denominator, a small $\epsilon$ is added in (5):

$$3\text{CosAdd} = \underset{e_D}{\operatorname{argmax}} \cos(e_D, e_B - e_A + e_C), \quad (4)$$

$$3\text{CosMul} = \underset{e_D}{\operatorname{argmax}} \frac{\cos(e_D, e_B)\cos(e_D, e_C)}{\cos(e_D, e_A) + \epsilon}. \quad (5)$$

We refer the reader to [40] for a detailed description of 3CosAdd and 3CosMul.

## 5.5 Analogy solving using ANNc

It is also possible to use ANNc to solve analogies, by maximizing the classification score over possible solutions. This method is most adapted to a
retrieval approach, but it is possible to envision generating candidate solutions (for instance, in as was done in Alea) or even to maximize the classification score in a machine learning generation algorithm (for example, using a genetic algorithm).

# 6 Pre-training, training and evaluation

In this section, we first explain in Subsections 6.1 and 6.2 how to pre-train our embedding models, and detail the technical aspects of training our analogy detection and analogy solving models in Subsections 6.3 and 6.4 respectively.

Pre-training is a wide-spread approach for embedding models, which consists of two steps: firstly, the embedding model is pre-trained on general-purpose data or data of the target domain, and secondly, the embedding model is transferred to serve as a component of the final model on the target task. For instance, it is possible to transfer a part of a model and reuse it as a component of a larger model, as is usually done with large pretrained embedding models such as Bert [41], wav2vec2 [50], or vision transformers [51]. Then, it is possible to either train only the new part of the final model, but it is also possible to finetune the embedding model during the training. The pre-training and finetuning approach allows to first bring the embedding model to a globally viable state before specializing it on the task.

Further details on the training hyperparameters are provided in appendix Subsection A.1.

## 6.1 Pre-training the CNN-based embedding model

In [15] we confirmed the benefits of pre-training the CNN-based embedding model before analogy solving with ANNr. The CNN-based embedding is not designed for a standalone pre-training, therefore it is pre-trained on the analogy detection task with an ANNc model, following the ANNc training procedure described below.

## 6.2 Pre-training the autoencoder embedding model

This subsection summarizes the task and criterion for pre-training the AE embedding model, following [16].

### Autoencoding task.

AEs are designed to be trained on an autoencoding task (AE task). The principle of this task is, in our setting, to present a training dataset containing words to the model. The AE is then trained to encode-decode each word of the training dataset, and to minimize the error in the predicted word.

### Training criterion.

We use the Cross-Entropy (CE) loss to train the AE, as it is the common way to train a model where each output is one among a series of class. In the
following formula, we write $\mathcal{L}_{AE}$ the loss used for our AE, which is exactly the CE loss but re-written in our setting:

$$ \mathcal{L}_{AE}(D, decoder(e_x)) = -\frac{1}{|D|} \sum_{i=1}^{|D|} \sum_{c \in V} D_{c,i} \log (decoder(e_x)_{c,i}), \quad (6) $$

where $|D|$ is the number of characters to predict including the EOS character, $V$ is the character vocabulary (the set of characters present in the training data), $D_{c,i}$ is 1 if the $i$-th character of $D$ is $c$ and 0 otherwise, $decoder(e_x)$ is the decoder output and $decoder(e_x)_{c,i}$ is the predicted probability that the $i$-th character of the prediction is $c$.

## 6.3 Training ANNc

Here we summarize the training process of ANNc that achieved the most competitive performance, but further details can be found in [13, 15].

### Data augmentation.

The axioms of APs described in Subsection 2.1 enable the generation multiple valid APs (positive examples) and invalid APs (negative examples) from each AP in our analogy dataset. This process is a data augmentation process that extends the amount of data available for training, and is necessary to obtain the negative examples required for training the classifier. Finally, it allows the models to learn how to fit the formal postulates of AP by becoming invariant to them.

By using the symmetry and central permutation axioms, we are able to generate 7 more valid APs from a valid AP $A: B :: C: D$, namely:

* $A: C :: B: D$,

* $D: B :: C: A$,

* $C: A :: D: B$,

* $C: D :: A: B$,

* $B: A :: D: C$,

* $D: C :: B: A$,

* $B: D :: A: C$.

From each of these valid APs, we can generate invalid APs that are in conflict with at least one axiom, following [10, 11]. Given a valid AP $A: B :: C: D$, we generate the following invalid APs: $B: A :: C: D$, $C: B :: A: D$, and $A: A :: C: D$, for a total of 24 invalid APs. In practice, data augmentation is applied after the embedding model, as shown in Figure 1.

In the experiments reported in [15, Appendix B], we identified that training ANNc with an imbalance between the positive and the negative examples resulted in a corresponding imbalance in the performance on the two classes of examples. Additionally, we identified that some invalid APs we generate have the form of valid APs, for instance when using sang : sang :: was : were (with
the underlying relation of going from the 1st to the 3rd person of singular), the invalid AP generation rule $A : B :: C : D \to A : A :: C : D$ produces sang : sang :: was : were, which we consider valid. To tackle these issues, we use the following data augmentation process:

1. given an AP $A : B :: C : D$ we compute the set of 8 valid and the set of 24 invalid APs mentioned above;

2. we remove APs from the invalid APs that have the same form as any of the valid APs;

3. if we have more than 8 invalid APs remaining, we sample without replacement 8 invalid APs from the filtered set;

4. if we have less than 8 invalid APs remaining, we take the filtered set of invalid APs and add the missing amount of APs by randomly sampling them with replacement from the filtered set of invalid APs; this process allows us to have at least one instance of each invalid AP.

### Training criterion.

To train ANNc we use the Binary Cross-Entropy (BCE) loss which is a standard for binary classification, and is written as follows:

$$ \mathcal{L}_{ANNc}(y, y_{pred}) = -y \log(y_{pred}) - (1 - y) \log(1 - y_{pred}) \\ = \begin{cases} -\log(y_{pred}) & \text{if } y = 1, \\ -\log(1 - y_{pred}) & \text{otherwise,} \end{cases} \qquad (7) $$

where $y$ is the true class (1 or 0) and $y_{pred}$ is the predicted probability for class 1 (i.e., the probability that the AP is valid, the classification score).

## 6.4 Training ANNr

Here we summarize the key elements we use to obtain the best results with ANNr. Further details on the training process can be found in [17].

### Data augmentation.

To train ANNr, we use a similar data augmentation process as for ANNc, but considering only the valid APs. In other words, given an AP equation $A : B :: C : x$ with expected $x = D$, we generate 7 other AP equations:

* $A : C :: B : x$ with expected $x = D$,

* $D : B :: C : x$ with expected $x = A$,

* $C : A :: D : x$ with expected $x = B$,

* $C : D :: A : x$ with expected $x = B$,

* $B : A :: D : x$ with expected $x = C$,

* $D : C :: B : x$ with expected $x = A$,

* $B : D :: A : x$ with expected $x = C$.
**Training criterion.**

When fine-tuning the embedding model while training ANNr, it is possible to observe collapse of the embedding space if the criterion is badly chosen: for instance, when using the mean squared error (MSE) (or L² distance, see Equation (8)), it is possible for the model to move all the embeddings in a smaller area of the embedding space by multiplying them by 10⁵; while this minimizes the MSE, it does not improve retrieval performance as the relative distance between embeddings does not change.

$$MSE(e_D, e_x) = \frac{1}{n} \sum_{i \in [1, n]} (e_{D,i} - e_{x,i})^2, \qquad (8)$$

where $e_{D,i}$ is the i-th component of $e_D$ and $n$ is the number of dimensions of the embeddings.

In [17] we experimented with multiple training objectives designed to avoid this phenomena. In the present work we consider only the most successful one, called $\mathcal{L}_{\text{norm. random}}$ in [17] that we rename $\mathcal{L}_{\text{ANNr}}$ for clarity:

$$\mathcal{L}_{\text{ANNr}}(e_D, e_x) = \frac{1 + MSE(e_D, e_x)}{1 + MSE(\text{batch shuffle}(e_D), e_x)}. \qquad (9)$$

As the model is trained with batches of randomly selected samples, we associate each prediction with an unrelated embedding by permuting $e_D$ along the batch dimension (batch shuffle($e_D$) in Equation (9)). Intuitively, we encourage the prediction $e_x$ of ANNr to be close to the actual $e_D$ and far from other embeddings by normalizing MSE($e_D, e_x$) based on distances between other embeddings.

**Inference.**

When used with the CNN-based embedding model, we do not have direct access to the word corresponding to the embedding predicted by ANNr. We consider all the words appearing in the non-analogical data as candidates for retrieval, which includes words that do not appear in any analogy, and retrieve the closest one based on the cosine distance between the ANNr prediction and the embedding of each candidate word. Previous experiments [15] showed that cosine distance slightly outperforms euclidean distance in that setting.

**Training ANNr when using the CNN-based embedding model.**

As reported in [17], we obtained the best results by first training only ANNr without finetuning the embedding until ANNr converges (i.e., untill there is no improvement in the loss on the development data), then training both models together. We limited the total training time to 50 epochs including both phases.
### Training ANNr when using the AE embedding model.

When training ANNr while finetuning the AE embedding model, we add a generation loss term $\mathcal{L}_{AE}$ to the training loss of the ANNr. In other words, $\mathcal{L}_{AE+ANNr}$ is a convex combination of $\mathcal{L}_{ANNr}$ and $\mathcal{L}_{AE}$. This allows us to maintain good decoding performance and to make use of the differences between the predicted and expected embedding to guide ANNr at the beginning of training. The $\mathcal{L}_{AE}(e_D, e_x)$ used here is the same as the one used to pre-train the AE:

$$ \mathcal{L}_{AE+ANNr}(e_D, e_x) = (1 - \lambda)\mathcal{L}_{ANNr}(e_D, e_x) + \lambda\mathcal{L}_{AE}(D, \text{decoder}(e_x)). \quad (10) $$

The parameter $\lambda = \min(\max(epoch/5, 0.01), 0.99)$ evolves during training, such that at the beginning of the training ($\lambda = 0.99$) ANNr is training almost alone and the AE performance is not altered, and after 5 epochs, only the AE loss is used ($\lambda = 0.01$), following that better performance is obtained by first training only ANNr [17].

In our early experiments, we attempted to use only $\mathcal{L}_{AE}(D, \text{decoder}(e_x))$ as the criterion for analogy solving with AE+ANNr, but achieved poor performance. More specifically, the models got stuck in poor local optima at the beginning of training that resulted in the degradation of the quality of the AE while not reaching satisfying analogy solving performance.
# Part II

## Empirical study

In this part we report and discuss the performance of our framework on analogy solving. We compare multiple variants of our analogy solving approach and two baselines on morphological analogies in 16 languages of the Siganalogies [25] dataset we describe in Section 7. This part focuses on latest results on analogy solving presented in Section 10, but we also report additional results on the pre-training of the AE and CNN embedding models in Subsections A.3 and A.4. For extended studies on the behavior of the ANNc models, we refer the reader to [13–15, 18]. We summarize the key findings of these works in Section 11.

## 7 Multilingual morphological analogies data

The Siganalogies [25] we proposed in previous work is built upon three datasets described below. We briefly survey the original datasets as well as the process used to generate analogies. Siganalogies is a dataset of morphological analogies built upon Sigmorphon 2016 [52], Sigmorphon 2019 [53], and the Japanese Bigger Analogy Test Set (JBATS) [54]. However in our experiments we only consider data from Sigmorphon 2016 and Sigmorphon 2019. A more detailed description of the Siganalogies dataset, including the number of analogies for each language, can be found on the GitHub page of the dataset<fnref n="4" />.

### Data from Sigmorphon 2016 and Sigmorphon 2019

The Sigmorphon shared tasks are a series of shared tasks focusing on tackling morphology-related tasks on multiple languages.

The Sigmorphon 2016 [52] shared task focuses on *morphological inflection*, which consists in applying an inflexional morphological transformation on a word that may already be the result of such a transformation. Taking the example from the authors of the task, going from “ran” to “running” corresponds to a transformation from (some person of) the past to the present participle of “to run”. The dataset covers 10 languages, namely Spanish, German, Finnish, Russian, Turkish, Georgian, Navajo, Arabic, Hungarian, and Maltese, which are “mostly languages with rich inflection” by the own words of the authors of [52]. The data is extracted from Wikitionary and underwent cleaning and other preprocessing steps. Sigmorphon 2016 contains 3 tasks, namely *inflection* (task 1), *reinflection* (task 2) and *unlabeled reinflection* (task 3). In our experiments we focus on the data for the inflection task, which contains triples such as “run, present participle, running”, although the morphological transformation is specified in more details. To do so we use a similar process as [37] by combining triples that share the same morphological transformation, for example if we have “dance, present participle, dancing” we

<footnote n="4">⁴https://github.com/EMarquer/siganalogies/blob/main/siganalogies_description.pdf</footnote>
can combine the two triple to obtain “run:running::dance:dancing”. A similar
process can be applied to the other two tasks.

The Sigmorphon 2019 [53] shared task aims for “universal morphological inflection” and covers close to 100 languages. Once again, 3 tasks are proposed, namely *crosslingual transfer for inflection generation* (task 1) and *morphological analysis and lemmatization in context* (task 2), as well as a task for competing on previous editions of Sigmorphon. Here we focus only on task 1, which was to leverage large amounts of data from a high resource language to perform morphological inflection on a low resource language with little amounts of data. The data contains pairs of high resource with low resource languages. For each language, the data contains pairs of words and the corresponding morphological transformation, as for task 1 of Sigmorphon 2016.

Data from the Japanese Bigger Analogy Test Set

The Japanese Bigger Analogy Test Set (JBATS) [54] is an analogy dataset covering 4 categories of relations: derivational and inflexional morphology as well as lexicographic and encyclopedic semantics. It was initially designed to evaluate the performance of the sub-character and character level embedding models in Japanese.

Both morphological categories contain pairs of words sharing the same
morphological transformation, that can be manipulated in a similar way as
the Sigmorphon data. For each second word of the pair, a kanji form and an
hiragana/katakana form are provided to cover possible alternate writings. In
previous studies, both writing were considered without distinction, but further
experiments in that regard would be interesting in further work.

Siganalogies

Siganalogies [25] covers more than 80 languages, however not all of them are
suited for our study. Our language selection process is done in two steps that
we detail in the following paragraphs: we first exclude languages that do not
have enough training analogies, then we train an AE model on each of the
remaining languages and keep only languages with above 80% accuracy for
the AE task.

We first put aside languages in Sigmorphon 2019 that do not contain
enough analogies for training the models, namely the low resource languages
as well as Basque and Uzbeck, as they contain less than 50000 analogies in the
training set. We also exclude Japanese as it contains kanjis, which require sig-
nificantly different hyperparameters to achieve acceptable performance. Note
that some languages appear in both Sigmorphon 2016 and Sigmorphon 2019,
but are counted as separate languages. Indeed, the pre-processing done in
these two editions of Sigmorphon is not the same, in particular with regards to
alphabets: in Sigmorphon 2016 everything is transformed in the Latin alphabet
while in Sigmorphon 2019 the original alphabet is kept.
At this step, we have 52 languages remaining: 10 languages from Sigmorphon 2016 and the remaining 42 from Sigmorphon 2019, with 6 languages (Arabic, Finnish, German, Russian, Spanish, and Turkish) appearing in both groups. The AE is trained on each of the remaining languages, with 5 different training / test split and 10 random initialization per model, for a total of 2600 models (5 × 10 × 50). The accuracy of the AE is detailed in Table A4 in appendix.

For experiments beyond training the AE on each language, we only consider 10 models trained on the first data split and only languages where the AE achieved an accuracy above 80% on the AE task. The apparent discrepancies between the performance reported in Table A4 and the languages we selected (in particular for Portuguese in Sigmorphon 2019 and Russian in Sigmorphon 2016) come from the differences in performance between the first data split and the other 9 splits, which can be appreciated from the large standard deviation reported.

We end up with 16 languages: Turkish, Hungarian, and Georgian of Sigmorphon 2016, and Adyghe, Arabic, Bashkir, English, French, Hebrew, Portuguese, Sanskrit, Slovak, Slovene, Swahili, Welsh, and Zulu from Sigmorphon 2019.

# 8 Training, development and test data

In our experiments, we use the standard of three dataset splits: the training set containing the examples learned by the model, the development set containing examples not seen during training and used to identify overfitting and interrupting the training when the performance stops increasing, and finally the test set to measure the final performance, with examples that appear in neither the training or development set.

For APs, we follow the training and test split from Sigmorphon 2016 [52], Sigmorphon 2019 [53], and JBATS [54] when possible, and split the training set into our training and development analogies. We exclude duplicates of the form *A : B :: C : D ↔ C : D :: A : B* (corresponding to the axiom of symmetry) but we keep analogies of the form *A : B :: A : B*. We randomly sample 500 development and 5000 test APs and at most 50000 training APs.

For the AE task, we consider all the words appearing in training and test data from Sigmorphon and JBATS, which includes words that do not appear in any AP (either because the AP is not used, or because there is no two pair of with the same morphological transformation containing the aforementioned word). We randomly sample 500 development and 500 test examples, and at most 40000 training examples, with no overlaps between the sets.

# 9 Analogy solving models

Our framework allows us to propose both retrieval-based models and generation-based models, that we compare with two baseline symbolic
approaches: the Alea approach from [8] as implemented in [12], and the *Kolmo*
approach from [12].

We consider 3 the retrieval model variants using the same CNN-based
embedding model pre-trained using ANNc on analogy detection: CNN+ANNc,
CNN+3CosMul, and CNN+ANNr. CNN+ANNc corresponds to re-using both
models without fine-tuning on the retrieval task, as mentioned in Subsec-
tion 5.5. CNN+3CosMul uses the 3CosMul [40] approach on the embeddings
obtained with the CNN-based model, once again without fine-tuning. Finally,
in CNN+ANNr the embedding model is finetuned together with the ANNr, as
described in Subsection 6.4. The CNN+3CosMul and CNN+ANNc retrieval
models serve as a baseline to measure the improvement brought by using ANNr
in a retrieval approach. In [17] we used the 3CosMul [40] and 3CosAdd [42]
approaches as baselines for our CNN+ANNr approach and showed signifi-
cant improvements when using CNN+ANNr. However, it can be argued that
embeddings trained for analogies using ANNc encode relevant features in a
way that is hard to manipulate with 3CosMul and 3CosAdd. We compare
CNN+ANNc with CNN+3CosMul to measure this effect, excluding 3CosAdd
which performed worse than 3CosMul in our previous studies. We also com-
pare CNN+ANNc with CNN+ANNr to ascertain the benefit of training an
ANNr model when the setting is limitted to retrieval.

In [16] we explored the potential of a character-level AE for analogy
solving, using a similar approach as [38] that we call AE+parallel here. Fol-
lowing the same approach, we pre-train an AE on our data without analogies
then compute the embedding of the solution using the parallelogram rule
(Subsection 5.3), and finally we use the decoder of the AE to generate the pre-
dicted solution word. The last model we experiment with is AE+ANNr which
combines the AE embedding model with ANNr to achieve higher performance.

10 Analogy solving performance

To evaluate the performance of all analogy solving models we use the accuracy.
For retrieval models, this corresponds to top-1 retrieval accuracy, additionally
hit rate at *k* for *k* ∈ {1, 3, 5, 10} are listed in appendix Subsection A.5. For
generative models, we use word accuracy *b* taking only the most likely predic-
tion for each character, additionally character accuracy and other metrics are
available in the detailed model outputs.

The retrieval and generative models performance are reported in Table 1
and Table 2 respectively, with Table A5 in appendix comparing the results
of both approaches in a single table. In this work we only consider models
trained on a single language, and we discuss the possibility of a multilingual
model in Section 11

General observations.

A first striking result is that the embedding based approaches outperform
the symbolic baselines in all settings using ANNr and in most settings with
<table><caption>**Table 1** Analogy solving accuracy (rate of fully well predicted words, in %) in the retrieval setting (closed world assumption), compared to the best generation model (AE+ANNr). For models depending on an embedding model, and thus sensitive to random initializations, we report performance as mean ± standard deviation over 10 random initialization in each setting.</caption><thead><tr><th rowspan="2">Language</th><th rowspan="2">CNN+ANNr</th><th colspan="2">Retrieval models</th><th rowspan="2">Best gen. model<br>AE+ANNr</th></tr><tr><th>CNN+3CosMul</th><th>CNN+ANNc</th></tr></thead><tbody><tr><td colspan="5" style="text-align:center; font-style:italic;">Sigmophon 2016</td></tr><tr><td>Georgian</td><td><strong>97.60 ± 0.23</strong></td><td>85.58 ± 7.37</td><td>76.77 ± 9.57</td><td>87.50 ± 2.08</td></tr><tr><td>Hungarian</td><td><strong>89.06 ± 1.71</strong></td><td>74.89 ± 5.27</td><td>72.95 ± 8.02</td><td>90.58 ± 0.63</td></tr><tr><td>Turkish</td><td><strong>84.75 ± 2.04</strong></td><td>52.42 ± 4.33</td><td>44.43 ± 13.95</td><td>79.81 ± 8.58</td></tr><tr><td colspan="5" style="text-align:center; font-style:italic;">Sigmophon 2019</td></tr><tr><td>Adyghe</td><td><strong>93.37 ± 0.97</strong></td><td>58.01 ± 8.66</td><td>80.11 ± 5.10</td><td>98.50 ± 0.25</td></tr><tr><td>Arabic</td><td><strong>72.08 ± 3.49</strong></td><td>21.67 ± 5.44</td><td>40.73 ± 9.93</td><td>83.99 ± 1.28</td></tr><tr><td>Bashkir</td><td>57.63 ± 5.48</td><td>37.76 ± 11.12</td><td><strong>65.38 ± 6.01</strong></td><td>95.15 ± 0.92</td></tr><tr><td>English</td><td><strong>92.29 ± 0.91</strong></td><td>66.83 ± 15.67</td><td>65.51 ± 6.25</td><td>92.29 ± 4.57</td></tr><tr><td>French</td><td><strong>93.15 ± 0.96</strong></td><td>80.37 ± 7.97</td><td>62.87 ± 17.65</td><td>88.25 ± 2.24</td></tr><tr><td>Hebrew</td><td><strong>66.52 ± 2.59</strong></td><td>15.47 ± 10.12</td><td>36.10 ± 4.28</td><td>92.50 ± 0.25</td></tr><tr><td>Portuguese</td><td><strong>93.12 ± 1.10</strong></td><td>58.52 ± 18.48</td><td>70.53 ± 9.88</td><td>93.11 ± 8.32</td></tr><tr><td>Sanskrit</td><td><strong>64.18 ± 2.62</strong></td><td>33.20 ± 9.34</td><td>42.59 ± 4.88</td><td>91.48 ± 0.78</td></tr><tr><td>Slovak</td><td><strong>56.23 ± 4.57</strong></td><td>49.43 ± 3.13</td><td>39.58 ± 3.37</td><td>78.90 ± 0.86</td></tr><tr><td>Slovene</td><td><strong>71.99 ± 2.24</strong></td><td>57.79 ± 8.59</td><td>51.88 ± 6.69</td><td>82.41 ± 10.98</td></tr><tr><td>Swahili</td><td><strong>68.56 ± 6.09</strong></td><td>44.84 ± 7.77</td><td>44.46 ± 3.85</td><td>97.49 ± 0.21</td></tr><tr><td>Welsh</td><td><strong>63.80 ± 3.13</strong></td><td>47.30 ± 4.67</td><td>47.58 ± 5.72</td><td>96.79 ± 0.23</td></tr><tr><td>Zulu</td><td><strong>76.59 ± 2.65</strong></td><td>58.53 ± 4.42</td><td>42.56 ± 6.55</td><td>93.42 ± 0.73</td></tr></tbody></table>

CNN+3CosMul, CNN+ANNc, and AE+parallel. This result is not surprising, as there is a well known trade-off between the performance of deep learning and the explainability of symbolic approaches. In particular, the contribution of Kolmo [12] to explainability is clear as it explicitly provides the underlying transformation used to solve the analogy. However, the contribution of Alea is more subtle as its main contribution in that regard are the theoretical grounding of the method and the ability to limit the space of solutions.

## Performance of the symbolic baselines.

The baseline generation models have very low performance on Arabic, while the deep learning models do not appear to suffer from the same effect. As reported in [16], further analysis of the Arabic data reveals that the character encoding used decomposes each character into multiple encoded characters, resulting in longer and more complex sequences of characters than expected. However, on Slovene the performance is also very low even if the characters involved are more or less the same as for Portugese and English. We propose the hypothesis that the languages where Alea and Kolmo perform poorly contain data with irregularities. Indeed, the data used to test Kolmo and Alea in [12], introduced in [37], has been filtered to exclude analogies that can not be solved without knowledge of the language, in other words, without knowledge of the irregularities in the morphology of the language.
<table><caption>**Table 2** Analogy solving accuracy (rate of fully well predicted words, in %) in the generation setting (open world assumption), compared to the best retrieval model (CNN+ANNr). For models depending on an embedding model, and thus sensitive to random initializations, we report performance as mean ± standard deviation over 10 random initialization in each setting. Symbolic baselines Alea [8] and Kolmo [12] were tested in the same setting as our models.</caption><thead><tr><th rowspan="2">Language</th><th rowspan="2">Best ret. model<br>CNN+ANNr</th><th colspan="4">Generative models</th></tr><tr><th>AE+ANNr</th><th>AE+parallel</th><th>Alea</th><th>Kolmo</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-style: italic;">Sigmorphon 2016</td></tr><tr><td>Georgian</td><td>97.60 ± 0.23</td><td><strong>87.50 ± 2.08</strong></td><td><strong>87.06 ± 6.28</strong></td><td>84.97</td><td>79.94</td></tr><tr><td>Hungarian</td><td>89.06 ± 1.71</td><td><strong>90.58 ± 0.63</strong></td><td>83.57 ± 6.63</td><td>35.24</td><td>32.07</td></tr><tr><td>Turkish</td><td>84.75 ± 2.04</td><td><strong>79.81 ± 8.58</strong></td><td><strong>83.03 ± 14.07</strong></td><td>42.09</td><td>39.45</td></tr><tr><td colspan="6" style="text-align: center; font-style: italic;">Sigmorphon 2019</td></tr><tr><td>Adyghe</td><td>93.37 ± 0.97</td><td><strong>98.50 ± 0.25</strong></td><td>81.90 ± 7.35</td><td>47.94</td><td>31.25</td></tr><tr><td>Arabic</td><td>72.08 ± 3.49</td><td><strong>83.99 ± 1.28</strong></td><td>78.40 ± 12.37</td><td>2.21</td><td>3.34</td></tr><tr><td>Bashkir</td><td>57.63 ± 5.48</td><td><strong>95.15 ± 0.92</strong></td><td>80.38 ± 18.55</td><td>22.29</td><td>29.89</td></tr><tr><td>English</td><td>92.29 ± 0.91</td><td><strong>92.29 ± 4.57</strong></td><td>65.61 ± 19.96</td><td>60.15</td><td>47.69</td></tr><tr><td>French</td><td>93.15 ± 0.96</td><td><strong>88.25 ± 2.24</strong></td><td>76.04 ± 10.20</td><td>54.48</td><td>54.39</td></tr><tr><td>Hebrew</td><td>66.52 ± 2.59</td><td><strong>92.50 ± 0.25</strong></td><td><strong>91.16 ± 7.29</strong></td><td>19.50</td><td>16.17</td></tr><tr><td>Portuguese</td><td>93.12 ± 1.10</td><td><strong>93.11 ± 8.32</strong></td><td>62.88 ± 24.26</td><td>78.01</td><td>71.28</td></tr><tr><td>Sanskrit</td><td>64.18 ± 2.62</td><td><strong>91.48 ± 0.78</strong></td><td>83.83 ± 5.37</td><td>42.80</td><td>28.83</td></tr><tr><td>Slovak</td><td>56.23 ± 4.57</td><td><strong>78.90 ± 0.86</strong></td><td>82.62 ± 6.66</td><td>30.66</td><td>28.81</td></tr><tr><td>Slovene</td><td>71.99 ± 2.24</td><td><strong>82.41 ± 10.98</strong></td><td><strong>80.95 ± 8.19</strong></td><td>2.64</td><td>5.43</td></tr><tr><td>Swahili</td><td>68.56 ± 6.09</td><td><strong>97.49 ± 0.21</strong></td><td>81.94 ± 21.80</td><td>60.23</td><td>43.02</td></tr><tr><td>Welsh</td><td>63.80 ± 3.13</td><td><strong>96.79 ± 0.23</strong></td><td>87.62 ± 12.69</td><td>14.47</td><td>19.15</td></tr><tr><td>Zulu</td><td>76.59 ± 2.65</td><td><strong>93.42 ± 0.73</strong></td><td>81.96 ± 15.81</td><td>26.17</td><td>27.69</td></tr></tbody></table>

Interestingly, English and Portugese where AE+parallel performs the worst are also the languages of Sigmorphon2019 where the symbolic baselines Alea and Kolmo perform the best. Portugese has a very regular inflexional morphology [55], and the design of Alea and Kolmo allows them to handle morphological transformations which are frequent in those two languages, in particular affixation [8, 12, 55]. Given that the performance of AE+ANNr and of models using the CNN-based embedding does not show the same tendency as AE+parallel, we hypothesize that the AE together with the AE pre-training task have trouble handling those transformations. However, due to the high variance in the performance it is hard to draw definite conclusions.

While it would be beneficial for the understanding of the limitations of Alea and Kolmo to perform an in depth morphological analysis of the data of languages were the performance is low, this falls out of the scope of this work.

### ANNr improves on analogy solving performance

The AE+parallel, CNN+ANNc, and CNN+3CosMul models have significantly lower performance on analogy solving than CNN+ANNr and AE+ANNr. While it is possible that this performance gap is caused by the finetuning on analogy solving present when ANNr is used but not the other models, we argue that this is not the main factor. Indeed, CNN+ANNc and CNN+3CosMul use an embedding model trained with analogy while AE+parallel does
not, yet the latter outperforms the former. By considering the gap in perfor-
mance between ANNr and the other models, we conclude that ANNr, which
was designed and trained specifically for analogy solving, performs better on
that task than general purpose embedding models.

As can be seen in Table 2, we obtain equivalent or better performance with AE+ANNr than we obtained from CNN+ANNr, even though the latter benefits from an embedding model designed specifically for morphology and from a closed set of possible solutions to retrieve from. The training procedure and architecture of AE+ANNr compared to CNN+ANNr differ mostly in the use of the decoder output to compute the loss in AE+ANNr. By having direct access to the characters corresponding to the output of ANNr, it is likely that the model learns to avoid small differences in the word form that are not well captured by the CNN-based embedding model. In depth comparison of the errors of the two models would be required to confirm this hypothesis. We leave such studies for further work.

***Analogical training reduces sensitivity to random initialization.***

Putting aside the actual performance achieved, we observe that using ANNr
significantly reduces the standard deviation of model performance compared
to using only the pretrained AE embedding. Indeed, as shown in appendix
Table A4, the AE has a very large standard deviation on the AE task, even
when considering 50 models. A similar observation can be made in Tables 1
and 2 for CNN+3CosMul, CNN+ANNc, and AE+parallel, while CNN+ANNr
and AE+ANNr have a significantly lower standard deviation.

There are only two differences between the two groups of models: the use
of ANNr and the finetuning of the embedding models using analogical data
augmentation. In [14], we also identified the significant importance of data
augmentation in the performance and stability of CNN+ANNc models on
morphological analogies. Moreover, we observed on a different task in (the
target sense verification [21]) that using ANNc and analogy during training
also reduces the sensitivity of the model to input encoding. Overall, it appears
that analogical training with ANNc or ANNr allows to compensate for high
variance in embedding model performance and sensitivity to input conditions.

***Use of ANNc for analogy solving.***

Our experiments with ANNc revealed an unexpected weakness of the
approach, namely, the time required to compute the score. For instance,
for Turkish from Sigmorphon2016, in average, running the retrieval with
CNN+ANNc took more than 20 minutes against 46 seconds for CNN+3Cos-
Mul and 40 seconds for CNN+ANNr. In comparison, it took 2.5 minutes
to train for CNN+ANNc and 2 minutes for CNN+ANNr (4.5 minutes if we
include the pretraining of the CNN-based embedding model), on a computer
equipped with an Nvidia RTX A5000 Mobile used at close to 100% of its pro-
cessing capabilities. In this example, the time required to retrieve the solution
with CNN+ANNc is an order of magnitude 10 times longer than the time
to train an CNN+ANNr models from the CNN+ANNc model and apply it.
Using CNN+3CosMul is even faster as it is not necessary to train a new model,
reaching a 30 times speedup. This issue is caused by having to repeat the
computation of the classification score for all the words of the vocabulary for
each analogy to solve, however careful engineering might reduce the impact
on run time. For example, selecting a subset of candidates using 3CosMul
before applying CNN+ANNc as a retrieval model would allow for a significant
speedup.

Additionally, the ANNc model is designed for classification and has a tency to distinguish poorly between comparatively meaningful solution to the AP equation. This can be confirmed by extending the results to the top 10 highest classification scores, which significantly increases the accuracy as shown in appendix Subsection A.5.

Despite these limitations, CNN+ANNc outperforms CNN+ANNr on Bashkir, and CNN+3CosMul on 7 languages (Adyghe, Arabic, Bashkir, Hebrew, Portugese, Sanskrit, and Welsh). As mentioned before, the results become even more interesting when we look at the performance within the top 3, 5, and 10 retrieved words, for which CNN+ANNc outperforms CNN+3CosMul respectively on 8, 11, and 12 languages out of 16. This result is in line with the properties of ANNc and 3CosMul for retrieval, as 3CosMul relies on an arbitrary retrieval formula, while ANNc is learned together with the embedding, the latter resulting in a more appropriate alignment between the embedding space and the retrieval approach used. In simpler terms, ANNc is what was used to train the CNN-based embedding model for morphological analogies, so the features learned by the CNN-based embedding model are easier to properly manipulate with ANNc than with another model.

# 11 Discussion

In this section we discuss the results of previous sections as well as the
ones of our previous papers on the proposed approach, and also provide
recommendations for using and further developing the framework.

## 11.1 Benefit of ANNc and ANNr over symbolic, non-parametric, and other parametric approaches

The framework we propose for analogy detection and solving uses ANNc and ANNr, two deep learning models inspired from the properties of APs. It can be argued that the architecture of ANNc and ANNr is not the most fitting for the manipulation of APs, yet it is relatively easy to grasp and achieves good performance. As shown in this paper, the performance achieved is better than the one of non-parametric models such as the parallelogram rule and 3CosMul. Additionally, [11] showed, for semantic APs, the benefit of the ANNc and ANNr models over multi-layer perceptron [56] (up to 5 layers), random forest [57] and support vector machine [58]. The performance improvement is
especially striking for ANNr, while for ANNc the benefits are seen mostly on the SAT-based task which is considered a harder semantic analogy task [11].

Our framework also outperforms symbolic approaches to morphological APs. In all our works we compare ourselves with Alea [8] and Kolmo [12], as well as with the approach of [24] in some cases. In [15] we compare CNN+ANNc with [24] as well as a relaxed formulation of Alea and Kolmo. It can be argued that comparing generative approaches to analogy solving with analogy detection approaches such as Alea and Kolmo is unfair, and this was taken into account before drawing conclusions. Additionally, the top 10 generated solutions were considered in particular for valid AP detection, resulting in a higher accuracy for the baselines. In this setting, ANNc very significantly outperforms the baselines on valid APs with no significant difference on invalid APs.

AE+ANNr and CNN+ANNr outperform the symbolic baselines by a very significant margin in all cases. Furthermore, by considering the 10 most likely retrieval results for CNN+ANNr (see Subsection A.5), the accuracy improves above 99% for all but 6 languages above 95%. We observed a similar phenomenon in [15] for CNN+ANNr, with above 98% accuracy for all but 2 languages (at 95% and 92% respectively) when considering the 10 most likely results.

In [16] we managed to significantly outperform our symbolic baselines with AE+parallel and to reach performance slightly lower performance with the generation approach than with the retrieval approach CNN+ANNr. Further analysis of the results revealed that AE+parallel was highly accurate with regular morphology and with certain permutations, while it struggled when the morphology was more irregular. These considerations resulted in us developing AE+ANNr, which outperforms CNN+ANNr on multiple languages as shown in Table 2.

## 11.2 Improvements in the training process

To achieve the best possible performance, we experimented with many different parameters of the training procedure to reach the current state of the framework.

Firstly, regarding ANNc, in [13] we identified an imbalance in the analogy detection performance due to the data augmentation process, as we later identified in [15]: using the 8 equivalent forms of a valid analogy together with the 24 corresponding invalid forms skewed the data in favor of invalid analogies, while using only the 3 invalid forms obtained before computing the 8 valid forms skewed the data in the other direction, resulting in a matching imbalance in the performance over the two classes for CNN+ANNc. A compromise was found by sampling 8 out of the 24 invalid forms, as presented in Subsection 6.3. Based on detailed analyses of results of some languages, in particular from [17, 18], we further refined the data augmentation to exclude invalid forms that also appear in the valid forms. As explained later and as can be seen in
[18, 21, 59], proper care should be given to the data augmentation process to make the most of the benefits it brings.

Secondly, the current training procedure of ANNr is the result of multiple observations from [17]. Indeed, pre-training the embedding is necessary to achieve good performance on analogy solving with ANNr, and this can be achieved with relative ease by using CNN+ANNc (with a 10% to 40% reported in [17]) or an AE task (as we identified in preliminary experiments to the present work). We also observed that when training only ANNr, the model performance was not satisfying, yet naively training CNN+ANNr with MSE resulted in a collapse of the embedding space. After experimenting with various training criterion based on these results, we achieved the best results with $\mathcal{L}_{ANNr}$ (as currently presented in Subsection 6.4). While $\mathcal{L}_{ANNr}$ is an easy to implement way to mitigate embedding space collapse, in many settings it achieves comparable performance to the other training criterion we considered, so we encourage the reader to consider [17, Table 1.] to have a better outlook when choosing the training criterion.

Other details, in particular with regards to the implementation of approach, were refined along our experiments. It would be beneficial to further refine our training process as well as the training parameters and hyperparameters, but we leave those considerations for further work due to time limitations.

## 11.3 Multilinguality

In this work we did not attempt to model multiple languages at once or to transfer models between languages, as these aspects were extensively studied for CNN+ANNc in [13, 14, 18]. In particular, in [14] we experimented with different forms of transfer from a source language to a target language without finetuning, within Sigmorphon 2016 and JBATS. Transferring the whole CNN+ANNc resulted in reasonable performance (often above 50% accuracy), however the transfer failed on some languages. Indeed, those languages used a different set of characters than the rest of languages, resulting in a significant amount of unrecognized characters. To tackle this issue that we call the *alphabet gap*, we used the CNN-based embedding model of the target language together with the ANNc of the source language. This did not improve the performance for all languages, likely due to a mismatch between the embedding space of the CNN-based embedding model (trained on the target language) and the one of the ANNc (trained on the source language).

In [14] we also experimented with a model covering multiple languages: either two representative languages or all languages at once, with several variants. All variants achieved comparable results, with a high performance on positive examples and a poorer performance on negative examples. In all cases, as expected, the overall performance was poorer than with models dealing with only one language, with a slightly higher performance for bilingual models and a more consistent performance for the models on all languages.
We performed further transfer experiments in [18] by transferring the full
CNN+ANNc on Sigmorphon 2019, which covers a wider range of languages.
We leveraged languages present in both Sigmorphon 2016 and Sigmorphon
2019 to confirm that our approach generalizes even when the distribution
of morphological transformations is not the same, with a strong correlation
between the performance loss of transferred models and the alphabet gap. A
striking example is that of Arabic, for which the character representation in
Sigmorphon 2016 is completely different than that of Sigmorphon 2019: the
former uses a romanized version of the words while the latter uses the actual
UTF8 encoding of the Arabic characters. Furthermore, the high-resource lan-
guages of Sigmorphon 2019 form 4 clusters (plus 4 outliers) corresponding to
different sets of characters, i.e., different alphabets, as identified in [18]. Within
each cluster and in particular the largest one, some alphabet gap remains (e.g.,
the accents present in French but not English), however it is no longer cor-
related with the transfer performance. Instead, we found striking similarities
between the latter and the language families that can be found in Wikipedia.
In particular, similar clusters were found in a hierarchical clustering based on
the transfer performance and the hierarchy of language families.

Based on our observations, it is possible to use the proximity in the
Wikipedia language families to predict the performance of transferred models,
which not only confirm the transferability of morphological analogies between
languages but also opens possibilities to empirically study morphological simi-
larities between languages. Moreover, these results can be used to identify the
language to use when dealing with low resource languages, by finding a closely
related high resource language to train the language on. Finally, the large
scale results with comparable transfer performance between closely related
languages can be used to design a multilingual analogy model similar to [14]
on a larger scale. One such approach may be studied in further work, for exam-
ple using different components of the multilingual model for each cluster of
languages.

11.4 Behavior of the models with regards to the axiomatic setting

In [18], transfer is also used to confirm that changing the axioms used in the
data augmentation process results in a significantly different model correspond-
ing to the changed axiomatic settings. This highlight the importance of the
choice of the axiomatic setting in our framework, in particular since some
axioms are discussed in some application settings [28, 59], as the model behav-
ior can significantly change depending on the chosen axioms. If enough data
is available, we hypothesize that it is possible to determine the most fitting
set of axioms by finding the axiomatic setting with the closest performance
to the analogical data. Experiments on a wider range of application domains
would be required to confirm this hypothesis.
When analyzing the results of Navajo and Georgian in [17], we identified
several interesting behaviors of CNN+ANNr: (i) there is no significant per-
formance difference based on permutation, which indicates that the model is
invariant to the axioms of APs, (ii) reflexivity and identity (A : B :: A : B
and A : A :: B : B respectively) are well handled, (iii) when an example
appears to violate strong reflexivity or strong inner reflexivity (examples like
A = B, C ≠ D and A ≠ B, C = D) the ANNr model frequently gives a (wrong)
answer which corresponds to a proper application of the violated axiom.

11.5 Model input

In [21] we applied the data augmentation process and ANNc to target sense
verification, and observed that using ANNc together with our data augmen-
tation during training reduces the sensitivity of the model to input encoding.
The results of the present paper also indicate that data augmentation reduces
the sensitivity to the initial setting of the model. Overall, it appears that our
framework results in models that are less sensitive to slight changes in their
input, as they are made invariant to changes in the input due to the axioms
of APs.

Additionally, we explored in [15] the sensitivity of CNN+ANNc and
CNN+ANNr to perturbations in their input, by applying dropout on the word
embeddings. By randomly replacing some embedding dimensions by 0 with
a given probability, we found that beyond a certain dropout probability the
performance of CNN+ANNc dropped drastically for invalid APs, and observed
a similar but less striking phenomenon for CNN+ANNc on valid APs and
for CNN+ANNr. The probability threshold depends on the language we con-
sider, and we observe a plateau of high performance before the threshold.
This allowed us to identify languages for which the embedding appears to con-
tain redundant information (hence there is little effect of removing a few) or
conversely languages that might benefit from an increase in embedding size
(where the plateau of high performance does not appear, and that are more
sensitive to dropout). In the latter category we find Japanese, which is in line
with the larger set of possible characters.

11.6 Limitations of the data used

Over our experiments, we identified several limitations of our work due to the
data we use for training our models.

Firstly, the encoding of characters used in Sigmorphon 2016 is significantly
different from the one of Sigmorphon 2019 and JBATS, as they respectively
a rewriting of the language using roman alphabet and the standard UTF8
encoding of the characters of the language. As discussed above, this difference
results in an alphabet gap between some languages, limiting the possibility of
transferring models Additionally, JBATS contains multiple writings for some
words, which has not been leveraged in our previous work on that language.
Secondly, for most languages in the Siganalogies dataset, the grammatical
categories represented are very skewed towards specific types of words. Among
other effects, this impacts the generalization ability of the AE embedding
model as can be seen when trying to apply the model on words of grammatical
categories underrepresented in the data. On the flip-side, from the results
of [18] it appears that CNN+ANNc overcomes this limitation and is able to
properly generalize, if we exclude the impact of the alphabet gap.

It is interesting to note that [12] uses a slightly different version of the
Sigmorphon 2016 data, where the analogies that cannot be inferred by relying
only on the characters of the source terms have been removed.

11.7 Limitations of the symbolic approaches

Along our experiments, and in particular in [17], we identified several limi-
tations of the Alea [8] and Kolmo [12] symbolic approaches that we use as
baselines in most of our experiments. For instance, longer words and words
with many repeated adjacent letters significantly reduce the speed of the two
approaches, with this phenomenon particularly striking for Kolmo, as was
already identified by the authors of [12]. We solved this issue by introducing
a timeout for Alea and Kolmo as described in [17], as it appears that for most
languages less than 0.5% of the AP equations took longer than 10 seconds to
solve. Additionally, in [13] we found that Kolmo struggled with APs obtained
in the data augmentation by using central permutation. It is interesting that
this approach, based on empirical observations of human behavior in analogy
solving, does not follow all the axioms of APs, as it further highlights the need
for flexibility with regards to the axiomatic setting.

11.8 Diversity in tackling analogical proportions

Our framework proposes multiple technical solutions, in particular for analogy
solving, each with their own benefits. For instance, while ANNr outperforms
3CosMul, the latter does not require training an additional model. Also, while
using ANNc for retrieval outperforms 3CosMul without requiring additional
training, using the former is not recommended in most cases due to two key
limitations: (i) the execution time for retrieval is greater than that of train-
ing an ANNr model and using traditional cosine-based retrieval, and (ii)
ANNc tends to give similarly high scores to multiple solutions, which entails
the expected solution appearing further in the ranking. However, (ii) can
be a desired property of the model, and (i) can be reduced by engineering
the process and preprocessing the candidates more effectively. Additionally,
as reported in [17], 3CosMul does not systematically perform better than
3CosAdd, so it can be relevant to test both models. Similarly, while we use
cosine distance for retrieval with CNN+ANNr based on preliminary exper-
iments, there was no significant difference between cosine and Euclidean
distance.
Beyond the choice of the model, using our framework requires to formulate
the problem to tackle as APs. This step can be challenging, as can be seen in
[59] or [21] where multiple formulations are explored, but the effort is usually
rewarded by good performance and the integration of analogical knowledge in
the model.

12 Conclusion

Our framework achieves very high analogy detection and solving performance, in a variety of settings. Both the CNN+ANNc for analogy detection and the AE+ANNr for analogy solving achieve consistent state of the art performance on all the languages used in our experiments, and outperform the symbolic baselines Alea [8] and Kolmo [12] by as much as 80% in some cases. This gain in performance results from, among other things, (i) a representation of words learned on the data for the purpose of analogy manipulation, (ii) the ability of the model to integrate the dependencies between the embedding dimensions, and (iii) the flexibility to go beyond arbitrary arithmetic formulas of APs such as 3CosAdd, 3CosMul, or the parallelogram rule. However, there is a known trade-off between the performance of deep learning models and their interpretability. In particular, it is usually difficult to understand why an artificial neural network obtains a particular output. This also applies to our framework, but further work might provide theoretical guarantees or empirical methods to tackle this limitation.

In the current work we performed two sets of new experiments to confirm
the benefit of using ANNc and ANNr. Firstly, we compared the retrieval per-
formance of ANNc to ANNr and 3CosMul and showed that ANNc can be used
to outperform 3CosMul, in particular when considering the top-10 retrieved
solutions. However, using ANNc in this manner results in significant increase
in retrieval time, which should be addressed in further work. Secondly, we
combined ANNr and the AE embedding model, and outperform the symbolic
baselines but also the previous approach relying on the parallelogram rule by
as much as 30% in some cases. This new model outperforms even the retrieval
approach for multiple languages, despite the latter benefiting from a closed
set of possible solutions.

Overall, our framework shows that it is possible to obtain high perfor-
mance when manipulating APs beyond arithmetic models on embeddings or
manually-designed approaches. The framework leverages the properties and
intuitions of APs in the design of the models, but also to augment data in
a way that benefits model performance, sensitivity to initial conditions, and
makes the model invariant with regards to permutations following the axioms
of APs. The approach can be applied to other types of data following our
guidelines, as can be seen in [10, 11, 21, 59].

**Funding.** This research was partially supported by the ANR project "Analogies: from theory to tools and applications" (AT2TA), ANR-22-CE23-0023, by
the TAILOR project funded by EU Horizon 2020 research and innovation program under GA No 952215, and by the Inria Project Lab “Hybrid Approaches for Interpretable AI” (HyAIAI).

**Code availability statement.** Our implementation is publicly available at GitHub (https://github.com/EMarquer/morpho-analogy-amai). Code used to generate and manipulate the Siganalogies [25] V1 dataset is available at GitHub https://github.com/EMarquer/siganalogies.

**Data availability statement.** The Siganalogies [25] V1 dataset (DOI: 10.12763/MLCFIE) is publicly available in the Dorel repository https://dorel.univ-lorraine.fr/dataset.xhtml?persistentId=doi:10.12763/MLCFIE. Our trained models will appear in the Dorel repository https://dorel.univ-lorraine.fr/dataset.xhtml?persistentId=doi:10.12763/15ED78, excluding the models reported in appendix due to their large volume.

**Conflict of interest.** The authors declare that they have no conflict of interest.
# Appendix A Additional experimental results

This appendix contains additional details for some of our experiments. Subsection A.1 covers the key hyperparameters used when training the model. Subsection A.2 reports the results of pre-trained semantic embedding on solving morphological AP equations on English. Subsection A.3 detains the performance of the CNN+ANNc model on analogy detection (the pre-training task of CNN+ANNr and CNN+3CosMul), and Subsection A.4 covers the pre-training performance of the AE embedding model. Finally, Subsection A.5 provide extended results for the analogy solving models.

## A.1 Training hyperparameters

The optimisation algorithm used for training our models is as follows:

* CNN+ANNc: Adam [60] with a learning rate of $10^{-3}$ and batches of 256;

* CNN+ANNr: Adam with a learning rate of $10^{-3}$ and batches of 256;

* AE pretraining: NAdam [61] (slightly better results than Adam in our preliminary experiments) with a learning rate of $10^{-2}$ and batches of 2048;

* AE+ANNr: Adam with a learning rate of $10^{-3}$ and batches of 512.

## A.2 Semantic pre-trained embedding models on morphology

Here we report the performance of ANNc using pre-trained semantic embeddings, in particular GloVe6B <fnref n="5" /> and fasttext <fnref n="6" />. For GloVe6B we use the models with 50, 100, 200, and 300 dimensions, and the fasttext embeddings have 300 dimensions. It would be unreasonable to expect the best performance from semantic embeddings on a morphological task, due to for instance synonyms for which the forms (and thus the morphemes) are not distinguished. However, in English as in many languages some semantic features translate directly to morphemes, for instance the plural “-s”, which may result in an amount of success.

A first issue encountered is that the pre-trained models cover only a small part of the word vocabulary: 37.69% for GloVe6B and 53.35% for fasttext. Taking only the words covered by both models (to get a comparable result) reduces the vocabulary to 37.15% of its original size. Analogies containing only covered represent 10.33% of the original APs. To obtain results that are more comparable to our CNN+ANNc model, which has the embedding trained for AP detection, we add a fully connected layer with output dimension 80 after the pre-trained embedding model, resulting on embeddings of comparable dimension to the CNN-based ones. Additionally, this layer is meant to handle finetuning for the embedding model. We present in Table A1 results for model trained and tested on the original data, and in Table A2 for models using only

<footnote n="5">⁵GloVe trained on a wikipedia dump, see https://nlp.stanford.edu/projects/glove/</footnote>

<footnote n="6">⁶https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en Payment</footnote>
covered APs. In each case we specify the dimension of the Glove6B model used, and models with a “+” sign are equipped with the fully connected layer.

As one could expect, models using pre-trained embeddings and trained and tested on non-covered analogies have a significantly lower performance than CNN+ANNc, as shown in Table A1, for the most part due to unrecognized words during training and testing. In this setting, there is no significant difference between the models, with improvements of 2% to 4% on the F1 when learning a fully connected layer after the embedding. If we consider only covered APs for training and testing the model, the F1 for GloVe6B jumps to 94% to 99% for models with the fully connected layer and to 81% to 84% for models without. Surprisingly, fasttext, which is based on embeddings of subwords (which are similar to morphemes), achieves a lower F1 of 52% without and 56% with the fully connected layer.

In addition to the advantages of character-based embeddings (avoiding unknown words) and to the performance gap, our morphological embedding approaches result in more lightweight embedding models, since it is not necessary to store an embedding for each word. For instance, our CNN-based embedding model for English weights less than 100 KB, while Glove6B weights 171 MB, 347 MB, 693 MB and 1 GB for 50, 100, 200, and 300 dimensions, respectively, and fasttext weights 6.6 GB.

<table><caption>**Table A1** Performance of the ANNc model on analogy detection (in %, mean ± std.) on English (Sigmorphon 2019), for all analogies. We consider several pre-trained embedding variants, for 5 random initialization of the model. Models with a “+” sign have a fully connected layer after the embedding and before ANNc. The balanced accuracy (Bal. acc.) is the average of the True Positive Rate (TPR) and True Negative Rate (TNR), weighted by the number of the positive and negative examples respectively.</caption><thead><tr><th>Model</th><th>F1</th><th>Bal. acc.</th><th>TPR</th><th>TNR</th></tr></thead><tbody><tr><td>FastText</td><td>52.21 ± 0.34</td><td>66.99 ± 0.28</td><td>59.69 ± 1.34</td><td>74.30 ± 1.44</td></tr><tr><td>GloVe6B 50</td><td>52.61 ± 0.21</td><td>67.30 ± 0.19</td><td>60.43 ± 2.33</td><td>74.17 ± 2.55</td></tr><tr><td>GloVe6B 100</td><td>52.82 ± 0.21</td><td>67.48 ± 0.18</td><td>60.05 ± 1.66</td><td>74.90 ± 1.78</td></tr><tr><td>GloVe6B 200</td><td>52.88 ± 0.33</td><td>67.52 ± 0.25</td><td>60.57 ± 1.96</td><td>74.47 ± 1.92</td></tr><tr><td>GloVe6B 300</td><td>52.70 ± 0.24</td><td>67.38 ± 0.21</td><td>60.07 ± 1.57</td><td>74.68 ± 1.86</td></tr><tr><td>FastText+</td><td>56.22 ± 0.35</td><td>70.08 ± 0.26</td><td>61.58 ± 1.75</td><td>78.58 ± 1.70</td></tr><tr><td>GloVe6B 50+</td><td>54.38 ± 0.34</td><td>68.70 ± 0.26</td><td>59.82 ± 1.37</td><td>77.57 ± 1.47</td></tr><tr><td>GloVe6B 100+</td><td>55.20 ± 0.25</td><td>69.31 ± 0.19</td><td>60.89 ± 1.10</td><td>77.74 ± 0.89</td></tr><tr><td>GloVe6B 200+</td><td>55.54 ± 0.34</td><td>69.59 ± 0.24</td><td>62.46 ± 1.53</td><td>76.72 ± 1.78</td></tr><tr><td>GloVe6B 300+</td><td>55.96 ± 0.24</td><td>69.92 ± 0.18</td><td>63.03 ± 1.71</td><td>76.81 ± 1.66</td></tr><tr><td>CNN+ANNc</td><td>99.39 ± 0.04</td><td>99.70 ± 0.05</td><td>99.76 ± 0.12</td><td>99.64 ± 0.03</td></tr></tbody></table>

## A.3 CNN embedding pre-training performance

We report in Table A3 the performance of the CNN+ANNc model on analogy detection. The models presented here are the same as the ones used for the
<table><caption>**Table A2** Performance of the ANNc model on analogy detection (in %, mean ± std.) on English (Sigmorphon 2019), for covered analogies only. We consider various variants of pre-trained embeddings, for 5 random initialization of the model, with all models trained and tested with covered analogies only. Models with a “+” sign have a fully connected layer after the embedding and before ANNc. The balanced accuracy (Bal. acc.) is the average of the True Positive Rate (TPR) and True Negative Rate (TNR), weighted by the number of the positive and negative examples respectively.</caption><thead><tr><th>Model</th><th>F1</th><th>Bal. acc.</th><th>TPR</th><th>TNR</th></tr></thead><tbody><tr><td>FastText</td><td>52.14 ± 0.19</td><td>66.93 ± 0.11</td><td>60.06 ± 2.14</td><td>73.79 ± 2.03</td></tr><tr><td>GloVe6B 50</td><td>81.70 ± 1.43</td><td>89.81 ± 0.80</td><td>95.30 ± 0.90</td><td>84.32 ± 1.88</td></tr><tr><td>GloVe6B 100</td><td>84.56 ± 1.27</td><td>91.26 ± 0.69</td><td>94.51 ± 0.55</td><td>88.02 ± 1.42</td></tr><tr><td>GloVe6B 200</td><td>84.03 ± 0.46</td><td>90.95 ± 0.28</td><td>94.36 ± 0.55</td><td>87.54 ± 0.62</td></tr><tr><td>GloVe6B 300</td><td>82.50 ± 0.42</td><td>90.09 ± 0.47</td><td>94.35 ± 2.52</td><td>85.83 ± 1.73</td></tr><tr><td>FastText+</td><td>56.24 ± 0.30</td><td>70.08 ± 0.24</td><td>61.08 ± 1.62</td><td>79.09 ± 1.44</td></tr><tr><td>GloVe6B 50+</td><td>94.83 ± 0.86</td><td>97.15 ± 0.47</td><td>97.77 ± 0.52</td><td>96.52 ± 0.68</td></tr><tr><td>GloVe6B 100+</td><td>98.37 ± 0.17</td><td>99.06 ± 0.11</td><td>99.11 ± 0.30</td><td>99.01 ± 0.19</td></tr><tr><td>GloVe6B 200+</td><td>98.76 ± 0.15</td><td>99.34 ± 0.10</td><td>99.51 ± 0.18</td><td>99.17 ± 0.12</td></tr><tr><td>GloVe6B 300+</td><td>98.83 ± 0.15</td><td>99.36 ± 0.09</td><td>99.47 ± 0.19</td><td>99.25 ± 0.14</td></tr></tbody></table>

CNN+ANNc model for analogy solving, and the CNN-based embedding model was reused for CNN+3CosAdd and finetuned for CNN+ANNr.

## A.4 Autoencoder embedding pre-training performance

We report in Table A4 the AE accuracy at word level of the AE embedding model. The results presented cover 5 random data splits and 10 random initialization of the model per split, resulting in 50 models per language. Only the 10 random initialization of the model were used for the first random data split were used in our other experiments involving AE+ANNr and AE+parallel.

## A.5 Extended analogy solving results

In this appendix we provide Table A5 summarizing the results of all our analogy solving approaches for easier comparison.

We describe and compare in the hit rate at *k* for *k* ∈ {1, 3, 5, 10} for our retrieval models in Tables A6 to A9. The hit rate corresponds to the rate of analogies for which the expected answer appears in the top *k* retrieved answers. For *k* = 1, the hit rate is the accuracy.
<table><caption>**Table A3** Performance of the CNN+ANNc model on analogy detection (in %, mean ± std.) for 10 random initialization of the model. The balanced accuracy (Bal. acc.) is the average of the True Positive Rate (TPR) and True Negative Rate (TNR), weighted by the number of the positive and negative examples respectively.</caption><thead><tr><th>Language</th><th>F1</th><th>Bal. acc.</th><th>TPR</th><th>TNR</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-style: italic;">Sigmorphon 2016</td></tr><tr><td>Georgian</td><td>98.26 ± 0.09</td><td>99.18 ± 0.07</td><td>99.39 ± 0.19</td><td>98.98 ± 0.09</td></tr><tr><td>Hungarian</td><td>99.82 ± 0.05</td><td>99.87 ± 0.04</td><td>99.80 ± 0.09</td><td>99.95 ± 0.03</td></tr><tr><td>Turkish</td><td>99.75 ± 0.08</td><td>99.83 ± 0.05</td><td>99.76 ± 0.09</td><td>99.91 ± 0.04</td></tr><tr><td colspan="5" style="text-align: center; font-style: italic;">Sigmorphon 2019</td></tr><tr><td>Adyghe</td><td>97.37 ± 0.05</td><td>98.94 ± 0.04</td><td>99.55 ± 0.16</td><td>98.33 ± 0.08</td></tr><tr><td>Arabic</td><td>99.96 ± 0.02</td><td>99.98 ± 0.02</td><td>99.97 ± 0.04</td><td>99.98 ± 0.01</td></tr><tr><td>Bashkir</td><td>93.04 ± 0.07</td><td>97.07 ± 0.07</td><td>99.07 ± 0.17</td><td>95.06 ± 0.07</td></tr><tr><td>English</td><td>99.39 ± 0.04</td><td>99.70 ± 0.05</td><td>99.76 ± 0.12</td><td>99.64 ± 0.03</td></tr><tr><td>French</td><td>99.86 ± 0.06</td><td>99.88 ± 0.06</td><td>99.78 ± 0.11</td><td>99.98 ± 0.01</td></tr><tr><td>Hebrew</td><td>98.33 ± 0.12</td><td>99.18 ± 0.09</td><td>99.28 ± 0.21</td><td>99.09 ± 0.10</td></tr><tr><td>Portuguese</td><td>99.83 ± 0.09</td><td>99.88 ± 0.09</td><td>99.82 ± 0.17</td><td>99.95 ± 0.00</td></tr><tr><td>Sanskrit</td><td>98.08 ± 0.07</td><td>99.08 ± 0.05</td><td>99.20 ± 0.12</td><td>98.96 ± 0.06</td></tr><tr><td>Slovak</td><td>96.09 ± 0.15</td><td>98.02 ± 0.14</td><td>98.25 ± 0.36</td><td>97.79 ± 0.13</td></tr><tr><td>Slovene</td><td>99.65 ± 0.03</td><td>99.82 ± 0.04</td><td>99.81 ± 0.09</td><td>99.83 ± 0.03</td></tr><tr><td>Swahili</td><td>99.25 ± 0.04</td><td>99.66 ± 0.02</td><td>99.75 ± 0.07</td><td>99.57 ± 0.05</td></tr><tr><td>Welsh</td><td>98.63 ± 0.10</td><td>99.33 ± 0.11</td><td>99.37 ± 0.30</td><td>99.28 ± 0.12</td></tr><tr><td>Zulu</td><td>99.48 ± 0.04</td><td>99.78 ± 0.04</td><td>99.87 ± 0.08</td><td>99.69 ± 0.03</td></tr></tbody></table>
<table><caption>**Table A4** Accuracy (in %, mean ± std.) at the word level, of the AE pre-trained for at most 100 epochs on 40,000 random words, for 5 different training / test splits and 10 random initialization of the model in each case. Languages in bold are the ones selected for further experiments.</caption><thead><tr><th>Language</th><th>Accuracy (%)</th><th>Language</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td colspan="2"><em>Sigmorphon 2016</em></td><td colspan="2"><em>Sigmorphon 2019</em></td></tr><tr><td>Arabic</td><td>75.72±13.98</td><td><strong>French</strong></td><td>76.04±10.20</td></tr><tr><td>Finnish</td><td>73.30±9.96</td><td>German</td><td>64.98±16.12</td></tr><tr><td><strong>Georgian</strong></td><td>87.06±6.28</td><td>Greek</td><td>39.44±21.68</td></tr><tr><td>German</td><td>69.50±15.03</td><td><strong>Hebrew</strong></td><td>91.16±7.29</td></tr><tr><td><strong>Hungarian</strong></td><td>83.57±6.63</td><td>Hindi</td><td>58.08±32.85</td></tr><tr><td>Maltese</td><td>77.07±25.07</td><td>Hungarian</td><td>61.51±13.81</td></tr><tr><td>Navajo</td><td>38.36±21.41</td><td>Irish</td><td>10.39±12.59</td></tr><tr><td><em>Russian</em></td><td>84.70±6.63</td><td>Italian</td><td>70.92±13.52</td></tr><tr><td>Spanish</td><td>78.36±15.85</td><td>Kannada</td><td>0.05±0.16</td></tr><tr><td><strong>Turkish</strong></td><td>83.03±14.07</td><td>Kurmanji</td><td>76.52±8.50</td></tr><tr><td colspan="2"><em>Sigmorphon 2019</em></td><td>Latin</td><td>69.90±13.80</td></tr><tr><td><strong>Adyghe</strong></td><td>81.90±7.35</td><td>Latvian</td><td>78.29±5.53</td></tr><tr><td>Albanian</td><td>51.07±13.56</td><td>Persian</td><td>59.14±23.10</td></tr><tr><td><strong>Arabic</strong></td><td>78.40±12.37</td><td>Polish</td><td>67.36±18.58</td></tr><tr><td>Armenian</td><td>77.94±4.87</td><td><strong>Portuguese</strong></td><td>62.88±24.26</td></tr><tr><td>Asturian</td><td>73.71±26.71</td><td>Romanian</td><td>62.76±22.14</td></tr><tr><td><strong>Bashkir</strong></td><td>80.38±18.55</td><td>Russian</td><td>67.76±11.51</td></tr><tr><td>Belarusian</td><td>61.15±15.30</td><td><strong>Sanskrit</strong></td><td>83.83±5.37</td></tr><tr><td>Bengali</td><td>47.78±25.46</td><td><strong>Slovak</strong></td><td>82.62±6.66</td></tr><tr><td>Bulgarian</td><td>76.07±7.18</td><td><strong>Slovene</strong></td><td>80.95±8.19</td></tr><tr><td>Czech</td><td>61.08±19.61</td><td>Sorani</td><td>77.43±11.88</td></tr><tr><td>Danish</td><td>73.88±11.70</td><td>Spanish</td><td>67.86±21.80</td></tr><tr><td>Dutch</td><td>63.90±19.41</td><td><strong>Swahili</strong></td><td>81.94±21.80</td></tr><tr><td><strong>English</strong></td><td>65.61±19.96</td><td>Turkish</td><td>68.03±12.29</td></tr><tr><td>Estonian</td><td>66.78±9.02</td><td>Urdu</td><td>52.02±28.24</td></tr><tr><td>Finnish</td><td>34.68±22.89</td><td><strong>Welsh</strong></td><td>87.62±12.69</td></tr><tr><td></td><td></td><td><strong>Zulu</strong></td><td>81.96±15.81</td></tr></tbody></table>
<table><caption>**Table A5** Accuracy (rate of fully well predicted words, in %) on the analogy solving tasks. For models depending on an embedding model, and thus sensitive to random initialization, we report performance as mean ± standard deviation over 10 random initialization in each setting. Symbolic baselines Alea [8] and Kolmo [12] were tested in the same setting as our models.</caption><thead><tr><th rowspan="2">Language</th><th colspan="3">Retrieval models</th><th colspan="3">Generative models</th></tr><tr><th>CNN+ANNr</th><th>CNN+3CosMul</th><th>CNN+ANNc</th><th>AE+ANNr</th><th>AE+parallel</th><th>Alea</th><th>Kolmo</th></tr></thead><tbody><tr><td></td><td colspan="3"><em>Sigmorphon 2016</em></td><td colspan="3"></td></tr><tr><td>Georgian</td><td><strong>97.60</strong> ± <strong>0.23</strong></td><td>85.58 ± 7.37</td><td>76.77 ± 9.57</td><td><strong>87.50</strong> ± <strong>2.08</strong></td><td><strong>87.06</strong> ± <strong>6.28</strong></td><td>84.97</td><td>79.94</td></tr><tr><td>Hungarian</td><td><strong>89.06</strong> ± <strong>1.71</strong></td><td>74.89 ± 5.27</td><td>72.95 ± 8.02</td><td><strong>90.58</strong> ± <strong>0.63</strong></td><td>83.57 ± 6.63</td><td>35.24</td><td>32.07</td></tr><tr><td>Turkish</td><td><strong>84.75</strong> ± <strong>2.04</strong></td><td>52.42 ± 4.33</td><td>44.43 ± 13.95</td><td><strong>79.81</strong> ± <strong>8.58</strong></td><td><strong>83.03</strong> ± <strong>14.07</strong></td><td>42.09</td><td>39.45</td></tr><tr><td></td><td colspan="3"><em>Sigmorphon 2019</em></td><td colspan="3"></td></tr><tr><td>Adyghe</td><td><strong>93.37</strong> ± <strong>0.97</strong></td><td>58.01 ± 8.66</td><td>80.11 ± 5.10</td><td><strong>98.50</strong> ± <strong>0.25</strong></td><td>81.90 ± 7.35</td><td>47.94</td><td>31.25</td></tr><tr><td>Arabic</td><td><strong>72.08</strong> ± <strong>3.49</strong></td><td>21.67 ± 5.44</td><td>40.73 ± 9.93</td><td><strong>83.99</strong> ± <strong>1.28</strong></td><td>78.40 ± 12.37</td><td>2.21</td><td>3.34</td></tr><tr><td>Bashkir</td><td>57.63 ± 5.48</td><td>37.76 ± 11.12</td><td><strong>65.38</strong> ± <strong>6.01</strong></td><td><strong>95.15</strong> ± <strong>0.92</strong></td><td>80.38 ± 18.55</td><td>22.29</td><td>29.89</td></tr><tr><td>English</td><td><strong>92.29</strong> ± <strong>0.91</strong></td><td>66.83 ± 15.67</td><td>65.51 ± 6.25</td><td><strong>92.29</strong> ± <strong>4.57</strong></td><td>65.61 ± 19.96</td><td>60.15</td><td>47.69</td></tr><tr><td>French</td><td><strong>93.15</strong> ± <strong>0.96</strong></td><td>80.37 ± 7.97</td><td>62.87 ± 17.65</td><td><strong>88.25</strong> ± <strong>2.24</strong></td><td>76.04 ± 10.20</td><td>54.48</td><td>54.39</td></tr><tr><td>Hebrew</td><td><strong>66.52</strong> ± <strong>2.59</strong></td><td>15.47 ± 10.12</td><td>36.10 ± 4.28</td><td><strong>92.50</strong> ± <strong>0.25</strong></td><td>62.88 ± 24.26</td><td>78.01</td><td>71.28</td></tr><tr><td>Portuguese</td><td><strong>93.12</strong> ± <strong>1.10</strong></td><td>58.52 ± 18.48</td><td>70.53 ± 9.88</td><td><strong>93.11</strong> ± <strong>8.32</strong></td><td><strong>91.16</strong> ± <strong>7.29</strong></td><td>19.50</td><td>16.17</td></tr><tr><td>Sanskrit</td><td><strong>64.18</strong> ± <strong>2.62</strong></td><td>33.20 ± 9.34</td><td>42.59 ± 4.88</td><td><strong>91.48</strong> ± <strong>0.78</strong></td><td>83.83 ± 5.37</td><td>42.80</td><td>28.83</td></tr><tr><td>Slovak</td><td><strong>56.23</strong> ± <strong>4.57</strong></td><td>49.43 ± 3.13</td><td>39.58 ± 3.37</td><td><strong>78.90</strong> ± <strong>0.86</strong></td><td>82.62 ± 6.66</td><td>30.66</td><td>28.81</td></tr><tr><td>Slovene</td><td><strong>71.99</strong> ± <strong>2.24</strong></td><td>57.79 ± 8.59</td><td>51.88 ± 6.69</td><td><strong>82.41</strong> ± <strong>10.98</strong></td><td><strong>80.95</strong> ± <strong>8.19</strong></td><td>2.64</td><td>5.43</td></tr><tr><td>Swahili</td><td><strong>68.56</strong> ± <strong>6.09</strong></td><td>44.84 ± 7.77</td><td>44.46 ± 3.85</td><td><strong>97.49</strong> ± <strong>0.21</strong></td><td>81.94 ± 21.80</td><td>60.23</td><td>43.02</td></tr><tr><td>Welsh</td><td><strong>63.80</strong> ± <strong>3.13</strong></td><td>47.30 ± 4.67</td><td>47.58 ± 5.72</td><td><strong>96.79</strong> ± <strong>0.23</strong></td><td>87.62 ± 12.69</td><td>14.47</td><td>19.15</td></tr><tr><td>Zulu</td><td><strong>76.59</strong> ± <strong>2.65</strong></td><td>58.53 ± 4.42</td><td>42.56 ± 6.55</td><td><strong>93.42</strong> ± <strong>0.73</strong></td><td>81.96 ± 15.81</td><td>26.17</td><td>27.69</td></tr></tbody></table>
<table><caption>**Table A6** Accuracy, i.e., hit rate at 1 (in %, mean ± std.) at the word level for the retrieval models. **: highest average performance; *: second highest average performance.</caption><thead><tr><th>Sigmorphon</th><th>Language</th><th>CNN+ANNr</th><th>CNN+3CosMul</th><th>CNN+ANNc</th></tr></thead><tbody><tr><td rowspan="3">2016</td><td>Georgian</td><td><b>97.60 ± 0.23</b> **</td><td>85.58 ± 7.37 *</td><td>76.77 ± 9.57</td></tr><tr><td>Hungarian</td><td><b>89.06 ± 1.71</b> **</td><td>74.89 ± 5.27 *</td><td>72.95 ± 8.02</td></tr><tr><td>Turkish</td><td><b>84.75 ± 2.04</b> **</td><td>52.42 ± 4.33 *</td><td>44.43 ± 13.95</td></tr><tr><td rowspan="10">2019</td><td>Adyghe</td><td><b>93.37 ± 0.97</b> **</td><td>58.01 ± 8.66</td><td>80.11 ± 5.10 *</td></tr><tr><td>Arabic</td><td><b>72.08 ± 3.49</b> **</td><td>21.67 ± 5.44</td><td>40.73 ± 9.93 *</td></tr><tr><td>Bashkir</td><td>57.63 ± 5.48 *</td><td>37.76 ± 11.12</td><td><b>65.38 ± 6.01</b> **</td></tr><tr><td>English</td><td><b>92.29 ± 0.91</b> **</td><td>66.83 ± 15.67 *</td><td>65.51 ± 6.25</td></tr><tr><td>French</td><td><b>93.15 ± 0.96</b> **</td><td>80.37 ± 7.97 *</td><td>62.87 ± 17.65</td></tr><tr><td>Hebrew</td><td><b>66.52 ± 2.59</b> **</td><td>15.47 ± 10.12</td><td>36.10 ± 4.28 *</td></tr><tr><td>Portuguese</td><td><b>93.12 ± 1.10</b> **</td><td>58.52 ± 18.48</td><td>70.53 ± 9.88 *</td></tr><tr><td>Sanskrit</td><td><b>64.18 ± 2.62</b> **</td><td>33.20 ± 9.34</td><td>42.59 ± 4.88 *</td></tr><tr><td>Slovak</td><td><b>56.23 ± 4.57</b> **</td><td>49.43 ± 3.13 *</td><td>39.58 ± 3.37</td></tr><tr><td>Slovene</td><td><b>71.99 ± 2.24</b> **</td><td>57.79 ± 8.59 *</td><td>51.88 ± 6.69</td></tr><tr><td rowspan="3">Zulu</td><td>Swahili</td><td><b>68.56 ± 6.09</b> **</td><td>44.84 ± 7.77 *</td><td>44.46 ± 3.85</td></tr><tr><td>Welsh</td><td><b>63.80 ± 3.13</b> **</td><td>47.30 ± 4.67</td><td>47.58 ± 5.72 *</td></tr><tr><td>Zulu</td><td><b>76.59 ± 2.65</b> **</td><td>58.53 ± 4.42 *</td><td>42.56 ± 6.55</td></tr></tbody></table>

<table><caption>**Table A7** Hit rate at 3 (in %, mean ± std.) at the word level for the retrieval models. **: highest average performance; *: second highest average performance.</caption><thead><tr><th>Sigmorphon</th><th>Language</th><th>CNN+ANNr</th><th>CNN+3CosMul</th><th>CNN+ANNc</th></tr></thead><tbody><tr><td rowspan="3">2016</td><td>Georgian</td><td><b>99.54 ± 0.23</b> **</td><td>93.34 ± 3.82 *</td><td>92.07 ± 4.92</td></tr><tr><td>Hungarian</td><td><b>97.49 ± 0.42</b> **</td><td>89.04 ± 3.87 *</td><td>87.96 ± 5.84</td></tr><tr><td>Turkish</td><td><b>98.20 ± 0.48</b> **</td><td>68.29 ± 4.04 *</td><td>63.64 ± 12.89</td></tr><tr><td rowspan="10">2019</td><td>Adyghe</td><td><b>99.63 ± 0.13</b> **</td><td>74.78 ± 7.80</td><td>95.93 ± 1.84 *</td></tr><tr><td>Arabic</td><td><b>91.33 ± 1.94</b> **</td><td>38.61 ± 7.44</td><td>65.41 ± 11.60 *</td></tr><tr><td>Bashkir</td><td>78.50 ± 3.00 *</td><td>55.72 ± 12.65</td><td><b>83.85 ± 3.96</b> **</td></tr><tr><td>English</td><td><b>98.66 ± 0.29</b> **</td><td>76.34 ± 14.94</td><td>80.50 ± 4.66 *</td></tr><tr><td>French</td><td><b>98.33 ± 0.30</b> **</td><td>91.60 ± 4.64 *</td><td>82.94 ± 13.10</td></tr><tr><td>Hebrew</td><td><b>88.06 ± 2.09</b> **</td><td>30.20 ± 15.80</td><td>59.69 ± 4.93 *</td></tr><tr><td>Portuguese</td><td><b>99.06 ± 0.21</b> **</td><td>79.18 ± 17.98</td><td>91.58 ± 6.02 *</td></tr><tr><td>Sanskrit</td><td><b>89.95 ± 1.26</b> **</td><td>53.90 ± 8.92</td><td>66.03 ± 4.88 *</td></tr><tr><td>Slovak</td><td><b>87.86 ± 1.56</b> **</td><td>67.81 ± 4.08 *</td><td>64.46 ± 4.00</td></tr><tr><td>Slovene</td><td><b>92.28 ± 0.72</b> **</td><td>78.08 ± 8.26 *</td><td>75.92 ± 6.94</td></tr><tr><td>Swahili</td><td><b>84.72 ± 4.70</b> **</td><td>60.57 ± 7.08 *</td><td>57.34 ± 5.10</td></tr><tr><td>Welsh</td><td><b>88.29 ± 1.92</b> **</td><td>71.66 ± 5.04</td><td>76.67 ± 6.09 *</td></tr><tr><td>Zulu</td><td><b>89.66 ± 1.66</b> **</td><td>77.92 ± 4.10 *</td><td>64.51 ± 7.17</td></tr></tbody></table>
<table><caption>**Table A8** Hit rate at 5 (in %, mean ± std.) at the word level for the retrieval models. **: highest average performance; *: second highest average performance.</caption><thead><tr><th>Sigmorphon</th><th>Language</th><th>CNN+ANNr</th><th>CNN+3CosMul</th><th>CNN+ANNc</th></tr></thead><tbody><tr><td rowspan="2">2016</td><td>Georgian</td><td><b>99.73 ± 0.14</b> **</td><td>94.92 ± 2.63</td><td>95.14 ± 3.16 *</td></tr><tr><td>Hungarian</td><td><b>98.62 ± 0.29</b> **</td><td>92.40 ± 3.29 *</td><td>91.75 ± 4.70</td></tr><tr><td rowspan="11">2019</td><td>Turkish</td><td><b>99.28 ± 0.20</b> **</td><td>73.95 ± 3.43 *</td><td>70.90 ± 11.30</td></tr><tr><td>Adyghe</td><td><b>99.85 ± 0.07</b> **</td><td>79.44 ± 7.22</td><td>98.08 ± 0.97 *</td></tr><tr><td>Arabic</td><td><b>95.19 ± 1.14</b> **</td><td>47.04 ± 8.23</td><td>74.75 ± 11.07 *</td></tr><tr><td>Bashkir</td><td>87.12 ± 2.04 *</td><td>64.30 ± 11.48</td><td><b>89.61 ± 2.40</b> **</td></tr><tr><td>English</td><td><b>99.01 ± 0.22</b> **</td><td>79.63 ± 14.15</td><td>85.14 ± 3.89 *</td></tr><tr><td>French</td><td><b>98.86 ± 0.18</b> **</td><td>93.91 ± 3.57 *</td><td>88.61 ± 9.81</td></tr><tr><td>Hebrew</td><td><b>93.13 ± 1.43</b> **</td><td>40.07 ± 17.04</td><td>69.63 ± 4.61 *</td></tr><tr><td>Portuguese</td><td><b>99.50 ± 0.17</b> **</td><td>85.79 ± 15.20</td><td>96.26 ± 3.61 *</td></tr><tr><td>Sanskrit</td><td><b>95.90 ± 0.48</b> **</td><td>63.92 ± 7.26</td><td>76.12 ± 4.11 *</td></tr><tr><td>Slovak</td><td><b>96.50 ± 0.46</b> **</td><td>74.26 ± 4.15</td><td>74.42 ± 3.83 *</td></tr><tr><td>Slovene</td><td><b>97.48 ± 0.28</b> **</td><td>84.38 ± 7.26</td><td>84.41 ± 5.89 *</td></tr><tr><td>Swahili</td><td><b>90.08 ± 3.70</b> **</td><td>66.97 ± 7.03 *</td><td>63.30 ± 5.61</td></tr><tr><td>Welsh</td><td><b>93.88 ± 1.64</b> **</td><td>79.10 ± 4.40</td><td>85.76 ± 4.72 *</td></tr><tr><td>Zulu</td><td><b>92.83 ± 1.21</b> **</td><td>84.28 ± 3.62 *</td><td>73.67 ± 6.59</td></tr></tbody></table>

<table><caption>**Table A9** Hit rate at 10 (in %, mean ± std.) at the word level for the retrieval models. **: highest average performance; *: second highest average performance.</caption><thead><tr><th>Sigmorphon</th><th>Language</th><th>CNN+ANNr</th><th>CNN+3CosMul</th><th>CNN+ANNc</th></tr></thead><tbody><tr><td rowspan="2">2016</td><td>Georgian</td><td><b>99.84 ± 0.07</b> **</td><td>96.19 ± 1.55</td><td>97.31 ± 1.68 *</td></tr><tr><td>Hungarian</td><td><b>99.21 ± 0.16</b> **</td><td>94.97 ± 2.49</td><td>95.01 ± 3.36 *</td></tr><tr><td rowspan="11">2019</td><td>Turkish</td><td><b>99.67 ± 0.09</b> **</td><td>79.96 ± 2.71 *</td><td>79.22 ± 8.81</td></tr><tr><td>Adyghe</td><td><b>99.95 ± 0.03</b> **</td><td>84.02 ± 6.46</td><td>99.30 ± 0.41 *</td></tr><tr><td>Arabic</td><td><b>97.82 ± 0.53</b> **</td><td>58.14 ± 8.82</td><td>84.30 ± 9.44 *</td></tr><tr><td>Bashkir</td><td><b>98.50 ± 0.50</b> **</td><td>74.58 ± 8.39</td><td>94.27 ± 0.86 *</td></tr><tr><td>English</td><td><b>99.31 ± 0.16</b> **</td><td>83.26 ± 13.18</td><td>89.99 ± 2.95 *</td></tr><tr><td>French</td><td><b>99.25 ± 0.13</b> **</td><td>95.94 ± 2.50 *</td><td>93.62 ± 6.07</td></tr><tr><td>Hebrew</td><td><b>96.96 ± 0.78</b> **</td><td>54.58 ± 15.98</td><td>80.83 ± 3.77 *</td></tr><tr><td>Portuguese</td><td><b>99.74 ± 0.10</b> **</td><td>91.98 ± 10.44</td><td>98.55 ± 1.50 *</td></tr><tr><td>Sanskrit</td><td><b>98.78 ± 0.24</b> **</td><td>75.24 ± 4.99</td><td>86.48 ± 3.03 *</td></tr><tr><td>Slovak</td><td><b>98.77 ± 0.31</b> **</td><td>80.71 ± 3.73</td><td>84.65 ± 3.10 *</td></tr><tr><td>Slovene</td><td><b>99.16 ± 0.19</b> **</td><td>89.77 ± 5.58</td><td>91.35 ± 4.13 *</td></tr><tr><td>Swahili</td><td><b>95.42 ± 2.17</b> **</td><td>75.30 ± 6.64 *</td><td>71.77 ± 5.76</td></tr><tr><td>Welsh</td><td><b>97.56 ± 0.84</b> **</td><td>85.97 ± 3.66</td><td>93.26 ± 2.71 *</td></tr><tr><td>Zulu</td><td><b>95.96 ± 0.64</b> **</td><td>90.70 ± 2.80 *</td><td>84.30 ± 5.04</td></tr></tbody></table>
References

[1] Chollet, F.: On the measure of intelligence. CoRR (2019)

[2] Mitchell, M.: Analogy making as a complex adaptive system. In: SFISSC, pp. 335–360 (2001)

[3] Fahandar, M.A., Hüllermeier, E.: Learning to rank based on analogical reasoning. In: AAAI, pp. 2951–2958 (2018)

[4] Fahandar, M.A., Hüllermeier, E.: Analogical embedding for analogy-based learning to rank. In: IDA. LNCS, vol. 12695, pp. 76–88 (2021)

[5] Hug, N., Prade, H., Richard, G., Serrurier, M.: Analogical proportion-based methods for recommendation - first investigations. FSS **366**, 110–132 (2019)

[6] Mitchell, M.: Abstraction and analogy-making in artificial intelligence. ANYAS **1505**, 79–101 (2021)

[7] Couceiro, M., Hug, N., Prade, H., Richard, G.: Analogy-preserving functions: A way to extend boolean samples. In: IJCAI, pp. 1575–1581 (2017)

[8] Langlais, P., Yvon, F., Zweigenbaum, P.: Improvements in analogical learning: Application to translating multi-terms of the medical domain. In: EACL, pp. 487–495 (2009)

[9] Fam, R., Lepage, Y.: Morphological predictability of unseen words using computational analogy. In: CAW@ICCBR, vol. 1815, pp. 51–60 (2016)

[10] Lim, S., Prade, H., Richard, G.: Solving word analogies: A machine learning perspective. In: ECSQARU, vol. 11726, pp. 238–250 (2019)

[11] Lim, S., Prade, H., Richard, G.: Classifying and completing word analogies by machine learning. IJAR **132**, 1–25 (2021)

[12] Murena, P.-A., Al-Ghossein, M., Dessalles, J.-L., Cornuéjols, A.: Solving analogies on words based on minimal complexity transformation. In: IJCAI, pp. 1848–1854 (2020)

[13] Alsaidi, S., Decker, A., Lay, P., Marquer, E., Murena, P.-A., Couceiro, M.: A neural approach for detecting morphological analogies. In: DSAA, pp. 1–10 (2021)

[14] Alsaidi, S., Decker, A., Lay, P., Marquer, E., Murena, P.-A., Couceiro, M.: On the Transferability of Neural Models of Morphological Analogies. In: AIMLAI@ECML-PKDD, vol. 1524, pp. 76–89 (2021)
[15] Alsaidi, S., Decker, A., Marquer, E., Murena, P., Couceiro, M.: Tackling morphological analogies using deep learning - extended version. CoRR (2021)

[16] Chan, K., Kaszefski-Yaschuk, S.P., Saran, C., Marquer, E., Couceiro, M.: Solving Morphological Analogies Through Generation. In: IARML@IJCAI-ECAI, vol. 3174, pp. 29–39 (2022)

[17] Marquer, E., Alsaidi, S., Decker, A., Murena, P.-A., Couceiro, M.: A Deep Learning Approach to Solving Morphological Analogies. In: ICCBR. LNCS, vol. 13405, pp. 159–174 (2022)

[18] Marquer, E., Murena, P.-A., Couceiro, M.: Transferring Learned Models of Morphological Analogy. In: ATA@ICCBR, Nancy, France (2022)

[19] Sadeghi, F., Zitnick, C.L., Farhadi, A.: Visalogy: Answering visual analogy questions. In: NeurIPS, pp. 1882–1890 (2015)

[20] Peyre, J., Laptev, I., Schmid, C., Sivic, J.: Detecting unseen visual relations using analogies. In: ICCV, pp. 1981–1990 (2019)

[21] Zervakis, G., Vincent, E., Couceiro, M., Schoenauer, M., Marquer, E.: An analogy based approach for solving target sense verification. In: NLPIR, Bangkok, Thailand (2022)

[22] Hofstadter, D., Mitchell, M.: The copycat project: A model of mental fluidity and analogy-making. In: FCCAs, pp. 205–267 (1995). Chap. 5

[23] Murena, P.-A., Dessalles, J.-L., Cornuéjols, A.: A complexity based approach for solving hofstadter’s analogies. In: CAW@ICCBR (2017)

[24] Fam, R., Lepage, Y.: Tools for the production of analogical grids and a resource of n-gram analogical grids in 11 languages. In: LREC, pp. 1060–1066 (2018)

[25] Marquer, E., Couceiro, M., Alsaidi, S., Decker, A.: Siganalogies - Morphological Analogies from Sigmorphon 2016 and 2019

[26] Lepage, Y.: Analogy and formal languages, vol. 53, pp. 180–191 (2001)

[27] Miclet, L., Bayoudh, S., Delhay, A.: Analogical dissimilarity: Definition, algorithms and two experiments in machine learning. JAIR 32, 793–824 (2008)

[28] Antic, C.: Analogical proportions. AMAI 90, 595–644 (2022)

[29] Barbot, N., Miclet, L., Prade, H.: Analogy between concepts. AI 275, 487–539 (2019)
[30] Lepage, Y., Ando, S.: Saussurian analogy: a theoretical account and its application. In: COLING, pp. 717–722 (1996)

[31] Prade, H., Richard, G.: A short introduction to computational trends in analogical reasoning. In: CAAR-CT. SCI, vol. 548, pp. 1–12 (2014)

[32] Lepage, Y.: De l'analogie rendant compte de la commutation en linguistique. Habilitation à diriger des recherches, Université Joseph-Fourier - Grenoble I (2003)

[33] Couceiro, M., Hug, N., Prade, H., Richard, G.: Behavior of Analogical Inference w.r.t. Boolean Functions. In: IJCAI, pp. 2057–2063 (2018)

[34] Couceiro, M., Lehtonen, E.: Galois theory for analogical classifiers (2023)

[35] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: EMNLP, pp. 1532–1543 (2014)

[36] Yvon, F.: Finite-state transducers solving analogies on words. Report GET/ENST&LTCI (2003)

[37] Lepage, Y.: Character-position arithmetic for analogy questions between word forms. In: CAW@ICCBR, vol. 2028, pp. 23–32 (2017)

[38] Wang, L., Lepage, Y.: Vector-to-sequence models for sentence analogies. In: ICACSIS, pp. 441–446 (2020)

[39] Vania, C.: On understanding character-level models for representing morphology. PhD thesis, University of Edinburgh (2020)

[40] Levy, O., Goldberg, Y.: Dependency-based word embeddings. In: Short Papers. ACL, vol. 2, pp. 302–308 (2014)

[41] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT, pp. 4171–4186 (2019)

[42] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: ICLR (2013)

[43] Bojanowski, P., Grave, E., Joulin, A., Mikolo, T.: Enriching word vectors with subword information. ACL 5, 135–146 (2017)

[44] LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. NeurComp 1(4), 541–551 (1989)
[45] Kramer, M.A.: Nonlinear principal component analysis using autoassociative neural networks. AIChE **37**(2), 233–243 (1991)

[46] Hochreiter, S., Schmidhuber, J.: LSTM can solve hard long time lag problems. In: NeurIPS, pp. 473–479 (1996)

[47] Chollet, F.: Character-level recurrent sequence-to-sequence model (2017)

[48] Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. NN **18**(5-6), 602–610 (2005)

[49] Mikolov, T., Yih, W.-T., Zweig, G.: Linguistic regularities in continuous space word representations. In: NAACL, pp. 746–751 (2013)

[50] Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. In: NeurIPS, pp. 12449–12460 (2020)

[51] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)

[52] Cotterell, R., Kirov, C., Sylak-Glassman, J., Yarowsky, D., Eisner, J., Hulden, M.: The sigmorphon 2016 shared task–morphological reinflection. In: SIGMORPHON 2016, pp. 10–22 (2016)

[53] McCarthy, A.D., Vylomova, E., Wu, S., Malaviya, C., Wolf-Sonkin, L., Nicolai, G., Kirov, C., Silfverberg, M., Mielke, S.J., Heinz, J., Cotterell, R., Hulden, M.: The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In: CRPPM@ACL, pp. 229–244 (2019)

[54] Karpinska, M., Li, B., Rogers, A., Drozd, A.: Subcharacter information in japanese embeddings: when is it worth it? In: RELNLP, pp. 28–37 (2018)

[55] Brakel, A.: Boundaries in a morphological grammar of portuguese. WORD **32**(3), 193–212 (1981)

[56] Haykin, S.: Neural Networks: a Comprehensive Foundation, (1994)

[57] Ho, T.K.: Random decision forests. In: ICDAR, vol. 1, pp. 278–282 (1995)

[58] Cortes, C., Vapnik, V.: Support-vector networks. ML **20**(3), 273–297 (1995)

[59] Alsaidi, S., Couceiro, M., Quennelle, S., Burgun, A., Garcelon, N., Coulet,
A.: Exploring analogical inference in healthcare. In: IARML@IJAI-ECAI. CEUR-WP, vol. 3174, pp. 40–50 (2022)

[60] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)

[61] Dozat, T.: Incorporating Nesterov Momentum into Adam. In: ICLR, pp. 1–4 (2016)