# Proceedings of the 3rd Workshop on the Interactions between Analogical Reasoning and Machine Learning (IARML 2024) co-located with the International Joint Conference on Artificial Intelligence (IJCAI 2024)

Miguel Couceiro, Pierre-Alexandre Murena

► To cite this version:

Miguel Couceiro, Pierre-Alexandre Murena. Proceedings of the 3rd Workshop on the Interactions between Analogical Reasoning and Machine Learning (IARML 2024) co-located with the International Joint Conference on Artificial Intelligence (IJCAI 2024). 3rd Workshop on the Interactions between Analogical Reasoning and Machine Learning (IARML 2024) co-located with the International Joint Conference on Artificial Intelligence (IJCAI 2024), Aug 2024, Jeju City, Jeju Island, South Korea. 2025. hal-04996117

HAL Id: hal-04996117

https://hal.science/hal-04996117v1

Submitted on 18 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Proceedings

IJCAI Workshop

“Interactions between Analogical Reasoning and Machine
Learning”

IARML 2024

August 4, 2024

Jeju, South Korea

Editors

Miguel Couceiro (U. Lorraine, CNRS, Loria & IST, U. Lisbon, INESC-ID)

Pierre-Alexandre Murena (Hamburg University of Technology)

https://iarm12024-ijcai.loria.fr
# Preface

Different ML & AI communities have a different views on “analogical reasoning”. However, they all share the idea of transferring knowledge from a source domain to a target domain, by leveraging both the similarities and dissimilarities between the two. While early discussions on analogy trace back to Greek philosophy, in particular Aristotle’s study of proportions, analogical reasoning has been intensively explored by cognitive sciences since the 1960s as a unique and remarkable capability of human cognition. The researches in artificial intelligence (AI) have since focused on either formalizing this ill-defined notion of “analogy” or attempting to replicate this very human capability. A significant shift has been observed lately with the rise of deep learning, opposing two trends. On the one hand, neuro-symbolic techniques pursue the systematization of analogical reasoning, by combining the strong representational power of symbolic AI with the computational efficiency of neural networks. On the other hand, analogy has been used more as a validation tool than as an end goal, helping to assess the reasoning performances of neural networks. This was the case already with the evaluation of the first embedding techniques like word2vec and, recently, in assessing the capabilities of Large Language Models.

The series of workshops *Interactions between Analogical Reasoning and Machine Learning (IARML)*, organized on behalf of the International Joint Conferences on Artificial Intelligence (IJCAI), aims to explore these two trends, to bridge gaps between the ML and the AI communities, and examine the status of analogical reasoning in the light of the current trends in AI and ML. Specifically, it aims to understand how analogical reasoning can be applied in machine learning, and conversely how machine learning techniques can enhance analogical reasoning. Over the last three editions, the workshop has brought together AI researchers at the cross roads of machine learning, cognitive sciences and knowledge representation and reasoning. This third edition of the IARML workshop was hosted by the 33rd International Joint Conference on Artificial Intelligence (IJCAI) and took place on August 4, 2024 in Jeju (South Korea). It welcomed submissions of research papers on all topics at the intersection of analogical reasoning and machine learning. The submissions were subjected to a strict double-blind reviewing process that resulted in the selection of six original contributions and of two plenary keynote talks.

## Plenary talks:

*Learning Semantic Concept Embedding from Language Models*, by Zied Bouraoui

*A Novel View of Analogical Proportion between Formulas*, by Emiliano Lorini

We are truly thankful to the IJCAI workshop chairs for their help in the organization of this event, and to the scientific committee for their reviews and suggestions for improving the accepted contributions.

Miguel Couceiro
Pierre-Alexandre Murena

<footnote>Copyright © 2024 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).</footnote>
**Organising Committee**

Miguel Couceiro (U. Lorraine, CNRS, Loria & IST, U. Lisbon, INESC-ID)

Pierre-Alexandre Murena (Hamburg University of Technology, DE)

**Scientific Committee**

Fadi Badra (Université Sorbonne Paris Nord, LIMICS, FR)

Nelly Barbot (Université de Rennes 1, IRISA, FR)

Tarek R. Besold (DEKRA DIGITAL, Eindhoven University of Technology, NL)

Myriam Bounhas (LARODEC-ISGT, TU, UAE)

Adrien Coulet (Inria Paris, FR)

Sebastien Destercke (CNRS, Université de Technologie de Compiegne, Heudiasyc, FR)

Claire Gardent (University of Lorraine, CNRS, LORIA, FR)

Eyke Hullermeier (University of Munich, DE)

Mehdi Kaytoue (Infologic, FR)

David B. Leake (Indiana University, USA)

Yves Lepage (Waseda University, JA)

Jean Lieber (University of Lorraine, CNRS, LORIA, FR)

Esteban Marquer (University of Lorraine, CNRS, LORIA, FR)

Laurent Miclet (Université de Rennes, FR)

Pierre Monnin (Orange, FR)

Amedeo Napoli (University of Lorraine, CNRS, LORIA, FR)

Henri Prade (CNRS, Université Paul Sabatier, IRIT, FR)

Irina Rabkina (OXY Occidental College, USA)

Steven Schockaert (Cardiff University, IR)
Table of Contents

<table><tr><td>Preface</td><td>II</td></tr><tr><td>Accepted papers</td><td>1</td></tr><tr><td>Transformer-based hierarchical attention models for solving analogy puzzles between longer, lexically richer and semantically more diverse sentences</td><td>3</td></tr><tr><td>Benming Yan, Haotong Wang, Liyan Wang, Yifei Zhou and Yves Lepage</td><td>3</td></tr><tr><td>Learning Analogies between Classes to Create Counterfactual Explanations</td><td>19</td></tr><tr><td>Xiaomeng Ye, David Leake, Yu Wang, Ziwei Zhao, David Crandall</td><td>19</td></tr><tr><td>Enhancing Analogical Reasoning in the Abstraction and Reasoning Corpus via Model-Based RL</td><td>34</td></tr><tr><td>Jihwan Lee, Woochang Sim, Sejin Kim, Sundong Kim</td><td>34</td></tr><tr><td>Towards a unified framework of numerical analogies: Open questions and perspectives</td><td>47</td></tr><tr><td>Yves Lepage, Miguel Couceiro</td><td>47</td></tr><tr><td>Testing proportional sentence analogies on SATS: from vector offsets to conditional generation</td><td>60</td></tr><tr><td>Yves Blain-Montesano, Philippe Langlais</td><td>60</td></tr><tr><td>Probing Large Language Models to Perform Analogical Transformation</td><td>75</td></tr><tr><td>François Olivier, Miguel Couceiro, Zied Bouraoui</td><td>75</td></tr></table>
Accepted papers
# Transformer-based hierarchical attention models for solving analogy puzzles between longer, lexically richer and semantically more diverse sentences

Benming YAN¹, Haotong WANG¹, Liyan WANG¹, Yifei ZHOU¹ and Yves LEPAGE¹,*

¹Waseda University, 2-7 Hibikino, Wakamatsu-ku, Kitakyushu-shi, 808-0123, Japan

## Abstract

We address the task of solving analogy puzzles between sentences. Various possible neural architectures are envisageable for this task. However the scarcity of datasets of sentence analogies to assess the efficiency of such proposals is a problem. Our contribution is thus twofold. We firstly propose a method to build a resource of sentence analogies that uses sentence templates and leverages word analogy datasets. Back translation is used to produce paraphrases. Our method could produce a resource in English that is shown to be richer along three dimensions compared with a previously publicly available resource: our sentences are longer, their vocabulary is richer and their semantics is more varied. Secondly, to solve analogy puzzles between sentences, we introduce an architecture that makes use of a hierarchical attention mechanism in the Transformer model in conformity to properties of analogy puzzles. Our experiments show superior results compared to a baseline and other possible architectures on the newly created dataset.

## Keywords

Sentence analogy puzzles, Datasets, Transformer, Attention

## 1. Introduction

With the advent of vector representations of language pieces in natural language processing (NLP), analogies have made a come-back. The over-repeated male : female :: king : queen has now long been known to every practitioner of NLP [1]. Solving analogy puzzles between words, like male : female :: king : x ⇒ x = queen, has been made possible, on the assumption that words in analogy constitute a parallelogram in embedding spaces, a hypothesis supported by the theoretical result that negative sampling skip-gram models are equivalent to a factorisation of a PPMI matrix of co-occurrences [2]. In order to assess the quality of various word embedding models, attempts at building various word analogy datasets have been made [1, 3, 4, 5].

This paper addresses the problem of solving sentence analogy puzzles, hence beyond

IARML@IJCAI'2024: Workshop on the Interactions between Analogical Reasoning and Machine Learning, at IJCAI'2024, August, 2024, Jeju, South Korea

*Corresponding author.

nathan_yan@toki.waseda.jp (B. YAN); wanghaotong0925@toki.waseda.jp (H. WANG); wangliyan0905@toki.waseda.jp (L. WANG); yifei.zhou@ruri.waseda.jp (Y. ZHOU); yves.lepage@waseda.jp.com (Y. LEPAGE)

http://lepage-lab.ips.waseda.ac.jp/ (Y. LEPAGE)

0000-0002-3059-4271 (Y. LEPAGE)

<figure><img src="image_2.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_3.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)
word analogy puzzles. For example: I talk to her . : He talked to her . :: I go to the park . : x ⇒ x = He went to the park . is an example of solving a simple sentence analogy puzzle that relies on elementary word analogy puzzles.

Analogies have properties that have been agreed upon since Antiquity and that have been reiterated in more recent works [6, 7, 8]. In particular, equivalent forms are in the number of eight:

<figure><img src="image_4.png" /></figure>

From the viewpoint of solving analogy puzzles, by letting the unknown *D* at the same position, there is only one equivalent form, which was called *permutando* in Latin, or *permutation of the means*.<fnref n="1" /> But in addition, it should be noticed that *A* does not play the same role as the means *B* and *C* relatively to the unknown *D*. As a matter of fact, in an analogy, *A* and *D* can be permuted thanks to the property of *permutation of the extremes*. The above two properties are indicated by frames in the list of eight equivalent forms above.

## 2. Previous dataset and newly created dataset

Datasets of sentence analogies are almost inexistant. The ACL pages relate to word analogies, not to sentence analogies<fnref n="2" />. The Penn Discourse Treebank, which annotates sentence pairs for discourse relations, is not *stricto sensu* a sentence analogy dataset [9] because the exchange of the means is not verified in general [10, 11]. The same holds for the ANALOGICAL dataset that is also a set of pairs of sentences, not strictly speaking a set of sentence analogies [12].

### 2.1. Previous dataset

The only known sentence analogy dataset has been introduced in [13].<fnref n="3" /> It consists of 5,607 semantico-formal sentence analogies in English (see Table 1), i.e., analogies which are partly formal (the sentences are almost fixed patterns) and partly semantic (word analogies are involved). The second example in Table 1 is an example of such analogies: it interleaves two syntactic patterns *I do not [like this N / feel like Ving]* . with a semantic word analogy *song : singing :: game : playing*. This dataset has been used as a benchmark to assess various methods for solving analogy puzzles between sentences in [15], including the use of compact language models like SBERT.

However, a thorough investigation of this dataset reveals flaws. Nearly half of the analogies fall within the category of formal analogy (first row in Table 1), while true semantico-formal analogies constitute a significantly smaller proportion (second and third row). Additionally, it contains many unacceptable sentence analogies because the analogies have been produced automatically by solving word analogy puzzles blindly. E.g., high frequent words, like, e.g., *girlfriend* (2489 occurrences) or *boyfriend* (1674 occurrences) were found in analogy with

<footnote n="1">¹In an analogy *A : B :: C : D*, *B* and *C* are the means and *A* and *D* are the extremes.</footnote>

<footnote n="2">²https://aclweb.org/aclwiki/Analogy_(State_of_the DI</footnote>

<footnote n="3">³The dataset has been extended to French in [14].</footnote>
<table><caption>**Table 1**</caption><thead><tr><th></th><th colspan="4">Example analogy</th><th># of such analogies</th></tr></thead><tbody><tr><td>Hurry up, or you 'll miss the bus.</td><td>Hurry up, or you will miss the bus.</td><td>II'll be late.</td><td>Hurry up, or you 'll be late.</td><td>Hurry up, or you will be late.</td><td>2,362</td></tr><tr><td>I do not like this song.</td><td>I do not feel like singing.</td><td>II'll be late.</td><td>I do not like this game.</td><td>I do not feel like playing.</td><td>303</td></tr><tr><td>I had a dreadful dream last night.</td><td>I had a horrible dream last night.</td><td>II'll be late.</td><td>I had a strange dream last night.</td><td>I had a weird dream last night.</td><td>303</td></tr><tr><td>I do not need a nap.</td><td>I do not have a beard.</td><td>II'll be late.</td><td>I do not need a boyfriend.</td><td>I do not have a girlfriend.</td><td>2,942</td></tr><tr><td>Total</td><td></td><td></td><td></td><td></td><td>5,607</td></tr></tbody></table>

Classification of analogies found in the previous analogy dataset. The last number is the number of analogies of this type. The first row is purely formal analogies; the second and third rows are for analogies that might be considered acceptable semantico-formal analogies (possibly after some slight change); the last one is for unacceptable analogies.

unrelated words like, i.e., nap and beard. This is caused by automatically solving analogy puzzles in the word embedding model used, and then selecting the closest sentence from Tatoeba to ensure correctness, but such analogies are unacceptable for the human eye. The last row in Table 1 is such an example

A statistical analysis of word and sentence lengths in the dataset shows that, on average, sentences consist of 7.1 words and span 26 characters. This points at the relative simplicity of the sentences in the dataset, its limited vocabulary and supposedly a limited semantic variety.

## 2.2. Newly created dataset

We address the flaws in the previous dataset and produce a sentence analogy dataset characterised by longer sentences, richer vocabulary and richer semantic content. For that, we leverage the Google Analogy Test Set [1]<fnref n="4" /> and the Tatoeba corpus<fnref n="5" />.

The Google Analogy Test Set encompasses various word analogy types of relation, from world-knowledge (capital : country) and semantic relations (male : female) to grammatical ones (adjective : opposite), as illustrated in Table 2. For each type of relation, the dataset offers a list of analogies, i.e., a pair of ratios. We remember the list of all word pairs for each type of relation.

We retrieve any sentence from the Tatoeba corpus that contains a word in two of the types of analogies capitals-common-countries and family of the Google Analogy Test Set (sentence A in Figure 1). We filter out too short sentences. We apply back-translation to all retrieved sentences, i.e., we translate into a language other than English (here we translate into Chinese) and keep the sentences that are different from the original ones. The purpose of back-translation is to introduce textual or syntactic variation while maintaining semantic similarity. Recent research

<footnote n="4">⁴http://download.tensorflow.org/data/questions-words.txt</footnote>

<footnote n="5">⁵https://tatoeba.org/en/</footnote>
<table><caption>**Table 2**</caption><thead><tr><th>Type of relation</th><th></th><th colspan="4">Example analogy</th></tr></thead><tbody><tr><td rowspan="2">capital-common-countries</td><td>Athens</td><td>Greece</td><td>Bangkok</td><td>Thailand</td></tr><tr><td>Beijing</td><td>China</td><td>London</td><td>England</td></tr><tr><td rowspan="2">currency</td><td>Algeria</td><td>dinar</td><td>Europe</td><td>euro</td></tr><tr><td>Canada</td><td>dollar</td><td>Thailand</td><td>baht</td></tr><tr><td rowspan="2">city-in-state</td><td>Chicago</td><td>Illinois</td><td>Houston</td><td>Texas</td></tr><tr><td>Huntsville</td><td>Alabama</td><td>Boston</td><td>Massachusetts</td></tr><tr><td rowspan="2">family</td><td>grandfather</td><td>grandmother</td><td>prince</td><td>princess</td></tr><tr><td>husband</td><td>wife</td><td>stepfather</td><td>stepmother</td></tr><tr><td rowspan="2">gram2-opposite</td><td>competitive</td><td>uncompetitive</td><td>informed</td><td>uninformed</td></tr><tr><td>known</td><td>unknown</td><td>tasteful</td><td>distasteful</td></tr></tbody></table>

Some example analogies from the Google analogy test set with the type of relation indicated on the left

<figure><img src="image_5.png" /><figcaption>**Figure 1:** Production of a sentence analogy by relying on word ratios. We start with sentence A that contains words from a given list of word ratios with the same type of relation. We get sentence C by back-translation, and sentences B and D by replacing words using word ratios from the given list.</figcaption></figure>

points at the fact that understanding syntactic structure might be crucial in understanding sentence analogies [16]. This provides us with pairs of sentences that we consider as sentence ratios (sentences A and C in Figure 1). We then revisit the set of word pairs in the same type of relation and replace the words in the original and back-translated sentences with corresponding words from other word pairs from the same type of relation. If this is not possible, we discard the sentence ratio. This creates two new sentences (sentences B and D in Figure 1) that form an analogy with the two previous sentences.

The entire process described above and illustrated in Figure 1 allows us to successfully expand sentence ratios into sentence analogies. Table 3 gives some examples of analogies. They show that the semantic type of relation between the sentences in analogy remains consistent. Simultaneously, the process guarantees the formation of correct analogies, because one or two words only are exchanged in one direction, while back-translation produces variations in the other direction.
<table><caption>**Table 3**</caption><tr><td>She cannot bear moonlight and she cannot bear darkness.</td><td>He cannot bear moonlight and he cannot bear darkness.</td><td>::</td><td>She couldn't bear the moonlight or the darkness.</td><td>:</td><td>He couldn't bear the moonlight or the darkness.</td></tr><tr><td>Your son is very bright, much more than his age.</td><td>Your daughter is very bright, much more than her age.</td><td>::</td><td>Your son is very intelligent, much older than his age.</td><td>:</td><td>Your daughter is very intelligent, much older than her age.</td></tr><tr><td>He plays at aerobics just to please his girlfriend.</td><td>She plays at aerobics just to please her boyfriend.</td><td>::</td><td>He only plays aerobics to please his girlfriend.</td><td>:</td><td>She only plays aerobics to please her boyfriend.</td></tr></table>

Some example analogies in the newly created sentence analogy dataset

<table><caption>**Table 4**</caption><thead><tr><th></th><th colspan="2">Sentence analogy dataset</th><th></th></tr><tr><th></th><th>Previous one [13]</th><th>Newly created</th><th>Increase</th></tr></thead><tbody><tr><td>Number of analogies</td><td>5,607</td><td>22,161</td><td>+295%</td></tr><tr><td>Sentence length</td><td></td><td></td><td></td></tr><tr><td>Number of words/sentence</td><td>7.1</td><td>10.1</td><td>+42%</td></tr><tr><td>Number of characters/sentence</td><td>26.0</td><td>45.4</td><td>+75%</td></tr><tr><td>Lexical richness</td><td></td><td></td><td></td></tr><tr><td>Vocabulary size</td><td>1,135</td><td>8,159</td><td>+619%</td></tr><tr><td>Semantic diversity</td><td></td><td></td><td></td></tr><tr><td>Std. dev. of vector dispersion</td><td></td><td></td><td></td></tr><tr><td>all-mpnet-base-v2</td><td>0.908</td><td>0.949</td><td>+4%</td></tr><tr><td>multi-qa-mpnet-base-dot-v1</td><td>4.094</td><td>5.228</td><td>+28%</td></tr><tr><td>all-distilroberta-v1</td><td>0.887</td><td>0.945</td><td>+7%</td></tr><tr><td>all-MiniLM-L12-v2</td><td>0.924</td><td>0.943</td><td>+7%</td></tr></tbody></table>

Comparison between the previous analogy dataset and the newly created one along several criteria. All criteria show an increase in favour of the newly created dataset.

## 2.3. Comparison of the two sentence analogy datasets

We compare the previous sentence analogy dataset and the newly created one, using several criteria. Table 4 details the numbers for this comparison.

**Number of analogies.** The newly created dataset contains more analogies with an increase by 295%, i.e., approximately four times more analogies.

**Sentence length.** The new dataset achieves an increase in average sentence length over the previous dataset, with +42% in average number of words per sentence or +75% in average number of characters per sentence, i.e., the sentences are longer by around a half.

**Lexical richness.** The vocabulary size shows a substantial increase in the newly created dataset: more than 7 times more words are used. This indicates a higher lexical richness.

**Semantic diversity.** To measure semantic diversity, we convert sentences into vector rep-
representations and, similarly to what is done in [17], we compute the standard deviations on each dimensions of all vectors from both datasets in four different embedding spaces [18]. We aggregate these standard deviation values by taking their norms (which is proportional to the quadratic mean). The greater the norm, the more dispersed the vectors in the sentence embedding spaces, hence, supposedly, the wider the semantic variety. The last rows in Table 4 give the results obtained by applying the same procedure to four different sentences representation models. They show an increase in semantic variety in all used embedding spaces.

To summarise the content of Table 4, the new dataset that we constructed demonstrates improvements along all mentioned criteria: in the newly created dataset, there are more analogies and the sentences are longer, lexically richer and semantically more diverse.

# 3. Previous models and proposed models

## 3.1. Previous models

The first sentence analogy dataset presented above has been built by merging two techniques: a formal approach to solve sequence analogy puzzles and a semantic approach to solve word analogy puzzles. New sentences have been generated by decomposing sentence analogies into word analogies [1] along edit distance traces between sentences [19]. This means that sentences were interpreted as sequences of vector representations of words.

**Vec2Seq.** It is also classical to represent a sentence by just one vector, the sum of the vector representations of its words. It is then possible to apply the classical arithmetic interpretation of analogy on such vector representations of sentences. I.e., the sentence solution *D* of an analogy between sentences *A*: *B* :: *C*: *D* is computed as the sentence corresponding to the vector $\vec{D} = \vec{B} - \vec{A} + \vec{C}$. This induces the necessity of a decoder that decodes the vector representation of $\vec{D}$ into a word sequence that makes sentence *D*. In the Vec2Seq model proposed in [15], the architecture chosen for such a decoder is that of a recurrent network, namely a Long Short-Term Memory model. Although ameliorations can be given, as is done in [20], we will use the Vec2Seq model as our baseline.

## 3.2. Proposed models

On the contrary to the Vec2Seq model which concentrates learning on the decoder side, we propose several models that focus on the encoder side. As our work is more recent, we rely on Transformers rather than on recurrent networks. We aim to leverage the attention mechanism on the three given sentences in an analogy puzzle, i.e., sentences *A*, *B* and *C*, in a manner that better exploits the properties in an analogy puzzle. We introduce several models that incorporate step-by-step the two properties seen in the Introduction: the permutation of the extremes and the permutation of the means.

**A.B.C-to-D Transformer.** A first naïve model to address the task of solving sentence analogy puzzles is to construe it as a sequence-to-sequence task where the input sequence is just made of the concatenation of the three given sentences *A*, *B* and *C*. The output sequence should be
<figure><img src="image_6.png" /><figcaption>**Figure 2:** Architecture of the A.B.C-to-D Transformer model. It is a naïve use of the basic encoder-decoder Transformer model for solving analogy puzzles between sentences. The input of the encoder is the concatenation of *A*, *B* and *C*. This figure is just a slight modification of the figure in [21].</figcaption></figure>

the fourth sentence *D*. A basic encoder-decoder Transformer model can be used for that as illustrated in Figure 2. This serves as a second, more robust, baseline in addition to the Vec2Seq model.

**(B.C)xA-to-D Transformer.** In an analogy puzzle, *A* plays a different role from *B* and *C*. This comes from the fact that, in an analogy, *A* and *D* can be exchanged by the permutation of the extremes. It pleads for a separation of *A* from *B* and *C* when solving an analogy puzzle.

Consequently, the second model we propose encodes *B* and *C* separately from *A*. The encoder side features two layers. The first layer comprises two self-attention blocks, one for the concatenation of sentences *B* and *C*, and another one for sentence *A*. The second layer consists of a cross-attention block. It takes the outputs of the two self-attention blocks from the first layer as inputs. The explanation for this cross-attention block is as follows.

As was the case in the basic encoder-decoder model where the attention block for *D* with respect to *A*, *B* and *C* was a cross-attention block, here too, on the encoder side, the attention block for *A* with respect to *B* and *C* should be of the same type, i.e., a cross-attention block. As a result, on Figure 3, two cross-attention blocks are to be seen: one for *A* with respect to *B.C* and another one for *D* with respect to *B.C* and *A* (notice the same blue arrows for Key (*K*)
<figure><img src="image_7.png" /><figcaption>**Figure 3:** Architecture of the (B.C)xA-to-D Transformer model</figcaption></figure>

and Value (V) and the same red arrows for Query (Q)).

**(BxC)xA-to-D Transformer.** Indeed, nothing prevents from inputting *C.B* rather than *B.C* in the previous (B.C)xA-to-D Transformer model because of the property of permutation of the means. We propose a model that copes with this property, i.e., *B* and *C* play the same role.

In this new model, as in the previous one, the encoder side features two layers. However, different from the previous model, the first layer of attention blocks provides two cross-attention blocks and one self-attention block, as illustrated in Figure 4. The first-layer cross-attention blocks each receive sentence *B* and sentence *C* as inputs and provide the required Key (*K*) and Value (*V*) for computing cross-attention one with another. Because of this double cross-attention mechanism, the positions of sentences *B* and *C* can be freely exchanged in conformity with the permutation of the means. Subsequently, the outputs of these two cross-attention blocks are combined (⊕ in Figure 4) and provide the Key (*K*) and Value (*V*) for the second-layer cross-attention block, where the Query (*Q*) comes from the self-attention block on input *A* in the first layer.

**Summary on Transformer models (B.C)xA-to-D and (BxC)xA-to-D.** In both models, the incorporation of cross-attention blocks, along with their respective *Q*, *K*, and *V*, conforms to the standard decoding side of the basic encoder-decoder Transformer model. In the decoder of
<figure><img src="image_8.png" /><figcaption>**Figure 4:** Architecture of the (BxC)xA-to-D Transformer model</figcaption></figure>

a Transformer model, during training, the input encompasses both the target text itself and the output of the encoder. When performing cross-attention operations in the decoder, the *K* and *V* in the input data of the cross-attention block are derived from the encoder's output, while *Q* originates from the result of self-attention on the target text, as depicted in the Figure 2. I.e., the knowledge from the encoder guides the generation in the decoder. Thus, in both models, the second layer of the cross-attention block aims to enable sentence *A* to glean valuable knowledge from sentences *B* and *C*, so as to facilitate the generation of sentence *D* in the decoder.

# 4. Experiments

## 4.1. Data and models used

The two datasets described in Section 2, for which Table 4 provided statistics, are used in the experiments. They are cut into training, validation, and test sets with the classical proportions of 80%, 10%, and 10%, respectively. The tested models are the previously four described models with only one stack of Transformers. They are trained from scratch with early stopping with a batch size of 128 analogies. The training task is language modeling by next word completion.
<table><caption>**Table 5**</caption><thead><tr><th>Dataset</th><th>Size</th><th>Model</th><th>BLEU</th><th>Accuracy (%)</th><th>Levenshtein<br>in words</th><th>Levenshtein<br>in chars</th></tr></thead><tbody><tr><td rowspan="3">Previous one</td><td rowspan="3">5,607</td><td>Recurrent model</td><td></td><td></td><td></td><td></td></tr><tr><td>Vec2Seq</td><td><b>90.02±1.89</b></td><td><b>82.2</b></td><td><b>0.4</b></td><td><b>1.4</b></td></tr><tr><td>Transformer models</td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">1/2 Previous one<br>+ 1/2 Newly cre-<br>ated</td><td rowspan="3">5,607</td><td>A.B.C-to-D</td><td>87.19±1.89</td><td>76.1</td><td>0.5</td><td>1.8</td></tr><tr><td>(B.C)xA-to-D</td><td>82.64±2.49</td><td>70.1</td><td>0.7</td><td>2.6</td></tr><tr><td>(BxC)xA-to-D</td><td>77.91±2.46</td><td>63.5</td><td>0.7</td><td>2.8</td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">5,607</td><td>Recurrent model</td><td></td><td></td><td></td><td></td></tr><tr><td>Vec2Seq</td><td>56.58±3.02</td><td>26.7</td><td>2.2</td><td>10.4</td></tr><tr><td>Transformer models</td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">22,161</td><td>A.B.C-to-D</td><td>70.91±3.38</td><td>59.9</td><td>1.7</td><td>7.2</td></tr><tr><td>(B.C)xA-to-D</td><td>60.77±3.49</td><td>47.8</td><td>2.4</td><td>9.5</td></tr><tr><td>(BxC)xA-to-D</td><td><b>75.84±2.63</b></td><td><b>59.4</b></td><td><b>1.1</b></td><td><b>5.2</b></td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">5,607</td><td>Recurrent model</td><td></td><td></td><td></td><td></td></tr><tr><td>Vec2Seq</td><td>52.89±2.72</td><td>12.3</td><td>3.0</td><td>14.1</td></tr><tr><td>Transformer models</td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">22,161</td><td>A.B.C-to-D</td><td>77.92±2.60</td><td>64.7</td><td>1.4</td><td>6.5</td></tr><tr><td>(B.C)xA-to-D</td><td>64.04±3.64</td><td>59.0</td><td>2.8</td><td>11.0</td></tr><tr><td>(BxC)xA-to-D</td><td><b>85.65±1.92</b></td><td><b>71.3</b></td><td><b>0.8</b></td><td><b>3.9</b></td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">22,161</td><td>Recurrent model</td><td></td><td></td><td></td><td></td></tr><tr><td>Vec2Seq</td><td>61.17±1.41</td><td>27.6</td><td>2.5</td><td>11.8</td></tr><tr><td>Transformer models</td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">Newly created</td><td rowspan="3">22,161</td><td>A.B.C-to-D</td><td>93.76±0.64</td><td>83.9</td><td>0.4</td><td>1.9</td></tr><tr><td>(B.C)xA-to-D</td><td>80.37±1.25</td><td>67.2</td><td>1.3</td><td>5.6</td></tr><tr><td>(BxC)xA-to-D</td><td><b>95.27±0.58</b></td><td><b>86.8</b></td><td><b>0.2</b></td><td><b>1.4</b></td></tr></tbody></table>

Results of the experiments using the four models on the two datasets and more. The (BxC)xA-to-D Transformer model performs worse on the previous sentence analogy dataset. It performs best on the newly created dataset with longer, lexically richer and semantically more diverse sentences under two data size conditions and on a mixture of data from the two datasets.

## 4.2. Experiments and results

**Comparing performance on the two datasets.** The first round of experiments consists in comparing the performance of each model on the two datasets. The results are shown on the first and fourth groups of rows in Table 5 (Previous one (5,607) and Newly created (9,579)). A surprising phenomenon is observed: the trend over the four models are opposite on the two datasets. The recurrent network model Vec2Seq is the best on the previous dataset, but it achieves an accuracy of less than 30% on the newly created dataset.<fnref n="6" /> This suggests that this model, which relies on a simple aggregation of word vectors for sentence representation and employs a fixed formula for deriving vector representations of the solution sentence, faces difficulties in dealing with longer, lexically richer and semantically more diverse sentences.

The (BxC)xA-to-D Transformer model exhibits the poorest performance on the previous

<footnote n="6">⁶Accuracy is the percentage of output sentences that are exactly the same as the reference sentences, hence possibly a clue for overfitting.</footnote>
analogy dataset. Nevertheless, its performance is not so low — a little bit less than 80 in BLEU. We find it hard to explain why it struggles to capture relevant information for sentence generation from the shorter and structurally simpler sentences within this dataset. We guess that this might come from the dataset itself.

Because half of the analogies in that dataset are formal ones and many are semantically hard to accept, the scores for the recurrent neural model might be explained by overfitting. This can be supported by the high accuracy (more than 80%, i.e., 4 in 5) on this dataset, and the low accuracy scores on the datasets involving newly created data (below 30%).

The decrease in scores for our three proposed Transformer models on the previous dataset might be explained by an inability to handle unacceptable analogies. This is somewhat supported by the results on the newly created dataset, where the last Transformer model, which integrates the properties of permutation of the extremes and permutation of the means, and should thus have higher competency in identifying semantically corresponding parts in sentences thanks to cross-attention, surpasses all other models.

We also observe that, against our expectations, the decrease on the previous dataset and the increase on the newly created dataset are not gradual from the A.B.C-to-D model to the (BxC)xA-to-D model, with the (B.C)xA-to-D model exhibiting worse scores.

In order to test the hypothesis about the datasets themselves impacting the results, and in order to have a fairer comparison, we perform another round of experiments.

**Comparing performance on equal size datasets.** To study performance independently of the size of the data and neutralise the effect of the different sizes of the two datasets, we make the amount of training data equal in both conditions: we reduce the number of sentence analogies in the newly created dataset (originally 22,161) to match the quantity in the previous dataset (5,607). This is the third group of rows in Table 5 (Newly created (5,607)). Even on a reduced amount of data from the newly created dataset, the (BxC)xA-to-D Transformer model performs the best. We observe again that the ordering of the (B.C)xA-to-D and A.B.C-to-D models is contrary to our expectations.

To inspect the influence of the type of data contained in the two datasets, we construct yet another dataset that equally mixes data from both datasets. We build it to reach a number of 5,607 sentence analogies, i.e., the size of the previous sentence analogy dataset. Half of the analogies in this new dataset are randomly sampled from each dataset. We expect results in-between the previously obtained results on equal size datasets. The second group of rows in Table 5 (1/2 Previous one + 1/2 Newly created (5,607)) does not follow our expectations: in BLEU, they are worse than on each dataset with equal size, except for the recurrent network model which is in-between. We are led to conclude that the data in the previous dataset have a strong impact on both types of models: a positive one on the recurrent network model and a negative one on the Transformer models. The ordering of Transformer models is the same as on the third and fourth groups of rows: (B.C)xA-to-D < A.B.C-to-D < (BxC)xA-to-D.

**Behaviour of the (B.C)xA-to-D Transformer model** As said above, the results of the (B.C)xA-to-D Transformer model are not in between the two other Transformer models; they are worse. In all experiments involving the newly created dataset, the Levenshtein distance in words or in characters is surprisingly much larger than in other Transformer models (and often
comparable to that of the recurrent model). In particular, an important decrease is observed
on all metrics on the newly created dataset with the same size as the previous dataset. This
questions the progressiveness of the integration of the two properties of analogy, and this might
even question whether the way we integrate the permutation of the extremes is correct, calling
for a better way of reflecting this property with the help of attention.

5. Conclusion

We addressed the problem of solving analogy puzzles between sentences, a task for which the scarcity of available datasets is problematic. We described an existing previous dataset, and, as a first contribution, created another dataset that corrects some of the weaknesses of the previous one. The newly created dataset contains more analogies and features longer sentences with a richer vocabulary and wider semantic variety in comparison to the previous dataset. The two datasets basically combine syntactic patterns with word analogies. The first dataset had the drawback of containing many formal analogies and semantically not so reliable analogies because it freely exploited word analogies in word embedding spaces. The newly created dataset introduces syntactic variations by relying on back translation to create paraphrases. In addition, by relying on the Google Analogy Test Set, the word analogies used are more constrained and therefore more reliable.

The second contribution of this paper is with models to solve sentence analogy puzzles. We introduced hierarchical attention models based on the encoder-decoder Transformer architecture to solve sentence analogy puzzles, i.e., produce a fourth sentence that is in analogical relationship with three given sentences. We introduced two properties of analogy into our models, namely the permutation of the extremes and the permutation of the means, by making use of cross-attention. The resulting models adopt a hierarchical architecture. Our experiments showed that our proposed architectures, especially the last one, can address the challenges posed by analogies which involve longer, lexically richer and semantically more diverse sentences.

Still, our results call for further inspection. As for datasets, it would be interesting to examine
datasets that would contain only formal analogies or that would contain only not so acceptable
analogies. Experiments on these datasets could allow us to get more insight at why models are
confused or whether they are more prone to overfitting on such datasets.

As for models, we introduced properties of analogy step-by-step, but our results did not
exhibit a gradual increase in performance. The ordering of results for the (B.C)xA-to-D and
A.B.C-to-D models is not the one we expected. This might call for the exploration of other
models, e.g., one of the type ((A.B)x(A.C))-to-D.

6. Acknowledgments

This work was supported in part by a grant from the Japanese Society for the Promotion of Science, Kiban C, n° 21K12038, entitled: "Theoretically founded algorithms for the automatic production of analogy tests in NLP".
## References

[1] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, in: Proceedings of the International Conference on Learning Representations (ICLR), volume CoRR abs/1301.3781, 2013. URL: https://arxiv.org/pdf/1301.3781.pdf.

[2] O. Levy, Y. Goldberg, Neural word embedding as implicit matrix factorization, in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Weinberger (Eds.), Proceedings of Advances in Neural Information Processing Systems 27 (NIPS 2014), Curran Associates, Inc., 2014, pp. 2177-2185. URL: https://papers.nips.cc/paper_files/paper/2014.

[3] A. Drozd, A. Gladkova, S. Matsuoka, Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen, in: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, The COLING 2016 Organizing Committee, 2016, pp. 3519-3530. URL: http://www.aclweb.org/anthology/C16-1332.

[4] A. Gladkova, A. Drozd, S. Matsuoka, Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't, in: Proceedings of the NAACL-HLT SRW, San Diego, California, 2016, pp. 47-54. doi: https://www.aclweb.org/anthology/N/N16/N16-2002.pdf.

[5] M. Abdou, A. Kulmizev, V. Ravishankar, MGAD: Multilingual generation of analogy datasets, in: N. C. C. chair), K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Paris, France, 2018.

[6] Y. Lepage, Languages of analogical strings, in: Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000), volume 1, Saarbrücken, 2000, pp. 488-494. URL: https://aclanthology.org/C00-1071.

[7] N. Stroppa, Définitions et caractérisation de modèles à base d'analogies pour l'apprentissage automatique des langues naturelles, Thèse de doctorat, École nationale supérieure des télécommunications, 2005. URL: https://hal.archives-ouvertes.fr/tel-00145147/.

[8] L. Miclet, H. Prade, Handling analogical proportions in classical logic and fuzzy logics settings, in: C. Sossai, G. Chemello (Eds.), Symbolic and Quantitative Approaches to Reasoning with Uncertainty, Springer Berlin Heidelberg, Berlin, Heidelberg, 2009, pp. 638-650.

[9] R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, B. Webber, The Penn Discourse TreeBank 2.0., in: N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, D. Tapias (Eds.), Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), European Language Resources Association (ELRA), Marrakech, Morocco, 2008. URL: http://www.lrec-conf.org/proceedings/lrec2008/pdf/754_paper.pdf.

[10] S. Afantenos, T. Kunze, S. Lim, H. Prade, G. Richard, Analogies between sentences: Theoretical aspects - preliminary experiments, in: J. Vejnarová, N. Wilson (Eds.), Symbolic and Quantitative Approaches to Reasoning with Uncertainty, Springer International Publishing, Cham, 2021, pp. 3-18.

[11] S. Afantenos, S. Lim, H. Prade, G. Richard, Theoretical study and empirical investigation
of sentence analogies, in: IJCAI-ECAI Workshop: Workshop on the Interactions between
Analogical Reasoning and Machine Learning (IAMRL 2022)@ IJCAI-ECAI 2022, volume
3174, CEUR-WS. org, 2022, pp. 15–28

[12] T. Wijesiriwardene, R. Wickramarachchi, B. Gajera, S. Gowaikar, C. Gupta, A. Chadha, A. N. Reganti, A. Sheth, A. Das, ANALOGICAL - a novel benchmark for long text analogy evaluation in large language models, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 3534–3549. URL: https://aclanthology.org/2023.findings-acl.218. doi:10.18653/v1/2023.findings-acl.218.

[13] Y. Lepage, Analogies between short sentences: A semantico-formal approach, in: Human Language Technology. Challenges for Computer Science and Linguistics: 9th Language and Technology Conference, LTC 2019, Poznan, Poland, May 17–19, 2019, Revised Selected Papers, Springer-Verlag, Berlin, Heidelberg, 2019, pp. 163–179. URL: https://doi.org/10.1007/978-3-031-05328-3_11. doi:10.1007/978-3-031-05328-3_11.

[14] Y. Lepage, Human Language Technology – Challenges for Computer Science and Linguistics, number 13212 in Lecture Notes in Artificial Intelligence, Springer Nature Switzerland, 2022, pp. 163–179. URL: https://link.springer.com/content/pdf/10.1007%2F978-3-031-05328-3_11. doi:https://doi.org/10.1007/978-3-031-05328-3.

[15] L. Wang, Y. Lepage, Vector-to-sequence models for sentence analogies, in: 2020 International Conference on Advanced Computer Science and Information Systems (ICACSIS), 2020, pp. 441–446. doi:10.1109/ICACSIS51025.2020.9263191.

[16] T. Wijesiriwardene, R. Wickramarachchi, A. N. Reganti, V. Jain, A. Chadha, A. Sheth, A. Das, On the relationship between sentence analogy identification and sentence structure encoding in large language models, in: Y. Graham, M. Purver (Eds.), Findings of the Association for Computational Linguistics: EACL 2024, Association for Computational Linguistics, St. Julian's, Malta, 2024, pp. 451–457. URL: https://aclanthology.org/2024.findings-eacl.31.

[17] W.-t. Yih, V. Qazvinian, Measuring word relatedness using heterogeneous vector space models, in: E. Fosler-Lussier, E. Riloff, S. Bangalore (Eds.), Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Montréal, Canada, 2012, pp. 616–620. URL: https://aclanthology.org/N12-1077.

[18] N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3982–3992. URL: https://aclanthology.org/D19-1410. doi:10.18653/v1/D19-1410.

[19] Y. Lepage, Solving analogies on words: an algorithm, in: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL’98) and the 17th International Conference on Computational Linguistics (COLING’98), volume I, Association for Computational Linguistics, Montreal, 1998, pp. 728–735. URL: https://www.aclweb.org/anthology/P98-1120/. doi:10.3115/980845.980967.

[20] W. Mao, Y. Lepage, Embedding-to-embedding method based on autoencoder for solv-
ing sentence analogies, in: Proceedings of the workshop Analogies: from Theory to Applications (ATA@ICCBR 2023), Aberdeen, Scotland, 2023, pp. 15–26.

[21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Neural Information Processing Systems, Neural Information Processing Systems (2017).
# Learning Analogies between Classes to Create Counterfactual Explanations

Xiaomeng Ye<fnref n="1" />*, David Leake<fnref n="2" />, Yu Wang<fnref n="2" />, Ziwei Zhao<fnref n="2" /> and David Crandall<fnref n="2" />

**Abstract**

Machine learning (ML) classification algorithms predict based on learned patterns of class characteristics. The patterns between different classes are less studied. Such patterns can be considered as analogies in classification domain: If *a* and *c* are from a class *C*₁, *b* and *d* are from a class *C*₂ and the two classes *C*₁ and *C*₂ are consistently similar in some features and different in some other features, then the analogy *a*: *b*:: *c*: *d* holds, because the similarities (resp. differences) between *a* and *b* are equivalent to those between *c* and *d*. A class-to-class siamese network (C2C-SN) is a classifier neural network that trains on sample pairs from two classes *C*₁ and *C*₂, learns an inter-class pattern—i.e., an analogy—between the two classes and then can decides whether any two samples *a* and *b* holds in the analogy *a*: *b*:: *c*₁: *c*₂ where *c*₁ and *c*₂ are representative samples from *C*₁ and *C*₂ respectively. If the analogy does exist, from sufficient random sample pairs the C2C-SN can learn without human knowledge or intervention. This paper proposes a general method to regularize any feature extraction ML model in a classification domain, by combining the ML model with a C2C-SN, to regulate the ML model to extract features important to inter-class analogies. It demonstrates an application of this method to learning and applying these analogies for counterfactual explanation in an image domain.

**Keywords**

Analogy, Case Adaptation, Case-based Reasoning, Counterfactual, Representation Learning

## 1. Introduction

Analogical reasoning research has been extensively studied for symbolic representations (e.g., [1, 2]), but is less commonly applied to non-symbolic data such as images until recent [3, 4]. Machine learning techniques have achieved great success for classifying complex data such as large language corpora, images, and audio data. However, models like neural networks generally learn the pattern from input to output while ignoring analogies between the categories. This paper presents research on learning analogies directly from non-symbolic data using machine learning techniques. Specifically, it focuses on analogy in an image classification domain, assuming that a similarity and difference pattern exists between a pair of classes *C*₁ and *C*₂. For *x*₁, *x*₂ of class *C*₁ and *y*₁, *y*₂ of class *C*₂ and *x*₁ is similar to and different from *y*₁ the same

IARML@IJCAI'2024: Workshop on the Interactions between Analogical Reasoning and Machine Learning, at IJCAI'2024, August, 2024, Jeju, South Korea

<figure><img src="image_9.png" /></figure>

xye@berry.edu (X. Ye); leake@iu.edu (D. Leake); yw173@iu.edu (Y. Wang); zz47@iu.edu (Z. Zhao);

djcran@iu.edu (D. Crandall)

<figure><img src="image_10.png" /></figure>

0000-0002-2289-1022 (X. Ye); 0000-0002-8666-3416 (D. Leake)

<figure><img src="image_11.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<footnote n="1">¹Berry College, Mount Berry GA 30149, USA</footnote>

<footnote n="2">²Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington IN 47408, USA</footnote>

<footnote>*Corresponding author.</footnote>
way as $x_2$ is for $y_2$, we consider $x_1 : y_1 :: x_2 : y_2$ as an analogy. The paper describes a general
method that can be used to guide any machine learning (ML) model trainable with gradient
descent, to extract features with analogy in mind.

We illustrate the method by applying it to use the analogies for counterfactual explanation [5].
Counterfactual explanations explain classifications by showing the effects of feature changes on
the classification. By doing so, they both clarify the scope of the concepts and potentially provide
actionable information for changing the outcome. For example, a counterfactual explanation
for denying a loan might be that the applicant would have been approved if the applicant had
had 5000 euro more savings. Given a domain described by symbolic features, counterfactual
explanations can be generated by selectively changing feature values to generate examples
across a decision boundary. For explaining classifications of image data, predefined feature sets
are not available. Consequently, research on counterfactual explanations of image classifications
has used ML techniques to extract and modify features [6, 7].

This paper demonstrates the usefulness of the learned analogies in counterfactual explanation
in two image domain tasks. The method regularizes the feature extraction process by pairing
it with a class-to-class siamese network (C2C-SN) and tuning it to extract features that are
significant in inter-class analogies. A C2C-SN can rank features based on their importance in
inter-class analogies. We hypothesize that modifying such features, which are most important
for distinguishing classes, leads to good counterfactual explanations.

The paper begins with related work. It then discusses the process of training an upstream
model with a C2C-SN to learn inter-class analogies and to rank features by their importance
in these analogies. It presents three experiments testing the method. The first experiment
combines the C2C-SN with a Variational Autoencoder (VAE) [8] and applies the resulted model
to two data sets, MNIST [9] and FashionMNIST [10], showing that modifying single features
top-ranked by C2C-SN feature importance often directly leads to desired class change. A
second experiment combines C2C-SN with the PIECE algorithm [6], which can modify features
exceptional for a class into their expected values, on the MNIST data set. A C2C-SN network
can rank features and suggest which features to change while PIECE can carry out the change to
generate a counterfactual. The counterfactuals generated are different but comparable in quality
to previous results, supporting that C2C-SN can work with different upstream ML models and
can guide a counterfactual explanation method with a new perspective. A third experiment
combines C2C-SN with a VAE on the cars3d data set [11], revealing limitations of the approach
and a potential direction for future research.

## 2. Background

### 2.1. Analogy Between Classes and Class-to-Class (C2C) Methodology

Machine learning classification methods generally classify based on similarity between members
of a class. Less studied are the patterns of similarities and differences between classes. The
pattern from one class to another is essentially an analogy between the two classes. For example,
given two images of the digit 1, in roman and italic fonts ("1" and "1"), and two digit 7s ("7"
and "7"), there exists the analogy that "1" is to "7" as "1" is to "7", matching the pattern of the
analogical proportion "A is to B as C is to D". Moreover, such an analogy applies to many pairs
of samples from the two classes: Here “1”s and “7”s share a vertical line, but “7”s always have a horizontal line on the top while “1”s do not. In other words, if 1s (or 7s) are similar to other 1s (or 7s) in a consistent way, then 1s and 7s are similar or different in a consistent way.

Inspired by siamese networks [12], which learn a similarity pattern between samples sharing the same class label, Ye et al. [13] developed class-to-class siamese networks (C2C-SNs) which learn the inter-class patterns where samples of one class are consistently different or similar on certain features when compared to samples of another class. Following the original paper, the inter-class pattern, or analogy, from a class $C_1$ to a class $C_2$ will be denoted as $C_1-C_2$. This paper modifies the original design slightly: Here a C2C-SN is a neural network that uses a shared network to extract features from two samples, calculates a feature difference $f_\Delta$ and a class label difference $C_\Delta$, and outputs a value $p$ (between 0 and 1) indicating whether the feature difference $f_\Delta$ matches the analogy between the two classes indicated in $C_\Delta$.

This paper presents a general usage of C2C-SNs to highlight features relevant to inter-class analogies learned from data. By pairing a C2C-SN with any feature extraction model and training both together, the C2C-SN can (1) regularize the training of the feature extractor so the features extracted are useful in identifying inter-class analogies and (2) rank features based on their importance in a specific analogy.

## 2.2. Machine Learning Models that Learn Analogies

As discussed in the survey by Bounhas et al., [14], the analogy community has studied analogies for the purposes of classification (e.g., [15, 16, 17]) and explanation [18]. Analogical models are mainly either logical, algebraic, or complexity-based [19]. This paper takes a different path by using a neural model to learn the inter-class analogies directly from image data. The resulting model does not require the process of choosing a triplet *a*, *b*, *c* to classify (or explain) a sample *d* based on an analogy *a* : *b* :: *c* : *d*. Instead, a C2C-SN simply decides whether a pair *a* : *b* match a pattern $C_1-C_2$.

Early analogy research explored concept analogies on small symbolic concepts [20], while more recent analogy research has applied modern ML techniques to domains such as tabular data [5], images [21] and natural language processing tasks [22, 23]. A comprehensive survey can be found in [24]. This paper presents a neural network method that can help any feature extraction model to find analogy-relevant features in classification domains, and demonstrates how the identification of such features can be used in counterfactual explanation in images. Lim [18] also identified the usage of analogy in factual and counterfactual explanations, but they experimented on Boolean data sets; Our work is in spirit aligned with Hüllermeier [5], who demonstrated analogy-based explanation in machine learning on a tabular data and suggested that analogy-based explanations can complement similarity-based explanations.

## 2.3. Counterfactual Explanation for Images

Counterfactual explanation in image domains has received much attention [7]. Generating counterfactual images often requires applying machine learning techniques to extract features, which are then modified for counterfactual explanation. For example, a VAE can encode sample inputs to embeddings that fall within a predetermined distribution (normally Gaussian). REVISE
creates counterfactual by modifying latent features extracted by a VAE [25]. Theobald et al. [26] shows that REVISE can generate unrealistic images and improves upon it with Clarity.

Kenny and Keane [6] propose the PlausIBLE Exceptionality-based Contrastive Explanations (PIECE) algorithm, which “identifies probabilistically-low feature-values in the test image (i.e., exceptional features) and modifies them to be their expected values in the counterfactual class (i.e., normal features)”. Feature importance methods measure how influential a feature is in a model’s output. Two popular examples are LIME [27] and SHAP [28]. A recent survey on feature importance methods [29] links feature importance to counterfactual explanations, as they can both serve as local explanations for a model’s behavior.

Our approach uses the Fisher et al. [30] model reliance method (as explained in Molnar [31]) to rank feature importance in inter-class analogies. This ranking guides a counterfactual explanation method to adapt a sample from a source class $C_1$ into a target class $C_2$ by prioritizing the modification of features important in the pattern $C_1-C_2$. It can be considered as an experience-guided counterfactual explanation method.

# 3. Proposed Method

Our method for counterfactual generation begins by extracting features regularized by a C2C-SN, then identifies the most important features for analogies, and generates counterfactuals by varying those features. This approach is generally applicable to any feature extractor trainable with backpropagation. This study illustrates the usage in combining C2C-SN with a VAE (where the feature extractor is the encoder as shown in Figure 1) and the PIECE algorithm (where the feature extractor is a convolutional neural network). If properly trained and regularized by the C2C-SN, the feature extractor would extract features that are both useful in its original downstream task (e.g. classification, reconstruction) and meaningful for inter-class analogies.

## 3.1. Training Process

The training process coordinates training of the C2C-SN and of the upstream ML feature extraction model, as shown in Algorithm 1. We highlight a few key points:

* Each batch of samples $s_1$ and the corresponding labels $C_1$ are used to calculate the original loss of the ML model as $L_f(f(s_1), C_1)$.

* A second batch of samples and labels is also created as $s_2$ and $C_2$ by shuffling $s_1$ and $C_1$ in the same order. A feature difference can be calculated as $f_Δ = f(s_1) - f(s_2)$. A class label difference $C_Δ = C_1 - C_2$ is constructed as the element-wise vector difference between the one hot encodings of $C_1$ and $C_2$ [32]. ($f_Δ, C_Δ$) serve as the positive training data ($X_{pos}, y_{pos}$) for the C2C-SN. Then $C_2$ is again shuffled into $\hat{C}_2$ so that every value is different and $\hat{C}_2$ is a batch of *wrong* labels for samples $s_2$. ($f_Δ, \hat{C}_Δ = C_1 - \hat{C}_2$) will serve as the negative training data ($X_{neg}, y_{neg}$) for the C2C-SN.

* The positive and negative training data are combined into one set ($X = X_{pos} + X_{neg}, y = y_{pos} + y_{neg}$) and shuffled. The C2C-SN is trained using contrastive loss $L_{c2c}(X, y)$ to best predict whether the input sample difference $f_Δ$ matches with the analogy of $C_Δ$.
<figure><img src="image_12.png" /><figcaption>**Figure 1:** Model Structure. Given a pair of samples $s_1$ and $s_2$ of class $C_1$ and $C_2$ respectively, the feature extractor $f$ (a VAE's encoder in this case) extracts features $f(s_1)$ and $f(s_2)$. The feature difference $f_{\Delta} = f(s_1) - f(s_2)$ is concatenated with a label difference $C_{\Delta}$ (that is either correct $C_1-C_2$ or incorrect $C_1-C_{\text{wrong}}$ where $C_{\text{wrong}} \neq C_2$). The expected output $y$ of the C2C-SN is 1 or 0 depending on whether $C_{\Delta}$ is correct or not.</figcaption></figure>

* Last, the training of the upstream model $f$ and the C2C-SN can be combined by optimizing their parameters together to lower the total loss

$$L = L_f(f(s_1), C_1) + \alpha * L_{c2c}(X, y) \quad (1)$$

where the parameter $\alpha$ controls the relative importance of the contrastive loss $L_{c2c}$.

## 3.2. C2C-SN Feature Importance

Once the C2C-SN is trained along with the feature extractor $f$, we apply a permutation feature importance method [30] (as explained in Molnar [31]) to calculate the importance of each value $f_{\Delta i}$ in the feature difference $f_{\Delta}$. The method randomly permutes each value $f_{\Delta i}$ in $f_{\Delta}$ and measures how much the output of the C2C-SN is changed when using the new feature $f'_{\Delta}$ instead of the original feature $f_{\Delta}$. If the value $f_{\Delta i}$ is important, then changing it will drastically change the final error of the C2C-SN. Because $f_{\Delta i}$ is calculated from $f_{\Delta i} = f(s_1)_i - f(s_2)_i$, the importance of $f_{\Delta i}$ is also the feature importance of $f(s_1)_i$ and $f(s_2)_i$ in the analogy $C_1-C_2$.

In contrast to other feature importance methods which permute one feature of the input and measure the variance in a model's output [30], the C2C-SN input includes both $f_{\Delta}$ and a class label difference $C_{\Delta}$. Therefore C2C-SN feature importance can *rank features locally* based on $C_{\Delta}$: The local feature important in one analogy can be of little significance in another analogy. The local feature importance is especially useful for counterfactual explanation by highlighting the most semantically meaningful difference between two classes. Samples of a class $C_1$ can be adapted into another class $C_2$ by changing the most important features in the analogy $C_1-C_2$, yielding a counterfactual. C2C-SN feature importance can also rank features as *global feature importance* across all $C_{\Delta}$ analogies.
The decoder (marked as $f'$ in Figure 1) is the reverse of the encoder: an embedding is passed to a linear layer ($in_features = latent_dim$, $out_features = c * 2 * 7 * 7$), the output of which is passed to two convolutional layers: Conv2d ($inull channels = c * 2$, $out\_channels = c$, $kernel\_size = 4$, $stride = 2$, $padding = 1$, ReLU) and Conv2d ($in\_channels = c$, $out\_channels = 1$, $kernel\_size = 4$, $stride = 2$, $padding = 1$, Sigmoid). As in standard VAEs, the encoder $f$ and decoder $f'$ together are trained to optimize a loss function $L_f$ that is composite of a reconstruction loss (so the reconstructed image $f'(f(s))$ is similar to the original input $s$) and a KL-divergence loss (so the distribution of $mus$ conforms to a prior Gaussian distribution whose $mu = 0$ and $std = 1$).

The C2C-SN is a composite of two hidden layers with leaky ReLU activation func-
tions: ($in\_features = latent\_dim + label\_size$, $out\_features = c2c\_capacity$) and
($in\_features = c2c\_capacity$, $out\_features = c2c\_capacity/2$), and an output linear layer
($in\_features = c2c\_capacity/2$, $out\_features = 1$). The $c2c\_capacity$ is set to 32. The en-
coder $f$ and the C2C-SN are trained together to minimize the constrastive loss $L_{c2c}$.
Therefore,
the encoder $f$, the decoder $f'$ and the C2C-SN are all trained together to minimize the total loss
$L = L_f + \alpha * L_{c2c}$ by following the procedure described in Section 3.1.

4.1.1. Experimental Results on MNIST

Given a sample *s* from MNIST, we use C2C-SN feature importance to rank features *f*(*s*) both locally on analogical patterns between two classes and globally across all patterns. Using an image “1” as an example, Figure 2a shows the reconstructed images after modifying each of the top 8 globally ranked features out of 32 features in total. In Figures 2a-4b, each row shows the effect of modifying one feature for 9 equally spaced values. The values fall between −1 and 1, because the embeddings conform to a Gaussian distribution of *mu* = 0 and *std* = 1. The images in the center conform to the average of the class, while the ones on the two ends are more irregular, potentially of a different class. Figure 2a shows that features that are globally important may not necessarily lead to class change when modified. For example, modifying the 2nd - 4th most important features do not modify this “1” into digits of other classes.

C2C-SN feature importance does well at ranking features based on their local importance in the pattern between two specific classes. As shown in Figures 3a and 3b, C2C-SN feature importance correctly suggests the top features that can lead to the desired class change. More specifically, Figure 3a ranks features based on the analogy between “1”s and “2”s. It shows that the digit “1” can be modified into “2” by altering a single feature, however, too much modification might distort the image unrealistically, because the modified feature is now at the boundary of the distribution of latent embeddings. Figure 3b ranks features based on the pattern between “1”s and “9”s. It shows another example in which the digit “1” can be modified into a “9” by altering one single feature. Note that the digit “1” is modified into a “7” before it changes to a “9”. This is because the embedding space of VAE is intrinsically smooth and other classes may be allowed between embeddings of two classes.

Figure 3c ranks features based on the pattern between “1”s and “5”s. Here, the C2C-SN failed to suggest any feature that can modify the “1” into a “5”. Although the C2C-SN encourages the VAE encoder to extract features that are important in analogies, the training does not guarantee that a single feature corresponds to a given class change. Changing multiple features might be
<figure><img src="image_13.png" /><figcaption>**Figure 2:** Gradual Changes of an Image when Modifying Top 8 Globally Ranked Features</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>**Figure 3:** Gradual Changes of the Image “1” when Modifying Top 5 Locally Ranked Features in the Pattern from “1”s to (a) “2”s (b) “9”s (c) “5”s</figcaption></figure>

required for the change between two vastly different classes (e.g. “1”s and “5”s).

### 4.1.2. Experimental Results on FashionMNIST

The same experiment was carried out on FashionMNIST. Using a Pants image as an example, Figure 2b demonstrates the reconstructed images after modifying each of the top 8 globally ranked features. The majority of the features do not lead to class change when modified.

Figure 4a ranks features based on the pattern between Pants and Long Sleeves. Figure 4b ranks features based on the pattern between Pants and Shoes. Note that modifying one feature may not be enough to cause a desired change, as the Shoes converted from the Pants have a gap in the middle. This aligns with our earlier discussion of the results for MNIST.

These experiments illustrate that the C2C-SN can help a ML model to extract features for analogy and C2C-SN feature importance can rank these features for counterfactual explanation. However, it is not guaranteed that modifying a top feature will lead to desired class change, especially when the differences between the two classes are complex.
<figure><img src="image_15.png" /><figcaption>**Figure 4:** Gradual Changes of a Pants Image when Modifying Top 5 Locally Ranked Features in the Pattern from Pants to (a) Long Sleeves (b) Shoes</figcaption></figure>

## 4.2. Quality of Counterfactuals Using C2C-SN Feature Importance in PIECE

The strategy of learning analogies with C2C-SN should be generally applicable when using any feature extractor. This section presents an experiment combining C2C-SN feature importance with the PIECE algorithm [6], which uses a convolutional neural network for feature extraction.

This experiment generates counterfactuals for misclassified samples from Kenny and Keane [6] but replaces the feature selection component of PIECE with a new algorithm that selects top features ranked by C2C-SN feature importance. The experiment applies different methods to generate counterfactual explanations for MNIST and compares the generated images both qualitatively and quantitatively. The original design of PIECE has four major steps [6]:

1. A convolutional neural network `cnn` is trained to classify MNIST samples. We find 41 hard samples misclassified by `cnn`. We denote a hard sample image as `s`, the true class label as `Ctrue` and the misclassified label as `Cpred`.

2. The CNN derives the feature vector `x` of a sample `s` as the activation of the penultimate layer of `cnn` when applied to `s`, denoted as `x = cnn(s)`. A GAN-inversion technique is used to generate an image using a GAN `G`. A GAN embedding `z` for image `s` is optimized so that `x = cnn(G(z))`.

3. The probability function of each feature value `x` is modeled as a hurdle model. A feature value `x` is considered exceptional if the value is rare in the true class `Ctrue` (probabilities are less than 0.05). PIECE only considers modifying `x` if doing so will increase the output activation for `Ctrue`. Assume `n` feature values are to be modified by PIECE.

4. All `n` feature values are modified into their expected value in `Ctrue`. The modified feature is denoted as `x'`. Last, `z` is optimized into `z'` such that `cnn(G(z')) ≈ x'`. The counterfactual explanation is `G(z')`.

Two sample counterfactuals are shown in Figures 5a and 6a. In both scenarios, the CNN classifier misclassifies a digit "7" is as a class other than "7", and the PIECE algorithm attempts to generate a counterfactual explanation by making the exceptional features normal. The modified image looks more like a standard "7". This explanation can be understood as follows: The model misclassified the query as non-"7" because it believes that the image needs to be like the counterfactual example to be considered a "7".

We first created a variation of the above algorithm (which we name as "C2C" for convenience), in which we replaced Step 3 in PIECE with the following: A C2C-SN is trained on the features
<figure><img src="image_16.png" /><figcaption>**Figure 5:** Comparison of Four Different Counterfactuals Generated for the Digit "7" in Figure 5a</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>**Figure 6:** Comparison of Four Different Counterfactuals Generated for the Digit "7" in Figure 6a</figcaption></figure>

extracted and provides a C2C-SN feature importance ranking for the pattern $C_{true}-C_{pred}$. The top *n* features are then modified in their expected value in Step 4. The choice of *n* depends on how many features are deemed exceptional by PIECE. This variation selects features based on the C2C-SN feature importance ranking but modifies features in the same manner as PIECE.

We also made a variation of PIECE (named “PIECE-step”) that, instead of modifying all exceptional features, modifies exceptional features one by one, starting from the most exceptional, until the modified sample is classified as $C_{true}$ by the cnn. Last, we made a variation of C2C (named “C2C-step”) which instead of modifying *n* features, modifies top features suggested by the C2C-SN feature importance one by one, until the modified sample is classified as $C_{true}$.

Thus we tested four variations: The original PIECE algorithm, PIECE-step that modifies features one by one until crossing the class boundary, the C2C algorithm that replaces PIECE’s Step 3 with C2C-SN feature importance ranking, and the C2C-step that modifies features one by one until crossing the class boundary. The four were run on all 41 hard samples. Figures 5 and 6 show counterfactual examples generated by different algorithms for two sample digit “7”s. We observe that PIECE and C2C both produce plausible counterfactuals, and their corresponding step variations also produce counterfactuals near the class boundary.

We compared the models quantitatively, reusing the following metrics the original evaluation of PIECE [6]: IM1 and IM2 are two measures of the interpretability of counterfactuals [33]; The posterior mean (MC-Mean) and posterior standard deviation (MC-STD) of MC dropout measures the model’s confidence and uncertainty on the counterfactuals [34]; Last, NN-Dist measures the distance between a counterfactual and a nearest neighboring training sample. These metrics quantitatively reflect how realistic the generated images are. Table 1 compares the counterfactuals generated by different algorithms for the 41 hard samples under the five metrics. Min-Edit is included as a baseline counterfactual generation algorithm. Note that PIECE and C2C are comparable to each other because they modify a fixed number *n* of features, while the rest of the algorithms modify features one by one until crossing the class boundary.
<table><caption>**Table 1**
Comparison of Counterfactual Generation Algorithms</caption><thead><tr><th></th><th>MC-Mean</th><th>MC-STD</th><th>NN-Dist</th><th>IM1</th><th>IM2</th></tr></thead><tbody><tr><td>PIECE</td><td><strong>90591</strong></td><td>248393</td><td><strong>0.411</strong></td><td>3.005</td><td>0.064</td></tr><tr><td>C2C</td><td>5937</td><td><strong>12574</strong></td><td>0.626</td><td><strong>2.338</strong></td><td>0.064</td></tr><tr><td>Min-Edit</td><td><strong>600</strong></td><td>1168</td><td>1.066</td><td>3.897</td><td><strong>0.067</strong></td></tr><tr><td>PIECE-step</td><td>395</td><td><strong>668</strong></td><td>1.029</td><td><strong>2.227</strong></td><td>0.072</td></tr><tr><td>C2C-step</td><td>409</td><td>811</td><td><strong>1.014</strong></td><td>2.607</td><td>0.073</td></tr><tr><td>which is better?</td><td>higher</td><td>lower</td><td>lower</td><td>lower</td><td>lower</td></tr></tbody></table>

<figure><img src="image_18.png" /><figcaption>**Figure 7:** Gradual Changes of a Car's Image when Modifying Top 5 Locally Ranked Features in the Inter-Elevation Pattern</figcaption></figure>

Both the qualitative and quantitative comparisons show that PIECE and C2C (PIECE-step and C2C-step correspondingly) generate counterfactuals with their own unique “tastes” but achieve similar overall performance.

### 4.3. Limitations of Analogies Found by C2C-SN

The assumption that there exists an analogy between two classes $C_1$ and $C_2$ may not hold when either class is widely dispersed. For $x_1, x_2$ of class $C_1$ and $y_1, y_2$ of class $C_2$, $x_1$ (or $y_1$) can be very different from $x_2$ (or $y_2$) and the feature difference $f(x_1) - f(y_1)$ can be not consistent with $f(x_2) - f(y_2)$. We demonstrate this issue in the following experiment combining C2C-SN with a Guided-VAE [35] on the Cars3d data set [11].

Cars3d is a multi-label data set where each sample is an image associated with one of 184 car models, one of 24 azimuths and one of 4 camera elevations. Feature entanglement is intrinsically difficult in the Cars3d data set. For example, two images of different car models may pose with different camera angles, obscuring the analogy between the models. We experimented with Guided-VAE which reportedly encourages feature disentanglement through supervised learning [35]. However, features are still entangled. Figure 7 shows changes in features extracted for inter-elevation pattern often change the azimuth or car model as a side-effect.

The sub-optimal experimental result can be explained in a few ways: (1) C2C-SN encourages features extracted to be relevant for a target analogy but does not prevent the entanglement of other analogies. (2) C2C-SN relies on the feature extractor. If the feature extractor fail to
disentangle features, C2C-SN suffers with it. (3) The analogical pattern between two complex classes may involve multiple features instead of one.

As a future direction, an analogical suitability test can be devised to check whether a data set with a feature extractor is suitable for analogy learning using C2C-SN.

## 5. Conclusion

This paper proposes a generally applicable method to extract analogies corresponding to general similarity and difference patterns between elements of classification domains, and illustrates the usefulness of such analogies in counterfactual explanation. An ML model with a feature extractor and a C2C-SN can be trained together where the loss of the C2C-SN functions as a regularizer for the original ML loss. The ML model is thus trained to extract features that are useful for (1) the original purpose of the ML model (e.g. classification or image reconstruction) and (2) describing analogies. This method can provide C2C-SN feature importance rankings and suggest features to modify for counterfactual explanations. Experimental results of quantitative evaluations show results comparable to an existing method while the proposed method is more generally applicable. An interesting future question is to apply C2C-SN to discover analogies in other domains such as natural language texts.

## Acknowledgments

This paper is adapted from "Selecting Feature Changes for Counterfactual Explanation: A Class-to-Class Approach," which appeared in the non-archival proceedings of the IJCAI 2023 Workshop on Explainable AI (XAI).

## References

[1] R. P. Hall, Computational approaches to analogical reasoning: A comparative analysis, Artificial Intelligence 39 (1989) 39–120. URL: https://www.sciencedirect.com/science/article/pii/0004370289900039. doi:https://doi.org/10.1016/0004-3702(89)90003-9.

[2] D. Hofstadter, M. Mitchell, The copycat project: A model of mental fluidity and analogy-making, in: K. Holyoak, J. Barnden (Eds.), Advances in Connectionist and Neural Computation Theory, volume 2, Ablex, 1993, pp. 31–11.

[3] S. E. Reed, Y. Zhang, Y. Zhang, H. Lee, Deep visual analogy-making, in: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL: https://proceedings.neurips.cc/paper_files/paper/2015/file/e07413354875be01a996dc560274708e-Paper.pdf.

[4] N. Ichien, Q. Liu, S. Fu, K. J. Holyoak, A. L. Yuille, H. Lu, Visual analogy: Deep learning versus compositional models, Proceedings of the 43rd Annual Meeting of the Cognitive Science Society (2021). URL: https://par.nsf.gov/biblio/10231806.

[5] E. Hüllermeier, Towards analogy-based explanations in machine learning, in: V. Torra, Y. Narukawa, J. Nin, N. Agell (Eds.), Modeling Decisions for Artificial Intelligence, Springer International Publishing, Cham, 2020, pp. 205–217.
[6] E. M. Kenny, M. T. Keane, On generating plausible counterfactual and semi-factual explanations for deep learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 11575-11585.

[7] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, S. Lee, Counterfactual visual explanations, in: International Conference on Machine Learning, PMLR, 2019, pp. 2376-2384.

[8] D. P. Kingma, M. Welling, Auto-Encoding Variational Bayes, in: 2nd International Conference on Learning Representations, ICLR 2014, 2014.

[9] Y. LeCun, C. Cortes, MNIST handwritten digit database (2010). URL: http://yann.lecun.com/exdb/mnist/.

[10] H. Xiao, K. Rasul, R. Vollgraf, Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017. URL: http://arxiv.org/abs/1708.07747, arxiv:1708.07747.

[11] S. E. Reed, Y. Zhang, Y. Zhang, H. Lee, Deep visual analogy-making, in: Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL: https://proceedings.neurips.cc/paper_files/paper/2015/file/e07413354875be01a996dc560274708e-Paper.pdf.

[12] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, R. Shah, Signature verification using a "siamese" time delay neural network, in: Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'93, Morgan Kaufmann, San Francisco, 1993, pp. 737-744.

[13] X. Ye, D. Leake, W. Huibregtse, M. Dalkilic, Applying class-to-class siamese networks to explain classifications with supportive and contrastive cases, in: Case-based reasoning research and development, ICCBR-20, Springer, 2020, pp. 245-260.

[14] M. Bounhas, H. Prade, G. Richard, Some recent advances in reasoning based on analogical proportions, 2022. arXiv:2212.11717.

[15] M. Bounhas, H. Prade, G. Richard, Analogy-based classifiers for nominal or numerical data, International Journal of Approximate Reasoning 91 (2017) 36-55. URL: https://www.sciencedirect.com/science/article/pii/S0888613X17305303. doi:https://doi.org/10.1016/j.ijar.2017.08.010.

[16] M. Couceiro, N. Hug, H. Prade, G. Richard, Behavior of analogical inference w.r.t. boolean functions, in: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, IJCAI, 2018, pp. 2057-2063.

[17] E. Marquer, F. Badra, M.-J. Lesot, M. Couceiro, D. Leake, Less is better: An energy-based approach to case base competence, in: ATA'23: Workshop on Analogies: From Theory to Applications, ICCBR 2023 Workshop Proceedings, CEUR, 2023, pp. 27-42.

[18] S. Lim, H. Prade, G. Richard, Using analogical proportions for explanations, in: F. Dupin de Saint-Cyr, M. Öztürk-Escoffier, N. Potyka (Eds.), Scalable Uncertainty Management, Springer International Publishing, Cham, 2022, pp. 309-325.

[19] H. Prade, G. Richard, Analogical proportions: Why they are useful in AI, in: International Joint Conference on Artificial Intelligence, IJCAI, 2021, pp. 4568-4576.

[20] M. Mitchell, Analogy-Making as Perception, MIT Press/Bradford Books, Cambridge, MA, 1993.

[21] S. Hu, Y. Ma, X. Liu, Y. Wei, S. Bai, Stratified rule-aware network for abstract visual reasoning, in: AAAI Conference on Artificial Intelligence, AAAI, 2020, pp. 1567-1574.

[22] E. Marquer, M. Couceiro, Solving morphological analogies: from retrieval to generation,
ArXiv abs/2303.18062 (2023). URL: https://api.semanticscholar.org/CorpusID:257900745.

[23] C. Allen, T. M. Hospedales, Analogies explained: Towards understanding word embeddings, in: International Conference on Machine Learning, PMLR, 2019, pp. 223–231.

[24] M. Mitchell, Abstraction and analogy-making in artificial intelligence, Annals of the New York Academy of Sciences 1505 (2021). URL: https://api.semanticscholar.org/CorpusID: 231985717.

[25] S. Joshi, O. Koyejo, W. D. Vijitbenjaronk, B. Kim, J. Ghosh, Towards realistic individual recourse and actionable explanations in black-box decision making systems, ArXiv abs/1907.09615 (2019). URL: https://api.semanticscholar.org/CorpusID:198179624.

[26] C. Theobald, F. Pennerath, B. Conan-Guez, M. Couceiro, A. Napoli, Clarity: an improved gradient method for producing quality visual counterfactual explanations, 2022. arXiv:2211.15370.

[27] M. T. Ribeiro, S. Singh, C. Guestrin, "why should i trust you?": Explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, Association for Computing Machinery, New York, NY, USA, 2016, p. 1135–1144. URL: https://doi.org/10.1145/2939672.2939778. doi:10.1145/2939672.2939778.

[28] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 4768–4777.

[29] S. Mishra, S. Dutta, J. Long, D. Magazzeni, A survey on the robustness of feature importance and counterfactual explanations, 2023. arXiv:2111.00358.

[30] A. Fisher, C. Rudin, F. Dominici, All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously, 2019. arXiv:1801.01489.

[31] C. Molnar, Interpretable Machine Learning, 2 ed., 2022. https://christophm.github.io/interpretable-ml-book.

[32] X. Ye, D. Leake, D. Crandall, Case adaptation with neural networks: Capabilities and limitations, in: Case-Based Reasoning Research and Development ICCBR-22, Springer, Cham, 2022, pp. 143–158.

[33] A. Van Looveren, J. Klaise, Interpretable counterfactual explanations guided by prototypes, in: Machine Learning and Knowledge Discovery in Databases. Research Track, Springer, Cham, 2021, pp. 650–665.

[34] Y. Gal, Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, JMLR.org, 2016, p. 1050–1059.

[35] Z. Ding, Y. Xu, W. Xu, G. Parmar, Y. Yang, M. Welling, Z. Tu, Guided variational autoencoder for disentanglement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7920–7929.
# Enhancing Analogical Reasoning in the Abstraction and Reasoning Corpus via Model-Based RL

Jihwan Lee¹,†, Woochang Sim¹,†, Sejin Kim¹,* and Sundong Kim¹,*

¹Gwangju Institute of Science and Technology

**Abstract**

This paper demonstrates that model-based reinforcement learning (model-based RL) is a suitable approach for the task of analogical reasoning. We hypothesize that model-based RL can solve analogical reasoning tasks more efficiently through the creation of internal models. To test this, we compared DreamerV3, a model-based RL method, with Proximal Policy Optimization, a model-free RL method, on the Abstraction and Reasoning Corpus (ARC) tasks. Our results indicate that model-based RL not only outperforms model-free RL in learning and generalizing from single tasks but also shows significant advantages in reasoning across similar tasks.

**Keywords**

Abstraction and Reasoning Corpus, Analogical Reasoning, Model-Based RL, DreamerV3, PPO

## 1. Introduction

The Abstraction and Reasoning Corpus (ARC)¹ is known as a benchmark for evaluating abstraction and reasoning abilities [1]. Examples of ARC tasks can be seen in Fig. 1. To solve an ARC task, one needs the ability to abstract a common rule among given demos and apply this rule to a new test input grid to infer the corresponding grid [2]. Additionally, to solve untrained tasks, one must have the ability to discover various knowledge during the learning process and selectively apply this knowledge in the task-solving process [2].

This study interprets the two abilities required by ARC not as separate skills but as one concept of analogical reasoning. Analogical reasoning refers to the process of understanding new situations or solving problems based on existing knowledge that can be applied to different contexts or problems [3]. First, the process of solving an ARC task can be seen as solving a new problem (test input grid) based on the transformations from input grids to output grids in each demo (existing knowledge), which exactly fits the definition described above. Second, for a model trained on various ARC tasks to solve a new ARC task, it must understand the demos of the new ARC task (new situations) based on the solution processes (existing knowledge) applicable to the training tasks (different problems) and solve the task, which also aligns with the concept of

IARML@IJCAI'2024: Workshop on the Interactions between Analogical Reasoning and Machine Learning, at IJCAI'2024, August, 2024, Jeju, South Korea

*Corresponding authors.

†Equal contribution.

jihwan.lee@gm.gist.ac.kr (J. Lee); woochang@gm.gist.ac.kr (W. Sim); sejinkim@gist.ac.kr (S. Kim); sundong@gist.ac.kr (S. Kim)

<figure><img src="image_19.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_20.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)

¹ARC dataset URL: https://github.com/fchollet/ARC-AGI
<figure><img src="image_21.png" /><figcaption>**Figure 1:** Each ARC task includes several demos and a test input. The objective is to identify the grid corresponding to the test input by applying a common transformation rule found across all demos. The term “CCW” in the third task means counterclockwise.</figcaption></figure>

analogical reasoning. Third, since the evaluation tasks in ARC are typically more complex and challenging than the training tasks [4], the ability to infer new problem-solving processes based on learned knowledge through analogical reasoning can enhance the performance on ARC.

In this study, noting the critical importance of analogical reasoning capabilities in solving ARC, we conducted reinforcement learning (RL) for training ARC tasks. Learning ARC with RL can offer the following potential benefits from an analogical reasoning perspective. First, RL can enhance its learning strategies through reward-based learning. In the process of learning ARC tasks, RL agents learn to maximize rewards across various tasks, thereby developing analogical reasoning capabilities. Second, RL can learn policies that effectively operate in new, yet similar environments based on experience. During the learning of various ARC tasks, RL agents can learn to apply rules learned in one scenario to another, a key element of analogical reasoning. Third, RL allows agents to produce their own data, enabling effective learning even without high-quality given data. In tasks like ARC, where data may be limited, RL agents can quickly infer patterns from a few demos and learn to apply these to the new input.

This study compared model-based RL and model-free RL to evaluate the efficiency of acquiring analogical reasoning abilities. The key difference between these two methodologies lies in the presence or absence of an internal environmental model. Model-based RL builds an internal model of the environment based on the agent's experiences and learns policies through predictable scenarios, which has the potential to significantly improve decision-making in complex situations. On the other hand, model-free RL learns to optimize rewards through direct interaction with the environment, allowing for the development of effective action policies without constructing an environmental model.

This study hypothesized that the internal model of model-based RL could aid in analogical reasoning. To test this hypothesis, we observed the learning processes of ARC tasks using DreamerV3 [5], a representative model-based RL algorithm, and Proximal Policy Optimization (PPO) [6], a general model-free RL algorithm, demonstrating the potential for RL agents to acquire analogical reasoning. Additionally, we evaluated the performance and learning efficiency of both algorithms by fine-tuning them on tasks similar to the pre-trained tasks and comparing the analogical reasoning capabilities they acquired.
## 2. Model-Based RL and Model-Free RL

RL has two main categories: model-based RL and model-free RL. Before conducting experiments to compare these categories from the perspective of abstraction reasoning, this paper briefly reviews the characteristics of each category and compares the algorithms that will be used in this experiment.

### 2.1. Model-Based RL

Model-based RL builds an internal model of the environment to predict various possibilities and establish more efficient policies based on these predictions. In problems requiring abstract thinking and pattern recognition like ARC, the model-based RL approach provides the ability to generalize to new situations that the agent has not experienced through its internal model. Representative algorithms utilizing the model-based RL approach include Dyna-Q [7], I2A [8], MBMF [9], and World Model [10]. Also, AlphaZero [11], which successfully surpassed human performance in board games such as Go, Chess, and Shogi based on Monte Carlo Tree Search (MCTS), is the most well-known. Recently, DreamerV3 [12], based on the World Model, has been the subject of extensive research.

A key feature of DreamerV3 is its ability to extract important features from input data and convert these into a latent representation, which is then used to learn a predictive model of the environment's dynamics. This model functions by taking the current state and actions as inputs and predicting the next state's representation and rewards. This approach allows for the simulation of future scenarios and the evaluation of various scenarios. In summary, DreamerV3 operates by repeating the following three processes: 1) collect data based on a random policy or the current policy, 2) update the agent's model based on the collected data, and 3) generate and evaluate virtual future scenarios using the updated model to improve the policy. This process excels in multi-step tasks like mining diamonds in Minecraft. We chose DreamerV3 for its superior handling of ARC, compared to other model-based approaches.

### 2.2. Model-Free RL

Model-free RL operates directly through interactions with the external environment, basing its action policies on reward information. This methodology utilizes only the agent's experiences without prior modeling of the environment, making it relatively simple to implement and fast to execute. These characteristics could constrain the flexible reasoning and rapid adaptation required in ARC. Representative algorithms of model-free RL include DQN [13], PPO [14], and SAC [15].

Among these, the Proximal Policy Optimization (PPO) algorithm is one of the most commonly used algorithms in RL, based on the policy gradient method. PPO is designed to achieve stable and efficient learning even in complex environments. Its process can be summarized similarly to DreamerV3 in the following two steps: 1) perform actions in the environment based on the current policy to collect data, and 2) incrementally improve the policy using the policy gradient technique based on the collected data.
## 3. Experiments

We conducted an experiment comparing the ARC task learning of DreamerV3 and PPO, which are representative algorithms of model-based RL and model-free RL, respectively, to demonstrate the differences in analogical reasoning capabilities between these methodologies. Through this experiment, we aimed to answer the following research questions (RQs):

RQ1. Can Model-Based RL Learn a Single Task? ($A \to A$)

RQ2. Can Model-Based RL Reason about Tasks Similar to Pre-Trained Task? ($A \to A'$)

RQ3. Can Model-Based RL Reason about Sub-Tasks of Pre-Trained Task? ($AB \to A$)

RQ4. Can Model-Based RL Learn Multiple Tasks Simultaneously? ($A, B \to A$)

RQ5. Can Model-Based RL Reason about Merged-Tasks of Pre-Trained Tasks? ($A, B \to AB$)

We scope our paper to provide answers on RQ1 and RQ2. By addressing RQ1 and RQ2 through experiments, we aim to demonstrate the abilities of model-based RL in efficiently learning and applying knowledge across similar tasks. The following descriptions are the common setting of every experiment.

**Action** The vast action space of ARC is one of the biggest obstacles to the RL [2]. According to the ARCLE framework [16], action is defined as a combination of operation and selection; where operation means the type of action to be performed, and selection determines grids to which these actions are applied. We force strict restrictions on these operations and selections to weaken the challenge posed by the vast action space. In experiments, RL agents can select only five operations: Rotate90, Rotate270, FlipH, FlipV, and Submit [16], and selections are always fixed as the entire grid. These limitations influenced the selection of tasks.

**Task** We found tasks from among the 400 training tasks that could be solved with the restricted operations and selections of actions, and 7 tasks satisfied this condition. Among these tasks, we selected 4 tasks for our experiment as shown in Fig. 1. First, we selected tasks that need diagonal flipping on a 3 × 3 grid or $N \times N$ grids. These tasks need the right combination of actions (rotate and flip) before submission. Thus, we thought these tasks were good for measuring analogical reasoning ability. Next, we chose tasks that need to rotate a 3 × 3 grid and flip horizontally on $N \times N$ grids. These tasks need only one action except submit, so analogical reasoning ability are not necessary for agents.

**Reward** We designed the reward based on a sparse reward such that an agent gets a reward of 1000 if it does the submit action and the state matches the correct grid. Additionally, even if the submit action is not executed, reaching the correct grid grants an extra reward of 1. Although sparse rewards can present challenges in learning tasks with a wide search space, we overcame these difficulties through simple tasks and restricted action spaces. Furthermore, by giving a small reward for just finding the correct grid, the RL agent can learn the importance of achieving the goal state, and by offering a large reward for the submit action, we prevented the agent from repeatedly reaching the correct grid without submitting in a single episode.
**Metric** In all experiments, we measured the data efficiency of learning through the accuracy relative to the number of environment steps used during training. We decided to use environment steps to ensure consistency with the experimental settings of DreamerV3 [12], as we wanted to monitor data efficiency throughout the learning process. Additionally, we employed the *pass@3* method used in ARC to measure accuracy [2]. The term *pass@3* grants three submission opportunities per episode, and an episode is considered successful if the correct answer is found within three attempts. If three incorrect submissions are made or the predetermined episode length (50) is exceeded, the attempt is judged unsuccessful.

**Training** All tasks used in the experiments were trained using the RL environment for ARC, ARCLE [16]. Each task was originally composed of a few demos and one test input; however, there was a concern that this could lead to overfitting during the learning process. Therefore, we augmented the training with 1,000 demos and used an additional 100 test inputs for evaluating the model. Furthermore, when training a single task, the agent was trained over 100,000 environment steps, whereas for fine-tuning a new task on a pre-trained model, the agent was trained for 50,000 environment steps.

## 3.1. Can Model-Based RL Learn a Single Task? ($\mathcal{A} \to \mathcal{A}$)

<figure><img src="image_22.png" /><figcaption>**Figure 2:** Performance of agents on four single ARC tasks with different RL algorithms. The above two results show that the model-based RL agent learned better on analogical reasoning tasks. The below two results show that the model-free RL agent could be better on simple tasks. Additionally, an interesting common result was shown in the learning curve of model-based RL: there always occurs an interval in the middle of learning where accuracy drops to 0. We argue that this interval is where model-based RL learns concepts for analogical reasoning.</figcaption></figure>
The first experiment focused on a single ARC task requiring analogical reasoning, where the objective was to learn the common rules from given demo pairs and apply these rules effectively to a test input. This experiment involved training on four ARC tasks, each illustrated in Fig. 1, using the DreamerV3 and PPO algorithms. Tasks requiring multiple actions to solve, such as Diagonal Flipping, demand analogical reasoning, whereas tasks that can be solved with a single action, such as CCW Rotation and Horizontal Flipping, are less complex. DreamerV3, a model-based RL algorithm, exhibited superior performance in Diagonal Flip tasks, which involve more complex reasoning. In contrast, PPO, recognized for its stable learning capabilities, performed comparably or even better in simpler tasks.

It is noteworthy that DreamerV3 achieved 100% performance on uniformly sized 3 × 3 tasks but only managed about 40% on tasks with varying sizes. Conversely, PPO's performance remained consistent regardless of grid size and was influenced solely by task difficulty. This suggests a performance drop in DreamerV3 during the encoding processes.

A common interval of zero performance was observed in DreamerV3's training across various tasks, followed by rapid performance improvement. This pattern suggests that DreamerV3 is not merely learning simple problem patterns but is developing a deeper understanding of concepts essential for analogical reasoning. Initially, DreamerV3 demonstrated about 40% effectiveness due to learning straightforward patterns. Later, the agent recognized that this approach was insufficient for fully solving tasks, prompting it to explore and learn from various trials. Consequently, DreamerV3 displayed significant performance improvements in later learning stages, outperforming earlier results. These observations imply that DreamerV3's latent representations capture conceptual content that enhances learning efficiency and illustrates the potential for analogical reasoning within model-based RL frameworks.

## 3.2. Can Model-Based RL Reason Tasks Similar to Pre-Trained Task? ($\mathcal{A} \to \mathcal{A}'$)

<figure><img src="image_23.png" /><figcaption>**Figure 3:** Comparing the performance of agents between DreamerV3 and PPO on two single ARC tasks with a pre-trained model about a similar task. Understandably, PPO did not gain any benefit from fine-tuning due to the low performance of the pre-trained model. In contrast, DreamerV3 showed very high performance when adapting from a pre-trained model that had performed well in the 3 × 3 Diagonal Flip task. However, when utilizing a pre-trained model with poor performance, DreamerV3 also displayed lower initial learning efficiency than when no pre-trained model was used. Lastly, at the end of the experiment where fine-tuning was successful, a sudden drop in performance occurred, which is presumed to be the same phenomenon as the interval in previous experiments.</figcaption></figure>
The second experiment evaluated the learning efficiency of an agent when facing a task similar to one it was previously trained on. For instance, in Fig. 3a, we observed the performance of diagonal flip tasks across various grid sizes for agents pre-trained on a 3 × 3 diagonal flip task. Conversely, in Fig. 3b, we reversed the roles of the pre-trained and adaptation tasks to evaluate both agents' performances.

In both scenarios, PPO consistently displayed a performance level of around 20%. This outcome was anticipated, considering that the agent's performance on pre-trained tasks had similarly been around 20%, as evidenced in previous experiments (Fig. 2a and Fig. 2b). The insufficient performance of the pre-trained model likely inhibited any positive influence on the adaptation process, resulting in no notable performance gains compared to the non-pre-trained scenario.

An interesting finding from the DreamerV3 results was the direct impact of the pre-trained model's performance on adaptation. For example, while the model without pre-training achieved about 40% performance on the same task, the pre-trained model approached 100%, mirroring the 3 × 3 Diagonal Flip task's success previously encountered. This pattern was further supported by the performance contrasts observed between Fig. 2a and Fig. 3b. Here, a task that once achieved 100% performance dropped to around 40% after pre-training, reflecting the outcomes seen with the *N* × *N* Diagonal Flip tasks. These outcomes illustrate that DreamerV3 tailored its problem-solving strategies based on insights drawn from pre-trained tasks, suggesting that better results could have been obtained if adaptation in Fig. 3b had also been initiated from a higher-performing pre-trained model.

Towards the latter part of training in Fig. 3a, a sudden decline in performance was noted. While the exact cause of this drop cannot be definitively established, we hypothesize that it might be due to the same type of interval observed in previous results (Fig. 2). Although the experiment's step limit prevented precise verification, it seems probable that the DreamerV3 agent was undergoing an interval similar to earlier tests, attempting to learn new concepts. Further experimentation beyond 50,000 steps and additional theoretical analysis are required to explore this phenomenon more deeply, which could further substantiate DreamerV3's capability for analogical reasoning.

# 4. Discussion

**Restriction of Action Space** Our study examined the analogical reasoning abilities of reinforcement learning (RL) algorithms and their ability to apply learned concepts. In the ARCLE, an agent's actions consist of operation and selection, requiring the policy to adeptly handle both components. However, the challenge of learning the selection process significantly increases the complexity of the tasks. To mitigate this, we simplified the action space by providing selections through an oracle in this experiment. This adjustment allowed the agents to focus more on mastering operations without the added difficulty of selection, streamlining the learning process. Future research should explore the possibility of incorporating the selection process more autonomously within the reasoning tasks. This would test the agent's ability to handle the complete process of decision-making, potentially leading to more analogical reasoning ability for agents.
**Restriction of Tasks** This paper focused on learning simple tasks involving just one or two operations, such as diagonal flipping tasks, a Rotating task, and a horizontal flipping task. Future research should explore additional challenges such as the Rotate and the Flip task, which could provide valuable insights into the model's ability to generalize and reason about sub-components of learned tasks. It is also crucial to assess the efficiency of learning multiple related actions (multi-tasking) simultaneously. To overcome this challenge, previously learned knowledge must be retained while new knowledge is acquired. In other words, it is essential to be robust against catastrophic forgetting. Should such an agent be developed, two types of experiments will be essential: 1) The agent should be trained on a large and diverse set of tasks to verify the absence of catastrophic forgetting. 2) The agent should be trained on a wide range of tasks with varying difficulties, followed by an evaluation of its capability to adapt to untrained tasks.

**Applying Meta-Learning** In addressing the ARC tasks, the ability to apply learned concepts to entirely new contexts is essential. Meta-learning, or 'learning to learn,' can be considered to adapt quickly to new tasks. Future studies should focus on developing meta-learning frameworks that can efficiently abstract underlying task structures, allowing for rapid generalization across varying problem settings. A deeper exploration into meta-learning could utilize frameworks such as Model-Agnostic Meta-Learning (MAML) [17] which has shown promise in various domains. Implementing and refining meta-learning techniques like MAML could lead to breakthroughs in developing AI models that can learn not only untrained tasks more efficiently but also their knowledge to solve untrained tasks.

**Applying Transfer Learning** Transfer learning can significantly enhance the model's ability to utilize knowledge acquired in one context and apply it to different yet related tasks. This process often involves the use of deep neural networks, specifically their ability to approximate and adapt policies through layers that capture generalizable features, which are crucial for the successful transfer of knowledge across tasks within the same domain or between different domains [18]. Conducting such studies would provide deeper insights into the flexible application of acquired knowledge, a critical aspect of analogical reasoning. These investigations will pave the way for creating adaptable and efficient learning systems that thrive in dynamically changing environments.

**Theoretical Analysis of Model-Based RL** Finally, our findings indicate that model-based RL can facilitate more efficient learning in analogical reasoning tasks compared to model-free approaches. The inherent capabilities of model-based methods to infer and generalize from limited data were particularly beneficial. However, our study also highlights the need for further investigation into the specific mechanisms through which these models store and retrieve task-specific patterns. A deeper understanding of these processes could inform the development of more robust model-based systems, enhancing their capability to handle a wider array of complex reasoning tasks.
# 5. Related Works

The winner of the ARC Challenge [19] utilized a total of 142 domain-specific languages (DSLs) to combine various transformed images [20]. The ARC Challenge later expanded into ARCathon [21], and the ARCathon 2022 winner's approach involved exploring using 166 DSLs that included more complex and diverse features [22]. This attempted to challenge the broad generalization of DSLs; however, it also introduced limitations due to the inclusion of very infrequently used features, making the DSLs too complex. Compared to humans can solve about 80% of ARC evaluation tasks [23], DSL-based search algorithms have shown performances around 30-40% [4, 21]. However, these approaches, being fundamentally simple search-based, have inherent weaknesses in complex tasks and are prone to overfitting to specific tasks defined by artificially designed DSLs, making them difficult to apply or generalize to other tasks. This fundamental limitation may render them unsuitable for analogical reasoning.

Subsequent studies have attempted to solve tasks using neural networks. Some research efforts were based on program synthesis, progressively recognizing increasingly complex patterns, while other studies have tried to learn the complexities of ARC through the computational abilities of Large Language Models (LLMs). However, studies without DSLs have shown relatively low performance. Even the most recent studies based on program synthesis [24] or LLMs [25] have only achieved performances of 6.5-6.75% on untrained tasks [26]. Recently, there was interesting research utilizing human-solving processes [27], but this study did not contain the performance on untrained tasks. The low performance of these studies is attributed to the following reasons: In the case of program synthesis, the method of combining all possessed knowledge to acquire new knowledge leads to poor learning efficiency [28], and LLMs are known to be weak in incremental reasoning [29]. These limitations can be particularly harmful in evaluation tasks, which are known to be comparatively more complex and difficult than training tasks [4]. Additionally, the research utilizing human-solving processes has a risk to untrained tasks due to their reliance on offline learning from given solutions [30].

Ensemble-based research that combines the previous approaches has also been announced [4, 26]. These methods were expected to compensate for the weaknesses of existing approaches through the ensemble. An ensemble-based study combining program synthesis and LLMs showed a performance of 14.75% [26], and the approach of the ARCathon 2023 winner exhibited about 33% performance, significantly surpassing previous studies. However, the performance of ensemble-based research that combined DSL, program synthesis, and LLMs was only 40.25% [4], which is just 0.25% higher than the performance of a baseline that used only DSL. This result means that just one more task out of 400 evaluation tasks was solved. Ultimately, the experimental results of these ensemble-based studies further highlighted the limitation that the presence of program synthesis and LLMs does not significantly impact performance compared to DSL.

Meanwhile, there are some attempts to apply reinforcement learning (RL) algorithms in a limited way based on program synthesis and LLMs methods [26, 31]. The limited application of RL algorithms can be attributed to factors such as the absence of appropriate rewards, high-dimensional complex states, and extensive search spaces, but fundamentally, the absence of an environment for ARC was the main reason. However, an environment for training ARC tasks [16] has been developed recently. Furthermore, this research demonstrated successful training of an ARC task using PPO, suggesting the potential applicability of various RL models.
# 6. Conclusion

In this paper, we compared analogical reasoning abilities between model-based and model-free RL, using a subset of ARC to demonstrate the effectiveness of model-based RL. This approach has shown superior learning efficiency and adaptability in tasks compared to model-free methods like PPO. Specifically, DreamerV3 outperformed in tasks similar to those it had previously solved, displaying a remarkable ability to generalize across various task dimensions.

The observed performance dips followed by rapid recoveries during DreamerV3's training are thought to represent periods of conceptual consolidation. These intervals align with theoretical expectations of analogical reasoning in AI systems and highlight DreamerV3's potential to manage complex cognitive tasks. Future research should further explore these learning dynamics to better understand the mechanisms that enable such advanced reasoning. This will help enhance RL agents capable of analogical reasoning and adaptation in dynamic environments, reducing the gap between the reasoning ability of humans and RL agents.

# Acknowledgments

This work was supported by the IITP (No. 2019-0-01842, RS-2023-00216011), AICA (HPC-AI) and the GIST (AI-based Research Scientist Project, HPC-AI) funded by the Ministry of Science and ICT, Korea. QCT also supported computing servers for this work.

# References

[1] M. Małkiński, J. Mańdziuk, A Review of Emerging Research Directions in Abstract Visual Reasoning, Information Fusion 91 (2023) 713–736.

[2] F. Chollet, On the Measure of Intelligence, CoRR abs/1911.01547 (2019).

[3] D. R. Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid, Basic books, 1979.

[4] M. Bober-Irizar, S. Banerjee, Neural Networks for Abstraction and Reasoning: Towards Broad Generalization in Machines, CoRR abs/2402.03507 (2024).

[5] D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap, Mastering Diverse Domains through World Models, CoRR abs/2301.04104 (2023).

[6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization algorithms, CoRR abs/1707.06347 (2017).

[7] R. S. Sutton, Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming, in: ICML, 1990, pp. 216–224.

[8] S. Racanière, T. Weber, D. Reichert, L. Buesing, A. Guez, D. Jimenez Rezende, A. Puigdomènech Badia, O. Vinyals, N. Heess, Y. Li, et al., Imagination-Augmented Agents for Deep Reinforcement Learning, in: NeurIPS, 2017.

[9] A. Nagabandi, G. Kahn, R. S. Fearing, S. Levine, Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning, in: ICRA, 2018, pp. 7559–7566.

[10] D. Ha, J. Schmidhuber, World Models, CoRR abs/1803.10122 (2018).
[11] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al., Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, CoRR abs/1712.01815 (2017).

[12] D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap, Mastering Diverse Domains through World Models, CoRR abs/2301.04104 (2023).

[13] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller, Playing Atari with Deep Reinforcement Learning, CoRR abs/1312.5602 (2013).

[14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, CoRR abs/1707.06347 (2017).

[15] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, in: ICML, 2018, pp. 1861–1870.

[16] H. Lee, S. Kim, S. Lee, S. Hwang, J. Lee, B.-J. Lee, S. Kim, ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learning, in: CoLLAs, 2024. URL: https://openreview.net/forum?id=EE33zbaLwa.

[17] C. Finn, P. Abbeel, S. Levine, Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, in: ICML, 2017, pp. 1126–1135.

[18] M. E. Taylor, P. Stone, Transfer Learning for Reinforcement Learning Domains: A Survey, Journal of Machine Learning Research 10 (2009).

[19] Kaggle, Abstraction and Reasoning Challenge, 2020. URL: https://www.kaggle.com/c/abstraction-and-reasoning-challenge/.

[20] J. S. Wind, DSL Solution to the ARC Challenge, 2020. URL: https://github.com/top-quarks/ARC-solution/.

[21] Lab42, ARCathon: Global AI Competition, 2024. URL: https://lab42.global/arcathon/.

[22] M. Hodel, Domain Specific Language for the Abstraction and Reasoning Corpus, 2023. URL: https://github.com/michaelhodel/arc-dsl.

[23] A. Johnson, W. K. Vong, B. M. Lake, T. M. Gureckis, Fast and Flexible: Human Program Induction in Abstract Reasoning Tasks, in: CogSci, 2021, pp. 2471–2477.

[24] J. Ainooson, D. Sanyal, J. P. Michelson, Y. Yang, M. Kunda, A Neurodiversity-Inspired Solver for the Abstraction & Reasoning Corpus (ARC) Using Visual Imagery and Program Synthesis, CoRR abs/2302.09425 (2023).

[25] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, A. Zeng, Large Language Models as General Pattern Machines (2023).

[26] N. Butt, B. Manczak, A. Wiggers, C. Rainone, D. Zhang, M. Defferrard, T. Cohen, CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay, CoRR abs/2402.04858 (2024).

[27] J. Park, J. Im, S. Hwang, M. Lim, S. Ualibekova, S. Kim, S. Kim, Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformer, in: ICML ILHF Workshop, 2023.

[28] K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, L. Morales, L. Hewitt, L. Cary, A. Solar-Lezama, J. B. Tenenbaum, Dreamcoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning, in: PLDI, 2021, pp. 835–850.

[29] S. Lee, W. Sim, D. Shin, S. Hwang, W. Seo, J. Park, S. Lee, S. Kim, S. Kim, Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus, CoRR abs/2403.11793 (2024).
[30] J. C. M. Tan, M. Motani, Large Language Model (LLM) as a System of Multiple Expert Agents: An Approach to Solve the Abstraction and Reasoning Corpus (Arc) Challenge, CoRR abs/2310.05146 (2023).

[31] S. Alford, A. Gandhi, A. Rangamani, A. Banburski, T. Wang, S. Dandekar, J. Chin, T. Poggio, P. Chin, Neural-Guided, Bidirectional Program Search for Abstraction and Reasoning, in: Complex Networks, 2021, pp. 657–668.
# Towards a unified framework of numerical analogies: Open questions and perspectives

Yves Lepage<fnref n="1" />*, Miguel Couceiro<fnref n="2" />,<fnref n="3" />

**Abstract**

A recently-proposed framework for modeling analogies exploits tight relationships between analogies and generalized means, and unifies different models of numerical analogies. In this contribution we further exploit this framework and discuss pertaining questions such as invariance properties, the semantics and the expressive power of such analogies. We also explore further topics related to equivalence classes and invariance properties, generalized mean approximations, manifold representations, as well as algorithmic aspects and machine learning applications such as machine classification.

**Keywords**

Analogy, Analogical inference, Formalization, Case-Based Reasoning, Knowledge Representation

## 1. Introduction and motivation

Analogical reasoning (AR) is a remarkable capability of human thought that exploits parallels between situations of different nature to infer plausible conclusions, by relying simultaneously on similarities and dissimilarities. Machine learning (ML) and artificial intelligence (AI) have tried to develop AR but the early works lacked theoretical and formalization support or are limited to Booleans. This situation started to change two decades ago when researchers adopted the view of the so-called analogical proportions as statements of the form “*a* relates to *b* as *c* relates to *d*”, usually denoted *a*: *b* :: *c*: *d*. Such proportions are at the root of the analogical inference mechanism, and several formalisms to study this mechanism have been proposed, which follow different axiomatic and logical approaches [1, 2, 3]. The study of relationships between two pairs of objects *A* and *B*, and of objects *C* and *D*, can focus on different aspects. For instance, one may want to judge whether the relationship between *C* and *D* is the same as that between *A* and *B*, and assess the quality of such relationships, or one may discuss the quality of attribute or relationship similarities [4], following the foundational work by Gentner [5]. One may also see *A* and *C* as problems, *B* as a solution to problem *A*, and ask whether the transposition of the ratio of *A* to *B* on *C* generates a *D*, and to what extent the

IARML@IJCAI’2024: Workshop on the Interactions between Analogical Reasoning and Machine Learning, at IJCAI’2024, August, 2024, Jeju, South Korea

*Corresponding author.

<figure><img src="image_24.png" /></figure>

yves.lepage@waseda.jp (Y. Lepage); miguel.couceiro@loria.fr (M. Couceiro)

<figure><img src="image_25.png" /></figure>

0000-0002-3059-4271 (Y. Lepage); 0000-0003-2316-7623 (M. Couceiro)

<figure><img src="image_26.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_27.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)

<footnote n="1">¹IPS, Waseda University, Japan</footnote>

<footnote n="2">²LORIA, Université de Lorraine, France</footnote>

<footnote n="3">³INESC-ID, IST, Universidade de Lisboa, Portugal</footnote>
generated *D* is a solution to problem *C*.

There are two basic tasks associated with AR. The first is analogy making that corresponds to the task of detecting and deciding whether a quadruple (*a*, *b*, *c*, *d*) corresponds to a valid analogical proportion. The second is analogy solving that refers to the task of finding or extrapolating, for a given triple *a*, *b*, *c* the value *x* such that *a*: *b*:: *c*: *x* is a valid analogy. This task is typically addressed in the literature by retrieval and adaptation, i.e., defining an *x* from a pool of retrieved candidate solutions to be suitably adapted. In fact, analogy solving is somewhat related to case-based reasoning (CBR) [6, 7, 8] where, given a set *P* of problems, a set *S* of solutions and a set *C* of cases (*x*, *y*) ∈ *P* × *S*, the CBR task is to find a solution *y*_t to a given target problem *x*_t. CBR basically consists in (1) selecting *k* source cases in the case base according to some criteria related to the target problem (retrieval step), and (2) reusing the *k* retrieved cases for proposing a target solution (adaptation step). For *k* = 1, the desired solution *y*_t corresponds to the solution of the analogical equation *x*: *y*:: *x*_t: *y*_t. For higher values of *k*, different models of analogy on *P* and *S* could be taken into account. For instance, when *k* = 3, the retrieval task consists in finding a triple of cases (*x*₁, *y*₁), (*x*₂, *y*₂) and (*x*₃, *y*₃), such that *x*₁: *x*₂:: *x*₃: *x*_t is valid and such that *y*₁: *y*₂:: *y*₃: *y* is solvable in *y*. In this setting the desired *y*_t would then be one of such solutions [9].

The latter idea was extended to analogy based classification [10] where objects are viewed as attribute tuples (instances) **x** = (*x*₁, ..., *x*ₙ). Here, analogical inference relies on the idea that if four instances **a**, **b**, **c**, **d** are in analogical proportion for most of the attributes describing them, then it may still be the case for the other attributes. Similarly, if class labels are known for **a**, **b**, **c** and unknown for **d**, then one may infer the label for **d** as a solution of an analogical proportion equation. Theoretically, it is quite challenging to find and characterize situations where such an **analogical inference principle** (AIP) can be soundly applied, but there have been several efforts in this direction [11, 12, 13]. In fact, the latter culminated recently in a Galois theory of analogical classifiers [14] illustrated over classical numerical analogies. In fact, the principle of analogical inference has been integrated into various machine learning tasks, e.g. preference learning and recommendation [15, 16, 17], and used to solve difficult reasoning tasks such as academic aptitude tests, visual question answering and target sense verification [18, 19, 20, 21], as well as support computational creativity [22], analogical transfer [23, 24], and explainable AI [25, 26].

Recently, analogies and AR became quite popular due to the successes of deep learning together with distributional representations (embeddings). For instance, [27, 28] show that vector representations of quadruples respecting certain linear transformations satisfy common properties of analogies, whereas [29] unveils the potential analogies as a benchmark to evaluate the quality of embedding models. This exploration has extended to complex structures such as knowledge graphs (KG) over multimodal domains, to address tasks such as named entity recognition, link prediction, relation discovery (abduction) and KG bootstrapping and completion [30, 31, 32], by leveraging multimodal knowledge embeddings [33, 34, 35, 36]. Despite these impressive results by such analogy-based approaches in rather complex tasks, many works have questioned the retrieved analogical relations, their dependency and limitation with respect to the underlying representation model, and even the evaluation procedure [37, 38]. Many works have been advocating for foundational mathematical frameworks and experiments to gain a better understanding of the analogical capabilities of embedding models as well as recent large language models [39, 40, 41, 42].
In this paper we revisit the recent unified framework of numerical analogies proposed in [43]. It is rooted in the intuitive and classical idea of “analogy”, where a quadruple (A, B, C, D) is said to constitute a valid analogy, denoted by A : B :: C : D, if the “mean of the extremes A and D is equal to the mean of the means<fnref n="1" /> B and C”. Here, we consider a generalized notion of mean that goes back to Hölder [44]. (see Section 2). Endowed with this notion, the authors of [43] introduced the so-called analogies in power p (see Section 2), and showed several noteworthy and rather surprising results, namely, that

1. any sequence of four increasing positive real numbers is an analogy in a unique suitable power,

2. any such analogy can be reduced to an equivalent arithmetic analogy, and

3. any analogical equation has a solution for increasing numbers.

In this position paper, we survey these and other results, and further explore the potential of the unifying framework proposed in [43]. In particular, we will address mathematical challenges revealed by this new analogy formalization, and discuss potential impacts in algorithmic approaches as well as downstream ML tasks. The paper is thus organized as follows. In Section 2 we briefly recall the framework proposed in [43] to formalize analogies. We then summarize a few surprising results in Section 3. The discussion on the open problems and perspectives is carried out in Section 4, and divided into mathematical challenges (Subsection 4.1) and ML impacts (Subsection 4.2).

**Notation and terminology.** We will use lower case letters *a*, *b*, *c* and *d* when the terms of the analogy are numbers. We will also use $\mathbb{R}$, $\mathbb{R}^*$, $\mathbb{R}^+ \setminus \{0\}$ and $\mathbb{C}$ to denote the sets of real numbers, real numbers without 0, positive numbers and complex numbers, respectively.

## 2. Generalized means and analogy

Recall that the *p*-generalized mean of real positive numbers $x_1, \dots, x_N$ is

$$m_p(x_1, \dots, x_N) = \lim_{r \to p} \sqrt[r]{\frac{1}{N} \sum_{i=1}^{N} x_i^r}$$

for all $p \in [-\infty, +\infty]$. Note that this notion subsumes the classical notions of:

* arithmetic mean:

$$m_1(x_1, \dots, x_N) = \frac{1}{N} \sum_{i=1}^{N} x_i;$$

* harmonic mean:

$$m_{-1}(x_1, \dots, x_N) = \frac{1}{\frac{1}{N} \sum_{i=1}^{N} \frac{1}{x_i}};$$

<footnote n="1">¹In this paper the term *mean* will be understood in two different ways. The *means* in an analogy will refer to the terms *B* and *C*, while *generalized means* of two numbers will refer to the notion of an average. Hopefully, the context will be clear and will not lead to any ambiguity.</footnote>
<figure><img src="image_28.png" /><figcaption>Figure 1: Visualization of generalized means of a = 2 and d = 5.</figcaption></figure>

• root mean square:

$$m_2(x_1, \dots, x_N) = \sqrt[2]{\frac{1}{N} \sum_{i=1}^{N} x_i^2};$$

• geometric mean:

$$\lim_{p \to 0} m_p(x_1, \dots, x_N) = \sqrt[N]{\prod_{i=1}^{N} x_i};$$

• maximum:

$$\lim_{p \to +\infty} m_p(x_1, \dots, x_N) = \max(x_1, \dots, x_N);$$

• minimum:

$$\lim_{p \to -\infty} m_p(x_1, \dots, x_N) = \min(x_1, \dots, x_N).$$

The generalized mean of two numbers a and d reduces to

$$m_p(a, d) = \lim_{r \to p} \sqrt[r]{\frac{1}{2}(a^r + d^r)},$$

for all $p \in \mathbb{R}$, and even for p equal to $-\infty$ or $+\infty$. Figure 1 illustrates the curve obtained for the particular values a = 2 and d = 5. It is noteworthy that the generalized mean function of two numbers does not behave like an odd function with respect to p. Although defined above for positive reals, this can be generalized to the case when a, d and p are complex numbers, except in some undefined cases. Figure 2 shows the curve obtained for the generalized means of two complex numbers $-20 + i20$ and $50 + i70$.

Based on this notion of generalized means, the authors of [43] proposed the following unified notion of numerical analogy.
<figure><img src="image_29.png" /><figcaption>**Figure 2:** Generalized means for the two complex numbers $a = -20 + i20$ and $d = 50 + i70$. The color is supposed to reflect the transition from $p = -\infty$ in blue on $a$ up to $p = +\infty$ in red on $d$, through all values in $\mathbb{R}$. The points around $p = 0$ are in green and they visibly occupy the main part of the curve. The point for $p = 0$. i.e., the geometric mean, is located near the vertical axis. The harmonic ($p = -1$) and the arithmetic means ($p = 1$) are also indicated.</figcaption></figure>

**Definition 1.** Let $a, b, c, d \in \mathbb{R}^+ \setminus \{0\}$, and $p \in \mathbb{R}$. The analogy in power $p$ is defined as follows:

$$a : b ::^p c : d \stackrel{\text{def.}}{\Leftrightarrow} m_p(a, d) = m_p(b, c) \Leftrightarrow \lim_{r \to p} \sqrt[r]{\frac{1}{2}(a^r + d^r)} = \lim_{r \to p} \sqrt[r]{\frac{1}{2}(b^r + c^r)}. \quad (1)$$

This definition generalizes the classical arithmetic, geometric, or harmonic analogies, as illustrated in Table 1. Note that when $p \in \mathbb{R}^*$, the limits in (1) are not necessary, neither is taking the $p$-th root, the one-half factor can be eliminated and demonstrations can then directly exploit the formula $a^p + d^p$ or the equality $a^p + d^p = b^p + c^p$. In the case $p = 0$, the simplified versions of the formulas for the geometric mean are used: $a \times d$ or $a \times d = b \times c$. Finally, in the two infinite cases, formulas with min and max are used.

The above defined relation between four terms defines a mathematical analogy in the expected sense of the term, i.e., firstly, $::$ is a dependence relation (reflexivity and symmetry). Indeed, transitivity is also verified so that, $::$ is an equivalence relation in this case. Secondly, it is easy to verify that the permutation of the means $a : c :: b : d$ and the inverse of the ratios $b : a :: d : c$ hold, which implies in total eight equivalent forms, that classical formalizations of mathematical analogy allow writing for the same analogy, by playing on the position of the terms:

$$\begin{align*}
a : b :: c : d & \qquad b : a :: d : c & \qquad c : a :: d : b & \qquad d : b :: c : a \\
a : c :: b : d & \qquad b : d :: a : c & \qquad c : d :: a : b & \qquad d : c :: b : a
\end{align*}$$
<caption>**Table 1**</caption>

Some illustrative examples subsuming instances of classical models of numerical analogies.

<table><thead><tr><th>Data</th><th>Equality of means</th><th>Analogy</th><th>Name of analogy</th></tr></thead><tbody><tr><td>5, 7, 10, 12</td><td><sup>1</sup>&frasl;<sub>2</sub>(5 + 12) = <sup>1</sup>&frasl;<sub>2</sub>(7 + 10)</td><td>5 : 7 ::<sup>1</sup> 10 : 12</td><td>Arithmetic</td></tr><tr><td>2, 4, 8, 16</td><td>&radic;<span style="text-decoration: overline;">2 &times; 16</span> = &radic;<span style="text-decoration: overline;">4 &times; 8</span></td><td>2 : 4 ::<sup>0</sup> 8 : 16</td><td>Geometric</td></tr><tr><td>8/5, 2, 4, 8</td><td><sup>1</sup>&frasl;<sub>2</sub>(<sup>1</sup>&frasl;<sub>2</sub>(<sup>1</sup>&frasl;<sub>8</sub> + <sup>1</sup>&frasl;<sub>8</sub>)</td><td><sup>8</sup>&frasl;<sub>5</sub> : 2 ::<sup>-1</sup> 4 : 8</td><td>Harmonic</td></tr><tr><td>2, 2, 5, 67</td><td>min(2, 67) = min(2, 5)</td><td>2 : 2 ::<sup>-&infin;</sup> 5 : 67</td><td>By minimum value</td></tr></tbody></table>

### 3. Main results of Lepage & Couceiro [43]

The main contributions of article [43] can be summarized in three rather surprising results. The first one states that analogies exist and are unique between any quadruple of positive real numbers in increasing order (the numbering of the results does not reflect the importance but the order afterwards).

**Theorem 2.** Given four increasing positive real numbers, all different, we can always see an analogy between them and this analogy is unique.

Note that the most common form of analogy between numbers is the arithmetic analogy, which is just the particular case $p = 1$. In addition to Theorem 2, the second main result states that any analogy between four positive reals can be thought of as an arithmetic analogy, as it can always be transformed into an arithmetic analogy.

**Theorem 3.** Any analogy in $p$ between four positive real numbers can be reduced to an arithmetic analogy.

As vector representations in machine learning make use of real numbers, the main results are presented with respect to real numbers, but extensions are possible. In particular, the third main result states that every analogical equation of the form $A : B :: C : x$ is solvable over the complex numbers.

**Theorem 4.** Conditioned on $p$, it is possible to solve any analogical equation with complex terms.

This mathematical formalism to model analogies reveals several challenges, both of mathematical and of ML nature, that we will discuss in the next section.
# 4. Perspectives opened by this formalization

We will discuss some challenges and perspectives opened by this formalization of numerical analogies. We identify two main directions of future work that correspond to the following sections.

## 4.1. Mathematical aspects

**Equivalence classes.** From the definition of $::^p$, it is not difficult to verify that for, $p \neq 0, +\infty, -\infty$, the relation $::^p$ is reflexive, symmetric and transitive, and thus constitutes an equivalence relation. It follows from Theorem 2 that the corresponding equivalence classes are in bijection. Furthermore, the authors of [43] also showed that $::^p$ remains an equivalence relation in the cases $p = +\infty, -\infty$, and that the corresponding equivalence classes remain in bijection. However, this situation seems to change for the case $p = 0$.

It is thus natural to ask for a structural study of these equivalence classes. The answer to this question would shed some light into potential axiomatizations of $::^p$ and some of the challenges below. For instance, we could determine and transfer invariance results between equivalence classes (see below). Also, it would make clear the meaning of $p$ (see Subsection 4.2). Furthermore, one may ask whether such a trichotomy with respect to $p$ still holds in the complex case.

**Invariance properties.** Let $R \subseteq S^n$ be an $n$-ary relation over $S \in \{\mathbb{R}, \mathbb{R}^*, \mathbb{R}^+ \setminus \{0\}, \mathbb{C}\}$, and let $f: S \to S$. Function $f$ is said to be a homomorphism of $R$, and relation $R$ is said to be an invariant<fnref n="2" /> under $f$ if for every $a = (a_1, a_2, \dots, a_n) \in R$, we have

$$f(\mathbf{a}) = (f(a_1), f(a_2), \dots, f(a_n)) \in R.$$

For instance, it was observed in [45, 43] that any affine transformation (translation) leaves the analogy invariant for $p = 1$. Similarly, the multiplication by any positive scalar leaves analogies invariant for any real $p$.

These facts ask for the connection between $p$ and invariance properties, e.g., what are the classes of $::^p$ invariant under a certain class of functions or, conversely, given a $p$, what is the class of homomorphisms. These questions can be extended to higher order relations with an immediate application in the study of analogical classifiers (see Subsection 4.2). Also, taking the “functional” definition of analogy under which $a : b :: c : d$ holds whenever there is a transformation $t$ such that $b = t(a)$ and $d = t(c)$, one may want to characterize the functions that leave such analogies invariant. Moreover, we propose to extend these questions to the richer realm of complex numbers.

**Behavior of analogies on manifolds.** Word embeddings are sets of vector representations of words based on the distributional hypothesis that the semantics of a word is characterized by its context. This has been implemented in efficient methods like Word2Vec and its successors. It has been claimed that, in such a space, analogies can be formalized by an arithmetic relation and theoretical arguments have been given in support of this [28, 46]. The Google analogy test

<footnote n="2">²Note that these notions naturally extend to any underlying set $S$.</footnote>
set [47] or BATS [39] are collections of analogies grouped by types of relations. For instance, the capital-common-country type contains analogies like: *Paris : France :: Berlin : Germany* or *Tehran : Iran :: Havana : Cuba*. We wonder whether significant information can be drawn from the observation of the various analogical powers that one can compute on each different dimension of each different analogy of the same type. Because the set of words contained in such a group of analogies can be seen as defining, by extension, a manifold in the representation space, an open question is whether observations on the analogical powers would allow to characterize this manifold.

Another way of addressing the above question is to have a view that separates the terms in an analogy. To illustrate with an example, given a number of points (e.g., a certain number of masculine titles like *emperor, count, duke and king*) and some corresponding points (feminine titles: *empress, countess, duchess*), one may try to characterize not only each of the two manifolds for these two sets of words, but also the correspondance between these manifolds, or even the manifold that separates the two sets of words, by reminiscence with classification and SVMs. How would that allow to deduce, on the basis of analogical powers, the fact that the last missing point in the list above should be *queen*?

## 4.2. Machine learning: semantics and practical aspects

**Meaning of p.** As suggested in the previous section about equivalence classes, a question is whether a meaning can be assigned to *p*. It comes naturally to mind that it could possibly reflect some strength of the analogy. This stems from the fact that the two powers $-∞$ and $+∞$ do not have exactly the same properties as the other powers. In particular, an analogical equation in $-∞$ or $+∞$ looks more relaxed than in other powers because, there may exist an infinite number of solutions. For instance $2 : 2 :: -∞$ 3 : *x* admits any value for *x* greater than 3 as a solution. Although it seems that this is a particular behaviour for the infinities, it also seems that, depending on the kind of data at hand, some analogical powers might have a heavier signification than, say other analogical powers. For instance, it could be the case in word embedding spaces, where arithmetic analogies are being felt as the most reasonable kind of analogy based on theoretical considerations [46], the dimensions on which the analogical power *p* departs greatly from 1 might be considered as less representative of the essence of the relationship expressed by the word analogy.

**Algorithmic aspects** In the current state of our implementation, the analogical power for a quadruple of real numbers is determined with a certain precision by dichotomic search. We implemented a Python package that wraps C functions.<fnref n="3" /> For a million quadruples of increasing positive real numbers drawn at random, the determination of the analogical power with a precision of $10^{-4}$ takes 2.3 seconds on an Apple M1 processor (3.2 GHz). Acceleration being always wanted, practical as well as theoretical questions can be considered.

Firstly, in practice, one may inspect whether exploiting some properties of invariance, e.g., by multiplication or reduction to arithmetic analogy, allows the use of tabular techniques to accelerate computation. Tables of pre-computed cases could be loaded into memory to allow

<footnote n="3">³http://lepage-lab.ips.waseda.ac.jp/projects/Kakenhi Project_21K12038/ > Experimental results > Python packages</footnote>
for fast access. In conjunction with such tabulation techniques, the use of quantization can also
be considered.

Secondly, more theoretically, one can ask whether the use of analytic forms of approximating
functions can be exploited. The analytic expression of the generalized mean function itself is
reasonably simple, but its derivative and in particular the derivative of differences, as needed
in analogy, have rather complex expressions that would entail heavy computations if one
would envisage gradient techniques. By contrast, the “popularity” of the sigmoid function as
an activation function in neural networks is to be found in its simple analytic properties, in
particular the simplicity of its derivative $\sigma'(x) = \sigma(x) \times (1 - \sigma(x))$, and the existence of a
central symmetry $\sigma(-x) = 1 - \sigma(x)$, something that the generalized mean function does not
have. It is however undeniable that, despite the previous remark, the generalized mean function
looks very much alike the sigmoid function. In the hope of faster computation to determine
analogical powers, one can ask whether approximations of the generalized mean function by,
e.g., the sigmoid function, could lead to accelerations. This first entails the determination of
suitable parameters to fit the sigmoid function onto the generalized mean function. The next
task is to estimate the error under such an approximation and to quantify the trade-off between
such an error and the acceleration, if any is obtained.

**Relationships between invariance properties and classification** As discussed in the Introduction, analogy solving and analogical transfer are tightly related to case-based reasoning (CBR) [8]. In particular, we mentioned the case where retrieval consists in finding a triple of cases such that analogy holds in the problem domain, and finding a solution consists in solving the corresponding analogical equation in the solution domain. This idea was extended to analogy based classification [10] where objects are viewed as attribute tuples (instances) x = (x₁, ..., xₙ). Here, analogical inference relies on the idea that if four instances a, b, c, d are in analogy for most of the attributes describing them, then it may still be the case for the other attributes. Similarly, if class labels are known for a, b, c and unknown for d, then one may infer the label for d as a solution of an analogical equation. Theoretically, it is quite challenging to find and characterize situations where such an *analogical inference principle* (AIP) can be soundly applied. In case of Boolean attributes, a first step for explaining the analogical mechanism was to characterize the set of functions for which AIP is sound (i.e., no error occurs) no matter which triplets of examples are used. In [12], it was shown that these so-called “analogy-preserving” (AP) functions coincide exactly with the set of affine Boolean functions. Moreover, when the function is close to being affine, it was also shown that the prediction accuracy remains high [13]. These results were extended in [11] to nominal domains when taking the *minimal model* of analogy, i.e., only patterns of the form x : x :: y : y and x : y :: x : y.

The latter were unified into a Galois theory of analogical classifiers [14] by exploiting the notion of “polymorphism”, in which formal models of analogy are put in a two-way correspondence with classifiers compatible with the analogical inference principle. To some extent, this reflects invariance properties of higher-order, in which homomorphisms may be *n*-ary, for *n* ≥ 2. This asks for revisiting the Galois framework of [14] under *Defintion 1*, and obtaining explicit descriptions of the Galois sets of analogical classifiers, possibly parametrized by *p*.
# 5. Conclusion

In this position paper, we surveyed the recently proposed formalization of numerical analogy, more precisely on positive real numbers, with its definition and the three main results associated with it. This framework opens new avenues in two main directions. The first direction we identified concerns the mathematical aspects for machine learning pertaining to invariance properties and the behavior of analogies. The second one pertains to theoretical and practical aspects, such as the semantics, explainability considerations, and impacts on downstream ML tasks.

# Acknowledgments

This work is the result of a collaboration made possible by the first author's sabbatical stay with the second author's organization. The authors would like to thank Waseda University Research Promotion Section and the LORIA International Chair for their funding.

This work benefited from a research grant from the Japan Society for the Promotion of Science, Kiban C, n° 21K12038, entitled "Theoretically grounded algorithms for the automatic extraction of analogy test sets in natural language processing" and was partially supported by the ANR project "Analogies: from theory to tools and applications" (AT2TA), ANR-22-CE23-0023.

# References

[1] Y. Lepage, Analogy and formal languages, Electron. Notes Theor. Comput. Sci. 53 (2001) 180-191.

[2] L. Miclet, S. Bayoudh, A. Delhay, Analogical dissimilarity: Definition, algorithms and two experiments in machine learning, J. Artif. Intell. Res. 32 (2008) 793-824.

[3] H. Prade, G. Richard, Analogical proportions: Why they are useful in AI, in: Proc. IJCAI-21, 2021, pp. 4568-4576. Survey Track.

[4] P. D. Turney, Similarity of semantic relations, Computational Linguistics 32 (2006) 379-416.

[5] D. Gentner, Structure mapping: A theoretical model for analogy, Cognitive Science 7 (1983) 155-170.

[6] F. Badra, M. Lesot, Coat-apc: When analogical proportion-based classification meets case-based reasoning, in: ATA@ ICCBR 2022, volume 3389 of *CEUR Workshop Proceedings*, Nancy, France, 2022, pp. 43-56.

[7] F. Badra, M.-J. Lesot, E. Marquer, M. Couceiro, Some Perspectives on Similarity Learning for Case-Based Reasoning and Analogical Transfer, in: IARML@IJCAI 2023, volume 3492, CEUR-WS.org, 2023, pp. 16-29.

[8] J. Lieber, E. Nauer, H. Prade, When revision-based case adaptation meets analogical extrapolation, in: ICCBR 2021, volume 12877 of LNCS, Springer, 2021, pp. 156-170.

[9] Y. Lepage, E. Denoual, Purest ever example-based machine translation: detailed presentation and assessment, Machine Translation 19 (2005) 251-282.

[10] M. Bounhas, H. Prade, G. Richard, Analogy-based classifiers for nominal or numerical data, IJAR 91 (2017) 36-55.
[11] M. Couceiro, E. Lehtonen, L. Miclet, H. Prade, G. Richard, When nominal analogical proportions do not fail, in: SUM 2020, volume 12322 of LNCS, Springer, 2020, pp. 68–83.

[12] M. Couceiro, N. Hug, H. Prade, G. Richard, Behavior of Analogical Inference w.r.t. Boolean Functions, in: IJCAI 2018, 2018, pp. 2057–2063.

[13] M. Couceiro, N. Hug, H. Prade, G. Richard, Analogy-preserving functions: A way to extend boolean samples, in: IJCAI 2017, 2017, pp. 1–7.

[14] M. Couceiro, E. Lehtonen, Galois theory for analogical classifiers, Ann. Math. Artif. Intell. 92 (2024) 29–47.

[15] M. A. Fahandar, E. Hüllermeier, Learning to rank based on analogical reasoning, in: AAAI-18, 2018, pp. 2951–2958.

[16] M. A. Fahandar, E. Hüllermeier, Analogical embedding for analogy-based learning to rank, in: IDA 2021, volume 12695 of LNCS, Springer, 2021, pp. 76–88.

[17] M. Mitchell, Abstraction and analogy-making in artificial intelligence, Annals of the New York Academy of Sciences 1505 (2021) 79–101.

[18] Y. Bitton, R. Yosef, E. Strugo, D. Shahaf, R. Schwartz, G. Stanovsky, VASR: visual analogies of situation recognition, in: (AAAI 2023) and (IAAI 2023) and (EAAI 2023), AAAI Press, Washington, DC, USA, 2023, pp. 241–249.

[19] J. Peyre, I. Laptev, C. Schmid, J. Sivic, Detecting unseen visual relations using analogies, in: IEEE ICCV 2019, 2019, pp. 1981–1990.

[20] F. Sadeghi, C. L. Zitnick, A. Farhadi, Visalogy: Answering visual analogy questions, in: NIPS 2015, 2015, pp. 1882–1890.

[21] G. Zervakis, E. Vincent, M. Couceiro, M. Schoenauer, E. Marquer, An analogy based approach for solving target sense verification, in: NLPIR 2022, ACM, 2022, pp. 144–151.

[22] A. K. Goel, Computational design, analogy, and creativity, in: Computational Creativity - The Philosophy and Engineering of Autonomously Creative Systems, Springer, 2019, pp. 141–158.

[23] A. Cornuéjols, P.-A. Murena, R. Olivier, Transfer learning by learning projections from target to source, in: IDA 2020, volume 12080 of LNCS, Springer, 2020, pp. 119–131.

[24] S. Alsaidi, A. Decker, P. Lay, E. Marquer, P.-A. Murena, M. Couceiro, On the transferability of neural models of morphological analogies, in: AIMLAI 2021, volume 1 of PKDD/ECML Workshops, Bilbao/Virtual, Spain, 2021, pp. 76–89.

[25] E. Hüllermeier, Towards analogy-based explanations in machine learning, in: MDAI 2020, volume 12256 of LNCS, Springer, 2020, pp. 205–217.

[26] M. T. Keane, B. Smyth, Good counterfactuals and where to find them: A case-based technique for generating counterfactuals for explainable AI (XAI), in: ICCBR 2020, volume 12311 of LNCS, Springer, 2020, pp. 163–178.

[27] A. Drozd, A. Gladkova, S. Matsuoka, Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen, in: COLING 2016, 2016, pp. 3519–3530.

[28] T. Mikolov, W. Yih, G. Zweig, Linguistic regularities in continuous space word representations, in: HLT-NAACL 2013, 2013, pp. 746–751.

[29] T. Schnabel, I. Labutov, D. M. Mimno, T. Joachims, Evaluation methods for unsupervised word embeddings, in: EMNLP 2015, 2015, pp. 298–307.

[30] A. Ushio, L. E. Anke, S. Schockaert, J. Camacho-Collados, BERT is to NLP what alexnet is to CV: can pre-trained language models identify analogies?, in: ACL/IJCNLP 2021, volume 1,
2021, pp. 3609–3624.

[31] N. Zhang, L. Li, X. Chen, X. Liang, S. Deng, H. Chen, Multimodal analogical reasoning over knowledge graphs, in: ICLR 2023, 2023, pp. 1–20.

[32] L. Jarnac, M. Couceiro, P. Monnin, Relevant entity selection: Knowledge graph bootstrapping via zero-shot analogical pruning, in: CIKM 2023, ACM, 2023, pp. 934–944.

[33] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, in: ICML 2016, volume 48 of JMLR, 2016, pp. 2071–2080.

[34] R. Xie, Z. Liu, H. Luan, M. Sun, Image-embodied knowledge representation learning, in: IJCAI 2017, ijcai.org, 2017, pp. 3140–3146.

[35] Z. Wang, L. Li, Q. Li, D. Zeng, Multimodal data enhanced representation learning for knowledge graphs, in: IJCNN 2019, IEEE, 2019, pp. 1–8.

[36] M. Wang, S. Wang, H. Yang, Z. Zhang, X. Chen, G. Qi, Is visual context really helpful for knowledge graph? A representation learning perspective, in: MM 2021, ACM, 2021, pp. 2735–2743.

[37] T. Linzen, Issues in evaluating semantic spaces using word analogies, in: RepEval@ACL 2016, 2016, pp. 13–18.

[38] N. Schluter, The word analogy testing caveat, in: NAACL-HLT 2018, Volume 2 (Short Papers), 2018, pp. 242–246.

[39] A. Gladkova, A. Drozd, S. Matsuoka, Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't, in: SRW@HLT-NAACL 2016, 2016, pp. 8–15.

[40] Z. Bouraoui, S. Jameel, S. Schockaert, Relation induction in word embeddings revisited, in: COLING 2018, Association for Computational Linguistics, 2018, pp. 1627–1637.

[41] C. Allen, T. M. Hospedales, Analogies explained: Towards understanding word embeddings, in: ICML 2019, volume 97 of *Proceedings of Machine Learning Research*, PMLR, 2019, pp. 223–231.

[42] M. R. Petersen, L. van der Plas, Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance, in: EMNLP 2023, 2023, pp. 16414–16425.

[43] Y. Lepage, M. Couceiro, Any four real numbers are on all fours with analogy, 2024 (under review).

[44] O. L. Hölder, Grenzwerthe von Reihen an der Convergenzgrenze, Math. Annalen 20 (1882) 535–549.

[45] M. Couceiro, Y. Lepage, Analogie et moyenne généralisée, in: Journées d'Intelligence Artificielle Fondamentale - Journées Francophones sur la Planification, la Décision et l'Apprentissage pour la conduite de systèmes - JIAF-JFPDA 2024, La Rochelle, France, 2024, p. no page number. URL: https://inria.hal.science/hal-04568339.

[46] O. Levy, Y. Goldberg, Neural word embedding as implicit matrix factorization, in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Weinberger (Eds.), Proceedings of Advances in Neural Information Processing Systems 27 (NIPS 2014), Curran Associates, Inc., 2014, pp. 2177–2185. URL: https://papers.nips.cc/paper_files/paper/2014.

[47] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, in: Y. Bengio, Y. LeCun (Eds.), ICLR 2013, 2013. URL: http://arxiv.org/abs/1301.3781.
# Testing proportional sentence analogies on SATS: from vector offsets to conditional generation

Yves Blain-Montesano*, Philippe Langlais

Université de Montréal, Canada

**Abstract**

Analogy, the correspondence between two things, has been hailed as an important reasoning capability. Proportional analogy, denoted *a : b :: c : d* and read "*a* is to *b* as *c* is to *d*", is a special case of this where a correspondence is made in the relation that holds between the two elements of two different pairs (*a*, *b*) and (*c*, *d*). A common task in this domain is to solve for the conclusion *d* given the premise (*a*, *b*, *c*). Few datasets of proportional sentence analogies exist which aren't limited in the variety of relations we hope to capture, including those of a semantic or common sense nature. In this work, we curate a dataset of pairs of sentences for which such relations hold, constructing 78,400 analogies involving 32 relations. Our experiments demonstrate little basis for analogical reasoning of this kind using offsets of vector embeddings, in agreement with previous work, for retrieval and generation. We leverage the representations learned by pretrained language models, and the natural language input interface they provide, to solve analogies by generating from a prompt, as well as finetuned in a sequence-to-sequence setting. From this we gain insights into their failure modes and disparity in task ability.

**Keywords**

natural language processing, sentence analogy, text generation, sentence embedding

## 1. Introduction

We study the analogy-solving ability of language models and representations derived therefrom, specifically as regards proportional sentence analogies (those said “A is to B as C is to D” and denoted *a : b :: c : d*) formed on the basis of syntax, semantics, and encyclopedic knowledge. That is, we are interested in finding the conclusion *d*—given the premise (*a*, *b*, *c*)—of paired natural language sentences (*a*, *b*) and (*c*, *d*) which are analogous in ways that can be considered common sense to many humans. For example, in the analogy *I'm happy : I'm angry :: I sang : I yelled*, there is a relation of opposition of mood that holds which is relatively intuitive, though ambiguous as the solution *I yelled* need not be unique. This is the kind of analogy we refer to in our work.

The framework of proportional analogies is drawn from classical mathematical analogies such as $5 - 3 = 12 - 10$ or $\frac{2}{1} = \frac{4}{2}$ in addition to conceptual ones, for example when Aristotle writes “as old age is to life, so is evening to day” [1]. These analogies are a quaternary relation

IARML@IJCAI'2024: Workshop on the Interactions between Analogical Reasoning and Machine Learning, at IJCAI'2024, August, 2024, Jeju, South Korea

<figure><img src="image_30.png" /></figure>

yves.blain-montesano@umontreal.ca (Y. Blain-Montesano); felipe@iro.umontreal.ca (P. Langlais)

<figure><img src="image_31.png" /></figure>

https://www-labs.iro.umontreal.ca/~felipe/ (P. Langlais)

<figure><img src="image_32.png" /></figure>

0000-0002-2602-2019 (P. Langlais)

<figure><img src="image_33.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_34.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)

<footnote>*Corresponding author.</footnote>
where for some $(a, b, c, d) \in X^4$ we denote it with $a : b :: c : d$ if and only if some relation $R \subseteq X^2$ holds both for $(a, b)$ and $(c, d):^1$

$$a : b :: c : d \iff \exists R \subseteq X^2 \text{ s.t. } aRb \land cRd$$

Previous work has postulated equivalence relations between some permutations of the terms in $a : b :: c : d$ which apply in formal languages and Boolean logic. Instead, we follow [2] in distinguishing this “parallelogram model” of analogy from NLP-style analogies. Rather than leveraging domain-specific properties to prove the equivalence of analogy permutations, the latter kind only admits analogies whose relations $R$ are in a predetermined set $S$, such as the Google analogy set [3] or BATS (Bigger Analogy Test Set) [4]. There is no guarantee of symmetry, i.e. $R \in S \iff R^{-1} \in S$ (hence $b : a :: d : c$), nor of central permutation, i.e. $\forall R_i \in S \text{ s.t. } a : b :: c : d, \exists R_j \in S \text{ s.t. } a : c :: b : d,^2$ as these must be deliberately included in $S$.

We narrow our interest onto distributional models of natural language, where interest in proportional analogies has persisted since the popularizing work of Mikolov et al. [5] on word embeddings, interest which has extended to sentence embeddings. In this tradition, the typical method for solving proportional analogies, called the **vector offset or vector arithmetic** method, is to take the embedding $E(\cdot)$ for each term of the analogy under a model of arithmetic analogy: assuming the embeddings of $(a, b)$ and $(c, d)$ define a parallelogram and thus have equal offset vectors $E(b) - E(a) = E(d) - E(c)$, then $E(d) = E(c) + E(b) - E(a)$. As the predicted $d$ vector will rarely correspond exactly to an embedding, a solution will be found by retrieval among a set of candidates $C$ from a vector space model (VSM) by maximum cosine similarity, in which case the method is called **3CosAdd**. In general, we will call a **vector solver** any function $f$ which aims to predict $E(d) = f(E(a), E(b), E(c))$. Above, $C$ refers to the set of candidates for retrieval. In word analogy, this is the vocabulary for which we have embeddings. As this work regards sentence analogy, $C$ is a set of sentences, which we'll take as the unique sentences from a set of analogies. Vector arithmetic has been used to retrieve [6] as well as decode [7] the solution to sentence analogies. Yet, analogical reasoning with a vector offset has been shown to lack empirical and theoretical justification by its assumption of binary opposition of relations, the commutative properties of addition used by the offset, and its dependence on spurious properties of the embedding vectors used [8, 9].

While sentence embeddings from pooled token encodings produced by large language models (LLMs) of the Transformer architecture are often used [6, 10, 11, 12, 13], less attention has been given to conditionally generating the solution $d$ by directly providing the premises $(a, b, c)$ as input to the language model. Language models have been shown to display impressive performance including at unseen tasks in a few-shot setting, scaling with size [14]. Such an approach may offer the improvements of (1) retaining all information rather than pooling encodings, and (2) using the natural language interface which the model has conveniently already learned in order to specify the problem context. As such we need not enforce a model of proportional analogy such as vector arithmetic, nor require sufficient training examples to fit one on the pooled embedding space of a language model which is already pretrained.

<footnote n="1">¹We define it this way while noting that trivial analogies under e.g. $R = X^2$ are not typically of interest.</footnote>

<footnote n="2">²In *pillow : bed :: headrest : car*, the relation is between head supports and their associated locations. It is less clear what relation holds between a bed and a car.</footnote>
In light of this, our contributions are as follows. We evaluate various models on their ability to solve proportional analogies of natural language sentences held in varied (e.g. syntactic, semantic) relational proportions. For this purpose, we elaborate a dataset of relational sentence analogies described in Section 2. Extracting sentence embeddings from several language models, we evaluate vector solvers via retrieval (see Section 3) and compare this approach to generating solutions. For generation, we compare decoding from a vector solver prediction to finetuning a pretrained language model on our analogies as a sequence-to-sequence task, as well as prompting pretrained LLMs ranging into the billion-parameter size, as described in Section 4.

## 2. Dataset

While several sentence analogy datasets have been elaborated in previous works, we do not overall find them sufficiently adapted to evaluating analogical reasoning on natural language held in intuitive relations.

Zhu and de Melo [6] use NLI entailment and negation pairs such as *There is no skilled person riding a bicycle on one wheel*. : *A skilled person is riding a bicycle on one wheel*. and template sentences pairs filled with word analogies from the Google analogy set [3] such as *I've never been to Amman*. : *I've never been to Jordan*. However, it has been shown that the Google analogy set's imbalanced relations are dominated by easier geographical and morphological word pairs such as *possible:impossible* or *long:longer* which can skew the evaluation of analogical reasoning with embeddings, while worse performance was found against BATS [4], which is a broader, more balanced selection of relational word analogies. We expect the minimal variation between template sentences coupled with the lack of distractor sentences (see Section 3.2) in their candidate set to preserve the Google set's skew.

Some previous work [7, 11] has used a set of 5,607 semantico-formal analogies extracted from the Tatoeba corpus [15]. This set does not use predefined relations, though contains many surface changes (e.g. pluralization, present versus past tense), and other unclear semantic analogies such as the example *I do not need a wheelchair*. : *I do not need a girlfriend*. :: *I do not have a cat*. : *I do not have a boyfriend*.

Others collate analogies from existing resources [10, 13], for example the DSBATS set [12] of paired definition sentences corresponding to encyclopedic relations from BATS [4], but these remain relatively restricted in the breadth of relations considered.

For these reasons, we collect and indeed write our own relational sentence pairs to ensure an (1) adequate diversity of relations and sentences, and (2) that they constitute relatively “valid” analogies by their manual curation. Our sentence analogy test set (SATS)<fnref n="3" /> is a collection of 32 relation sets, listed in Table 1, each of 50 pairs of sentences (totalling 1,600 pairs from 3,024 unique sentences) from which 2450 non-identity quadruples (i.e. not *a : b :: a : b*) can be made by combining pairs of a same relation, for a total of 78,400 analogies. We coarsely categorize each relation set as Encyclopedic, Lexical, Syntactic, or Semantic for later aggregation of results.

We manually construct syntactic pairs such as *My parents turned on the TV*. : *My parents turned the TV on* (from the canonical-verb-particle-movement relation set), and also include relations of declarative sentences and questions taken from the QA2D dataset [16], which are

<footnote n="3">³Available at https://github.com/rali-udem/sats-sentence-analogy.</footnote>
<table><caption>**Table 1**
SATS relations by category and split</caption><thead><tr><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td></td><td>Encyclopedic</td><td>hypernym-animal</td><td>Encyclopedic</td><td>capital-country</td></tr><tr><td></td><td>Encyclopedic</td><td>misc-hypernym</td><td>Encyclopedic</td><td>country-language</td></tr><tr><td></td><td>Encyclopedic</td><td>person-occupation</td><td>Encyclopedic</td><td>invention-creator</td></tr><tr><td></td><td>Lexical</td><td>present-past</td><td>Encyclopedic</td><td>member-band</td></tr><tr><td rowspan="7">Training</td><td>Semantic</td><td>informal-formal</td><td>Lexical</td><td>idiom-literal</td></tr><tr><td>Semantic</td><td>sentence-opposite</td><td>Lexical</td><td>numeral-spelled</td></tr><tr><td>Semantic</td><td>sentiment-good-bad</td><td>Lexical</td><td>numeric-approximation</td></tr><tr><td>Syntactic</td><td>because-so</td><td>Lexical</td><td>past-future</td></tr><tr><td>Syntactic</td><td>canonical-extraposition</td><td>Test</td><td>Semantic</td><td>cause-effect</td></tr><tr><td>Syntactic</td><td>qa2d-declarative-howmany</td><td>Semantic</td><td>description-state</td></tr><tr><td>Syntactic</td><td>qa2d-declarative-when</td><td>Semantic</td><td>home-outdoors</td></tr><tr><td></td><td>Syntactic</td><td>qa2d-declarative-who</td><td>Semantic</td><td>simple-implicative-entailment</td></tr><tr><td rowspan="4">Validation</td><td>Encyclopedic</td><td>meronym-substance</td><td>Syntactic</td><td>active-passive</td></tr><tr><td>Lexical</td><td>present-future</td><td>Syntactic</td><td>canonical-verb-particle-movement</td></tr><tr><td>Semantic</td><td>phrasal-implicative-entailment</td><td>Syntactic</td><td>qa2d-declarative-howmuch</td></tr><tr><td>Syntactic</td><td>qa2d-declarative-what</td><td>Syntactic</td><td>qa2d-declarative-where</td></tr></tbody></table>

formed by replacing a constituent by a wh- question word (e.g. *She opened the car door : What did she open?*).<fnref n="4" /> Those in the Semantic category have a more ambiguous relation, such as the home-outdoors relation (e.g. *He gave himself a haircut in the bathroom. : He went for a haircut at the barbershop.*) or the informal-formal relation (e.g. *We gotta get going. : We need to start moving.*). Encyclopedic relations are inspired by the encyclopedic and lexicographic relation sets in BATS [4]. Here, where possible, sentences are chosen which match a corresponding word pair,<fnref n="5" /> specifically for the hypernym-animal, misc-hypernym, person-occupation, meronym-substance, capital-country, and country-language relations. The sentences themselves are collected from the first sentences of Wikipedia articles corresponding to those words, e.g. from the meronym-substance relation we have *A lens is a transmissive optical device which ... : Glass is a non-crystalline ... : A mirror or looking glass is ... : Glass is ...*, from which it can also be noted that several such relations are non-unique.

We also note that many attempts to train models on proportional analogies, while they use data splits, do so within relation types [6, 7, 12, 17], rather than between them, thus confounding whether they generalize analogical reasoning as such to unseen relations. For this reason we split our data as outlined in Table 1.

### 3. Retrieving solutions in a VSM

The first approach we examine is to retrieve the solutions to our analogies from a set of candidate sentences by maximum cosine similarity. The two questions this experiment seeks to answer are: (1) How do different models' embeddings lend themselves to analogical reasoning? (2) Do vector-based models of analogy (such as vector arithmetic) vary in their ability to recover useful features for this purpose?

<footnote n="4">⁴This example pair is not contained in SATS.</footnote>

<footnote n="5">⁵For example, for the BATS pair *Mozart : composer*, we will take the first sentence from the corresponding Wikipedia articles for "Mozart", and for "Composer"</footnote>
## 3.1. Models

To the first point, the embeddings we use are those mean-pooled from the final encodings of transformers,<fnref n="6" /> namely Flan-T5⁷ [19], BERT [20], RoBERTa [21], DeBERTa V1 and V3⁸ [22, 23], SBERT<fnref n="9" /> [18], and summed embeddings from the English Common Crawl CBOW FastText embeddings [24]. We use the Base checkpoint for all models where applicable.

To the second point, we examine three vector solvers: 3CosAdd, a feedforward network—as has been attempted by others [6, 7]—and an Abelian neural network [17] which is a universal approximator of Abelian Lie operations<fnref n="10" /> (such as addition, used in vector arithmetic). Unlike 3CosAdd, the feedforward (hereafter shortened to FF) and Abelian neural networks have parameters which must be trained<fnref n="11" /> for each different embedding model. Both trained solvers are feedforward neural networks composed of five blocks.

The FF blocks are composed of two affine transformations using the GELU activation function with residual connections and layer normalization. All hidden states retain the input dimensionality, though the first block has the dimensionality of the concatenation of the premise sentence embeddings $E(a) \circ E(b) \circ E(c)$, followed by an affine transformation which reduces it to the original embedding dimension.

The Abelian neural network is an invertible neural network $\phi$ which transforms embeddings into a space of the same dimensionality before applying the vector arithmetic method. The predicted solution is then found by using the inverse $x = \phi^{-1}(\phi(E(c)) + \phi(E(b)) - \phi(E(a)))$. For this we use the AllInOneBlock invertible block provided by the FrEIA Python module [25], which takes an inner function which we compose from two affine layers with a GELU activation function.

For each embedding model, the Abelian and FF solvers are trained on batches<fnref n="12" /> of 8 SATS analogies, with additive Gaussian noise on the order of $10^{-2}$, using the Adafactor optimizer [26] with a learning rate of $3 \times 10^{-5}$. As a loss, we use negative cosine similarity,<fnref n="13" /> i.e. $loss(x, y) = -\frac{\langle x, y \rangle}{\|x\| \cdot \|y\|}$. We observe overfitting over two epochs of training, thus the best checkpoint is selected on the validation split using the harmonic mean of the top-1 and top-5 retrieval accuracy to encourage choosing a model that at least predicts the neighborhood of the solution.

The Flan-T5 model being an encoder-decoder, for a sequence of token length $L$, it performs cross-attention on all $L$ token encoding vectors in order to autoregressively generate an output (which we explore further in Section 4). We can use this to gauge whether a model trained jointly to solve analogies in vector space and decode the solution results in better retrieval accuracy by adapting Flan-T5 to decode solely by cross-attention on a single bottleneck vector, which

<footnote n="6">⁶Mean-pooling has been found to outperform the [CLS] token's encoding [18].</footnote>

<footnote n="7">⁷Note that this is the only encoder-decoder model we use. All others have an encoder-only architecture.</footnote>

<footnote n="8">⁸The latter differs by the use of a replaced token classification task rather than the usual masked language modeling objective.</footnote>

<footnote n="9">⁹Specifically the all-mpnet-base-v2 checkpoint.</footnote>

<footnote n="10">¹⁰This is a differentiable operation, over some set, which has associativity and commutativity, as well as identity and inverse elements.</footnote>

<footnote n="11">¹¹Architecture and hyperparameters were manually tweaked. While the FF solver remains essentially a feedforward neural network, our tweaking prevents direct comparison with e.g. the feedforward solver of [7].</footnote>

<footnote n="12">¹²Here and elsewhere, batches are obtained by a random shuffling of all analogies in a given split.</footnote>

<footnote n="13">¹³We found regression with a mean square error loss to perform worse overall.</footnote>
<figure><img src="image_35.png" /><figcaption>**Figure 1:** A visualization of the end-to-end vector solver and decoder architecture. First each sentence is encoded and its encodings mean-pooled. *f* is the vector solver used, e.g. the offset method. In a retrieval task, *x* is used as the predicted solution vector for retrieval. When training (and for generative evaluation in Section 4), the decoder performs cross-attention on *x* to autoregressively generate *d*.</figcaption></figure>

we take to be a vector solver prediction $f(E(a), E(b), E(c))$. All vector solvers mentioned previously being differentiable, we can train such an end-to-end solver-decoder model (E2E-Flan-T5, visualized in Figure 1) for each different $f$, backpropagating gradients into the solvers' as well as the pretrained model's parameters. We use these finetuned models both for retrieval in this experiment—since at test time we can simply extract the output of the embedded vector solver—and for generation in Section 4 by decoding from the vector solver's output.

In order to use Flan-T5 as a vector bottleneck sentence autoencoder, we must first “warm up” the pretrained model by finetuning it for this purpose using the mean-pooled encoding vector, rather than all *L* encodings. Using a negative log-likelihood (NLL) loss, we train it on a next-token prediction task on interleaved, padded (up to 500 tokens) batches of 72 sentences split from the Reddit comments dataset provided by the Sentence-Transformers team<fnref n="14" /> and from the Online Language Modelling [27] dataset of Wikipedia articles dated 20-12-2022.<fnref n="15" /> This finetuning was performed in a single epoch (with no intentionally repeated data) for 281,600 batches,<fnref n="16" /> at which point it reached a validation loss of 0.372 (compared to the pretrained Flan-T5’s 0.30) on some 1,024 held-out sentences. We use a learning rate of 3 × 10⁻⁵ and the same Adafactor optimizer [26] as the model was originally trained with.

This checkpoint is used as the initialization of the end-to-end model with each of the vector solvers mentioned previously, trained to generate the solutions to analogies from the SATS training split, using the same NLL objective and optimizer, a learning rate of 10⁻⁴, and a batch size of 64. Observing overfitting over a single epoch, we select the best checkpoint on the validation split by METEOR score [28].

## 3.2. Evaluation

One aspect of evaluation that shouldn't be overlooked is the choice of candidate set, which often excludes the premises (*a*, *b*, *c*), thereby artificially improving accuracy [29], as the offset vector often points away from *d* and indeed every other candidate but *c* [9]. While we use the 3,024

<footnote n="14">¹⁴https://huggingface.co/datasets/sentence-transformers/refox-title-body</footnote>

<footnote n="15">¹⁵https://huggingface.co/datasets/olm/olm-wikipedia-20221220</footnote>

<footnote n="16">¹⁶At this point the model had not finished converging, due to the abundance of data.</footnote>
<caption>**Table 2**</caption>

Retrieval accuracy of the analogy solution using the 3CosAdd and FF solvers. Columns describe accuracy under different candidate sets. Neither: premises (a, b, c) excluded, no distractors added; +distractors: distractors added; +(a,b,c): premises retained; Both: both distractors and premises included.

<table><thead><tr><th rowspan="2">Model</th><th colspan="4">3CosAdd</th><th colspan="4">FF</th></tr><tr><th>Neither</th><th>+distractors</th><th>+(a,b,c)</th><th>Both</th><th>Neither</th><th>+distractors</th><th>+(a,b,c)</th><th>Both</th></tr></thead><tbody><tr><td>E2E-Flan-T5</td><td>0.60</td><td>0.25</td><td>0.15</td><td>0.14</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td></tr><tr><td>Flan-T5</td><td>0.61</td><td>0.21</td><td>0.10</td><td>0.08</td><td>0.36</td><td>0.10</td><td>0.15</td><td>0.07</td></tr><tr><td>SBERT</td><td>0.82</td><td>0.10</td><td>0.07</td><td>0.03</td><td>0.50</td><td>0.04</td><td>0.15</td><td>0.02</td></tr><tr><td>RoBERTa</td><td>0.57</td><td>0.27</td><td>0.15</td><td>0.12</td><td>0.23</td><td>0.12</td><td>0.12</td><td>0.08</td></tr><tr><td>BERT</td><td>0.61</td><td>0.16</td><td>0.07</td><td>0.05</td><td>0.32</td><td>0.12</td><td>0.13</td><td>0.08</td></tr><tr><td>DeBERTa</td><td>0.59</td><td>0.20</td><td>0.08</td><td>0.07</td><td>0.22</td><td>0.03</td><td>0.15</td><td>0.03</td></tr><tr><td>DeBERTa-V3</td><td>0.34</td><td>0.20</td><td>0.08</td><td>0.06</td><td>0.01</td><td>0.00</td><td>0.01</td><td>0.00</td></tr><tr><td>FastText</td><td>0.57</td><td>0.00</td><td>0.21</td><td>0.00</td><td>0.01</td><td>0.00</td><td>0.01</td><td>0.00</td></tr></tbody></table>

unique sentences in SATS as our candidates, they are very few in number. Thus, we evaluate accuracy under different candidate sets: with or without premises, and with or without added distractor sentences which we automatically create as follows.

For each unique sentence in SATS, we construct distractors by sampling whitespace-tokenized pairs of words to swap, or individual words to remove or replace with a randomly chosen nearest neighbour from the FastText vocabulary.<fnref n="17" /> For example, from *You're in trouble, friend*, we might swap to get *friend*. *You're trouble, in*, delete to get *You're ∅ trouble, friend*. , or replace to get *You're inside trouble, friend*. We sample from each sentence at most 5 times (length permitting) per type of modification, producing 41,356 unique distractor sentences.

We test the retrieval accuracy of 3CosAdd and the trained FF and Abelian vector solvers for each embedding model. To this we add the E2E-Flan-T5 model (for each embedded vector solver). We find mostly similar retrieval accuracy for the Abelian and 3CosAdd methods.<fnref n="18" /> Hence, in Table 2, we present the retrieval accuracies for 3CosAdd and FF only.

FF, even with premises and distractors excluded, attains only roughly half the accuracy of the Abelian solver and 3CosAdd. In the hardest condition it performs at a similar level, depending on the model. We'll note that it did not converge in training for E2E-Flan-T5, attaining zero accuracy. It also obtains near-zero test accuracy for DeBERTa-V3 on all candidate sets. Insofar as it succeeds with Flan-T5, BERT, and RoBERTa embeddings, the majority of its accuracy comes from the declarative-question, canonical-verb-particle-movement, numeral-spelled, and numeric-approximation relations.

This concentration of accuracy in only some relations is not a particularity of the FF solver or of training. 3CosAdd and the Abelian solver obtain near or above 20% accuracy for Lexical and Syntactic relations, depending on the embedding model, using the most difficult candidate set, and near zero for Encyclopedic and Semantic ones. Relatedly, we find that for base sentence pairs (a, b) in SATS, the nearest neighbor to a is b over 80% of the time for most embedding models

<footnote n="17">¹⁷We select the top 20 nearest neighbours with fewer than 25 characters.</footnote>

<footnote n="18">¹⁸We surmise that the Abelian solver's invertible neural network has preserved much of the original embedding space. Perhaps the algebraic properties of any Abelian operation will impede embedding-based solvers.</footnote>
<caption>**Table 3**</caption>

Example analogy from the member-band relation and top two retrieved solutions for SBERT and 3CosAdd. When the premises are removed, the correct solution is found. When they or their distractors are included, it is one of the premises which is retrieved. In the hardest case, it is in fact a distractor for *c* which ranks first.

<table><thead><tr><th colspan="4">Gene Simmons ... : <u>Kiss</u> ... :: <u>Bradford Phillip Delson</u> ... : <u>Linkin Park</u> ...</th></tr><tr><th>Neither</th><th>+(a,b,c)</th><th>+(a,b,c) +distractors</th><th></th></tr></thead><tbody><tr><td><strong>Linkin Park</strong> is an American rock band ...</td><td><u>Bradford Phillip Delson</u> (born December 1, 1977) is an American musician, best known as ...</td><td><u>Bradford Phillip Delson</u> 1, 1977) is an musician, best known as ...</td><td></td></tr><tr><td>Def Leppard are an English ...</td><td><u>Kiss</u> (stylized as ...) is an ...</td><td><u>Bradford Phillip Delson</u> ...</td><td></td></tr></tbody></table>

for Lexical and Syntactic relations, 30-50% for Semantic ones, and roughly 0% for Encyclopedic ones—except for SBERT, where it's 46%. Nor is the decrease in accuracy simply an effect of our distractors. It is still the case that when a distractor is picked, it is of the premises (*a*, *b*, *c*) rather than of *d*, as exemplified in Table 3. Such caveats are in line with results from the word analogy literature, where keeping premises in the candidate set reduces accuracy immensely [29], and where some surface relations are preferred [4].

We will note that when premises are included, FastText embeddings attain 40% and 46% for the Lexical and Syntactic categories but near zero in other categories, and even this drops to zero when distractors are added, as summed word embeddings cannot account for word order. However, this summation does mean that vector offsets can correspond to differences in used vocabulary.

It has been shown by [9] for lexical analogies that the cosine similarity to *d* of the offset prediction *c* + *b* − *a* is dependent on the similarity of the offsets of the two pairs (*a*, *b*) and (*c*, *d*) and the similarity of the terms within a single pair (*c*, *d*), which can be high or low for spurious reasons. Instead, they introduce the pairing consistency score (PCS), which measures the linear distinguishibility of a relation for a particular embedding space. To compute PCS for a given relation, we treat sets of offset vectors of true pairs and those of false shuffled pairs as observations for a binary classification task where the predictive score of each set of offset

<caption>**Table 4**</caption>

PCS for test split relations by model and category. Higher is better, 0.5 is chance level.

<table><thead><tr><th></th><th>Encyclopedic</th><th>Lexical</th><th>Semantic</th><th>Syntactic</th></tr></thead><tbody><tr><td>E2E-Flan-T5 (arithmetic)</td><td>0.52</td><td>0.85</td><td>0.64</td><td>0.94</td></tr><tr><td>Flan-T5</td><td>0.55</td><td>0.83</td><td>0.62</td><td>0.86</td></tr><tr><td>SBERT</td><td>0.82</td><td>0.80</td><td>0.61</td><td>0.74</td></tr><tr><td>RoBERTa</td><td>0.54</td><td>0.83</td><td>0.66</td><td>0.87</td></tr><tr><td>BERT</td><td>0.62</td><td>0.84</td><td>0.63</td><td>0.81</td></tr><tr><td>DeBERTa</td><td>0.58</td><td>0.83</td><td>0.67</td><td>0.89</td></tr><tr><td>DeBERTa-V3</td><td>0.51</td><td>0.68</td><td>0.58</td><td>0.65</td></tr><tr><td>FastText</td><td>0.57</td><td>0.86</td><td>0.62</td><td>0.87</td></tr></tbody></table>
vectors is the average cosine similarity of all (n/2) combinations of offsets. With this the area under the receiver operating characteristic (AUROC) is computed.<fnref n="19" /> As with [9], we take PCS as the average AUROC over N = 50 different false pair shufflings per relation.

Table 4 shows how embedding models have high PCS, i.e. linear separability of true offsets, for Lexical and Syntactic relations. PCS is much closer to chance for Semantic and, most notably, Encyclopedic relations. There are two exceptions to this. SBERT has exceptionally high PCS for Encyclopedic relations, likely due to its contrastive training on sources such as Wikipedia, whereas it has been found that contrastive loss leads to parallel offsets [30]. The second exception is DeBERTa-V3, which has exceptionally low PCS, and yet is a performant state of the art model. This is likely due to DeBERTa-V3's replaced token detection task, whereas all other embedding models are trained on a language modeling objective. The latter objective has been shown to result in linear representations [31]. We will suggest from these results that even when an embedding space adequately represents a relation in terms of parallel offsets—and even this should not be taken for granted among otherwise performant models like DeBERTa-V3-vector arithmetic fails to successfully utilize this regularity, often leading to the retrieval of a premise.

## 4. Generating solutions

We explore several approaches to generating the solution to analogies. First, we directly decode it with the E2E-Flan-T5 model.<fnref n="20" /> Second, we finetune the Flan-T5 Base and Large checkpoints on our analogies as a sequence-to-sequence (Seq2Seq) task. Third, we directly prompt the Flan-T5 model in its parameter sizes of Base (250M), Large (780M), XL (3B) and XXL (11B). When sampling, we use η-sampling [32] with η = 6 × 10⁻⁴. We use BLEU [33], METEOR [28], word error rate (WER), and exact match accuracy, which we report in Table 5.

The Seq2Seq Flan-T5 models (see Figure 2) are finetuned on SATS analogies to conditionally generate *d* from (a, b, c). First, we input to the encoder portion of the transformer the concate-

<caption>**Table 5**</caption>

SATS test split generation metrics. Copy rates refer to the exact match of the generated prediction with a premise sentence.

<table><thead><tr><th></th><th></th><th>Exact Match ↑</th><th>WER ↓</th><th>METEOR ↑</th><th>BLEU ↑</th></tr></thead><tbody><tr><td>(a, b) Baseline</td><td></td><td>—</td><td>0.86</td><td>0.54</td><td>0.23</td></tr><tr><td rowspan="2">E2E-Flan-T5</td><td>Base</td><td>0.03</td><td>0.92</td><td>0.38</td><td>0.14</td></tr><tr><td>Large</td><td>0.01</td><td>0.92</td><td>0.42</td><td>0.16</td></tr><tr><td rowspan="4">Seq2Seq</td><td>Base</td><td>0.00</td><td>1.10</td><td>0.31</td><td>0.10</td></tr><tr><td>Large</td><td>0.01</td><td>0.92</td><td>0.42</td><td>0.16</td></tr><tr><td>Base</td><td>0.00</td><td>1.84</td><td>0.27</td><td>0.05</td></tr><tr><td>Large</td><td>0.00</td><td>0.94</td><td>0.40</td><td>0.14</td></tr><tr><td rowspan="3">Prompted</td><td>XL</td><td>0.01</td><td>0.98</td><td>0.37</td><td>0.13</td></tr><tr><td>XXL</td><td>0.02</td><td>0.94</td><td>0.41</td><td>0.16</td></tr></tbody></table>

<footnote n="19">¹⁹Thus, if a relation is linearly distinguishable in an embedding space, as we increase the threshold for positive classification from 0 to 1, we expect the similarity of false offsets (false positives) to fall below it quickly, and that of true offsets (true positives) to do so slowly.</footnote>

<footnote n="20">²⁰We use the vector arithmetic variant since we find it performs identically to the Abelian one.</footnote>
<figure><img src="image_36.png" /><figcaption>**Figure 2:** Depiction of the sequence-to-sequence analogy task. The bolded text refers to sentence *c* which is prepended to the generated solution.</figcaption></figure>

nated input sentences $a \oplus b$. The decoder portion of the transformer performs cross-attention on the encodings of $a \oplus b$, and auto-regressively generates *d*, given the *c* premise sentence which is prepended to decoder output (and masked from the loss computation, as it is an input). The rationale is that the decoder may extract the change in the pair (*a*, *b*) and complete (*c*, *d*) in a similar fashion.<fnref n="21" /> We use a NLL loss for 3 epochs with batches of 64, using the Adafactor optimizer with a constant learning rate of $3 \times 10^{-5}$ for the Base models and $10^{-5}$ for the Large models. NLL loss on the validation set is used to pick the best checkpoint.

Seq2Seq improves for the Large model, though its output mostly consists of attempted copies of premise sentences. While E2E-Flan-T5 obtains better evaluation metrics, we can note (see Table 6) that it has also overfit to training examples from the person-occupation relation. Its output is often similar to training data, if not nonsensical.

Given that the pretrained Flan-T5 model is trained on text-to-text instruction-prompted tasks, we informally settled on a few-shot prompt incorporating three new example analogies not found in SATS, which elicits the model to output an appropriate response separated by a (FINAL ANSWER) substring without any finetuning. We automatically separate the solution from the generated sequence using the final (if any) occurrence of this answer separator.

We find that as model size increases the exact match accuracy increases. Copy rates hover around or below 5%, only shooting up to a *c* copy rate of 14% and 24% for the Flan-T5-Large Seq2Seq and Prompted outputs, respectively, returning to low levels for the XL and XXL sizes. Importantly, Table 6 shows that the largest model checkpoint exhibits understanding of the task, though it too ultimately copies the *c* premise after repeating the answer separator substring.

As an informal experiment, we prompted ChatGPT-3.5, which is finetuned by reinforcement learning with human feedback [34], and davinci-002, which is trained solely on the language modeling objective, both of which we expect to have 175 billion parameters [35]. As shown in Table 6, while davinci-002 falls into the familiar premise copying trap, ChatGPT-3.5 answers ideally, indicating that a language modeling objective alone may not suffice for immense models.

## 5. Conclusion

This paper presented a SATS, a novel sentence-level proportional analogy dataset which we use to evaluate common methods of analogical reasoning using vector embeddings which originate in work on word analogies. We identify similar gaps in the ability of such methods to solve sentence analogies as have previously been found for lexical analogies. To what

<footnote n="21">²¹Providing (*a*, *b*, *c*) as one to the encoder could potentially improve upon this, though we opted not to.</footnote>
<caption>**Table 6**</caption>

Comparison of example analogies and different models' predictions. E2E-Flan-T5 reproduces an unrelated sentence from the person-occupation relation in the training set. Flan-T5-XXL first correctly identifies the relation in question, though later repeats the answer separator and copies the *c* premise sentence, thus failing. The davinci-002 model also copies the *c* premise sentence. ChatGPT-3.5, however, correctly identifies the relation and solves the analogy, though the solution was manually extracted as it disregarded the answer separator.

<table><tbody><tr><td colspan="4" style="text-align: center; font-weight: bold;">E2E-Flan-T5 (capital-country)</td></tr><tr><td>Taipei ...</td><td>:</td><td>Taiwan ...</td><td>:: Berlin ...</td><td>:</td><td>Germany ...</td></tr><tr><td colspan="6" style="text-align: center;">*An author is someone who writes music ...</td></tr><tr><td colspan="4" style="text-align: center; font-weight: bold;">Flan-T5-XXL (capital-country)</td></tr><tr><td>Lisbon ...</td><td>:</td><td>Portugal ...</td><td>:: Manila ...</td><td>:</td><td>The Philippines ...</td></tr><tr><td colspan="6" style="text-align: center;">(FINAL ANSWER) The change between sentences one and two is the country that Lisbon is the capital and largest city of ... (FINAL ANSWER) The city of Manila</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">OpenAI (person-occupation)</td></tr><tr><td>John Christopher</td><td>:</td><td colspan="3">An actor or actress is</td><td>:: Christopher Columbus ...</td></tr><tr><td>Depp ...</td><td>...</td><td colspan="3"></td><td>davinci-002</td></tr><tr><td colspan="6" style="text-align: center;">(FINAL ANSWER) An actor or actress is a person who ...</td></tr><tr><td colspan="6" style="text-align: center;">ChatGPT-3.5</td></tr><tr><td colspan="6" style="text-align: center;">An explorer or navigator is a person who completes voyages across the Atlantic ...</td></tr></tbody></table>

extent analogical reasoning can be achieved with sentence embeddings is uncertain. While we evaluate a variety of approaches, further exploration is necessary. The use of 3CosAdd could be compared to 3CosMul [36] or retrieval by pairwise similarity [37]. Trained vector solvers may see improvements from exhaustive hyperparameter search. Comparisons to existing neural solver architectures [7, 11] should be made.

We experiment with solving sentence analogies generatively using the Flan-T5 language model by finetuning on them as a sequence-to-sequence task, and by few-shot prompting, finding improvements as models grow into the many billion parameter range. However, better generative metrics should be found for predictions and targets which have high surface similarity. Prompt engineering should also improve the performance of the pretrained Flan-T5 checkpoints by inducing less premise copying, better chain-of-thought reasoning, and fewer incidences of answer separator repetition

Our dataset is an important limitation. The reoccurrence of sentences in training analogies may lead to overfitting, and we find it likely that their small quantity is the main contributor to the poor performance of trained solvers. Finally, SATS should be improved by a principled method for collecting and validating a variety of humanlike proportional analogies.

## References

[1] N. Barbot, L. Miclet, H. Prade, Analogy between concepts, Artificial Intelligence 275 (2019) 487–539. doi:10.1016/j.artint.2019.06.008.