of all the stays belonging to a patient. We organize our dataset in way where each pair of stay is associated to the patient it belongs to: ⟨$S_1, S_2, PATIENT_ID⟩$, where $S_1$ corresponds to $s_{t_1}^{i_1}$, $S_2$ corresponds to $s_{t_2}^{i_2}$, and the associated $PATIENT_ID$ that represents $i_1$. We obtain a dataset made of 46,986 triples, where for each two pairs of stays we produce an analogy. For our experiments, we use all hospital stays associated with randomly selected 200 patients. We use the data augmentation process to generate positive and negative examples. For training and evaluation, we perform a random split (using a fixed random seed) in a training set of 70% of the extracted analogies, the remaining 30% serving as the test set. We end up with 939,638 analogies for training and 402,703 for testing. To maintain reasonable training and evaluation time, we randomly selected 50,000 analogies from the training set and 50,000 analogies from the testing set.

# 5. Experiment Setup

We now present the three experiments that we conducted in the *Identity* setting. In Section 5.1, we describe the patient-stay features that we consider and the data preprocessing that we performed for structured and unstructured data. We describe the implementation details in Section 5.2. The results of our experiments are reported in Section 5.3 and discussed further in this section. The code used for our experiments is written in Python 3.9 and PyTorch and is available in the repository https://github.com/Safa-98/patient-stay-analogy.

## 5.1. Stay Features and Data Preprocessing

We consider both structured (i.e., demographics and admission-related information) and unstructured data (i.e., clinical notes) to define our analogies. In this subsection, we describe the patient-stay features that are utilized by our model and some data preprocessing details.

**Static information.** In our experiments, our static information consists of demographic information and admission-related information. For demographic information, we extract patient's age, gender, marital status, ethnicity, and insurance information. We keep only adult patients (i.e., patients aged 18 and above). We split the age into 5 groups [18, 25[, [25, 45[, [45, 65[, [65, 89[ and [89, +∞[. For admission-related information, we include admission type as features.

**Clinical notes.** Nursing, Nursing/Other, Physician, and Radiology notes make up the majority of clinical notes in MIMIC-III database. For each hospital stay, we only kept notes that belong to these 4 categories. We excluded notes that have an error tag and notes that lack a hospital admission id.

## 5.2. Implementation Details

To build our corresponding cohorts, we performed the preprocessing described in the previous section to obtain our patient-stay features. Patients without any records of clinical notes or with notes that do not belong to the 4 categories defined above were removed. We computed the median of notes per hospital admission to determine the number of clinical notes to extract
<caption>**Table 1**</caption>

Accuracy and F1 score (both in %) of 3 runs of the classification model. Embeddings used are concatenation of static information and clinical notes.

<table><thead><tr><th>Epochs</th><th>Valid</th><th>Invalid</th><th>F1</th></tr></thead><tbody><tr><td>40 epochs</td><td>98.41 ± 1.56</td><td>68.22 ± 1.94</td><td>95.79 ± 0.59</td></tr><tr><td>20 epochs</td><td>94.89 ± 1.74</td><td>72.08 ± 1.68</td><td>94.30 ± 0.80</td></tr><tr><td>10 epochs</td><td>96.85 ± 1.75</td><td>70.31 ± 1.94</td><td>95.20 ± 0.71</td></tr></tbody></table>

per hospital admission. Therefore, we kept the first 12 notes, and used padding (i.e., completion with zeros) for hospital admissions with less than 12 notes.

For the unsupervised Doc2Vec model [18], we finetune it on the training set to obtain the document-level embeddings using the Gensim toolkit [23]. For the training algorithm, we use PV-DBOW (Paragraph vector-Distributed Bag of Words). We set the number of training epochs as 30, the initial learning rate as 0.025, the learning rate decay as 0.0002, and the dimension of vectors as 200 to train. The Fusion CNN model is trained with Adam optimizer with a learning rate of 0.0001 and ReLU as the activation function. The chosen batch size is 64.

In this paper we perform three experiments. In the first, we consider both structured and unstructured data. Therefore, we obtain our patient-stay representation by concatenating the representations of clinical notes along with static information. In this experiment, we verify if a particular hospital stay belongs to a patient by looking at both the structured and unstructured data associated with each stay. In the second, we only consider unstructured data, which means that our patient-stay representations are based solely on the representations of clinical notes. Therefore, by looking at clinical notes associated with a single hospital stay, we check if a particular hospital stay belongs to a patient. In the third, we only consider structured data, which means that our patient-stay representations are based solely on the representations of static information (i.e., demographics and admission-related information). Therefore, we verify if a particular hospital stay belongs to a patient by looking at the static information that is associated with a hospital stay.

## 5.3. Results and Discussion

As mentioned previously, we conducted three experiments that mainly differ in what type of data was used to obtain our patient-stay representations. For all the experiments, we used 50,000 analogies for training and evaluation, and applied the same procedure for data augmentation. We report the accuracy and F1 score for each experiment. The F1 score gives a better measure of the incorrectly classified cases than the accuracy metric.

For the first experiment, we fed our embedding model with both structured (i.e., demographics and admission-related information) and unstructured data (i.e., clinical notes). Our patient-stay representations are thus made of the concatenation of static information and clinical notes. We chose the epochs where the training loss is at the local minimum. We trained our model for 10, 20, and 40 epochs, with 3 different random initializations in each case. Our results are detailed in Table 1. Our model performs the best for positive examples. For 40 epochs, the model gives the best result for valid analogies and performs best for invalid analogies for 20 epochs.
<caption>**Table 2**</caption>

Accuracy and F1 score (both in %) of 3 runs of the classification model. Embeddings used are based on only clinical notes.

<table><caption>**Table 3**</caption><thead><tr><th>Epochs</th><th>Valid</th><th>Invalid</th><th>F1</th></tr></thead><tbody><tr><td>40 epochs</td><td>88.52 ± 6.91</td><td>77.20 ± 6.92</td><td>95.64 ± 1.27</td></tr><tr><td>20 epochs</td><td>97.71 ± 1.69</td><td>67.89 ± 2.43</td><td>94.74 ± 0.93</td></tr><tr><td>15 epochs</td><td>92.05 ± 6.42</td><td>74.45 ± 7.39</td><td>94.90 ± 1.25</td></tr></tbody></table>

Accuracy and F1 score (both in %) of 3 runs of the classification model. Embeddings used are based on only static information.

<table><thead><tr><th>Epochs</th><th>Valid</th><th>Invalid</th><th>F1</th></tr></thead><tbody><tr><td>40 epochs</td><td>99.98 ± 0.001</td><td>66.14 ± 0.04</td><td>96.37 ± 0.01</td></tr><tr><td>20 epochs</td><td>99.98 ± 0.004</td><td>65.96 ± 0.31</td><td>96.35 ± 0.03</td></tr><tr><td>15 epochs</td><td>99.98 ± 0.002</td><td>66.12 ± 0.13</td><td>96.33 ± 0.05</td></tr></tbody></table>

For our second experiment, we used only unstructured data, i.e., the $Z_{note}$ part of the embedding for the patient-stay representations. Our patient-stay representations thus consisted of only the representation of clinical notes. The training loss was at the local minimum for 15, 20, and 40 epochs. Therefore, we trained our model for 15, 20, and 40 epochs, with 3 different random initializations in each case. As shown in Table 2, our model performs the best for positive examples when we train by 20 epochs.

For our third experiment, we used only structured data, i.e., $Z_{static}$, to represent our patient-stay representations. Our training loss was at the local minimum for 15, 20, and 40 epochs. Therefore, we trained our model for 15, 20, and 40 epochs, with 3 different random initializations in each case. We report our results in Table 3. As seen, the accuracy for positive examples is high for all cases compared to negative examples where the accuracy drops.

In all our experiments, we can see that our model performs the best for positive examples regardless of whether we use [$Z_{static}$; $Z_{note}$], only $Z_{note}$, or only $Z_{static}$ for the patient-stay representations. This can be explained as a result of the imbalance between positive and negative examples in the training data. Balancing the data would be the next step as it proved to be a good solution for [6] to get similar results for positive and negative examples. The accuracy for valid analogies is the highest when our embedding model is fed with only static information. Between the first and the second experiment, the accuracy is the highest for valid analogies when the patient-stay representations are made of the concatenation in contrast to when our patient-stay representations are made of only clinical notes. This indicates that adding or using static information when learning patient-stay representations, as in the first and third experiment, improves the performance of our model, where it allows the model to better distinguish the stays and to match them to the patient they belong to. We also notice that the accuracy for invalid analogies is the highest when the embedding model is fed with only clinical notes. For all performed experiments, the F1 score is high, which indicates that our model is able to correctly classify analogies to the class they belong to (i.e., valid or invalid).
To gain more insight into how our models perform, we conducted an error analysis where we noticed that most misclassifications were spotted in two cases.

1. **Cases where $i_1 = i_2$**.

To recall, we do not generate invalid analogies for cases where $i_1 = i_2$; therefore, invalid analogy forms ($D: A :: B: C$ and $A: C :: B: D$) should be considered valid in these cases. In our error analysis, we noticed that when the four stays belong to the same patient, our model classifies the above-mentioned invalid analogy forms as invalid instead of valid. We believe that our model was not trained enough to distinguish these forms of analogies as there were less analogies with four stays belonging to the same patient generated in our dataset.

2. **Cases where representations are made of only clinical notes.**

To recall, in our second experiment we only used the representations of clinical notes to obtain patient-stay representations. We noticed that when the category of the clinical notes is similar between two hospital stays or when two hospital stays have less than five clinical notes, our model struggles to distinguish between the two hospital stays. This indicates that in some cases using only clinical notes to learn patient-stay representations might not be sufficient as these notes might not contain enough information to help our model differentiate between two similar stays that belong to two distinct patients. As a result, the model would incorrectly match these two similar stays to the same patient.

In these experiments, we did not include temporal data, where we only used demographics and admission-related information as structured data. It would be interesting to also include temporal signals (i.e., vital signs) along with demographics and admission-related information as structured data. Our patient-stay representations would then be made of the concatenation of the representations of static information and temporal signals as structured data and the representation of clinical notes as unstructured data.

6. **Conclusion and Perspectives**

We adapted the approach in [3, 6] from semantic and morphological analogies to patient-stay analogies. Our prototypical architecture has some limits, but seems promising for the task of patient identification. Our classification model is flexible in terms of the analogies that it classifies. Changing the way the data is augmented will change the way the model behaves. Our model can be adapted to different healthcare applications through dedicated embedding models [24]. Inspired by [14], we implemented a model to build patient-stay representations. As mentioned in Section 5.3, there are multiple plausible improvements to our approach, in terms of balancing valid and invalid analogies as well as including other types of data to build our patient-stay representations. As we limited ourselves to analogy detection, a future work would be to address analogy solving in the same setting that would allow the generation of synthetic patient-stays.
## Acknowledgments

Experiments presented in this paper were carried out using computational clusters equipped with GPU from the Grid'5000 testbed (see https://www.grid5000.fr).

The research work of the second and third named authors is partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation program under GA No 952215, and the Inria Project Lab "Hybrid Approaches for Interpretable AI" (HyAIAI).

## References

[1] R. Fam, Y. Lepage, Morphological predictability of unseen words using computational analogy, in: Workshops Proceedings for the Twenty-fourth International Conference on Case-Based Reasoning (ICCBR), volume 1815, 2016, pp. 51-60.

[2] S. E. Reed, Y. Zhang, Y. Zhang, H. Lee, Deep visual analogy-making, in: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2015, pp. 1252-1260.

[3] S. Lim, H. Prade, G. Richard, Solving word analogies: A machine learning perspective, in: Proceedings of the Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU), volume 11726, 2019, pp. 238-250.

[4] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532-1543.

[5] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, D. Salesin, Image analogies, in: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), 2001, pp. 327-340.

[6] S. Alsaidi, A. Decker, P. Lay, E. Marquer, P.-A. Murena, M. Couceiro, A neural approach for detecting morphological analogies, in: Proceedings of the 8th IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2021, pp. 1-10.

[7] M. A. Casteleiro, J. D. Diz, N. Maroto, M. J. F. Prieto, S. Peters, C. Wroe, C. S. Torrado, D. M. Fernandez, R. Stevens, Semantic deep learning: Prior knowledge and a type of four-term embedding analogy to acquire treatments for well-known diseases, JMIR Medical Informatics 8 (2020) 1-28.

[8] E. Dynamant, R. Lelong, B. Dahamna, C. Massonnaud, G. Kerdelhué, J. Grosjean, S. Canu, S. J. Darmoni, et al., Word embedding for the french natural language in health care: comparative study, JMIR medical informatics 7 (2019) 118-122.

[9] N. N. Rather, C. Patel, S. A. Khan, Using deep learning towards biomedical knowledge discovery, International Journal of Mathematical Sciences and Computing, IJMSC 3 (2017) 1-10.

[10] A. E. W. Johnson, T. J. Pollard, L. Shen, L. wei H. Lehman, M. Feng, M. M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, R. G. Mark, Mimic-iii, a freely accessible critical care database, Scientific Data 3 (2016).

[11] Y. Si, J. Du, Z. Li, X. Jiang, T. Miller, F. Wang, W. Jim Zheng, K. Roberts, Deep representation learning of patient data from electronic health records (ehr): A systematic review, Journal of Biomedical Informatics 115 (2021) 1-42.
[12] J. Zhang, K. Kowsari, J. H. Harrison, J. M. Lobo, L. E. Barnes, Patient2vec: A personalized interpretable deep representation of the longitudinal electronic health record, IEEE Access 6 (2018) 65333-65346.

[13] S. Madhumita, S. Simon, L. Kim, D. Walter, Patient representation learning and interpretable evaluation using clinical notes, Journal of biomedical informatics 84 (2018) 103-113.

[14] D. Zhang, C. Yin, J. Zeng, X. Yuan, P. Zhang, Combining structured and unstructured data for predictive models: a deep learning approach, BMC Medical Informatics and Decision Making 20 (2020) 280.

[15] P. Waruhari, A. Babic, L. Nderu, M. C. Were, A review of current patient matching techniques, in: Informatics Empowers Healthcare Transformation (ICIMTH), volume 238, 2017, pp. 205-208.

[16] F. N. Wirth, T. Meurers, M. Johns, F. Prasser, Privacy-preserving data sharing infrastructures for medical research: systematization and comparison, BMC Medical Informatics Decision Making 21 (2021) 242.

[17] B. H. Just, D. T. Marc, M. Munns, R. H. Sandefer, Why patient matching is a challenge: Research on master patient index (mpi) data discrepancies in key identifying fields., Perspectives in health information management 13 (2016) 1e.

[18] Q. V. Le, T. Mikolov, Distributed representations of sentences and documents, in: Proceedings of the 31st International Conference on Machine Learning (ICML), volume 32, 2014, pp. 1188-1196.

[19] L. Miclet, S. Bayoudh, A. Delhay, Analogical dissimilarity: Definition, algorithms and two experiments in machine learning, Journal of Artificial Intelligence Research 32 (2008) 793-824.

[20] Y. Lepage, De l'analogie rendant compte de la commutation en linguistique, Habilitation à diriger des recherches, Université Joseph-Fourier - Grenoble I, 2003.

[21] C. Antic, Analogical proportions, ArXiv abs/2006.02854 (2020).

[22] A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, H. E. Stanley, Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals., Circulation 101 23 (2000) E215-20.

[23] R. Řehůřek, P. Sojka, Software Framework for Topic Modelling with Large Corpora, in: Proceedings of the LREC Workshop on New Challenges for NLP Frameworks, 2010, pp. 45-50.

[24] S. Alsaidi, M. Couceiro, A. Burgun, N. Garcelon, A. Coulet, Exploring analogical inference in healthcare, in: Workshop on Interactions between Analogical Reasoning and Machine Learning (IARML), 2022 (to appear).
# The Application of Qualitative Metadata to Analogical Reasoning

Dave Raggett

W3C/ERCIM, Sophia Antipolis, France

## Abstract

Analogical reasoning can be used for plausible inferences based upon direct similarities or structural mappings involving properties and relationships. This can be implemented on top of a combination of symbolic knowledge plus sub-symbolic qualitative metadata, with matching based upon structural or causal similarities, and noticing interesting differences, in essence, abstracting from similarities and dissimilarities, and will be applied to examples of the form "A is to B as C is to ?X". A further challenge is to support the use of literal and figurative analogies in natural language, e.g., comparing life to the wheel of fortune, when you want to highlight the role of chance. An easy-to-use syntax will be presented for expressing knowledge, along with a web-based proof of concept demonstrator, and a unifying cognitive architecture for human-like AI. This builds upon pioneering work by Alan Colins on plausible reasoning, and Dedre Gentner on analogies.

## Keywords

Plausible reasoning, Human-like AI, analogies, Cognitive Architecture

## 1. Introduction

The paper starts with an introduction to plausible reasoning before moving on to analogical reasoning and how this can be supported as an extension of plausible reasoning. This very much work in progress, and part of a long term drive to realise human-like memory, reasoning and learning in cognitive agents.

## 2. Plausible Reasoning

We are learning all the time, and revising our beliefs and understanding as we interact with others. As such our knowledge is imperfect and subject to uncertainties, incompleteness and inconsistencies. This is challenging both for conventional mathematical logic, and for statistical approaches such as Bayesian inference due to the difficulties in obtaining the required statistics. Evolution has equipped humans with the means to deal with imperfect knowledge in a rational way based upon sound judgement, albeit subject to various kinds of cognitive biases, see, e.g., Daniel Kahneman [1].

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<footnote>ICCBR Analogies'22: Workshop on Analogies: from Theory to Applications at ICCBR-2022, September, 2022, Nancy, France</footnote>
People have studied the principles for plausible arguments since the days of Ancient Greece, e.g., Carneades and his guidelines for argumentation. This was developed further by a long line of philosophers, including Locke, Bentham, Wigmore, Keynes, Wittgenstein, Pollock and many others. Plausible reasoning is everyday reasoning, and the basis for legal, ethical and business discussions. It is now timely to exploit plausible reasoning with imperfect knowledge in support of human-machine cooperative work. This will enable computers to analyse, explain, justify, expand-upon and argue in human-like ways.

Consider $A \rightarrow B$, which means if A is true then B is true. If A is false then B may be true or false. If B is true, we still can't be sure that A is true, but if B is false then A must be false. We can go further with a little knowledge. Consider a more concrete example: if it is raining then it is cloudy. This can be used for inferences in both directions. Rain is more likely if it is cloudy, and likewise, if it is not raining, then it might be sunny, so it is less likely that it is cloudy. Such arguments draw upon qualitative terms in lieu of quantitative statistics.

In essence, plausible reasoning draws upon prior knowledge as well as on the role of analogies, and the consideration of examples as precedents. Mathematical proof is replaced by reasonable arguments, both for and against a premise, along with how these are assessed. In legal proceedings, for instance, arguments are laid out by the Prosecution and the Defence, the Judge decides what evidence is admissible, whilst guilt is assessed by the Jury.

During the 1980's Alan Collins and his co-workers developed a theory of plausible reasoning [2] based upon analysis of recordings of how people reasoned. They found that:

* There are several categories of inference rules that people commonly use to answer questions.

* People weigh the evidence that bears on a question, both for and against, rather like in court cases.

* People are more or less certain depending on the certainty of the premises, the certainty of the inferences, and whether different inferences lead to the same or opposite conclusions.

* Facing a question for which there is an absence of directly applicable knowledge, people search for other knowledge that could help given potential inferences.

Plausible knowledge can be expressed using a combination of symbolic graphs and associated metadata. This paper introduces the plausible knowledge notation (PKN) as an easy-to-read extensible syntax accompanied with an implementation as a JavaScript library for use in web page demos for different kinds of plausible reasoning [3], and as part of work for the W3C Cognitive AI Community Group [4]. PKN supports a variety of different kinds of statements:

## Properties

flowers of England includes daffodils, roses (certainty high)

where flowers is a property of the referent England, and the use of includes signifies that the property is an open set with values daffodils and roses. For a closed set, use is instead of includes. Trailing round brackets are used to list qualitative metadata, in this case declaring that the statement has a high certainty.

## Relationships
robin kind-of songbird

duck similar-to goose for habitat

duck dissimilar-to goose for neck-length

where robin is declared as a subclass of songbird, and duck is declared as being similar to goose for habitat and dissimilar to goose in respect to neck length.

### Dependencies

climate depends-on latitude

pressure decreases-with altitude

current increases-with voltage

where climate depends on latitude in some unspecified way, whilst pressure decreases with increasing latitude and current increases with increasing voltage.

### Implications

temperature of ? place is warm &

rainfall of ? place is heavy

implies grain of ? place includes rice

Implications are a form of if-then rules where variables are prefixed with a question mark.

Metadata can be given with all kinds of PKN statements. Relationships, dependencies and implications can be used for inferences in both directions, subject to any associated metadata. Following Collins, PKN supports several kinds of statement metadata relevant to different kinds of inferences:

**Typicality** in respect to other group members, e.g., robins are typical song birds.

**Similarity** to peers, e.g., having a similar climate.

**Strength** as conditional likelihood, e.g., the strength of climate for determining which kinds of plants grow well. The forward and backward strengths may differ, e.g., rain is a strong indicator of cloudy weather, whilst cloudy weather is a weak indicator of rain.

**Frequency** as the proportion of children with a given property, e.g., most species of birds have the ability to fly.

**Dominance** as the relative importance in a given group, e.g., the size of a country's economy.

**Multiplicity** as the number of items in a given range, e.g., how many different kinds of flowers grow in England.

The web demonstrator [3] allows you to pick from an assortment of queries, and to then see a trace of the reasoning, proceeding from the facts to the premise. The inference engine itself works backwards from the premise to the facts, and the explanation is subsequently generated from the trace of execution. Here is an example of the reasoning associated with the query whether daffodils are grown in England:
Premise: flowers of England includes daffodils

Evidence supporting the premise:

flowers of England includes temperate-flowers
and daffodils kind-of temperate-flowers
therefore flowers of England includes daffodils

flowers of Netherlands includes daffodils, tulips
and Netherlands similar-to England for flowers
therefore flowers of England includes daffodils

Suggesting: flowers of England includes daffodils is likely

This develops two lines of argument in favour of the premise in the query. The first is based on recognising that daffodils are a sub-class of temperate flowers, which are known to grow in England. The second makes use of knowledge that England and the Netherlands are similar in respect to the flowers grown. The inference engine uses a fixed strategy for searching for and applying relevant inferences. This may involve the use of graph algorithms such as spreading activation to propose and prioritise potential inferences as suggested by Collins. Other algorithms are used to compute certainties of inferences based upon statement metadata, and for assessing and combining multiple lines of argument. Future work will explore a wide range of reasoning, including spatial, temporal, causal and social reasoning, along with metacognition for problem solving, and support for System 1 and 2 cognition [1].

### 3. Analogical Reasoning

What benefits are potentially possible for analogical reasoning by cognitive agents? A starting point is to distinguish between literal and figurative analogies. The former involves things that are really quite similar, whilst the latter are not obviously comparable at first glance. Analogies can help agents to generalise their knowledge based upon a few examples. This has potential applicability for the properties of things, understanding their behaviours, as well as for problem solving by drawing upon previous experience in similar situations.

Analogies are further related to similes and metaphors in language. Similes involve a comparison that explicitly emphasises some comparable characteristic, e.g., "his words were like a punch in the guts" as a way to establish the impact of the words on the listener, whilst metaphors involve an implicit comparison, e.g., "to get cold feet" is to have second thoughts about some proposed course of action. People commonly use similes and metaphors to communicate thoughts in ways that are more vivid and interesting, as well as to structure perceptions and understanding, see Lackoff and Johnson [5]. As such, this is expected to be an important aspect of human-machine communication, albeit one that is very challenging, at least in the near future.

Dedre Gentner [6] notes that analogies may involve matching based upon structural or causal similarities, and noticing interesting differences, in essence, abstracting from similarities and dissimilarities
Gentner cites the example of plumbing in that electrical circuits can be likened to a plumbing system for water, e.g., equating voltage to pressure, and electrical current to water flow. Causal relationships for the source can be used to suggest similar relationships for the target, e.g., higher voltage leads to greater current just as higher water pressure leads to greater water flow.

Two situations can be identified as similar if they share some of the same properties, with the implication that you may be able to infer properties of the target from properties of the source. You may also be able to infer relationships, e.g., part/whole or cause/effect. More generally, the situations have different properties, that can however be mapped one to another (as in voltage to pressure). Such mappings have to be learned or guessed from matching relationships. Thus, if two situations/contexts have several properties or relationships in common, then we may consider them as analogical equivalents.

The notion of similarity introduced by Collins [2] supports inferences on shared property values at least in some given context, see the similar-to and dissimilar-to statements in PKN above. A generalisation is to relate pairs of different properties, e.g., voltage corresponds to pressure, and current to flow when making an analogy between electrical circuits and plumbing. Such pairings can be represented by adding a corresponds-to statement to PKN:

voltage corresponds-to pressure for circuit
current corresponds-to flow for circuit
flow increases-with pressure
# thus allowing us to infer
current increases-with voltage

We also need a way to describe that voltage and current are characteristics of electrical circuits, which are a sub-class of circuits, e.g.

electrical-circuit kind-of circuit
voltage property-of electrical-circuit
current property-of electrical-circuit

An open question is how people learn such knowledge from examples and being taught by others. That relates to the notion of syntagmatic and paradigmatic learning. Syntagmatic learning deals with learning co-occurrence patterns within episodes, whilst, paradigmatic learning involves identifying generalisations, and is believed to develop at a later age in childhood.

**Analogies as part of critical thinking.** It is easy to find web sites that propose the use of analogies for teaching purposes. These are based upon simple patterns, e.g., synonyms, antonyms, part/whole, cause/effect, etc. Here are some examples:

battery is-to torch as ?x is-to car # engine powers a car
itch is-to scratch as ?x is-to cold # virus causes a cold
wall is-to brick as bottle is-to ?x # a bottle is made of glass

Solving such queries involves identifying the pattern, and then applying background knowledge. The first step is to recognise the query as using an analogy. The next step is to use the pair that doesn't involve a variable to identify likely patterns, e.g., battery/torch in the first example. The knowledge base may contain plenty of facts and relationships, and it will be important
<figure><img src="image_44.png" /><figcaption>**Figure 1:** Cognitive architecture with multiple cognitive circuits loosely equivalent to shared blackboards – where semantic integration mimics the role of the Anterior Temporal Lobe as a hub for unimodal spokes.</figcaption></figure>

to look for patterns that also occur for the pair with the variable. It may be the case that two
pairs use different relationships, in which case, we need to find plausible evidence that they are
comparable patterns.

Simple analogies are amenable to a fixed strategy plus associated graph algorithms. Qualitative metadata can be used to reason about certainty and to prioritise processing. What about more complicated analogies? The work by Jaime Carbonell on derivational analogies [7] is inspiring. The paper describes a problem solver that searches for analogies with previously solved problems, adapting the solution as needed based upon an analysis comparing the old and new problems.

4. Cognitive Architecture

The quest for realising human-like AI owes a huge debt to many pioneers over many decades. To mention just a few: Daniel Kahneman, a Nobel-prize willing psychologist who studied System 1 & 2 thinking along with cognitive biases [1]; Philip Johnson-Laird, a cognitive scientist renowned for his work on how humans reason in terms of mental models rather than logic and statistics [8]; John R. Anderson, a cognitive scientist renowned for his work on the ACT-R cognitive architecture for sequential cognition [9]; and Alan Collins, a cognitive scientist renowned for his work on plausible reasoning and intelligent tutoring systems [2].

Figure 1 illustrates a high-level cognitive architecture inspired by the structure and function of the human brain.

**Memory** is based on graph databases and associated graph algorithms. It combines symbolic graphs with sub-symbolic information, mimicking the human cortex, and defined at a conceptual level above that of RDF and Property Graphs (including NGSI-LD). Recall is stochastic reflecting prior knowledge and past experience. This involves activation boost/decay, spreading activation, the forgetting curve and spacing effect.
**Perception** interprets sensory data at progressively higher levels of abstraction, and places the resulting models into the cortex. Cognitive rules can set the context for perception, and direct attention as needed. Events are signalled by queuing chunks to cognitive buffers to trigger rules describing the appropriate behaviour. A prioritised first-in first-out queue is used to avoid missing closely spaced events.

**System 1** is about intuitive/emotional thought, and prioritising what's important. The limbic system provides rapid automatic assessment of past, present and imagined situations without the delays incurred in deliberative thought. Emotions are perceived as positive or negative, and associated with passive or active responses, involving actual and perceived threats, goal-directed drives and soothing/nurturing behaviours.

**System 2** is slower and more deliberate thought, involving sequential execution of rules to carry out particular tasks, including the means to invoke graph algorithms in the cortex, and to invoke operations involving other cognitive circuits. Thought can be expressed at many different levels of abstraction, and is subject to control through metacognition, emotional drives, internal and external threats.

**Action** is about carrying out actions initiated under conscious control, leaving the mind free to work on other things. An example is playing a musical instrument where muscle memory is needed to control your finger placements as thinking explicitly about each finger would be far too slow. The cerebellum provides real-time coordination of muscle activation actively guided by perception.

This architecture has been partially realised with a suite of web-based demos developed for the W3C Cognitive AI Community Group [4]. This includes the chunks and rules specification, and an implementation as a JavaScript library. Chunks are essentially collections of name/value pairs, where values are literals or references to other chunks, or lists thereof. Chunks are associated with decaying activation levels to mimic the characteristics of human-memory. Chunk rules support sequential reasoning (System 2).

Further work is underway to incrementally realise the requirements identified by Kahneman for System 1, and to understand how plausible reasoning, learning and metacognition can be layered on top of System 1 and 2. This will include the intuitive and deliberative reasoning involved in natural language processing, and the human ability to reason about the past, present, and imagined situations.

# 5. Conclusions

AI today can be broadly split into symbolic AI, statistical techniques, and approaches based upon deep learning and multi-layer artificial neural networks. Work in the cognitive sciences suggests a middle ground that combines symbols and sub-symbolic metadata, and is open to distributed representations (e.g., as vectors in noisy high dimensional spaces) where this would yield computational benefits. Traditional symbolic AI is hard to scale, relying on hand-coded knowledge, along with difficulties in dealing with imperfect knowledge, whilst deep learning scales well, but has challenges with reasoning and transparency. This paper draws attention
to the potential for mimicking human-like memory, reasoning and learning, inspired by the
wealth of research in the cognitive sciences.

References

[1] D. Kahneman, Thinking, fast and slow, Macmillan, 2011.

[2] A. M. Collins, R. S. Michalski, The logic of plausible reasoning: A core theory, Cogn. Sci. 13 (1989) 1-49. URL: https://doi.org/10.1207/s15516709cog1301_1. doi:10.1207/s15516709cog1301\_1.

[3] D. Raggett, Plausible reasoning demo, visited on July 24, 2022. URL: https://www.w3.org/Data/demos/chunks/reasoning/.

[4] W3C Cognitive AI Community Group, visited on July 24, 2022. URL: https://github.com/w3c/cogai#readme.

[5] G. Lakoff, M. Johnson, Metaphors we live by, University of Chicago press, 1980.

[6] D. Gentner, Structure-mapping: A theoretical framework for analogy, Cogn. Sci. 7 (1983) 155-170. URL: https://doi.org/10.1207/s15516709cog0702_3. doi:10.1207/s15516709cog0702\_3.

[7] J. G. Carbonell, M. Veloso, Integrating derivational analogy into a general problem solving architecture, in: Proceedings of the First Workshop on Case-Based Reasoning, Morgan Kaufmann Tampa, FL, 1988, pp. 104-124.

[8] P. Johnson-Laird, How we reason, Oxford University Press, 2006.

[9] J. R. Anderson, How can the human mind occur in the physical universe?, Oxford University Press, 2007.
# Towards efficient scoring of student-generated long-form analogies in STEM

Thilini Wijesiriwardene<fnref n="1" />,*, Ruwan Wickramarachchi<fnref n="1" />, Valerie L. Shalin<fnref n="1" />,<fnref n="2" /> and Amit P. Sheth<fnref n="1" />

## Abstract

Switching from an analogy pedagogy based on comprehension to analogy pedagogy based on production raises an impractical manual analogy scoring problem. Conventional symbol-matching approaches to computational analogy evaluation focus on positive cases, and challenge computational feasibility. This work presents the Discriminative Analogy Features (DAF) pipeline to identify the discriminative features of strong and weak long-form text analogies. We introduce four feature categories (semantic, syntactic, sentiment, and statistical) used with supervised vector-based learning methods to discriminate between strong and weak analogies. Using a modestly sized vector of engineered features with SVM attains a 0.67 macro F1 score. While a semantic feature is the most discriminative, out of the top 15 discriminative features, most are syntactic. Combining this engineered features with an ELMo-generated embedding still improves classification relative to an embedding alone. While an unsupervised K-Means clustering-based approach falls short, similar hints of improvement appear when inputs include the engineered features used in supervised learning.

## Keywords

Descriptive analogies, Analogical features, Analogy scoring, Long-form analogies,

## 1. Introduction

Analogical reasoning relies on the ability to draw on the relational similarities between two systems of objects in different contexts [1, 2, 3]. Analogies appear in several disciplines such as engineering design, scientific reasoning, and often in STEM education. However, the dominant pedagogical paradigm requires students to comprehend curated analogies. In this work, we are focusing on the evaluation of *student-generated* analogies in their first undergraduate biochemistry course.

Problem sets, specifically created to explore the underlying mechanisms of analogical reasoning, consist of visual and verbal analogies [4, 5]. Verbal analogies have two primary forms; analogical proportions and long-form analogies. Analogical proportions follow a four-term

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<footnote n="1">¹AI Institute, University of South Carolina, Columbia, SC, USA</footnote>

<footnote n="2">²Department of Psychology, Wright State University, Dayton, OH, USA</footnote>

<footnote>ICCBR Analogies'22: Workshop on Analogies: from Theory to Applications at ICCBR-2022, September, 2022, Nancy, France</footnote>

<footnote>*Corresponding author.</footnote>

<footnote>thilini@sc.edu (T. Wijesiriwardene); ruwan@email.sc.edu (R. Wickramarachchi); valerie.shalin@wright.edu</footnote>

<footnote>(V. L. Shalin); amit@sc.edu (A. P. Sheth)</footnote>

<footnote>0000-0001-8431-8443 (T. Wijesiriwardene); 0000-0001-5810-1849 (R. Wickramarachchi); 0000-0001-8135-2793</footnote>

<footnote>(V. L. Shalin); 0000-0002-0021-5293 (A. P. Sheth)</footnote>

<footnote>© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).</footnote>

<footnote>CEUR Workshop Proceedings (CEUR-WS.org)</footnote>
format such as "A to B as to C to D" or A:B::C:D [6]. Recent work on computational analogy
making focuses on analogical proportions [7, 8, 9]. Our interest here lies in long-form analogies
consisting of a narrative/ description of a target unfamiliar situation/ system (for the context
or lesson to be learned) using several sentences and a familiar source (base) [10, 3]. While the
objects across the two descriptions differ, they employ similar relations between these objects.
An example of a well-known long-form analogy is between the solar system (source) and the
Rutherford-Bohr model of the atom (target) [11] where small objects revolving around a large
central object provide relational similarity with the target. The solar system and the atom can
each be described using several sentences. Parallels between these two systems can then be
drawn, making the two descriptions analogous.

The atom-solar system analogy exemplifies the curated analogies in STEM textbooks. [12] has developed algorithms for evaluating correct or slightly incorrect long-form analogies. In [13] we solicited analogies from STEM students, with the expectation, relative to a comprehension exercise, that analogy production is both more engaging and allows students to employ existing familiar knowledge to scaffold the acquisition of new knowledge. No matter how pedagogically successful, manual scoring is impractical for an analogy production pedagogy. Production pedagogy elevates an analogy scoring problem for computational solution.

We aim to identify discriminative features between strong and weak, long-form *student generated* verbal analogies collected in a college biochemistry class (see Section 2.1) to support efficient computational scoring. To this end, we develop the Discriminative Analogy Features (DAF) pipeline.

We use a long-form analogy dataset, instructor-graded as strong or weak. We explore both supervised and unsupervised learning classifiers using vectors based on embeddings, engineered features and both. Given manually annotated data for a supervised learning classifier (i.e. SVM), we identify the discriminative features of strong and weak analogies.

We introduce DAF, a pipeline to identify the discriminative features of strong and weak
analogies. We also introduce four feature categories – semantic, syntactic, sentiment, and
statistical used in supervised learning to discriminate between strong and weak analogies.
We show that “unique attribute count”, a *semantic feature*, is the most discriminative when
identifying between strong and weak analogies. Out of the top 15 discriminative features, most
are *syntactic*. Unsupervised learning is unable to obtain comparable success, though it slightly
improves with features corresponding to the above categories.

The rest of this paper is organized as follows: Section 2 introduces and describes the DAF
pipeline and identifies the discriminative features. Section 3 presents the discussion with
findings, insights, limitations, and future work subsections. Section 4 concludes the paper.

## 2. Discriminative Analogy Features (DAF) Pipeline

To identify discriminative features, we introduce the pipeline illustrated in Figure 1. In the
subsequent subsections, we describe each pipeline component.
<figure><img src="image_49.png" /><figcaption>**Figure 1:** Illustration of the DAF Pipeline. The analogies are split into sentences to create the analogy dataset. Single sentences are sent through Analogy Processing. Features of the sentences are extracted and used in SVM-based classification and K-means clustering. An analysis is then conducted on SVM-based classification and K-means clustering results.</figcaption></figure>

## 2.1. Dataset and Input

The dataset used in this work was drawn from 500 student-created analogies, collected in a college classroom. An instructor explained a process in the domain of biochemistry, e.g., Glycolysis (source analogy), and requested students to construct a scenario analogous to the explained process from a domain of their choice (target analogy). The instructor then evaluated 31 student-generated analogous scenarios as a strong or weak analogy based on its correspondence to the Biochemistry concept. A strong analogy corresponds well with the target analogy, and a weak analogy minimally corresponds with the target analogy. To increase the size of the 31 exemplar data set from the original we split each analogy into its constituent sentences, generating a data set of 526 strong exemplars and 140 weak exemplars. Each constituent sentence of an analogy falls into the same annotation category as the original analogy. Ergo, the initial input to the DAF pipeline is a sentence. This work does not distinguish between the analogy's target domains (Enzyme Kinetics and Glycolysis). Table 1 presents the summarized statistics of the dataset.

<table><caption>**Table 1**
Dataset statistics</caption><thead><tr><th></th><th>Strong</th><th>Weak</th></tr></thead><tbody><tr><td>Num. of analogies</td><td>25</td><td>6</td></tr><tr><td>Num. of analogies (sentences)</td><td>586</td><td>140</td></tr></tbody></table>
## 2.2. Input Processing

Sentences were processed and used as inputs to a Support Vector Machine (SVM) classifier (supervised learning) and K-means clustering (unsupervised learning) separately. In the following section we briefly review the background of input processing techniques, learning methods and implementation details.

### 2.2.1. Background

SVM is a supervised learning technique that creates functions to map inputs to pre-existing annotations [14]. SVM is an easy-to-understand classifier providing competitive performance in classification, regression, and outlier detection tasks [15]. The following paragraphs detail the background of four feature groups of interest here.

The obviously relevant features are semantic. Abstract Meaning Representation (AMR) is a semantic representation language that expresses a sentence's logical meaning by converting it to a rooted, directed, acyclic, edge-labeled, and leaf-labeled graph. [16]. To abstract away from syntactic idiosyncrasies, AMR assigns the same AMR graph to sentences with the same meaning. Nodes of an AMR graph are labeled as concepts, edges as relations, and concept properties as attributes. Concepts are either English words, PropBank framesets [17] or special keywords. There are approximately 100 relations [16]. AMR is used as a semantic representation of text in several NLP tasks such as summarization [18], machine comprehension [19], and event extraction [20, 21]. In this work we use AMR representations to extract concepts, relations and attributes present in sentence analogies. Figure 2 illustrates the AMR for a sentence from the dataset.

Sentiment-based features potentially reveal student engagement. Sentiment analysis aims to identify emotional or affective tendencies in user-generated content such as tweets, product reviews, and feedback [22]. Subjectivity detection and polarity determination are two common tasks in sentiment analysis [23]. Subjectivity quantifies the personal opinions versus factual information contained in the text. High subjectivity indicates the text contains more personal opinions compared to factual information [23]. Polarity describes the sentiment of a piece of text as positive, negative, or neutral [22].

We extract three groups of syntactic features. The first feature group is Part of Speech (POS), a grammatical classification of the word types in a sentence. These POS tags commonly include nouns, verbs, adjectives, etc. [24]. Named Entities Recognition (NER), the second feature group, is used to identify occurrences of named entities such as people, organizations, times, and locations in a sentence [25]. The third feature group is sentence type. Sentences in the dataset are identified as complex or compound sentences and simple sentences. In linguistics, complex sentences are sentences with two or more clauses connected with a subordinate conjunction. Simple sentences contain one independent clause [26].

We use four routine and straightforward statistical features, word count, character count, the average word length of a sentence (character count/ word count), and the number of unique words in a sentence.

K-means is a non-deterministic, iterative, and unsupervised machine learning technique to produce clusters from data [27]. Unsupervised learning here serves as both a baseline for
<figure><img src="image_50.png" /><figcaption>**Figure 2:** AMR representation of the sentence "Adding more marbles to the box will not increase the amount of product produced since it relies heavily on red marbles being oriented into the grooves." from the dataset.</figcaption></figure>

comparison with supervised learning results, and as a long term goal in itself, independent of any manual annotation. In the simplest test, we converted the input sentences to embeddings and cluster them using K-means clustering. Sentence embeddings were created using two techniques, context-based Embeddings from Language Models (ELMo) and knowledge-graph-based ConceptNet Numberbatch (CNNB). The following two paragraphs give a brief overview of these two embedding techniques.

ELMo embeddings are deep, contextualized representations of words computed using a two-layer bidirectional language model (biLM), which is pretrained on a large text corpus [28]. ELMo is robust in creating embeddings for out-of-vocabulary (OOV) words because it incorporates subword and character-level information when creating embeddings. Handling OOV words is particularly important in this work as most of the sentences often contain domain-specific keywords such as "Glucose-6-p", "DHP", "GAP" which can fall into the OOV category. ELMo sentence vectors are 1024-dimensional.

ConceptNet is a semantic network of knowledge about word meanings [29]. CNNB embeddings [30] are semantic word vectors created by encoding the knowledge in ConceptNet [29]. ConceptNet Numberbatch sentence embeddings are produced by taking the mean of single word
embeddings in a sentence. CNNB sentence vectors are 300-dimensional. CNNB embeddings
are not as robust as ELMo embeddings when handling OOV words, yet the percentage of OOV
words in the current dataset is rather small (6%). Hence we use CNNB as the second embedding
technique to create sentence embeddings.

2.2.2. Implementation Details

We use Pandas DataFrames [31] to process and manipulate the sentence features. We also
used other external libraries used in the extraction of sentence features as follows. To extract
semantic features, the sentences are sent through a transition-based AMR parser named CAMR
[32]. Textblob [33] is used to assess the subjectivity and polarity scores of the sentences. POS tag
and NER-related features (in syntactic features category) are extracted using spaCy <fnref n="1" />. Matplotlib
² and seaborn ³ are used for the visualizations.

ConceptNet Numberbatch embeddings are static representations for words available publicly
⁴. ELMo sentence embeddings were created using the model available at Tensorflow Hub⁵.

## 2.3. Analysis

In the following sections, we look at semantic, sentiment-based, syntactic, and statistical feature
distributions for strong and weak analogies. We then compare the performances of an SVM
classifier and K-means clustering.

Figure 3 illustrates the distributions of counts of concepts, relations, attributes, unique concepts, unique relations, and unique attributes of strong and weak analogies. Figure 4 presents the polarity and subjectivity distribution of strong and weak analogies. As shown in the plots, both strong and weak analogies contain sentences with neutral polarity and less subjectivity. Seventeen POS tags are present in the dataset. Distributions of the three most prevalent POS tags in strong and weak analogies are depicted in Figure 5 to utilize space effectively. Nevertheless, we used all 17 POS tags in the SVM classifier as features. Out of the fourteen named entities in the dataset (that are used in the SVM classifier), the distributions of the top three (ORG, CARDINAL, and PERSON) are plotted in Figure 6. We use the spaCy English pipeline ⁶ for NER tagging. Analogies written by students contain several references to biochemicals. These are misidentified as organizations (ORG) by spaCy, resulting in the ORG tag being the top named entity in the dataset. Figure 7 shows the distribution of simple and complex/ compound sentences. Weak analogies tend to have a slightly higher number of complex/ compound sentences, and strong analogies have slightly more simple sentences. Figure 8 presents the distributions of word counts, character counts, average word lengths, and unique word counts of strong and weak analogies. Modest discrepancies between distributions suggest the potential for such features to distinguish between strong and weak analogies. Therefore a feature vector combining the abovementioned features (engineered features) was then used in

<footnote n="1">¹https://spacy.io/
²https:// matplotlib.org/
³https://seaborn.pydata.org/
⁴https://github.com/ commonsense/conceptnet-numberbatch
⁵https://tfhub.dev/google/elmo/3
⁶https://spacy.io/models/en#en_core_web_md</footnote>
an SVM classifier to classify strong and weak analogies.

We use five variants of sentence vectors as inputs to the SVM classifier and K-means clustering. The first variant is the ELMo embeddings vector (ELMo). The second variant is the CNNB embeddings vector (CNNB), and the third variant is the engineered features vector with the vector dimension of 44. The fourth variant is a simple concatenation between ELMo embeddings and the engineered feature vector (ELMo composite). The fifth variant is a simple concatenation between CNNB embeddings and the engineered feature vector (CNNB composite).

<figure><img src="image_51.png" /><figcaption>**Figure 3:** Plots illustrating the distributions of (a) Concepts counts, (b) Relations counts, (c) Attributes counts, (d) Unique concepts counts, (e) Unique relations counts, and (f) Unique attributes counts of sentence analogies.</figcaption></figure>

<figure><img src="image_52.png" /><figcaption>**Figure 4:** Plots illustrating the distributions of (a) Polarity, (b) Subjectivity across sentence analogies.</figcaption></figure>

We opted to train an SVM classifier with stratified K-fold cross validation due to the limited size of our dataset (less than 1000 data points). Due to the imbalanced nature of the dataset and
<figure><img src="image_53.png" /><figcaption>**Figure 5:** (a) VERB, (b) DETERMINER, (c) NOUN POS tags counts distribution across sentence analogies.</figcaption></figure>

<figure><img src="image_54.png" /><figcaption>**Figure 6:** (a) ORG, (b) CARDINAL, and (c) PERSON NER tag counts distribution across sentence analogies.</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>**Figure 7:** Sentence type statistics of strong and weak analogies.</figcaption></figure>

identifying strong and weak analogies were equally important in this initial analysis, we used macro-F1 as the performance metric [34]. Performance of the SVM classifier with five variants of sentence vectors are listed in table 2.3.

We further inspect the contributions of the engineered features from the four feature categories mentioned in section 2.2 when discriminating between strong and weak analogies. We observe (see Figure 9) that most of the top 15 discriminating features belong to the syntactic
<figure><img src="image_56.png" /><figcaption>**Figure 8:** Plots illustrating the distributions of (a) Word count, (b) Character count, (c) Avg. word length, and (d) Number of unique words across sentence analogies.</figcaption></figure>

feature category, but a semantic feature contributes the most to discriminate between strong and weak analogies.

<figure><img src="image_57.png" /><figcaption>**Figure 9:** Top 15 discriminative features between strong and weak analogies</figcaption></figure>

We use K-means to cluster the five variants of sentence vectors mentioned above with cluster centers randomly selected and K set to two. Based on the Rand index <fnref n="7" />, the clusters are not well

<footnote n="7">⁷https://scikit-learn.org/stable/modules/clustering.html#rand-index</footnote>
<caption>**Table 2**</caption>

Different sentence vector variants and their performance on SVM classifier measured by *macro* averaged Precision/Recall/F1-score along with their K-means cluster qualities given by Rand Index.

<table><thead><tr><th rowspan="2">Vector Variant</th><th colspan="3">SVM Classifier</th><th>K-Means Clustering</th></tr><tr><th>Precision</th><th>Recall</th><th>F1-score</th><th>Rand Index</th></tr></thead><tbody><tr><td>ELMo embeddings</td><td>0.96</td><td>0.91</td><td>0.93</td><td>0.51</td></tr><tr><td>CNNB embeddings</td><td>0.86</td><td>0.82</td><td>0.84</td><td>0.52</td></tr><tr><td>Engineered features</td><td>0.75</td><td>0.64</td><td>0.67</td><td>0.54</td></tr><tr><td>ELMo composite</td><td>0.96</td><td>0.96</td><td>0.96</td><td>0.50</td></tr><tr><td>CNNB composite</td><td>0.83</td><td>0.82</td><td>0.81</td><td>0.52</td></tr></tbody></table>

segregated in any variant, yet engineered features embeddings performed slightly better (see Table 2.3).

# 3. Discussion

This section presents our findings and insights, followed by the limitations and future work.

## 3.1. Findings and Insights

We introduce the DAF pipeline to identify discriminative features of strong and weak analogies. We show that just a few engineered features does a surprisingly good job as input to SVM. To be sure, the ELMo composite sent through the SVM classifier performs better than the rest of the sentence vector variants. Nevertheless, the ELMo composite score is slightly higher (~0.03) than the ELMo. This increase highlights that the engineered features encode some aspects of the analogies not well-captured by the ELMo embeddings. Although the SVM's performance with the engineered feature vector is 26% lower than that of the ELMo embedding, its embedding size is ~23 times smaller than the ELMo. This phenomenon hints that considerable performance gains can be achieved with a much smaller number of better hand-crafted features, and most importantly, the better performance is explainable. We also note that the CNNB composite vector's performance in SVM is slightly poorer than that of the CNNB itself (~0.02). Although further exploration is required to explain this phenomenon clearly, we suspect this may be the result of feature multicollinearity specific to the manner in which CNNB creates its embeddings, combined with idiosyncracies of the subsets constructed in cross-validation.

We show that, among the features passed to the SVM classifier, the most discriminative feature for classification is a semantic feature (unique attribute count) and three out of the four semantic features (unique relations count, unique concepts count, concepts count) fall in the list of top 15 discriminative features. Also, among the top 15 discriminative features, syntactic features have the most representation. Overall, the engineered features are few in number, meaningful, and relatively cheap to calculate. Given the range of content in the data set –anything from marbles to cake–the modest success reported here is impressive. These features will contribute to our future efforts based on more computationally intensive semantic analysis. A successful unsupervised learning method would liberate classifier training from
dependence on manual annotation. Unsupervised learning results remain largely unimpressive. Nevertheless, there are some hints of promise. Engineered features improve clustering results relative to embeddings alone or embeddings and engineered features. This reinforces our claim that such features are identifying discriminators that are not captured by embeddings.

## 3.2. Limitations and Future Work.

The dataset used in this work is imbalanced, with more strong analogy data points than weak ones. This may cause the “uniform effect” where K-means produces clusters of the same size, even when the “true” cluster sizes of the dataset are varied [35]. To overcome such issues we plan to improve class imbalance through SMOTE [36], GANS[37], and the expansion of the manually-annotated corpus.

The natural language processing techniques employed in this work do not handle the particular nature of the dataset. For example, the spaCy model we use is trained on a generic English corpus <fnref n="8" />. However, we plan to use models/ techniques trained on subject-specific corpora to overcome issues like misidentifying biochemical terms as organizations in NER. Also, students use the term “like” in their target analogies to signify the similarity between their analogy and the source domain (biochemistry) concept. These are wrongly picked up by the sentiment analysis tool when evaluating polarity. Modified corpora will allow us to better manage these issues.

We classified analogy strength using individual sentences, which is both a benefit and a limitation. As a result, we identified very simple discriminators. However, some sentences in the dataset might not contribute when creating strong/ weak analogies. Constraining analysis to the sentence level requires annotation to eliminate this potential source of noise. However, the long-term goal is to evaluate analogies at the document level, for their epistemic quality. Though still vector based, our ongoing work in this area employs referent knowledge bases for both the target and variable student sources, to guide semantic interpretation.

## 4. Conclusion

This work introduces the DAF pipeline to identify discriminative features between strong and weak long-form analogies. We show that an SVM-based supervised-learning approach can successfully discriminate component sentences drawn from strong and weak analogies. Semantic and several syntactic features are the main contributors to discrimination, helping us to realize our goal of efficient evaluation of student generated long-form analogies.

## Acknowledgments

We thank Dr. Biplav Srivastava for his valuable feedback and Dr. Nitin Jain for providing the data used in this work. We also thank the reviewers for their constructive comments.

<footnote n="8">⁸https://spacy.io/models/en#en_core_web_md</footnote>
## References

[1] D. Gentner, Structure-mapping: A theoretical framework for analogy, Cognitive science 7 (1983) 155-170.

[2] D. J. Chalmers, R. M. French, D. R. Hofstadter, High-level perception, representation, and analogy: A critique of artificial intelligence methodology, Journal of Experimental & Theoretical Artificial Intelligence 4 (1992) 185-211.

[3] K. J. Holyoak, P. Thagard, Mental leaps: Analogy in creative thought, MIT press, 1996.

[4] D. C. Krawczyk, R. G. Morrison, I. Viskontas, K. J. Holyoak, T. W. Chow, M. F. Mendez, B. L. Miller, B. J. Knowlton, Distraction during relational reasoning: The role of prefrontal cortex in interference control, Neuropsychologia 46 (2008) 2020-2032.

[5] R. G. Morrison, D. C. Krawczyk, K. J. Holyoak, J. E. Hummel, T. W. Chow, B. L. Miller, B. J. Knowlton, A neurocomputational model of analogical reasoning and its breakdown in frontotemporal lobar degeneration, Journal of cognitive neuroscience 16 (2004) 260-271.

[6] N. Ichien, H. Lu, K. J. Holyoak, Verbal analogy problem sets: An inventory of testing materials, Behavior research methods 52 (2020) 1803-1816.

[7] H. Prade, G. Richard, Analogical proportions: Why they are useful in ai., in: IJCAI, 2021, pp. 4568-4576.

[8] S. Lim, H. Prade, G. Richard, Classifying and completing word analogies by machine learning, International Journal of Approximate Reasoning 132 (2021) 1-25.

[9] A. Ushio, L. Espinosa-Anke, S. Schockaert, J. Camacho-Collados, BERT is to NLP what alexnet is to CV: Can pre-trained language models identify analogies?, in: ACL 2022 Workshop on Commonsense Representation and Reasoning, 2022. URL: https://openreview.net/forum?id=BdWgrMFxdW9.

[10] B. A. Spellman, K. J. Holyoak, Pragmatics in analogical mapping, Cognitive psychology 31 (1996) 307-346.

[11] B. Falkenhainer, K. D. Forbus, D. Gentner, The structure-mapping engine: Algorithm and examples, Artificial intelligence 41 (1989) 1-63.

[12] M. McLure, S. Friedman, K. Forbus, Extending analogical generalization with near-misses, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.

[13] R.Shrem, T. Vonderhaar, V. Shalin, N. Jain, Use of student-generated process analogies to enhance student engagement, in: (in preparation), 2022.

[14] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (1995) 273-297.

[15] J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, A. Lopez, A comprehensive survey on support vector machine classification: Applications, challenges and trends, Neurocomputing 408 (2020) 189-215.

[16] L. Banarescu, C. Bonial, S. Cai, M. Georgescu, K. Griffitt, U. Hermjakob, K. Knight, P. Koehn, M. Palmer, N. Schneider, Abstract Meaning Representation for sembanking, in: Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, Association for Computational Linguistics, Sofia, Bulgaria, 2013, pp. 178-186. URL: https://aclanthology.org/W13-2322.

[17] P. Kingsbury, M. Palmer, From TreeBank to PropBank, in: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02), European Language Resources Association (ELRA), Las Palmas, Canary Islands - Spain, 2002. URL:
http://www.lrec-conf.org/proceedings/lrec2002/pdf/283.pdf.

[18] F. Liu, J. Flanigan, S. Thomson, N. Sadeh, N. A. Smith, Toward abstractive summarization using semantic representations, in: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Denver, Colorado, 2015, pp. 1077-1086. URL: https://aclanthology.org/N15-1114. doi:10.3115/v1/N15-1114.

[19] M. Sachan, E. Xing, Machine comprehension using rich semantic representations, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016, pp. 486-492.

[20] S. Rao, D. Marcu, K. Knight, H. Daumé III, Biomedical event extraction using Abstract Meaning Representation, in: BioNLP 2017, Association for Computational Linguistics, Vancouver, Canada., 2017, pp. 126-135. URL: https://aclanthology.org/W17-2315. doi:10.18653/v1/W17-2315.

[21] L. Huang, T. Cassidy, X. Feng, H. Ji, C. Voss, J. Han, A. Sil, Liberal event extraction and event schema induction, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 258-268.

[22] C. Puschmann, A. Powell, Turning words into consumer preferences: How sentiment analysis is framed in research and the news media, Social Media+ Society 4 (2018) 2056305118797724.

[23] E. Kasmuri, H. Basiron, Subjectivity analysis in opinion mining—a systematic literature review, Int J Adv Soft Comput Appl 9 (2017) 132-159.

[24] A. Chiche, B. Yitagesu, Part of speech tagging: a systematic review of deep learning and machine learning approaches, Journal of Big Data 9 (2022) 1-25.

[25] A. Mikheev, M. Moens, C. Grover, Named entity recognition without gazetteers, in: Ninth Conference of the European Chapter of the Association for Computational Linguistics, 1999, pp. 1-8.

[26] B. Das, M. Majumder, S. Phadikar, A novel system for generating simple sentences from complex and compound sentences, International Journal of Modern Education and Computer Science 11 (2018) 57.

[27] J. MacQueen, Classification and analysis of multivariate observations, in: 5th Berkeley Symp. Math. Statist. Probability, 1967, pp. 281-297.

[28] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 2227-2237. URL: https://aclanthology.org/N18-1202. doi:10.18653/v1/N18-1202.

[29] R. Speer, J. Chin, C. Havasi, Conceptnet 5.5: An open multilingual graph of general knowledge, in: Thirty-first AAAI conference on artificial intelligence, 2017.

[30] R. Speer, J. Chin, An ensemble method to produce high-quality word embeddings, arXiv preprint arXiv:1604.01692 (2016).

[31] W. McKinney, et al., pandas: a foundational python library for data analysis and statistics, Python for high performance and scientific computing 14 (2011) 1-9.

[32] C. Wang, S. Pradhan, X. Pan, H. Ji, N. Xue, Camr at semeval-2016 task 8: An extended
transition-based amr parser, in: Proceedings of the 10th international workshop on semantic evaluation (semeval-2016), 2016, pp. 1173-1178.

[33] S. Loria, et al., textblob documentation, Release 0.15 2 (2018).

[34] M. Sokolova, G. Lapalme, A systematic analysis of performance measures for classification tasks, Information processing & management 45 (2009) 427-437.

[35] H. Xiong, J. Wu, J. Chen, K-means clustering versus validation measures: A data-distribution perspective, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39 (2009) 318-331. doi:10.1109/TSMCB.2008.2004559.

[36] N. V. Chawla, K. W. Bowyer, L. O. Hall, W. P. Kegelmeyer, Smote: synthetic minority over-sampling technique, Journal of artificial intelligence research 16 (2002) 321-357.

[37] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014).