dataesr/scientific-paragraphs-categorization huggingface dataset:
List of previously published tracking data used in our analysis. The table lists major characteristics of the datasets we used to show within-individual speed variability. The # of tracks indicates the number of individual tracks used for the analysis. Du to the initial study designs and questions, tracks may represent repeated measures of the same (Guppy: 20 individuals, RoboFish 1 replica) or different individuals (Molly single ind, Molly groups: 8 with 4 ind. per group). Please find exact study designs in the respective references.
For instance, electrons are attracted by the nucleus as planets are attracted by the sun, leading to electrons : nucleus :: planets : sun where the underlying relation is "orbit around". That is why most computational models of analogical reasoning assume that a binary relation exists between the pairwise components of an analogical proportion. Unfortunately, there is no clear knowledge about how these abstract relations are established by a human brain  [34]. Very often, representations of formal relations are given in a top-down manner by assuming a grammar of relations (see  [33]  for instance). Or these binary relations are extracted from data as it is the case in  [42]  for instance. But very few of these models of relational learning have been applied to real-life datasets such as the ones which are currently manipulated by the NLP community.
In this paper, we choose option (b). More precisely, we proceed as follows: 1. We try machine learning algorithms as binary classifiers of analogies. Such a classifier takes as input a quadruple of words (a, b, c, d) and output 1 if a : b :: c : d holds, 0 otherwise. 2. We also test algorithms for solving analogical equations such as a : b :: c : x. The solving process is a regression task where we have to approximate a hidden function f such that the solution of the equation a : b :: c : x is x = f (a, b, c). For experimenting our approach on the two tasks above, we need to get datasets of quadruples of words (a, b, c, d) which are considered as cognitively acceptable analogical proportions. Moreover, if we consider that an analogical proportion (a : b :: c : d) can be built from two pairs of words (a, b) and (c, d) sharing the same relation R, we can also take advantage of dataset of pairs to build new analogical proportions.
and 4. 1 As a consequence, we experiment on 4 datasets: 1. The first one is the standard Google Analogy Test Set having more or less 20,000 analogical proportions 2. 2. The second one is BATS (Bigger Analogy Test Set)  3   [18] , which is made of pairs, and as such departs from Google dataset. From the pairs in BATS, we can obtain 99,200 quadruples of words viewed as analogical proportions in 40 morphological and semantic categories. 3. The third one, usually called DiffVec 4   [59]  is also made of pairs. Vylomova et al. [59]  have introduced this large dataset covering many well-known lexical relation types, leading to a total of 36 classes of pairs, highly unbalanced in terms of size (from a class of 6 examples to a class of 3,583 examples), with a total number of 12,458 pairs. DiffVec refers to Difference Vectors method.
Bouraoui et al. [8]  keeps the name DiffVec for the dataset of pairs introduced in  [59]. We shall continue to use DiffVec for referring to the set of quadruples that we build by taking pairs of pairs in the same class (as we do with BATS). 4. Finally, the fourth one is SAT (Scholastic Assessment Test), a former college entrance exam  [57]   5. This dataset is quite small with only 374 analogies such as livid : anger :: radiant : happiness but it is so challenging that even a very good student will get no more than 60% success rate. This paper is a fully revised and expanded version of a conference paper  [31]. Apart from adding more extensive sets of experiments, we have also investigated in more detail the impact of the formal modeling of analogical proportions on the machine learning implementation.
We make several contributions to the field as summarized below: 1. We formally investigate what can be expected from an analogical relation. We show that an intuitive property like unicity postulate does not fit with word analogies. As such, it has to be rejected. 2. Starting from the previous formal properties expected from an analogy relation, we build positive and negative training sets from our datasets. Doing so, we are able to build classifiers from a proper set of examples. 3. Apart from testing diverse machine learning classifiers, we design and implement a neural network to decide whether a quadruple of words is an analogy or not, with better accuracy than state of the art algorithms, and even higher accuracy than an average human on the SAT dataset. 4. Finally, we depart from the traditional view of analogical proportions where finding x such that a : b :: c : x holds is based on a parallelogram view where the missing x is just the vertex of the parallelogram (a, b, c, x).
The machine learning approaches outperforms state of the art approaches on Google, BATS and DiffVec datasets. As far as we know, neither analogy classification nor analogy completion have been investigated in the same way as we have proposed in this paper, namely learning a model, instead of starting from the parallelogram model. The paper is structured as follows. Section 2 recalls the postulates characterizing analogical proportions and identifies a rigorous method for enlarging a set of examples and counter-examples (also known as data augmentation in the machine learning community). Section 3 and its subsections provide a new approach to the recognition/classification of analogical proportions, experimenting with several algorithms (SVM, random forests, neural networks) for this purpose.
Moreover, we also propose a learning approach for solving analogical proportion equations in natural language (often referred to as analogy completion in the natural language processing literature). Section 4 presents and discusses the experimental settings and Section 5 reports results for datasets Google, BATS and DiffVec. Experiment results show that machine learning-based techniques are more accurate than the state of the art, both for classification and regression tasks. We dedicate Section 6 to experiments and results on the SAT dataset. Related work is discussed in Section 7, before concluding in Section 8.
As we understand, distinguishing the two types of analogical proportions might have consequences not only on the logical modeling but also on its experimental counterpart. However, there are also examples of analogical proportions involving two conceptual spaces where central permutation is tolerable, for instance with: "Tokyo": "Japan"::"Paris":"France" This proportion is based on the relation T oBeT heCapitalOf shared by the two pairs. For a long time, there have been many discussions about what makes an analogical proportion valid or fallacious  [21, 50]. Generally speaking, it may seem natural to assume that an analogical proportion can simply be defined as: a : b :: c : d iff for some relation R, R(a, b) and R(c, d) hold. This assumption is implicit when using datasets such as BATS and DiffVec.
• If R is symmetrical, i. • The relation R may be arbitrary, quite general or vague, as in: "camera":"water"::"attention":"baby" which would hold because the number of syllables of the first word in a pair is just the number of syllables of the second word increased by 1. In summary, the only properties that hold for sure with relation-based analogical proportions are 'symmetry', 'internal reversal', and 'complete reversal', while other properties are debatable. Ultimately, the acceptable orderings of 4 words making a proper analogical proportion may be viewed as a matter of convention, even if some orderings are certainly more cognitively valid than others. This is not an issue for the machine learning approaches proposed in this paper since they consider as valid analogical proportions whatever is given in the dataset of positive examples.
This leads to the well-known parallelogram view of analogies. In fact, this view has to be seriously tuned to detect analogies in a diverse corpus of data  [18, 30, 8]. Given the embedding process (here GloVe in the experiments), we depart from this assumption and tackle the issue from another viewpoint. Instead of assuming that a word analogy a : b :: c : d is valid iff embed(a) -embed(b) embed(c) -embed(d), which allows to classify a quadruple of words, we learn from word analogies examples how to classify without assuming any underlying predefined formula. Equipped with the observations from Section 2, we have a rigorous way to augment the initial datasets, as it is often needed in machine learning. Let us first investigate some works related to what we do and which could, in one way or another, serve as baselines, before presenting our approaches to analogy classification and completion.
The regression model relaxes the simplistic assumption of the difference-based approach from the parallelogram view, but is generally outperformed by the translation model. None of these two approaches are used in an analogy completion perspective. In view of the above descriptions, it is natural to compare our works with: • The current best method from the translation (and regression) model  [8]  for analogy classification task, since it outperforms the other approaches; • 3CosM ul and LRCos for analogy completion task. In fact, none of the previous works have been experimented on SAT dataset. As a consequence, regarding our experiences with SAT, we will compare with the works of  [56]. We now describe our approach for classifying word analogies, and then for analogy completion.
Experimental settings As explained in Section 1, in all our experiments, we use GloVe  [43]  as the embedding model. Glove provides several publicly available pre-trained R n word vectors which can be downloaded from https://nlp. stanford. edu/projects/glove/, with dimension n in {50, 100, 200, 300} (Wikipedia dump). Ultimately, our datasets are loaded as CSV files. In the case of our CNN for classification, each row is compiled into a real-valued matrix of dimension n × 4 that we consider as an image. The GloVe embedding originally contained 400,000 words. To speed up processing time, we removed non-alphabetical words (numbers, punctuations, underscores), and we are left with 317,544 words. This has no effect on the embedding process as the removed words do not appear in the analogy datasets  11 Before presenting our four datasets (Google, BATS, DiffVec and SAT) in detail, and the way we build a training set for machine purposes, we discuss the link between relational induction and analogy classification, and we propose a measure of the quality of a dataset as a repository of analogical proportions, given a word embedding.
• The relation AT T RIBU T E Action : ObjectAttribute is satisfied by the two word pairs (paint, house) and (wine, drink). Still, it is hard to think of paint : house :: wine : drink as a genuine analogical proportion, since the explanation to support it seems quite intricate. The parallel between two pairs sharing some relation does not always seem to be a sufficient requirement for making widely acknowledged and thus cognitively valid analogical proportions, as discussed in detail in Subsection 2. 4. The above remarks tend to suggest that, strictly speaking, both BATS and DiffVec cannot be considered as proper analogical proportions datasets. A measure of the analogical quality of a dataset, supporting this claim, is presented in the next Subsection. However, as mentioned before, the proposed machine learning procedures still apply to "weak" analogical proportions.
Average analogical dissimilarity of a dataset As suggested by Linzen  [32] , the parallelogram equation for solving word analogies has become a standard evaluation tool for words embedding (such as GloVe). In the following, we reuse this idea in a different perspective. We thus assume that given a reasonably good word embedding, such as GloVe, datasets made of high quality analogical proportions should lead to parallelograms of better quality than datasets containing weak proportions. Let us explain how we have checked this assumption on the 4 datasets. by BATS, DiffVec and SAT. Surprisingly, the deviations of SAT and DiffVec are very similar, but SAT is regarded as a repertory of high quality proportions. This is likely due to the fact that SAT proportions involve sophisticated vocabularies, which occur less frequently in the corpus documents so the models for those words may be less accurate.
Extending datasets for classification Due to the properties of analogical proportions reviewed in Section 2, it is easy to rigorously extend a dataset of analogies by applying the diverse postulates. This process is commonly known as data augmentation, since one uses properties or structures of the specific data available to produce new ones. Although the most common application is in image processing (where results should be invariant by rotation, zoom, etc. ), it can also be applied to analogies using their structural properties. Starting from an initial dataset of n distinct analogies, each one involving 4 different words, we end up with 8×n valid analogies which constitute the positive examples. But in machine learning, negative examples are also needed. By using the same initial dataset, we get invalid analogies just by permuting 2 elements of a valid analogy a : b :: c : d: • Starting from n valid analogies and permuting the 1st and the 2nd elements provides an invalid analogy.
Then permuting 8 times this invalid analogy, we still get 8 invalid analogies leading to 8 × n invalid analogies. • Then, applying the same process after permuting the 1st and the 3rd ele-ments, leading again to 8 × n invalid analogies. We get a final extended dataset of n×24 examples, among which n×8 are positive examples and the remaining n × 16 are negative examples. Note that we apply central permutation in a systematic way, although we pointed out in Subsection 2. 4 that it may be debatable. However it does not affect the results as shown in experiments; indeed we never test both (a, b, c, d) and (a, c, b, d). Another important point to note: the negative examples are only permutations of the positive ones. As it is highly unlikely that 4 random words (a, b, c, d) constitute a valid analogy, we have also experimented by adding such random negative examples.
But this random extension did not bring better results than the logical extension. This should not come as a surprise: adding negative examples involving the same words as the positive examples is likely to be much more informative than adding random negative examples, since the negative examples obtained by permutation are "closer" to the positive ones. In the following, we denote a dataset of n analogies by T S and the extended dataset of 24×n elements by Ext(T S). We have diverse ways to use our extended datasets as explained below.
Working with extended datasets for classification From a machine learning perspective, we have to split a dataset T S into 2 subsets train(T S) and test(T S). Let us denote split(S) a function leading to a pair of disjoint subsets train(S), test(S) from a set S. It is quite clear that creating a pair of training and testing set, starting from an initial dataset T S can be done in three ways: (Ext(train(T S)), test(T S)) Method 1 and Method 2 have exactly the same number of examples in total. This is not the case for Method 3 as there are fewer elements to be tested. Intuitively, the performances from these methods may be different: Method 2 and Method 3 can be considered tougher in the sense that there is no permutation of any training analogy in the testing set: the training and testing sets are entirely disconnected.
Experimental comparison between Method 1, Method 2 and Method 3 Before launching a full batch of experiments which are very time consuming, we have decided to investigate if one method is definitely better than the other. We consider the Google dataset as clean and simple for analogy classification task, so we started our experiments with it in order to get some indication about the most relevant method to use. Using the CNN described in Section 3, we experimented in dimension 50, with a split ratio train/test of 0. 2, i. e 20% of the dataset are retained for testing, the remaining 80% used for training. The resulting accuracies and losses (cross-entropy) are in Table  3   of one method over another (at least with these parameters). As a consequence of these indicative experiments, we choose Method 2 in the following, because we consider this method provides a clean separation between training and testing sets (which is not the case with Method 1) and generates a bigger test set than Method 3.
Experimental results for Google, BATS and DiffVec This section is devoted to the results on the classification and regression tasks for datasets Google, BATS and DiffVec; Section 6 covers the SAT dataset. Note that for the DiffVec dataset, we exclude classes larger than 62,000 analogies, as some algorithms require more resources than what we have: Classes 13 and 36 are excluded for this reason. For SVM, we also exclude class 14 and 16 as they are too big for SVM. For LRCos, we also exclude Class 1 and Class 15, as there are only six word pairs and LRCos requires more samples.
Analogy testing as a classification task We employ the commonly used metrics for classification tasks: precision, recall, F1, and accuracy collected from 10-fold cross-validation. A summary of the results showing the performance of the SVM, Random Forest, Neural Network (NN) and CNN classifiers is given in Table  4  for Google dataset, in Table  5  for BATS and in Table  6  for DiffVec. Complementary tables are provided in annexes. The NN was trained using 10, 30, 100, 200, 300 and 400 epochs. We present the results for 30 and 100 epochs just because there are no big changes for more than 100 epochs. The CNN was trained using 3, 5 and 10 epochs and we present the results for 10 epochs. Tables 4, 5 and 6 show that the results are uniformly very good for Google, BATS and DiffVec, whatever the machine learning algorithm used.
For Google dataset (Table  4 ), the accuracies for all four classifiers are nearly 100%. Note that simpler classifiers such as Random Forest and simple Neural Network still achieve very high accuracy even at dimension 100, and the performance is about the same in higher dimensions. For CNN, the accuracy increases by about 2% when the number of dimension increases from 50 to 100. However, the gain appears to be less significant for the higher dimensions, as the accuracy for 200 dimensions is only 0. 42% higher than 100 dimensions, and the accuracy for 300 dimensions is actually slightly lower (0. 16% lower) than 200 dimensions. It appears that using dimension 100 is more than enough to get a clear view of the value of the 4 metrics. This is why in Tables  5  and 6  we only use at most dimension 100.
For BATS, we provide average values for the metrics per BATS class and for DiffVec, a sample of four classes and an average of all classes. As can be seen in Table  5 , the performance of the algorithms vary with the classes. For instance, for BATS, the accuracy of CNN decreases to 69. 81% on the Lexicographic-semantics subcategory 'antonyms-binary' (see Table  B. 14). This should not come as a surprise since the antonym relationship is a very abstract relation and some instances may be even debatable analogical proportions for people, e. g. , 'before:after::dive:emerge'. Table  6  shows the individual results of 4 diverse categories for DiffVec and the average of 34 classes; for the complete results please see Appendix C. For the DiffVec dataset, NN and Random Forest perform the best, followed by CNN, and lastly SVM.
Analogy completion as a regression task Here, we also used R n word embeddings with n ∈ {50, 100, 200, 300}. The network proposed in Section 3. 4 was trained for 50 epochs 10 fold cross-validation. As suggested in Subsection 3. 1, the results for NN have to be compared with the works of  [19]  for 3CosM ul (code provided by  [30] ) and LRCos (code provided by  [15] ). The results are in Table  7  for Google dataset, Table  8  for the BATS dataset and Table  9  for the DiffVec dataset. As seen from all three tables, NN regression outperforms 3CosM ul and LRCos by a large margin -LRCos was implemented using leave-one-out cross validation. However, 3CosM ul and LRCos have some advantages over NN: they are much faster to execute and require less resources. Note that the NN was trained for each category.
In our previous work  [31] , the NN model was trained on all categories for the Google dataset, but we feel that it would be a fairer comparison with LRCos if the NN was trained for each category, as the relation in LRCos was extracted for each category. In Section 3. 4, we explained that both NN and LRCos try to exploit the relationship of d with a, b and c (as 3CosM ul). In LRCos, a (e. g. , 'Japan') serves as  a negative example and b (e. g. , 'Tokyo') serves as a positive example, and that d should be close to c (e. g. , 'France'), but the relationships perhaps may not be well captured during the regression process. Using NN, the relationship of d with a, b and c is captured under the form g(f 1 (a, b), f 2 (a, c)) via the NN structure. This may explain the better results obtained with NN.
Tables 7, 8 and 9 also demonstrate that the accuracies of both 3CosM ul and LRCos increase as the dimension of the embedding increases. Interestingly, this is not the case for NN: its accuracies are less dependent on the dimension -only in the DiffVec dataset that the average accuracy increases by 3. 85% from dimension 50 to dimension 100, and the increase is insignificant (only 0. 02%) from dimension 100 to dimension 200. This appears to support the argument put forward by Drodz et al. [15]  that a better method can overcome the shortcoming of an embedding. Up to a point, this argument is acceptable, when the relationships are simple, such as those presented in the Google, BATS and DiffVec datasets. Nevertheless, when the relationships are very complex, such as those in SAT, then it would be difficult to capture those relationships using NN as discussed in Section 6.
Semantic analogies : SAT dataset As we have previously seen, the machine learning approaches are quite successful in recognizing word analogies from Google, BATS and DiffVec datasets. Nevertheless, all the datasets are mainly made of morpho-syntactic analogies, not really semantic analogies. We are not in a position to provide a clear and concise definition of semantic analogies but everybody understands the difference between 'man:men::woman:women' (morpho-syntactic) and 'ostrich:bird::lion:cat' (semantic, extracted from SAT). SAT is generally considered as a well-accepted set of semantic analogies: this is the set of analogies used in the SAT college entrance test. It is a set of 374 multiple-choice questions where a pair of words (a : b) has five options (c : d) but only one option makes a valid analogy with the first pair.
So we have 374 × 5 examples among which only 374 of the examples are valid semantic analogies. Thus, SAT provides both positive and negative examples -which is not the case for Google and BATS datasets where we only have positive examples, at least initially. Below is an example from the SAT dataset: With his SuperSim (for 'supervised similarity') approach, Turney  [56]  achieves a score of 54. 8% of correct answers for the classification task, which reveals the inherent difficulty of this dataset. In order to work with a balanced dataset, Turney selectively increases the number of positive examples by applying symmetry postulate and internal reversal property (see Section 2)  13. Apart from classifying analogies, SuperSim can also perform other tasks.