Year: 2021

Abstract:
Antonymic and synonymic pairs may both occur nearby in word embeddings spaces because they have similar distributional information. Different methods have been used in order to distinguish them, making the antonymy-synonymy discrimination a popular NLP task. In this work, we propose the repelling parasiamese neural network, a model which considers a siamese network for synonymy and a parasiamese network for antonymy, both sharing the same base network. Relying in the antagonism between synonymy and antonymy, the model attempts to repel siamese and parasiamese outputs making use of the contrastive loss functions. We experimentally show that the repelling parasiamese network achieves state-of-the-art results on this task.

text: Antonymic and synonymic pairs may both occur nearby in word embeddings spaces because they have similar distributional information. Different methods have been used in order to distinguish them, making the antonymy-synonymy discrimination a popular NLP task. In this work, we propose the repelling parasiamese neural network, a model which considers a siamese network for synonymy and a parasiamese network for antonymy, both sharing the same base network. Relying in the antagonism between synonymy and antonymy, the model attempts to repel siamese and parasiamese outputs making use of the contrastive loss functions. We experimentally show that the repelling parasiamese network achieves state-of-the-art results on this task.

heading: 1 INTRODUCTION
text: Semantic opposition is a binary relation of central importance in the cognitive baggage of human languages. It establishes that one term contradicts the other, that both cannot be satisfied simultaneously. In the context of lexical semantics, it corresponds to antonyms (e.g. light and dark), whose recognition is essential for natural language usage. For instance, this capability is crucial for text entailment and paraphrasing, which are basic abilities for different NLP tasks.
Most of modern NLP is using word embeddings (i.e. vectors for word meanings built from word contexts and subword information). These word representations have the potential to cluster words according to their distributional information on a corpus. However, since antonyms tend to occur in similar contexts, word embeddings may have close vectors in the space. Faced to this problem, different approaches have been proposed to re-encode the word embeddings in a supervised learning setup for the antonymy-synonymy discrimination task (Mrkšić et al., 2017; Etcheverry & Wonsever, 2019; Samenko et al., 2020; Xie & Zeng, 2021).
In this work, we deepen in the parasiamese network as an antitransitive relationship learning approach, and we propose the repelling parasiamese neural network: a model that simultaneously opposes the siamese and parasiamese outputs (of a same base network). We present two independent alternatives to do so: (1) pair and (2) triplet based approaches. We experimentally evaluate different alternatives and we introduce a formulation to enforce symmetry through the network structure. We carry out our experiments in three datasets: the publicly available antonymy-synonymy dataset introduced by Nguyen et al. (2016), a here introduced dataset confeccionated from Samuel Fallow’s antonym’s dictionary (accessed through the Gutenberg project) and in a version of the Nguyen et al. (2016)’s dataset splitted without lexical intersection between train, validation and test introduced by Xie & Zeng (2021). We show that the repelling parasiamese neural network achieves better performance than its predecessor, the (non-repelling) parasiamese network, and the best performing models found in the literature.

heading: 2 SOME PRELIMINARIES
text: Before getting into the repelling parasiamese neural network, let’s introduce some preliminary concepts concerning antitransitivity, metric learning for antonymy and the parasiamese network.

heading: 2.1 ANTONYMY AND ANTITRANSITIVITY
text: Antonymy can be considered as an antitransitive relationship1. If two lexical units are antonyms of a third (e.g. huge and enormous being opposite of small) then they will not oppose each other; in fact, they will often present semantic similarity (Edmundson, 1967). In table 1 we sample antonyms for some words from Fallow’s dictionary. Supporting the claimed antonymy antitransitivity, it can be seen that the words on each antonymy list (i.e. common antonyms of a word) do not oppose between them, and many cases of similarity can be detected (e.g. savage and wild).

heading: 2.2 A METRIC FOR ANTONYMS
text: Siamese networks are among the best performing approaches for text semantic similarity tasks (Tran et al., 2020; Ranasinghe et al., 2019; Mueller & Thyagarajan, 2016). The properties that similarity relations tend to have, such as reflexivity, symmetry and transitivity; are suitable for metrics and particularly for siamese networks. To clarify, suppose a metric d, reflexivity and symmetry arise directly from the metric definition, precisely from d(x, x) = 0 and d(x, y) = d(y, x). Concerning transitivity, it is related to triangular inequity. The triangular inequity establishes that for any triplet (x, y, z) of words:
d(a, c) ≤ d(a, b) + d(b, c) (1)
So, given two pairs of related words (a, b) and (b, c), then due to the triangular inequity, d(a, c) is bounded by the sum of d(a, b) and d(b, c), which are expected to be low values since (a, b) and (b, c) are related. This makes the pair (a, b) to tend to be related as well, and therefore, the relation transitivity.
If instead of a metric for similarity we consider a metric for opposition, e.g. antonyms, the aforementioned regarding triangular inequity is a drawback. The metric function will not be suitable for the antitransitivity of the opposition relation, tending to return low values (i.e. treat them as related) for the unrelated pair of words in the anti-transitive triangles. In other words, for each pair of words with a common antonym (e.g. short and brief as antonyms of long), the metric will tend to wrongly treat them as antonyms as well.
In conclusion, the triangular inequity is beneficial for transitivity but it is problematic for antitransitivity. In the following section we describe the parasiamese network, a siamese-like neural network that does not satisfy triangular inequity and is suitable for learning antitransitive relations.

heading: 2.3 THE PARASIAMESE NETWORK
text: The parasiamese network (Etcheverry & Wonsever, 2019) was introduced as inspired by the siamese network, being better suited for the learning of antitransitive relations. Just like the siamese network, it consists of a model that consumes two vectors and returns a non-negative value; and it relies on a base neural network that is applied more than once, sharing its weights, to compute the output. The parasiamese network differs from the siamese formulation in the fact that the base network is applied once to one input and twice to the other, instead of once to each input, as in the siamese network. The double application of the base network in the parasiamese network, imposes that the base network must have the same dimension for its input and output. The output of the parasiamese network is the distance between both branches (see Figure 1).
1We remind that a binary relation R is called antitransitive iff ∀a, b, c (a R b ∧ b R c→ a 6R c)
Let Fθ : Rn → Rn be a neural network with trainable parameters θ. Then, the parasiamese network with base network Fθ is defined by
ΦFθ (x, y) = ||Fθ(x)− Fθ(Fθ(y))||2, (2)
where ||.||2 is the Euclidean norm. The model is trained through the contrastive loss function. Concretely, ΦFθ is trained through mini-batch stochastic gradient descent on:
L = ∑
(x,y)∈P
[ΦFθ (x, y)− µp]+ + ∑
(x′,y′)∈N
[µn − ΦFθ (x′, y′)]+, (3)
where P and N are, respectively, the positive and negative pairs in the dataset; and µp and µn are the positive and negative thresholds, respectively. The [.]+ notation corresponds to the ReLU function. The training attempts to pull closer than µp the related elements and push away unrelated pairs further than µn.
This definition, unlike the siamese network, does not enforce transitivity even when the parasiamese output of the two related pairs in the antitransitive triangle are strictly zero. Moreover, the relation given by ΦFθ and a threshold µ (i.e. RΦFθ ,µ = {(a, b) : ΦFθ (a, b) ≤ µ}) is benefited concerning antitransitivity if ΦFθ (w,w) > µ, which is consistent with the fact that antitransitive relations are necessarily antireflexive2.
In addition, the unrelated pair in the anti-transitive triangles will present a low value for the siamese formulation using the same base network. If (a, b, c) is an antitransitive triangle with unrelated pair (a, c), then:
||Fθ(a)− Fθ(c)||2 = ||Fθ(a)− Fθ(c) + Fθ(Fθ(b))− Fθ(Fθ(b))||2 ≤ ||Fθ(a)− Fθ(Fθ(b))||2 + ||Fθ(c)− Fθ(Fθ(b))||2
and the parasiamese formulations of (a, b) and (c, b) are expected to output low values since they are both related.
A possible interpretation for the parasiamese definition is thinking the base network F as an opposition transformation. So, if we consider two opposite terms a and b (i.e. a ∼ ¬b), it is expected that opposition remains when both terms are negated (i.e. ¬a ∼ ¬¬b), which brings the parasiamese formulation: F (a) ∼ F (F (b)).
2To support this claim we rely in ΦFθ (a, c) ≤ ΦFθ (a, b) + ΦFθ (b, b) + ΦFθ (b, c).

heading: 3 THE REPELLING PARASIAMESE NETWORK
text: The repelling parasiamese network is based on contrasting the siamese and parasiamese networks in a differentiable formulation (Figure 2). By doing this, we consistently observe a performance improvement in comparison to its predecessor (i.e. the parasiamese network without repelling its siamese counterpart).
The siamese counterpart of a parasiamese network may reflect the similitude-like relation that emerges from being opposed to the same elements through the antitransitive relation. Recalling the case of lexical semantics, the siamese formulation that comes from a parasiamese network that models antonymy, may be suitable to model synonymy, since the words that share antonyms may tend to be synonyms. Hence, given the antagonism between antonymy and synonymy, it seems suitable that both outputs should not be simultaneously low for the same outputs.
Let a, b, c be three inputs with the pairs a, b and b, c being related and therefore returning low parasiamese outputs. Then, as commented in the previous section, the output for the siamese formulation of the same base network for the pair a, c will present a low value. Simultaneously, its parasiamese output is expected to be greater than the acceptance threshold, because of the antitransitivity. This suggests contrariness between siamese and parasiamese formulations in the unrelated pairs of antitransitive triangles.
Moreover, if a pair (a, b) presents low siamese and parasiamese outputs, it would implies a reflexive pair within the antitransitive relation, since:
ΦFθ (a, a) = ||Fθ(a)− Fθ(Fθ(a))||2 = ||Fθ(a)− Fθ(Fθ(a)) + Fθ(b)− Fθ(b)||2 ≤ ||Fθ(a)− Fθ(b)||2 + ||Fθ(a)− Fθ(Fθ(b))||2
which is inconsistent with antitransitivity. So, it may be suitable to contrast the parasiamese and siamese outputs (using a same base network) when one of them returns a low value.
To describe the repelling parasiamese network let introduce the following notation:
• Parasiamese left and right branches: We will refer as left and right branches of the parasiamese network to the transformations applied to the left and right parts of the relationship (that correspond to the left and right terms of the distance in equation 2), and we will write them as αθα : Rn → Rn and βθβ : Rn → Rn, respectively. So, in the non-repelling proposal αθ(x) = Fθ(x) and βθ(x) = Fθ(Fθ(x)).
• Parasiamese output function: We will use the notation Φαθα ,βθβ for to the binary function that given the left and right branches, αθα and βθβ , returns the distance between them, i.e. Φαθα ,βθβ (x1, x2) = ||αθα(x1)− βθβ (x2)|| 2 2. Notice that Φαθα ,αθα (x1, x2) corresponds to
the siamese network formulation.

heading: 3.1 SIAMESE-PARASIAMESE REPULSION
text: In order to formulate the repelling parasiamese network, lets consider the parasiamese output function Φαθα ,βθβ as the following two functions:
• Φ(p)αθα ,βθβ : R d × Rd → R≥0 as a function that returns a low value when the parasiamese
output presents a low value and the siamese network a high value (e.g. higher than a threshold).
• Φ(s)αθα ,βθβ : R d × Rd → R≥0 as a function that returns low value when the siamese net-
work presents a low value and the parasiamese network a high value (e.g. higher than a threshold).
So, Φ(p)αθα ,βθβ and Φ (s) αθα ,βθβ are the parasiamese and siamese networks, respectively, with its respective counterparts repelled on its formulation; using both the same base network. While the parasiamese network will be suitable to learn antitransitive relations (or anti-Euclidean in case of nonsymmetry), its siamese counterpart will be useful to learn the similitude-like relation that emerges from the former. The training is performed through the minimization of the following loss function:
∑ (x,y)∈P Φ (p) α,β(x, y) + ∑ (x′,y′)∈N Φ (s) α,β(x ′, y′),
where P and N are antonymy and synonymy instances, respectively. Notice that contrast details and thresholds (if some) are delegated to each Φ function definition.
We propose two formulations inspired and aligned to two of the main approaches to deep metric learning: pair and triplet based. Essentially, both formulations consider the same information, the outputs of the parasiamese and siamese networks, but the repelling is driven in slightly different ways. Given the branches α and β and the input (x, y), the pair based approach minimizes (α(x), β(y)) maximizing (α(x), α(y)), while the triplet based approach considers the triplet (α(x), β(y), α(y)) attempting to get (α(x), β(y)) closer than (α(x), α(y)).

heading: 3.1.1 PAIR BASED
text: This approach considers positive and negative pairs, pretending the distance between related pairs to be lower than a given threshold µp and the distance between unrelated pairs to be higher than a threshold µn, by terms of hinge expression. The pair based Φ functions are written as:
Φ (p) αθα ,βθβ (x1, x2) = [Φαθα ,βθβ (x1, x2)− µp]+ + [µn − Φαθα ,αθα (x1, x2)]+
Φ (s) αθα ,βθβ (x1, x2) = [Φαθα ,αθα (x1, x2)− µp]+ + [µn − Φαθα ,βθβ (x1, x2)]+;
where µp and µn are the positive and negative margins, respectively. A disadvantage of this formulation is that the same margin is applied to every pair. This is addressed, theoretically, by means of the triplet loss function (Musgrave et al., 2020) that we consider in the following section.

heading: 3.1.2 TRIPLET BASED
text: The triplet loss is based on the triplet network concept (Hoffer & Ailon, 2015). Given a set T of triplets of elements (a, p, n) where p is related to a and n unrelated to a, the triplet loss function
attempts to make the distance between a and p smaller than the distance between a and n by a margin µ, through minimizing:
∑ (a,p,n)∈T [||a− p||2 − ||a− n||2 + µ]+
For the triplet based Φ(p)α,β and Φ (s) α,β functions, given a pair (x, y), we consider the triplets: (α(x), β(y), α(y)) and (α(x), α(y), β(y)), respectively. Hence, the triplet based Φ functions are written as:
Φ (p) α,β(x, y) = [Φα,β(x, y)− Φα,α(x, y) + µt]+
Φ (s) α,β(x, y) = [Φα,α(x, y)− Φα,β(x, y) + µt]+
where µt is the separation between the positive and negative samples.

heading: 3.2 PARASIAMESE BRANCHING
text: The repelling between the outputs of the parasiamese and the siamese networks does not rely on any particular way of the right branch of the parasiamese network. The base network double application is just an alternative that allows to share the weights between both branches and it is inspired in the logic negation, but only is needed to have both branches distinguished. In the following we detail the two variants that we consider in this work.
• (Standard) Parasiamese: Corresponds to the original formulation of the parasiamese network, where αθ(x) = Fθ(x) and βθ(x) = Fθ(Fθ(x)).
• Half-twin Parasiamese: This formulation completely unties the weights between both branches. Both branches consist on entirely different networks, without shared weights. Accordingly, αθα(x) = Fθα(x) and βθβ (x) = Gθβ (x).
Notice that both definitions are suitable for the previously introduced repulsion formulations. In the case of the half-twin parasiamese case, the siamese network (for repelling) corresponds to the left branch.

heading: 3.3 SYMMETRIC PARASIAMESE NETWORK
text: The parasiamese network is not symmetric. Etcheverry & Wonsever (2019) included symmetry driven by data, by augmenting the dataset with the reversed pair of each instance, showing that the model performance is improved when symmetryzed pairs are seen during training.
In this work, we propose to enforce symmetry through the architecture definition. We refer as right (left) parasiamese to the formulation with the right (left) branch distinguished (by double application or different base network in the half-twin parasiamese case). The parasiamese network can be formulated as a symmetric function if the two versions of the parasiamese network (left and right) are combined to be minimized. We experiment through adding them. In section 4.3, we compare the performance of the symmetric and non-symmetric variants showing in most of cases a better performance for the symmetric in the antonymy detection task.

heading: 3.4 PARASIAMESE AND SIAMESE SUB-NETWORKS
text: The here presented repelling parasiamese network has been designed to discern opposite from similar elements. However, it is possible to afford unrelated pairs (i.e. neither similar nor opposite terms) with it, when both sub-networks, the parasiamese and siamese outputs, present high values, suggesting that the elements of the candidate pair are neither similar nor opposite.
The table 2 shows the antonymy-synonymy classification using the repelling parasiamese network according to the parasiamese and siamese sub-networks outputs. Besides antonymy and synonymy regions, the region denoted by (1) refers to pairs that are simultaneously synonymic and antonymic or a self-antonymic term (e.g. rent). And the region (2) belongs to pairs that are neither synonyms nor antonyms.
As a future work we comment that a 3-way loss could be defined considering opposition, similarity and unrelatedness. A challenge on this direction is the unrelated pairs mining strategy, which contains a large space of possibilities.

heading: 4 EXPERIMENTS AND DISCUSSION
text: In the following we detail the settings and results of the experiments we conducted, in the antonymysynonymy discrimination task using general purpose pre-trained word embeddings as input.

heading: 4.1 DATASETS AND WORD EMBEDDINGS
text: To perform our experiments we need a number of words pairs, labelled with if they are synonyms or antonyms. The datasets we consider are split into train, validation and test. We consider the following datasets:
• Nguyen’s: This dataset was built by Nguyen et al. (2016) using WordNet (Miller, 1995) and Wordnik3. It consists of 15, 632 pairs of words with a balanced amount of synonyms and antonyms, over a vocabulary of 9, 405 words.
• Fallows’s: We introduced a dataset for synonym/antonym distinction from the book "Complete Dictionary of Synonyms and Antonyms", by Samuel Fallows; available through the Gutenberg project4. We automatically processed the electronic version of the book, obtaining a number of 25, 419 antonym and 32, 302 synonym pairs, with a vocabulary of 15, 698 distinct words with 5, 810 in common to Nguyen’s dataset.
• Xie’s: This dataset was built by Xie & Zeng (2021) using Nguyen et al. (2016)’s dataset but split (into train,validation and test) without any word in common between each part (i.e. without lexical intersection). It consists of 12, 732 pairs of words with a balanced amount of synonyms and antonyms, over a vocabulary of 9, 404 words.
We perform all our experiments using the word embeddings from the pretrained fastText (Joulin et al., 2016) model for English available in the fastText site5. This model was trained using Wikipedia6 and Common Crawl7. The resulting vectors are in dimension 300 and there are not any out of vocabulary word over any dataset since fastText considers subword information.

heading: 4.2 EVALUATION PROCEDURE
text: On recent works, deep metric learning advances have been criticized for its evaluation methodology (Musgrave et al., 2020; Fehervari et al., 2019). It has been detected unsuitable hyperparameter settings, leading to unfair comparisons. In order to avoid that, we perform an independent random search over each model to obtain a suitable hyperparameter configuration against the validation set. Then, we report the results of the best performing configuration against the test set.
3https://www.wordnik.com/ 4https://www.gutenberg.org/ebooks/51155 5https://fasttext.cc/docs/en/crawl-vectors.html 6https://www.wikipedia.org/ 7https://commoncrawl.org/
The hyperparameter space is considerably complex in these models. We divide it on three classes of hyperparameters: parasiamese, base network and training. The parasiamese hyperparameters corresponds to each margin values (e.g. positive, negative and acceptance). The branching type (vanilla, half-twin), model symmetry and the repelling type (pair or triplet) are considered as different models instead of hyperparameters; to compare results between them. Regarding the base network, in our experiments the considered base networks are feed forward networks. The base network hyperparameters corresponds to number of layers, each layer size and activation function. Finally, the training hyperparameters include the optimization algorithm, learning rate and batch size. We include the random search details in Appendix A. As evaluation metrics we use precision, recall and F1 scores.

heading: 4.3 RESULTS
text: We present the obtained results on antonym-synonym distinction task in Table 3. We compare the repel-parasiamese network to its predecessor, the (non-repelling) parasiamese network (Etcheverry & Wonsever, 2019), the Distiller (Ali et al., 2019), and MoE-ASD8 (Xie & Zeng, 2021). We consider half-twin variants for the repelling and non-repelling networks; and we include the best result obtained using a siamese network for comparison purposes.
8We do not consider dLCE vectors (Nguyen et al., 2016), since those are already specialized to antonyms and synonyms.
It can be observed that the repelling parasiamese network consistently outperforms its non-repelling predecessor. Regarding symmetry, the symmetric variants obtains the best results; and the halftwin branching improves the results for the symmetric formulations, while it degrades for the nonsymmetric variants.
In comparison to the other reported methods, the repelling parasiamese network achieved better results at least in two of the three sub-sets of Nguyen’s dataset (original and lexically splitted). Compared to MoE-ASD it is worth mentioning that the repelling parasiamese network reach competitive results without explicitly considering that the antonymic salient dimensions in the semantic space may vary for different antonymic pairs.

heading: 4.3.1 ANTITRANSITIVE LINKS
text: Since the main design consideration of the model is concerning antitransitivity, in Table 4 we show the model sub-network outputs (i.e. parasiamese and siamese) on antitransitive triplets (i.e. two words having an antonym in common).
We expose parasiamese and siamese outputs for pairs taken from the validation dataset partition of Fallow’s dataset. Each block in the table contains three pairs that form triangles where antitransitive property should be satisfied. For example, in the triplet (real, aerial,notional) the parasiamese output represents the antitransitivity (see psiam column) and the siamese output is not below the acceptance margin in any case, which is correct according to the dataset, since none of them are marked as synonyms. In the triplet (valid, bad,false), the antitransitivity is stated by the parasiamese output and the siamese output correctly indicates similarity between bad and false. The triplet (bottom, lateral,top) may be debatable. The model inferred it as antitransitive as in the dataset, and similarity between lateral and top is slightly high which we take as a mistake. Lastly, for the triplet (realistic, fantastic, utopian), the antitransitivity is also represented (rightly according to the dataset) and any of the pairs is stated as similar by the siamese network.

heading: 5 CONCLUSION
text: In this work we deepen the parasiamese network and we introduce the repelling parasiamese network. We show that it is beneficial, in the parasiamese formulation, to repel the siamese counterpart using the same base network; to distinguish antonyms and synonyms. The model achieves better results than its predecessor, using the weights of one (or two in the case of half-twin formulations) few-layered fully connected feed forward network. We perform our experiments in the Nguyen et al. (2016) publicly available dataset, the lexically split version provided by Xie & Zeng (2021), and we introduced a new dataset built from Samuel Fallow’s antonym’s dictionary accessed through the Gutenberg project. We show that the model performs well, encoding meaningful information in terms of opposition and similarity.

heading: A RANDOM SEARCH DETAILS
text: This appendix includes random search set up and the best hyperparameter configuration obtained for each model. Each run was early stopped with a patience chosen between 3 or 5. Every model was trained using adam (Kingma & Ba, 2014) and a learning rate chosen from [0.0001, 0.001, 0.01]. The batch size is 64 for every case. We perform 100 trials to search for the hyperparameters of each model.
Regarding the margins, they were selected using the following criteria:
• Positive margin is uniformly chosen from [0, 0.2, 0.5, 1.0, 1.25, 1.5, 2.0]. • Negative margin is uniformly chosen from the values of
[.5, 1., 1.5, 2., 2.5, 3., 3.5, 4., 4.5, 5.] that are greater than the previously chosen positive margin.
• Acceptance margin is the result of multiplying the negative margin with a factor uniformly chosen from [.75, 1., 1.25, 1.75].
In Table 5 we detail the parameters obtained from the random search.

Review summaries:
The paper presents the new approach for distinguishing antonyms and synonyms. This task could be interesting for further applications such as paraphrasing and text entailment. The authors present proposed the repelling parasiamese neural network. The model with several setups was tested on three datasets of antonyms and synonyms pairs showing the improvement over the baselines. 

This paper proposes a simple new architecture for the task of antonymy detection. The architecture is an extension of a previous architecture for the same task, known as the parasiamese network. The new architecture, called the repelling parasiamese network, essentially contains a copy of a siamese network and a parasiamese network tied together using a euclidean loss. Experimental results on several datasets demonstrate that this architecture can detect antonymy reliably and sometimes even beat the state of the art. 

The task is a binary classification task.  The system is given a pair of words that are either synonyms or antonyms.  The task is to say which pairs are synonyms and which are antonyms.  Table 3 reports results on two datasets: (1) Nguyen's and (2) Fallows's.   The second dataset is a contribution.   This paper points out the usefulness of https://www.gutenberg.org/ebooks/51155  The paper proposes a new model, REPELLING PARASIAMESE NEURAL NETWORK.  Results in table 3 are impressive, especially on Fallows's  The suggestion of using triplets is nice.