Antonymy-Synonymy Discrimination through the Repelling Parasiamese Neural Network

Antonymic and synonymic pairs may both occur nearby in word embeddings spaces because they have similar distributional information . Different methods have been used in order to distinguish them , making the antonymy-synonymy discrimination a popular NLP task . In this work , we propose the repelling parasiamese neural network , a model which considers a siamese network for synonymy and a parasiamese network for antonymy , both sharing the same base network . Relying in the antagonism between synonymy and antonymy , the model attempts to repel siamese and parasiamese outputs making use of the contrastive loss functions . We experimentally show that the repelling parasiamese network achieves state-of-the-art results on this task . 1 INTRODUCTION . Semantic opposition is a binary relation of central importance in the cognitive baggage of human languages . It establishes that one term contradicts the other , that both can not be satisfied simultaneously . In the context of lexical semantics , it corresponds to antonyms ( e.g . light and dark ) , whose recognition is essential for natural language usage . For instance , this capability is crucial for text entailment and paraphrasing , which are basic abilities for different NLP tasks . Most of modern NLP is using word embeddings ( i.e . vectors for word meanings built from word contexts and subword information ) . These word representations have the potential to cluster words according to their distributional information on a corpus . However , since antonyms tend to occur in similar contexts , word embeddings may have close vectors in the space . Faced to this problem , different approaches have been proposed to re-encode the word embeddings in a supervised learning setup for the antonymy-synonymy discrimination task ( Mrkšić et al. , 2017 ; Etcheverry & Wonsever , 2019 ; Samenko et al. , 2020 ; Xie & Zeng , 2021 ) . In this work , we deepen in the parasiamese network as an antitransitive relationship learning approach , and we propose the repelling parasiamese neural network : a model that simultaneously opposes the siamese and parasiamese outputs ( of a same base network ) . We present two independent alternatives to do so : ( 1 ) pair and ( 2 ) triplet based approaches . We experimentally evaluate different alternatives and we introduce a formulation to enforce symmetry through the network structure . We carry out our experiments in three datasets : the publicly available antonymy-synonymy dataset introduced by Nguyen et al . ( 2016 ) , a here introduced dataset confeccionated from Samuel Fallow ’ s antonym ’ s dictionary ( accessed through the Gutenberg project ) and in a version of the Nguyen et al . ( 2016 ) ’ s dataset splitted without lexical intersection between train , validation and test introduced by Xie & Zeng ( 2021 ) . We show that the repelling parasiamese neural network achieves better performance than its predecessor , the ( non-repelling ) parasiamese network , and the best performing models found in the literature . 2 SOME PRELIMINARIES . Before getting into the repelling parasiamese neural network , let ’ s introduce some preliminary concepts concerning antitransitivity , metric learning for antonymy and the parasiamese network . 2.1 ANTONYMY AND ANTITRANSITIVITY . Antonymy can be considered as an antitransitive relationship1 . If two lexical units are antonyms of a third ( e.g . huge and enormous being opposite of small ) then they will not oppose each other ; in fact , they will often present semantic similarity ( Edmundson , 1967 ) . In table 1 we sample antonyms for some words from Fallow ’ s dictionary . Supporting the claimed antonymy antitransitivity , it can be seen that the words on each antonymy list ( i.e . common antonyms of a word ) do not oppose between them , and many cases of similarity can be detected ( e.g . savage and wild ) . 2.2 A METRIC FOR ANTONYMS . Siamese networks are among the best performing approaches for text semantic similarity tasks ( Tran et al. , 2020 ; Ranasinghe et al. , 2019 ; Mueller & Thyagarajan , 2016 ) . The properties that similarity relations tend to have , such as reflexivity , symmetry and transitivity ; are suitable for metrics and particularly for siamese networks . To clarify , suppose a metric d , reflexivity and symmetry arise directly from the metric definition , precisely from d ( x , x ) = 0 and d ( x , y ) = d ( y , x ) . Concerning transitivity , it is related to triangular inequity . The triangular inequity establishes that for any triplet ( x , y , z ) of words : d ( a , c ) ≤ d ( a , b ) + d ( b , c ) ( 1 ) So , given two pairs of related words ( a , b ) and ( b , c ) , then due to the triangular inequity , d ( a , c ) is bounded by the sum of d ( a , b ) and d ( b , c ) , which are expected to be low values since ( a , b ) and ( b , c ) are related . This makes the pair ( a , b ) to tend to be related as well , and therefore , the relation transitivity . If instead of a metric for similarity we consider a metric for opposition , e.g . antonyms , the aforementioned regarding triangular inequity is a drawback . The metric function will not be suitable for the antitransitivity of the opposition relation , tending to return low values ( i.e . treat them as related ) for the unrelated pair of words in the anti-transitive triangles . In other words , for each pair of words with a common antonym ( e.g . short and brief as antonyms of long ) , the metric will tend to wrongly treat them as antonyms as well . In conclusion , the triangular inequity is beneficial for transitivity but it is problematic for antitransitivity . In the following section we describe the parasiamese network , a siamese-like neural network that does not satisfy triangular inequity and is suitable for learning antitransitive relations . 2.3 THE PARASIAMESE NETWORK . The parasiamese network ( Etcheverry & Wonsever , 2019 ) was introduced as inspired by the siamese network , being better suited for the learning of antitransitive relations . Just like the siamese network , it consists of a model that consumes two vectors and returns a non-negative value ; and it relies on a base neural network that is applied more than once , sharing its weights , to compute the output . The parasiamese network differs from the siamese formulation in the fact that the base network is applied once to one input and twice to the other , instead of once to each input , as in the siamese network . The double application of the base network in the parasiamese network , imposes that the base network must have the same dimension for its input and output . The output of the parasiamese network is the distance between both branches ( see Figure 1 ) . 1We remind that a binary relation R is called antitransitive iff ∀a , b , c ( a R b ∧ b R c→ a 6R c ) Let Fθ : Rn → Rn be a neural network with trainable parameters θ . Then , the parasiamese network with base network Fθ is defined by ΦFθ ( x , y ) = ||Fθ ( x ) − Fθ ( Fθ ( y ) ) ||2 , ( 2 ) where ||.||2 is the Euclidean norm . The model is trained through the contrastive loss function . Concretely , ΦFθ is trained through mini-batch stochastic gradient descent on : L = ∑ ( x , y ) ∈P [ ΦFθ ( x , y ) − µp ] + + ∑ ( x′ , y′ ) ∈N [ µn − ΦFθ ( x′ , y′ ) ] + , ( 3 ) where P and N are , respectively , the positive and negative pairs in the dataset ; and µp and µn are the positive and negative thresholds , respectively . The [ . ] + notation corresponds to the ReLU function . The training attempts to pull closer than µp the related elements and push away unrelated pairs further than µn . This definition , unlike the siamese network , does not enforce transitivity even when the parasiamese output of the two related pairs in the antitransitive triangle are strictly zero . Moreover , the relation given by ΦFθ and a threshold µ ( i.e . RΦFθ , µ = { ( a , b ) : ΦFθ ( a , b ) ≤ µ } ) is benefited concerning antitransitivity if ΦFθ ( w , w ) > µ , which is consistent with the fact that antitransitive relations are necessarily antireflexive2 . In addition , the unrelated pair in the anti-transitive triangles will present a low value for the siamese formulation using the same base network . If ( a , b , c ) is an antitransitive triangle with unrelated pair ( a , c ) , then : ||Fθ ( a ) − Fθ ( c ) ||2 = ||Fθ ( a ) − Fθ ( c ) + Fθ ( Fθ ( b ) ) − Fθ ( Fθ ( b ) ) ||2 ≤ ||Fθ ( a ) − Fθ ( Fθ ( b ) ) ||2 + ||Fθ ( c ) − Fθ ( Fθ ( b ) ) ||2 and the parasiamese formulations of ( a , b ) and ( c , b ) are expected to output low values since they are both related . A possible interpretation for the parasiamese definition is thinking the base network F as an opposition transformation . So , if we consider two opposite terms a and b ( i.e . a ∼ ¬b ) , it is expected that opposition remains when both terms are negated ( i.e . ¬a ∼ ¬¬b ) , which brings the parasiamese formulation : F ( a ) ∼ F ( F ( b ) ) . 2To support this claim we rely in ΦFθ ( a , c ) ≤ ΦFθ ( a , b ) + ΦFθ ( b , b ) + ΦFθ ( b , c ) . 3 THE REPELLING PARASIAMESE NETWORK . The repelling parasiamese network is based on contrasting the siamese and parasiamese networks in a differentiable formulation ( Figure 2 ) . By doing this , we consistently observe a performance improvement in comparison to its predecessor ( i.e . the parasiamese network without repelling its siamese counterpart ) . The siamese counterpart of a parasiamese network may reflect the similitude-like relation that emerges from being opposed to the same elements through the antitransitive relation . Recalling the case of lexical semantics , the siamese formulation that comes from a parasiamese network that models antonymy , may be suitable to model synonymy , since the words that share antonyms may tend to be synonyms . Hence , given the antagonism between antonymy and synonymy , it seems suitable that both outputs should not be simultaneously low for the same outputs . Let a , b , c be three inputs with the pairs a , b and b , c being related and therefore returning low parasiamese outputs . Then , as commented in the previous section , the output for the siamese formulation of the same base network for the pair a , c will present a low value . Simultaneously , its parasiamese output is expected to be greater than the acceptance threshold , because of the antitransitivity . This suggests contrariness between siamese and parasiamese formulations in the unrelated pairs of antitransitive triangles . Moreover , if a pair ( a , b ) presents low siamese and parasiamese outputs , it would implies a reflexive pair within the antitransitive relation , since : ΦFθ ( a , a ) = ||Fθ ( a ) − Fθ ( Fθ ( a ) ) ||2 = ||Fθ ( a ) − Fθ ( Fθ ( a ) ) + Fθ ( b ) − Fθ ( b ) ||2 ≤ ||Fθ ( a ) − Fθ ( b ) ||2 + ||Fθ ( a ) − Fθ ( Fθ ( b ) ) ||2 which is inconsistent with antitransitivity . So , it may be suitable to contrast the parasiamese and siamese outputs ( using a same base network ) when one of them returns a low value . To describe the repelling parasiamese network let introduce the following notation : • Parasiamese left and right branches : We will refer as left and right branches of the parasiamese network to the transformations applied to the left and right parts of the relationship ( that correspond to the left and right terms of the distance in equation 2 ) , and we will write them as αθα : Rn → Rn and βθβ : Rn → Rn , respectively . So , in the non-repelling proposal αθ ( x ) = Fθ ( x ) and βθ ( x ) = Fθ ( Fθ ( x ) ) . • Parasiamese output function : We will use the notation Φαθα , βθβ for to the binary function that given the left and right branches , αθα and βθβ , returns the distance between them , i.e . Φαθα , βθβ ( x1 , x2 ) = ||αθα ( x1 ) − βθβ ( x2 ) || 2 2 . Notice that Φαθα , αθα ( x1 , x2 ) corresponds to the siamese network formulation .

Summary: The task is a binary classification task.  The system is given a pair of words that are either synonyms or antonyms.  The task is to say which pairs are synonyms and which are antonyms.  Table 3 reports results on two datasets: (1) Nguyen's and (2) Fallows's.   The second dataset is a contribution.   This paper points out the usefulness of https://www.gutenberg.org/ebooks/51155  The paper proposes a new model, REPELLING PARASIAMESE NEURAL NETWORK.  Results in table 3 are impressive, especially on Fallows's  The suggestion of using triplets is nice.