Antonymy-Synonymy Discrimination through the Repelling Parasiamese Neural Network

Antonymic and synonymic pairs may both occur nearby in word embeddings spaces because they have similar distributional information . Different methods have been used in order to distinguish them , making the antonymy-synonymy discrimination a popular NLP task . In this work , we propose the repelling parasiamese neural network , a model which considers a siamese network for synonymy and a parasiamese network for antonymy , both sharing the same base network . Relying in the antagonism between synonymy and antonymy , the model attempts to repel siamese and parasiamese outputs making use of the contrastive loss functions . We experimentally show that the repelling parasiamese network achieves state-of-the-art results on this task . 1 INTRODUCTION . Semantic opposition is a binary relation of central importance in the cognitive baggage of human languages . It establishes that one term contradicts the other , that both can not be satisfied simultaneously . In the context of lexical semantics , it corresponds to antonyms ( e.g . light and dark ) , whose recognition is essential for natural language usage . For instance , this capability is crucial for text entailment and paraphrasing , which are basic abilities for different NLP tasks . Most of modern NLP is using word embeddings ( i.e . vectors for word meanings built from word contexts and subword information ) . These word representations have the potential to cluster words according to their distributional information on a corpus . However , since antonyms tend to occur in similar contexts , word embeddings may have close vectors in the space . Faced to this problem , different approaches have been proposed to re-encode the word embeddings in a supervised learning setup for the antonymy-synonymy discrimination task ( Mrkšić et al. , 2017 ; Etcheverry & Wonsever , 2019 ; Samenko et al. , 2020 ; Xie & Zeng , 2021 ) . In this work , we deepen in the parasiamese network as an antitransitive relationship learning approach , and we propose the repelling parasiamese neural network : a model that simultaneously opposes the siamese and parasiamese outputs ( of a same base network ) . We present two independent alternatives to do so : ( 1 ) pair and ( 2 ) triplet based approaches . We experimentally evaluate different alternatives and we introduce a formulation to enforce symmetry through the network structure . We carry out our experiments in three datasets : the publicly available antonymy-synonymy dataset introduced by Nguyen et al . ( 2016 ) , a here introduced dataset confeccionated from Samuel Fallow ’ s antonym ’ s dictionary ( accessed through the Gutenberg project ) and in a version of the Nguyen et al . ( 2016 ) ’ s dataset splitted without lexical intersection between train , validation and test introduced by Xie & Zeng ( 2021 ) . We show that the repelling parasiamese neural network achieves better performance than its predecessor , the ( non-repelling ) parasiamese network , and the best performing models found in the literature . 2 SOME PRELIMINARIES . Before getting into the repelling parasiamese neural network , let ’ s introduce some preliminary concepts concerning antitransitivity , metric learning for antonymy and the parasiamese network . 2.1 ANTONYMY AND ANTITRANSITIVITY . Antonymy can be considered as an antitransitive relationship1 . If two lexical units are antonyms of a third ( e.g . huge and enormous being opposite of small ) then they will not oppose each other ; in fact , they will often present semantic similarity ( Edmundson , 1967 ) . In table 1 we sample antonyms for some words from Fallow ’ s dictionary . Supporting the claimed antonymy antitransitivity , it can be seen that the words on each antonymy list ( i.e . common antonyms of a word ) do not oppose between them , and many cases of similarity can be detected ( e.g . savage and wild ) . 2.2 A METRIC FOR ANTONYMS . Siamese networks are among the best performing approaches for text semantic similarity tasks ( Tran et al. , 2020 ; Ranasinghe et al. , 2019 ; Mueller & Thyagarajan , 2016 ) . The properties that similarity relations tend to have , such as reflexivity , symmetry and transitivity ; are suitable for metrics and particularly for siamese networks . To clarify , suppose a metric d , reflexivity and symmetry arise directly from the metric definition , precisely from d ( x , x ) = 0 and d ( x , y ) = d ( y , x ) . Concerning transitivity , it is related to triangular inequity . The triangular inequity establishes that for any triplet ( x , y , z ) of words : d ( a , c ) ≤ d ( a , b ) + d ( b , c ) ( 1 ) So , given two pairs of related words ( a , b ) and ( b , c ) , then due to the triangular inequity , d ( a , c ) is bounded by the sum of d ( a , b ) and d ( b , c ) , which are expected to be low values since ( a , b ) and ( b , c ) are related . This makes the pair ( a , b ) to tend to be related as well , and therefore , the relation transitivity . If instead of a metric for similarity we consider a metric for opposition , e.g . antonyms , the aforementioned regarding triangular inequity is a drawback . The metric function will not be suitable for the antitransitivity of the opposition relation , tending to return low values ( i.e . treat them as related ) for the unrelated pair of words in the anti-transitive triangles . In other words , for each pair of words with a common antonym ( e.g . short and brief as antonyms of long ) , the metric will tend to wrongly treat them as antonyms as well . In conclusion , the triangular inequity is beneficial for transitivity but it is problematic for antitransitivity . In the following section we describe the parasiamese network , a siamese-like neural network that does not satisfy triangular inequity and is suitable for learning antitransitive relations . 2.3 THE PARASIAMESE NETWORK . The parasiamese network ( Etcheverry & Wonsever , 2019 ) was introduced as inspired by the siamese network , being better suited for the learning of antitransitive relations . Just like the siamese network , it consists of a model that consumes two vectors and returns a non-negative value ; and it relies on a base neural network that is applied more than once , sharing its weights , to compute the output . The parasiamese network differs from the siamese formulation in the fact that the base network is applied once to one input and twice to the other , instead of once to each input , as in the siamese network . The double application of the base network in the parasiamese network , imposes that the base network must have the same dimension for its input and output . The output of the parasiamese network is the distance between both branches ( see Figure 1 ) . 1We remind that a binary relation R is called antitransitive iff ∀a , b , c ( a R b ∧ b R c→ a 6R c ) Let Fθ : Rn → Rn be a neural network with trainable parameters θ . Then , the parasiamese network with base network Fθ is defined by ΦFθ ( x , y ) = ||Fθ ( x ) − Fθ ( Fθ ( y ) ) ||2 , ( 2 ) where ||.||2 is the Euclidean norm . The model is trained through the contrastive loss function . Concretely , ΦFθ is trained through mini-batch stochastic gradient descent on : L = ∑ ( x , y ) ∈P [ ΦFθ ( x , y ) − µp ] + + ∑ ( x′ , y′ ) ∈N [ µn − ΦFθ ( x′ , y′ ) ] + , ( 3 ) where P and N are , respectively , the positive and negative pairs in the dataset ; and µp and µn are the positive and negative thresholds , respectively . The [ . ] + notation corresponds to the ReLU function . The training attempts to pull closer than µp the related elements and push away unrelated pairs further than µn . This definition , unlike the siamese network , does not enforce transitivity even when the parasiamese output of the two related pairs in the antitransitive triangle are strictly zero . Moreover , the relation given by ΦFθ and a threshold µ ( i.e . RΦFθ , µ = { ( a , b ) : ΦFθ ( a , b ) ≤ µ } ) is benefited concerning antitransitivity if ΦFθ ( w , w ) > µ , which is consistent with the fact that antitransitive relations are necessarily antireflexive2 . In addition , the unrelated pair in the anti-transitive triangles will present a low value for the siamese formulation using the same base network . If ( a , b , c ) is an antitransitive triangle with unrelated pair ( a , c ) , then : ||Fθ ( a ) − Fθ ( c ) ||2 = ||Fθ ( a ) − Fθ ( c ) + Fθ ( Fθ ( b ) ) − Fθ ( Fθ ( b ) ) ||2 ≤ ||Fθ ( a ) − Fθ ( Fθ ( b ) ) ||2 + ||Fθ ( c ) − Fθ ( Fθ ( b ) ) ||2 and the parasiamese formulations of ( a , b ) and ( c , b ) are expected to output low values since they are both related . A possible interpretation for the parasiamese definition is thinking the base network F as an opposition transformation . So , if we consider two opposite terms a and b ( i.e . a ∼ ¬b ) , it is expected that opposition remains when both terms are negated ( i.e . ¬a ∼ ¬¬b ) , which brings the parasiamese formulation : F ( a ) ∼ F ( F ( b ) ) . 2To support this claim we rely in ΦFθ ( a , c ) ≤ ΦFθ ( a , b ) + ΦFθ ( b , b ) + ΦFθ ( b , c ) . 3 THE REPELLING PARASIAMESE NETWORK . The repelling parasiamese network is based on contrasting the siamese and parasiamese networks in a differentiable formulation ( Figure 2 ) . By doing this , we consistently observe a performance improvement in comparison to its predecessor ( i.e . the parasiamese network without repelling its siamese counterpart ) . The siamese counterpart of a parasiamese network may reflect the similitude-like relation that emerges from being opposed to the same elements through the antitransitive relation . Recalling the case of lexical semantics , the siamese formulation that comes from a parasiamese network that models antonymy , may be suitable to model synonymy , since the words that share antonyms may tend to be synonyms . Hence , given the antagonism between antonymy and synonymy , it seems suitable that both outputs should not be simultaneously low for the same outputs . Let a , b , c be three inputs with the pairs a , b and b , c being related and therefore returning low parasiamese outputs . Then , as commented in the previous section , the output for the siamese formulation of the same base network for the pair a , c will present a low value . Simultaneously , its parasiamese output is expected to be greater than the acceptance threshold , because of the antitransitivity . This suggests contrariness between siamese and parasiamese formulations in the unrelated pairs of antitransitive triangles . Moreover , if a pair ( a , b ) presents low siamese and parasiamese outputs , it would implies a reflexive pair within the antitransitive relation , since : ΦFθ ( a , a ) = ||Fθ ( a ) − Fθ ( Fθ ( a ) ) ||2 = ||Fθ ( a ) − Fθ ( Fθ ( a ) ) + Fθ ( b ) − Fθ ( b ) ||2 ≤ ||Fθ ( a ) − Fθ ( b ) ||2 + ||Fθ ( a ) − Fθ ( Fθ ( b ) ) ||2 which is inconsistent with antitransitivity . So , it may be suitable to contrast the parasiamese and siamese outputs ( using a same base network ) when one of them returns a low value . To describe the repelling parasiamese network let introduce the following notation : • Parasiamese left and right branches : We will refer as left and right branches of the parasiamese network to the transformations applied to the left and right parts of the relationship ( that correspond to the left and right terms of the distance in equation 2 ) , and we will write them as αθα : Rn → Rn and βθβ : Rn → Rn , respectively . So , in the non-repelling proposal αθ ( x ) = Fθ ( x ) and βθ ( x ) = Fθ ( Fθ ( x ) ) . • Parasiamese output function : We will use the notation Φαθα , βθβ for to the binary function that given the left and right branches , αθα and βθβ , returns the distance between them , i.e . Φαθα , βθβ ( x1 , x2 ) = ||αθα ( x1 ) − βθβ ( x2 ) || 2 2 . Notice that Φαθα , αθα ( x1 , x2 ) corresponds to the siamese network formulation .

Summary: The paper presents the new approach for distinguishing antonyms and synonyms. This task could be interesting for further applications such as paraphrasing and text entailment. The authors present proposed the repelling parasiamese neural network. The model with several setups was tested on three datasets of antonyms and synonyms pairs showing the improvement over the baselines.