Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors

1 INTRODUCTION . Natural languages are able to encode sentences with similar meanings using very different vocabulary and grammatical constructs , which makes determining the semantic similarity between pieces of text a challenge . It is common to cast semantic similarity between sentences as the proximity of their vector representations . More than half a century since it was first proposed , the Bag-of-Words ( BoW ) representation ( Harris , 1954 ; Salton et al. , 1975 ; Manning et al. , 2008 ) remains a popular baseline across machine learning ( ML ) , natural language processing ( NLP ) , and information retrieval ( IR ) communities . In recent years , however , BoW was largely eclipsed by representations learned through neural networks , ranging from shallow ( Le & Mikolov , 2014 ; Hill et al. , 2016 ) to recurrent ( Kiros et al. , 2015 ; Conneau et al. , 2017 ; Subramanian et al. , 2018a ) , recursive ( Socher et al. , 2013 ; Tai et al. , 2015 ) , convolutional ( Kalchbrenner et al. , 2014 ; Kim , 2014 ) , self-attentive ( Vaswani et al. , 2017 ; Cer et al. , 2018a ) and hybrid architectures ( Gan et al. , 2017 ; Tang et al. , 2017 ; Zhelezniak et al. , 2018 ) . Interestingly , Arora et al . ( 2017 ) showed that averaged word vectors ( Mikolov et al. , 2013a ; Pennington et al. , 2014 ; Bojanowski et al. , 2016 ; Joulin et al. , 2017 ) weighted with the Smooth Inverse Frequency ( SIF ) scheme and followed by a Principal Component Analysis ( PCA ) post-processing procedure were a formidable baseline for Semantic Textual Similarity ( STS ) tasks , outperforming deep representations . Furthermore , Wieting et al . ( 2015 ; 2016 ) and Wieting & Gimpel ( 2018 ) showed that averaged word vectors trained supervised on large corpora of paraphrases achieve state-of-the-art results , outperforming even the supervised systems trained directly on STS . Inspired by these insights , we push the boundaries of word vectors even further . We propose a novel fuzzy bag-of-words ( FBoW ) representation for text . Unlike classical BoW , fuzzy BoW contains all the words in the vocabulary simultaneously but with different degrees of membership , which are derived from similarities between word vectors . Next , we show that max-pooled word vectors are a special case of fuzzy BoW . Max-pooling significantly outperforms averaging on standard benchmarks when word vectors are trained unsupervised . Since max-pooled vectors are just a special case of fuzzy BoW , we show that the fuzzy Jaccard index is a more suitable alternative to cosine similarity for comparing these representations . By contrast , the fuzzy Jaccard index completely fails for averaged word vectors as there is no connection between the two . The max-pooling operation is commonplace throughout NLP and has been successfully used to extract features in supervised systems ( Collobert et al. , 2011 ; Kim , 2014 ; Kenter & de Rijke , 2015 ; De Boom et al. , 2016 ; Conneau et al. , 2017 ; Dubois , 2017 ; Shen et al. , 2018 ) ; however , to the best of our knowledge , the present work is the first to study max-pooling of pre-trained word embeddings in isolation and to suggest theoretical underpinnings behind this operation . Finally , we propose DynaMax , a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair . DynaMax outperforms averaged word vector with cosine similarity on every benchmark STS task when word vectors are trained unsupervised . It even performs comparably to Wieting & Gimpel ( 2018 ) ’ s vectors under cosine similarity , which is a striking result as the latter are in fact trained supervised to directly optimise cosine similarity between paraphrases , while our approach is completely unrelated to that objective . We believe this makes DynaMax a strong baseline that future algorithms should aim to beat in order to justify more complicated approaches to semantic similarity . As an additional contribution , we conduct significance analysis of our results . We found that recent literature on STS tends to apply unspecified or inappropriate parametric tests , or leave out significance analysis altogether in the majority of cases . By contrast , we rely on nonparametric approaches with much milder assumptions on the test statistic ; specifically , we construct bias-corrected and accelerated ( BCa ) bootstrap confidence intervals ( Efron , 1987 ) for the delta in performance between two systems . We are not aware of any prior works that apply such methodology to STS benchmarks and hope the community finds our analysis to be a good starting point for conducting thorough significance testing on these types of experiments . 2 SENTENCES AS FUZZY SETS . The bag-of-words ( BoW ) model of representing text remains a popular baseline across ML , NLP , and IR communities . BoW , in fact , is an extension of a simpler set-of-words ( SoW ) model . SoW treats sentences as sets , whereas BoW treats them as multisets ( bags ) and so additionally captures how many times a word occurs in a sentence . Just like with any set , we can immediately compare SoW or BoW using set similarity measures ( SSMs ) , such as Jaccard ( A , B ) = |A ∩B| |A ∪B| , Otsuka ( A , B ) = |A ∩B|√ |A| × |B| , and Dice ( A , B ) = 2|A ∩B| |A|+ |B| . These coefficients usually follow the pattern # { shared elements } # { total elements } . From this definition , it is clear that sets with no shared elements have a similarity of 0 , which is undesirable in NLP as sentences with completely different words can still share the same meaning . But can we do better ? For concreteness , let ’ s say we want to compare two sentences corresponding to the sets A = { ‘ he ’ , ‘ has ’ , ‘ a ’ , ‘ cat ’ } and B = { ‘ she ’ , ‘ had ’ , ‘ one ’ , ‘ dog ’ } . The situation here is that A ∩ B = ∅ and so their similarity according to any SSM is 0 . Yet , both A and B describe pet ownership and should be at least somewhat similar . If a set contains the word ‘ cat ’ , it should also contain a bit of ‘ pet ’ , a bit of ‘ animal ’ , also a little bit of ‘ tiger ’ but perhaps not too much of an ‘ airplane ’ . If both A and B contained ‘ pet ’ , ‘ animal ’ , etc . to some degree , they would have a non-zero similarity . This intuition is the main idea behind fuzzy sets : a fuzzy set includes all words in the vocabulary simultaneously , just with different degrees of membership . This generalises classical sets where a word either belongs to a set or it doesn ’ t . We can easily convert a singleton set such as { ‘ cat ’ } into a fuzzy set using a similarity function sim ( wi , wj ) between words . We simply compute the similarities between ‘ cat ’ and all the words wj in the vocabulary and treat those values as membership degrees . As an example , the set { ‘ cat ’ } really becomes { ‘ cat ’ : 1 , ‘ pet ’ : 0.9 , ‘ animal ’ : 0.85 , . . . , ‘ airplane ’ : 0.05 , . . . } Fuzzifying singleton sets is straightforward , but how do we go about fuzzifying the entire sentence { ‘ he ’ , ‘ has ’ , ‘ a ’ , ‘ cat ’ } ? Just as we use the classical union operation ∪ to build bigger sets from smaller ones , we use the fuzzy union to do the same but for fuzzy sets . The membership degree of a word in the fuzzy union is determined as the maximum membership degree of that word among each of the fuzzy sets we want to unite . This might sound somewhat arbitrary : after all , why max and not , say , sum or average ? We explain the rationale in Section 2.1 ; and in fact , we use the max for the classical union all the time without ever noticing it . Indeed , { ‘ cat ’ } ∪ { ‘ cat ’ } = { ‘ cat ’ } and not { ‘ cat ’ : 2 } . This is simply because we computed max ( 1 , 1 ) = 1 and not sum ( 1 , 1 ) = 2 . Similarly { ‘ cat ’ } ∪∅ = { ‘ cat ’ } since max ( 1 , 0 ) = 1 and not avg ( 1 , 0 ) = 1/2 . The key insight here is the following . An object that assigns the degrees of membership to words in a fuzzy set is called the membership function . Each word defines a membership function , and even though ‘ cat ’ and ‘ dog ’ are different , they are semantically similar ( in terms of cosine similarity between their word vectors , for example ) and as such give rise to very similar membership functions . This functional proximity will propagate into the SSMs , thus rendering them a much more realistic model for capturing semantic similarity between sentences . To actually compute the fuzzy SSMs , we need just a few basic tools from fuzzy set theory , all of which we briefly cover in the next section . 2.1 FUZZY SETS : THE BARE MINIMUM . Fuzzy set theory ( Zadeh , 1996 ) is a well-established formalism that extends classical set theory by incorporating the idea that elements can have degrees of membership in a set . Constrained by space , we define the bare minimum needed to compute the fuzzy set similarity measures and refer the reader to Klir et al . ( 1997 ) for a much richer introduction . Definition : A set of all possible terms V = { w1 , w2 , . . . , wN } that occur in a certain domain is called a universe . Definition : A function µ : V→ L ⊆ R is called a membership function . Definition : A pair A = ( V , µ ) is called a fuzzy set . Notice how the above definition covers all the set-like objects we discussed so far . If L = { 0 , 1 } , then A is simply a classical set and µ is its indicator ( characteristic ) function . If L = N≥0 ( non-negative integers ) , then A is a multiset ( a bag ) and µ is called a count ( multiplicity ) function . In literature , A is called a fuzzy set when L = [ 0 , 1 ] . However , we make no restrictions on the range and call A a fuzzy set even when L = R , i.e . all real numbers . Definition : Let A = ( V , µ ) and B = ( V , ν ) be two fuzzy sets . The union of A and B is a fuzzy set A ∪B = ( V , max ( µ , ν ) ) . The intersection of A and B is a fuzzy set A ∩B = ( V , min ( µ , ν ) ) . Interestingly , there are many other choices for the union and intersection operations in fuzzy set theory . However , only the max-min pair makes these operations idempotent , i.e . such thatA∪A = A and A ∩ A = A , just as in the classical set theory . By contrast , it is not hard to verify that neither sum nor average satisfy the necessary axioms to qualify as a fuzzy union or intersection . Definition : Let A = ( V , µ ) be a fuzzy set . The number |A| = ∑ w∈V µ ( w ) is called the cardinality of a fuzzy set . Fuzzy set theory provides a powerful framework for reasoning about sets with uncertainty , but the specification of membership functions depends heavily on the domain . In practice these can be designed by experts or learned from data ; below we describe a way of generating membership functions for text from word embeddings .

Summary: This submission presents a simple model for sentence representation based on max-pooling of word vectors. The model is motivated by fuzzy-set theory, providing both a funded pooling scheme and a similarity score between documents. The proposed approach is evaluated on sentence similarity tasks (STS) and achieves very strong performance, comparable to state-of-the-art, computationally demanding methods.