Abstract: Attention is a very efficient way to model the relationship between two sequences by comparing how similar two intermediate representations are. Initially demonstrated in NMT, it is a standard in all NLU tasks today when efficient interaction between sequences is considered. However, we show that attention, by virtue of its composition, works best only when it is given that there is a match somewhere between two sequences. It does not very well adapt to cases when there is no similarity between two sequences or if the relationship is contrastive. We propose an Conflict model which is very similar to how attention works but which emphasizes mostly on how well two sequences repel each other and finally empirically show how this method in conjunction with attention can boost the overall performance.

Introduction: Modelling the relationship between sequences is extremely significant in most retrieval or classification problems involving two sequences. Traditionally, in Siamese networks, Hadamard product or concatenation have been used to fuse two vector representations of two input sequences to form a final representation for tasks like semantic similarity, passage retrieval. This representation, subsequently, has been used to compute similarity scores which has been used in a variety of training objectives like margin loss for ranking or cross-entropy error in classification. We have also witnessed word or phrase level similarity to create alignment matrices between two sequences BIBREF0 , BIBREF1 . These alignment matrices has proved to be very useful to model the relationship between two word representations as well fuse the relevant information of one sequence into another. Empirical evidences have shown this alignment procedures have significantly performed better then simple concatenation or element-wise multiplication, especially for long sentences or paragraphs. Attention works on creating neural alignment matrix using learnt weights without pre-computing alignment matrix and using them as features. The main objective of any attentive or alignment process is to look for matching words or phrases between two sequences and assign a high weight to the most similar pairs and vice-versa. The notion of matching or similarity maybe not semantic similarity but based on whatever task we have at hand. For example, for a task that requires capturing semantic similarity between two sequences like "how rich is tom cruise" and "how much wealth does tom cruise have", an attentive model shall discover the high similarity between "rich" and "wealthy" and assign a high weight value to the pair. Likewise, for a different task like question answering, a word "long" in a question like "how long does it take to recover from a mild fever" might be aligned with the phrase "a week" from the candidate answer "it takes almost a week to recover fully from a fever". Thus, attention significantly aids in better understanding the relevance of a similar user query in a similar measurement task or a candidate answer in a question answering task. The final prediction score is dependent on how well the relationship between two sequences are modeled and established. The general process of matching one sequence with another through attention includes computing the alignment matrix containing weight value between every pair of word representations belonging to both of the sequences. Subsequently, softmax function is applied on all the elements of one of the two dimensions of the matrix to represent the matching probabilities of all the word of a sequence with respect to one particular word in the other sequence. Since attention always looks for matching word representations, it operates under the assumption that there is always a match to be found inside the sequences. We provide a theoretical limitation to it and propose another technique called conflict that looks for contrasting relationship between words in two sequences. We empirically verify that our proposed conflict mechanism combined with attention can outperform the performance of attention working solely.

Question: Which neural architecture do they use as a base for their attention conflict mechanisms?
Answer: GRU-based encoder, interaction block, and classifier consisting of stacked fully-connected layers.