Interactive Model with Structural Loss for Language-based Abductive Reasoning
Authors: Linhao Li
Ming Xu
Yongfeng Dong
Xin Li
Jianhua Tao
Qinghua Hu
Venue: ICLR 2022 Submitted (2022)
Keywords: abductive  natural  language
abductive reasoning
BiLSTM
joint loss function.

Abstract: The abductive natural language inference task ($\alpha$NLI) is proposed to infer the most plausible explanation between the cause and the event. In the $\alpha$NLI task, two observations are given, and the most plausible hypothesis is asked to pick out from the candidates. Existing methods model the relation between each candidate hypothesis separately and penalize the inference network uniformly. In this paper, we argue that it is unnecessary to distinguish the reasoning abilities among correct hypotheses; and similarly, all wrong hypotheses contribute the same when explaining the reasons of the observations. Therefore, we propose to group instead of ranking the hypotheses and design a structural loss called "joint softmax focal loss" in this paper. Based on the observation that the hypotheses are generally semantically related, we have designed a novel interactive language model aiming at exploiting the rich interaction among competing hypotheses. We name this new model for $\alpha$NLI: Interactive Model with Structural Loss (IMSL). The experimental results show that our IMSL has achieved the highest performance on the RoBERTa-large pretrained model, with ACC and AUC results increased by about 1% and  5% respectively.

Under review as a conference paper at ICLR 2022
INTERACTIVE MODEL WITH STRUCTURAL LOss FOR LANGUAGE-BASED ABDUCTIVE REASONING
Anonymous authors Paper under double-blind review
ABSTRACT
The abductive natural language inference task (aNLI) is proposed to infer the most plausible explanation between the cause and the event. In the oNLI task. two observations are given, and the most plausible hypothesis is asked to pick out from the candidates. Existing methods model the relation between each candidate hypothesis separately and penalize the inference network uniformly. In this pa- per, we argue that it is unnecessary to distinguish the reasoning abilities among correct hypotheses; and similarly, all wrong hypotheses contribute the same when explaining the reasons of the observations. Therefore, we propose to group in- stead of ranking the hypotheses and design structural loss called "joint softmax focal loss" in this paper. Based on the observation that the hypotheses are gen- erally semantically related, we have designed novel interactive language model aiming at exploiting the rich interaction among competing hypotheses. We name this new model for aNLI: Interactive Model with Structural Loss (IMSL). The experimental results show that our IMSL has achieved the highest performance on the RoBERTa-large pretrained model, with ACC and AUC results increased by about 1% and 5% respectively.
INTRODUCTION
Abductive natural language inference (aNLI) (Bhagavatula et al. (2020) is a newly established branch of natural language inference (NLI) and an interesting task the area of natural language processing (NLP) based commonsense reasoning. Originating from NI.I which targets at the seman- tie relationship between the two sentences, aNL.I further estimates the abductive reasoning of each sentence by explicitly deducing its cause. In the past years, oNLI has attracted increasing attentions as it makes NLP tools more explainable and comprehensible. As of today, typical applications of aNLI include knowledge graph Completion (Yu et al. (2020) (Bauer & Bansal (222)), ) question answering (Ma et al. (021)). sentence in in-filling (Huang et al. (2020)) knowledge integration (Zhou
et al. (2021) and so
To better motivate this work, we have shown a comparison between NLI and aNLI in Table I For NLJ, the task is to judge the relationship between the premise statement P and the hypothetical sentence based on the given information in P. Options of the answer can be implication, neutrality, or contradiction. For oNLI, pair of observations (O, and O2) and some hypotheses (c.g., two competing hypotheses H and H2 in the example) are given. The task of aNLI is to deduce the more plausible reason between H and H that can explain the situational change from O, Og- In addition to constructing the aNLI task, the authors of (Bhagavatula et al. (2020)) has released new challenge data set, called ART and reported comprehensive baseline performance for aNLI by directly employing and retraining solution for NLI, i.e., ESIM+EL et al. (2017). Peters) al. (2018). They also found that the pretrained language model can apparently influence the performance of an algorithm and demonstrated some test results with the latest language models like (aa(ood et al. (2018)) and (Eelin et al. (201)))
We note that there is still a considerable gap between the human performance and the class of base- line models in (Bhagavatula et al. (2020)). More recently, (Zhu et al, (2020) argued that the former framework cannot measure the rationality of the hypotheses, and reformulated oNLI learning- to-rank task for abductive reasoning. In their approach, RoBERTa et al. (2019)), BERT(Devlin al. 2019)), and ESIM al. (2017)) are all tested to work the pretrained language model.

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of NLI tasks and oNLI tasks, where E, N, and C represent entailment, neutral and contradiction, respectively Task
Context
Answer
P: A man inspects the uniform of figure in some East Asian country.
E, N.Nor C
H: The man is sleeping.
P: An older and younger man smiling.
NLI
N,Nore or
H: Two men are smiling and laughing at the cats playing on the floor.
P: A soccer game with multiple males playing-
H: Some men are playing sport.
Oj: Dotty was being very grumpy.
H Dotty ate something bad.
oNLI
H H
H²: Dotty call some close friends to chat.
O2: She felt much better afterwards.
introduces novel multi-head knowledge attention model which learns to focus on multiple pieces of knowledge at the same time, and is capable of refining the input representation in a recursive manner for oNLI
Under this new ranking- based framework, (Paul & Frank (2020)
Despite the performance improvement achieved by the ranking framework, there are still some weaknesses calling for further investigation. For instance, a practical training sample (e.g., two observations and four hypotheses) from ART is shown in Figure [ It is casy to conclude that both H' and H² are correct answers; while the other two (H3, H*) are false. However, in previous munking-based aNLI method such as L2R² (Learning to Rank for Reasoning) (Zhu et al., 2020, four hypotheses will be trained simultaneously by treating one of the two correct answers as a more cor- rect one. Similarly, the wrong answers also treated as wrong the ranked hypotheses are trained separately, but the sum of their probabilities is set as a fixed value e.g., the probability of correct hypothesis H decreases when the probability of answer H² increases.
and worse one. Meanwhile,
0 : Josh bought a parrot as a pet.
O2 Josh was so excited that he taught his parrot how to say it's name!
H' Josh realized that the parrot could talk!
H The parrot repeated Josh's 's moming greeting.
H Josh started teaching the parrot things.
H He is scared of birds.
S(H S(H)))S(H))=S(HT)) S(H))>S(**==S(HH)
S(#)))S((EF)-S(())>>{{)}})
L2R²
IMSL
Figure 1: Comparison of L2R2 method and IMSL method. Among them, O,, O2 represent obser- vations, H1 H² are correct answers, H3, H wrong answers, S(H) represents the of the i-th hypothesis correctness.
In this paper, advocate new approach for aNLI shown in Figure Our principle of abductive reasoning is constructed based on following two arguments: 1) hypothesis is correct because its meaning explains the change of the observations. In practice, the causes of situational changes are often diverse, and therefore the answers are seldom unique. It follows that we do not need to intentionally distinguish or rank the correct answers. 2) hypothesis is wrong because it can not explain the cause of some event. Therefore, all wrong answers contribute the same i.c.. it is plausible to treat all wrong hypotheses equally in the process of constructing our reasoning network. We argue that the proposed abductive reasoning principle is closer to commonsense reasoning by humans than previous ranking- approaches.
Based the above reasoning, propose abductive reasoning model called Interactive Model with Structural Loss (IMSL) for aNLI as shown in Figure 2 The IMSL model mainly consists of two components: interactive model and structural loss. On the hand, note that in the process
2

---

Under review as a conference paper at ICLR 2022
Softmax
joint Softmax- Focal loss
Score
Score
Score
Score
Score
Model
Model
Model
Model
Model
S(H')
S(E+)
S(H')
S(H²)
S(H))
Original model
IMSL
Figure 2: Comparison of the interaction between the traditional model and the IMSL model, where S (H)) represents the input sequence containing the i-th hypothesis
of extracting the language features of an arbitrary hypothesis, its relationship to other hypotheses should also be considered because the hypotheses often semantically related Pearl (1986). To this end, we can design an information interaction layer to capture the relationship between different hypotheses and produce more discriminative language feature vectors. On the other hand, we have constructed a new loss function called "joint softmax focal loss" inspired by a recent work (Lin et al. (2177). It is essentially structural softmax based Focal loss formed by sequentially constructing a loss for each score group that composed by a correct hypothesis and all wrong hypotheses. When compared with conventional models, we argue that IMSL more powerful for the task of aNLI by jointly exploiting the rich relation among competing hypotheses. The main technical contributions of this work can summarized as follows.
1) For oNL task, we claim that, the correct hypotheses of a given observation pair are often diverse, and there no need to tell them apart. The wrong hypotheses contribute the same to the task. We regroup instead of ranking all hypotheses, shown in Figure 1
2) Aiming at the problem of incorrect probability distribution between correct hypotheses in training process, joint softmax focal loss is proposed. For the hypotheses groups formed in the rearrange process, we design a filtmax-based focal loss for each group and combine them into a joint loss.
3) In view of the problem that traditional models cannot capture the language relationship between different hypotheses, have added information interaction layer between different hypothesis models. The information interaction layer increases the area under the receiver's operating charac- teristic curve (AUC) by about 5%.
4) Impressive abductive reasoning performance is achieved by IMSL when tested using RoBERTa the pretrained language model. The best language model DeBERTa (He al. (202I) is not tested due to the constraint by our limited GPU resources (4-piece RXT 2080Ti). In our experiment, compared with all recent algorithms whose codes have been made publicly available, the IMSL method has achieved state-of- oate--of-the-ar art results in ACC and AUC on both the validation set and test Besides, on the public leaderboard"] IMSL the best Bon-DeBERTa based algorithm and ranks 4/56 in all (including both DeBERTa based and non-DeBERTa based) competing methods.
2 RELATED WORK
NNL task solves abductive reasoning problem based natural language inference (NLI). In the past years, there has been an explosion NL.I benchmarks, since the Recognizing Textual Entail-
https::/leaderbbatt 1enai .rgg an1 submi ssi ssions/publlee

---

Under review as a conference paper at ICLR 2022
ment (RTE) Challenges introduced by (Dagan et al. (2005) in the early 2000s. Then, in order to find the most reasonable explanation of the incomplete observations, (Bhagavatula et al. (2020)) studied the feasibility of language-baseer abductive reasoning and proposed the task of aNLI It pays more attention to the information provided in the premise than the RTE task. For traditional RTE, the main task is to judge the relationship between the premise sentence and the hypothetical sentence, but the main objective of aNLI is to select the most plausible hypothesis among the hypotheses given Iwo observations.
oNLI is the first language-basee abductive reasoning study. This shift from logic-based to language- based reasoning draws inspirations from significant body of works on language-bart based entailment al.](2011))))
(Bowman et al. (2015); WWlllias et al. (2018)) 8)), language-bel baseee logic (Lakoff (1970); MacC Crrnny & :
ZellerZellery /Zllers et (2018)) In addition to establish oNLI, (Dagan et al. (2005) have also released a new challenge dataset, i.e., ART, which can be visited through the first footnote in this paper. The authors have also formulate the task as multiple-choice task to support easy and reliable automatic evaluation. Specifically, from a given context, the task is to choose the more reliable explanation from a given pair of hypotheses choices
Manning (2007)) and language-based commonsense reasoning (Mostafazadeh et al. (2016);
However, discriminating correct from wrong does not measure the plausibility of a hypothesis in aNLI (Zhu et al. (2020)). So. to fully model the plausibility of the hypotheses, Zhu et al. turn to the perspective of ranking and propose a novel learning to rank for reasoning (T.2R²) approach for the task. The authors rank the hypotheses based on the number of times they appear in the dataset, and use some pairwise rankings as well as listwise ranking as loss. Pairwise rankings contains Ranking SVM (Herbrich et al. (2000), RanrNet(Burges et al. (2005)). ). LambdaRank (Burges et al. (2006)) ) and Listwise Ranking contains ListNet(Cao et al. (2007)) ListMLE (Li et al. (2020) and ApproxNDCG@Qin al. (2010)) The experiments on the ART dataset show that reformulating the oNLI task as ranking task really brings obvious improvements. After that, (Paul & Frank (2020))
proposes novel multi-head knowledge attention model that encodes semi-structured commonsense inference rules and learns to incorporate them in a transformer based reasoning cell. The authors still prove that a model using counterfactual reasoning is useful for predicting abductive reasoning tasks. Accordingly, they have established new task called Counterfactual Invariance Prediction (CIP) and provide a new dataset for this.
In addition to the abductive reasoning models, the pre- pre-teained language model still plays impor- tant role in aNLI task. Early ways for language inference are constructed directly by some simple statistical measures like bag-of-words and word matching. Later, various kinds of neural network architectures are used to discover useful features in the languages, like word2vec Mikooovv et al.) (2013) and Glo GiennnPennnnooon et al. (2014)). Recent works have developed contextual word rep- resentation models, E...eedddings from Language Models (ELMO) by Peters et al. (2018) and Bidirectional Encoder Representations from (eoesoormersBBBRRTT by Devlin al. (2019). The orig- inal implementation and architecture of BERT has been outperformed by several variants and other transformer- beaserr models, such as RoBERTa, DeBERTa and UNIMO. RoBERTa et al. (2019) replaces training method of BERT and uses larger batches and more data for training. DeBERTaie (He jet al. (202T)) uses the disentangled attention mechanism and an enhanced mask decoder to improves the BERT and RoBERTa models. In order to effectively adapt to unimodal and multimodal under- standing task, [i et al. (2021) proposes the UNIMO model. In this paper, however, restricted by our computing resources, RoBERTa is selected as our language model.
3 INTERACTIVE MODEL WITH STRUCTURAL LOss (IMSL) METHOD
IMSL model consists of two components: context coding layer and information interaction layer (the backbone network) as well as a joint softmax focal loss (objective function). The design of the model architecture and loss function are described in detail below.
3.1 INFORMATION INTERACTION MODEL
Model input: Under the framework of IMSL,, a training sample X includes two given observa-
tions (i.e., O and O2) and group of candidate hypotheses denoted by H = { (N is the
W are assigned to each hypoth- number of candidate hypotheses). Then, binary labels y {yا}
4

---

Under review as a conference paper at ICLR 2022
h
4
Pretrained Modil
X
h2
72
Xv1
S
hy
2+1
he
01
X4
H
O2
Score
Context coding layer
Information interaction layer
Figure 3: The proposed IMSL model consists of context coding layer (using pre-trained model) and an information interaction layer (characterizing the relationship among different hypotheses).
esis by yj = when HD is the correct, and 0 when Hi is the wrong. The task of abductive reasoning can be characterized by a mapping from X to y. For explicitly estimating the relation between each hypothesis and the two observations,
construct triad for each hypothesis as Ty = HP; O2] (H² E H). This way, each sample in the training set can be represented by [21, 22. INl [31, 1/2 ... 3NN].
Context coding layer: We use a pre-trained language model (RoBERTa-large is used in our exper- iment) to calculate the contextual representation of the text. For each word single input Ty, an embedding vector with context information is generated. For each sentence in a single input Ij, sentence-level embedding matrix vj = encode (Ix) is first obtained, where encode(-) denotes the pre-trained model for encoding. Then we can sum the word embedding dimensions in the feature matrix to generate the feature vector 2j-
Information interaction layer: Traditional models only consider single input zz during scoring as shown in Fig. 2] which makes it difficult to capture the relationship between different inputs (c.g., Ty and xj+1). To exploit the dependency between two different inputs, propose to construct novel information interaction layer as follows. First, a pair feature vectors and 2j+1 can be generated after zj and Iz+1 are passed through the context encoding layer. Second, plug 2 and Zj+1 into the information interaction layer and use BiLSTM to acquire the distributed feature representation fj and fj+1. Finally, fully connected module outputs the corresponding scores 8; and s,+1. A flowchart of the context coding and information interaction layers is shown in Figure
To efficiently use contextual information, we use Zj as the input of BiLSTM, which aim exploiting the dependency relationship between the feature vectors. BiLSTM uses a forward LSTM and
backward LSTT for each sequence to obtain separate hidden states: Tiy, h j. The key idea of BiLSTM is to generate the final output at time by concatenating these two hidden states:
h
(1)
After passing the BiLSTM layer, we can get bidirectional hidden state vector hj. then use the fully connected layer to generate the final score sj. For computational efficiency, linear regression formula adopted here for prediction score:
= W, hy + bj.
(2)
where W, 2dxx by E Id..
5

---

Under review as a conference paper at ICLR 2022
3.2 JOINT SOFTMAX FOCAL LOSS FUNCTION
Based on the output score layer of the IMSL model, we propose to design a new structural loss function based the principle of abductive reasoning. Instead of ranking-based approach, the proposed loss function for each sample is formulated as a linear combination of softmax focal losses for several rearranged groups. which is called joint softmax focal loss. The detailed procedure of generating multiple rearranged groups is shown in Figure 4 We note that it is unnecessary to compare the group of correct hypotheses; while the exploration of the relationship between correct hypothesis and wrong hypotheses is sufficient for the task of score prediction. Therefore, we can rearrange the set N prediction scores into several groups, each which only contains a single correct hypothesis. A toy example is given in Figure where the two hypotheses are correct, and all other hypotheses are wrong. In this example, the total N scores can be divided into two groups associated with two correct hypotheses, respectively.
Score
S
S3
Sy
S
Softmax +oocal loss
S2
S3
Sy
S
Sw
Softmax loss
S
Sy
Sy
correct hypothesis
joint Softmex Focal loss
wrong hypothesis
Figure 4: An example of rearrange groups for joint softmax focal loss.
With the above construction, we can first apply the softmax operation to each rearranged group and then combine them into joint loss function. In the first step, each prediction score is given by:
Yn 1. is
Un
Yn O.
where yn is the correct/wrong label, and in represents a predicted value. The normalization factor K = Σ, My represents the number of correct hypotheses. Note that sy represents the scores of the wrong hypotheses, where i is the position of the false label. Similarly, st indicates the score of the correct hypotheses.
In addition to the softmax loss, we have borrowed the idea of focal loss from (Lin et al. (2017)) and introduce a balancing factor (0, 1) control the shared weight of the correct hypothesis and the wrong ones. Here, a is used for the correct hypotheses, and - is used for the wrong hypotheses, i.e.
An = Yn a + (1 - yn) (1 a). Putting things together, we can rewrite the joint softmax focal loss as
(4)
L = F((u = Σ Bn (1 pm) log (px).
(5)
where
Pr. = 8n + (1 -0)-)))- 3n) (1
)m.) +
(6)
Here, the parameter y is included for regulating the model's attention to hard hypotheses during the training of IMSL model. As suggested in (Lin et al. (007))) 7 E [0.5, 5]. Then, small positive real number E of 1e-8 is used to avoid the numerical instability problem. In practice, both a and r are used hyper-parameters which can be tuned by cross-validation. For the example shown in Figure

---

Under review as a conference paper at ICLR 2022
by assuming that the softmax focal losses for the two groups are Lgroupi and Egroup2, Obtain the overall loss by Leample = Lgroupl + groupp. Furthermore, the total loss for all training samples can be estimated by the sum of the losses over individual samples.
4 EXPERIMENT
In this section, the experimental results on public data sets are presented to evaluate the method proposed in this paper.
In recent years, more and more Pre-Training models have been proposed, such as DeBERTa (He et al. (202T)), UNIMO (Li et al. (202T)), etc. They use more data for training and have more parameters. Due to limited computational resources, we did not conduct comparative experiments with these high-performing yet computationally demanding pretrained models.
Evaluation indicators: AUC and ACC are the most common evaluation indicators. Since the original ACC cannot evaluate the model that is far away from the test data, AUC is added as an additional evaluation index to handle skewed sample distribution. AUC is measurement method that is statistically consistent and more discriminative than ACC.
4.1 EXPERIMENTAL. SETUP
Tasks and settings: The aNLI task uses the ART dataset, which is the first large-scale benchmark dataset used for abductive reasoning in narrative texts. consists of about 20,000 observations and about 200,000 pairs of hypotheses. The observations come from collection of manually curated stories, and the hypotheses are collected through crowdsourcing. In addition, the candidate hypothe- for each narrative context in the test set are selected through the adversarial filtering algorithm with BERT-L (Large) as the opponent. The input and output formats are shown in Table 2
Table 2: The format of data input and output in oNLI task
Task
Input Format
Output Format H' H
oNLI
[CLS] O [SEP] H' [SEP] O2 (SEP)
Hyperparameter details: Due to the difference in the amount of data, the focusing parameter and the amount of training data will vary. For different training data, select the hyperparameter that the set. Specifically, the learning rate is fixed 1e-6, the
produces the best performance batch size is fixed at 1, and the training batch will vary with the amount of training data. Training uses Cross Softmax+Focal Loss. For the validation set, ACC and AUC are used for evaluation. Use the results of five different seeds to evaluate the performance of the test set.
Baseline: We have used the following four baseline models for comparison: A) BERT (Devlin al. 2019) is language model that uses a masked language model and predicts the next sentence as the target training. For example, it masks certain words in the input, and then trains and predicts the words that are blocked. B) RoBERTa (Liu et al.) 2019) has the same structure as BERT, but there is no prediction (NSP) for the next sentence. RoBERTa-B (ase) and RoBERTA-L. (arge) use more data and larger batches for training. C) Learning to Rank for Reasoning (T22R² Zhu et al. (2020) reprogrammed the aNLI task as a ranking problem, using a learning ranking framework that includes score function and loss function. D) Multi- Knowledge Attention (MHKA) (Paul & Frank (020) proposed new multihead knowledge attention model, and used a novel knowledge integration technology.
4.2 EXPERIMENTAL RESULTS
Our experimental results in the aNLI task are shown in Table The baseline comparison models are: Majority, GPT, BERT-L, RoBERTa-L, L2R² and MHKA related results. It can be observed that the IMSL method improves about 3.5% ACC and about 7.5% in AUC compared with RoBERTa- L. The results show that the improvement of ACC is mainly attributed the new IMSL loss function,
7

---

Under review as a conference paper at ICLR 2022
Table 3: Results the oNLI task: The results are quoted from (Bhagavatula al. 2020
L.=Large
Model
Dev(ACC%)
Dev(AUC%)
Test(ACC%)
Human Perf
91.40
Majority
50.80
GPT
62.70
62.30
BERT-L
69.10
69.03
68.90
RoBERTa-L
85.76
85.02
84.48
T22R2
88.44
87.53
86.81
MHKA
87.85
87.34
Ours
RoBERTa-L+MSSL
89.20
92.50
87.83
and the improvement of AUC is mainly attributed to the exploitation of the relationship between the hypotheses by the proposed information interaction layer.
Low-resource setting: Testing the robustness of the model to sparse data on aNLI tasks refers to the low-resource scenario where the MHKA model uses 10, % training data respectively. Figure Shows how the model improves on MHKA, RoBERTa-Large and L2R². The experimental results show that the model in this paper achieve better results in the case of low resource setting. When using 1% training data only. the improvement brought by IMSL is the most significant, which about 4% higher than that of T.2R² and MHKA. Experimental results show that method performs consistently better than other competing methods on low-resource data sets.
90
RoBERTa-Large
RoBERTa-Large
L2R²
RoBERTa-Large IMSL
Accuracy%
80
75
(1. (1)7k)
(3.4k)2%
(8.5k)5%
(17k)10%
(169k)100%
Figure 5: The accuracy of oNLI under low resource settings
DETAILED ANALYSIS VIA ABLATION STUDY
To more clearly show the contribution of each module, we have done corresponding comparative experiments on both information interaction layer and hyperparameter tuning.
5.1 ABLATION STUDY OF THE INFORMATION INTERACTION LAYER
First, we have conducted ablation study experiments on the related BiLSTM to investigate the role played by the information interaction layer. The hyperparameters of Focal Loss will be fixed to reduce the impact on BiLSTM. Through the experimental results, can be found that the addition of BiLSTM greatly improves the AUC, but does not have significant impact ACC. The following
8

---

Under review as a conference paper at ICLR 2022
Figure 6 shows the visualization results the validation set. In the plot, the abscissa is the score of hypothesis and the ordinate the score of hypothesis 2. The red points in the upper left corner correspond to the subset of correct hypotheses, do the blue points in the lower right corner. It can be seen that the introduction of the information interaction layer pushes all points further away from each other and toward the four corners. It follows that the margin between the positive and negative samples is larger, implying improved discriminative power.
125
18.0
35
-35-
-10
-5.0
-2.5
2.5
5.0
2.5
10.0
12.5
(a) using the information interaction layer
(b) without the information interaction layer
Figure 6: Score distribution using and without using the information interaction layer
5.2 PARAMETER COMPARISON
When we select the parameters of Focal Loss, several experiments were carried out on two hyperparameters of the balancing factor and the focusing parameter. The focusing parameter y in Eq. equation[5Jcan automatically doigh the contribution of easy examples during training and rapidly focus the model hard examples; while the balancing factor 0 < controls the tradeoff between correct and incorrect hypotheses. Figure 7 below shows the ACC performance of IMSL model with different focusing parameters and balance factors. In this study,
in Eq. equation 4
{1, 2, 3} is used as the option of focusing parameter, and {0.45, 0.5, 0.55) is used as the set of balancing factor. It can be observed that as the most effective parameter couple given by 7 2, 0.55.
89.1
89.0
848.87
89900
0/9.1
88.6
0.45
0.5
0.5s
balance factor
Figure The ACC results of adjusting the balance factor y and focusing parameter a.
SUMMARY
In this paper, IMSL method is proposed for commonsense abductive reasoning. It includes information interaction layer that captures the relationship between different hypotheses, and a joint loss for our proposed way of grouping the correct/wrong hypotheses. Experimental results show that on aNLI tasks, IMSL has better performance on ACC and AUC, especially in low-resource settings, IMSL can significantly improve the accuracy.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Lisa Bauer and Mohit Bansal. Identify, align, and integrate: Matching knowledge graphs to com- monsense reasoning tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pp. 2259-2272. Association for Computational Linguisties, 2021.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 632-642. The Association for Computational Linguistics, 2015.
Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning to rank using gradient descent. In Machine Learning, Proceed- ings of the Twenty-Second International Conference (ICMI. 2005), Bonn, Germany, August 7-1 I, 2005, volume 119 of ACM International Conference Proceeding Series, pp. 89-96. ACM, 2005.
Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19, Proceedings of the Twenti- eth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pp. 193-200. MIT Press, 2006.
Zhe Cao, Tao Qin, Tie-Ya Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach listwise approach. In Machine Learning, Proceedings of the Twenty-Fourth Interna- tional Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, volume 227 of ACM International Conference Proceeding Series, pp. 129-136. ACM, 2007.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Papers, pp. 1657-1666 Association for Computational Linguistics, 2017.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, volume 3944 of Lecture Notes in Computer Science, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLTT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4188 Association for Computational Linguistics, 2019.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhanee enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 202 Openkeview.net, 2021.
Ralf Herbrich, Thore Graepel, Klaus Obermayer, et al. Large margin rank boundaries for ordinal regression. Advances in large margin classifiers, 88(2):115-332 132, 2000.
Yichen Huang, Yizhe Zhang, Oussama Elachqar, and Yu Cheng. INSET: sentence infilling with inter-sentential transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2502-2515. Association for Computational Linguistis,, 2020.
George Lakoff. Linguistics and natural logic. Synthese, 22(1):151-271, 1970.
10

---

Under review as a conference paper at ICLR 2022
Minghan Li, Xialei Liu, Joost van de Weijer, and Bogdan Raducanu. Learning to rank for active learning: A listwise approach. In 25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021, pp. 5587-5594. IEEE, 2020.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang- UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learn- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 11th International Joint Conference Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pp. 2592-2607. Association for Computational Linguistics, 2021.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2999-3007. IEEE Computer Society, 2017.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019.
Kaixin Ma. Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. Knowledge-drivee driven data construction for zero-shot evaluation in commonsense question answer- ing. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thiry-Third Confer- ence on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13507-13515. AAAI Press, 2021.
Bill MacCartney and Christopher D. Manning. Natural logic for textual inference. In Proceedings of the ACL-PASCAL @ ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic, June 28-29, 2007, pp. 100-200. Association for Computational Linguistics, 2007.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In NAACL 2016, The 2016 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA. June 12-17, 2016, pp. 839 849. The Association for Computational Linguisties, 2016
Debjit Paul and Anette Frank. Social commonsense reasoning with mult knowledge attention. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16- 20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 2969-2980. Association for Computational Linguistics, 2020.
Judea Pearl. On evidential reasoning in a hierarchy of hypotheses. Artificial Intelligence, 28(1): 9-15, 1986.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2014, October 25-29, 2014, Dohu, Qatar, meeting of SIGDAT, a Special Interest Group of the ACI., pp. 1532-1543. ACI,, 2014.
Matthew E. Peters, Mark Neumann, Mohit Tyyer, Matt Gardner, Christopher Clark, Kenton Lee. and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pp. 2227-2237. Association for Computational Linguistics, 2018.
Tao Qin, Tie- Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Inf. Retr. 13(4):375-377, 2010.

---

Under review as a conference paper at ICLR 2022
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL-HLI 2018, New Orleans. Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018.
Changlong Yu, Hongming Zhang, Yangqiu Song, Wilfred Ng, and Lifeng Shang. Enriching large- scale eventuality knowledge graph with entailment relations. In Conference on Automated Knowl- edge Base Construction, AKBC 2020, Virtual, June 22-24, 2020, 2020.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 1044 Association for Computational Linguistics, 2018
Wangchunshu Zhou, Dong-Ho Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. Pre-training text-to-text transformers for concept-centrie common sense. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Yunchang Zhu, Liang Pang, Yanyan Lan, and Xueqi Cheng. L2r²: Leveraging ranking for abductive reasoning. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp- 1961-1964. ACM, 2020.
12


Reviews:
[{"review_id": "mWW_23plCDa", "reviewer": "Reviewer_imYG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The approach shows significant improvements on ACC/AUC compared to previous methods.\n- The approach seems to be motivated well. It is reasonable that it might not be helpful to distinguish amongst the set of correct hypotheses and the set of incorrect hypotheses.\n\nWeaknesss:\n- Presentation: The paper is not written in a clear and concise manner. For example, there are some loose claims about abductive natural language inference in the first paragraph. Description of previous work such as Paul and Frank, 2020 is not presented clearly. Finally, there are some details that might not be necessary to include in the paper (description of BiLSTM, repetitions in description of alphaNLI). Also, Figure 2 is not easy to understand and I would suggest revising it.\n- Impact: Do the authors think their approach could be used for tasks other than alphaNLI? I'm a bit concerned that the proposed approach is too specific to the task of alphaNLI. While a better approach specific for alphaNLI is valuable in itself, but it does limit the generality of the approach. \n- Results: Some of the results are missing. The AUC/ACC scores corresponding to ablation in sec 5.1 are missing. Also, the experiment in sec 5.2 is not particularly useful and could simply be summarized by saying that the \"approach is not as sensitive to hyperparameter settings\".\n- Error analysis: It seems like model performances on the ART task are nearing human performance. In this scenario, it would have been helpful to see some examples where the model still fails to predict the right answer and also examples where their proposed model helps. There is lack of any error analysis presented in the paper.\n\nQuestions: \n- Can you provide the AUC/ACC scores corresponding to ablation performed in sec 5.1?\n- Do you have any thoughts on how one might adapt this framework for abductive natural language generation?\n\nOther typos / language:\n- For in-line citations, please use \\citet.", "summary_of_the_paper": "This paper proposes a novel method called Interactive Model with Structural Loss (IMSL) to tackle the task of abductive natural language inference. Their approach is based on the observation that it may be unnecessary to intentionally distinguish among the set of correct hypotheses and the set of incorrect hypotheses. Specifically, they propose a novel loss function called the joint softmax focal loss function (based on Lin et al, 2017) that groups together a single correct hypothesis together with all the incorrect hypotheses. Their approach first uses a pretrained language model to get encodings for each of the hypotheses which are then passed to a BiLSTM to get scores for each hypothesis. The first component is called the context coding layer and the second one is called an information interaction layer.\n \nThe authors perform experiments on the ART dataset from Bhagavatula et al, 2020 and show improvements on area under curve (AUC) and accuracy (ACC) compared to previous methods. Further, they present an ablation experiment where they remove the information interaction layer and show that it helps with distinguishing correct and incorrect hypotheses.", "summary_of_the_review": "I think the paper proposes a useful method for the task of abductive natural language inference based on good intuition.\nHowever, I'm afraid the impact of this work might be limited (disregarding the fact that larger models like DeBERTa outperform the proposed approach) and the presentation/analyses needs quite a bit of work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tk3zY5sQiPW", "reviewer": "Reviewer_hk5y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nS1) The paper obtains a high score on the $\\alpha$NLI dataset, surpassing most models on the public leaderboard except for some recent ones that use improved pre-trained representations (DeBERTa and UNIMO). \n\n*Weaknesses*\n\nW1) I have one major question about the evaluation (see question Q1a below) which makes me wonder whether the results are comparable to some of the past work (e.g. the leaderboard results cited).\n\nW2) It was really difficult to tell what's responsible for the improvement in results over past work, especially given that I found it difficult to understand the motivation for parts of the approach (see below). I would have really appreciated some ablation experiments that individually removed the parts (the info interaction layer, and the focal loss) individually and in combination to help explain what's the cause of these strong results (and give more confidence that these strong results on this single dataset aren't just due to e.g. better hyperparameter tuning or optimization). Section 5.1 makes it seem like one of these ablations was done, but there don't seem to be any evaluation results reported from it which struck me as odd.  The closest the paper came to presenting evaluation results was the RoBERTa baseline in the results table, but the paper didn't give any details (that I could find) about this RoBERTa baseline's implementation in comparison to the full proposed method. \n\nW3) The model was quite complex, and a few choices seemed strange to me.\n- Is the ordering of the candidate hypotheses 1...N significant? If so, how is it determined? If not, why use a BiLSTM in the information interaction layer?\n- There's a number of changes from the L2R^2 method, if I understand correctly (at least from the way L2R^2 was described): switching from a total ordering to a partial ordering, switching from a ranking loss to a softmax, and using a focal loss. While I feel like the introduction and Figure 1 help motivate using a partial ordering, I wasn't clear about why the other changes are necessary, or if they help (e.g. why not use one of the more standard pairwise rankings cited in Section 2)?\n- Why use a focal loss? If I understand correctly, focal loss is motivated in class imbalance, but it wasn't totally clear to me where that imbalance comes from here.\n\nW4) I felt like the impact of the paper is likely to be limited, with the paper in its current form: it's a relatively complex model for one single task, with little analysis of why the method improves or indication that the approach could be more broadly useful, and it relies heavily on one feature of this dataset (having multiple annotated hypotheses for a single pair of observations).\n\nW5) Parts of the writing were pretty unclear to me. In particular I found it difficult to follow the description of the joint softmax layer in section 3.2 (see questions below), and (to a lesser extent) the information interaction layer.\n\n*Questions*\n\nQ1) How are predictions made from the model during evaluation? In particular:\n\nQ1a) Is the interaction layer used in evaluation, and if it is used, what candidate hypotheses are used for a given observation pair? If I understand the original $\\alpha NLI$ setup, there are just a pair of hypotheses for each obesrvation pair (a binary classification task); is that also the setup here in evaluation (I understand it's not, in training)? \n\nQ1b) How is the classification obtained in evaluation? Which quantity in section 3.2 is used to select a hypothesis?\n\nQ2) In Section 4.1,  \"For different training data, select the hyperparameter that produces the best performance on the *test set*\". I'm assuming this is a typo, but just to confirm, what data split was used to tune the hyperparameters (validation or testing)?\n\nQ3) Section 4.1: \"Use the results of five different seeds to evaluate the performance of the test set\". What does this mean -- how were the seeds used? To compute an average? If so, it would help to report standard deviations along with the results in Table 3.\n\nQ4) \"Specifically, the learning rate is fixed at 1e-6, the batch size is fixed at 1, and the training batch will vary with the amount of training data.\" What does it mean to have a varied training batch but a fixed batch size? And does \"amount of training data\" here refer to the low-resource setting at the end of 4.2, or something else?\n\nQ5) I was pretty confused by the $s_i^0$ and $s_i^1$ notation in section 3.2, and in general was unclear on how Equation (3) was motivated or derived. It would be really helpful to explain this in more detail.\n\nQ6) Could more details be given about training: what optimizer is used, and is any early stopping performed (and if so what's the stopping criterion)?\n", "summary_of_the_paper": "This paper proposes a model and loss function for the task of abductive natural language inference ($\\alpha$NLI), which is a commonsense reasoning task that requires a model to choose which of two possible *hypothesis* sentences best goes in between two *observation* sentences, to explain the second observation (e.g. for observation sentences \"Josh bought a parrot\" and \"Josh taught his parrot how to say its name\", selecting the hypothesis \"Josh realized the parrot could talk\" over \"He is scared of birds\"). \n\nThe paper, following past work (Zhu et al. 2020), uses the fact that the training set for this dataset contains multiple instances with the same pair of observations (but different hypotheses), allowing training using a ranking loss rather than just a binary classification loss. This paper proposes to use a BiLSTM to aggregate information across RoBERTa encodings of hypotheses for the same observations, groups together subsets of the hypotheses, and applies the focal loss of Lin et al. 2017 on top of a grouped softmax. \n\nThe full model improves performance substantially on the dataset in comparison to baselines and other past work that also uses a RoBERTa encoder.", "summary_of_the_review": "I didn't feel like this paper is strong enough in its current form -- while it obtains strong results on the $\\alpha$NLI dataset, I felt like there's unfortunately not much here that other work would be able to build on. I had trouble understanding why the proposed method does well, or whether it would be likely to work on other tasks (or even other datasets for the same task if they lacked the multi-view property this one has). The paper would be a lot stronger with clearer motivation for the technique, some ablations and more thorough comparisons to baselines, and improved writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PdvTVd44Hsl", "reviewer": "Reviewer_cRzc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths of the paper\n- The paper is overall well written. The text describes the problem well. The proposed model changes are easy to follow as well as the focal loss.\n\n- The experimental analysis show strong, but not state-of-the-art results, on the aNLI (ART) dataset.\n\n- The most interesting result, to me, are the results on the artificially low resource setting (1-2% of the training data). This result shows there seems to be some value of the proposed approach in improving sample efficiency during learning.\n\n### Weaknesses of the paper\n\n- One main point that I struggled to understand from this work is whether the techniques proposed here are specific to the ART dataset, or whether they make sense to apply in other settings. Are there other multiple-choice tasks in which the combination of late-stage interaction of representations and focal losses improves learning over, for example, learning-to-rank approaches?\n\n- Another primary concern relates to the modest overall improvement in performance for ART. Specifically, it seems that ART, by itself, is not such a useful task. But it is a useful task to (1) evaluate progress in natural language understanding more generally, and (2) may transfer learning to other tasks. In fact, in the original aNLI paper (https://arxiv.org/abs/1908.05739 ) Section 7 describes how learning from ART can be transferred over to other tasks. This work does not show whether the new techniques will build better representations to transfer to new tasks or whether better performance on ART is derived solely from new information learned in the newly introduced interaction layers.\n\n- I found Section 5 a bit lacking in terms of details that I would have expected in order to fully comprehend the contributions of this work. For example, Section 5.1 attempts to study the contribution of the interaction layer. However, besides the scatter plot in Figure 6 (which is quite hard to interpret, it doesn’t seem convincing that (a) discriminates items better than (b)), we don’t have a quantitative analysis of the impact of the interaction layer. How much of the performance is degraded/improved by adding or removing the interaction layers? This crucial information is missing,\n\n- Similarly in Section 5.2, we see the results of modifying some hyperparameters for the focal loss. However, the most important result in my opinion, is observing the effect of removing the focal loss altogether (falling back to simpler cross-entropy softmax or learning-to-rank loss). \n- As it stands, it is not possible to infer how much of the improvement described in Section 4.2 (Table 3 and Figure 5) is derived from the interaction layer and how much is derived from the focal loss.\n\n- One final weakness of this work is that none of the two techniques used here (interaction layers and focal loss) are new to this paper, as the Related Work section of this paper clearly states. What is new is their application to the aNLI task. As a result I would have expected a much more thorough analysis of their impact on this task, clearly outlining why these techniques should be applied in tandem when modeling abductive reasoning.\n\n### Other comments\n\n- In Section 3, the per-example representation is derived in the “context coding layer” by summing the individual word (token?) representation. Is this correct? If so this seems quite unintuitive, and warrants more explanation. In particular when pooling representations, particularly of variable-length sequences, there is a normalization problem that occurs (e.g., short vs. long sentences contain different numbers of tokens). That is why pooling methods use average or max, or attention, or even a reserved [CLS] token that can attend to the entire sequence. I would have liked to see some explanation for this choice, and why this method is expected to work well.\n\n- It would have been useful to compare how the focal loss compares to a simpler weighted binary cross-entropy loss where each positive is compared to each negative in the batch.\n\n- To improve the paper, I would have focused more on the low resource setting which seems to be the main clear benefit of the proposed techniques (Figure 5). In my opinion, the impact of this work could be much greater by (1) finding other related tasks where a  multiple-choice task is being modeled, and (2) comprehending whether they also improve in low resource settings with the techniques described here. With better analysis of the impact of late-stage interaction layer vs. focal loss, one may be able to get more insight into why sample efficiency seems to be improved. Finally, I would focus on a qualitative analysis of the wins in the low resource setting attempting to classify the types of examples that are improved using the techniques here versus a L2R2 or other approaches.\n", "summary_of_the_paper": "This paper proposes a new model architecture, along with a new loss, to solve the abductive natural language inference (aNLI) task. Specifically, this work hypothesizes that models benefit from the interaction between multiple choice options, and thus introducing a “information interaction layer” to allow the interaction of representations of individual options. Further, it proposes the use of a “focal loss” to rebalance the loss weight between positives and multiple negative examples during training. This work evaluates proposed changes on the ART dataset, extending the evaluation to an artificial low-resource setting. Finally, the paper presents preliminary analysis of the effects of the interaction layer and focal loss on the results.\n", "summary_of_the_review": "While this study has some preliminary results that indicate that late-stage interaction between options in a multiple-choice task setting is useful, I found that the work has a few flaws and is slightly premature for publication. Firstly, it relies heavily on specific behavior of the ART dataset, specifically, multiple gold positives and multiple gold negative examples in the training set. Also, the experiments need to be improved a bit to better disentangle the impact of late-stage interaction and focal losses on the performance of this task. Finally, besides the very low resource setting, it's not clear that the proposed changes make a significant impact in the performance of task; the reported gains seem quite modest.  My recommendation to the authors is to potentially improve this work by focusing on the low-resource setting, where the proposed techniques show some potential improvements in sample efficiency (see comments above for specific recommendations on how one could explore such a study).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NLkRVBD-CEc", "reviewer": "Reviewer_3Vwo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper proposes a state-of-the-art model on the ART dataset which ranks the fourth in the leaderboard (see footnote 1 in the paper), only lower than UNIMO and DeBERTa based models.\nWeaknesses:\n(1) The proposed joint softmax focal loss seems not applicable to the experimental ART dataset. If I do not misunderstand, the footnote 1 provides a link to the leaderboard of the ART dataset. However, the experimental dataset in this leaderboard consists of only instances having exactly two hypotheses, where one is correct and the other is wrong. The joint softmax focal loss is designed specifically for instances having three or more hypotheses where the correct hypotheses are multiple. In other words, this proposed loss is not evaluated in the experiments.\n(2) The ablation study for the BiLSTM-based information interaction layer is unclear. The main results are shown in Figure 6, but it is unclear what the red points are and what the blue points are. It is said in the paper that both the red points and blue points refer to correct hypotheses. It is also unclear why the separation of major red points and major blue points into two different corners implies a better performance.\n(3) The evaluation on the effectiveness of the information interaction layer is also incomplete. It is insufficient to show this effectiveness based on a single pretrained language model RoBERTa-Large. Although the authors declare that they have no experimental resources for UNIMO and DeBERTa, they could still conduct experiments on UNIMO-base and DeBERTa-base to show that extending these base models (requiring similar computation resources as RoBERTa-Large) outperforms the corresponding baselines.\n", "summary_of_the_paper": "This paper proposes a new model for the abductive natural language inference (alphaNLI) task. The model extends a pretrained language model with a BiLSTM-based information interaction layer and the joint softmax focal loss. Experimental results on the ART dataset show that the proposed model built on top of RoBERTa-Large outperforms its baseline RoBERTa-Large as well as state-of-the-art learning-to-rank-based models.", "summary_of_the_review": "Please provide a short summary justifying your recommendation of the paper.\nThe paper has marginal novelty on two extensions of a pretrained language model, but one extension (the BiLSTM-based information interaction layer) is not completely evaluated, while the other (the joint softmax focal loss) is even inapplicable to the experimental dataset. This makes the contributions of the paper unconvincing at the current stage.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9maE3AcPa0M", "reviewer": "Reviewer_W1Sp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The authors propose to use common techniques for modeling the abductive NLI task, which might be potentially useful for improving the performance of pre-trained language models.\n\nWeaknesses:\n\n- The writing is not clear and the paper has a lot of grammatical errors and stylistic issues in particular around citations. It needs proofreading.\n- It seems that the proposed approach lacks novelty where the authors use bi-directional modeling and focal loss that are common techniques in the field. Even if the improvement on the task is significant, its contribution might be incremental.\n- Given a small improvement (1% for the test accuracy) against the previous best model (MHKA), the standard deviation should be reported.\n- In Section 5.1, the authors mention that ablating the component of modeling hypotheses together doesn't have a significant impact on the test ACC. Given that the model should be evaluated on the test set rather than the dev set, the authors could not conclude that the proposed approach significantly improves the task performance.\n- There is no qualitative analysis on examples where modeling hypotheses together is expected to be meaningful and better than re-ranking them.\n\nComment:\n- Use \\cite and \\citet properly.", "summary_of_the_paper": "This paper proposes a method for the abductive natural language inference task. Different from previous state-of-the-art models that score and rank candidate hypotheses, the authors propose to model them together and compute a joint loss. The proposed method implemented on top of RoBERTa-large shows improvement by 1% accuracy in the test set from the previous best method, and 3% improvement from the simple RoBERTa-large.", "summary_of_the_review": "Lack of novelty, not a significant improvement, and no qualitative analysis on the authors' hypothesis (i.e., it's useful to model hypotheses together).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies improving model for abductive natural language inference task. Specifically, they introduce information interaction layers and the joint softmax focal loss. 

On positive notes, their method shows persuasive empirical gains. However, reviewers found (1) the technical novelty of the approach to be limited (reviewer croc, 3Vwo, W1Sp), (2) approaches (especially focal loss) not well motivated (reviewer hk5y), (3) there are limited take-away from the paper (reviewer imYG, hk5y) and (4) claims not well supported and experimental details missing (reviewer hk5y). The reviewers further provided detailed comments that would be helpful for authors to improve the paper. Because of such limitations, in its current form, the paper is not ready for publication.
Decision: Reject