A smashed glass cannot be full: Generation of Commonsense Explanations through Prompt-based Few-shot Learning
Year: 2023

Abstract:
We assume that providing explanations is a process to elicit implicit knowledge in human communication, and propose a general methodology to generate commonsense explanations from pairs of semantically related sentences. We take advantage of both prompting applied to large, encoder-decoder pre-trained language models, and few-shot learning techniques, such as pattern-exploiting training. Experiments run on the e-SNLI dataset show that the proposed method achieves state-of-the-art results on the explanation generation task, with a substantial reduction of labelled data. The obtained results open new perspective on a number of tasks involving the elicitation of implicit knowledge.

heading: 1 Introduction
text: When exchanging information, it is typical to exclude details that appear self-evident or insignificant, so that only part of the message is articulated verbally while other details are implied (Becker et al., 2020, 2021a). This is particularly true for information involving commonsense knowledge, which represents the backbone of everyday communication and reasoning. For example, consider the following sentences:
(1) The glass is broken into pieces.
(2) The glass is full.
We can intuitively assert that these two sentences contradict each other, and if we were asked to explain why, our answer would most probably appeal to some implicit knowledge about the world (“A glass broken into pieces cannot contain any liquid” and “A glass cannot be broken and full of a liquid at the same time”, etc.) to various degrees of depth. This implicit, shared knowledge can easily be inferred by people, but represents a challenge for computational systems. Crucially, a request for explanation is often a request to make explicit
something that is only implied or omitted in conversation. While this is evident in everyday communication, it is also true for more specialised domains, for example a doctor-patient scenario where the patient is given a diagnosis (“Your clinical case and tests indicate that you have type 2 diabetes”) and asks for explanations to the doctor (“Why do you believe that?”).
Therefore, the underlying assumption of this paper is that commonsense explanations are to some extent based upon the notion of implicitness, and that the information they rely on can not be fully derived from their textual input alone. Being able to elicit commonsense implicit knowledge is a relevant step forward not only towards providing explanations, but also towards better natural language understanding.
In this work, we state the “implicit knowledge problem” as the capacity to automatically generate explanations regarding the semantic relations between two sentences. We define a general methodology to elicit implicit knowledge from pre-trained large language models, motivated by the intuition that they contain most of the knowledge needed, and are able to generalize over unseen instances. In fact, a fully-supervised fine-tuning would require a large training set where each input-sentence pair should be labeled with one or more explanations, a setting which is unrealistic in an open-domain scenario.
To achieve this, we propose a combination of prompting and few-shot learning techniques, which are well-suited to exploit the generative capabilities of language models and, at the same time, make use of limited supervision. Similar methods have been applied to some popular NLP tasks such as text classification, inference, summarization (Schick and Schütze, 2021a,b) and, more recently, to teach language models to leverage external tools via APIs (Schick et al., 2023). However, the generation of implicit knowledge poses a further challenge
18
to these techniques as it requires to generate text based on information or reasoning structures that are outside the input texts1.
For these reasons, in our experiments, we select the e-SNLI dataset Camburu et al. (2018), where sentence pairs traditionally employed for a textual inference task (pairs are labelled with entail, contradict or neutral tags) were augmented with explanations for the relation tag, collected through crowd-sourcing. We compare three generation methods: unsupervised, fine-tuning and ensembling, showing that the ensembling method achieves the best results, en-par with state-of-theart while making use of limited supervision. Moreover, we find that this method mitigates the potential negative effects of “bad” prompts, which is a desirable feature whenever prompt optimization is not possible. However, we also underline that evaluation and comparison with SOTA results is still critical, as evaluation metrics are not usually directly comparable on the explanation generation task.
The innovative contributions of this paper are the following:
• We propose a general methodology to elicit implicit knowledge from language models with very limited training data.
• We compare the effects of prompting, fewshot fine-tuning and ensembling on a set of different language models, indicating which strategy suits best for each type.
• We show that one of our proposed methods for implicit knowledge generation is able to mitigate the negative impact of badly designed prompts.
The paper is structured as follows: Section 2 provides relevant background on language modelling and elicitation of implicit knowledge. Section 3 introduces the general approach to implicit knowledge generation. Sections 4 and 5 report, respectively, the experimental setting and the results we have obtained on the e-SNLI dataset. Finally, Section 6 provides relevant context of recent approaches to implicit knowledge elicitation.
1The code is available at github.com/ andreazaninello/explanationgeneration

heading: 2 Background

heading: 2.1 Language modelling and transfer learning
text: Recent advances in NLP, particularly in Natural Language Generation (NLG), have been driven by the success of transfer learning techniques applied to neural language models, pre-trained on very large textual corpora in a self-supervised fashion (Howard and Ruder, 2018; Radford et al., 2019). These general models can be trained on in-domain datasets or on specific downstream tasks with much less resource expense through fine-tuning.
Transformers (Vaswani et al., 2017), based on the attention mechanism, currently represent the state-of-the-art in most of NLP tasks. In our experiments, we employ Transformers’ encoder-decoder (sequence-to-sequence) models, like BART (Lewis et al., 2020), T5 (Raffel et al., 2020) or PEGASUS (Zhang et al., 2020). In these models, encoder attention can access the whole initial sequence, while decoder attention can only attend to previous words; these models perform best on language generation tasks that depend on a sequential input, such as machine translation or text summarization thus we hypothesize that the encoder-decoder kind of models are particularly suited to the task of generating explanations, because the sequence to be generated (the explanans) has to be conditioned on a full input sequence (the explanandum).

heading: 2.2 Prompting
text: The core technique that we use to elicit implicit knowledge is prompting (see Liu et al. (2021) for an extensive survey). Prompting consists in reframing an NLP task as a language modelling task: from a practical viewpoint, it corresponds to feeding a very large language model an input prompt that describes the desired task in natural language (and/or gives some examples of the desired output), and constructing a function that maps the desired label (e.g., positive in a sentiment analysis task) onto a series of natural language verbalizers (e.g. good, great, excellent). Given this prompt as an input, the language model is let generate the output as if it were a language modelling task such as next word or masked word prediction.
This new trend, which is especially appealing as it requires much less training signal compared to regular fine-tuning, has led to a shift from objective engineering to prompt engineering: this includes both the manual design of templates (Petroni et al., 2019) and automatic prompt learning (Jiang
et al., 2020), as well as various options to ensemble (Schick and Schütze, 2021b) and compose (Han et al., 2022) multiple prompts.
When using prompts, we can simply generate with a language model with no parameter update, giving the model some answered prompts as example to direct generation, or we can update the prompts’ parameters through the supervision given by training examples (Liu et al., 2021); additionally, some promising recent approaches have attempted to apply few-shot training techniques based on prompting to language generation, too. Schick and Schütze (2021c), for example, propose a method called pattern-exploiting training (PET) and use hand-crafted patterns as task instructions to train intermediate models, in combination with a small set of unlabeled gold training examples which they use to ensemble those models. This technique has proved successful in performing few-shot downstream NLG tasks on sequence-to-sequence models (Zhang et al., 2020) with many fewer training examples than state-of-the-art benchmarks, especially when the generated output is tightly connected to the input (e.g. text summarization). While our method is closely inspired by this line of work, to the best of our knowledge this has not been applied to tasks where models need to use implicit information outside the input text.

heading: 2.3 Evaluating generation
text: As we anticipated, we model the “implicit knowledge problem” as the capacity to automatically generate explanations regarding the semantic relations between two sentences, which brings up the issue of evaluating the quality of the generated predictions. Unlike classification and regression tasks, the evaluation of generation is known to be critical, and has often relied on human judgments (Wiegreffe et al., 2022), which is however expensive and difficult to scale.
On the other hand, automatic assessment is usually done by measuring the overlap between at least one reference generation and the system’s output, as with popular metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002). Other metrics aim to measure the semantic similarity between generations and references using contextualised embeddings, for example BERT-Score (Zhang et al., 2019) or Sentence-BERT (Reimers and Gurevych, 2019), which consider word and, respectively, sentence-embeddings.
Nevertheless, evaluating generation is a critical aspect and there is still no consensus on what metric is to be taken as a reference for system comparison, especially on explanation generation. In our experiments, we align with our selected benchmark to facilitate results’ comparison, and thus report BLEU, ROUGE and BERT-Scores. However, we are aware that results may not be fully conclusive in terms of comparison with SOTA, and that metrics are rather to be taken as a relative indication of systems’ performance.

heading: 3 Methodology
text: Our proposed methods belong to the “fixed-prompt language model tuning” types (Liu et al., 2022): the LM’s parameters are updated through few-shot fine-tuning, after both training and test examples have been modified by textual templates, meaning that the prompts’ parameters are fixed while the model’s parameters are updated.
While some recent studies aim at discovering the best prompts or performing prompt optimization (Li et al., 2021; Shin et al., 2020), our methods aim to provide a general framework that may work well even without prompting optimization, minimizing the negative impact of “bad” prompts. Finding the best patterns and prefixes would clearly contribute to the task but falls outside the scope of this study, and would be task dependent, while we aim to provide a general framework potentially applicable to different generation tasks.
Our method is closely related to Gao et al. (2021)’s, who apply a similar procedure to classification and regression tasks, and is inspired by Schick and Schütze (2021b)’s GenPET, who apply it to a set of generation tasks close to summarization, not involving the elicitation of implicit external knowledge.

heading: 3.1 Problem formalization
text: We state the “implicit knowledge generation” problem as the task of automatically producing an explanation for the semantic relation between a pair of textual sentences ta and tb. We set M to be a language model of vocabulary V , pre-trained on a masked language modelling task. We define an input x ∈ X , an output sequence y ∈ Y , a label l ∈ L, a label verbalizer v ∈ V ∗ that maps the labels to a natural language sequences (for example mapping a neutrality label to the sequence “does not entail”), and the sequence result-
ing from applying a prompt p ∈ P to input x as z = fprompt(x, p), with z containing one masked sequence <mask>. The fprompt function then simply takes the input texts ta, tb, and the verbalized label l, and returns a modified, natural language version of the input.
To obtain the prompts, we need to define (1) a set of task prefixes (pref1, ..., prefn) that introduce the explanation y ∈ Y , and are either processed as final part of the input or pre-pended to the reference explanation at training time (or to the mask sequence at test time), and (2) a set of patterns (patt1, ..., pattm), which we combine with (pref1, ..., prefn), resulting in n×m prompts. Prompts are used to rewrite each input example, by sampling randomly across all labels. We refer to this modified training set as T , which is n × m times the size of X . We summarize and give an example of our prompting design in Section A.1.

heading: 3.2 Training objective
text: The models we use are pre-trained on masked language modelling task, so their objective is calculating the probability of pM = (y|z). We have the choice of (1) processing the task prefix pref using the decoder, as part of the generated sequence, or (2) with the encoder, as part of the input. Thus, assuming some model M , some task prefix prefi and some pattern pattj , the model needs to compute the probability of y as follows, respectively:
p(y|x) = pM (prefi; y|pattj(x)) (1)
p(y|x) = pM (y|patti(x); prefi) (2)
Schick and Schütze (2021b) indicate that processing it with the decoder has a stronger impact on generations, and we apply it to BART and Pegasus, as in (1). However, this is not possible with T5 as encoding it with the decoder pushes the model to produce empty strings. With T5, we therefore encode it as part of the encoder (2). Modified training and test instances are used in different zero- and few-shot training configurations, namely UNSUPERVISED (3.3), FINE-TUNING (3.4) and ENSEMBLING (3.5). We synthesize the three methods in Figure 1.

heading: 3.3 Method 1: UNSUPERVISED
text: In a first zero-shot configuration, which we name UNSUPERVISED and we take as our baseline, we simply evaluate the model’s predictions without
any training, so no parameter update is performed, nor do we use any training instances. However, to have the prompts influence the model’s generations at inference, we modify each test instance with each one of the prompts, as explained in Section 3.1. We evaluate the model’s predictions for each prompt separately after modifying the test instances with that prompt through function fprompt, as detailed in Section 4.4. Moreover, we define a null prompt p0 for which both pref and patt are an empty string, resulting in the simple concatenation of x and the <MASK> token, and evaluate its generations.

heading: 3.4 Method 2: FINE-TUNING
text: In a second configuration, which we call FINETUNING, at training time we apply all the prompts p to every input through function fprompt obtaining a dataset T of all training instances X (where X can be empty). In addition, in order to avoid overfitting and ensure regularization, we also take a set of 1000 unlabeled instances U (for which the explanation y is not given), modify them with the patterns p ∈ P and have the untrained model M generate an output for each of them. We therefore obtain a synthetically generated dataset TFINE-TUNED, which we append to our training instances.
We use T to fine-tune M with teacher forcing, by minimizing the cross-entropy between the model’s prediction and the target sentences, obtaining a fine-tuned model MFINE-TUNED. In a zero-shot setting, only U is used for fine-tuning. In few-shot setting, we use training sets of increasing size. As in method 1 (3.3), at test time we assess the predictions of each pattern separately and of all patterns together using p0.

heading: 3.5 Method 3: ENSEMBLING
text: In the previous method we were able to assess each prompt separately at inference time. However, it may not always be possible to know which pattern works better for a certain task and poor performance of one prompt could hurt the overall performance of the fine-tuned model2. Moreover, in few-shot scenarios, models tend to overfit the training data or copy part of the original input, resulting
2We are aware that several methods for prompt search and optimization have been recently proposed, and our method would certainly benefit from better quality prompts. However, our aim is to mitigate the potential impact of bad prompts, while prompt search currently falls outside the scope of this study.
in poor quality of the generations. In this configuration, we then aim to generalize over all possible patterns given at training time, without having to choose a specific pattern at test time.
Therefore, we perform a form of knowledge distillation through prompt-ensembling by taking the fine-tuned model MFINE-TUNED and the unlabeled instances U = (u1, ..., un), and use the set of patterns in P = (p1, ..., pm) to generate a set of candidate outputs C = (y1, ..., ym) for each u ∈ U . Then, we modify the instances in U with the null pattern p0 and ask an untrained model M (that did not see any of the training examples) to assign a probability to each candidate generation in C given the modified inputs from U . To assign a final score to the generation, we take the exponentiated average of all the log-likelihood assigned by the untrained model across all patterns, and take the best scoring generation as our prediction.
By doing so, we obtain a new dataset TENSEMBLE, where inputs are u’s from U modified by p0 and y’s are the best ranking y’s from C according to M . The so obtained explanations should not be biased towards one particular pattern. Moreover, we empirically set a cutoff lower threshold at the bottom 20% of the instances ranked by their probability, so that low quality explanations are discarded.
We use this final dataset TENSEMBLE to fine-tune a final model MENSEMBLE with a procedure similar to method 2, but we only evaluate it using the null prompt as explained in Section 4.4.

heading: 4 Experiments

heading: 4.1 Pre-trained language models
text: We experiment on three different Transformerbased encoder-decoder language models: BART, Pegasus, and T5.
Bart (Lewis et al., 2020) uses a standard sequence to sequence architecture with a bidirectional encoder and a left-to-right decoder. It is pre-trained by firstly corrupting text with a noising function, then learning a model able to reconstruct the original text. BART was evaluated on several benchmarks and proved particularly suitable for generation tasks, a reason why we decided to employ it. In all experiments, we use the BART-large model.
Pegasus (Zhang et al., 2020) has a similar architecture to BART and trained in a self-supervised way by masking important sentences in text and have the model generate them as a single output
conditioned on the remaining sentences. Thus, its training objective is similarl to a summarization task. We use the PEGASUS-large implementation in all our experiments.
T5 (Raffel et al., 2020) is also an encoderdecoder model, however unlike the previous two it was pre-trained on a mix of NLP tasks prompted in a text-to-text format, where inputs and outputs are text strings, as opposed to BERT-style models. For this reason, it is particularly suitable for methods exploiting textual verbalizations to condition generations.

heading: 4.2 Dataset
text: For all our experiments, we use the e-SNLI dataset (Camburu et al., 2018), an extension of the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) enriched with crowd-sourced natural language explanations. The SNLI dataset is an influential dataset widely used for the task of Recognizing Textual Entailment (RTE) (Dagan et al., 2005): given two text fragments (called premise and hypothesis), the aim of RTE is deciding whether the premise entails, contradicts or neither entails nor contradicts the hypothesis, labelling the relationship between the two texts with an entailment, contradiction, or neutrality label. The SNLI dataset contains 570K premise-hypothesis pairs, evenly distributed across labels. E-SNLI contains an extra layer of information, represented by a crowd-sourced natural language explanations for each instance for the training, testing and development splits. An example is given below.
1 {"guid": "test-3", 2 "idx": "3", 3 "label": "NEUTRALITY" 4 "meta": {} 5 "explanations": ["the woman
could’ve been old rather than young"]
6 "premise": "’A woman with a green headscarf, blue shirt and a very big grin.’"
7 "hypothesis": ’The woman is young.’}
Few other datasets exist, such as the CoS-E dataset (Rajani et al., 2019) (an expansion of the CommonsenseQA (Talmor et al., 2019) dataset) that provide explanations based on commonsense. However, CoS-E’s explanations mainly focus on one single term or phrase as the given explanation refers to
one of the five possible answers to a commonsense question. Moreover, many times the open-ended explanations in this dataset are not formed as full sentences, while we aim to generate self-contained, linguistically complete explanations, so this dataset is not suitable for our experiments.
On the other hand, the task of recognizing textual entailment involves general reasoning as well as understanding some subtle facts about the language and the referents, while heavily relying on commonsense knowledge. For this reasons, many studies which aim to investigate the ability of a system to elicit implicit knowledge have turned onto this dataset, and we choose to experiment our methods on this challenging dataset enriched with natural language explanations.
For each training instance x (and for each corresponding output y), we obtain a set of four patterns. As can be seen from table in Section A.1, odd patterns present the explanation after the two sentences, while even patterns presents the explanation before the sentences, which are then introduced by the “Text:” string.

heading: 4.3 Experimental setup
text: For the FINE-TUNING and ENSEMBLING methods, we experiment with training sizes = (0, 10, 100, 500), test and unlabeled sizes = 1000, sampled uniformly across the original examples and labels. For UNSUPERVISED, we set T = 0 and do not perform any parameter update, we simply let the model generate given the modified inputs. For FINE-TUNING and ENSEMBLE we train on a single NVIDIA GeForce RTX GPU with 10GB RAM for 3 epochs, for about 15 hours. We optimize with Adafactor with square root learning rate decay, dropout rate = 0.1 and learning rate = 104, following (Schick and Schütze, 2021b). We train each model with 8 gradient accumulation steps using a batch size of 2 as our computing resources are limited, and generate using greedy decoding. For all models, we use the Pytorch Transformers library implementation (Wolf et al., 2019).

heading: 4.4 Evaluation
text: At test time, we evaluate each model on each pattern separately, by modifying each testing instance with one prompt at a time. Secondly, as in realworld scenarios it is not always possible to know in advance which patterns will perform better, we also evaluate each model on the null pattern p0, where the input precedes the masked sequence to-
ken and we use an empty task prefix. We evaluate using common metrics for generation, comparing the predicted output with the reference explanation, and thus report BLEU-1, ROUGE-1 and BERTScore. Kayser et al. (2021) present the most extensive, current study on the correlation between human judgments and generation, focusing in particular on the explanation generation task, and show that BERT-Score is the one that best matches human judgments, as also confirmed by other studies (Becker et al., 2021b). Therefore, we set BERTScore as our reference metric to assess the best model and method.

heading: 5 Results and Discussion
text: In Table 1 we report the results of our experiments for each of the considered models. T5 represents the best scoring model, which most benefits from the proposed methods, achieving a BERT-score of 91.23 both for the P0-FINE-TUNING method and for the ENSEMBLING method, as confirmed by both BLEU and ROUGE scores. Similarly, Pegasus benefits from the proposed methods, with a slight decrease on the ENSEMBLING method, which may be due to the error margin of the metrics.
Interestingly, both models have very low scores in both zero-shot configurations, indicating that the “fixed-prompt fine-tune” strategies may be particularly suitable for them, even without prompt optimization, as indicated by the low figures of the null prompt. On the other hand, BART displays a stronger baseline and is more sensitive to prompt design, as displayed by the decreasing values especially relevant on the p0-FINE-TUNING method. This indicates that when using BART, which was not specifically trained to accept prompts as inputs, prompt optimization may be a better strategy. On the other hand, for all three models the ENSEMBLING method is able to mitigate the negative effects of shallow prompt design.
While a clear benchmark for the explanation generation task does not yet exist, in Table 2 we report the results of related studies on the same task and dataset. Specifically, we compare our methods with studies using the same underlying model and with comparable settings and show that our methods achieve better results but with a significant reduction in training size. In particular, Marasovic et al. (2022) use T5 with 48 training examples and achieve a significantly lower Bert-Score compared to our T5-Ensembling method with 10
training examples. Becker et al. (2021b) achieve a slightly lower BERT-Score using BART, but with 18K training examples (which we compare with our BART-Ensemble method with 500 training examples). We provide further details on the related works in Section 6.
We also manually analyzed several generations and compared the different models and methods. We notice that in the unsupervised settings, models tend to hallucinate, while with zero-shots BART and PEGASUS tend to copy (part of) the input, while T5 often returns single words. In Table 3 we report an example of the generations produced by T5 under the different configurations. Notice that generations under settings T10 and T100 are already correct. However, the bigger the train size, the closer the generation to the reference. Finally, manual inspection also highlighted that in some cases the model learns to reproduce some patterns, such as the contradiction explanation pattern “X cannot Y and Z at the same time”, where X is the common referent to the sentences and Y and Z are the states described by the two sentences, respectively. However, the pattern repetition also characterises many human-generated sentences, a phenomenon that deserves further attention in future investigations if we aim at general, better natural language explanations.

heading: 6 Related work
text: Being a relatively recent area of interest, generation of free text explanations is not a well consolidated task. Particularly, evaluations metrics are still being discussed (Golovneva et al., 2022) (Wiegreffe et al., 2022), attempting both to capture the explanation informativeness and to improve the correlation toward human judgements. Here we report related works which are most focused on the e-SNLI dataset, being more comparable with our work (see Table 2).
Generating explanation with implicit knowledge has traditionally been addressed either by constraining generations with general knowledge paths (Ribeiro et al., 2020), by fine-tuning on specific or general knowledge datasets (Fatema Rajani et al., 2019), or with a combination of both methods (Becker et al., 2021a).
Camburu et al. (2018) train four different models with the aim to generate an explanation given only the hypothesis, generate an explanation without knowing the label, jointly predict a label and
generate an explanation for the predicted label, and generate an explanation and then predict the label. Their work uses straightforward recurrent neural network architectures so it is does not achieve state-of-the-art results, but it establishes a strong baseline.
Becker et al. (2021a) generate implicit knowledge connecting sentences in text, similarly to Camburu et al. (2018). They perform fine-tuning on corpora enriched with implicit information, by constraining them with relevant concepts and connecting commonsense knowledge paths, combining data augmentation and graph-to-text methods.
Marasovic et al. (2022) both present FEB, a standardized collection of four existing Englishlanguage datasets and associated metrics, and results based on template-based prompting. In our work we show that specific prompting design for the e-SNLI task results in significant improvements with respect to more general purposes prompts.
Li et al. (2022), based on the intuition that explanation generated through single-pass prompting often lacks sufficiency and conciseness, propose a two-step approach where the first-pass output from the pretrained language model is polished, and then regenerated retaining the information that supports the contents being explained.
Ye et al. (2022) show that both the computation trace (the way the explanation is decomposed) and the natural language of the prompt, contribute to the effectiveness of explanations. According to this finding they propose automatic prompt selection that focus on prompt diversity, rather than complementarity only.

heading: 7 Conclusion
text: In this work, we argued that providing explanations is often a process of eliciting implicit knowledge. We proposed a general methodology to generate commonsense explanations from pairs of semantically related sentences, taking advantage of both prompting applied to large pre-trained language models and few-shot learning techniques. Experiments run on the e-SNLI dataset show that the proposed methods achieve SOTA results on the explanation generation task, with a substantial reduction of labelled data. The obtained results open new perspective for a number of tasks based on eliciting implicit knowledge.

heading: 8 Limitations
text: Although we showed significant improvements in explanation generation using prompt-based few-
shot learning, our work still has some limitations. First, we experimented only on the e-SNLI dataset: although e-SNLI is a reference for the task, it would be interesting to extend the proposed methodology
to other datasets with natural language explanations (see Wiegreffe and Marasović (2021) for an extensive review).
Second, we did not attempt to automatic prompt optimization: although this may bring further minor improvements, we decided to leave optimization to a next step, as it does not change the core contribution of our work.
Third, we believe there is an intrinsic limitation in comparing our results with SOTA, as there is not a clear consensus on which metric is to be taken as the reference metric for benchmarking, along with the fact that measures sometimes disagree on scoring one system better than another. We hope that in the future this task and its evaluation will consolidate into a shared benchmark.
Finally, as for our our use of e-SNLI, we are assuming that for all sentence pairs in the dataset there is an implicit explanation of the semantic relation between the sentences. Under this assumption we always generate an explanation, even when the explanation is already explicit in one of the sentences. We think that a better capacity to detect those cases would bring relevant insight to our approach.

heading: Acknowledgements
text: This work has been partially supported by the chistera ANTIDOTE Project.

heading: A Appendix
text: A.1 Prompt design
A.2 System diagrams

heading: Future Work
text: We hope that in the future this task and its evaluation will consolidate into a shared benchmark. Finally, as for our our use of e-SNLI, we are assuming that for all sentence pairs in the dataset there is an implicit explanation of the semantic relation between the sentences. Under this assumption we always generate an explanation, even when the explanation is already explicit in one of the sentences. We think that a better capacity to detect those cases would bring relevant insight to our approach