Deep Natural Language Feature Learning for Interpretable Prediction
Authors: Felipe Urrutia
Cristian Buc Calderon
Valentin Barriere
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Reasoning
Explanability
BERT

Abstract: We propose a general method to break down a main complex task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. Our method allows for representing each example by a vector consisting of the answers to these questions. We call this representation Natural Language Learned Features (NLLF). 
NLLF is generated by a small transformer language model (e.g., BERT) that has been trained in a Natural Language Inference (NLI) fashion, using weak labels automatically obtained from a Large Language Model (LLM). We show that the LLM normally struggles for the main task using in-context learning, but can handle these easiest subtasks and produce useful weak labels to train a BERT. 
The NLI-like training of the BERT allows for tackling zero-shot inference with any binary question, and not necessarily the ones seen during the training.
We show that this NLLF vector not only helps to reach better performances by enhancing any classifier, but that it can be used as input of an easy-to-interpret machine learning model like a decision tree. This decision tree is interpretable but also reaches high performances, surpassing those of a pre-trained transformer in some cases.
We have successfully applied this method to two completely different tasks: detecting incoherence in students' answers to open-ended mathematics exam questions, and screening abstracts for a systematic literature review of scientific papers on climate change and agroecology.

Deep Natural Language Feature Learning for Interpretable Prediction
Felipe Urrutial2, Cristian Buc!, Valentin Barriere 'Centro Nacional de Inteligencia Artificial, Macul, Chile
DDpartmment of Computer Science, Universidad de Chile, Santiago, Chile furrutia@dim, uchile.ch. name. castname@cenia..
Abstract
justice (Dass et al., 2022), or finances (Heaton al., 2017). This pervasiveness is intrinsically linked the huge success of Deep Learning models, which have shown to scale particularly well to complex decision-making problems. However, this success comes at cost. To solve complex (high-stakes) decisions, these models develop inner hidden rep- resentations which are hard to understand and inter- pret even by researchers implementing the models (Castelvecchi, 2016).
We propose a general method to break down main complex task into set of intermediary easier sub-tasks, which are formulated natu- ral language binary questions related the final target task. Our method allows for repre- senting each example by a vector consisting of the answers to these questions. We call this rep- resentation Natural Language Learned Features (NLLF). NLLF is generated by small trans- former language model (e.g.. BERT) that has been trained a Natural Language Inference (NI.I) fashion, using weak labels automatically obtained from Large Language Model (LLM). We show that LIM normally struggles the main task using in-context learning, but can handle these casiest subtasks and produce use- ful weak labels to train BERT. The NLI-like training of the BERT allows for tackling zero- shot inference with any binary question, and not necessarily the ones seen during the train- ing. We show that this NLLF vector not only helps to reach better performances by enhanc- ing any classifier, but that can be used input of an casy-to-interpret machine learning model like decision tree. This decision tree is in- terpretable but also reaches high performances, surpassing those of a pre-trained transformer in some cases. We have successfully applied this method to two completely different tasks: detecting incoherence in students' answers to open-ended mathematics exam questions, and screening abstracts for systematic literature review scientific papers on climate change and agroecology.
Regulations like the European Union's "Right to Explanation" (Goodman and Flaxman, 2017) or Russell al. (2015) requirements for safe and se- cure AT in safety-critical tasks fostered advances explainability. Therefore, there has been a grow- ing interest in developing techniques allowing to explain the mechanisms subtending these so-called "black-box" models (Guidotti et al., 2018; Fel al., 2022). Notably, this endeavor has given rise an entire research field called Explainable AI (XAI), which focuses on providing iamanninterpretabbl in- formation on the models' behavior (Gunning et al., 2019; Arrieta et al., 2020).
Explainable Deep Learning in Natural Language Processing (NLP) can be decomposed in two cat- egories: representational and practical. Represen- tational XAI in NLP focuses on understanding the underlying structure of the representations. For in- stance, studies have shown that Transformer-base architectures develop abstract symbolic, or com- positional, representations (Lovering and Pavlick. 2022; Li et al., 2022b). Similarly, it has been shown that conceptual knowledge can have sparse repre- sentations, which can thus be located and edited to induce different predictions (Meng et al., 2022). Practical methods can analyze the outputs of the models, for example when perturbing the inputs (Tulio Ribeiro et al., 2016; Fel al.. 2023; Lund- berg and Lee, 2017). Recently, practical XAT in NLP focuses on prompting to increase explainabil- ity. For instance, chain-of-thone -fttuughtt prompting is a
1 Introduction and Related Work
The use of AI models is becoming increasingly pervasive in today's society. As such, applica- tions of these models have seen the light in do- mains where decisions can have dramatic conse- quences such as healthcare (Norgeot et al., 2019),
'oode available at https: /gt t.ub. com/ furrutiav/nl 1 f--mmlp----- 2-023

---

One Main Complex Task
abstract a paper tesessing the impael agroecolugical practimes climate change milligation or climate change adaptation?
Binary Subtask Questions
Linguistic Features
Decision Tree Path
Docs the article analvse agruecology affects nitrogen
grrass
tillage
practices
Does article address the relationchip between betwemmagrrooo agroncological practioss and dimette change?? A: You
dynamics?
grannouse
ngrioslltre
Does the study specifirally evaliaate grenhouse gas emissions?
Article Abstract
convent
...
Contribation of crop residue, soil, and fertilizer nltrogen to nitrous oxide emissions varies with long-term crop
conservation
mitigation
The text has a word with the prefix comsent
rotation and tillage
Dawn the article discuss ecrtified organic production?
mode]
emission'
organic
Agriculture importur contribator to
No
N20 emissions potent greenhouse with ligh peaks occuriig wlen soil mineral aitrogge (N) is high leg. atte mineraltattion of organic N and N fertilizer applcation) Nitrogen dynamics in soil and conuqqeenly NaO emissiona are affieted by crop and soil management practices leg. crop rotation and tillage). an effect mostly assaaeed in the litesture through compasisons of tubil NaO emission. Honse informatioo is scarce on the effess of these management practiecs on suesile It sources aflecting N2O emissious (i.., N fertilizer, soll, above and belowgrondd crop ressiues) a knowledge zap exploced in this suuy with the use of N-15 tracess. The solope appeoach quabled... ... (more)
crop
manggement prattices
Q: Dacs the article analyse how agroecology affects uitrogen
NLL Fintures
Expert Fiothons
dynamios? A: Yes
Feature Extraction:
OOO
Interpretable model
Does the article agroecological practices' impact climate change? A:
Explonation
Final Label:
Inchude
Figure I: Overview of the proposed system: Extraction of Natural Language Learned Features and Expert Features in order to understand the decision process of an interpretable model for complex task solving.
method that implements a sequence of interposed NLP steps leading to a final answer (Wei et al., 2022; Wang et al., 2022: Zhao et al., 2023; Lyu et al., 2023). This method has the advantage to pro- vide insights on the logical reasoning steps behind a model's behavior, and thus allows to understand (at a higher level) the predictive success or failure of LLM (Zhou et al., 2022; Diao et al., 2023; Wang al., 2022).
2022). Interpretability relies on the possibility know exactly why the model is making a predic- tion because they are inherent to the prediction and faithful to what the model actually computes. However, methods like CoT which should be in- terpretable (because outputting explanations with their predictions) have not always shown to give faithful explanations (Radhakrishnan et al., 2023). It is also arguable that our method relies on learned representations from a BERT, which decreases its overall interpretability.
Although advances in XAI within the NLP do- main have offered interesting insights on the under- lying mechanisms and representational structures of LLM, there has been recent push to solely focus on interpretable models for high-stakes de- cisions (Rudin, 2019). This push is motivated by dramatic errors made by these models in real life situations, such as asscssing criminal risk at large scale (Angwin et al., 2016) or incorrectly denying bail for criminals (Wexler, 2017). The main ratio- nale behind this idea holds in that there will always be certain level of error (or information loss) as- sociated to the explanations of black-bor box models. Indeed, these explanations can, by definition, only partially incorporate information of the model's reasoning process.
Motivation and Contributions In this work, we aim to reconcile the abilities of black-box LLM and interpretable machine learning (ML) models, by leveraging the impressive zero-shot abilities of LLM, instructed (Peng et al., 2023; Ouyang et al., 2022; Chung et al., 2022) or not (Wei et al., 2021), to improve the performance of ML models in more complex tasks. One particularly stunning feature of LLM is compositional systematicity (Lake and Baroni, 2018; Bubeck et al., 2023): the ability to decompose concepts into their constituent parts that can be recombined to produce entirely novel con- cepts. Such compositional representations is at the basis of systematic generalization in novel contexts (Brown et al., 2020). For instance, experimental work has shown that LLM are zero-shot learners, and can further improve this skill when encouraged to reason sequentially (Kojima et al., 2022a).
Following Rudin (2019), we make the difference between an explainable black-box model and an in- terpretable white-box model. Explainability relies on algorithms aiming to explain the model predic- tions by showing cues to the user like LIME, or other ad-hoc methods (Fel et al., 2022; Colin et al.,
In our approach schematize in colors on Figure I, we leverage the ability of LLM to decompose

---

complex tasks in simpler sub-tasks, and use these sub-tasks with medium-size language model to create interpretable features (the Binary Subtask Questions (BSQ) and Natural Language Learned Features (NLLF) in green) and to improve the per- formance of ML classifiers. This classifier can be a simple interpretable model such as Decision Tree with readable decision path (in blue). The uniqueness of our work is combine the strength of LLM and the explainability of ML classifiers, as opposed to similar previous work which as mainly focused on leveraging LLM to increase the reason- ing abilities of smaller LM (Li et al., 2022a).
zero-shot ability of LLM in order label exam- ples regarding the binary subtasks. Next, subsec- tion 2.3 (in orange in Figure 3) contains details on how we train a BERT-like model (Devlin et al., 2018; Cañete et al., 2020) in a natural language inference (NLI) fashion to resolved those subtasks. Finally, subsections 2.4 and 2.5 (in blue in Figure 3) describe the process to generate interpretable representations and how integrate them in explainable model to solve the main task.
2.1 Lower-level Subtasks Generation
In this step, Questions (BSQs), which using LLM. This step not mandatory as a human practitioner could do it manually.
generating C Binary Subtask
This work makes three main contributions. First, compared with tfain-of-thoughhe methods, which are often computationally expensive and are effec- tive in solving certain types of reasoning problems, our approach is a computationally cheap and uni- versal solution. Indeed, it can be applied to any problem that can be reasonably decomposed in sim- pler tasks (see methods below). Second, we present a method that allows simpler and interpretable mod- cls to solve complex reasoning tasks; tasks which are usually outside the realm of solutions for these type of models. Third, we show that those inter- pretable models can surprisingly outperform state- of-the-art LLM models in reasoning-baan based classifi- cation tasks.
In order to identify subtasks of the main task, we randomly select small percentage Py of samples from the training set. With the help of LLM that we prompt using an instruction-based template (visible in Appendix E), generate set of 5 basic binary questions per sample useful solve the main task as shown Figure 2.
Lower-level Subtaskes Generation
Irputt
Output
Prompt
LLM
BBC
classify
Labe
Template
Fioe Bage
To demonstrate the usefulness of the proposed method we focus on two different languages tasks that requires high levels of reasoning, and can be decomposed subtasks with lower difficulty levels of reasoning, in English and in Spanish. We aim to show the generalization power of our method by (i) classifying the coherence of fourth grade students' answers to mathematical ques- tions (Urrutia Vargas and Araya, 2023; Urrutia and Araya, 2023) (IAD: Incoherent Answer Detection), (ii) classifying scientific papers regarding a topic of interest in the context of a systematic literature re- view about agroecology and climate change (SAC:
<T Iresce clasee fy l/T>
Give me 5 attributes, in yes/no question format, general and verifiable the text to check that the text to elossify is <ll. [Tabel] /L.
Sulraskk Geseraation
Puppreent selanct
Sabcaek
Questions
Figure 2: Automatic generation of BSQs using prompt templates with an LLM and manual grouping.
This process lead to a large set of Q binary ques- tions obtained from the Pg subpart of the dataset. In order to reduce the redundancy in this large set, similar questions were manually grouped together into C groups, and each group was reformulated into a unique general question and verifiable yes/no binary questions. This process leaves us with a set of questions.
Scientific Abstract Classification.
2 Methods
This section describes the different parts of the pro- posed method, which are summarized in Figures 2 and 3. Subsection 2.1 shows how utilize an instructed LLM to generate natural language bi- nary subtasks questions that can be useful solve a more complex task. Subsection 2.2 (in green in Figure 3) explains the process to leverage the
2.2 Zero-shot Subtasks Labeling with an LLM
In this phase, we generate labels on some of the training examples regarding the C' subtasks. In or- der to achieve this, we are leveraging the zero-shot

---

Natural Language Learned Features Generator Training
Natural Language Learned Features for Interpretable Prediction
Zero-shot Subtask Labelling with an LLM
Hinary Subtask Questions
Sed ofC quasalionte to moofre the took
Yes
No
Impur
Ourput
dissilyy
Small snbbrt of the Thilsser
BSQq Arg.
Rephinability
NLLPG Toanng &
350-
Id0
Text to classity
NLLFG Preacn @ $
liff
peroeer Small subset
Toxt to classi
Interpretable NLLFeatures
Interpeotuile
BSQ Labels
sclesiion
Figure 3: Full process of subtask labelisation, NLLFG training, NLLF generation and integration.
learning capacity of LLM solve simple tasks. By prompting an LLM with samples from the training dataset with a low-level subtask binary question, we are able to annotate each example according to the C' binary subtasks. In the end, we obtain synthetics labels on a limited percentage of the training set, which totals to C pt percents of the initial training dataset size. The template of the prompts can be seen in Appendix E.
2.4 Natural Language Learned Feature Generation
BSQ augmentation Only C' question were used for the training of the NLLFG because of budget cost, but way more might be used represent a sample as the NLLFG can generate an answer to a question never seen during training, a zero- shot way. Hence, we augment the set of questions by adding new questions from the expert domain: we used all the available BSQs before the manual clustering, translation of expert linguistics features into natural language, paraphrases of the C' BSQs and human-made BSQs. This leads to a set of C+ questions.
2.3 Natural Language Learned Feature Generator Training
In this stage, use the examples tagged with low-level weak labels obtained through large model, fine-tune smaller BERT-like trans- former modef'in in a natural language inference (NLI) way. Given that the BSQs can be expressed and in- serted in natural language inside the transformer, an NLI-type of inference means that the text to classify and the BSQ are seen as premise and hypothesis. It has two strong advantages: (i) it leverages the semantic knowledge encoded during pre- treiirinnin understand the label, (ii) it can be applied in zero-shot manner using new labels formulated as natural language binary question ( (Yin et al., 2019; Vamvas and Sennrich, 2020; Barriere and Jacquet, 2022).
NLLF construction For every example, we use the NLLFG with the C, binary questions in order to create a vector of NLLF. The vector of NLLF was constructed by taking the sigmoid of the logit instead of the softmax, in order to keep the infor- mation about the confidence of the prediction for both classes (i.e., sometimes the predictions far away from the decision hyperplan for both classes). Which means that for each BSQ, there are two values between 0 and I: one representing the prob- ability of the Yes answer, and one representing the probability of the No answer. This gave us a NLLF of size 2C..
In the end, this model is able to predict, for every pair of sample associated with any binary question, if the answer to the question is yes or no. We call this model Natural Language Learned Feature Generator (NLLFG). More details are available in Appendix A.
Feature selection Finally, we ensure to select only the most effective features by removing the predicting useful representations with fea- ture selection. We employed genetic algorithm for feature selection (Fortin et al., 2012), using deci- sion trees as backbones. We executed the algorithm in a 15-fold cross-validatie setting and selected the features that were selected at least one third of
'which has 1000 times less parameters than the I.LM used heforehand

---

the times. In other words, we removed the BSQs that lead unsure answers from the model.
Incoherent Answer Detection To evaluate our method on special domain text, we foeus on the task of coherence detection students' answers to an open-ended mathematical test questions. We used the dataset of Urrutia Vargas and Araya (2023) composed of 15,435 answers to 700+ different open-ended questions collected using the online e-learning platform Conectaldeas. The answers' (in)coherence were manually annotated by several teachers. The test set only contained examples that were annotated similarly by least three annota- tors. Both the train and test datasets are imbalanced between the classes, with respectively 13.3% and 20. 10.1% of incoherent examples.
2.5 NLLF-boosted Explainable Model
the NLLF vector generated from sample is con- trolled representation, which can then be added to augment any existing classifier. In this work, given that we are focusing on interpretable mod- elization, we chose use as input of a Decision Tree (Breiman et al., 1984). We also enhanced the NLLF representation with Expert Features (EF) de- rived from linguistic patterns, which are known to be very precise but generalize poorly. In this way. we take the best of both world with an hybrid model benefiting from the robustness and high accuracy of deep transformers well the fine-gri -raigna precision of linguistic rules (Barriere, 2017).
3.2 Baselines and proposed methods
In this section, we deseribe the baseline models that evaluated in our experiments.
3 Experiment and Results
Vanilla ChatGPT Because this model is known to have good performances zero- and few-shot inference, we evaluate it in 0/4-shot prompt strate- gies.
3.1 Datasets
We used two datasets in order to validate our model. First, we present dataset abstracts and titles of English scientific articles, labeled regarding their pertinence towards a systematic literature review on Climate Change and Agroecology. Second, we present a dataset of coherent and incoherent stu- dents answers open-ended questions of mathe- matical test.
CoT ChatGPT Chain-of-Thought has been shown to improve the performances of LLM for reasoning tasks. Hence, we enhance the model with this technique. We used the technique of Kojima et al. (2022b) for the zero-shot COT.
Scientific Abstract Classification To evaluate our method on complex text, we use dataset anno- tated the context of systematic literature review about the impact of agroecological practices on cli- mate change mitigation and climate change adapta- tion. More than 15k articles were retrieved from the Web of Science database using extensive set of keywords related to Agroccology and Climate Change. The first 2,000 articles were tagged by two annotators, using the title and abstract of the article, regarding whether or not the article was relevant for the systematic literature review. there was no consensus between the two annotators, third annotator was called to arbitrate, which happened the case 14% of the time. The articles with missing abstracts removed from this study, which left total of 1,983 articles, from which 50.1 % labeled included and 49.9 labeled excluded.
Self-ask ChatGPT Self-ask (Press al., 2023) enhances compositional reasoning by explicitly for- mulating and answering follow-up questions before addressing the initial query to significantly reduces the compositionality gap.
We evaluate different models based on a BERT transformers (Devlin et al., 2018), process- ing the raw text. The Vanilla version connects one fully-connected layer after the [CLS] token. The other versions concatenate (previously extracted) expert features and NLLF with the [CLS] represen- tation. For the IAD dataset, which is in Spanish, used the Spanish version of BERT called BETO (Cañete et al., 2020).
BERT
Decision Tree Decision trees were used plainable models, with low height and only inter- pretable features. We used the same features for BETO, plus added -fg-of-N-Grams (BoNG, variant of the Bag-of Bag-offWords; Harris, 1954) in order to model the text content.
'article to be published from the Agroecology re- search group of Sant" Anna di Pisa: attpp://i /www. santannapiss acienze-dee delle www.llepppanneggreeeeeeeee
cerce/

---

IAD
SAC
Model
Variant
Params
Explainability
Prec.
Rec.
F1
Prec.
Rec.
F1
0-shot
19.70
76.47
31.33
76.57
50.27
35.23
4-shots
X
24.80
90.44
38.92
66,92
51.66
38.92
0-shots CoT
/
23.14
84.56
36.33
44.93
46.31
41.59
ChatGPT
10t1
4-shots CoT
42.18
85.29
56.45
65.00
63.26
62.72
0-shots SA
21.29
82.35
33.84
63.65
55.15
48.13
4-shots SA
/
51.71
77.94
62.17
70.31
62.42
59.50
Vanilla
58.47
78.68
67.08
67.74
67.80
67.72
BERT
EF
~ 10
X
78,40
72.06
75.10
67.65
66.93
66.90
NLLF
X
67.10
76.47
71.48
68.97
68.98
68.75
NLLF+EF
X
80.49
72.79
76.45
73.66
73.61
73.63
BoNG
100.0
8.09
14.97
65.38
65.13
65.15
EF
83.33
66.18
73.77
68.18
66.49
64.95
Decision Tree
NLLF
75.00
44.12
55.56
62.41
62.43
62.25
-0
NLLF+EF
85,22
72.06
78.09
68.02
68.01
67.75
NLLF+BONG
82.28
47.79
60.47
66.21
66.26
66.20
NLLF+EF+BONG
85.22
72.06
78.09
68.17
67.43
67.41
Table Precision, Recall and Fl-score of all the configurations and models for Incoherent Answer Detection (IAD);
and (Macro) Precision, Recall and Fl-score of all the configurations and models for the Scientific Abstract Classification (SAC). Using Expert Features (EF), NLLF, and Bag-of-N-Grams (BoNG).
3.3
Experimental Protocol
3.3.3 Method parameters
Scientific Abstract Classification We used a Pg of 1.3% (21 examples) to generate BSQ and
3.3.1 Dataset splitting
Scientific Abstract Classification We randomly split the data into training, validation and a test sets following the proportion 70/10/20.
of 10% to train the NLLFG. C' was set up to 13 questions, and the number of questions for the gen- eration C, was 109 questions.
Incoherent Answer Detection We train the clas- sifiers on the 2019 data and tested on sample of 677 perfect-labeled answers from the 2017 dataset. The study used the different open-ended questions and answers, but the same definition of incohcrence throughout, despite different students and teachers in each year.
Incoherent Answer Detection We used a Py of .15% (21 examples) to generate BSQ and a of
10% to train the NLLFG. C' was set up to 10 ques- tions, and the number of questions for the genera- tion C. obtained was 66.
3.3.4 Implementation
The formers library (Wolf et al., 2019) was used to access the pre-trained model and train our models. We used BERT and BETO" as backbones for the NLLFG. The decision trees were trained using scikit-learn (Pedregosa et al., 2012). We used the 03/23/23 version of ChatGPT (Ouyang et al., 2022) LLM. Other details can be found in Appendix and C.
NLLFG training For each task, we randomly split 90% of the weakly labeled examples into a training set and keep the last 10% for the model validation.
3.3.2 Evaluation Metrics
Scientific Abstract Classification As both the classes are important, we have adopted the macro- averaged precision, recall, and Fi-score metrics as our evaluation criteria.
3.4 Results
The results are visible in Table for respectively the IAD task and the SAC task. The last six columns detail the precision, recall and Fi-score of the "incoherent" class for the IAD task, and the macro precision, recall and Fi-score for the SAC task. In both cases, the best results overall are
Incoherent Answer Detection To maintain con- sistency with the aforementioned work (Urru- tia Vargas and Araya, 2023; Urrutia and Araya, 2023), we have adopted precision, recall, and F1- score metrics for the positive class (incoherent) as our evaluation criteria.
-bert-base-cae bert baat-base-- caase-spantie spanisinnsee spanieh-wwn-caed
and

---

the ones from models enhanced by NLLF and EF, reach the Fl-scores of 78.09% and 73.63%.
NLLF generated by the BERT-like model, and also the weak labels by the LLM.
ChatGPT For the SAC task, ChatGPT models display high precision, but overall low recall, and very low Fi-score coming from low Fi-score for the exclude task, except for the 4-shot CoT /SAC versions that reach a Fl-score of 62.72% / 59.50%. This is because ChatGPT tends to catego- rize almost all of the articles as included. For the IAD task, ChatGPT models display poor precision and Fi-score metrics, but overall high recall, mean- ing these models tend to categorize most of the answers as incoherent. Moreover, as expected, the 4-shots SAC variant outperform all other Chat- GPT variants in F1 Fl-score (62.17%).
NLLFG Training We analyze the performance of the NLLFG the validation set during it's train- ing in order quantify how good NLI-like BERT transformer can reproduce the weak labels of an LLM. From the results shown in Table 2, we can see that the performances for the LAD task are much higher than the ones of the SAC task.
Task
Label
Prec.
Rec.
Fl
Acc.
Yes
74
86
79
SAC
73
No
71
52
60
Yes
97
96
97
IAD
95
No
92
94
93
BERT BERT-like models display the high met- rics across the board in precision, recall and F1- score. In particular, the models incorporating both NLLF and EF obtains the highest overall F1- score (76.45% and 73.63%) in both tasks. Surpris- ingly, enhancing the transformer with NLLF (resp. EF) provokes drop in the performances for the IAD (resp. SAC) task. Note however, that BERT models belong to the class of models that are not explainable.
Table NLLFG performance on the validation set during the weak label training.
Nevertheless, we can see that the NLLFG tend to overelassify the examples in the Yes class. This due to the skewness the weak labels distribu- tion, which is mainly composed of Yes labels. The distribution of Yes/No labels towards with regard to each question is visible in 4.
Finally, within each task the Fl-scores pretty similar between each of the classes: .69 and .70 for the SAC task and .97 and .93 for the IAD task.
Decision Tree The DT models also reached high performances across the board (with the exception of the BoNG variant for the IAD task). Specifi- cally, the variant using NLLF+EF displays highest Fl-score (78.09% and 67.75%) in that model class, and it is notable that adding BoNG features does not improve the performances. The DT models are simple and fully interpretable, and significantly outperforms LLM like ChatGPT, while reach- ing performance metrics competitive with deep learning (black-box) model. This approach pro- vides interpretable steps to explain decision making within the tree (see Appendix H). The DTs using EF have very competitive results. Even though it is not relying on deep neural nets, it needs many complex handerafted features coming from expert knowledge (Table 9).
Validation by an Expert Here we analyze how accurate were the NLLF generated by the BERT- like model, and also the weak labels by the LLM. We took 100 examples from the validation used to train the NLLFG, and asked an expert to manu- ally label them regarding the labels of a BSQ. We compare the labeling of the expert with the outputs of the NLLFG and CharGPT models, using classi- cal classification metries such as precision, recall and FI F1-score. We focus only on the SAC task has just saw carlier that is the most challenging for the NLLFG during its training.
The results for both the tasks are available Table The LLM obtain a better Fl-score than the smaller transformer model, which was expected. It is interesting to note that the accuracy of the NLLFG model is of 0.68, which is not its accuracy on the weak labels validation set times the accuracy of the LLM (0.70 0.78) that is 0.55. This suggests that the NLLFG compensate some errors of the LLM.
4
Model Analysis
4.1 NLLF Accuracies
We quantify the error of the output of the decision tree using classical metrics, but not the error on the input of the tree, which is the error when creating the NLLF Here we analyze how accurate were the
Another important details regarding the propaga- tion of the NLLF errors in the tree, is that the tree

---

Number of weak labels per BSQ
1. Does the article assess the impact of agroecology on nitrogen dynemics?
Number weak labels
2. Does the article discuss the impact of methane
140
(CH4) emissions?
Does the article discuss the role of
120
climate resilience agricultura systems?
article
limitations, mnallenges. potential
100
article
specific geographic
80
Does article address pollcy implications?
Does the article cover climate change adaptation?
8. Does the article assess agroecological practices impact climate change?
9. Does the article discuss agroecological practices?
10. Dees the article focus primarily en climete change
20
mitigacien?
11. Dees the study specifically evaluate greenhause gas emissions?
12. Does the article comprehensively cover climate change and environmental aspects?
13
BSQ
13. Does the article target stakeholders??
Figure 4: Weak label distributions (Yes/No) for each binary-subtask question of the SAC task.
is not using as input class label but the probability
design, but just with some keywords spotting using regular expressions.
of each label, which contains more information.
Causality In addition to provide interpretability, our method tends foster causal learning. Indeed, by allowing the user to directly write the features to use as input, this method prevents the model to rely too heavily on latent correlational patterns that are specifically associated to certain classes (Gilpin et al., 2018; Angwin et al., 2016). Never- theless, feature selection still relies on data distribu- tion which makes the system not completely causal even though tends
Model
Label
Prec.
Rec,
Acc.
Yes
71
79
ChatGPT
No
88
77
Yes
60
96
74
NLLFG
68
No
92
43
59
Table 3: NLLFG and ChatGPT performances on set 100 examples annotated manually by expert.
4.2 Features
4.3 Path visualization
Decision Tree Selected Features The DTs com- bining distinct features (see last three rows of Table 1) are free to select the features they deem best to solve the main decision-making task. For the IAD task, the decision tree combining NLLF + EF considers 25 features of which 14 are NLLF and 11 are EF. An exemplar is shown in Figure 15 on the Appendix, where the tree starts by checking for less than 3 tokens and the presence of pronouns. Otherwise, checks whether the answer provides evidence or reasons, vowels, binary words, of calculations, and adequate knowledge of the topic raised the question. For the SAC task, the de- cision tree combining NLLF + EF considers 22 features of which 14 are NLLF and are EF.
The possible path visible in Figure 15 for the IAD task and in Figure 16 for the SAC task. All the possible paths are composed of mixed type of features with EF and NLLF.
For the SAC task, we can see that the first de- cision derives from the answer to "Does the ab- stract address the relationship between agroecolog- ical practices and climate change?" which allow to coarsely separate the samples between the two classes. Then, if the prefix "convent" is contained in the text, 5.6 times more probable (158/28) that the text comes from include class according to the GINI value. Examples of decisions with their associated paths in the tree are shown in Figures 13 and 14.
Correlation with Main Tasks Labels In the TAD task, we observed that some classes correlated with some EFs more strongly than with NLLFs, due meticulous feature design versus intuitive BSQ design, respectively. However, when looking at the results of the SAC task, it is possible to vali- date that our approach functioning even to tasks that lack a known powerful, hand-crac cand-crafted feature
Conclusion and Future Work
We proposed new method to leverage low-level reasoning knowledge related more complex task from large language model, and integrate this into smaller transformer. The transformer has been trained a way that it can be used for zero- shot inference with any low-level reasoning having

---

the task formulated in natural language. This al- lows the practitioners to formulate easily their own features related about the task. The Natural Lan- guage Learned Feature vector can then be used as a representation in any other classifier. We show that it is easy to train an interpretable model like Decision Tree, leading to both competitive results and interpretabiliity. Our method can be applied any predictive task using text as input. Future work should focus on investigating the potential impacts of this approach real-world educational settings, and especially inspect the preferences of the practitioners regarding different explainability methods in order to help them taking complex de- cision (Jesus et al., 2021).
Ethics Statement
This work is in compliance with the ACL Ethics Policy as it allows to create models that might be more interpretable and more causal.
Acknowledgments
The authors would like to thank Angelica Marchetti for her useful help on creating the expert features for the SAC dataset, and for the manual assess- ment of the weak labels in Table 3. This work was funded by National Center for Artificial Intel- ligence CENIA FB210017, Basal ANID.
References
Angwin, Larson, S. Mattu, and L. Kirchner. 2016. Machine bias.
Limitations
Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Garcfa, Sergio Gil-López. Daniel Molina, Richard Benjamins, et al. 2020. Ex- plainable artificial intelligence (xai): Concepts, tax- onomies, opportunities and challenges toward respon- sible ai. Information fusion, 38:82-115.
Although, on paper, our method is universal, we need to show that our results can generalize to other tasks where LLM (such as ChatGPT) struggle in their reasoning process, e.g., theory of mind tasks (Ullman, 2023). Moreover, our approach has been demonstrated on binary classification, and it re- mains to demonstrate that our approach can scale well more categories. Otherwise, more com- plex tasks like multi-hop reasoning would be a late target for our system, as a simple classifier can- not solve this as it is, which would require many adaptations.
Valentin Barriere. 2017. Hybrid Models for Opinion Analysis in Speech Interactions. In ICMI, pages 647- 651.
Valentin Barriere and Guillaume Jacquet. 2022. CoFE New Dataset of Multilinugl Multi-targe target Stance Classification from an Online European Par- ticipatory Democracy Platform. AACL-IJCNLP.
Despite the fact that obtained promising re- sults with our approach, the performance of both BERT and the decision tree using NLLF alone was not exceptional. This may be affected by the perfor- mance of NLLFG. In particular, we used a limited set of examples to train our NLLFG. is possi- ble that training with a large set of answers and BSQs, or using a prompt-based approach (Schick and Schütze, 2022) useful in few-shot setting, may improve the results. Especially, have also seen during our experiments that the number of exam- ples shown to the NILFG during training was correlated with its performances, for this reason we would like to monitor the performances of the NLLFG when trained with more weak labels from the LLM.
Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone. 1984. Classification and regression trees. wadsworth int. Group, 37(15):237-251.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint 12X1v22303.12712.
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou- Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish Pre- Fre-Trained BERT Model and Evaluation Data. In PML4DC at ICLR 2020.
Finally, we claim that choosing the features fos- ters causality but without rigorous experimentation. This could be tested by using a dataset with a de- liberately inflated bias (Reif and Schwartz, 2023).
Davide Castelvecchi. 2016. Can we open the black box of ai? Nature News, 538(7623):20.

---

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-ros, Marie Pellat, Kevin Robinson, Dasha Val- ter, Sharan Narang, Gaurav Mishra, Adams Yu, Vin- cent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H Chi, Jeff Dean, Jacob Devlin, Adam Robert, Denny Zhou, Quoc V Le, and Jason Wei. 2022. Scaling Instruction Fineiiund Language Models.
Bryce Goodman and Seth Flaxman. 2017. European union regulations on algorithmic decision making and "right to explanation". Al Magazine, 38(3):50- 57.
Riccardo Guidotti, Anna Monreale. Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51 42.
David Gunning, Mark Stefik, Jaesik Choi, Timo- thy Miller, Simone Suumff, and Guang-Zhong Yang. 2019. Xai Xai- Xai-explainable explainable artificial intelligence. Science robotics, 4(37)
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. 2022. What cannot predict, i do not un- derstand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems. 35:2832-2845.
7ellig S. Harris. 1954. Distributional Structure. Word. 10(2-3):146-11
James Heaton, Nick G Polson, and Jan Hendrik Witte. 2017. Deep learning for finance: deep portfolios. Applied Stochastic Models in Business and Industry, 33(1):3-12. 12.
Rahul Kumar Dass, Nick Petersen, Marisa Omori, Tamara Rice Lave, and Ubbo Visser. 2022. Detect-
ing racial inequalities in criminal justice: towards an equitable deep learning approach for generating and interpreting racial categories using mugshots. Al & SOCIETY, pages 1-22.
Sérgio Jesus. Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, and João Gama. 2021. How can choose an explainer?: An Application-groundee Evaluation of Post-hoc Ex- planations. FAccT 2021 Proceedings of the 2021 ACM Conference on Faimess, Accountability, and Transparency, pages 805-815.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: tre-training of deep bidirectional transformers for language understand- ing. arXiv preprint ar Xiv: 10.008805.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022a. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.1 11966.
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302.1233
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022b. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
Thomas Fel, Melanie Ducoffe, David Vigouroux. Remi Cadene, Mikael Capelle, Claire Nicodeme, and Thomas Serre. 2023. Don't 't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. In CVPR.
Brenden Lake and Marco Baroni. 2018 Generaliza- tion without systematicity: On the compositional skills of International conference on machine learning, pages 2873 2882. PMLR.
Thomas Fel, Lucas Hervier, David Vigouroux, An- tonin Poche, Justin Plakoo, Remi Cadene, Math- ieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. 2022. Xplique: A Deep Learning Explainability Toolbox. In Workshop on Explainable Artificial Intelligence for Computer Vision (XAI4CV), pages 5-8.
recurrent networks. In
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang. Jing Qian, Baolin Peng. Yi Mao, et al. 2022a. Explanations from large language models make small reasoners better. arxiv preprint arXiv:2210.06722
Felix Antoine Fortin, François Michel De Rainville, Mare Andfe Gardner, Mare Parizeau, and Chris- tian Gagne. 2012. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171-2111
Siyan Li, Riley Carlson, and Christopher Potts. 2022b. Systematicity in gpt-3's interpretation of novel english noun compounds. arXiv preprint arXiv:2210.0949)
Charles Lovering and Ellie Pavlick. 2022. Unit testing for concepts in neural networks. Transactions of the Association for Computational I. Linguistics, 10:1 10:1 1208.
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Ba- jwa, Michael Specter, and Lalana Kagal. 2018. Ex- plaining explanations: An overview of interpretabil- ity of machine learning. Proceedings 2018 IEEE 5th International Conference on Data Science and Advanced Analytics, DSAA 2018, pages 80-89.
Scott Lundberg and Su-In Lee, 2017. A Unified Ap- proach to Interpreting Model Predictions. In NIPS.

---

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thoughh reasoning. arXiv:2301.13377
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?" Ex- plaining the Predictions of Any Classifier. In KDD '16 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1 1144, San Francisco, California, USA.
arXiv preprint
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov, 2022. Locating and editing factual as- sociations in gpt. Advances Neural Information Processing Systems, 35:17359-1733 17372.
Tomer Ullman. 2023. Large language models fail trivial alterations to theory-of-mint tasks. arXiv preprint arXiv:2302.06399
Beau Norgeot, Benjamin S Glicksberg, and Atul Butte. 2019. A call for deep-learning healthcare. Nature medicine. 25(1):14-15. 15.
Felipe Urrutia and Roberto Araya. 2023. Who the Best Detective LLMs vs MLL in Detecting Inco- herent Fourth Grade Math Answers. arXiv.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Ameida, Car- roll Wainwright, Pamela Mishkin, C Mar, Jacob Hilton, Amanda Askell, Paul Christiano, Jam Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv, https://op.
Felipe Ignacio Urrutia Vargas and Roberto Araya. 2023. Automatic detection of incoherent written responses to open-ended mathematics questions of fourth graders. Preprint, pages 0-35.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel. Bertrand Thirion. Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg. Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Per- rot, and Édouard Duchesnay. 2012. Scikit-learn Machine Learning Python. Journal Machine Learning Research, 12:2825-2830.
Jannis Vamvas and Rico Sennrich. 2020. X-stance: A Multilingual Multi- Multi-Target Dataset for Stance Detec- tion. In SwissText.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed Chi, and Denny Zhou. 2022. Self-consistnncy im- proves chain of thought reasoning in language mod- els. arv preprint arXiv:2203.11 11171.
Baolin Peng, Chunyuan Li, Pengcheng He. Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu. Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2 1090052.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824- 24837.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukošiūte, al. 2023. Question decomposition im- proves the faithfulness of model generated reasoning. arXiv preprint arXiv:2307.11766 11768.
Rebecca Wexler. 2017. When a computer program keeps you in jail: How computers are harming crimi- nal justice. New York Times, 13.
Yuval Reif and Roy Schwartz. 2023. Fighting Bias with Bias: Promoting Model Robustness by Amplify- ing Dataset Biases. In Findings of ACL: ACL 2023, pages 13169-13189.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint 111vv11110.03771.
Cynthia Rudin. 2019, Stop explaining black box ma- chine learning models for high stakes decisions and interpretable models instead. Nature machine intelligence, 1 155:2206-215.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019, Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. EMNLP-UJNNPP 2019, pages 3914
Stuart Russell, Daniel Dewey, and Max Tegmark. 2015, Research priorities for robust and beneficial artificial intelligence. Al Magazine, 36(4):105-11
Timo Schick and Hinrich Schütze. 2022. True few- shot learning with prompts- real-world perspective. Transactions of Association Computational Linguistics, 10:716-731.
Hongyu Zhao, Kangrui Wang, Mo Yu, and Hongyuan Mei. 2023. Explicit planning helps language models in logical reasoning. arXiv preprint arXiv:2303.15714.

---

We show in Figures 5 and 6 the prompts used by the LLM order to create the BSQ and the associated label for respectively the IAD and the SAC tasks. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Le. and Fd Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint
Model
Label
Prec.
Rec.
Fl
Acc.
Yes
71
89
79
ChatGPT
78
No
88
68
77
Yes
60
96
74
NLLFG
68
1rXiv:2205.10625.
No
92
43
59
Yes
78
85
81
A NLI-like Training
CoT
No
81
72
76
79
The BSQ (a binary subtask question in natural language) were integrated inside the transformer input as follow:
Table 4: NILFG, ChatGPT and CoT performances on a set of 100 examples annotated manually by expert.
target metric focused on the minority class, accu- racy was deemed less pertinent).
[CLS] Input text
[SEP]
BSQ
[SEP]
We used a pre-trained BERT (resp. BETО) model in English (resp. Spanish) language for the SAC (resp. TAD) task. We fine-tuned one only model for all the subtasks, by integrating the sub- task as string and using the binary low-level sub- tasks labels Yes No as Entailment and Contra- diction in NLI. We trained the model for 7 epochs, using batch size of 16 and the Cross Entropy loss, with the Adam optimizer and a learning rate of 8 x 10
D Chain-of-Thought for Weak Labels
We used zero-shot CoT (Kojima et al., 2022b) to generate the weak labels. As shown in the Table 4, it allows to slightly improve the results, but as tested over 100 manually labeled examples only this not significant. Using OpenAl API comes with cost, the prompts and the outputs are longer (in average 0. 104 USD more per thousand queries).
Decision Tree Training
E Prompt Templates
we used the gini impurity as criterion to optimized and fixed the maximum depth of the tree 5 in or- der to keep comprehensible for the practitioners. During the learning phase, we used the Gini impurity as the criterion to optimized and fixed the maximum depth of the tree to 5 in order to keep it comprehensible for the practitioners. For the the model with BoNG only, we augmented the maximum depth 10 because of the sparsity of the features. The model with NLLF-BoNG had a minimum impurity of 1.2 x 10-3 while the other models had a minimum impurity decrease of zero.
The prompts to train the LLM to tackle the IAD- task are in Figures 7 and 8, for Vanilla and CoT., respectively. We also insert in the prompt the defi- nition of coherence we gave to the annotators. This process did not change the results.
On the other hand, the prompts to train the LLM to tackle the SAC task are in Figures 10 and 11, for Vanilla and CoT. respectively.
F Binary Questions
The BoNG were also implemented using scikit- learn, we choose 1000 the number of feature in order keep the dimension small and computed the if-idf for each a-gram.
We show in Table 6 the N binary questions that were obtained by several means, before the feature selection process for the IAD task; and Tables 7 and 8 for the SAC task.
C BERT fine-tuning
G Expert Features
We used 8 epochs, a batch size of 32 and the Cross Entropy loss, with the Adam optimizer and learn- ing rate of 1 5 and 5x 10-6 for the augmented transformers because of the concatenation of high- level features before the output layer. We selected the best model on the validation set using best ac- curacy for SAC and best loss for IAD (because the
We show in Table 9 the expert features by threc categories, called Traditional, Semantic and Con- textual features for the LAD task. On the other side, We show in Table 10 the expert features by two categories, called Keyword and Prefix features for the SAC task.

---

Template: Question generation
Question: <Q Question
/Q>.
Answer: <A
Answer
/A>.
Give me attributes (in yes/no question format) general and verifiable in the and/or question to verify that the answer is <L Label
/L> to the question.
Template:
Subtask labelisation
Question:
Question
Answer:
Answer
Tow- 1 leve question
Figure 5: Prompts used generate the subtasks binary questions, and create the labels regarding the subtasks questions for the IAD task, using LLM. Translated from Spanish.
Template: Question generation
You are expert agroecology and impact climate change. You have to decide which article is include or excluded in our literature review. Based on the example, give me very general (i.e., independent of the given example), verifiable and short attributes (in yes/no question format) in the abstract and/or title to verify that the following article is Label in the literature review of studies that assess the impact agroecological practices climate change mitigation or elimate change adaptation.
Title: Title
Abstract: Abstract
Template: Subtask labelisation
Title:
Title
Abstract: Abstract
Based on the abstract/title, Low level question
(answer "Yes" or "No")
Figure 6: Prompts used to generate the subtasks binary questions, and create the labels regarding the subtasks questions for the SAC task, using a LLM.
SYSTEM
You of incobereace in
open-ended questions from lourth grade math studentss will give different question mark questions
and answers, and you will say whether answer coherent or incoherent to question. between <Q/Q> and answers between <A /A2. You will turn the incoberent coherent label between <L./I.
USER
<Q Agustin wants to store some dice of cubic centimeter in box that is cm long, em wide, and em high. His brother that 15 diee will fit in the box. Is what Agustin's brother correct? Explain in own words. /Q>, <A wrong because 20 the box /Az.
1" shot
ASST.
Coherent /1,>
<Q Maria and husband yesterday cooked tortilla, they divided equal parts. 3/6. What fraction of the tortilla was left? /Q>, <A there was one piece of tortilla left and the last one was left and the tortilla /Az.
lushand
ASST:
Coherent /1,>
USER
<Q Claudio bas 12 pencils distribute in equal amounts in pencil do so, his friend Paula tells him that be should pot 6 pencils
have any Clandio's friend right? (Kxplain your words answer) /Q>. AA paula is not correct because she asks me in more pencil cases /A>.
3" shot
ASST.
/L>
USRR
<Q Francisco and Carilaa have collection of action figures of their favorite series, Francisco has 4/8 of the collection and Camila has 2/8, the total number of figures that Prancisco and Camila luve lose 5/8. How many figures of the collection do they have let?? (Write your answer appropriately//Q> <A IT'S GOOD /A>. <L
4" shot
ASST.
Incoherent/LL
< C Cueattion
/Q>.
Anawer
/A>,
Figure 7: Vanilla ChatGPT prompt templates for the IAD task. System/ Destem/Usse/Aaee User// (asst.) are the roles in the ChatGPT API. Translated from Spanish.

---

SYSTEM
You are incoherence in answers open-ended questions fourth grade students. will give you and answers, and then you will have to say whether the answer is coberent or incoberent to the question. We will mark questions between <Q /Q> can still be coberent, since incoberent means that the answer does not make sense with what is being asked and not necessarily with its Conrectness.
question
will in the incoherent coherent label between
Incorrect
USER
<Q Agustin wants to store some dice o cubic centimeter in box that is em long. em wide, and 2 cm high. His brother says 15 because 20 dice fil in the box /A>.
Agustin's brother says Explain your own words. /Q2. wrong
shot
ASST.
The
it is wrong' and further explains why. <l. Coherent
<Q Maria and husband yesterday cooleed tortilla, they divided equal parts. 2/6 hushand 3/6. What fraction of the tortilla was left? /Q=. <A there was one piece of tortilla left and the last one was left and the lortilla /A>,
g" shot
ASST.
The answer has fraction in words. So, <L. Coherent /T.>
USER
<Q Claudio has 12 pencila to distribute in equal amounts in 3 pencill cases, to do so, his friend Paula tells him that he should have any
put pencils in each answer) /Q>, <A paula is not correct becouse she asks me in more peneil cases /A>,
Claudio's frieud right? (Explain with your words
3 shot
ASST.
The says 'she not right' and explaios why. So, <L Cohereat /L>
<Q and Camila collection of action figures of Useir favorite Camila has 2/8, the total number figures that Francisco and Camila have they lose 5/8. How many figures of the collection do they have lett? (Write your answer appeopriately) /Q>. <A IT'S GOOD/AN /A>
Francisco lass 4/8 of the cullection and
4th
answer not give numerical representation. nor explain your answer. Incoherent /L>
USER 0-shot
<Q Cuestical
Q>.
<A Answer
/A> Let's think step by step. <L
Questica
QQ>
<A Answer
/A> Let's think step by step.
4-shol
Figure CoT ChatGPT prompt templates for IAD task. System/User/Asssien (asst) are the roles in the ChatGPT API. The phrase under the User role indicates which template is used in 0-shot or 4-shot format. Translated from Spanish.

---

SYSTEM
You are rater of incoherence in answers to open-ended questions from fourth grade math students, I will give you a different question and and then you will have whether the answer is coherent or incoherent to the question. We will mark questions between <Q /Q> and auswers between AA>..
USER
<Q Agustin wants to store some dice of 1 cubic centimeter in a box that is 5 cmn long, em wide, and 2 cm high, His brother says that 15 dice will in the box. Is what Agustin's ' brother says correct? Explain in your owa words. /Q², <A wrong because 2o dioe fit in the box /A>.
1" shot
ASST.
Yes. Follow-up: What is student expected to do when faced with question like Uhis? Intermediate response: The student is expected to give his or her opinion about the character's statement in the story and then justify or explain that opinion. Follow-up: What is the student's oginion in his or her response? Intermediate respiose: The student's oginion is that Augustine's brother's ss stalement is consistent. Follow-up:
the student also justify his or her respouse? Intermediate Yes, the student juslifies his response by explaining why he believes the statement is coberent. Fiual Answer: The student's response coherent.
USER
<Q Maria and husband yesterday cooked a tortilla, they divided into equal parts. Maria ale 2/6 and hushand 3/6. What fraction of the tortilla was leit? /Q=. <A there was one piece of tortilla left and the last one was left and the tortilla /A>,
2a shot
ASST.
Yes. Follow-up: What a student expected to do with question this? Intermediate response: The student expected to calculate the fraction of the toriilla that was left after Maria and ber busband ale their parts and provide his answer. Follow-up: Does student's answer mention pleces or fractions the tortilla? Intermediate response: Yes, student response meations tortilla pieces, which is another way of referring to tortilla fractions. Follow-up: How does the studenttss response relate what is being asked? Intermediate Response: The student's response relates the question by mentioning tortilla pieces, which demonstrates that the student understands that fractions are equivalent to parts or pieces. Final Answer: The stident's response consistent with the question.
<Q Clandio has 12 pencils distribute equal amounts in 3 pencil cases, do so, his friend put 6 pencils in each case and so be will not have any Jeft over. Is Claudio's friend right? (Esplaia with your words the correct becsuse she asks in more pencil cases /A>.
him he
3" shot
answer) /Q>. paula
ASST.
Yes. Follow-up: What is a student expected to do with question like this? Intermediate respoase: The student expected analyze whether Claudio's friend correct in stating that putting 6 pencils in each pencil case there will be none left over and then provide or ber opinion and explanation. Follow-up: What is the student's opinion in response? Intermedinte response: The student's opinion is that Claudio's friend is not correct. Follow-up: Does the student's response provide an explanation of or opinion? Intermediate respouse: Yes, the student provides explanation his response by pointing out that Claudio's is friend asks to pul 6 pencils in each pencil case, which would lead to distzibuting the more than pencil cases, contradicting the information given the question. Final Answer: The student's response is consistent with what asked.
USRR
<Q Prancisco and Camila have collection of action figures of their favorito series, Francisco has 4/8 of the collection and Camila has 2/3, of the total number of figures that Prancisco and Camila have they lose 5/8. How many figures of the collection do they have left? (Write your answer appropriately) /Q>. <A IT'S GOOD /A>.
4 shot
ASST.
Yes. Follow-up:
a student expected to with question this? Intermediate response: The student expected to calculate the number of action figures Francisco and Camila have left after they lose 5/8 of their collection and provide their response appropriately. Follow-up: Does the student's response make the number of action figures remaining? Intermediate response: No, the student response provides general statement TT is GOOD', but does not perform the calculation provide the specific number of action figures remaining. Follow-ug: How does the student's responae relate to what is being asked? Intermediate Responae: The student's response does not adequalely relate what is being asked because the student does not provide the number of action figures remaining perform necessary calenlation. Final Answer: student's a response inconsistent with what is heing askeed.
calenlation and provide the
USER 0-shot
Questioon| /Q> <A Lanseez /AA> Question: Is the anawer between <A /A> is incoherent or cobereat to the question between <Q /Q=? Provide linal suswer using internediate auswers follow-up questions. a single outpat, create an initial follow-up question, answer it, and continne asking and answering follow-up questions until you decide you have enough information; this point, issue "So the final answer is:" before providing the final answer:
Instamce: <Q
USER 4-shot
Instamce: <Q
Question Q+> <A Answez /A>. Let's think step by step. Question: answer between /A>
incoberent or coherent to the question between <Q /Q=? Are follow-up questions needed here?
Figure 9: Self-ask ChatGPT prompt templates for the IAD task. System/User//Ase Assisttt (asst.) are the roles in the ChatGPT API. The phrase under the User role indicates which template is used in O-shot or 4-shot format. Translated from Spanish.

---

SYSTEM
You are expert on agroroology and impact climate change. You have to decide which article is inchude or excluded in our literature
Impacts intensifying expanding cereal eropping sub-Saharan on greenhouse gas emissions security. Abstract: Abstract-1] Based the abstract/úile, the abstract INCLUDE EXCLUDE literature review? (anawer "INCLUDE" "EXCLUDE")
shot
ASST.
EXCLUDE the abstract
Title: 8ffect mid-acason drainage on CH4 NoO emisalon grain yield ecosystem: A meta-analysis. Abstract: Abatract-2
Based abstract/fitle, "EXCLUDE")
INCLUDE
(answer "INCLUDE"
2" shot
ASST.
INCLUDE the abstract
Using Organic
Vietnam Mekong Delta Region. Abstract: Abs --act=3 abstract/Lille, "EXCLUDE")
INCLUDE EXCLUDE
"INCLUDE"
3 shot
ASST.
EXCLUDE the abstract
tilliil rye cover crop can plobal warming potential yield-scalod global warming potential the long term organic soybean field. Abstract: Abstract-4 Based the abstract/tille, the abstract INCLUDE EXCLUDE in literature review? (anawer "INCLUDE" "FXCLUDDR")
4th
ASST.
INCLUDE the abstract
Based ahstract/title, Is the abstract or EXCLUDE our literature review? (answer "INCLUDE "EXCLUDE")
Title Abstract: Abstrack
Figure 10: Vanilla ChatGPT prompt templates for the SAC task. The abstracts used are Table Sys- tem/User/Assiitant (asst.) are the roles in the ChatGPT API.
H Examples of Decision Process in the Tree
We show Figures and 14 some examples along with the decision tree prediction and precision on the pathway it used.

---

SYSTEM
You expert agroecology and impact climate change. You have decide which article include excluted literature review.
USER
Title: Impacts of intensifying or expanding cereal cropping sub-Sabaran Africa on greeahouse gas emissions food security. Abstract: [Abstrect-
Based ahstract/Hitle
abstract INCLUDDE EXCLUDE literature review?
1" shot
ASST.
Step 1: Abstract discusees cereal cropping in aub-Sabaran Africa's impact on emissions and food security. Step 2: Abstract provides findings projected emissions. elimate change. Step 4 Abstract should be EXCLUDED.
Abstract doesn't primarily address agroccological practiins impact
USER
Title: Effiect of mid-season drainage on CI14 and N2O emission and grain yleld in rice ecosystem: meta-analysis. Abstract: castrect-2 Hesed on The abstrack Lille, Is the abstract INCLUDE or EXCLUDE in our literature review?
2 shot
ASST.
Abstract explores mid-season drainage's effect emissions and grain yield rice ecosystems, Step Abstract presents quantitative findings showing reduced CHa emissions and increased NzO emissions with mid season drainage. Step 3: Abstract discusses global warming potential (GWP) and influencing factors. Step Abstract direcily addresses agroccological practice's impact emissions and aligns the review's objectives, so should INCLUDRD..
USER
Title:
Reduction in Greenhouse Gas Emission from Seedless Lime Callivation Using Organic Fertilioer in a Province in Vietnam Mekong Delta Region. Abstract: Based on the abstract/tille. Is the abstract INCLUDE or EXCLUDE in our literature review?
3" shot
ASST.
Abstract evaluates greenhouse gas emissions from seedless lime cultivation Vietnam. Step Abstract presents emission reduction results from replacing chemical fettilizers. Step 3: Abstract doesn't explicitly assess agroecological practices' impact on climate change. Step Abstract should be EXCLUDED.
Title:
No-tillage with cover crop reduce global warming potential and yleld-scaled plobal warming potential in the long-term organic soybean field. Abstract: Aisst Abstract--) Based thstract/title, Is the abstract INCLUDDE or EXCLUDE in literature review?
4" shot
ASST.
Abstract dismusses effects long term no- tillage systems and cover crops on emissions and soil carbon sequestration soybean fields, Slep Abstract presents findings of reduced CH4 emissions and increased NeO emissions with different cover crop trealments and tillage methods. Slep 3: Abstract highlights significant reductions net global warming potential (GWP) and yleld-scaled GWP with no-tillage and rye cover crop. Step Abstract directly addresses the impact of agroeoological practices ino-tillage rye cover crop) reducing emissions aligns should be INCLUDED.
Step 1:
objectives,
Title: Pitle
Abstract:
Abstract
Based the abstract/title, the abstract INCLUDE EXCLUDE
O-shot
literature review? Think step by step.
Title: Pitle
Abstract:
rbst
Based on the abstract/title, Is the abstract INCLUDE EXCLUDE in our
4-shot
review?
Figure 1I: CoT ChatGPT prompt templates for the SAC task. The abstracts used are in Table 5. Sys- Usmm/sss/Assisttant (asst.) are the roles the ChatGPT API. The phrase under the User role indicates which template is used in (--hot or 4-shot format.

---

SYSTEM
You expert agroecology and impacc climate change. You have decide which article is include excluded literature review.
USER
Title: Impucts of intensitying or expanding cereal cropping in sub- Saharan Africa on greenhouse gas emissions and food security. Abstract: [Abetract-]] Question: Is the abstract INCLUDE or EXCLUDE our literature review? Are follow up questions necded here:
1" shot
ASST.
Yes. Follow up: Does the study mention an agroecologica practice? Intermediate auswer: Il does not seem that the study mentions agroorological practices, Follow up: Does the abstract diseuss greenhouse gas emissions and food security? Intermediate answer: Yes, the ahstract addresses the impact of cereal cropping in sub-Saharan Africa on greenhouse gas emissions and food security. Follow up: Does the abstract primarily focus the impact of agroecnlogical practices climate change? Intermediate answer: No, the abstract's primary focus on the impact of cereal cropping scenarioe on greeabouse gas emissiios and food securily. rather than agroevological practices. Follow up: Considering the main objertive of your literaturee review, which is to explore the impact of agroecology on elimate change, does abstrnct align well with this objective? Intermediate answer: No, the abstract's emphasis on cereal cropping scenatios and emissions review's primary focus an agraecolagy's effects climate change. So final
outsomes doesn't closely align answer is: Exclude the abstract from the literature review.
USER
Title: Edfecc of seaso- drainage on CH4 and NaO emission and grain yield in rice ecosystem: A meta -nallis.alysi.. Abstract:
Question: the abstract INCLUTER RXCLUDE in our literature review? Are follow up questions needed here:
2" shol
ASST.
Yes. Follow up: What is the agroecological practice mentioned in this artidle Intermediate answer: Yes, the study mentions mid-season drainage. Follow up: Over what does the study explore the impact of mid-seaton drainage? Intermediate answer: The study investigates the eflect of med-on drainage global warming potential (GWP). Specially CII4 emissions and N2O emissions. Follow up: Does the study present quantitative findings regarding reduced CHI emissions and increased NeO emissions due to mid-ocason drainage? Intermediste answer: Yes, the abstract provides numerical remnlts showing Given the main objectives of the literature review; does the abstract align with the focus agroroological practices' impact emissions? Intermediate answer: Yes, the abstract directly discusses the impact an agroecological practice mnid-sesoon drainage) on emissions, making it relevant to the review's s objectives. So the final anawer: Include the abstract in the literature review.
mid sea-se drainage leads decreased CH4 emissions
inereased
emissions. Follow up:
USER
Title: Reduction in Greenhouse Gas Emission from Seedless Lime Cultivation Using Organic Fertilizer in a Province in Vietnam Mekong Delta Region. Abstrnet: [Abstraes=3] literature review? Are follow up questions needed here:
Question: Is the abstract INCLUDE or EXCLUDE our
5 shot
ASST.
Yes. Follow up: Does this abstract mention agroecological practice at the farm levet? Intermediate answer: Yes, the abstract mentions the replacement of chemical fertilizers with organic fertilizers, Follow up: Does abstract assess impact of this practice on climate change? Intermediate answer: Yes, the abstract assesses the impact of replacing chemical fertilizens with organic fertilizers on greenhouse gas emissions. Follow up: Intermediate answer: Yes, it is because they substitute chemical fertilizer for organic fertiliaer. Follow up: Given the main objectives of the literature review, which exclude substitution practices, should we include this article? Intermediate answer: No, the ahstract's primary foeus is on evaluating emission reduction through fertilizer substitution, rather than agroecological redesigning practices. So the final answer: Exclude the absiract from the literalure review.
practice a subsiittionn practice?
USER
Title: tillaae with rye cover crop can reduce not glohal warming potential and yield-scaled global warming potential the long-term term organic soybean field. Abstract: Aboeract-4 J. Question: Is the abstract INCLUDE or EXCLUDE in our literature review? Are follow questions needed here:
1" shol
ASST.
Yes. Follow up: What is the agronoologieal praction mentioned in the study? Intermediate answer: The study mentions rye cropping and tillage practices. Follow up: Does the study find link between these practices and climate change mitigation or reduction? Intermediate answer: Yes the study discusses the impact of these practices on greenhouse gas (GHG) emissions and soil carbon sequestration reductions global warming potential (GWP) and yiold-scaled GWP crop? Intermediate answer: Yes, the abstract emphasipes notable decreases in not GWP and field-sealed GWP V through the implementation no-tillage practices combined with rye cover crops. So final answer: Include the abstruct in literature review.
soybean Gelds. Follow up: Does the abstract hightight siguificant
combination no-tillage and rye cover
USRR O-shot
Title: Picle] Abstract: Abbs rast Question: the abstract INCLUDK EXCLUDE in our literature review?
Provide tinal answer using intermediate answers to follow-up questions. In a single autput, create an initial follow-up question, answer it, and continue asking and answering follow-up questions antil you decide you have mough information: al Uhis point, issue "So the final answer is:" before providiag the final answer:
USRR 4-shot
Title:
ritle] Abstract: [Aberrect Question: Is the abstract INCLUDE EXCLUDE in our literature review?
Are follow up questions needed here:
Figure 12: Self-ask ChatGPT prompt templates for the SAC task. The abstracts used are in Table 5. Sys- tem/User/Assistaann (asst.) are the roles the ChaiGPT API. The phrase under the User role indicates which template is used in O-shot or 4-shot format.

---

Id
Abstract
Abstract-1
Cropping is responsible for substantial emissions of greenhouse gasses (GHGs) worldwide through the use of fertilizers and through expansion of agriculuual land and associated carbon losses. Especially aab-Saharan Africa (SSA), GHG emissions from these processes might increase steeply in coming decades, due to tripling demand for food until 2050 to match the steep population growth. This study assesses the impact of achieving cereal self- sufficieeyy by the year 2050 for 10 SSA countries on GHG emissions related to different scenarios of increasing cereal production, ranging from intensifying production to agrriultural assessed different nutrient management variants in the intensification Our analysis revealed that irreseectvve of intensification or extensification, GHG emissions of the 10 countries jointly are at least 50% higher in 2050 than in 2015. Intensification will come, depending on the nutrient use efficiency achieved, with large inereases in nutrient inputs and associated GHG emissions. However, matching food demand through conversion of forest and grasslands to coreal area likely results in much higher GHG emissions. Morcover, many countries lack enough suitable land for cereal expansion to match food demand. In addition, we analysed the uncertainty in our GHG estimates and found that it is caused primarily by uncertainty in the IPCC Tier coefficient for direct N2O emissions, and by the agronomic nitrogen use efficiency (N-AE). In conclusion, intensification scenarios are clearly superior to expansion scenarios in terms of climate change mitigation, but only if current N-AE is increased to levels commonly to be feasible in some locations in SSA. As such, intensifying cereal production with good agronomy and nutrient management is essential to moderate inevitable increases in GHG emissions. Sustainably increasing crop production in SSA is therefore a daunting challenge in the coming decades
expansion. We also
localy achieved in, for example, the United States, and which have been demonstrated
Abstract-2
Paddy rice cultivation is an important souree of glohal anthropogenic methane emissions. Drainage the flonded soils can reduce methane suhstamtially, but N20 emission occur concurrently, which would offset the reduction of methane emission. It remains unelear how mid-scason drainage affects the glohal warming potential (GWP) of CH4 and N2O emissions. In this study, meta-analysis was conducted to investigate the effect of mid-scason drainage on GWP and the factors that control the response of GWP to mid season drainage. Results showed that mid-sen mid-season drainage decreased CH4 emission by 52% while increased N20 emission by 242%. The GWP under sed-seeson drainage decreased by 47% compared to continuously flooding. The yield scaled GWP under mid- mid-scason drainage decreased by 48%. Mid-seasor drainage had no effect rice grain yield Although soil drainage times and organic matter amendment are important factors affecting CH4 and N2O emissions in rice paddy field, the study showed that neither of them had effect on the response of GWP to mid-scason drainage. The reduction rate of the GWP under mid-scason drainage increased when N fertilization application rate increases from 50 kg ha(-I) to > 200 kg ha(-1). This study demonstrated that CH4 is still dominant greenhouse gas in rice paddies under water management with mid-scason drainage. Nitrogen fertilization is an important factor GWP to mid-scason drainage. High nitrogen fertilization rate would decrease the
that regulates the response overall emission of CH4 and N20 under mid season drainage, However, increasing drainage times applying organic fertilizer under mid- season does not change the overall emission rate of CH4 and N20
Abstract-3
This study aimed to evaluate greenhouse gas (GHG) emissions from conventional cultivation (SI) of seedless lime (SL) fruit in Hau Giang province, in the Mekong Delta region of Vietnam. We adjusted the scenarios by replacing 25% and 50% of nitrogen chemical fertilizer with respective amounts of N-based organic fertilizer (S2 and 53). Face-to-face interviews were conducted to collect primary data. Life eycle assessment (1 (LCA) methodology with the cradle to gate approach was used estimate GHG emission based on the functional unit of one hectare of growing area and one tonnage of fresh fruit weight. The emission factors of agrochemicalss, fertilizers, electricity, fuel production, and internal combustion were collected from the MiL.CA software, IPCC reports, and previous studies. The S1. S2, and S3 emissions were 7590, 6703, and 5884 kg-CO2 equivalent (CO(2)e) per hectare of the growing area and 273.6, 240.3, and 209.7 kg- CO(2)e for each tonnage of commercial fruit, respectively. Changing fertilizer- based practice from SI to S2 and S3 mitigated 887. 00 1706 kg- CO(2)e ha(-1) (11. 22.55% and 33.3-63.9 9 kg. CO(2)e f(-1) (12.2-25.658) respectively, These results support a solution to reduce emissions by replacing chemical fertilizers with organic fertilizers
Abstract-4
No-tillage (NT) and the introduction of cover crops, owing to their positive effects on soil organic carbon (SOC) soquestration and crop yields, are potential agricultural practices that both support s fond security under
the new realities of climate change and alleviate greenhouse gas (GHG) emissions. However, the effects of the combination of long-term NT systems and cover crops on non-carbon dioxide (CO2) emissions and SOC sequestration have not been adequately documented, particularly in East Asia. We conducted a split-plot field experiment involving two tillage systems INT and moldboard plowing (MP) and three cover crops, namely, fallow (FA), hairy vetch (HHV), and rye (RY). NT had slightly higher soybean yield than MP, although tillage methods and cover crop treatments had no significant effects on soy beybean yield. Cover crop treatments rather than tillage methods significantly affected methane (CH4) emissions; under FA and RY treatments, we observed CH4 uptakes, whereas under HV, we observed CH4 emissions. In contrast, rather than cover crop treatments, tillage methods affected nitrous oxide (N20) emissions. Higher WFPS and soil bulk density under NT resulted in significantly higher annual N2O emissions than those under MP. However, under NT, the annual SOC sequestration rate Poaeeiinntotttaattttt by net CO2 retention under NT. Additionally, treatment under NT reduced
significantly increased compared with that under MP, the global warming potential (GWP) caused by CH4 and N20 emissions was fully GWP and yield-scaled GWP to significantly greater degree than did treatment under MP. Treatments under NT with RY cover crop had the lowest net GWP (-2324 kg CO2 equivalent ha(-l) year(-1) 1)) and yield scaled GWP (-1037 kg CO2 equivalent Mg-1 soybean yield). These findings suggest that treaúments under NT with cover crop systems- expecilly RY cover crop-in the long-term organic soybean field maintains sustainable crop production and reduces net GWP and yield-scaled GWP, which will be an effective elimate-smas smart agriculture practice in the humid, suhtropical regions prevailing in Kanto, Japan
Table 5: Abstracts used in the prompt templates for the SAC task.

---

Questions
Origin
Is the answer clear and uses appropriate language and spelling for the question asked?
LLM
Does the answer provide useful information without any joking, sarcastic, or ambiguous tones?
LLM
Does the question imply that the student should evaluate the logic and coherence of the character's answer in the story?
LLM
Are the processes shown to obtain the value?
LLM
Does the answer rule incorrect options with justification?
LLM
Does the answer limit itself to sarcastic comment instead of providing useful and coherent answer?
LLM
Does the answer provide relevant and accurate information related to the question asked? Is calculation methodology present in answer?
LLM
LLM
Does the
include calculations processes?
LLM
Does the answer support its position with facts?
LLM
Does the answer show the calculations or processes to arrive at a numerical value? Does the answer appear to be joke or humorous answer?
LLM
LLM
Does the answer show clear understanding of the mathematical concepts involved in the question? Does the answer substantiate its claim with data?
LLM
LLM
Does the answer contain spelling or grammatical errors? present any contradictions
LLM
Does the
ambiguities with the question asked?
LLM
Does the answer indicate mathematical superiority of the character? Does the answer provide justification for ruling out incorrect options? Does the answer consider all possible options?
LLM
LLM
LLM
Does the question imply deciding whether statement is correct incorrect?
LLM
Does the question present character who must make a mathematical decision?
LLM
Does the answer consider all possible options and provide justification for ruling out incorrect options?
LLM
Does the selected character know more mathematies than others mentioned? Does the answer include evidence or justification?
LLM
LLM
Does the answer offer arguments or examples to support its claim?
LLM
Does the answer provide direct answer related to question asked? Does the
LLM
consider all options and rule incorrect ones?
LLM
Does the answer provide the calculations performed? Is the selected character the most skilled in mathematic??
LLM
LLM
Does the answer indicate logical and coherent skills?
LLM
Does the
specific examples to support the correct character choice?
LLM
Does the answer demonstrate adequate knowledge and understanding of the topic raised in the question? Is the answer to the question "yes" or "no"? Does the Does the answer the question imply verifying whether mathematical operation was performed correctly?
LLM
LLM
demonstrate logical and coherent mind?
LLM
LLM
Does the answer clearly indicate whether the character's statement is correct or not?
LLM
Does the answer to the question provide a clear explanation of why is correct or incorrect? Does the answer indicate the selected character has or less mathematical knowledge than other characters mentioned in the question?
LLM
LLM
Does the answer reflect reasoning and coherence skills? Does the answer present evidence or reasons?
LLM
LLM
Does the answer reveal coherent thinking skills? Is the answer clear, concise, and uses casy-to- to-understand and precise languago? Is justification requested for the answer to the question?
LLM
LLM
LLM
Does the answer provide justification for chonsing the correct character?
LLM
Does the answer reference other relevant sources of information that may support the correct character's choice?
LLM
Is answer brief and not elaborated? Does the answer reflect clear understanding of the context and situations described in the question? Does the chosen character have more mathematical skills than others? Does the answer use language and spelling consistent with the question? Does the answer consider all options and justify discards?
LLM
LLM
LLM
LLM
LLM
Does the answer demonstrate the student's ability reason logically and follow coherent thought process?
LLM
Does the answer show understanding of the question and evidence of an attempt to solve the mathematical problem posed?
LLM
Does the answer indicate whether the character is mathematical than others? Does the answer reveal the calculation process?
LLM
LLM
Does the answer show reasoning and cohesion? Does the answer defend its assertion with solid arguments?
LLM
LLM
Does the answer make sense?
Ling
Does the answer contain any of the words "yes" or "no"?
Ling
Does the Does the answer descrihe the pracess used to obtain the result reach the conclusion? Is the answer a personal opinion?
contradict the question?
Ling
I Ling
I Ling
Does the answer involve the use of numbers or digits?
Ling
Does the answer have reasonable number of tokens?
Hum
Does the answer have a reasonable maximum length of repeated characters? Does the question suggest that something is correct or that someone right?
Hum
Hum
Does the answer have reasonable proportion of non-numeric characters?
Hum
Does the answer have a reasonable proportion of vowels?
Hum
Does the question include proper name and is it also present in the answer?
Hum
Table 6: Binary subtasks questions and their origin (human-made, LLM-made, or natural language translation of linguistics rules) before the feature selection process for the IAD task. Everything was translated from Spanish

---

Questions
Origin
Does the abstract cover climate change adaptation?
LLM
Does the abstract assess impact of agroecology on nitrogen dynamics?
LLM
Does the abstract address limitations, challenges, and potential risks?
LLM
Does the abstract discuss the role of eereecology in enhancing climate resilience in agricultural systems?
LLM
Does the abstract assess agroecological practices' impact on climate change?
LLM
Does the abstract discuss the impact of methane (CH4) emissions?
LLM
Does the abstract target stakeholders?
LLM
Does the study specifically evaluate greenhouse gas emissions?
LLM
Does the abstract discuss measures to miligate climate change?
LLM
Does the abstract evaluate agroecology's impact uitrogen dynamics? Is agroecological practices discussed in the abstract?
LLM
LLM
Does the abstract touch upon policy implications?
LLM
Does the abstract include comprehensive discussion on climate change and environmental aspects?
LLM
Does the abstract
limitations, challenges, and potential risks?
LLM
Does the abstract discuss limitations, challenges, and potential risks?
LLM
Does the abstract address methane (CH4) emissions' impact?
LLM
Does the abstract thoroughly address climate change and environmental aspects?
LLM
Does the abstract examine the implications of methane (CI14) emissions?
LLM
Does the abstract mention limitations, challenges, and potential risks?
LLM
Does the abstract provide a comprehensive coverage of climate change and environmental aspects? Does the abstract discuss policy implications?
LLM
LLM
Does the abstract consider policy implications?
LLM
Does the abstract analyze how agroecology affects nitrogen dynamics?
LLM
Does the abstract examine the impact of agroccology on nitrogen dynamies?
LLM
Does the abstract analyze the role of agroccology
carbon sequestration?
LLM
Does the abstract discuss agroecology s benefits for biodiversity? Does the abstract examine the impact these practices?
LLM
LLM
Does the abstract lack empirical evidence scientific research?
LLM
Does the abstract suggest a lack of correlation?
LLM
Does the abstract primarily address peat emissions and their quantification? findings?
LLM
Do the statistical analyses support
LLM
Does the abstract explore soil organic sequestration rate?
LLM
Does the abstract compare organic and conventional arable farming practices?
LLM
Does the abstract discuss agroccology's 's benefits for ecosystem services?
LLM
Does the ahstract discuss cultural aspects of agroecology?
LLM
Does the abstract focus primarily on economic aspects of agroccology?
LLM
Does the abstract discuss biodiversity conservation?
LLM
Does the abstract review previous studies?
LLM
Does the abstract provide evidence of the impact?
LLM
Does the abstract discuss social aspects of agroecology?
LLM
Does the abstract mention optimized timing of grass-clover ley phase removal?
LLM
Does the abstract discuss N20 emissions?
LLM
Does the abstract focus NH3 fluxes?
LLM
Does the abstract discuss soil health?
LLM
Does the ahstract compare industrial agriculture practices?
LLM
Does the ahstract discuss certified organic production?
LLM
Does the abstract specifically examine the impact on GHG protiles?
LLM
Does the abstract evaluate rubber-leguninous shrub systems?
LLM
Does the abstract focus small-scal scale or family farming systems?
LLM
Does the abstract specifically analyze nitrous oxide emissions? Is there evidence of NH3 and GHG fluxes?
LLM
LLM
Does the abstract measure field plots?
LLM
Does the abstract analyze yicld-soe vield-scaled global warming potential?
LLM
Does the study focus on conventional cultivation methods?
LLM
Does the ahstract examine nitrogen dynamics?
LLM
Does the ahstract provide recommendations? Does the abstract discuss biofuel production?
LLM
LLM
Does the abstract offer proof for its conclusions?
LLM
Does the abstract solely focus on cradle-to to-e-to-iio farm-gate activities?
LLM
Does the abstract measure net global warming potential?
LLM
Does the abstract emphasize the United States?
LLM
Does the abstract lack any new empirical data or fresh insights?
LLM
Does the abstract primarily emphasize economic aspects agroecnlogy??
LLM
Does the ahstract specifically address GHG profiles in its examination?
LLM
Is the main focus of the abstract on the cconomic aspects of agroccology?
LLM
Does the abstract center around the economic aspects of agroecology? Does the abstract specifically assess the effects of these practices? Does the abstract Does the abstract cover biofuel production?
LLM
LLM
agroecology's positive impact on biodiversity?
LLM
LLM
Does the abstract address the benefits of agroccology for biodiversity? Does the abstract examine the specific impact of these practices? Does the abstract examine how agroccology henefits hiodiversity?
LLM
LLM
L.M
Table 7: Binary subtasks questions and their origin (LLM-madde before the feature selection process for the SAC task.

---

Questions
Origin
Does the abstract mention any terms starting with "bio'?
Ling
Does the abstract specifically mention CH4?
Ling
Does the abstract discuss emissions?
Ling
Does the abstract discuss reducing something?
Ling
Does the abstract make reference the concept of cover? Is the concept of intercropping mentioned in the abstract?
Ling
Ling
Does the abstract discuss strategies?
Ling
Does the abstract address the topic of GIGG emissions?
Ling
Does the abstract refer the application of type of organic fertilisation practice? Does the abstract refer to the impact (or effect) of these practices on NitrogenNN2O/uitogen oxide emissions?
Hum
Hum
Does the abstract refer the impact (or effect) of these practices the carbon sequestration in the soil?
Hum
Does the abstract refer the impact (or effect) of these practices
Carbon/CH4/methann emissions?
Hum
Does the abstract refer to the application of one or more Slimate-Smart Agriculture practices?
Hum
Does the abstract refer the impact (or effect) of these practices on climate change mitigation?
Hum
Does the abstract refer the application of one
Sustainable Rice Intensification practices?
Hum
Does the abstract refer the impact (or effect) of these practices on climate change adaptation?
Hum
Does the abstract refer to the impact (or effect) of these practices on greenshouse gasses (GIIG) emissions?
Hum
Does the abstract refer the application of type Bio-control practice?
Hum
Does the abstract refer the application of one
agroecological practices?
Hum
Does the abstract refer to the application of type of ecological or mechanical weed management practice?
Hum
Does the abstract refer the application of one or more Diversified farming practices?
Hum
Does the abstract evaluate agroecological practices' impact climate change?
Hum
Does the abstract mention any organic agriculture practices being applied?
Hum
Does the abstract discuss the substitution of different varieties or cultivars?
Hum
Does the abstract explore the connection between agroecological practices and climate change?
Hum
Does the abstract analyze how agroecological systems affect climate change?
Hum
Does the abstract mention the replacement of various varieties or cultivars?
Hum
Does the abstract mention the implementation of Sustainable Rice Intensification practices?
Hum
Does the abstract address how these practices affect soil carbon storage?
Hum
Does the abstract discuss the application of Regenerative agriculture methods?
Hum
Does the abstract discuss form of Residues management practice?
Hum
Does the ahstract mention the use intercropping practices?
Hum
Does the abstract mention the of Regenerative agriculture practices?
Hum
Do the contents of the abstract pertain to agroecological practices?
Hum
Is the abstract discussing the application of agroecological methods?
Hum
Does the abstract mention the application of cover crops or mulching?
Hum
Does the abstract discuss implementing type of water collection practice?
Hum
Table Binary subtasks questions and their origin (human-made natural language translation linguistics rules)
before the feature selection process for the SAC task.

---

Expert Feature
Category
The unswer is blank
Traditional
Number of non-space characters in the response
Traditional
Proportion punctuation symbols the response
Traditional
Proportion of non-vowel punctuation symbols in the response
Traditional
Proportion of vowels in the response
Traditional
Number of valid words in the response
Traditional
Maximum length of consecutive characters the response
Traditional
Proportion of characters that digits in the response
Traditional
Maximum length of consocutive vowel symbols in the response
Traditional
Proportion symbols that are not digits or con-mathematical punctuation symbols the response
Traditional
Number of alphabetical symbols the response
Traditional
Proportion of symbols that are vowels in the response
Traditional
Number of words in the response
Traditional
Length of the response
Traditional
Number of numerical representations the response
Traditional
Number of mathematical punctuation symhols in the response
Traditional
Number of digits the response
Traditional
Number of punctuation symbols in the response
Traditional
Number of non-number words in the response
Traditional
Proportion of symbols that are vowels in the response
Traditional
Proportion symbols that are not numbers in the response
Traditional
Proportion of punctuation symbols the response
Traditional
There is a numerical representation (integer, real, fraction) in the response
Traditional
Number of symbols the longest number in the response Proportion of punctuation symbols digits the response Proportion of non-mattemmatial punctuation symhols in the response Maximum length of consecutive non-vowel symbols in the response The unswer digit
Traditional
Traditional
Traditional
Traditional
Traditional
Frequency the letter "k" in the response Frequency of the letter "g" in the response Frequency of the letter "y" in the response Frequency the letter "!" in
Traditional
Traditional
Traditional
response
Traditional
Frequency of the letter "h" in the response Frequency of the letter "x" in the response Frequency the letter "w" the response Frequency the letter "fi" in the response
Traditional
Traditional
Traditional
Traditional
The answer is an emoticon
Semantic
The unswer "I don't I know"
Semantic
The answer greeting
Semantic
The answer contains offensive language
Semantic
The answer contains emoticons
Semantic
Proportion non-emin non-emoticon faces the response
Semantic
Proportion of keywords in the response
Semantic
Number of emoticons in the response
Semantic
Numher of words in RAF (Real Academia Española) in the response Number of words in the Urbun Dictionary the response
Semantic
Semantic
Number of popular words in the response
Semantic
Number of keywords in the response
Semantic
Proportion of words in the response that are the Royal Spanish Academy dictionary Proportion of words the response that
Semantic
urban dictionary
Semantic
Proportion of popular words in the response
Semantic
Proportion of keywords in the response Proportion emoticons the response
Semantic
Semantic
Intersection between words in the response and words in the question that are nominal subjects The question asks for which, who or what
Contextual
Contextual
The question asks if something possible
Contextual
The unswer is binary, yes
Contextual
The question asks why somcone is right or wrong
Contextual
The question asks something or someone is okay
Contextual
The answer has reason is binary (yes no)
Contextual
Intersection between pronouns in the question and the response
Contextual
The question asks for who or which
Contextual
The guestion asks if somcone is correct or right
Contextual
The question asks if somcone is correct The question asks if somcone is right Intersection between words in the question the response
Contextual
Contextual
Contextual
Table 9: Expert features and their category (Traditional, Semantic or Contextual) before the feature selection process for the IAD task. Everything was translated from Spanish

---

Linguistic Feature
Category
Linguistic Feature
Category
intensification rainfed
Keyword
meta-analysis
Keyword
Keyword
rice
Keyword
impact of net
Keyword
CH4
Keyword
Keyword
vineyard
Keyword
systems climate
Keyword
crop
Keyword
Keyword
cconomy
Keyword
agricultural
Keyword
farm
Keyword
grass
Keyword
till
Keyword
soils
Keyword
practices
Keyword
organic
Keyword
water
Keyword
productivity
Keyword
methane
Keyword
storage
Keyword
emission
Keyword
sccnario
Keyword
tillage
Keyword
farms
Keyword
conservation
Keyword
signiticantly
Keyword
scasonal
Keyword
cover
Keyword
social
Keyword
N2O
Keyword
GHG
Keyword
change
Keyword
agroforestry
Keyword
model
Keyword
potential
Keyword
gas
Keyword
soil
Keyword
strategies
Keyword
agriculture
Keyword
system
Keyword
experiment
Keyword
synthetic
Keyword
impact
Keyword
livestock
Keyword
greenhouse
Keyword
lower
Keyword
intercropping systems
Keyword
global warm
Prefix
fallow
Keyword
bio
Prefxx
higher
Keyword
reduc
Prefix
predict
Keyword
emission
Prefix
emissions
Keyword
nitr
Prefix
conventional
Keyword
convent
Prefix
soybean
Keyword
ecolog
Prefix
agroforestry systems
Keyword
integrate
Prefix
carbon
Keyword
mitig
Prefix
intercropping
Keyword inereas
Prefix
Table 10: Linguistic features and their category (Keyword and Prefix) before the feature selection process for the
SAC task.

---

Higure 13: Explanation of the decision trees results using the features NLL ALLF- E+EE for the IAD task. Inc. is incoherent. and Coh. is coherent. The check mark means that the prediction is correct. Translated from Spanish.
Final Label
Intopretubi HLL mode!!
Cdh
Inc
>.0.85
84.4%
$90.8
81
Number of promooms that answer In the question
Proportion of answer
The ancwer clear, conciso or
tses langttage that Is understand or procise
answer not denonst mite adceuutte kaswwhelger und inderstanding of the subject matter of question
> 66.1
> -9.0%
2g.0%
23.25
6
The answer does show Inderstaodlin of the question
evidene of any attempt to solve the malbenatien maliaal problem
The anowwr does not consider all possibe options provide nationale
for discarcing incorres optinas
The answer is neihhe iumorouu nor inappropriate to the quest lon
The anewer clean and uses langiage and spelling appropritt the question posed
The answer clear und tates longuuge and spolling
appropsiate to the question posed
Explenation
78-7%
78.7%
-0
>4
<4
Treee
Number of hinory the answer
Maximum number consountts rypoated words of answer
Maximum number comsomunts iipoatte words of the answer
The anover clear concise uoses language that
The answer is not clear, contiste or langnage not easy to understnnd or presise
imderstandde predis
RP
NI.LF
$13,300
513:39
$13:3%
s 0.325
answer does possible
aut consider all
proside rationale
for discordinng incorvect optieus
The answer does not consider all possible aptions or provide a rationale for dlsearding ding Ensurres uptions
anewer not consider all possible uptions provide rationale for dinarrinng Encurreet options
sIS Proportian of sooml lettors in the
vowel betters in the answer
Binnry Subtaak Questions
Feature Extraction
Expert Peatures
>15
>15
>15
S1S
Length the answer
Length the answer
the the mother givterriiees gives him 15 gree green cars. How many cars diess Pedrro have in total?
Length answer
Longth answer
Length
is the answer incoberent or coherent to the open ended question?
Answer to on Open--nned Question
Agustiiu lus plostic boxes wibiih sbe she horpe dolls in each, Her cowsin told coldber ber that she hos 18 dolls in says Agustina's shoald sbe correct, what operation
Q: if the side AB a triuugle neceures cm, how long the perimeter figire eog considering that its throe sidess sides have
uucoswres?? thave sides have equual mesture?
Q: Mrs. Marin goos to the sppermarket and legg box
ivocaaos for $2700. a killo of evecidos for$2700 $2700, How mach did spend in totai? Explain what stops you performeed arzive andwwell welke the operation to resalt.
bank Podro takes out the money and Bow has total of $ 500, How much manry did morber him? State the equaation.
uperation the resule? What she perform to get resalt? kaplain in your own words.
liter milk 51000 for S L5200,1 Jillo
plegy
One Main Comples: Task
A: 21 mullpppeee 7x3 and got 21
cars bis
Q: Pedro has seved a 500. His morder put more money
arriive
500 000 2700-4 I0o
As agulinn bas 20 dolls
X
perform
Pedro
g.5.5

---

Higure 14: Explanation of the decision trees results using the features NLLE LFFF for the SAC task, ii for Include, and E for Exclude. The check mark means that the prediction is correct.
Final Label
0.2t
0..%
Znerginttaiee ML mande
The aheaartlle exumiio the of agroceeooog on nitrogen dynamies
The artiele does diecuss
agroecologioo
prictices
s 97.22
>3
Number of times the systems appears in the
The articce climate change mitigatioon
Explonation
60.9%
this an abstract of' pager assessing the impuct agroecologccll practices on climate change mitigation or climate change adapiation?
s &
Decision Tree
The The article evver agrocologioal pracives
The artiele evaluat ugrocellog's import on nitrogan Opnnmics
EPP
NELF
$1
One Comple
Number of times %%
appeurs in the text
Number of times the the
prefix conpent prefixccomeett appoars in the text
Binary Subtook Questtions
Linguistle Features
Feature Extraction
The explore the impoctt of agroceologial practees on elimate change
The artide explore the impocct agroeoologiaa practiess on climates chmmatechhnge
suitaiiityof 4 greenouse gos (GB(C) miligaionn optiion and one GIGG offect option for improvement
GHG balanesof a representaal Swize suckler cow farm honsing 35 Livestook units and grosslend,
colltvating uniosion per lilogrum moat in the eoomomic optinum prodaction systems difeer between the
from to 21.0 2 kg CO2 es, As meat Only GBI offect by agroforsstty gostems showed significantiy these emistion. Depending on the production system appoforvstry systns could
net GBGG emiistins by 66% to 7:3 (now)
An overiew Consersaion rottioon agriz agrisnitre (CS) changes soil poll properties and processes ses compared conventional agrialluuee These changes can, in tum, affer the delivery of eccosstem services, including climale
regulation sequestration and grwahouse gite
carbon
through soil umisdins and provision of regulation plasiccal chemiead and biological propertiex. Comservation agriesltuze affect
und
underfying lindivenity that supports many moogyttm servicss in this thisooon ovurieww we sommariee
of the sciance, the gaps in undersardinn sanding and hightihh some reseurch priorities for ecoousten services conserational agrieniture. The riviews.... (more)
Greenhouse jass mitigation and offect options
Conservation aggreauturee and ecosystem
an econonic compartson for the the Swiss Swiss case
suckler
Article Abstract
throogh
potential
We accessed

---

Figure 15: Decision tree with NLLF and EF for the IAD task. Translate lated from Spanish. Inc. is incoherent, and Coh. is coherent. The NLLF are in white and the EF are in grey. The numbers down boxes are the threshold of the decision. The Gini value of a tree node corresponds to the ratio of minoritarian class elements over the dominant ones the subtree.
dessrie HiT CRRME
The aerregree
The SEOVE rot rotarti
Wampeen aooliill mAth OET
Kendee inte SOURE ad
Ronde of
The DMRE het
AEMEE FIN axttio Teiing
The atmerri
Reatnr oo mateepeee the eooeraa
1014

---

Figure 16: Decision tree with NLLF and LF for the SAC task. is for Include, and E for Exclude. The NLLF are white and the EF are in grey. The numbers down the boxes are the threshold of the decision. The Gini value a tree node corresponds to the ratio of minoritarian class elements over the dominant ones in the subtree.
Serter
The UPAK UnMR oRee ieiiet
Neaderd fire
Threetak braddd shogg
Rickruiiin ripe in leteel
Poedee Clom The cettrottin
daar Sueg and Informaort cogas


Reviews:
[{"review_id": "WFVZhSYA70", "reviewer": "Reviewer_rrHR", "summary": "", "questions": "1) Why well-known baselines are not implemented?\n\n2) Why the limitations section doesn't contain above time complexity and regarding cost-effective solution. I think the proposed algorithm is not a cost effective solution neither faster at training and generating predictions.\n\n3) If the dataset is large. Will the proposed algorithm work?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer rrHR for the feedback, we will try to answer all of your questions and comments.\n\n**Regarding the time and complexity costs**\n\nOur method relies on three components: an LLM – GPT3.5-turbo, a medium-size LM – BERT, and a very small model – decision tree. Regarding the cost of the method: \n* Only the decision tree is trained over the full dataset. This is cost effective and very quick (the max depth of the tree is set to 5)\n* The medium-size LM is trained only over $p_l$ = 10% of the dataset (per $C=10$ BSQ). This has the same cost as training a BERT (we did not even search for the best hyperparameters). Because we add the Binary Subtask Question in the input string, it adds some 10-15 tokens (which we think is negligible).\n* The LLM is used to weakly label $p_l$ = 10% of the dataset (per $C=10$ BSQ) in a zero-shot way.The output of the LLM is in the form of a yes/no answer, hence it is the costless setting when using chatGPT. The price depends on the number of generated tokens, and of the input tokens: it is far cheaper to do this than using 4-shot CoT for example.\n\nThe initial part to create the Binary Subtask Question used only pq percent (0.015% for IAD, 1.688% for SAC) of the data, which is very small, 27 examples in both cases.\n\nWe will add in the limitation section that it needs computational resources for huge datasets because our method relies on a LLM like chatGPT (GPT3.5-turbo), like any of the methods relying on LLM or LLM themselves. Nevertheless, we know that in the future, smaller open-source alternatives like Alpaca, Vicuña or PaLM-2 will drastically reduce this cost. \n\nWe will also add in the CR version the cost in dollars of each experiment, which in total is less than two dollars for both experiments: 1.264 USD for SAC and 0.188 USD for AID.\n\nAt inference time, the already trained BERT needs to generate only the NLLF that were selected (14 for SAC, 13 for IAD). This is 14 times more costly than a BERT, but far less than an LLM. Extracting the handcrafted features and running the decision tree is very fast. We will surely add these details that we forgot to mention in the Limitation section.  \n\n**About the other baselines**\n\nFollowing your remarks and the ones of reviewer wign, we added comparison with other baselines, and not only CoT. We compared our system with self-ask [1] (using GPT3.5-turbo as a backbone). The results are the following (Prec., Rec. and F1 like in Table 1): \n\nIAD: \n\n0 shot: 21.29, 82.35, 33.84\n\n4 shots: 51.71, 77.94, 62.17\n\nSAC:  \n\n0 shot: 63.65, 55.15, 48.13\n\n4 shots:  70.31, 62.42, 59.50\n\nThis makes self-ask a better baseline than Chain of Thoughts for IAD (62.2 vs 56.5), but not for SAC (59.5 vs 62.7). We can implement  and add other recent baselines for the CR version, like self-consistent chain of thoughts [2] or tree of thoughts [3] if the reviewers wish so. \n\nFinally, we believe that the BERT baseline is strong enough as a baseline for text classification. Can you be more precise in what you were expecting so we can answer this point more precisely and add other baselines that you would like to see? \n\n[1] O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis, “Measuring and Narrowing the Compositionality Gap in Language Models,” 2022.\n\n[2] X. Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” pp. 1–24, 2022.\n\n[3] S. Yao, D. Yu, J. Zhao, and T. L. Griffiths, “Tree of Thoughts : Deliberate Problem Solving with Large Language Models,” no. 1, pp. 1–11.\n\n**About the dataset size**\n\nFinally, there is no reason why our algorithm should not work on datasets bigger than 15k training examples (IAD). With more data, it is possible to get more BSQs, improve the performances of the NLLF Generator, and eventually augment the depth of the decision tree.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The idea of breaking the main task into intermediatory sub tasks is quite interesting and can be helpful for the further research.\n\n+ The visualizations and diagrams are quite expressive.", "reasons_to_reject": "- The experiments conducted didn't involve well-known baselines.\n\n- The proposed algorithm also takes time and needs computational resources for huge datasets. This point should be mentioned in the limitations section.", "questions_to_authors": "1) Why well-known baselines are not implemented?\n\n2) Why the limitations section doesn't contain above time complexity and regarding cost-effective solution. I think the proposed algorithm is not a cost effective solution neither faster at training and generating predictions.\n\n3) If the dataset is large. Will the proposed algorithm work?", "paper_topic_and_main_contributions": "The authors have proposed a novel technique for breaking down the main task into intermediary sub-tasks, which is quite impressive.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VRMtgQ5gt9", "reviewer": "Reviewer_wign", "summary": "", "questions": "A: Authors mention that the proposed method can be applied to any predictive task using text as input. Have you tried other more complex tasks other than binary classification? Have you tried more high-stake problems like multi-hop reasoning? If yes, how is the result comparable with sota?\n\nB: Have you tried other explainable models? How is the result compared with yours?\n\nC: Line532-538 claims that the proposed method fosters causality by allowing the user to write the features directly as the input, and thus this method could prevent the model from relying too heavily on latent correlational patterns. I don't agree with this statement. Feature selection and tree construction still rely on data distribution. You cannot guarantee the usage of these man-crated features. It is still possible that some high-correlation features, such as length, are artifacts.\n\nD: ChatGPT version: Line 408 mentioned the used version is 03/23/23 ChatGPT (without the model name), while the citation refers to GPT-4. Which specific version of ChatGPT is used in the experiments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our work carefully and for her/his useful and relevant comments to improve it.  \n\n###  About the reasons to reject\n\n**Interpretability vs explainability**\n\nIndeed, we had not addressed the distinction between interpretability and explainability, and we have now added this in the CR version. We based our definitions on the _Nature Machine Intelligence_ paper of Rudin (2019), where the author makes the difference between explaining a black box model with interpreting a white-box model, or interpreting a model that outputs predictions as par with explanations.\n\nExplainability relies on algorithms aiming to explain the model predictions by showing cues to the user like LIME, or other ad-hoc methods (Fel et al., 2022; Colin et al., 2022). Interpretability relies on the possibility to know exactly why the model is making a prediction because they are inherent to the prediction and faithful to what the model actually computes.  \n\nHowever, methods like CoT which should be interpretable (because outputting explanations with their predictions) have not always shown to give faithful explanations (Radhakrishnan, 2023). It is also arguable that our method relies on learned representations from a BERT, which decreases its overall interpretability.  \n\nWe will add more details (including the ones answered above) in the introduction and related works in order to make our point more clear. We also modify Table1 to change Explainability by Interpretability.  \n\nCynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215.\n\nA. Radhakrishnan et al., “Question Decomposition Improves the Faithfulness of Model-Generated Reasoning,” 2023.\n\nT. Fel et al., “CRAFT: Concept Recursive Activation FacTorization for Explainability,” in CVPR, 2023.\n\nJ. Colin, T. Fel, R. Cadene, and T. Serre, “What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods,” Adv. Neural Inf. Process. Syst., no. April, 2022.\n\nF. Jourdan, A. Picard, T. Fel, L. Risser, J. M. Loubes, and N. Asher, “COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP tasks,” 2023.\n\n\n**Other baselines** \n\nWe thank you for the useful reference. In the CR version, we will add comparison with self-ask [1] for both our datasets and using GPT3.5-turbo as a backbone. We will also add the templates that we used in the appendix as we did for all the other configurations. The results are the following (Prec., Rec. and F1 like in Table 1): \n\nIAD: \n\n0 shot: 21.29, 82.35, 33.84\n\n4 shots: 51.71, 77.94, 62.17\n\nSAC:  \n\n0 shot: 63.65, 55.15, 48.13\n\n4 shots:  70.31, 62.42, 59.50\n\nThis makes self-ask a better baseline than Chain of Thoughts for IAD (62.2 vs 56.5), but not for SAC (59.5 vs 62.7). We can implement  and add other recent baselines for the CR version, like self-consistent chain of thoughts [2] or tree of thoughts [3] if the reviewers wish so. \n\n[1] O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis, “Measuring and Narrowing the Compositionality Gap in Language Models,” 2022.\n\n[2] X. Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” pp. 1–24, 2022.\n\n[3] S. Yao, D. Yu, J. Zhao, and T. L. Griffiths, “Tree of Thoughts : Deliberate Problem Solving with Large Language Models,” no. 1, pp. 1–11.\n\n**Task difficulty**\n\nOur method has been applied here in the context of binary classification tasks, but it can be applied to any multinomial classification or regression. However, it is important to mention that the tasks we tackled in our work are not that easy: for SAC a BERT model is far from reaching high performances like 95% of Accuracy (like in SST). Moreover we show that these tasks are overly difficult for a LLM like GPT-3.5turbo using ICL. \n\n###  Questions to authors\n\n**Answer A**: We stated in l145-147: “Indeed, it can be applied to any problem that can be reasonably decomposed in simpler tasks”. This was a mistake on our part: it can be applied to any classification or regression problem, but the proposed method cannot be applied as it is now to a generative task! We will rephrase this.  \n\nWe did not try any other tasks than binary classification, but our method is indeed applicable to any multinomial classification or even regression. We tried our setting on these two binary classification tasks where the practitioners (teachers and agronomy researchers) would like to have more explanations on the prediction of the machine learning models, and we believe this applied setting is interesting. \nWe haven't tried our method on multi-hop reasoning yet as this was not the focus of our work (and we find our paper already very dense and complex for 8 pages). However, we are currently working on extensions of our work, and multi-hop reasoning is a good direction for future work, with many adaptations though. We will add this information to the future work section.  \n\n**Answer B**: We conducted new experiments with the self-ask method, as stated above. We can add other baselines for the CR version if you wish so.  \n\nFinally, we are also right now running experiments with other explainable post-hoc methods for classifiers like LIME, SHAP and COCKATIEL (Jourdan et al., 2023), asking humans their preferences between the different type of explanations, based on questionnaires similar to (Jesus et al., 2021). However we believe our paper is dense enough to keep this for a future publication as this drives many results and analyses over the results and the models. We add this in the future work section. \n\nF. Jourdan, A. Picard, T. Fel, L. Risser, J. M. Loubes, and N. Asher, “COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP tasks,” 2023.\n\nS. Jesus et al., “How can i choose an explainer?: An Application-grounded Evaluation of Post-hoc Explanations,” FAccT 2021 - Proc. 2021 ACM Conf. Fairness, Accountability, Transpar., pp. 805–815, 2021.\n\n**Answer C**: This is a very good point, we agree with the reviewer that feature selection and tree construction still rely on data distribution and are thus not completely causal. We will rephrase these lines to reflect this, and remove the initial claim. \n\nOne way of testing our hypothesis would be to construct a synthetic dataset that allows us to test it, for example in a dataset where the bias has been deliberately inflated (Reif et al., 2023). Due to time constraints, we leave this task for future development and assessment of our model. Nevertheless we are adding details in our paper regarding this issue\n\nY. Reif and R. Schwartz, “Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases,” in Findings of ACL: ACL 2023, 2023, pp. 13169–13189.\n\n**Answer D**: We used ChatGPT (GTP3.5-turbo, version from 23 March 2023) and not GPT4, this was a mistake, thanks for spotting it, we changed the citation. \n\n###  Typos\nFinally, we took into account all of your comments regarding the typos and presentation. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is an interesting work towards interpretable prediction. The authors propose to utilize LLM to extract attributes and use the data generated by LLM to train BERT-like model to obtain Natural Language Learned Feature (NLLF). NLLF and other man-crafted features are used in the decision tree which has more explainability than the black-box models. The experimental results are promising which outperform the sota models (ChatGPT with/without CoT and BERT-like model).", "reasons_to_reject": "- The authors only conduct experiments on 2 binary datasets: Scientific Abstract Classification (SAC) and Incoherent Answer Detection (IAD). These two tasks are simple and, in my view, not high-stake enough. It is still doubt that how the framework works on other difficult tasks.\n- The paper introduces explainable AI in the introduction; however, the topic of the paper is interpretable methods. The authors do not explain the definition of interpretability and do not explain the difference between these two concepts. The authors also do not provide the work in this line.\n- In the experiment, the only comparison model with explainability is ChatGPT with CoT. It is better to see at least one more explainable model. Also, self-ask* performs better in high-stake questions. It would be better if there are some comparisons with self-ask. \n\n*: Press, Ofir, et al. \"Measuring and narrowing the compositionality gap in language models.\" arXiv preprint arXiv:2210.03350 (2022).", "questions_to_authors": "A: Authors mention that the proposed method can be applied to any predictive task using text as input. Have you tried other more complex tasks other than binary classification? Have you tried more high-stake problems like multi-hop reasoning? If yes, how is the result comparable with sota?\n\nB: Have you tried other explainable models? How is the result compared with yours?\n\nC: Line532-538 claims that the proposed method fosters causality by allowing the user to write the features directly as the input, and thus this method could prevent the model from relying too heavily on latent correlational patterns. I don't agree with this statement. Feature selection and tree construction still rely on data distribution. You cannot guarantee the usage of these man-crated features. It is still possible that some high-correlation features, such as length, are artifacts.\n\nD: ChatGPT version: Line 408 mentioned the used version is 03/23/23 ChatGPT (without the model name), while the citation refers to GPT-4. Which specific version of ChatGPT is used in the experiments?", "paper_topic_and_main_contributions": "The authors introduce a novel framework for interpretable classification. The framework first leverages LLM to extract Binary Subtask Questions(BSQs) (the attributes) and then fine-tunes a smaller BERT-like model with the text and BSQs to get Natural Language Learned Feature (NLLF). In addition to man-crafted and linguistic features, the authors propose to use the NLLF to improve the performance of the decision tree. The results show that the proposed framework could have better performance than black box BERT-like models and ChatGPT with/without CoT and at the same time the proposed model has fewer parameters and better explainability. \n\nThe main contributions are: (i) the proposed method is interpretable; (ii) the computation cost is low; (iii) the performance outperforms the sota models on 2 binary classification tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Missing the line of work in the interpretable models and the discussion of the difference between explainability and interpretability."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The model in Table 1 is better to be BERT instead of BETO to be consistent with the main content (line349 and line432). BETO is only for IAD dataset, maybe it is better to use BETO/BERT. \n\n- Line 282 The first word should be capitalized. \n\n- It would be better to keep the same order when referring to the two datasets. Sometimes authors mentioned SAC before IAD (line305-336, line366-375, line392-401, etc), but sometimes vice versa (line161-168, Table1, line511~521, etc).  \n\n- Line 470: there should be a space in ‘IADtask’ \n\n- Table2: task should be IAD \n\n- 4.3 is in the main content; however the visualization figures (Figure 11-14) are all in the appendix. It is better to keep them in the same page, e.g. provide details in the appendix and provide simple descriptions in the main content. \n\n- Line535: prevents the model from relying.....\n\n- Line899: ‘we’ instead of ‘We’"}}, {"review_id": "qUsat9IDoL", "reviewer": "Reviewer_kRx7", "summary": "", "questions": "1) Is there any chance of using filtering mechanisms in the weakly labeled data (BSQs with LLMs labeled data) to improve the performance further? Is it okay to use all LLMs labeled data on BSQs to train BERT? This is because it is possible that LLMs generate noisy labeled data which degrade the performance if directly used. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for her/his relevant comments, which we address in the following replies. \n\n**About the complexity to apply to real-world scenarios**\n\nWe really want our method to be used in practice by users like in our case it could be math teachers or agronomy researchers. The implementation of our method relies on well-known python libraries like `scikit-learn`, huggingface's `transformers` and the OpenAI python API. Our framework is actually easily applicable to any new dataset, using our code made of adaptable python objects for programmers or one-liners for non-expert. We will release it after the review phase, here is a first anonymous version: https://anonymous.4open.science/r/nllf-emnlp-2023-E07C/ \n\n**Question 1**:\n\nThat's a very good proposition. We show in Table 3 that indeed the weak labels are not perfect, as we compared them with ones made by a human expert (78% accuracy for chatGPT that produces them). The weak labels obtained from the LLM are better than the ones generated by a BERT trained on them (68% of accuracy). \n\nFor example, a method like CleanLab [1] could be used to detect the noisy labels within the weak labels and filter or correct them. We will add this to the future work section. If reviewer kRx7 wish so we can  \n\nWe could also use CoT in order to create better weak labels with ChatGPT. In fact, we tried this on the 100 examples from Table 3 and obtained the following results (Accuracy first, then Precision, Recall, F1): \n \nChatGPT (already in paper): 78\n\nYes: 71 89 79\n\nNo: 88 68 77\n\nCoT: 79\n\nYes: 78    85    81\n\nNo: 81    72    76\n\nIt allows to slightly improve the results, but as we tested over 100 manually labeled examples only this is not significant. Using OpenIA comes with a cost, as the prompts and the outputs are longer (in average 0.104 USD more per thousand queries). \n\nFinally, we show it is okay to use all LLMs labeled data on BSQs to train BERT. Even though our future work will focus on increasing the quality of weak labels, our method still allows reaching competitive results. Other interesting details about the weak labels are discussed in the paper in the subsection “Validation by an Expert”. First, we show the BERT actually compensates for errors of the ChatGPT. Second, the decision tree also can compensate for errors of the BERT, because it takes as input the BERT’s last layer before the softmax. \n\n\n[1] C. G. Northcutt, L. Jiang, and I. L. Chuang, “Confident learning: Estimating uncertainty in dataset labels,” J. Artif. Intell. Res., vol. 70, pp. 1373–1411, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is very novel. The idea is very interesting in terms of decomposing tasks into sub-tasks.\n", "reasons_to_reject": "The proposed method is somewhat complicated to apply to real-world scenarios. While it presents innovative concepts and potential benefits, its implementation may pose challenges for practical use. The intricate nature of the approach demands a high level of expertise and resources, which could limit its adoption in real-life applications. ", "questions_to_authors": "1) Is there any chance of using filtering mechanisms in the weakly labeled data (BSQs with LLMs labeled data) to improve the performance further? Is it okay to use all LLMs labeled data on BSQs to train BERT? This is because it is possible that LLMs generate noisy labeled data which degrade the performance if directly used. \n", "paper_topic_and_main_contributions": "The authors proposed a method of breaking down a main task into a set of intermediary sub-tasks which are formulated as binary questions that are related to the final target task. Accordingly, the authors provide a way of understanding the decision-making process that occurs in deep learning models. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a general strategy for decomposing a complicated task into a set of easier sub-tasks. Specifically, the proposed method represents each example with a vector consisting of a list of binary questions. By this decomposition, the proposed method not only improves prediction performance but also can generate more interpretable predictions (or be used as inputs to interpretable machine learning models). The reviewers commonly agreed that the proposed method is interesting and novel, although expressed their concerns about the experiment setup (e.g., baseline methods) as well as whether the method is generally applicable to real-world applications.
Decision: Accept-Main