CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks
Authors: Hoang H Nguyen
Ye Liu
Chenwei Zhang
TAO ZHANG
Philip S. Yu
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
natural language understanding
chain-of-thought
multi-step reasoning
slot filling
intent detection
semantic parsing
abstract meaning representation
TL;DR: Proposal of Coarse-to-Fine-grained Chain-of-Thought prompting to solve multi-grained NLU tasks

Abstract: While Chain-of-Thought prompting is popular in reasoning tasks, its application to Large Language Models (LLMs) in Natural Language Understanding (NLU) is under-explored. Motivated by multi-step reasoning of LLMs, we propose Coarse-to-Fine Chain-of-Thought (CoF-CoT) approach that breaks down NLU tasks into multiple reasoning steps where LLMs can learn to acquire and leverage essential concepts to solve tasks from different granularities. Moreover, we propose leveraging semantic-based Abstract Meaning Representation (AMR) structured knowledge as an intermediate step to capture the nuances and diverse structures of utterances, and to understand connections between their varying levels of granularity. Our proposed approach is demonstrated effective in assisting the LLMs adapt to the multi-grained NLU tasks under both zero-shot and few-shot multi-domain settings.

CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks
Hoang H. Nguyen', Ye Liu', Chenwei Zhang', Tao Zhang', Philip S. Yu' Department of Computer Science, University of Illinois at Chicago, Chicago, IL, USA Salesforce Research, Palo Alto, CA, USA
3 Amazon, Seattle, WA, USA
{hnguy7, (zinuy7,tzhang90 psyujeuic.edu, yeliu@salesforce com, cwzhang@amazon.coom
Abstract
Show me dates for music restivals In 2018
While Chain-of Thought prompting is popu- lar in reasoning tasks, its application to Large Language Models (LLMs) in Natural Language Understanding (NLU) under-explored. Moti- vated by multi-step reasoning of LLMs, pro- pose Coarse-to-Fine Shain-of-Thoughh (CoF- CoT) approach that breaks down NLU tasks into multiple reasoning steps where LLMs can learn to acquire and leverage essential con- cepts solve tasks from different granularities. Moreover, we propose leveraging semantic- based Abstract Meaning Representation (AMR) structured knowledge to capture the nuances and diverse structures of utterances, and to understand connections between their varying levels of granularity. Our proposed approach is demonstrated effective in assisting the LLMs adapt to the multi grained NI.U tasks under hoth zero-shot and few-shot multi-domain settings
LABEL
AMR
ja ARGI tes) rEse O" mualcteshhg
grained): Get eveir
Slot (F [Fine-grained): CATESORY
Dalee 1100
2016
display
Figure 1: Illustration of Abstract Meaning Represen- tation (AMR) of two structurally different but semanti- cally similar utterances with the same fine-grained and coarse-grained labels. Each colored node represents an AMR concept matching the colored word or phrase existent in the corresponding utterances.
intermediate step
valuable guidance towards subsequent fine-ge mne-graieed tasks, allowing for deeper semantic understand- ing of diverse utterances across different domains within NLU systems (Firdaus et al., 2019; Weld et al., 2022; Nguyen et al., 2023a). For instance. consider the utterance "Remind John the meeting time at 8am" " under reminder domain, recogniz- ing GET REMINDER DATE_ TIME intent is crucial for correctly understanding the existence of PERSON REMINDED slot type rather than CONTACT or ATTENDEE slot type.
1 Introduction
Natural Language Understanding (NLU) of Dia- logue systems encompasses tasks from different granularities. Specifically, while intent detection requires understanding of coarse-grained sentence- level semantics, slot filling requires fine-grained token-level understanding. Moreover, Semantic Parsing entails the comprehension of connections between both token-level and sentence-leve tasks.
Chain-of of Thought (CoT) (Wei al., 2022) pro- vides an intuitive approach to elicit multi-step rea- soning from LLMs automatically. However, there remain two major challenges with the current CoT approach: (1) LLMs entirely rely on their uncon- trollable pre-trained knowledge to generate step- by-step reasoning and could result in unexpected hallucinations (Yao et al., 2022: Zhao et al., 2023a), (2) Additional beneficial structured knowledge can- not be injected into LLMs via the current CoT.
Large Language Models (LLMs) possess logi- cal reasoning capability and have yielded excep- tional performance (Zoph et al., 2022; Zhao et al., 2023b). However, they remain mostly restricted reasoning tasks. On the other hand, mutli- step reasoning can take place when solving multi- ple interconnected tasks in a sequential order. In practical NLU systems, as coarse-grained tasks are less challenging, they can be solved first be- fore proceeding to fine-grained tasks. Therefore, the coarse-grained tasks' outcomes can provide
On the other hand, structured representation demonstrates the effectiveness in enhancing the capability of Pre-trained Language Models (PLMs) (Xu al., 2021; Bai et al., 2021; Shou al., 2022). In Dialogue systems, the dependencies among dif- ferent dialogue elements together with the existent diversely structured utterances necessitate the inte-

---

Show me dates for music festivals in 2018
CoF-CoT
Direct Prompt
Inpult Sentenoo, Domai
Step 1. Genenaa AMd
Outputt
:ARC1 (m line)
SARG2 (ol cass endty
show-01
medt (i music fostiva)
Step 2. Generane Intent
IoSuLL Seronne: Intert bocsbllay, AMR. Domaán
Output
GIIT PMINNT
X
IN:GET. TIME EVENT
[SLIIITE EVENT music Testivars) [SLIDATE_ TIME 2018]]
Outpult Slot Values
Input: Senmincs AMRR, Stor Type) parss
(Show Ol.
(music lestivll CATEGORY (i 2015, DATE, (ATE_TIME)
Stot values, Intent. Domain Outputt (Slot Value, Stot Type) pairs
Input:: Sentence intent, Damain
[Stoe Valua, Slot Typs) pairs Output: Logio Form
Generane Logic Form
ISLICATEGORY EVENT music festiolls [SL:DATE TIME 2010
[INGET EVENT
Figure 2: Illustration of CoF-CoT and its counterpart Direct approach. The left side illustrates the proposed CoF-CoT. The right side illustrates the naive Direct Prompt approach. Red and Green represent sentence-level and token-level annotations captured in the Logic Form respectively. For CoF-CoT, the prompt at each step starting from Step is conditioned on the relevant output predicted from the previous step(s).
gration of additional structured representation. For instance, observed in Figure 1, by leveraging Ab- stract Mcaning Representation (AMR) (Banarescu al., 2013), it is possible map multiple seman- tically similar but structurally different utterances with similar coarse- coarse-grained and fine-gra meeggraneed labels into the same structured representation, allowing for effective extraction of intents, slots, and their interconnections within the Dialogue systems. In our work, explore the capability of LLMs in NLU tasks from various granularities, namely multi-grained NLU tasks. Motivated by CoT, we CoT in solving multi- grained UI. tasks with an integration of structured knowledge from AMR Graph. Our contribution can be summarized as follows:
best among the most complex outputs. They pro- pose complexity metries and leverage them to se- lect demonstration samples and decoding outputs. Unlike previous CoT approaches, we leverage CoT to solve multi-grained NLU tasks.
Structured Representation Structured Repre- sentation has been widely incorporated in language models to further enhance the capability across var- ious NLP tasks (Bugliarello and Okazaki, 2020; Zhang et al., 2020). Structured representation can be either in the syntax-based structure (Bai et al., 2021; Xu et al., 2021) such as Dependency Parsing (DP) Graph, Constituency Parsing (CP) Graph or remantic-basedd structure (Shou et al., 2022) such as Abstract Mcaning Representation (AMR) Graph (Banarescu et al., 2013). Unlike previous works, we aim at leveraging structured representation intermediate step in the multi-step reasoning approach extract essential concepts from diverse utterances in the multi-domain Dialogue systems.
propose an adaptation
conduct the first preliminary study of LLMs' capability in multi-grained NLU tasks of the Dialogue systems.
To the best of our knowledge,
We propose leveraging a CoT- based upproach to solve multi coalli-grrined UL. tasks a coarse-to- line-grained sequential reasoning order.
Proposed Framework
We propose integrating structured knowledge represented via AMR Graph in the multi-step rea- soning capture the shared semanties across di- verse utterances within the Dialogue systems.
In this section, we introduce our proposed Coarse- to-Fine Chain-of of-thought (CoF-CoT) approach for NLU tasks as depicted in Figure 2. Specifically, we propose a breakdown of multi multi-grained NLU tasks into 5 sequential steps from coarse-grained to fine- grained tasks. At each step. LLMs leverage the information from the previous steps guidance towards the current predictions. As domain name could provide guidance NLU tasks (Xie et al., 2022; Zhou al., 2023), each step, we condition the domain name of the given utterance the input prompt. The model's output in the format Logic Form (Kamath and Das) which encapsulates
2 Related Work
Chain-of-Thought (CoT) CoT (Wei et al., 2022) proposes leveraging intermediate steps to extracts logical reasoning of LLMs and succeeds in vari- ous reasoning tasks. Wang et al. (2022) enhances CoT by selecting the most consistent output an- swers via majority voting. Additionally, Fu et al. (2022) argues majority consistency voting works

---

Table 1: Experimental results on MTOP and MASSIVE under zero-shot and few-shot multi-domain settings.
MTOP
Model
Zero-shot
Few-shot
NLU
Semantic Parsing
NLU
Semantic Parsing
Intent Acc
Slot F1
Frame Ace
Exact Match
Intent Ace
Slot Fl
Frame Acc
Exact Match
Direct Prompe
31,50 + 1,80
21.84 + 2,83
8,33 + 1,44
6.00 + 1.32
51.33 + 3.40
28.35
3.24
11.00 + 1.80
8.33 + 1.00
CoI
31.83 +: 2.02
22.40 = 1.61
8.67
0.35
6.33 = 1.04
47.67 5.20
28.46
3.10
11.83 = 1.53
8,50 - 1.04
SC-CoT
32.50 +: 1.89
22.71 #: 2,44
10.05 4 0.87
6.83 4: 0.76
53.50 à: 3.04
29.53
1.99
12,50 à: 1.80
9.00 +: 0.87
ComplexCoT
32.67 +: 2.00
22.86
= 3.17
10.83 + 0.29
7.16 +:
48.83 +: 2.47
29.21
2.65
13.17
= 0.58
+
Least-to-Most
45.67 +: 0.58
21.84
1.91
14.50 = 0.50
4.54
27.28
2.41
16.00
0.50
8.83
Plan-and-Solve
45,00 4.000
22.45
2.28
9.50 : 1.61
8.25 = 2.25
CoF-CoT
57.67 2.75
23.47
4.09
14.33 #: 1.52
9.00
1.00
61.50
4.93
30.12
3,93
15.00
1.32
11.00 1.61
MASSIVE
Direct Prompt
72.50 1:: 4.58
33.24 + 3.34
24.17 = 3.79
20.67 3.28
75.17 = 0.58
42.36 : 2.98
29:00 55.39
24.50 : 4.07
CoT
71,83 2.57
36.32 + 1.94
24.50 +: 2.29
21.66
1: 3.40
76.83 +: 3,82
44,89 1: 2.50
31.33 = 0,87
25.83
1: 2.25
SC-CaT
73.05 + 1.27
37.06 + 2.54
27.
+3.21
22.50
2.65
77.33 + 2.89
47.02 + 4.60
34.00 3.21
27.16
3.50
ComplexCoT
73.66 3.65
37.64 3.51
25.83
= 2.25
2.51
77.83
= 1.83
46,59
+ 2.43
36.50 = 2.89
28.00
3.69
Least-to- to Most
72.83
37.62
31.50
1.26
3.28 77.00
45.93 4:
32.50
29.00
5.11
Plon-and-Solive
69.33 + 2.47
38.07
2.07
32.00
1.26
29.00
1.26
CoF-CoT
89.00 +: 2.29
38.66
3.25
33.17
4.04
25.50
2.64
92.00 2.29
47.06 +: 4,63
37.50 1.89
29,50 3.12
coarse-grained intent label, fine-grained slot labels and slot values. Further details of Logic Form's structure and its connections with multi-grained NLU tasks are provided in the Appendix A.
lar Step 3, we condition the generated output with the predictions from previous steps, including AMR and intent. Both AMR and intent provide additional contexts for slot type predictions of the given slot values besides the input utterance.
Our multi-step reasoning is designed in the fol- lowing sequential order:
5. Generate Logic Form: The last step involves aggregating the predicted intents together with se- quences of slot type and slot value pairs to construct the final Logic Form predictions.
1. Generate AMR: Given the input utterance, LLMs generate the AMR structured representation (Banarescu et al., 2013). The representation is pre- served in the Peo-Davidsonian format as demon- strated in Figure 1,2. Each node in AMR graph refers to a concept, including entity, noun phrase, pre-defined frameset or special keyword. Edges connecting two nodes represent the relation types.
Experiments
4.1 Datasets & Preprocessing
We evaluate our proposed framework on two multi- domain NLU datasets, namely MTOP (Li et al., 2021) and MASSIVE (Bastianelli et al., 2020; FitzGerald al., 2022). As the innate capability of language understanding is best represented via the robustness across different domains, we evaluate the frameworks under low-resource multi-domain settings, including zero-shot and few-shot. Details of both datasets are provided in Appendix B.
2. Generate Intent: In this step. LLMs generate coarse- suars--grained intent label prediction when con- ditioned on the given input and its corresponding AMR Graph. AMR concepts could provide addi- tional contexts to ambiguous utterances, leading to improved ability to recognize the correct intents.
Generate Slot Values: In this stage, to gen- erate the fine-grained slot values existent in the in- put utterance, besides the utterance itself, prompts for LLMs are conditioned on the generated AMR structure and predicted intent label. As AMR graph captures the essential concepts existent in the ut- terance while abstracting away syntactic idiosyn- crasies of the utterance, can help extract the im- portant concepts mentioned in the utterances. In order to further couple the connections berween slots and intents (Zhang et al., 2019; Wu et al., 2020), predicted intents from the Step 2 are also concatenated to construct input prompts for Step 3. Generate Slot Value, Slot Type pairs: After obtaining slot values, LLMs label each identified slot value when given the slot vocabulary. Simi-
To provide comprehensive evaluation for coarse-grained, fine-grained NLU tasks, well as the interactions between the two, we conduct extensive study on both NLU and Semantic Pars- ing metrics, including: Slot FI Fl-score, Intent Accu- racy, Frame Accuracy, Exact Match. Intent Accu- racy assesses the performance on coarse-grained sentence-level tasks, while Slot Fl metric evaluates the performance on more fine-grained leven tasks. The computation of Frame Accuracy and Exact Match captures the ability to establish the accurate connections between sentence-level and token-level elements. For more details of individ- ual metric computation from the Logic Form, we refer readers to (Li et al., 2021).
To conduct the evaluation with efficient API

---

calls, following (Khattab et al., 2022), we construct test sets by randomly sampling 200 examples cover- ing a set of selected domains, namely test domains. We repeat the process with different seeds to gen- erate corresponding test sets. Reported perfor- mance is the average across 3 different seed test sets with standard deviations. For few-shot settings, we randomly select a fixed k samples from disjoint set of domains, namely train domains. These sam- ples are manually annotated with individual step labels as commonly conducted by other in-context learning CoT approaches (Wei et al., 2022: Wang et al., 2022). Additional implementation details as well as the prompt design and sample outputs are provided in Appendix C and D respectively.
Table 2: Comparison between FT and LLM approaches on MTOP dataset.
Method
Ascmmption Supersised
Intm
She
Froms Ace
Fient March
ROBERT,FT FT
67.19 + 2.00
75.17
1.08
43.57= 4.18
30.10= 1.08
RoBERTaFE FT
ZSL
0
12.68 4: 1.25
0
ROBERG FT
PSL.
13.75 + 1.22
7ST.
275
CaP-Col
ESL
30.32 3.33
15.0
11.00
Table 3: Ablation study on the effectiveness of differ-
ent structured representations on MTOP dataset under zero-shot settings. CP, DP, AMR denote Constituency Parsing, Dependency Parsing and Abstract Meaning Representation respectively.
Inseni Ac:
Fl
Frame Acc
Exact Match
(wlp stuucurr)
57.16
17.50
2.2
12.16
4.67 + 3,33
CF-CoT
57,33
19.4
1316 1166100 101
5,50 1.32
DP-CaT
$7.50
1723 + 253
1267 104
588 2,08
AMR-CoT
57.67 2.75
23,47 1 4.00
14:33 4 1.2
9.00 1: 1.00
exist no overlapping domains between train and test domains under ZSL and FSI. setting for both FT and CoT approaches. This leads to 3 different scenarios for FT approaches, including:
4.2 Baseline
Chain-of-Thoupp (CoT) Approach Compari- We compare our proposed method with the
Fully Supervised: Samples sharing similar do- mains with test sets are used for training.
son
current relevant state-of-the- CoT approaches:
Direct Prompt: Naive prompting to generate the
ZSL: We utilize samples from domains different
Logic Form given the intent and slot vocabulary.
from test domains for training.
CoT (Wei et al., 2022): Automatic generation of
FSL: We leverage samples from domains differ- ent from test domains in conjunction with a fixed number of k-shot test domain samples.
series of intermediate reasoning steps from LLMs
SC-CoT (Wang al., 2022): Enhanced CoT via majority voting among multiple reasoning paths. Complex-CoT (Fu et al., 2022): Enhanced CoT by selecting and measuring the consistency of the most complex samples. In our case, we leverage the longest output as the complexity measure.
5 Result & Discussion
As observed in Table 1, our proposed CoF-CoT achieves state-of state-of-the-art performance across differ- ent evaluation metrics on MASSIVE and MTOP datasets under both zero-shot and few-shot settings. The performance gain over the most competitive baseline is more significant in terms of Intent Ac- curacy (25% and 15.34% improvements on MTOP and MASSIVE respectively in zero-shot settings). Additional case studies presented in Appendix E further demonstrate the effectiveness of CoF-CoT. In addition, we observe consistent improvements of different CoT variants over the Direct Prompt. It implies that CoT prompting allows the model to reason over multiple steps and learn the connec- tions between different NLU tasks more effectively. In comparison with MASSIVE, performance of all methods is significantly lower on MTОР. It is mainly due to the more complex Logic Form struc- tures existent in MTOP. It is noticeable that MAS- SIVE datasets contain samples of fewer average number of slots, leading significantly better per- formance on Semantic Parsing tasks (i.e. Frame Accuracy and Exact Match).
Least-to-Most (Zhou et al.. 2022) Enhanced CoT by first automatically decomposing the in- hand problems into series of simpler sub-problems, and then solving each sub-problem sequentially.
Plan-and-Solve (Wang et al., 2023) Enhanced CoT by guiding LLMs to devise the plan before solving the problems by prompting "Let's 's first un- derstand the problem and devise a plan to solve the problem. Then, let's 's carry oul the plan and solve the problem step by step."
Fine-tuning (FT) Approach Comparison As one of the early studies in leveraging LLM for NLU tasks, we also conduct additional compar- isons with traditional FT approaches. Specifically, we leverage RoBERTa PLM (Zhuang et al., 2021) with joint Slot Filling and Intent Detection objec- tives (Li et al., 2021) as the FT model. Unlike LLM, traditional FT operates under cloled-world assump- tion which requires sufficient data learn domain- specific and donainnagnostic feature extraction in multi-domain settings. For a fair comparison with LLM, we impose an essential constraint that there
Our CoF-CoT shares certain degrees of similari- ties with Least-to-Moos (Zhou et al., 2022), Plan- and-Solve prompting (Wang et al., 2023). How-

---

Table 4: Ablation study of step ordering on MASSIVE dataset. CoF and FoC denote ceoss--to-iine-griine and Cime-toCCoarse-eee order respectively.
Method
Assumption
Inient Acc
Slot
Prame Acc
Lxact Match
Random-Con
Random Order
80.67 +3.60
27.14 + 2.47
26.50 +1.00
16.50 + 1.04
FoC-CoT
FoC order
83,00 + 2.88
32.11 +: 2.50
28.50 +: 3.21
18.00 = 3.50
CoF-CoT (wlo step
No AMR
81.50 + 4.36
33.68 + 2.40
27.50 + 2.65
18.00 = 0.76
CoF-CoT (w/o step 2)
No intent
78.17 +: 4.80
27.66 +: 1.93
23.50 +: 2.78
14.50 = 2.25
CoF-CoT (w/o step 3)
No separate KP
82.33 + 1.04
34.63 3.10
32.83 2,47
23,00 = 1.80
Col-CoT (w/o step 4)
No sepurate slot prediction for KP
10.17:40000
32.92 5.02
31.50 3.50
21.83 = 3.17
CoF-CoT (w/o step 344)
No separate slot prediction
81.33 + 4.19
31.31 #3.77
27.67 + 5.34
21.00 = 4.92
CoF-CoT
CoF order (F (Full)
89,00 + 2.29
38,66 1: 3.25
33.17
1: 4.04
25,50 = 2.64
-Conditioning
No domain
84.50 2.75
36.80 + 2.08
32.50 + 1.73
24.83 = 0.58
ever, unlike the two aforemenniooed baselines that rely heavily on the existent pre-trained knowledge of LLMs, CoF-CoT provides a controllable num- ber of sequential steps and conditioning inputs for each step, allowing for flexible adaptations and cus- tomizations future downstream tasks that LLMs might not be familiar with.
with hypothetical individual step removal. Table 4 demonstrates CoF logical ordering yields the best performance with significant improvements on the challenging Exact Match metrics (9.00 and 7.50 points of improvement over random and FoC re- spectively). Random, FoC ordering together with CoF ordering with missing individual steps ne- glect the natural connections of problem-solving from high-level (rearse-grained) to low-level (fine- grained) tasks, Icading to worse performance across different metrics. For CoF-CoT, when step 1 or step 2 is removed (no AMR or intent information), we observe the most significant performance decrease, implying the essence of coarse- -ranieegrainddd knowledge for LLMs solve the later sequential steps.
Comparison with FT Under 7SL. and FSL set- tings, the FT model suffers from the aforemen- tioned domain gap issues. Specifically, as observed in Table 2, since there exist minimal overlapping intent labels between train and test domains, with- out sufficient data in ZSL and FSL settings, the FT approaches are unable to learn transferable multi- domain features, leading to 0 performance in Intent Accuracy. This behavior also results in 0 perfor- mance for both Frame Acc and Exact Match as the correct intents are the prerequisites for correct semantic frame and exact match metrics. On the other hand, Fully supervised FT approach acquires domain-specific knowledge of target domains from training data and performs the best across differ- ent evaluation metrics. However, this assumption does not directly match ZSL/FSL settings in which LLMs are currently evaluated.
Impact of Conditioning The major advantage of our multi-step reasoning the ability to explic- itly condition the prior predictions in later steps. As observed in Table 4, conditioning prior knowl- edge in multi-step reasoning improves the overall performance of CoF-CoT across different metrics with the most significant gain in Intent Accuracy (+4.50%). This observation implies the importance of conditioning the appropriate information on CoT for an improved performance of LLMs under chal- lenging zero-shot multi-domain settings.
Impact of Structured Representation Besides AMR Graph, there exist other structured represen- tations that directly link to semantic and syntactic understanding of utterances, including DP..P. Our empirical study presented in Table 3 reveals that AMR-COT unanimously achieves the best perfor- mance, demonstrating its effectiveness in capturing the diversity of input utterances when compared with other structured representations.
Conclusion
In this work, we conduct a preliminary study of LLMs' capability in multi-grained NLU tasks of Di- alogue systems. Moreover, motivated by CoT, we propose a novel CoF-CoT approach aiming to break down NLU tasks into multiple reasoning steps where (1) LLMs can learn to acquire and lever- age concepts from different granularities of NLU tasks, (2) additional AMR structured representa- tion can be integrated and leveraged throughout the multi-step reasoning. We empirically demonstrate the effectiveness of CoF-CoT in improving LLMMs capability in multi -ollieeraiied NLU tasks under both zero-shot and sho- domtin settings.
Impact of Step Order To further understand the importance of the designed CoF step or- der, we conduct additional ablation studies on 3 different scenarios: (1) random ordering (step 3-+1+2-+4-5) (2) Fine-to-Coarse (FoC) order- ing (step 1 1-+3 +4-+ +2 5)), and (3) CoF ordering

---

Limitations
in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Our empirical study is restricted to English NLU data. It is partially due to the existent English- bias of Abstract Mcaning Representation (AMR) structure (Banarescu et al., 2013). We leave the adaptation of the CoF-CoT to multilingual settings (Nguyen and Rohrbaugh, 2019; Qin et al., 2022; Nguyen et al., 2023b) as future directions for our work.
Emanuele Bugliarello and Naoaki Okazaki. 2020. En- hancing machine tramslation with dependency-awer self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1618-1627, Online. Association for Computational Linguistics.
lñigo Casanueva, Ivan Vulić, Georgios Spithourakis. and Pawel Budzianowski. 2022. Nlu++: A multi- label, slot-rich, generalisable dataset for natural lan- guage understanding in task-oriented dialogue. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1998-2013.
Our work empirically studied on the Flat Logic Form representation. other words, Logic Form only includes one intent followed by set of slot sequences. There are two major rationales for our empirical scope. Firstly, the early preliminary study multi-graines NLU tasks which unify both Semantic Parsing and NLU perspectives, we de- sign small and controllable scope for the experi- ments. Secondly, as most NLU datasets including MASSIVE (FitzGerald et al., 2022) are restricted to single-intent utterances, Flat Logic Form is a viable candidate reconciliating between traditional NLU and Semantic Parsing evaluations. We leave explorations the more challenging, Nested Logic Form where utterances might contain multiple in- tents for future work.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Mauajama Firdaus, Ankit Kumar, Asif Ekbal, and Push- pak Bhattacharyya. 2019. multi-task hierarchi- cal approach for intent detection and slot filling. Knowledge Based Systems, 183:104846.
Jack FitzGerald, Christopher Hench, Charith Peris. Scott Mackie, Kay Routmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2022. Massive: Im-example multilin- gual natural language understanding dataset with 51 apploogialllyddierss languages.
7 Acknowledgement
We thank the anonymous reviewers for their con- structive feedback which we incorporated in the final version of this manuscript.
This work is supported in part by NSF under grant III-2106758.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-base prompt- ing for multi-step reasoning. arXiv preprint arXiv:2210.00720.
References
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang.
Aishwarya Kamath and Rajarshi Das. A survey semantic parsing. In Automated Knowledge Base Construction (AKBC).
Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- BERT: Improving pre-trained transformers with syn- tax trees. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 301 1011-00 Online. Association for Computational Linguistics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search predict: Composing retrieval and language mod- els for knowledge-intensin nlp. arXiv preprint arXiv:2212.14024.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgeseu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguis- tie annotation workshop and interoperubility with discourse, pages 178-186.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2950-2962, Online, Association for Computa- tional Linguistics.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojan- ski, and Verena Rieser. 2020. SLURP: A spoken lan- guage understanding resource package, In Proceed- ings of the 2020 Conference on Empirical Methods

---

Hoang Nguyen and Gene Rohrbaugh. 2019. Cross- lingual genre classification using linguistic groupings. Journal of Computing Sciences in Colleges, 34(3)911- 96.
understanding ACM Computing Surveys, 55(8):1- 38.
Di Wu, Liang Ding, Fan Lu, and Jian Xie. 2020. SlotRe- fine: A fast non-autoregressive model for joint intent detection and slot filling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1932-1933 On- line. Association for Computational Linguistics.
Hoang Nguyen, Chenwei Zhang, Ye Liu, and Philip Yu. 2023a. Slot induction via pre-trained language model probing and multi-level contrastive learning. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, pages 470-481, Prague, Czechia. Association for Computa- tional Linguisties.
Congying Xia, Chenwei Zhang, Hoang Nguyen, Jiawei Zhang, and Philip Yu. 2020. Cg-bert: Conditional text generation with bert for generalized few-shot intent detection. arXiv preprint arXiv:2004.01
Hoang Nguyen, Chenwei Zhang, Congying Xia, and S Yu Philip. 2020. Dynamic semantic matching and aggregation network for few-shot intent detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2009-1218.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Vic- tor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. UnifiedSKG: Unifying and multi-tas multi-tasking structured knowledge grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 602-631, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hoang Nguyen. Chenwei Zhang, Tao Zhang, Eugene Rohrbaugh, and Philip Yu. 2023b. Enhancing cross- lingual transfer via phonemic transcription integra- tion. In Findings of the Association for Compu- lational Linguistics: ACL. 2023, pages 9163-9175, Toronto, Canada. Association for Computational Lin- guistics.
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Guang Lou, Wanxiang Che, and Min-Yen Kan, 2022. GI-clef: A global-local contrastive learning frame- work for cross- longs-lingual spoken language understand- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 2677-2686.
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong, Xiaojun Quan, Daxin Jiang, and Nan Duan. 2021. Syntax- -etitxx-enhanedd pre-trained model. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 1412-522..
Ziyi Shou, Yuxin Jiang, and Fangzhen Lin, 2022. AMR- DA: Data augmentation by Abstract Meaning Rep- resentation, In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 3082-3098, Dublin, Ireland. Association for Computational Lin- guistics.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint vrXiv:2210.03622
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models. In Pro- ceedings of the 6lst Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers). pages (609-2634, Toronto, Canada. Associ- ation for Computational Linguistics.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and S Yu Philip. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5259 5259-5267.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020. Sg-net: Syntax-guided machine reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9636-9643.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quac Lee, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint Xrxi:
Ruochen Zhao Xingxuan Li. Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023a. Verify-and-edit: A wwwledge-enhancee arXiv preprint ar0iv:2305. 033668.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 11900....191.....
framework.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023b. A survey of large language models. arXiv preprint arXiv:2303. 18223..
Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. survey of joint intent detection and slot filling models in natural language

---

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh Interna- sional Conference Learning Representations.
Table 5: Details MTOP and MASSIVE datasets
Dataset
МТOP
MASSIVE
# Domains
11
18
# Train Domains
8
14
Test Domains
4
Intents
117
60
# Slots
78
55
Han Zhou, Ignacio lacobacci, and Pasquale Minervini. 2023. XQA-DST: Multi-domain and multi-lingus dialogue state tracking. In Findings of the Associ- ation for Computational Linguistics: EACL 2023, pages 999-1009, Dubrovnik, Croatia. Association for Computational Linguistics.
Sentence Length
6,14 - 2.30
6,34 4: 2.94
# Slots per sample
1.87 0.81
0.73 + 0.63
A Connections between Semantic Parsing and NLU Tasks via Logic Form
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized bert pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-12211
Logic Form not only captures the coarse coarse-grained intent labels and fine-grained slot labels of the terances but also encapsulates the implicit connec- tions between slots and intents.
As observed in Table 7, Logic Form is con- structed as the flattened representation of the depen- dency structure between intents and slot sequences. Semantic Frame constructed intent type(s) fol- lowed by sequence of slot types be directly extracted from the Logic Form. In addition, via the Logic Form, the coarse grrsin- intent label CREATE REMINDER, fine-grained slot TODO, DATE_TIME labels together the respective slot values (message Mike, at 7pm tonight) can all be extracted and converted to appropriate format (i.e. BIO format as the traditional sequence labeling ground truths (Zhang et al., 2019)). Therefore, Logic Form can be considered the unified label for- mat bridge the gap between Semantic Parsing (Li et al., 2021; Xie et al., 2022) and traditional Intent Detection and Slot Filling tasks in NLU systems (Xia et al., 2020; Nguyen et al., 2020; Casanueva et al., 2022).
Barret Zoph, Colin Raffel, Dale Schuurmans, Dani Yo- gatama, Denny Zhou, Don Metzler, Ed H. Chi, Jason Wei, Jeff Dean, Liam B. Fedus, Maarten Paul Bosma, Oriol Vinyals, Percy Liang. Sebastian Borgeaud, Tat- sunori Hashimoto, and Yi Tay. 2022. Emergent abilities of large language models. TMLR.
B Dataset Details
We provide the details of MTOP and MASSIVE datasets in Table 5. As compared to MASSIVE, MTOP dataset only contains more slot types and intent types but also tends to cover more slot types per sample the Logic Form. This chal- lenging characteristic explains the consistent lower performance across all methods on MTOP when compared to MASSIVE, as observed in Section 5.
C Implementation Details
As the proposed step-by-step reasoning can be ap- plied any LLMs, our proposed method is LLM- agnostic which empirically studied in Appendix F. For simplicity and consistency, in our main em- pirical study, we leverage gpt-3.5-tu from Ope-

---

nAI as the base LLM model. Following (Wang al., 2022), we set the decoding temperature T=0.7 and number of outputs n=10.
As observed in Figure 5, our CoF-CCT provides the predictions closest the ground truth while other baselines struggle to (1) generate the correct intent type (i.e. GET DATE_ DATE_TIME, TIME_ EIE_EVEET in- tent type from Direct Prompt in comparison with GET EVENT intent from ground truth) (2) iden- tify the correct slot values (i.e. everything slot value generated from CoT), (3) generate the cor- rect slot type for the corresponding slot values. (i.e. EVENT_ TYPE slot type for music festivals slot values from Complex- Complex-CoTiee instead of CATE- GORY_EVENT EEEEEEEETTTT slot type).
As domain names provide essential clues for lan- guage models in multi-domaij settings for multi- grained NLU tasks (Zhou al., 2023), to safe- guard the fairess in baseline comparisons, consistently include the domain name in the input prompts for all baselines unless stated otherwise. Specifically, the only exception is presented Ta- ble for CoF-CoT(CoF (nder)-Conditioning
For few-shot (i.e. k-shot) learning settings, we randomly sample k examples and manually prepare the necessary labels for different baseline variants. We experiment with k=5 in our empirical study.
F LLM-Agnostic Capability
Our proposed CoF-CoT is LLM-agnostic since the focus of the work is on the prompt design, which can be applied any LLMs. As most LLMs rely on the high quality of designed prompts, our proposed CoF-CoT prompt design can be used as input to any LLMs for zero-shot and in-context learning settings. This is also similarly observed in CoT (Wei et al., 2022), SC-CoT (Wang et al., 2022) and other comparable CoT methods. For further clarification, report additional empirical results of our proposed Col--CoT applied both of the backbone PaLM (Chowdhery et al., 2022) and GPT3.5 LLMs on the MTOP dataset under both ZSL and FSL settings in Table As observed in Table 8, CoF-CoT prompting consistently out- performs the two backbone LLMs across all NLU and Semantic Parsing tasks, demonstrating both the effectiveness and LL capability of proposed CoF-CoT
Domains of Demonstration Samples To repli- cate more realistic scenario where the domains of k-shot demonstration samples are generally un- known, we assume that k-shot demonstration sam- ples come from different domains from the test samples. The relaxation of constraints on the as- sumption regarding the domain similarity between demonstration samples and test samples allows for broader applications and encourages LLMs to cumulate and extract the true semantic knowledge from k-shot demonstrations and avoid overfitting any specific domains. For completeness, we also conduct additional empirical studies to compare the FSL performance of CoF-CoT under both sce- narios: (1) k demonstration samples are from the same domain test samples, (2) k demonstrations are drawn from different domains from the test samples. As observed in Table 6, additional con- straint of similar domains between k-shot demon- stration samples and test samples leads to improve- ments in the evaluation performance across NLU and Semantic Parsing tasks. This might be intuitive since LLMs can extract domain-relevant informa- tion from the given k comain-similan samples to assist with inference process test samples.
D Prompt Design
Prompts for individual steps of our CoF-CoT are presented in Figure 3. Additional output samples are also provided in Figure 4.
Qualitative Case Study
We present additional Qualitative Case Study com- paring the outputs between different baseline meth- ods and our proposed CoF-CoT in Figure 5.

---

Table 6: FSL Results of CoF-CoT with k-shot demonstration samples selected from different and similar domains in comparison with domains of test samples on MTOP dataset.
Method
Assumption
Intent Acc
Slot F1
Frame Acc
Exact Match
CoF-CoT
k domain--ifferent samples
61.50 = 4.93
30.12 3.93
15.00 1.32
11.0 1.61
CoF-CoT
k domain-similar samples
70.00 = 1.33
38.16 5.42
20.50 + 2.00
15.00 = 1.00
Table 7: Sample utterance with its Logic Form under both Semantic Parsing and NLU tasks' metrics, // denotes the separation between tokens of the given utterance.
Metric
Granularity Level
Format
Ground Truth
upli if
Loglc
TUDO: message mike ISLDDAEE at tonggh]]
Insest Accuuasy
Course grained
Inteut Label
INCRRAATE REMINDER
NLU Taaks
Stot Fi
Finz-grained
BID Stot Sequence
ORONORORO B B-TODO LTODO J'B. DATE, TIME y LALETEE TIME LDDAE, TIME
Pansig
Fosel
Ausuracy
Lugic Furn
DNCOREATE: REMINDER [St. (SL-TODOO message mike]] [SI (SL-DATE TIME: 7pm lnnigh])
INCREATTE DESINDERSS PODDOSLLAATT
geneente single comnopaniiin Mearang Deprosemattoo
Use anoe: (uzerance] [sommii))
AMR Giapl: (ANI))
Grepo, Inllnwnng Vivamstary byne
Use uneo: (uterance)
[donain)) AMR Geaph: (AMI)]
iment Veosturlry: (mtern vocatj
Slap Sased on the utarance, demmaas It AMT Gruph and la intant, gas enaze ley pi phasss for the uterannes Key pfeasss can made up from mutile consapts.
uterannc appears koty phoase Key phrases need contain consecutive given untertnce. commas.
all words
(utteanno))
AMR Groph: (MMP))
Imeett: (riene)
phrases: phrase)
Step Gaven wecats lery. utemenne, cimmn Eruphh intent, identity the commepending seel type typee the unctturtry for wach bey prrmon. nonz of the types in the vocalu fits, lotum the slot type as
beey poet vecatt Utternnon: prteennon)
Domaor prinssr)
AMi Grush, (Ab(i))
(lay phrace. Slor Type) pain: paar
diet ganerate logic forn of the use aiterance in the tomal of IIN. (SA. [SS.. where IN. lnvied by intent type and SLL lolloweed by siot ope and skn value par seperated by wit space. number [SL: unimiied The rumoee
Given
domain
Use anoo: (uzerance] Domain: [Domai]]
(Itent)
Typa) gazt
Logie Form:
Figure 3: CoF-CoT Prompt Design Template. {-) denotes the placeholder argument.
Table Experimental results on MTOP dataset under zero-shot few-shot multi-domain settings with different LLM backbone architectures (PaLM (Chowdhery et al., 2022) and GPT3.5).
MTOP
Model
Zero-shot
Few-shot
NLU
Semantic Parsing
NLU
Semantic Parsing
Intent Ace
Slot Fi
Frame Ave
Exact Match
Intent Acc
Slot Fi
Frame Ace
Exact Match
PaL.M
16.67 a 2.52
7.24
1,00
3,17 0.76
1.17
0.76
48.83
4.54
14.24 J: 1.58
4.17 2,02
2.33 0.76
PaLM Col-Coo
42,33 :: 3.33
13.73 2.88
4.01 0.15
3.50
1.32
57.17 3.79
21.47 1: 3.79
10.33 3.06
6.67 2.75
GPT3.5
31,50 + 1,80
21.84 + 2.83
8.33 1.44
6.00
1.32
51.33
3.40
28.35 +: 3.24
11.00 1,80
8.33 1.00
CoF-CoT
9.00 1.00
493
30.12 3.93

---

Sample 1:
(utterance)- show dates for music festivals in 2018
(ground truth) = [INGEE_EVENN) [SLCATTGOOYY (domain) event
festivalls [SL:DATE__
(AMR)= = (cl show-01
:ARG1 (m /me)
:ARG2 date-entity
:mod (tl music-festival
'year 2018))
(intent) = GET EVENT
(key phrase] Show me, dales, music fesitval, in 2018
(Key phrase, Slot Type) = (Show me: O), (dates: O), (music festival: CATEGORY, EVENT), (in 2018: (ATE_TIME)
(In) = (IN.GET_EEVV [SL:CATEGORY EVENT music festivals [SLIDATE_ TIME in 2018 ]]
Sample 2:
[utterance)- Set my timer for my tabata workout.
(ground truth) [NNCREATE, TIVEER_TTRRRRR_E WSLLMETHOD_TMAEE TIMER timer [SL.TIMER_ MOMEERR_MMEEEE NAME tahata workout ] (domain) timer
(AMR) = (set-01
JRGG (I)
JARG1 (iimer-02
:ARG (my-03)
:0p1 (workout-05
JRRGO (my-04)
:ARG1 (tabata-06))))
[intent)= = CREATE_ TIMER
(key phrase) = Set timer, tabata workout
(Key phrase, Slot Type)= = (Set my time: МETHOD. TIMER), (tabata workout TIMER NAME) (In) = [INCCREATE_TIEE CEREEEETTTTERRRRREEEE [SLMETTHOD_ TIMER Sei timer [SL:TIMER_MM] NAME tabala workout ]]
Figure 4: Sample output from our CoF-CoT. (-) denotes the placeholder corresponding to template in Figure 3.
Utterance 1:
Show me dates for music festivals in 2018
Ground Truth:
[INGETT [SL:CATEGORY EVENT festivals [SL:DATE,
Prompt:
IN:GET
EVENT (SL:TITLE EVENT festivals [SLIDATE_ TIME:2018 ]]
CoT:
[SL:TITLE EVENT music festivals ] [N:GET EVENT [SL:DATE_ TIME 2018 [SL:TITLE EVENT music festivals ]
[N:GET EVENT [SL:DATE TIME 2018
SC-CoT:
COmplex-CoT:
IN.GET DATE TIME EVENT [SLEVENT TYPE music festivals ESEEET YEAR 2018
CoF-CoT:
[INGETT EVENT [SL:CATEGORY EVENT music festivals [SL:DTTE TIME 2018
Utterance
want know everything that breaks about the dam breaking california
Ground Truth:
[NNNEWS QUERY (SL:PLACE NAME california
Direct Prompt:
[IN: WEATHER QUERY] [SL: PLACE NAME california [SL: EVENT NAME dam breaking
COT:
IN: NEWS QUERY (SL: WEATHER DESCRIPTOR everything [SL: PLACE_NAME california ]]
SC-CoT:
[INNEWS QUERY SLINEWS TOPIC dam breaking in califomia [INNEEWS _UNRR (SL:NEWS_TOR dam breaking in califomia.
Complex-CoT:
CoF-CoT:
[IN: NEWEEWWWWEO QUERY [SL: PLACE_NAME california ]]
Figure 5: Qualitative Case Study among baseline variants and the proposed CoF-CoT. Ground Truth is shown in red.


Reviews:
[{"review_id": "hXoAC23bIw", "reviewer": "Reviewer_GkrJ", "summary": "", "questions": "1. Page 7, line 508. The paper says ”our proposed methods is LLM-agnostic”. But the experiments only involve gpt-3.5-turbo without comparing with other LLMs. Could you add the analysis of different LLMs?\n2. Table 1. The experiments demonstrate different results of different prompts (zero/few-shot settings) without considering other related works. Could you add the results of other methods to make more comprehensive comparison? (e.g., full training. You can search the citations of MTOP[1], MASSIVE[2])\n\n [1] Li H, Arora A, Chen S, et al. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark[J]. arXiv preprint arXiv:2008.09335, 2020.\n [2] FitzGerald J, Hench C, Peris C, et al. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages[J]. arXiv preprint arXiv:2204.08582, 2022.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. We respond to your questions as follows:\n\n**(1) LLM-Agnostic Capability of CoF-CoT (Q1, RR1):**  \nOur proposed method is LLM-agnostic since the focus of the work is on the prompt design, which can be applied to any LLMs. This is further validated via additional empirical studies presented in Table A. As most LLMs (i.e. GPT-3, PaLM, etc.) rely on the quality of the designed prompts, our proposed CoF-CoT prompt design can be used as input to any LLMs for zero-shot and in-context learning. This is similar to CoT, SC-CoT and other methods. We provide a detailed Prompt Design Template in Figure 3 (Appendix) for future extensions towards other LLMs.  For further clarification, we report additional results of our proposed method over the backbone PaLM LLM [1] on the MTOP dataset [2] below (Zero-shot Learning (ZSL) and Few-shot Learning (FSL) results are reported in the respective order from left to right). The format of the table below is similar to our main result Table 1 in the manuscript. We will update the newly reported results in the final version.\n\n|  | Intent Acc | Slot F1 |Frame Acc | Exact Match | Intent Acc | Slot F1 |Frame Acc | Exact Match |\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|----------|\n| PaLM  | 16.67 $\\pm$ 2.52 | 7.24 $\\pm$ 1.00 |3.17 $\\pm$ 0.76 | 1.17 $\\pm$ 0.76 | 48.83 $\\pm$ 4.54 | 14.24 $\\pm$ 1.58  | 4.17 $\\pm$ 2.02  |2.33 $\\pm$ 0.76 |\n| **PaLM + CoF-CoT** | **42.33 $\\pm$ 3.33** | **13.73 $\\pm$ 2.88** | **4.01 $\\pm$ 0.15**  | **3.50 $\\pm$ 1.32** | **57.17 $\\pm$ 3.79** | **21.47 $\\pm$ 3.79** | **10.33 $\\pm$ 3.06** | **6.67 $\\pm$ 2.75** |\n\nTable A: Performance of CoF-CoT integration with PaLM LLM on MTOP dataset. \n \n**(2) Comparison with fine-tuning (FT) approaches of Pre-trained Language Models (PLM) (Q2, RR2):**  \nWe argue traditional FT (i.e. full parameter training as proposed in [2],[3]) and LLM ZSL/FSL approaches are not directly comparable. Traditional FT operates under closed-world assumption which requires sufficient data to learn domain-specific and domain-agnostic feature extraction in multi-domain settings. Without sufficient data in  ZSL/FSL settings, FT approaches might not be able to capture these features, leading to inferior performance, especially when the domain gap between training and testing is significant. On the other hand, LLM provides an open-domain approach where fine-tuning on specific domains is not needed. \n\n\nWe conducted a preliminary study and observed the aforementioned gap between the two approaches. For fair comparison between FT and LLM approaches, our study imposes the constraint that  there exist no overlapping domains between train and test domains under ZSL and FSL setting. We experimented with 3 different scenarios for FT approaches, including:\n\n* *Fully Supervised:* Samples sharing similar domains with test sets are used for training\n* *ZSL:* We use samples from domains different from test domains for training\n* *FSL:* We use samples from domains different from test domains in conjunction with a fixed number of  k-shot test domain samples. \n\nWe leverage Roberta PLM [4] with joint Slot Filling and Intent Detection objectives as presented in [2],[3] as the FT model. Under ZSL and FSL settings, the FT model suffers from the aforementioned domain gap issues. Specifically, as observed in Table B, since there exist minimal overlapping intent labels between train and test domains, without sufficient data in ZSL and FSL settings, the FT approaches are unable to learn transferable multi-domain features, leading to 0 performance in Intent Accuracy. This behavior also results in 0 performance for both Frame Acc and Exact Match as the correct intents are the prerequisites for correct semantic frame and exact match metrics. On the other hand, Fully supervised FT approach acquires domain-specific knowledge of target domains from training data and performs the best across different evaluation metrics. However, this assumption does not match ZSL/FSL settings. \n\n\n|  Method | Assumption | Intent Acc | Slot F1 |Frame Acc | Exact Match | \n|-----------|-----------|-----------|-----------|-----------|-----------\n| Roberta FT  | Fully Supervised | 67.19 $\\pm$ 2.90 | 75.17 $\\pm$ 1.08 |43.57 $\\pm$ 4.18 | 36.10 $\\pm$ 1.08 | \n| Roberta FT | ZSL| 0| 12.68 $\\pm$ 1.25 | 0 | 0|\n| Roberta FT | FSL| 0| 13.75 $\\pm$ 1.22 | 0 | 0|\n| GPT3.5 + CoF-CoT | ZSL| 57.67 $\\pm$ 2.75 | 23.47 $\\pm$ 4.09 |  14.33 $\\pm$ 1.52| 9.00 $\\pm$ 1.00|\n| GPT3.5 + CoF-CoT | FSL| 61.50 $\\pm$ 4.93 | 30.12 $\\pm$ 3.93 |  15.00 $\\pm$ 1.32| 11.00 $\\pm$ 1.61|\n\nTable B: Complete comparison between FT and LLM approaches on MTOP dataset. \n\n**Reference:** RR1: Reason to Reject 1, Q1: Question 1\n\n[1] Chowdhery et al., Palm: Scaling language modeling with pathways. 2022. arXiv preprint arXiv:2204.02311.\n\n[2] Li et. al, MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark. EACL 2021. \n\n[3] FitzGerald et al., MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. 2022. arXiv preprint arXiv:2204.08582.\n\n[4] Liu et. al, RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019. arXiv:1907.11692", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work conducts the first preliminary study of LLMs’ capability in multi-grained NLU tasks, which is an interesting topic to be explored. It clearly describes the background (multi-grained NLU, LLMs, CoT and structured representation) before introducing the details of methods.\n2. The method proposed in this paper is incremental, simple but reasonable, which decomposes the NLU tasks into multiple steps, and each step target at extracting certain information of the tasks.", "reasons_to_reject": "1.\tThe weakness of this paper is that the experimental analysis is not strong enough. See \"Questions For The Authors\".\n2.\tA crucial question should be answered in the paper. What is the advantage of applying LLMs to NLU tasks? How about the comparison with the existing deep learning based NLU methods? As this is the first try to applying LLMs to multi-domain NLU tasks. Only when this question is answered, the proposed method makes sense.", "questions_to_authors": "1. Page 7, line 508. The paper says ”our proposed methods is LLM-agnostic”. But the experiments only involve gpt-3.5-turbo without comparing with other LLMs. Could you add the analysis of different LLMs?\n2. Table 1. The experiments demonstrate different results of different prompts (zero/few-shot settings) without considering other related works. Could you add the results of other methods to make more comprehensive comparison? (e.g., full training. You can search the citations of MTOP[1], MASSIVE[2])\n\n [1] Li H, Arora A, Chen S, et al. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark[J]. arXiv preprint arXiv:2008.09335, 2020.\n [2] FitzGerald J, Hench C, Peris C, et al. Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages[J]. arXiv preprint arXiv:2204.08582, 2022.", "paper_topic_and_main_contributions": "This paper studies the application of Chain-of-Thought (CoT) to Large Language Models (LLMs) in Natural Language Understanding (NLU). Specifically, they propose Coarse-to-Fine CoT (CoF-CoT) approach to enable LLMs to acquire and leverage essential concepts to solve tasks from different granularities. Moreover, they integrate structured knowledge based on semantics via Abstract Meaning Representation (AMR), which helps to capture the nuances and diverse structures of utterances and understand connections between their varying levels of granularity. They evaluate the proposed method on two multi-domain NLU datasets (MTOP, MASSIVE) under both zero-shot and few-shot multi-domain settings. The results show that their method could improve LLMs’ capability in multi-grained NLU tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Page 2, line 105 and 117. You only use “Unlike previous works” to mention related works. Could you summarize their characteristics and therefore highlight the difference of your work?"}, "extra_text": {}}, {"review_id": "yM3C0YQlpW", "reviewer": "Reviewer_ajre", "summary": "", "questions": "1) Could the proposed method be combined with self-consistency?\n\nI will update the final score according to the authors responses.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We address your questions as follows. \n\n**(1) Consistency integration (Q1)**: Our CoF-CoT is independent from the proposal of self-consistency. In other words, it can be combined with self-consistency (by allowing multiple outputs from LLMs and selecting the most consistent one as the final output) or can stand alone by itself. However, as self-consistency is not our main claim or contribution, we focus on verifying the effectiveness of our proposed (1) multi-step reasoning from coarse-grained to fine-grained , (2) integrating structured knowledge from AMR graphs in multi-step reasoning for multi-grained NLU tasks.\n \n\n**(2) Zero-shot and Few-shot Settings setups (RR5):** For zero-shot learning (ZSL), the setup is as mentioned in the test set construction (Line 207-213). Regarding the few-shot learning (FSL) setup, our k-shot setup is presented in Line 513-516. An important note is that for FSL, the domains of k-shot are distinct from the domains of the test sets. We will clarify with additional details in our final manuscript regarding the setups for further transparency.  \n\n**(3) Further CoT baselines (RR4):** The major difference between our method and Least-to-Most prompting [1], Plan-and-Solve prompting [2] is that we present a controllable number of sequential steps and controllable conditioning inputs for each sequential step. Both Least-to-Most and Plan-and-Solve prompting rely heavily on the pre-trained knowledge of LLMs, leading to potential hallucinations and worse performance if LLMs are not familiar with the downstream tasks (in our case, multi-grained NLUs). Table A below presents our empirical study on MTOP dataset. As Plan-and-Solve was proposed for ZSL, we only report ZSL performance for this method. We will also update the completed results for the MASSIVE dataset in the final version.\n\nAs observed in our empirical study in Table A, under the same setup, our proposed CoF-CoT achieves consistent improvements on NLU tasks and comparable performance with the other prompting techniques on other semantic parsing metrics. \n\n|  | Intent Acc | Slot F1 |Frame Acc | Exact Match | Intent Acc | Slot F1 |Frame Acc | Exact Match |\n|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|----------|\n| Least-to-Most  | 45.67 $\\pm$ 0.58 | 21.84 $\\pm$ 1.91 | **14.50 $\\pm$ 0.50** | 8.00 $\\pm$ 0.50 | 49.83 $\\pm$ 4.54 | 27.28 $\\pm$ 2.41  | **16.00 $\\pm$ 0.50**  |8.83 $\\pm$ 0.76 |\n| Plan-and-Solve  | 45.00 $\\pm$ 4.00 | 22.45 $\\pm$ 2.28 |9.50 $\\pm$ 1.61 | 8.25 $\\pm$ 2.25 |--|--|--|--| \n| **CoF-CoT** |  **57.67 $\\pm$ 2.75** | **23.47 $\\pm$ 4.09** |  14.33 $\\pm$ 1.52| **9.00 $\\pm$ 1.00** | **61.50 $\\pm$ 4.93** | **30.12 $\\pm$ 3.93** |  15.00 $\\pm$ 1.32| **11.00 $\\pm$ 1.61**|\n\nTable A: Performance of additional relevant CoT baselines and the proposed CoF-CoT on MTOP dataset.\n\n\n**(4) Comparison with fine-tuning (FT) approaches of Pre-trained Language Models (PLM) (RR3):**  \nWe argue traditional FT (i.e. full parameter training as proposed in [3],[4]) and LLM ZSL/FSL approaches are not directly comparable. Traditional FT operates under closed-world assumption which requires sufficient data to learn domain-specific and domain-agnostic feature extraction in multi-domain settings. Without sufficient data in  ZSL/FSL settings, FT approaches might not be able to capture these features, leading to inferior performance, especially when the domain gap between training and testing is significant. On the other hand, LLM provides an open-domain approach where fine-tuning on specific domains is not needed. \n\n\nWe conducted a preliminary study and observed the aforementioned gap between the two approaches. For fair comparison between FT and LLM approaches, our study imposes the constraint that  there exist no overlapping domains between train and test domains under ZSL and FSL setting. We experimented with 3 different scenarios for FT approaches, including:\n\n* *Fully Supervised:* Samples sharing similar domains with test sets are used for training\n* *ZSL:* We use samples from domains different from test domains for training\n* *FSL:* We use samples from domains different from test domains in conjunction with a fixed number of  k-shot test domain samples. \n\nWe leverage Roberta PLM [5] with joint Slot Filling and Intent Detection objectives as presented in [3],[4] as FT model. Under ZSL and FSL settings, the FT model suffers from the aforementioned domain gap issues. Specifically, as observed in Table B, since there exist minimal overlapping intent labels between train and test domains, without sufficient data in ZSL and FSL settings, the FT approaches are unable to learn transferable multi-domain features, leading to 0 performance in Intent Accuracy. This behavior also results in 0 performance for both Frame Acc and Exact Match as the correct intents are the prerequisites for correct semantic frame and exact match metrics. On the other hand, Fully supervised FT approach acquires domain-specific knowledge of target domains from training data and performs the best across different evaluation metrics. However, this assumption does not match ZSL/FSL settings. \n\n|  Method | Assumption | Intent Acc | Slot F1 |Frame Acc | Exact Match | \n|-----------|-----------|-----------|-----------|-----------|-----------\n| Roberta FT  | Fully Supervised | 67.19 $\\pm$ 2.90 | 75.17 $\\pm$ 1.08 |43.57 $\\pm$ 4.18 | 36.10 $\\pm$ 1.08 | \n| Roberta FT | ZSL| 0| 12.68 $\\pm$ 1.25 | 0 | 0|\n| Roberta FT | FSL| 0| 13.75 $\\pm$ 1.22 | 0 | 0|\n| GPT3.5 + CoF-CoT | ZSL| 57.67 $\\pm$ 2.75 | 23.47 $\\pm$ 4.09 |  14.33 $\\pm$ 1.52| 9.00 $\\pm$ 1.00|\n| GPT3.5 + CoF-CoT | FSL| 61.50 $\\pm$ 4.93 | 30.12 $\\pm$ 3.93 |  15.00 $\\pm$ 1.32| 11.00 $\\pm$ 1.61|\n\nTable B: Complete comparison between FT and LLM approaches on MTOP dataset. \n\n**(5) Backgrounds of NLU Tasks and CoT (RR1, RR2):** We will update further details of the NLU and CoT backgrounds in the Related work in our final version. \n\n**Reference:** RR1: Reason to Reject 1, Q1: Question 1\n\n[1] Zhou et al., Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ICLR 2023.\n\n[2] Wang et al., Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. ACL 2023.\n\n[3] Li et. al, MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark. EACL 2021.\n \n[4] FitzGerald et al., MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. 2022. arXiv preprint arXiv:2204.08582. \n\n[5] Liu et. al, RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019. arXiv:1907.11692.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed idea is interesting and reasonable.\n2) The paper is easy to follow", "reasons_to_reject": "1) The paper lacks content on related work regarding NLU, Multi-grained NLU, Multi-grained NLU of dialogue systems, and Multi-domain NLU.\n\n2) The related work on CoT is not well written.\n\n3) This paper lacks baselines with full parameter tuning or non-CoT baselines.\n\n4) The paper does not compare to multi-step prompting methods like the few-shot based least-to-most prompting [1] and the zero-shot based plan-and-solve prompting [2].\n\n5) It is not clear how to implement the proposed method in zero-shot and few-shot scenarios.", "questions_to_authors": "1) Could the proposed method be combined with self-consistency?\n\nI will update the final score according to the authors responses.", "paper_topic_and_main_contributions": "The paper discusses Natural Language Understanding (NLU) in Dialogue systems and the challenges associated with different granularities of NLU tasks. It proposes an adaptation of the Chain-of-Thought (CoT) approach to address multi-grained NLU tasks in a coarse-to-fine-grained sequential reasoning order. The authors also suggest integrating structured knowledge represented via Abstract Meaning Representation (AMR) Graph to enhance the reasoning capabilities of Large Language Models (LLMs) in Dialogue systems.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 34: mutli-step -> multi-step\n2) Line 25&42: understanding -> the/an understanding"}, "extra_text": {}}, {"review_id": "3r3wa0uRQe", "reviewer": "Reviewer_ZdgB", "summary": "", "questions": "1. As mentioned in weakness, will the proposed CoF-CoT still work when the intermediate ground truths are not available?\n2. What is the importance of each reasoning step (e.g., What if I remove one of the first four step)?\n3. Have you conducted any experiments for the selection of k-shot examples? For example, what if I select examples that are within the same domain of the test example?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We would like to clarify a few major points:\n\n**(1) Intermediate Ground Truth requirements (Q1, RR):** Generally speaking, our CoF-CoT does not require intermediate ground truths to work. \n\n* For Zero-shot Learning (ZSL), the input is the utterance sample. Each step (from step 2) of our proposed CoF-CoT leverages the output from the previous step as an additional source of input information. This is simply the benefits when we leverage multi-step reasoning. Throughout this process, no intermediate ground truth labels (i.e. Abstract Meaning Representation (AMR) Graph) are needed or provided. \n\n* For Few-shot Learning (FSL),  the input is the utterance together with a fixed number of k-shot demonstration samples  (k=5 in our study) for in-context learning. Similar to other CoT approaches, we prepare labels of individual steps for a fixed number of k-shot demonstration samples in order to represent the in-context samples in a multi-step reasoning process. In our case, we only need to generate AMR Graph for k demonstration samples as presented in Line 513-516. This procedure is simple and efficient in our case since the k-shot demonstration samples are fixed and the AMR graph can be auto-generated by external tools. \n\n\nThe gap between ZSL and FSL evaluation does not purely come from the inclusion of intermediate ground truths. It is also due to the introduction of in-context demonstration samples, which helps (1) LLMs familiarize themselves with the downstream tasks in terms of expected output, format, etc. (2) LLMs acquire multi-step reasoning procedure knowledge from in-context samples and apply it to test samples. \n\n**(2) Importance of individual reasoning steps (Q2):** We conducted ablation studies to study the importance of multiple steps in our framework. Firstly, removing all of the first four steps will be equivalent to Direct Prompting as shown in Table 1.  Removing step 1 is equivalent to discarding the leverage of AMR Graph, which is demonstrated in Table 3. As the remaining steps are more coherent and produce necessary outputs for the aggregation step, instead of removing these steps, we verified the importance of the designed sequential ordering  by conducting additional experiments in (1) reversing the CoF to FoC order, and (2) random ordering as presented in Table 3 and Line 263-275.\n\n **(3) K shot demonstration/ example selection (Q3):** Although we hypothesize and empirically validate in Table A that additional constraint of similar domains between k-shot demonstration samples and test samples could help further improve the evaluation performance, this setting might be unrealistic as the domains of test samples are generally unknown. For broader applications, we relax this constraint in our settings so that LLMs need to accumulate and extract the true semantic knowledge from k-shot demonstrations and avoid overfitting any specific domains. In our setting, k-shot demonstrations are from domains that are different from the test samples.    \n\n|  Method | Assumption | Intent Acc | Slot F1 |Frame Acc | Exact Match | \n|-----------|-----------|-----------|-----------|-----------|-----------\n| CoF-CoT | k domain-different samples| 61.50 $\\pm$ 4.93 | 30.12 $\\pm$ 3.93 |  15.00 $\\pm$ 1.32| 11.00 $\\pm$ 1.61|\n| CoF-CoT | k domain-similar samples| 70.00$\\pm$ 1.33 | 38.16 $\\pm$ 5.42 |  20.50 $\\pm$ 2.00 | 15.00 $\\pm$ 1.00|\n\nTable A: FSL results of CoF-CoT with k-shot demonstration samples selected from different and similar domains in comparison with test samples on MTOP dataset.\n\n**Reference:** RR1: Reason to Reject 1, Q1: Question 1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using CoT for multi-step reasoning on LLMs is a novel an interesting problem, and the paper propose a success exploration in the NTU domain.\n2. According to the paper, CoF-CoT has a significant improvement in performance compared to other variants of CoT.", "reasons_to_reject": "For MTOP and MASSIVE, ground truths of intermediate reasoning results are provided, which make the few-shot learning of CoF-CoT possible. However, as shown in the paper, currently all the CoT methods (including CoF-CoT) perform poorly under zero-shot setting, the method may have limited ability when a new scenario where the intermediate ground truths are not available.", "questions_to_authors": "1. As mentioned in weakness, will the proposed CoF-CoT still work when the intermediate ground truths are not available?\n2. What is the importance of each reasoning step (e.g., What if I remove one of the first four step)?\n3. Have you conducted any experiments for the selection of k-shot examples? For example, what if I select examples that are within the same domain of the test example?", "paper_topic_and_main_contributions": "This paper propose a novel approach called CoF-CoT, which is a modification of CoT that enables multi-step reasoning for natural language understanding in dialogue. The core idea of CoF-CoT is to prompt the large language model (LLM) to perform multiple reasoning steps from coarse to fine-grained levels. This includes generating abstract meaning representation, generate intention, generate slot values, generate slot value-type pairs, and generate logic form. The paper evaluates CoF-CoT on two datasets and shows that it outperforms previous CoT variants. The paper also analyzes the impact of different structured representations and step orders on the reasoning performance and demonstrates the benefits of a coarse-to-fine-grained reasoning process for LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the application of Chain-of-Thought (CoT) to Large Language Models (LLMs) in Natural Language Understanding (NLU). The proposed CoF-CoT approach enables LLMs to acquire and leverage essential concepts for solving tasks at different granularities. The authors also integrate structured knowledge via Abstract Meaning Representation (AMR) to capture the nuances and diverse structures of utterances. The paper evaluates the method on two multi-domain NLU datasets and shows improvement in LLMs' capability in multi-grained NLU tasks.

The reviewers generally find the topic and main contributions of the paper interesting. The proposed method is considered reasonable and incremental. However, there are concerns about the experimental analysis, lack of strong baselines, and comparisons with other methods. Reviewers also suggest exploring the advantages of different LLMs and providing more comprehensive comparisons with related works. Additionally, there are questions about implementing the proposed method in zero-shot and few-shot scenarios and its applicability when intermediate ground truths are not available.

Overall, the reviewers acknowledge the merits of the paper, such as reporting state-of-the-art results and presenting a novel idea. However, there are also key weaknesses, and the paper would benefit from another round of revision.
Decision: Accept-Main