MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation
Year: 2023

Abstract:
Prompting large language models has enabled significant recent progress in multi-step reasoning over text. However, when applied to text generation from semi-structured data (e.g., graphs or tables), these methods typically suffer from low semantic coverage, hallucination, and logical inconsistency. We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. MURMUR is a best-first search method that generates reasoning paths using: (1) neural and symbolic modules with specific linguistic and logical skills, (2) a grammar whose production rules define valid compositions of modules, and (3) value functions that assess the quality of each reasoning step. We conduct experiments on two diverse datato-text generation tasks like WebNLG and LogicNLG. The tasks differ in their data representations (graphs and tables) and span multiple linguistic and logical skills. MURMUR obtains significant improvements over recent few-shot baselines like direct prompting and chain-ofthought prompting, while also achieving comparable performance to fine-tuned GPT-2 on out-of-domain data. Moreover, human evaluation shows that MURMUR generates highly faithful and correct reasoning paths that lead to 26% more logically consistent summaries on LogicNLG, compared to direct prompting.1

heading: 1 Introduction
text: Data-to-text generation (McKeown, 1992; Reiter and Dale, 1997; Wen et al., 2015; Dušek and Jurcicek, 2015; Mei et al., 2016; Novikova et al., 2017; Gatt and Krahmer, 2018) is the task of generating fluent, faithful, and consistent summaries of semi-structured data. Recent works have introduced different data-to-text generation tasks
1Supporting code available at https://github.com/ swarnaHub/MURMUR

heading: Table Topic: Reinhold Roth
text: wherein the data is represented in diverse structures, like meaning representations (Novikova et al., 2017), graphs (Gardent et al., 2017), or tables (Lebret et al., 2016; Parikh et al., 2020; Chen et al., 2020a). Text generation from such data is challenging because it extends surface realization of the input content and requires various reasoning and compositionality skills, such as filtering a table based on a certain criterion, retrieving the maximum value from a table column, etc.
Existing works fine-tune pre-trained language models (Radford et al., 2019; Raffel et al., 2020) as the de-facto standard for building supervised
11069
data-to-text generation systems (Kale and Rastogi, 2020; Agarwal et al., 2021). However, this requires a large amount of domain-specific parallel data, which is expensive to obtain, and training models on such data also affects out-of-domain generalization (Laha et al., 2020; Dušek et al., 2020).
Motivated by the recent success of few-shot prompting in multi-step reasoning over text (Wei et al., 2022; Nye et al., 2021; Wang et al., 2022a; Dohan et al., 2022), we pose data-to-text generation as multi-step reasoning over data.2 Reasoning over data for text generation brings its own set of challenges: (1) Generation Quality: Firstly, directly prompting large language models (LLMs) can cause models to suffer from low semantic coverage, hallucinations, and logically inconsistent generations (see red marked phrases for the Direct Prompting summaries in Fig. 1). Other prompting methods like Chain-of-Thought (CoT) encourage LLMs to also generate intermediate reasoning steps (Wei et al., 2022) but it compromises the transparency, faithfulness,3 and correctness of the reasoning process due to the lack of explicit conditioning between the reasoning steps (Creswell and Shanahan, 2022). (2) Transformation-invariance: Text is a sequence of tokens while data is typically represented as a set of elements (e.g., a graph is a set of edges, a table is a set of rows, etc). Hence, a model that reasons over data must be transformation-invariant (Wang et al., 2022a). For instance, the summary generated from a table should be invariant to randomly shuffling the rows of the table. Thus, prompting methods that linearize the data in an arbitrary order, can be prone to some variance (see Table 3 and 6).
We propose MURMUR, a few-shot Modular Multi-step Reasoning approach to text generation from data (§3). It is a best-first search algorithm (§3.4) that generates reasoning paths (see examples in Fig 1) with three features: (1) Modularity (§3.1): MURMUR defines a set of few-shot neural and symbolic modules with diverse input/output data types that constitute multiple steps in a reasoning path. Neural modules perform linguistic skills that LLMs are good at (e.g., the Surface Realization module in Fig. 1 converts a reasoning path to a natural language summary) and symbolic mod-
2By data, we mean semi-structured data such as graphs or tables. Henceforth, we will refer to it as just ‘data’.
3Faithful reasoning refers to an underlying causal structure in the reasoning process. This is different from a text’s faithfulness to an input context which will be called hallucinations.
ules perform logical skills that they mostly struggle with (Wang et al., 2022b; Gao et al., 2022) (e.g., the argmin module in Fig. 1 finds the row with the minimum points); (2) Grammar (§3.2): MURMUR introduces a grammar whose production rules specify valid compositions of modules. For instance, in the second path of Fig. 1, MURMUR first generates the module filter_eq followed by the avg module, because the former outputs a table data type which is also the input data type to the latter; (3) Value functions (§3.3): To evaluate the quality of each plausible reasoning step and choose the best modules at each step, MURMUR defines value functions that score, rank, and select the best steps. For example, in the second path of Fig. 1, an avg module is perhaps more salient than a max or min module (which only finds the maximum or minimum points).
Our findings are: MURMUR can perform multistep generative reasoning on simple to complex semi-structured data-to-text generation tasks including WebNLG (Gardent et al., 2017), a graph-totext task (§5) and LogicNLG (Chen et al., 2020a), a table-to-text task (§6). We compare MURMUR with state-of-the-art supervised (end-to-end and pipeline) and few-shot prompting methods. On WebNLG, MURMUR obtains significant improvements in semantic coverage and hallucinations of generated summaries over other few-shot baselines like direct prompting and CoT prompting. Additionally, MURMUR demonstrates good out-ofdomain generalizability by obtaining comparable performance to fine-tuned LMs like GPT-2. On LogicNLG, human evaluation demonstrates that MURMUR significantly improves the logical consistency of summaries over direct prompting (by up to 26%), showcasing the strength of a neurosymbolic approach for data-to-text generation.

heading: 2 Definitions: Reasoning Step and Path
text: A Reasoning Step is a triple (M,X , y) where a module M performs a certain skill by conditioning on an input X to generate an output y. For example, in Fig. 2, the module argmin takes a table and a column (points) as input and outputs the row with the minimum points. A Reasoning Path is defined as a sequence of such reasoning steps {(Mi,Xi, yi)}ri=1. Fig. 2 shows an example of a reasoning path, represented as a nested structure. It consists of three reasoning steps for three modules (argmin, hop, and eq). The argmin module outputs
the row in the table with minimum points, which is the input to the next module hop that selects a column from that row. MURMUR generates textual summaries by constructing such reasoning paths that are then converted to the final outputs through a Surface Realization module, as shown in Fig. 2.4

heading: 3 MURMUR Approach
text: MURMUR consists of four components: (1) a set of modules, (2) a grammar, (3) value function(s), and (4) a search algorithm that brings all the previous three components together. The search algorithm constructs reasoning paths by first identifying plausible modules at each reasoning step according to the grammar and then determining the best modules (and their corresponding inputs) with the help of value functions. Fig. 2 shows a working example of MURMUR, in which given an input table, it searches for a reasoning path (of three steps), and finally converts it into a summary. The specifics of MURMUR’s components vary based on the task at hand. As case studies, we consider two datato-text generation tasks: WebNLG (Gardent et al., 2017), a graph-to-text generation task and LogicNLG (Chen et al., 2020a), a complex table-to-text generation task where the goal is to generate logical summaries from salient parts of the table.

heading: 3.1 MURMUR Modules
text: MURMUR defines a set of modules {Mi}mi=1 that perform specialized reasoning skills for the corresponding task. Formally, each module Mi is defined as a multi-variate function Mi : X → y that maps an n-tuple input X = (x1,· · · , xn) to an
4For better illustration, we show Surface Realization outside of the reasoning path but ideally, it can be considered as another (final) step in the reasoning path.
output y. Each input variable xi and output y can have their own expected data types di and dy respectively. These data types could be user-defined5 like Table, Triple, etc or standard ones like String, Number, Bool, etc. For example, in Fig. 2, the module Margmin : (t, c) → r takes a table t (with data type Table) and a column c (with data type String) as input and outputs a row r (with data type row) with the minimum value in column c. The modules are implemented as few-shot neural models or symbolic functions. We choose few-shot neural modules for linguistic skills that LLMs typically excel at and symbolic modules for logical operations that LLMs mostly struggle with (Wang et al., 2022b; Gao et al., 2022). Reasoning over semistructured data allows us to implement symbolic modules with PYTHON functions. Below we provide examples of neural and symbolic modules for the two tasks.
Neural Linguistic Modules. In any modular data-to-text generation approach, one of the modules is responsible for the transition from structured data to unstructured text. We call it Surface Realization. In particular, for WebNLG, we define it as Msr : t → s that converts a triple t (with data type Triple) into a short sentence s (with data type String). For LogicNLG, we define it as Msr : (t, p) → s that takes a table t (with data type Table) and a reasoning path p as input and converts it into a summary s (with data type String). As we show later, in WebNLG, Surface Realizations are the first reasoning steps, while in LogicNLG, it
5The modules are analogous to function definitions with expected IO types. Similarly, user-defined data types can be thought of as class definitions. For instance, a data type Triple can be implemented as a class consisting of a subject, a relation, and an object (all with data type String).
is the last step. For WebNLG, we also define a Text Fusion module Mtf : (s1, s2) → s that combines two strings s1 and s2 into a coherent text s. Text Fusion iteratively combines intermediate generations at each step, enabling more controllability in generation (Tan et al., 2021).
Symbolic Logical Modules. For LogicNLG, drawing motivation from prior work (Chen et al., 2020c), we define different categories of symbolic modules that perform logical operations over tables (see Table 1 and refer to Table 8 for the detailed list). WebNLG requires summarizing an input graph and hence, does not involve any logical modules.

heading: 3.2 Grammar over Modules
text: The role of the grammar is to determine a set of plausible modules in a reasoning step and how they should be composed. The production rules of the grammar capture possible transitions from an input data type to an output data type(s) (see Fig. 2 and Table 2). Each production rule thus defines multiple permissible modules. For example, the production rule ‘Table → Number’ (meaning that a number can be generated from a table) is valid for both max and min modules. When MURMUR searches for reasoning paths, the grammar reduces the search space (over all possible modules) by only selecting the ones that can be composed at each reasoning step. We provide examples below of how such grammars are constructed.
Grammar for Graph-to-Text Generation. Table 2 shows the grammar for Graph-to-Text generation. It consists of two production rules, one for Surface Realization and another for Text Fusion. Past pipeline approaches for graph-to-text generation (Xiang et al., 2022) also perform sur-
face realization followed by fusion, as explained through the grammar.
Grammar for Table-to-Text Generation. Generating logical summaries from a table is a more challenging task. Based on the types of modules introduced previously, we define a grammar, as shown in Table 2. As an instance, the first rule encodes the knowledge that given an input of type Table, one can output a Table, a Row of the table, a Number, or a Boolean.

heading: 3.3 Value Functions
text: While the grammar helps reduce the search space by defining permissible compositions of modules, each reasoning step can still have multiple plausible modules and each module can also have multiple plausible inputs to choose from. Thus, MURMUR introduces value functions (see Fig. 2) that assess the quality of each plausible reasoning step by scoring, ranking, and selecting the best step(s).

heading: Value Function for Graph-to-Text Generation.
text: In a Graph-to-Text generation task, each intermediate reasoning step r generates a summary yr for a subset of edges (triples) Gr from the input graph (see Fig. 7 for an illustration). The value functions evaluate the following two aspects of the generated summary yr. First, Fluency is measured by log-likelihood of the generated text similar to BARTScore (Yuan et al., 2021):
Sf (yr) = exp{ 1
l
l∑
i=1
log pθ(y i r|y<ir )}
Second, Semantic Consistency measures the average logical entailment probability Pe(·) between the generation yr and triples Gr6 and vice-versa:
Ssc(Gr, yr) = 0.5× (Pe(Gr, yr) + Pe(yr, Gr))
We use an NLI model to compute entailment probabilities. The both-way entailment scores capture
6We concatenate the surface realizations of the triples to construct the sequence for the NLI model.
equivalence between the triples and the generation, ensuring that the latter not only covers all the triples but also does not hallucinate any new information. Overall score is an ensemble of the two scores, given by αSf (yr) + (1− α)Ssc(Gr, yr).

heading: Value Function for Table-to-Text Generation.
text: Our value function chooses the best module(s) at each reasoning step, as well as the best input(s) for the corresponding module(s).7 For instance, if a reasoning step generates a number from a table (according to the grammar), the value function should determine the best module(s) between max, min, etc, as well as which column the max or min module should be computed on. Taking inspiration from past work on verifying intermediate reasoning traces over text (Creswell and Shanahan, 2022; Yang et al., 2022), we train a value function S : (T, Pr) → p that judges the correctness of a partial reasoning path Pr for an input table T . In particular, we train a binary classifier on samples with correct and incorrect partial reasoning paths. We call this value function a saliency metric because it selects the best reasoning steps that reason over salient parts of the table. We discuss the model and training data for our saliency metric in § 4.2.

heading: 3.4 Search Algorithm
text: We now describe how all the three components discussed above come together in generating reasoning paths for MURMUR (see Fig. 2). We propose a best-first search algorithm that operates as follows. It takes as input a set of m modules {Mi}mi=1, a grammar G, a value function V , and number of reasoning paths or summaries to generate p. Additionally, it considers a hyperparameter, the beam size b of the search (b ≥ p). The search begins by initializing an empty priority queue that maintains the beam (best b partial reasoning paths to be explored anytime during the search). Next, at each step, MURMUR (1) pops an element from the queue, (2) identifies the data type of the element (e.g., Table), (3) looks up the grammar to find all possible transitions from that data type (e.g., Row, Number), (4) selects all modules for each such transition (e.g., argmax and argmin for ‘Table → Row’, max and min for ‘Table → Number’), and (5) constructs all plausible reasoning steps consisting of modules and their corresponding inputs (e.g., all numerical columns for argmax). It then scores all these rea-
7In Graph-to-Text, we only need to choose the best inputs because at each step there is only one plausible module.
soning steps using the value function, ranks them, and only keeps the top-b paths in the queue. For WebNLG, the search terminates when all triples have been iterated. For LogicNLG, a reasoning path is complete when the current module outputs a boolean variable that evaluates to true (e.g., the eq module). Upon termination of the search, we return the top-p paths and the corresponding summaries.

heading: 4 Experimental Setup

heading: 4.1 Graph-to-Text Generation
text: We report results on both seen and unseen splits of the test set of WebNLG (Gardent et al., 2017).8
Modules. We implement both modules, Surface Realization and Text Fusion as few-shot neural models by prompting OPT-175B (Zhang et al., 2022) with skill-specific prompts (see Appendix D) and greedy decoding.
Value Function. As defined in §3.3, we compute fluency using the log probabilities estimated by OPT-175B. The entailment probability for the semantic scorer is based on a DeBERTa-base model (He et al., 2020) trained on a collection of eight NLI datasets.9 The mixing ratio α is set to 0.05. At each reasoning step, MURMUR scores and ranks the intermediate generations in the queue using the value function. Subsequently, it only explores the highest scoring intermediate generation in the next step of the search and prunes the rest.10

heading: 4.2 Table-to-Text Generation
text: Modules. We implement all logical modules, as described in §3.1, with PYTHON functions. We again prompt OPT-175B for the Surface Realization module (see Appendix D for the prompt).
Value Function. Our saliency metric is a binary classifier. Specifically, we train a BERT-base model that takes a table (as a sequence of rows) and a partial reasoning path as input and classifies it as correct or incorrect. During inference, we consider the correct class probability as the saliency score. We obtain training data from the Logic2Text dataset (Chen et al., 2020c) that annotates opendomain tables with gold reasoning paths. Given a
8The 2017 version of the dataset is available at https: //webnlg-challenge.loria.fr/challenge_2017/.
9https://huggingface.co/MoritzLaurer/ DeBERTa-v3-base-mnli-fever-docnli-ling-2c.
10This is equivalent to performing greedy search. We experimented with larger beams for a more exhaustive search without observing any noticeable improvement in performance.
gold reasoning path, we create correct partial paths by breaking it at each intermediate step/module and incorrect paths by performing two types of perturbations on every correct partial path: (1) replacing the module at the current step with another module of same data type (e.g., replacing module max with module min); (2) replacing the inputs to the module with other plausible inputs (e.g., replacing max over column c1 with max over another column c2). See Appendix C.4 for an illustration of the training data creation process. We choose 221 (table, reasoning path) pairs from the Logic2Text dataset and convert them into 1500 correct and incorrect training samples consisting of (table, partial reasoning path) pairs. While choosing the samples, we ensure that the corresponding tables have no overlap with those in the test and validation sets of LogicNLG. We choose the beam size of the search to be 20 (see further analysis of beam sizes in Appendix C.3).

heading: 5 Experiments on Graph-to-Text

heading: 5.1 Comparison of MURMUR with supervised and few-shot methods
text: Baselines. We compare with both supervised and few-shot baselines. • Supervised. We compare with MELBOURNE, a
non-pretrained encoder-decoder model (Gardent et al., 2017) and two fine-tuned LMs, GPT-2large (Radford et al., 2019) and T5-large (Raffel et al., 2020). We also compare with a SOTA modular pipeline approach, Neural Pipeline (Kasner and Dušek, 2022) that first converts triples to sentences using hand-designed templates and subsequently orders and fuses the sentences by fine-tuning on a large synthetic corpus of 934k samples. • Few-shot. For direct comparisons, we consider two few-shot baselines, Direct Prompting (DP)
that directly prompts the OPT-175B model to generate a summary of the graph, and Chainof-Thought Prompting (CoT) (Wei et al., 2022) that prompts the model to generate the summary step-by-step (see Appendix D for prompts). We choose the demonstrations randomly from the training data and keep them consistent across all few-shot methods.
Metrics. Following prior work, we perform automatic evaluation using BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005).
Results. For all few-shot methods, we report mean and variance of three random triple orderings. Table 3 shows the results. MURMUR significantly outperforms DP and CoT by up to 8 points in BLEU and METEOR (p < 0.001), when using a single demonstration (k=1).11 MURMUR even outperforms DP with five demonstrations (k=5). Prompting an LLM by simply concatenating the intermediate steps for CoT does not work well for text generation. MURMUR also outperforms a supervised baseline MELBOURNE and obtains comparable performance to fine-tuned models like GPT-2 on the unseen test split. Through its modular treatment, MURMUR generates outputs with more coverage of triples and lesser hallucinations, as reflected in the improved scores and further demonstrated in §5.2 through human evaluation. Finally, MURMUR is transformation-invariant because it treats the graph as a set (not sequence) of triples. Refer to Appendix B for experiments studying the number and variation in demonstrations.
11A single demonstration for DP can get decomposed into multiple in-context examples in MURMUR. However, like CoT, the decompositions are all from the same demonstration.

heading: 5.2 Human Evaluation of Final Summaries and Intermediate Reasoning Steps
text: Evaluation of Final Summaries. We compare the summaries generated by DP (our best baseline) and MURMUR. Two NLP experts take part in the study with 50 randomly chosen test samples (having an average of 3.8 triples). They count the number of omissions, hallucinations, and disfluencies in the generated outputs.12 Our results in Table 4 demonstrate that MURMUR benefits significantly from a step-wise generative process and reduces omissions by 24% and hallucinations by 9%. We do observe a slight drop in fluency in MURMUR’s generations because of its iterative fusion process.

heading: Evaluation of Intermediate Reasoning Steps.
text: We also evaluate the quality of the individual reasoning steps of MURMUR. For every reasoning step of a data point, we provide the annotators with the (1) generation, and (2) the previous steps that the current step is conditioned on. We conduct this study on six randomly chosen test examples, spanning 50 reasoning steps (28 Surface Realization and 22 Text Fusion). Annotators judge the generations for their grammaticality, module faithfulness (i.e., if the module is doing what it is supposed to do), and correctness (e.g., whether the fusion is correct). From Table 5, we conclude that both modules are almost always grammatical, and highly faithful and 64% of fusion operations are also fully correct.

heading: 6 Experiments on Table-to-Text

heading: 6.1 Comparison of MURMUR with supervised and few-shot methods
text: Baselines. We compare with several nonpretrained and pretrained supervised methods as well as few-shot methods. • Non-pretrained Supervised. We compare MUR-
MUR with a non-pretrained transformer model, Field-Infusing + Trans (Chen et al., 2020a).
12All metrics including hallucinations and disfluencies are counted at the level of triples or facts. This allows us to compare the raw counts of both methods with respect to the average number of triples in the input samples.
• Pretrained Supervised. Next, we compare with three pre-trained LMs based on BERT and GPT-2, BERT-TabGen, GPT-TabGen, GPTCoarse-to-Fine (Chen et al., 2020a) and a deconfounded variational encoder-decoder model, DCVED (Chen et al., 2021). • Few-shot. We also compare with Direct Prompting (DP) and CoT Prompting. Additionally, we evaluate the effect of our search algorithm and saliency metric. First, in BART + SR, instead of searching for reasoning paths, we fine-tune a BART model that generates reasoning paths in one go. As training data, we leverage the (table, gold reasoning path) pairs that are used for training the saliency metric. The surface realization (SR) step is left unchanged. Second, we remove the saliency metric by randomly selecting a module at each step (but according to the grammar). All few-shot methods use one random demonstration.
Metrics. Following Chen et al. (2020a), we compare all methods with BLEU scores. They also propose metrics to evaluate logical consistency but we found such learned metrics do not correlate well with humans. Instead, we conduct more reliable human evaluations of logical correctness in §6.2.
Results. Table 6 shows the results on the test set of LogicNLG. MURMUR significantly improves upon DP and CoT prompting by up to 2.4 points in BLEU-3 (p < 0.001). We attribute this to two factors: (1) leveraging symbolic modules for logical skills that ensure their correctness, (2) delegating the task of converting a path to natural language to an LLM. Both CoT and BART+SR, while generating intermediate reasoning paths, do not use executable modules and hence cannot guarantee valid compositionality or logical correctness of the reasoning steps. MURMUR also improves upon the supervised Field-Infusing model. Finally, MURMUR obtains some improvement with the saliency metric, indicating that it helps in choosing more salient paths. Refer to Appendix C for studies on the number and variation in demonstrations.

heading: 6.2 Human Evaluation of Logical Correctness
text: Next, we conduct human evaluation to compare the logical correctness of the generations from DP and MURMUR. Two NLP experts annotate 40 randomly chosen generations from eight different tables. In particular, they take part in two studies. First, they classify each generation into whether it is (a) ungrammatical, (b) grammatical but incorrect, (c) grammatical but partially correct, or (d) grammatical and also fully correct. Next, for each fully correct generation, they annotate whether it involves any underlying logical operation (like counting, summation, etc) or are mere surface realizations of the table content. We observe from Table 7 that MURMUR not only generates 26% more correct outputs, but about 95% of those generations also involve some logical operations. In summary, MURMUR is most beneficial in two scenarios: (1) generations that require many steps of reasoning, (2) generations that require logical reasoning. The first capability comes from the fact that MURMUR is specifically designed to compose multiple steps of reasoning through its grammar and value functions. The second benefit is because of the presence of symbolic modules that ensure logical correctness. These two capabilities are specifically required in long complex tables involving numerical columns where there is a need to summarize
content (e.g., by filtering, averaging a numerical column, etc). Generating reasoning paths through logical modules ensures that almost all generations are logical derivations from the table, an ability that is significantly harder to achieve through direct prompting. See Fig. 8 in the appendix for an illustrative example of the generations of MURMUR for long complex tables.

heading: 7 Related Work
text: Multi-step Reasoning over Text. Recent developments in LLMs (Brown et al., 2020; Zhang et al., 2022; Thoppilan et al., 2022; Chowdhery et al., 2022) have enabled significant progress in fewshot methods for logical reasoning tasks (Wei et al., 2022; Creswell et al., 2022; Nye et al., 2021; Wang et al., 2022c; Zelikman et al., 2022; Zhou et al., 2022; Dasgupta et al., 2022; Kojima et al., 2022; Dohan et al., 2022). Representative methods like CoT prompting output intermediate reasoning steps before generating the final answer. However, the reasoning steps are all generated in one go from a single model, potentially leading to unfaithful reasoning due to the lack of explicit conditioning between the steps (Creswell and Shanahan, 2022). MURMUR overcomes this issue by developing granular modules that are capable of performing specialized skills by explicitly conditioning on the outputs from previous reasoning steps. Conceptually, MURMUR bears similarity with the SelectionInference modular architecture (Creswell et al., 2022; Creswell and Shanahan, 2022). However, their focus is on QA and reasoning over textual context (Saha et al., 2020, 2021b; Tafjord et al., 2021; Dalvi et al., 2021; Bostrom et al., 2022). A few concurrent works have also proposed neuro-symbolic approaches for reasoning over text (Gao et al., 2022; Wang et al., 2022b; Chen et al., 2022; Cheng et al., 2022). Different from these, we tackle a more challenging setup of multi-step reasoning for controlled generation from semi-structured data.
Modular Reasoning over Text. Neural Module Networks learn and execute compositional programs over modules (Andreas et al., 2016; Jiang and Bansal, 2019; Gupta et al., 2020; Subramanian
et al., 2020; Saha et al., 2021a). While their modules typically output attention maps, prior works have also used text-in text-out modules whose input/output data types are strings (Khot et al., 2021, 2022; Saha et al., 2022). MURMUR’s modules are a generalization of text-in text-out modules since they can capture operations involving complex data types (like tables) and strings, among others. The data to text transition is also clearly represented through the compositions of our modules, unlike attention maps-based modules whose interpretability has often been debated (Serrano and Smith, 2019).
Data-to-Text Generation. Existing methods for data-to-text generation include (1) supervised methods that finetune seq2seq LMs (Kale and Rastogi, 2020; Chen et al., 2020b; Ribeiro et al., 2021; Ke et al., 2021; Xiang et al., 2022), (2) pipeline modular methods (Reiter and Dale, 1997; Reiter, 2007; Laha et al., 2020; Kasner and Dušek, 2022), and (3) few-shot methods that assume access to a large corpus of unlabeled examples for data augmentation or retrieving similar examples (Puduppully et al., 2019; Zhao et al., 2020; Trisedya et al., 2020; Su et al., 2021). Unlike prior modular methods, MURMUR uses few-shot neural or symbolic modules. Unlike past few-shot methods, MURMUR works well with as few as one demonstration, without requiring access to any unlabeled corpus.

heading: 8 Conclusion
text: We presented MURMUR, a neuro-symbolic modular reasoning approach for data-to-text generation. MURMUR shows the benefits of building interpretable modular text generation systems by breaking a task down into sub-problems and then solving them through separate modules, without requiring module-specific supervision. It utilizes the power of LLMs in solving linguistic sub-tasks through incontext learning while delegating the logical subtasks to symbolic modules. MURMUR generalizes the concept of modules by treating them as functions and defining their behaviors through expected input/output data types and compositions with a grammar (analogous to function compositions).

heading: Limitations
text: MURMUR relies on large language models for fewshot linguistic skills like surface realization and text fusion. It is probable that smaller models do not work as well, in which case one may curate additional training data to train these modules. We
also note that our choice of logical modules is motivated by the characteristics of the task. Hence, it is conceivable that other data-to-text generation tasks might benefit from incorporating additional modules. MURMUR does not make any assumptions about the type or implementation of the modules and it should be straightforward to extend our method to other data-to-text generation tasks.
We limit our experiments to English datasets. We also adopt a simple prompting strategy for converting a reasoning path to a natural language summary by representing the path as a string. This works well in practice and OPT is typically able to resolve the module names and their arguments correctly. However, more future work is needed to understand when this fails so that better prompting methods can be developed. Despite the known limitations of standard automatic metrics like BLEU and METEOR, we use them to compare our method to previous works. While this is not ideal, we have performed comprehensive human evaluation for both tasks to further verify our claims.

heading: Ethics Statement
text: Large Language Models can be prone to generate toxic and unwanted content (Weidinger et al., 2021). Since MURMUR uses focused modules to accomplish specific skills, we believe that this might help limit inadvertent negative impacts. Furthermore, the presence of specific modules should provide users with more trust and control in realworld scenarios, allowing one to verify, debug, and improve the capabilities of these modules.

heading: B Additional Experiments on WebNLG (Cont. from §5)

heading: B.1 Effect of Number of Demonstrations

heading: Effect of number of Demonstrations
text: In Fig. 3, we compare the METEOR scores of DP and MURMUR by varying the number of demonstrations. DP shows improved performance with more demonstrations, while MURMUR’s improvements are marginal. In the process of providing more demonstrative examples, DP implicitly learns the underlying step-wise reasoning process, while such phenomenon is explicitly captured through one demonstration in MURMUR.

heading: B.2 Effect of Variations of Demonstrations
text: In Table 9, we compare the performance of fewshot baselines on the validation set of WebNLG and
analyze the effect of different choices of random demonstrations on in-content learning. Using three different random seeds, we show that all methods are fairly robust to randomness in demonstrations.

heading: C Additional Experiments on LogicNLG (Cont. from §6)

heading: C.1 Effect of Number of Demonstrations

heading: Effect of Number of Demonstrations
text: In Fig. 4, we compare BLEU-3 scores of DP and MURMUR by varying the number of demonstrations from 1 to 3. Unlike WebNLG, we do not observe any noticeable improvements in incontext learning capabilities with more demonstrations, possibly because of the inherent difficulty of generating logical summaries from tables.

heading: C.2 Effect of Variations in Demonstrations
text: In Table 10, we study the effect of randomness in the choice of a single demonstration for LogicNLG. We report mean and variance of BLEU scores for each method with a randomly chosen demonstration from the training examples. Similar to WebNLG, all methods are fairly robust to the choice of demonstrations and exhibit comparable variance in performance.

heading: C.3 Effect of Different Beam Sizes in Best-first Search of MURMUR
text: At each step of the search, MURMUR keeps track of the highest scoring reasoning paths. Table 11 compares the effect of the beam size for our search algorithm on the LogicNLG validation set. Perhaps unsurprisingly, maintaining a bigger beam i.e., conducting a more exhaustive search leads to some
improvements in BLEU scores, however, the gain mostly saturates with beam sizes of around 50-100.
C.4 Further Analysis of Saliency Metric (Cont. from §4.2)
Training Data Construction. In Fig. 5, we show an illustrative example of the training data creation process for our saliency metric. In ‘Incorrect Partial Path-1’, when we perturb the avg module with the sum module, we aim to teach the model that although both are valid reasoning steps, averaging over the column ‘points’ is a more salient and informative reasoning step than summing over the column ‘year’. Similarly, in ‘Incorrect Partial Path2’, when we perturb the input to the module avg by performing average over the column ‘wins’, we want the model to learn the salient columns to reason over for a given module.
Effect of Varying Supervision on Metric Accuracy and Downstream Performance. We conduct an in-depth analysis of the saliency metric used to score the reasoning steps in MURMUR. As shown in Table 12, we vary the amount of supervision for training the saliency metric and study its effect on the validation set accuracy (in identifying whether a partial reasoning path is correct or not) and also on the downstream LogicNLG BLEU scores. Our key takeaway is that a small number of gold reasoning paths (about 200, spanning 100 tables) is enough to train a good saliency metric that not only achieves a high classification accuracy of 76% but also leads to a BLEU-3 score of 10.8 on LogicNLG. Increasing the training data further
to 7k gold paths (equivalently, 42k correct and incorrect partial paths) increases the classification accuracy to 82% but does not impact LogicNLG performance much.

heading: D Prompts (Cont. from §4)
text: WebNLG. Table 13 shows an example of direct prompting (Zhang et al., 2022) for WebNLG. In Table 15 and Table 16, we show the prompts for the surface realization and text fusion modules in MURMUR. Note that the single demonstration for direct prompting is decomposed into individual reasoning steps for the two modules in MURMUR.
LogicNLG. Table 17 shows an example of direct prompting for LogicNLG. Table 18 shows an example prompt for the surface realization module in LogicNLG. We only provide the table topic, table header, and the reasoning path in the prompt. We do not add the table content to the prompt because all the information needed by the model to generate the summary is typically present in the reasoning path. Any other contextual information about the table can also be inferred from the table header and topic. We observe that adding the table
content makes the model more prone to hallucinations because it may not limit its generation to the information provided in the reasoning path alone.

heading: E Examples of MURMUR Summaries
text: In Fig. 6, we show representative examples of summaries generated by Direct Prompting and MURMUR for WebNLG. Fig. 7 shows the step-wise summary generation process of MURMUR for WebNLG. In Fig. 8 and 9, we show representative examples of the reasoning paths and summaries generated for two tables in LogicNLG.
Table Topic: List of Latvian submissions for the academy award for best foreign language film
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 10
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 5, 6
3 B1. Did you cite the creators of artifacts you used? 5, 6
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5, 6
C 3 Did you run computational experiments? 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4, 5, 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5, 6, Appendix B, C
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5, 6
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 5, 6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 5, 6
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, more future work is needed to understand when this fails so that better prompting methods can be developed. Despite the known limitations of standard automatic metrics like BLEU and METEOR, we use them to compare our method to previous works. While this is not ideal, we have performed comprehensive human evaluation for both tasks to further verify our claims