Weakly Supervised Semantic Parsing with Execution-based Spurious Program Filtering
Authors: Kang-il Lee
Segwang Kim
Kyomin Jung
Venue: EMNLP 2023 Main (2023)
Keywords: weakly supervised semantic parsing
spurious programs

Abstract: The problem of spurious programs is a longstanding challenge when training a semantic parser from weak supervision. To eliminate such programs that have wrong semantics but correct denotation, existing methods focus on exploiting similarities between examples based on domain-specific knowledge. In this paper, we propose a domain-agnostic filtering mechanism based on program execution results. Specifically, for each program obtained through the search process, we first construct a representation that captures the program's semantics as execution results under various inputs. Then, we run a majority vote on these representations to identify and filter out programs with significantly different semantics from the other programs. In particular, our method is orthogonal to the program search process so that it can easily augment any of the existing weakly supervised semantic parsing frameworks. Empirical evaluations on the Natural Language Visual Reasoning and WikiTableQuestions demonstrate that applying our method to the existing semantic parsers induces significantly improved performances.

Weakly Supervised Semantic Parsing with Execution-based Spurious Program Filtering
Kang-il Lee
Segwang Kim
Kyomin Jung!.3t
'Dept. of ECE, Scoul National University
Samsung Electronics Mobile eXperience 9IPAI, Scoul National University {4bkang, ksk5693, jjung)esnu ac.
Abstract
tion result of the program) rather than the program itself.
The problem spurious progrums is long- standing challenge when training semantic parser from weak supervision. To eliminate such programs that have wrong semantics but correct denotation, existing methods focus on exploiting similarities between examples based domain-specifie knowledge. In this pa- we propose a domain-agnoste filtering mechanism based on program execution results. Specifically, for each program obtained through the search process, we first construct a repre- sentation that captures the program's 's seman- tics as execution results under various inputs. Then, we run a majority vote on these represen- tations identify and filter out programs with significantly different semantics from the other programs. In particular, our method is orthog- onal to the program search process that it can easily augment any of the existing weakly supervised semantic parsing frameworks. Em- pirical evaluations the Natural Language Vi- sual Reasoning and WIKITABLEQUESTOON demonstrate that applying our method the existing semantic parsers induces significantly improved performances. Code is available at https:/ggghubb.om//eeee7/exeee
There blue square
w: [I [solor: blue, shape: square), [color; black, shape: circle) l..].
=: ob//:xxstsssquuaaacae (squaeebllullll _bbj_cts)))
2 (bjEEiist((lacck(irileellllll objects)))
y: True
Rank
Nation
Gold
Silver
Bronze
Total
Soviet Union
50
27
States
East Germany (GDR)
20
23
23
66
Germany
Japan
13
Australia
Hungary
13
16
86
Bulgaria
10
Italy
How many nations than ten silver medals?
w: [Rank: 11, (Nation: Soviet Union), (Gold: s0)...I.... ...]
8 wow.tttilleruummrrrretttiiilllli
colurnn-Siiver, 10))
#t select(fiterlnnallRoww, column:Nation, Japan), column:Rank)
Figure Overview task setup on Natural Language Visual Reasoning (top) and WIKITABLEQUESTTONS (bottom) dataset. The datasets include only utterance ar, world and denotation y (ground truth program is not given). Spurious programs like z', whose meaning is wrong but execution result correct, are major chal- lenges of the task.
1 Introduction
Semantic parsing is the task of mapping natural language utterances into machine-executable mean- ing representations, often referred as programs. Most deep learning-based based semantic parsing stud- ies take the supervised learning approach requir- ing utterance-progee program paired dataset. However, annotating such pairs demands expensive expert annotations. Instead, weakly-supervised semantic parsing, i.e. learning from denotation, has drawn much attention (Clarke et al., 2010; Liang al., 2011; Berant et al., 2013). this setup, a semantic parser trained with cheaper denotation (execu-
Without supervision for correct programs, train- ing weakly supervised semantic parser typically entails a program search process. In this process, given the natural language utterance, a search al- gorithm such as beam search generates pool of likely programs. Among these programs, there may be some programs that have incorrect seman- tics but derive the correct denotation by chance, z' in Figure 1. These programs are called spuri- ous programs and introduce undesirable noise on the training signal. Hence, filtering these spurious programs is great interest in weakly supervised
Work done while at Scoul National University. Corresponding author.

---

semantic parsing (Pasupat and Liang, 2016; Gold- man et al., 2018).
tations indicating the semantic of z; in each world. Our goal is to train model such that when given 2i as input, produces a program 2j which re- turns (each member of) yi when executed on (each member of) Wi--
Prior works on weakly supervised semantic pars ing attack the spuriousness problem by introduc- ing some domain-speciio knowledge, such as ab- stracted utterance- prograa pairs (Goldman et al., 2018) or utterance groups (Gupta et al., 2021). Un- like these works, we propose a domain-agnostic filtering mechanism based on majority vote over program execution results to alleviate the spurious- ness issue. Our intuition is that programs whose execution results largely deviate from those of other programs in the pool are likely to be spurious, thus filtering them out would improve the training of weakly supervised semantic parsers, To effectively quantify the degree of deviation, novel representation scheme of programs based on the execution results. Here, the entries of a representation vector are the program's 's execution results on worlds retrieved from other training ex- amples. To exclude spurious programs, we run a majority vote on these representations to construct "centroid representation" and filter out the pro- grams whose representation is dissimilar to it. Our method can be applied to any weakly supervised semantic parser with minimal modification, long as involves a program search step in the training process.
2.2 Datasets
NLVR Natural Language Visual Reasoning (Suhr et al., 2017) is dataset of blocks world do- main that requires complex reasoning abilities. The world is given structured representations of various objects and the utterance is a statement about the properties or relations of the objects in the world, as shown in Figure I (Here, we graphically display the world to help the reader understand). The de- notations Boolean values representing whether the given utterance is true false in the world. There are 3,163 unique training examples consist- ing of one utterance and four world-denotation pairs. Also, there are development, public test, and hidden test sets with 267, 266, and 266 examples each.' !
propose
WTQ WIRITABLEQUESTIONSS (Pasupat and Liang, 2015) a table semantic parsing dataset with complex queries and large natural language variation. Worlds are structured representations of Wikipedia tables and utterances are questions about the tables. Unlike NLVR, denotations can have val- ues in the table cells values obtained by applying some elementary functions to the cell values. The WTQ training set consists of 11,321 training exam- ples and 2,831 development examples. It provides 4,344 test examples with unseen tables to measure the model's 's generalization performance.
We evaluate our filtering mechanism on two chal- lenging datasets with distinct characteristies: Nat- ural Language Visual Reasoning (NLVR) (Suhr al., 2017) and WIK/TABLEEQESTIONS (WTQ) (Pasupat and Liang, 2015). When added on the base models (Gupta et al., 2021; Wang et al., 2019), our filtering mechanism shows significant improve- ment over the baselines without using additional domain-specific knowledge. Finally, we quantita- tively analyze the effectiveness of our approach in detecting spurious programs and conduct error analysis failure case.
3 Execution-based Filtering
To climinate spurious programs, we devise a novel execution-based filtering mechanism. Intuitively, among consistent programs, the spurious ones are likely to semantically deviate. However, measur- ing defining semantic distance is challenging. Thus, we instead loosely capture the semantics of programs by executing them against reasonably se- lected worlds. Then, we filter out the programs whose execution results deviate most from others by performing a majority vote.
2 Background
In this section, we formalize weakly supervised semantic parsing problems and introduce two datasets: NLVR and WTQ.
2.1 Problem Definition
The dataset for weakly supervised semantic parsing consists of N examples (a,; wi; vr) N where x; is a natural language utterance, wy is set of worlds that x; can be evaluated on, and yi is a set of deno-
Formal Setups Consider an example with utter- ance x, world w. and denotation y.² The programs
'The hidden test set now made public by dataset creators. WW omit the data index for brevity.

---

Hard Vote In order get centroid program rep- resentation rs with hard vote, we run majority vote along each entry of (ri)
gold programs
spurtoss programe
r. argmaa EER
Σ - =
(1)
"S
where r is the j-th entry of vector and is an indicator variable. E denotes the set of denota- tions obtained by executing programs the pool. Note that Ys not necessarily the same as the rep- resentation of the real gold program. Given the representation Ti, we use to define the score as follows:
Figure Illustration of our program representation scheme and filtering based majority vote. Retrieved worlds (ve '9) partition the programs into several groups by their execution results and are represented lines in the figure.' By running majority vote based on the execution results, programs the gray regions may be filtered.
R
81 Σ 1(7- -1)
(2)
found in the search step are executed against w, and only those with correct denotations remain in the program pool / Still, many of these k programs may be spurious; they do not reflect the meaning of given utterance but coincidentally derive correct denotation y.
Finally, we filter out the programs with a score lower than the heuristically chosen threshold r. We can improve this mechanism by weighting representations when performing the majority vote, proportional to some scalar metric of the good- ness of the program. This metric can be either defined manually with domain-specific knowledge or learned from data. We add this weight term (()) to equation as follows:
3.1 Program Representation
In order to capture the semantics of programs, we devise a representation scheme based on their exe- cution results against set of worlds. Mathemati- cally speaking, represent semantic of program 2 as n-dimensional sparse vector r; whose j-th entry is the exccution result of 2 against world wj. Regarding the worlds [wy] we collect them from the training set using two different selection strategies for NLVR and WTQ, on which we elab- orate in section 4 and appendix B.
r arggmax Σ W()) =c)
(3)
eCE
As metric W-),, one can use a program's model likelihood or hand-crafted scores such as lexicon coverage used by Dasigi et al. (2019) and Gupta et al. (2021). Note that this weighting technique is different from soft vote, which we describe next.
Soft Vote One drawback hard vote is that it cannot take into account the proportion of deno- tations because only the eventual winner is repre- sented as centroid representation. Instead, we can incorporate the denotation proportion into the score 8; by counting the number of programs with the same execution result as 24.
3.2 Filtering Programs with Majority Vote
The program representations {ri) be under- stood as points on space and are partitioned by {wr}"=-- As shown Figure 2, r, can be classi- fied based on which region it resides. We hypoth- esize that the programs far from the "centroid" of {ribke are more likely to be spurious. Here, we employ two vote techniques: (1) a "hard" vote that constructs explicit centroid representation utilizing only the winning denotations and (2) "soft" vote that considers the proportion of each denotation.
(4)
As in hard vote, each program's contribution can be weighted by W(-):
*In NLVR. one utterance typically four worlds denotations. Therefore, program remains the pool when it correctly produces all four denotations.
n
"In this illustration, we assume binary denotations so that a world partitions programs into two groups.
$;
((z)1(/--/))
(5)

---

We normalize the scores {s such that the high- est score becomes 1 and filter out the programs whose normalized score lower than some thresh- old r.
Source table
Target
Teem
Wine Locses
Pent
Nation
Gold Saver
Dionon
Total
Sutrel tigare
lanord Blu Jaya
Engune
Boeton
Mivouuice Browers
4 Collecting Execution Results
Spor
Bootard
10 Commosawea
Farly
0
In the previous section, the program representa- tion is defined by the program's 's execution results on multiple worlds. Now we describe methods to select the worlds and obtain programs' exccution results on those worlds. We discuss two separate approaches for NLVR and WTQ, considering their vastly different world configurations.
12
Denmask
0
0
2) selecctarggmaxxllleeosss column:Winss, column:Team Team)
25 Dountf/leeNuubberrrrrteaaalalli column:Wins, 100))
seleci(argaxxxallFoww. column:Silver), column:Nation)
Sountiileeuumbberrrrtttrrallllll column:Silver, 2))
Figure 3: Illustration of column and entity replacement.
Here, žι and 22 are programs conditioned on the source table, and zi and 2/2 are their counterparts modified to be executed on the target table. Within the programs, column and entity names of the same type are displayed in the same color.
4.1 Collecting Execution Results for NLVR
NLVR, a program representation r; consists of Boolean execution results of 2; on various worlds {w))}=ı- In order to effectively identify outliers among the programs, the representations {ri) should provide helpful information when perform- ing the majority vote. For example, a world wi that returns 'True' for all 2;'s 's is useless because we would not get any information to identify outliers from the j-th entry. That is, worlds used for build- ing representations should be able to classify 2; ss according to their execution results. Our hypothesis is that the worlds whose corresponding utterance is similar to z would be more informative than others because these worlds are more likely to be relevant to the meaning of x. To this end, we collect worlds from the training set by retrieving top-n (n = 80 in NLVR) worlds {wo } based on the BLEU score between x and each wj's corresponding utterances.
single name in the target table. Specifically, we first find all the column and entity names in Z and t and identify their types, e.g., string, number, etc. Then, for each name N in Z, randomly sample name N' with the same type in w and replace all the occurrences of N with N'. We illustrate the column and entity replacement in Figure 3, where programs 21 and 2/2 are modified to = and 2/2- Note that all the occurrences of column:Wins are replaced with column:Silver consistently.
During the execution of modified programs on target table, some return execution errors for vari- ous reasons. For example, the function argmax in the domain language used by Wang et al. (2019) re- turns an error when the input is a list of length onc. When there are too many errors (when more than 10% of programs return eтror in our experiments) for particular table, we perform the sampling and replacement process again. The resampling proce- dure may be repeated up to 10 times per table, and we discard the table if no satisfactory replacement found after 10 iterations. The program represen- tation is constructed by repeating this process until the number of worlds used hits n (n = 40 in WTQ). More details on the table collection is described in appendix B.
4.2 Collecting Execution Results for WTQ
As each program in WTQ is conditioned on a spe- cific table (denoted as source table henceforth) and therefore cannot be used on others, propose method for modifying programs that they can be executed on target lable we want. In partic- ular, we replace entity and column names in the programs with those in the target table to make the programs executable while maintaining the seman- tic relationship between the programs.
Experiments
Consider an example with a program pool Z { and target table w. Our goal is to replace all the column and entity names in Z with those in w while maintaining the semantic relationships between the programs in Z. Thus, we replace the names consistently, i.c., the constants of identical name in multiple programs should be replaced with
To validate the efficacy of our filtering method, we conduct experiments on NLVR and WTQ datasets. First, for each dataset, we characterize existing parsers we use and explain where our method is applied. After explaining our filtering implemen- tation on each of these datasets, we present our

---

experimental results and ablation studies.
2. We defined lexicon recall score R(A(x), 2) which is the number of tokens included in both A(x) and 2, divided by |A(z)]. When perform- ing a beam search, we doubled the beam size and performed the beam search again until the pro- gram pool { had at least one program with R(A(z), zi) = Doubling the beam size was stopped if program with a score of I was not found after 5 iterations. We used lexicon recall score R(A(z), 2) as a proxy to the example diffi- culty (and the number of consistent programs in the beam) because when the utterance is complex, the model usually produces a very small number of consistent programs with low lexicon recall scores, We empirically observed that there is no computa- tional issue with such a large beam size because most of the nodes in the beam reach dead ends due to the grammar constraint.
5.1 NLVR Implementation Details
NLVR Base Parser We use the Iterative Search (Dasigi et al., 2019) and Consisteney-bast bassed based Parser (Gupta al., 2021) construct a base parser. In this section, we provide background on these two methods.
Iterative Search (Dasigi et al., 2019) uses grammar-constraing RNN encoder-decoder model of Krishnamurthy et al. (2017). Its training under- goes alternating two steps: Maximum Marginal Likelihood (MML) and Minimum Bayes Risk (MBR). MML, when given the utterance z; and denotation y/, the parser is trained to maximize the likelihood of marginalization of programs 2 ss that are consistent with yi- In MBR, the model is initialized with the previous MML checkpoint and trained to minimize cost functions for denotation accuracy and lexicon coverage.
For the filtering mechanism, we used this lexicon recall score R(A(z), 2) as the weight term W(.) described in section 3.2. We used hard vote on NLVR and chose the filtering threshold 0.8.
Consistency-based Parser (Gupta et al., 2021) improves the consistency of Iterative Search by introducing Logical Language Design (LLD) and Consistency Reward (CR). In LLD, they replace some macro functions with generic functions that are more reusable across different contexts. Also, they introduce the Consistency Reward (CR), which encourages consistency between programs for related utterances.
5.2 WTQ Implementation Details
WTQ Base Parser As a base parser for WTQ. we use the Structured Attention (Wang et al., 2019), which is strong and liigt-weighted system suit- able for testing our filtering mechanism. Among the systems that do not utilize pre-training ternal datasets, is state-of-the-art on WTQ.
In both approaches, there program search step between every MBR and MML to construct a dataset to train the model in the subsequent MMI.. The model performs a beam search in this search step to produce pool of likely programs. Our filtering mechanism is attached here to minimize undesirable noise from spurious programs being included in the next step's MML dataset.
Unlike Iterative Search and Consistency-basee Parser, they exhaustively search the programs con- sistent with the given denotation prior to the train- ing. The exhaustive search is manageable because the space of programs significantly reduced by introducing abstract programs and restricting the number of production rules. In this case, we take a fine-tuning approach: the model is first trained as proposed by Wang et al. (2019), and then fine- tuned with programs filtered with our method. The model was trained using the official code released by the authors, achieving slightly worse results (dev accuracy of 43.2 and test accuracy of 44.4) than those reported in the paper. Then, we fine-tuned this reproduced model for 5 epochs with programs filtered with our method.
Lexicon-based Program Search Our initial im- plementation without modifying the base model exhibited decrease in performance. presumably due to the beam size not being suitable for our method. As our filtering mechanism relies on the majority vote, it would be advantagcous if a large and informative candidate program pool is obtained during the search step. To this end, instead of sim- ply enlarging the beam size, we apply the more efficient beam search augmented by the lexicon, which is utilized by Dasigi et al. (2019) and Gupta al. (2021).
We used the instantiation model likelihood of the base parser (Wang et al., 2019) as weight terms W(.) representing how well the program aligns with the given utterance. Besides, we employed soft vote with filtering threshold T = 0.2. More
With the utterance x, its lexicon was given a set of program tokens A(x) that should be in program

---

Dev.
Test-P
Test-H
Test
Approach
Ace.
Con.
Acc.
Con.
Acc.
Con.
Con.
Abs. Sup. ReRank (Goldman et al., 2018)
85.7
67.4
84.0
65.0
82.5
63.9
64.5
Iterative Search (Dasigi et al., 2019)
85.4
64.8
82.4
61.3
82.9
64.3
62.8
LLD (Gupta et al., 2021)
88.2
73.6
86.0
69.6
87.2
70.1
69.9
LLD CR (Gupta et al., 2021)
89.6
75.9
86.3
71.0
89.5
74.0
72.5
LLD (w/ modified beam search)
90.8
77.8
88.3
73.4
89.0
74.6
74.0
+ Execution-based Filtering
90.5
78.8
89.4
74.2
89.4
76.3
75.2
LLD CR (w/ modified beam search)
90.3
77.5
87.8
72.8
87.8
72.2
72.5
Execution-based Filtering
90.9
78.7
88.7
74.9
88.8
72.5
73.7
Table I: Accuracy and consistency of our approach and prior works on NLVR development, test-public (Test-P), and test-hidden (Test-H) sets. The rightmost column (Test) shows the average consistency Test-P and Test-H. LLD: Logical Language Design. CR: Consistency Reward.
implementation details for NLVR and WTQ are described in appendix A.
Approach
Dev.
Test
Zhang et al. (2017)
40.4
43.7
5.3 Main Results
Liang et (2018)
42.3
43.1
Dasigi et al. (2019)
42.1
43,9
In both NLVR and WTQ, we evaluate the models with accuracy, which considers the correctness of the execution result for only one world-denotation pair. For NLVR, we also use consistency, which counts a program as consistent if it is correct in all four world-dennttaton pairs. We report the average value of 4 runs with different random seeds.
Agarwal ct al. (2019)
43.2
44.1
Wang et (2019)
43.7
44.5
+ Execution- Filtering
43.2
44.8
Table 2: Accuracy of our approach and previous works on WIKITABLEEUUESTION development and test sets.
NLVR Our modification on beam search de- scribed in section 1.1 yields quite different results depending on the setting; there is a significant im- provement when the LLD is used alone, but rather small change in LLD CR setup as shown in the Table 1. Interestingly, the consistency reward is helpful in our modified versions. When our filtering mechanism applied to these modified base models, improves the test accuracy and con- sistency in both of base models as shown Table I. When we add our filtering mechanism on our mod- ified version of LLD, the average test consistency improves by 1.2%. Adding our mechanism on the modified LLD CR also shows improvement of 1.2% in test consistency. We follow Gupta et al. (2021) and perform a statistical test on the signifi- cance of the improvements with Deep Dominance (Dror et al., 2019). All improvements mentioned turn out to be statistically significant (p < 0.05), implying that our approach effectively filters out spurious programs.
NLVR
WTQ
Vote Type
Dev.
Test
Dev.
Test
Hard Vote
78.8
75.2
42.8
44.3
Soft Vote
77.0
73.4
43.2
44.8
Table Consistency (NLVR) and accuracy (WTQ)
with hard vote and soft vote.
in the test set, which consists of the tables unseen during the training. This result demonstrates that our method successfully filters out spurious pro- grams and thus the parser learns more generaliz- able regularities in the mapping of natural language the program. Our approach achieves the highest reported test accuracy among semantic parsers that do not make use any external data.
5,4 Impact of Vote Types
As mentioned in section 5.1 and 5.2, we use dif- ferent vote types for NLVR and WTQ. Empirical result in table shows a distinct superiority of the vote methods in each domain. In NLVR, hard vote
WTQ As shown in table 2, our filtering mecha- nism improves the base parser (Wang et al., 2019)

---

Weight Type
Dev.
Test
Precision
Recall
Fl-score
Lexicon Recall Score
78.8
75.2
0.8
99.5
40.0
49.5
Model Likelihood
77.0
72.5
0.9
99.6
57.8
66.3
w/o Weight Term
78.8
74.1
1.0
99.4
82.0
85.7
Table 4: Consistency on NLVR with different weight types.
Table 5: Spurious program detection performance on 30 NLVR training examples at last search step, with various threshold All the values calculated indi- vidually for each train example and averaged afterward.
outperforms soft vote, but the opposite is true in WTQ. This tendency can be explained by the dis- tinct characteristics of two domains. NLVR has Boolean denotation, thus the winning denotation always gets more than 50% of the vote and can rep- resent the programs' semantics. However, in WTQ, the denotations can have various values, making the winner much less representative of the entire program pool compared that in NLVR. Soft vote can alleviate this issue by considering the denota- tions of programs in the pool.
Spurious Program Detection In table 5, we port the precision, recall, and Fl-score spurious program detection for various thresholds т. High precision values show that the majority of semanti- cally correct programs have scores very close to 1.0. This suggests that the centroid representation ob- tained through majority vote closely approximates the true gold program representation.
We also discovered that the optimal threshold for detecting spurious programs does not align with the optimal threshold for NLVR task performance. In our main experiment, the best NLVR test consis- tency is attained when using a threshold value of = 0.8. However, it appears that this threshold is somewhat generous, as 60% of spurious programs remain in the program pool. When we raised the threshold to 1.0, which is the optimal value for detecting spurious programs, we noticed decrease in NLVR test consistency. One potential reason for this phenomenon is that high threshold produces too many false positives in the early stages of train- ing, which hampers the search space exploration throughout the training process. To address this trade-off, one possible direction for future work is to utilize adaptive threshold for each step of the search process.
5.5 Impact of Weight Term (-))
The choice of weight term (..) is another impor- tant factor in our filtering mechanism. In this sec- tion, we analyze the effectiveness of three different types of vote weighting in NLVR: (1) lexicon recall score R(A(z), 2). (2) model likelihood p(=|z), and (3) vote without weight term (which corresponds to the equation I).
As shown in table 4, the use of lexicon recall score exhibits the most improvement over the base parser, indicating the effectiveness of weighting votes with a metric of alignment between the utter- ance and program. Vote without any weight term shows weaker performance improvement. Interest- ingly, weighting with model likelihood deteriorates the performance, presumably due to positive feed- back of up-weigtting spurious programs through the training process.
Correlation Statistics To further analyze our method depth, we report some correlation statis- tics between the programs' score and their spuri- ousness. First, the Pearson correlation coefficient between the spuriousness label (1 if spurious, 0 if not spurious) and score described in equation 2, is 0.358. Also, the ROC-AUC score stands 0.738 when using the score classify whether the program is spurious or not. Finally, the mean and standard deviation of scores 8; for correct programs are 0.997 and 0.029 respectively, while for spurious programs, they are 0.899 and 0.155.
6 Analysis
Setup In this section, we quantitatively analyze the effectiveness of our approach in filtering out spurious programs. To assess its effectiveness in distinguishing between spurious and correct pro- grams, we randomly select 30 examples from the NLVR training and manually label the pro- grams obtained through the last program search step. Next, we evaluate the performance of spuri- ous program detection, by classifying all programs with scores lower than the threshold r as spurious.
where 8q is program

---

(Successful case) Sentence: There is at least black item closely touching the bottom of box.
Score
Program
((* (* (object_ greater greater_eeuuas 1) black) touch touch__ootton bottom) all objects)
((* (* object exists black) touch bottom) all objects)
0.85
touch_bottom) ((* (* (* (object_cnnie count count_greater greater_equals 1) black) touch bottom) bottom) all all__ojeects
0.58
((* (* (object count greater equals 2) black) touch, bottom) all objects)
0.50
(hox_ moo__ gergatte_equuee
(box_ filter all boxes (* (* (object_ colustt__ootnn__ee cowuttggreaterrrrtteeeeeeee greater_equak 1) black) touch_ )oth_o)ttom))
(Failure case) Sentenee: There are 2 black blocks
Score
Program
1.0
((* (ohject_ count count_ggeaterrrrrt_teeeeeoo preater_equalk black) all oojectt))
1.0
(object_exiss) sejject_exists (object_in box all_ boxee))
0.63
(hox mnox_count___ couut_uuuuuuuueeeeeeeee equull 2 (box_ filter all all_bxxes object_exisse _eiistsssssssseeeeeeessttetsseee black)))
0.62
((* (object_conee certl__oot____eeeeeeee _quulleeulllss 2) black) (object _n__ox all boxeb)))))
0.62
((* (object count, equals 2) black) all objects)
Table 6: Successful (top) and failure (bottom) case of our filtering mechanism. Boldfaced programs are semantically correct programs and the others are spurious programs.
Successful and Failure Cases Table 6 shows both successful and failure cases in the last search step on NLVR train set. In the successful case, all programs with score lower than 0.8 are filtered out and turn to be all spurious. However, failure occurred when the majority vote selected false program between two programs with seman- tically similar functions (object cobjjct_couun___oe _eunll and abject_counte www.oooonnnnteeeegggeeooooo _equalse_eeuals..
table semantic parsing.
Unlike these approaches, our method can be ap- plied to existing semantic parsers with minimal domain-specific engineering and a relatively small amount of data.
7.2 Identifying Programs with Execution Results
Recently, there has been a growing interest in using the execution result of program to guide the train- ing and inference of deep learning model. Odena and Sutton (2020) introduce notion of property signature, which represents hypothetical program specified by given input-output pairs. The main difference between property signature and our rep- resentation scheme when representing program is that the former uses a set simpler programs, and the latter uses a set related worlds from other examples.
7 Related Work
7.1
Weakly Supervised Semantic Parsing
Recent research on semantic parsing has focused on weakly-supervixee semantic parsing, learning from denotations, whose goal is to learn semantic parser without manual program annotation (Clarke et al., 2010; Liang et al., 2011; Berant et al., 2013). an effort to reduce the program search space and minimize the noise from spurious programs, previ- ous works exploit domain-specific knowledge such as utterance groups (Gupla et al., 2021) and ab- stract programs (Goldman et al., 2018; Wang et al.. 2019). Other studies focus enforeing alignments between relevant utterances and program parts in lexicon level (Dasigi al., 2019) or phrase level (Wang et al., 2019).
In natural language to code translation, there are attempts leverage execution results cluster syntactically different but semantically identical programs and submit the program in the largest cluster for evaluation (Li et al., 2022; Shi et al., 2022). These methods are somewhat similar to soft vote without weight term in our approuch, while their goal is to pick the best program at the infer- ence stage rather than filtering spurious programs.
More recently, pre-trained language models have demonstrated outstanding performance on seman- tic parsing, especially in the table semantic parsing domain, thanks to the large corpus of tables and surrounding natural language utterances (Yin et al., 2020; Yu al., 2021). These models are trained with tasks devised for natural language and table understanding with the enormous amount of tables, which may not be available when we construct a semantic parser in more scarce domains other than
In recent times, the use self-consistent Chain- ohouhtt (CoT) prompting (Wang et al., 2023) has greatly enhanced the reasoning capabilities of large language models. This self-consistency is achieved by decoding multiple reasoning paths and selecting the with the highest vote score.
Program execution results can be utilized for var- ious purposes. Zhong et al. (2020) propose test

---

suite accuracy based on the programs' execution results on tables which are constructed to be likely to distinguish false programs from the gold pro- gram. Pasupat and Liang (2016) construct a set of "fictitious worlds" such that the denotations of those worlds are most effective in filtering out spurious programs when annotated by humans. Here, the effectiveness of the world is approximated using the execution results of the programs.
K. Jung is with Automation and Systems Research Institute (ASRI), Scoul National University.
References
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to general- ize from sparse and underspecified rewards. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma- chine Learning Research, pages 130-140. PMLR.
8 Conclusion
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
We proposed a domain-agnosiin approach to filter out spurious programs in weakly supervised seman- tic parsing based on execution results and majority vote. Our assumption was spurious programs are outliers in terms of meaning; thus, we introduced a representation scheme that captures the seman- ties of programs. Based on these representations, we ran the majority vote to identify and exclude spurious programs from the pool. Our approach showed significant improvements over base mod- els on NLVR and WTQ test set performance, also reporting new state-of-the-art on NLVR with less domain-specifit knowledge than the previous best model.
James Clarke, Dan Goldwasser, Ming- Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In Proceedings of the Four- teenth Conference on Computational Natural Lan- guage Learning, pages 18-27, 27, Uppsala, Sweden. As- sociation for Computational Linguistics.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 2669-2680, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Limitations
One limitation of our approach is that it does not help when the program pool is too small. Also, the weight term (-)) played a significant role in achieving tfate-of-the-ar performance. The appli- cability of our method on domains without any metric of alignment somewhat questionable, al- though the use of such metrics (e.g. lexicon cov- erage) is quite widespread in weakly supervised semantic parsing fields and the construction of the metric has to be done only once for a particular domain. Improving its robustness is one possible future work direction.
Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance how properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2773-2785, Florence, Italy, Associa- tion for Computational Linguistics.
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018 Weakly su- pervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 1809-1819, Melbourne, Aus- tralia. Association for Computational Linguistics.
Acknowledgements
This work was supported by National Research Foundation of Korea (NRF) grant funded by the Ko- rea government (No. B02IRIA2C20082 stitute of Information & communications Technol- ogy Plamning& & (Valuation(IITP) grant funded by the Korea (overnment(MMSIT) [No. 2022-0-0018 84, Development and Study of AI Technologies to In- expensively Conform to Evolving Policy on Ethics & N0.2021---01343, Artificial Intelligence Gradu- ate School Program (Seoul National University)).
Nitish Gupta, Sameer Singh, and Matt Gardner. 2021. Enforcing consistency in weakly supervised semantic parsing. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 1tth International Joint Conference Natu- ral Language Processing ( (Volume Short Papers). pages 168-174, Online. Association for Computa- tional Linguisties.
and In-
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gard- ner. 2017. Neural semantic parsing with type con- straints for semi-strrctured tables. In Proceedings of

---

the 2017 Conference on Empirical Methods in Natu- val Language Processing, pages 1516-1526, Copen- hagen, Denmark, Association for Computational Lin- guistics.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with la- tent structured aligmments and abstract programs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCCNPP) pages 3774- 3785, Hong Kong, China. Association for Computa- tional Linguisties.
Yujia Li. David Choi, Junyoung Chung, Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d'Atuuee Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbi, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-leve code generation with alpha- code.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Fed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Chen Liang Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and se- mantic parsing. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, On- line, Association for Computational Linguistics.
Percy Liang, Michael Jordan, and Dan Klein. 2011. Learning dependency-baid -basdd compositional seman- tics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 590-599, Port- land, Oregon, USA. Association for Computational Linguistics.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang. Dragomir Radev, richard socher, and Caiming Xiong, 2021. Gra(pp]a: Grammar-augmented pre-training for table semantic parsing. In International Conference on Learning Representations.
Yuchen Zhang, Panupong Pasupat, and Percy Liang. 2017. Macro grammars and holistic triggering for ef- ficient semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1214-1223 1223, Copenhagen. Denmark. Association for Computational Linguis- tics.
Augustus Odena and Charles Sutton. 2020. Learning to represent programs with property signatures. In In- termational Conference on Learning Representations.
Panupong Pasupat and Percy Liang. 2015, Composi- tional semantic parsing semi-structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 1470- 1480, Beijing, China. Association for Computational Linguistics.
Ruiqi Zhong. Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396-411, Online. Association for Computa- tional Linguistics.
Panupong Pasupat and Percy Liang, 2016. Inferring log- ical forms from denotations. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 23-32, Berlin, Germany. Association for Computa- tional Linguistics.
A Experimental Details
All the experiments were performed using one Nvidia RTX2080ti GPU.
NLVR
We used the same model architecture and hyperparameters as Gupta et al. (2021), except for the beam search described in section 5. Regard- ing the number of worlds n, we retrieved top-20 utterances, so n. = 80 in most cases. We reported the accuracy and consistency on the hidden test set of LLD in Table 1 based on our own experiment because Gupta et al. (2021) do not provide these values. We calculated BLEU score in section with nitk sentence bleu function. A full training process took 20 to 30 hours depending on the settings and random seeds.
Freda Shi. Daniel Fried, Marjan Ghazvininejad. Luke Zettlemoyer, and Sida I. Wang. 2022. Natural lan- guage to code translation with execution. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3533- 3546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alane Subr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reason- ing. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 217-223, Vancouver, Canada, Association for Computational Linguistics.

---

WTQ Before we fine-tuned the WTQ base parser (Wang al., 2019) with our filtering mechanism, was trained for 15 epochs without any modification. The best model with the highest dev accuracy of 43.2 achieved a test accuracy of 44.4. We fine- tuned this model with 4 different random seeds and reported the average in table 2. Note that the official dev/test accuracy reported in the paper is 43.7/44.5. The program filtering process took 6 to hours and was only done once.
high frequency. The program representation is con- structed by sequentially exccuting programs on these ranked and filtered tables until the number of worlds used hits n (n = 40).
Table Ranking and Filtering for WTQ
Here, suggest way rank and filter the ta- bles since some tables have fewer names than the program pool Z, thus less capable or infeasible to support the column and entity replacement process described in section 4.2.
To enhance the feasibility of replacement, we rank the tables according to their likelihood of sup- porting the replacement process. First, we sort the tables by score Stable [S /T//| where S and I are the multiset of the source table's column types and the target table's column types, respec- tively. Intuitively, the Stable assigns high score the target table has more columns than the source table for each column type, prioritizing big tables with various column types that are more likely to facilitate the replacement.
Furthermore, we exclude non-qualifying tables that have strictly fewer column types than the pro- gram pool Z. Given table w, we first find all oc- currences of column and entity names in Z and w. Then we construct two dictionaries for type count- ing: Cz and Cu., whose keys are the column/entity types, e.g., string, number, etc, and values are the number of names of that type in Z and w, respec- tively. Finally, training set tables and exclude all the tables with fewer names than Z in any of the types. For ample, if Cz = 3 number: 2), table with C =[string:4, number:2) 23 remains but Cu =(string:2, number :4 is excluded. This filtering process ensures that all the names Z can have a unique name of the same type in the target table.
compare the Cz and Cw'ss 's of
Additionally, do not utilize the tables with blank cells as target tables because the programs exccuted on such tables tend to return errors in
'Here, the multiset, or bag, is generalized notion of the set which allows multiple elements with the same value. For example, the source table has 2 columns with type str string, S would have two string elements.


Reviews:
[{"review_id": "LgsSIMwPSF", "reviewer": "Reviewer_E67k", "summary": "", "questions": "1. How are r_i vectors encoded for different result types such as strings, numbers, and boolean?\n\n2. It is not clear how entity replacement works (described in section 4.2)? \n\n3. At t=0.8, 60% of the spurious programs remain in the program pool -> why at t=1.0, do we see too many false positives? This part could be made clearer.\n\n4. Is the code for this work going to be made public?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for providing valuable feedback. Below, we address some of the concerns raised in your comments.\n\nRegarding the Results in Table 5:\n\nWe acknowledge that the result presented in Table 5 might appear to be a concern. It is important to note that the result in Table 5 represents the outcome of the final search step within our iterative search process. During this last step, the programs within the beam are mostly accurate. This high accuracy leads to a high optimal threshold for identifying spurious programs (tau=1.0). However, it's worth considering that earlier in the iterative search process, the parser's accuracy is not as accurate as in the last step. This circumstance favors a lower threshold, which in turn facilitates exploration.\n\nNotations and Presentations:\n\nThank you for pointing out the potential clutter within our notations. We will reflect them if the paper is accepted.\n\nAddressing Questions:\n\n1. The entries of r_i vectors of different types are stored as they are. This is posible because in the majority vote, we only compare (same/not same) the values in the same position within the r_i vectors. \n2. To help understanding the entity replacement process, Figure 3 is offered. This illustration demonstrates how we substitute entities occurring multiple times within the program pool (e.g., \"column:Wins\") with a consistent entity (e.g., \"column:Silver\"). The new entity is chosen randomly from the set of target table entities.\n3. In Table 5, the precision reaches 99.4% with a threshold of tau=1.0. This translates to low rate of false positives (0.6%).\n4. If the paper is accepted, we will make the code publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper clearly demonstrates the usefulness of their approach for the NLVR and WTQ datasets.\n\n2. The idea to use a voting mechanism based on the execution of programs on various worlds is quite useful. ", "reasons_to_reject": "1. Table 5, is an issue for me. At t=1.0, the f1-score is highest in eliminating the spurious programs, yet it doesn't help in increasing the overall accuracy. at t=0.8, with spurious programs, the gains are higher. Is there an analysis of the kinds of spurious programs that are worse than others? Are there other confounding factors that are overlooked causing this phenomenon? A detailed explanation on this is warranted.\n\n2. I feel that the notations/presentations could be made clearer so as to easily follow the work. ", "questions_to_authors": "1. How are r_i vectors encoded for different result types such as strings, numbers, and boolean?\n\n2. It is not clear how entity replacement works (described in section 4.2)? \n\n3. At t=0.8, 60% of the spurious programs remain in the program pool -> why at t=1.0, do we see too many false positives? This part could be made clearer.\n\n4. Is the code for this work going to be made public?", "paper_topic_and_main_contributions": "This paper presents a domain-agnostic approach aimed at addressing the issue of spurious programs in weakly supervised semantic parsing. The core concept revolves around executing generated programs on compatible worlds and utilizing a voting mechanism to identify and remove spurious examples.\n\nThe hard voting scheme involves obtaining centroids for each execution result in every world and calculating a score that reflects the distance from these centroids. By setting a hyper-parameter \"t\" as a threshold, programs with lower scores are considered more likely to be spurious. In addition, the authors introduce a soft-voting mechanism that takes into account the proportion of denotations.\n\nThe experimental evaluation conducted on the NLVR and WTQ datasets demonstrates the efficacy of their filtering mechanism when applied to state-of-the-art weakly supervised parsing methods for both datasets. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Looks good to me."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The problem definition described in section 2.1 looks a bit problematic. Each x_i is evaluated on worlds w_i (which is a set) - this warrants a subscript j for each world for example x_i. Similarly, another subscript is warranted for y, since y_ij is the evaluation of x_i on each world w_ij. \n\n2. Notations are harder to follow in section 4. j subscript represents the number of worlds, i subscript represents the number of programs. This could be made simpler. "}}, {"review_id": "ZxafapaRw0", "reviewer": "Reviewer_jkaF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. As you pointed out, there are some related works that are applicable to the tasks we addressed in our paper. \nIn terms of utilizing PLMs, we believe that transitioning from the LSTM-based models employed in our study to PLMs is a straightforward process. This involves replacing the LSTM seq2seq with a pretrained seq2seq model. It is important to note that most PLMs demand substantial computational resources for effective fine-tuning. Given this consideration, we chose to assess our approach using relatively compact LSTM-based models.\nIn addition, the original intention behind Self-consistent CoT did not encompass weakly supervised semantic parsing. Instead, it employed a majority vote mechanism to determine the most likely reasoning path in natural language reasoning tasks. Although our paper acknowledges Self-consistent CoT in the related work section due to its utilization of the majority vote, the primary focus diverges.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a simple method, Execution-Based Filtering, that can augment existing methods such as Logical Language Design to filter candidate programs. The paper shows that on NLVR, Execution-Based Filtering improves accuracy and consistency, and that on WTQ, Execution-Based Filtering improves accuracy.\n", "reasons_to_reject": "It would be helpful to see some additional comparisons with methods mentioned in the Related Work section, such as ones using pre-trained language models, self-consistent chain-of-thought prompting, etc. The paper mentions these as relatively recent advancements, so it would be helpful to understand the relationship between Execution-Based Filtering and these methods, e.g. how does the raw accuracy compare, could Execution-Based Filtering be applied to these methods, is there a theoretical relationship, etc.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the problem of spurious correlations in semantic parsing: incorrect programs may coincidentally return the correct answer when they are executed. To address this problem, the paper proposes a method called Execution-Based Filtering. The method relies on the intuition that even though a semantically incorrect program may coincidentally return the correct answer once, it is unlikely to consistently return the same answer as semantically correct programs when run multiple times over other configurations. \n\nThe paper evaluates its method on two datasets: Natural Language Visual Reasoning (NLVR) and WikiTableQuestions (WTQ) and shows improvements after it is used to augment existing methods such as Logical Language Design.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vut7iRlRpr", "reviewer": "Reviewer_ddAc", "summary": "", "questions": "A. Can you report some correlation statistics between (1) a program's similarity to the majority vote, and (2) whether a program is spurious? I would love to know the pearson-r correlation, AUC-ROC score, and the mean and std of the similarity scores when a program is spurious/not spurious.\n\nB. This is based on how I understand your algorithm: effectively the proposed algorithm is defining a metric space in the programs, where the pair-wise similarity is how often they agree. What if you filter out all programs that has an average distance too far away from the rest, instead of first computing the majority vote and then filter out those programs that disagree with the vote? (this is just my own curiosity, would not influence the score, though it'd be a plus if the performance actually improves)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's valuable feedback. In response, we'd like to address several points raised.\n\nAnalyzing the Intuition:\n\nIn order to address the reviewer's concerns, we've included additional statistics as requested. The Pearson-r correlation between the inverted program score (1-s_i) and the spuriousness label (0 for correct, 1 for spurious) is 0.358. The ROC AUC stands at 0.738. Additionally, the mean and standard deviation of scores (s_i) for correct programs are 0.997 and 0.029 respectively, while for spurious programs, they are 0.899 and 0.155.\n\nRegarding Question B:\n\nWe want to clarify that the mechanism you described aligns precisely with our soft vote approach. This approach demonstrates enhancements in WTQ, yet it leads to a decline in performance in NLVR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach seems interesting. \n- The paper is well-written", "reasons_to_reject": "- The gain in table 1 and 2 feels very incremental to me. The complexity involved in implementing this method might outweigh the empirical benefit. The problem exacerbates as the voting procedure introduces many other hyper-parameters to tune. \n- The paper adds little novelty to Shi et al. 2022 (this might not be a fair reason to reject though, given that this paper might have been written before 2022 since all baselines were pre-2020)\n- I am unsure how well this intuition holds empirically and can extend to more complex semantic parsing tasks: \"programs whose execution results largely deviate from those of other programs in the pool are likely to be spurious, thus filtering them out would improve the training of weakly supervised semantic parsers.\" I'd prefer having more statistics and results to back this assumption (which the algorithm is based on). See questions below.", "questions_to_authors": "A. Can you report some correlation statistics between (1) a program's similarity to the majority vote, and (2) whether a program is spurious? I would love to know the pearson-r correlation, AUC-ROC score, and the mean and std of the similarity scores when a program is spurious/not spurious.\n\nB. This is based on how I understand your algorithm: effectively the proposed algorithm is defining a metric space in the programs, where the pair-wise similarity is how often they agree. What if you filter out all programs that has an average distance too far away from the rest, instead of first computing the majority vote and then filter out those programs that disagree with the vote? (this is just my own curiosity, would not influence the score, though it'd be a plus if the performance actually improves)", "paper_topic_and_main_contributions": "The paper tries to address the \"ruling out spurious programs\" problem in weakly supervised semantic parsing. During the stage of generating several different candidate semantic parses to further the training, the author proposes to cluster the proposed semantic parses based on their \"denotation similarity\", as approximated by how often two semantic parses output the same results on a set of inputs. They show that the approach is effective on WTQ and NLVR.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of spurious correlations in weakly supervised semantic parsing with a filtering approach based on "denotation similarity" approximated by how often two programs output the same results on a set of inputs. 

Two out of three reviewers acknowledged the rebuttal.

All reviewers value the simple but effective method (on NLVR, Execution-Based Filtering improves accuracy and consistency, and on WTQ it improves accuracy), and give relatively high soundness and excitement scores (3.3 on average for each).  Still, the shortcomings mentioned by reviewers are quite serious, such the fear that the complexity may outweigh the benefits, and that the method may not extend well to more complex tasks, as well as the need for additional comparisons with methods using pre-trained language models, self-consistent chain-of-thought prompting etc. The latter of these is partly addressed in the rebuttal.
Decision: Accept-Main