SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation Metric. To comprehensively assess the capability of synthesizing programs based on provided inputs and outputs, we evaluate all models using three metrics: (1) Exact Match. It considers a generated program to be an exact match if every token in it is identical to the gold program. This is expressed mathematically as:  $\sum_{i=1}^{N} \Phi[\mathbf{TKN}_t(\mathcal{P}_i^*) = \mathbf{TKN}_t(\mathcal{P}_i)], \forall t \in [1, L]$  where  $\mathbf{TKN}_t(\mathcal{P}_i)$  refers to the  $t$ -th token of program  $\mathcal{P}_i$ ,  $\Phi[\cdot]$  yields 1 if the condition is satisfied and 0 otherwise, and  $L$  denotes the length of the program. (2) Semantic Match. It takes into account that a synthesized program may be semantically equivalent but not syntactically identical to the gold program. This metric is expressed as:  $\sum_{i=1}^{N} \Phi[\mathcal{P}_i^*(I_i^k) = \mathcal{P}_i(I_i^k) = O_i^k], \forall I_i^k, O_i^k \in \{IO\}_i^K$ . (3) Generalization Match. To genuinely evaluate the generalization ability of a model, it is crucial to assess its ability to generalize on unseen samples in addition to semantic consistency. To achieve this, we include additional examples and evaluate the model's performance as:  $\sum_{i=1}^{N} \Phi[\mathcal{P}_i^*(I_i^k) = \mathcal{P}_i(I_i^k) = O_i^k], \forall I_i^k, O_i^k \in \{IO\}_i^K \cup \{IO\}_i^{K_{test}}$ . Kindly note that our primary concern regarding the model's performance is the metric of generalization match.
Overall Synthesis Performance. Table 1 presents the primary outcomes of our EVAPS in comparison to prior works [7, 10, 43, 49] for VizDoom program synthesis. Notably, we have implemented the beam search technique for Inferred Trace, Latent Execution, and Transformer to enable the generation of multiple candidate programs while preserving their original architecture. Additionally, we have incorporated a convolutional neural network on top of the Transformer to facilitate the learning of specification embedding, given that the specification is in the form of grids rather than sequences, and the Transformer serves as the decoder. The supplementary material contains the detailed parameters of all the compared methods.
We can observe that EVAPS consistently outperforms other methods across three metrics, with an approximate improvement of  $+9.6\%$  compared to the runner-up. This indicates its superior capability to resolve potential semantic errors by incorporating environmental contexts, thereby enhancing its generalization ability while preserving syntax modeling ability. Moreover, EVAPS surpasses SED primarily due to the fact that SED only enhances program tokens based on execution outcomes and
Table 2: The average accuracy (with standard deviation) of all ablation methods evaluated on three metrics across Vizdoom tasks, assessed over 5 random seeds. The best results are highlighted in bold, and Gen.  $\Delta$  represents the mean improvement in generalization match.
Methods	Exact Match	Semantic Match	Generalization	Gen.Δ %
Top-1	Naïve	3.42% (1.80%)	32.29% (5.41%)	30.29% (5.36%)	-
EVAPS+O	23.56% (5.02%)	46.80% (2.53%)	45.24% (2.49%)	+14.95%
EVAPS+S	23.42% (1.84%)	42.47% (2.87%)	41.09% (2.79%)	+10.80%
EVAPS	36.40% (2.52%)	50.29% (2.18%)	48.90% (2.23%)	+18.61%
Top-5	Naïve	7.38% (3.15%)	52.11% (1.10%)	49.60% (1.50%)	-
EVAPS+O	40.15% (4.52%)	61.49% (1.85%)	59.71% (1.72%)	+10.11%
EVAPS+S	40.90% (4.80%)	58.18% (3.42%)	56.58% (3.20%)	+6.98%
EVAPS	50.62% (2.07%)	69.82% (1.72%)	67.82% (1.81%)	+18.22%
Top-20	Naïve	12.87% (3.25%)	64.80% (1.56%)	61.96% (1.57%)	-
EVAPS+O	55.35% (4.90%)	71.60% (2.54%)	69.63% (2.22%)	+7.67%
EVAPS+S	56.29% (5.26%)	69.85% (2.79%)	67.78% (2.56%)	+5.82%
EVAPS	62.22% (3.28%)	79.20% (1.08%)	76.84% (1.13%)	+14.88%
specifications, whereas EVAPS places greater emphasis on the semantic locality, supplemented by utilizing partial observations. While the attention mechanism of Transformer facilitates its ability to learn syntax, its generalization ability is relatively poor. Although LGRL can achieve comparable results in terms of generalization, it faces challenges in producing program sequences that are identical to the gold programs due to its tendency to generate semantically equivalent programs in a more complex environment. Inferring traces can provide information gain, but the partially observed environment restricts the model from capturing subtle semantic features. This limitation also applies to the latent executor, as the global status becomes agnostic to the model, thereby limiting its ability to extrapolate the execution states and learn the latent effects of each token.
Ablation Study. To thoroughly verify the efficacy of the partial observation leveraging module and the code symbol alignment module of EVAPS, we perform an ablation study to authenticate their capabilities and scrutinize their impacts as Table 2 presents. We consider the following ablations:
- Naïve: a variation of the program synthesis baseline LGRL [7] that utilizes greedy decoding (i.e., implements a beam search with beam size  $B = 1$ ). It employs a standard encoder-decoder architecture that acquires the ability to directly synthesize a program from scratch.
- EVAPS+O: an ablation of EVAPS incorporating partial environmental observations to modify tokens and resolve semantic conflicts in synthesized programs.
- EVAPS+S: an ablation of EVAPS aggregating long-range semantic and syntax connections of neighboring code symbols.
- EVAPS (EVAPS+O+S): incorporates both modules, allowing for a more comprehensive information flow and leveraging environmental context to facilitate program synthesis.
Based on the ablation outcomes presented in Table 2, we can deduce that utilizing only partial environmental observations or code symbol alignment can enhance the joint semantic and syntax modeling ability. However, the information gain is limited if only one of these techniques is employed. This is because the former concentrates primarily on the environmental context of a particular token, disregarding the implicit connections between different code segments in semantic spaces. On the other hand, the latter focuses mainly on information aggregation rather than environmental contexts. Consequently, combining these techniques can endow the model with supplementary information and further boost the program synthesis process, resulting in an approximate enhancement of  $+16\%$  in terms of generalization.
Task Complexity. Considering the varying complexities of different robot tasks, generating corresponding codes to accomplish them may pose varying levels of difficulty. Hence, we endeavor to delve deeper into the efficacy of each method in handling diverse levels of complexity. The issue of program aliasing, wherein two programs are semantically equivalent but not identical (e.g., while  $(r = 3)$  : turnLeft and turnRight), renders the evaluation of complexity using program
Figure 3: The generalization capability of all methods ranging from Top-1 to Top-20 evaluated across six categories that were partitioned based on the minimum number of steps required to complete the task. The tasks that can be accomplished with at least seven steps were denoted as  $7+$ .
length or control flow presence imperfect. Hence, we propose partitioning the tasks into different complexity levels based on the minimum number of steps required to complete the task, which we denote as steps.
To be specific, we have divided the testing set into six distinct categories based on their level of complexity, with each category requiring 2, 3, 4, 5, 6, or  $7+$  steps to complete, respectively. The  $7+$  category is reserved for tasks that demand no fewer than seven steps to accomplish. We then proceed to train and evaluate the model's performance in each category separately, and the results of this analysis are presented in Figure 3. Firstly, it is evident that as the complexity of the task increases, all methods experience a certain degree of performance degradation. This is reasonable since the embedding space becomes more intricate to learn as the task becomes more complex. Furthermore, we can infer that the performance gap between EVAPS and other methods widens as the complexity increases. This suggests that EVAPS is better equipped to adapt to more intricate tasks, owing to its ability to break down complex objectives into simpler behaviors by leveraging partial environmental contexts. On the other hand, the modeling ability of Latent Execution becomes relatively inferior, particularly for Step  $= 7+$ , which indirectly implies that it is challenging for a model to deduce the impact of complex behaviors in a single attempt.
Interestingly, for most techniques, the ability to generalize experiences a surge at approximately the Top-5 candidate programs. However, the rate of improvement tends to decelerate and the curve becomes flat. We surmise that this phenomenon occurs because certain semantic representations are conspicuous and uncomplicated to acquire, whereas others are more nuanced. Even with more candidate programs, the semantic features corresponding to these programs are somewhat overlapped, thereby reaching the limit for capturing semantic subtleties and hindering further generalization.
Handling Noisy Observations. For robot programming, real-world observations are collected through sensors in real-time, making it inevitable for occasional noise to occur. Consequently, we tend to evaluate the ability of our method to handle such noise. Given that the VizDoom dataset lacks noisy examples, we artificially introduce noise into the dataset by invalidating a certain percentage of observation grids based on uniform random probability [15].
The results regarding different levels of noise (with a maximum noise of  $20\%$ ) are illustrated in Figure 4. It can be inferred that the presence of noisy observations inevitably undermines the capacity of EVAPS to learn program semantics and generalize. Nevertheless, EVAPS still manages to exhibit a decent modeling ability. If we consider all Top-K potential candidates, the maximum performance drop obtained is merely about  $7.5\%$ . Meanwhile, with fewer candidate programs, the performance gap widens, but it is bound to be  $12\%$ . This outcome aligns with our expectations, thus we can conclude that EVAPS is capable of recognizing significant semantic patterns and maintaining robustness when encountering noises.
Figure 4: The generalization capacity of EVAPS in the presence of diverse levels of noise, accompanied by a  $95\%$  confidence interval band.
Discussion. The partial observation that the proposed approach relies on is often available in practice, thus it is theoretically feasible to apply this approach to the real world. Certainly, we anticipate some practical difficulties during the transition from the simulation environment to the real world. For instance, it can be labor-intensive to collect enough environmental data from the real world for the purpose of training. Further investigation into the efficient acquisition of environmental data while sustaining its quality is yet to be explored in future works.
Programming By Example. In the realm of program synthesis, the synthesis of programs from examples has long been a preoccupation of researchers. In recent years, deep learning methods have made many breakthroughs [37, 13, 36], and many researchers have proposed different methods for Improve the accuracy of synthesis. Symbolic search is regarded as an efficient technique for addressing program synthesis challenges [40, 38, 51, 26]. Wang et al. have devised a language for abstract queries, which facilitates the generation of SQL [51]. Moreover, Spoc has recognized the significance of pseudocode and explores its potential for aiding code search, focusing on alternative translations of pseudocode [26]. While symbolic search can yield promising outcomes within specific training domains, it lacks the ability to expand, and it is relatively hard to generate code snippets beyond the training datasets.
To tackle the issue of code fragment homogeneity, two approaches have gained prominence: the utilization of hidden variables and the employment of reward mechanisms. In order to effectively utilize the information contained in latent variables, some researchers consider using latent generative models [3, 10, 48]. Aim at obtaining the intermediate execution results of partial programs in C language, Chen et al. approximate by learning the latent representation of partially generated programs to facilitate program token search [10]. Additionally, reinforcement learning has been integrated into synthetic neural network training in numerous studies [7, 16, 9, 48, 19]. Bunel et al. combined program synthesis with reinforcement learning [7], and avoided the problem of Program Aliasing by defining proper rewards; Trivedi et al. considered a two-stage learning scheme on this basis [48]. First, a program embedding space is learned in an unsupervised manner, and then the most suitable program is searched in the embedding space with a given reward.
In our approach, we also use a reward mechanism for generating candidate programs. However, we extend this by adopting the concept from SED [19] to train an evaluator that assesses the candidate programs for further repair. While both EVAPS and SED incorporate environmental observation to train the neural program debugger, there are notable distinctions between the two: i) The execution feedback of SED relies on a global perspective, which is only available in some special cases. Conversely, EVAPS embraces partial observation, which is more achievable in real-world scenarios. ii) SED treats the program as a whole for training, while EVAPS pays more attention to the execution context. iii) EVAPS establishes a connection between the program execution context and the partial observable environment. Distinguishing itself from SED, EVAPS places greater emphasis on the partially observable environment in which the robot operates and the local context of the program.
We believe that this approach enables the network to concentrate more precisely on the actual scope of the impact of the program.
Program Repair. Program repair has also been a widely researched topic. Most researchers pay attention to the grammatical features [30, 29, 45] and semantic information [52, 54] of programs. Rolim and Hua [39, 20] employ abstract syntax trees to derive effective outputs, while Tfix [4] treats programs as text and trains a general Transformer model for repairs. While these approaches have achieved breakthroughs in different problem domains, they often overlook the semantic aspects of programs. Wang et al. address this by incorporating the learning of semantic embeddings from program execution trajectories [52], while Ye et al. enhance neural networks using a loss function based on program compilation and execution information [54]. Our work takes both semantic and grammatical information into account. Our evaluation network, which is based on the execution information of the program in the environment and the program's grammar domain-specific language, can effectively accomplish program repairs.
Regarding traditional program repair approaches, we are aware of a branch of rich works in the program repair area, which can be classified into three categories: i) Search-based. This type of approach considers program repair as a search problem, exploring the space of all possible program candidates to identify one that satisfies the given weak specification, i.e., test cases. One of the prior works in this area is GenProg [31]. ii) Semantic-based. This type of approach extracts semantic information from the program under repair (typically represented as path constraints) and then generates patches by solving those constraints. A well-known work is SemFix [35]. iii) Learning-based. This type of approach leverages a number of patches generated by developers to learn a model that repairs programs. An earlier work is Prophet [33]. Those approaches depend on high-quality test suites to validate patch candidates, which are unavailable in the setting our approach is targeted at, i.e., VizDoom.
This paper introduces EVAPS, a novel approach to augment the applicability of neural program synthesis by integrating partial environmental observations. By utilizing both the environmental context leveraging module and the code symbol alignment module, the ability to rectify semantically erroneous program segments and generalize across various tasks is substantially enhanced. Furthermore, we verify that EVAPS exhibits greater resilience when confronted with noise and can adeptly capture subtle features even under more intricate tasks. The proposed approach presents a promising direction for robot program synthesis by incorporating partial environmental observations. The ability of the framework to rectify semantically erroneous program segments, generalize across tasks, and maintain robustness in the presence of noise showcases its potential for practical applications in robot program synthesis.
This work was supported by Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103).
[1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
[2] Maria Bakhanova and Ilya Makarov. Deep reinforcement learning in vizdoom via dqn and actor-critic agents. In Proceedings of IWANN, 2021.
[3] Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In Proceedings of ICLR, 2017.
[4] Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. Tfix: Learning to fix coding errors with a text-to-text transformer. In Proceedings of ICML, 2021.
[5] Matko Bošnjak, Tim Roktáschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable forth interpreter. In Proceedings of ICML, 2017.
[51] Chenglong Wang, Alvin Cheung, and Rastislav Bodik. Synthesizing highly expressive sql queries from input-output examples. In Proceedings of PLDI, 2017.
[52] Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embeddings for program repair. In Proceedings of ICLR, 2018.
[53] Marek Wydmuch, Michal Kempka, and Wojciech Jaskowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games, 2019.
[54] He Ye, Matias Martinez, and Martin Monperrus. Neural program repair with execution-based backpropagation. In Proceedings of ICSE, 2022.
[55] Haiyan Yin, Jianda Chen, Sinno Jialin Pan, and Sebastian Tschiatschek. Sequential generative exploration model for partially observable reinforcement learning. In Proceedings of AAAI, 2021.
[56] Amit Zohar and Lior Wolf. Automatic program synthesis of long programs with a learned garbage collector. In Proceedings of NeurIPS, 2018.