SlowGuess/NeurIPS_2023_OCR huggingface dataset:
tianychen21@m.fudan.edu.cn
Qidi Wang
21210240038@m.fudan.edu.cn
Zhen Dong†
zhendong@fudan.edu.cn
Liwei Shen
shenliwei@fudan.edu.cn
Xin Peng
pengxin@fudan.edu.cn
Program synthesis aims to automatically generate an executable program that conforms to the given specification. Recent advancements have demonstrated that deep neural methodologies and large-scale pretrained language models are highly proficient in capturing program semantics. For robot programming, prior works have facilitated program synthesis by incorporating global environments. However, the assumption of acquiring a comprehensive understanding of the entire environment is often excessively challenging to achieve. In this work, we present a framework that learns to synthesize a program by rectifying potentially erroneous code segments, with the aid of partially observed environments. To tackle the issue of inadequate attention to partial observations, we propose to first learn an environment embedding space that can implicitly evaluate the impacts of each program token based on the precondition. Furthermore, by employing a graph structure, the model can aggregate both environmental and syntactic information flow and furnish smooth program rectification guidance. Extensive experimental evaluations and ablation studies on the partially observed VizDoom domain authenticate that our method offers superior generalization capability across various tasks and greater robustness when encountering noises.
Program synthesis endeavors to produce an executable program that fulfills the prescribed specifications. Given that formulating a definitive and lucid specification can often prove more arduous than physically authoring the program, a widely-used method is to furnish input/output (I/O) examples as a close approximation of the requisite specification [16, 17, 27, 34], also known as Programming By Example (PBE). It allows a more straightforward presentation of the desired program functionality. The utilization of PBE enables machines to acquire the capacity to implicitly capture a set of regulations and generalize them to analogous scenarios. Over the past few years, PBE has demonstrated its remarkable potential to elevate automation and ease human labor in numerous domains, including but not limited to array transformation [3, 46, 56], graphic design [16], and comprehension of human demonstrations [21, 42].
In the realm of robot programs, a crucial factor is the interaction between the robot and its environment. In practice, a robot can solely rely on its sensors to perceive the environment by gathering diverse types of data, which forms the basis for every decision. Consider a robot designed for food delivery,
†Corresponding author
which is tasked with transporting food to a specified destination. The robot is faced with a decision between two paths: one that is obstructed by barriers and leads to a dead-end, and another that is more circuitous and distant. If the robot is furnished with comprehensive knowledge of the environment, it may be possible to create a simple program consisting of only a few actions. However, if the robot can only obtain partial observations, such as RGB image data from its current perspective, a more intricate program with sophisticated control flow may be necessary to navigate out of the dead-end and reach the intended destination. Consequently, the latter program can generalize better in accomplishing more complex tasks. The variation in the level of observation can significantly impact the ultimate outcome. Therefore, the challenge in robot program synthesis arises from the fact that robots are limited to partial observations of their surroundings, making it difficult to assess the global impact of the generated program tokens toward the desired output. This is due to the fact that a comprehensive understanding of the environment is not achievable.
At present, there exists a diverse range of deep learning techniques aimed at solving PBE problems using neural network models [5, 8, 32]. Empirical studies [3, 13] have shown that recurrent neural network (RNN) architectures can learn strategies that generalize across problems and remain robust to moderate levels of noise. However, these methods have limitations in their ability to effectively utilize environmental observations due to their reliance on the RNN as the backbone. Additionally, recent studies have introduced reinforcement learning [7, 28], debugger method [19] and latent generative methods [10, 48] into the field. Furthermore, other studies have revealed that incorporating intermediate states produced by executing partially generated programs can improve the capability to address program synthesis tasks, especially for sequential programming tasks [9, 43, 47, 56]. These techniques have primarily been evaluated on the Karel [12, 18, 44] dataset, in which the model possesses complete awareness of the entire environment. To clarify, even if a particular location in the environment is inaccessible or unknown to the robot, the model can still generate a program with knowledge of that spot. Nonetheless, this assumption of acquiring a global view of the environment is arduous to achieve in reality, and their aptitude to generate accurate programs with only partial observations remains to be explored.
To tackle this issue, we introduce our novel Environmental-context Validated lAtent Program Synthesis framework (EVAPS). Drawing inspiration from the successful previous work SED [19], as well as the current trend of large language models [1, 22, 23], we believe that the trail-eval-repair loop provides valuable guidance for the evolution of programs, leading to better generalization ability. In order to leverage partial environmental observations, EVAPS initially obtains candidate programs by employing the same neural program synthesizer component used in SED. Subsequently, by executing the candidate program, EVAPS acquires the partial environment before and after each action (i.e., the environmental context). By concurrently modeling both the environmental and syntax contexts, EVAPS iteratively alters the programs that do not produce the correct output in an effort to resolve semantic conflicts across program tokens, ultimately correcting erroneous program fragments.
The framework was assessed in the partially observed VizDoom domain [24, 55], where a robot operates in a 3D world and interacts with its environment, including objects and adversaries. The results of the experiment demonstrate that the proposed approach is superior in modeling program semantic subtleties and resolving potential errors, thereby enhancing its generalization ability. Additionally, we authenticate the efficacy of leveraging environmental contexts and aligning them with code syntax by carrying out an ablation study. Further experiments show that our method is more capable of solving complex tasks compared to prior works. Furthermore, we validate that our method remains robust when encountering various levels of noise.
In a typical PBE task, we are given a set of  $N$  training samples, represented as  $\left\{\{IO\}^K,\mathcal{P}\right\} ^N$ . Each sample consists of  $K$  I/O pairs that act as specifications, and a gold program  $\mathcal{P}$  that correctly maps the input state  $I$  to the output state  $O$ , such that  $\mathcal{P}(I_i^k)\to O_i^k$ , where  $\forall i\in [1,N],\forall k\in [1,K]$ . Previous studies have investigated techniques to synthesize a candidate program  $\hat{\mathcal{P}}$  solely based on I/O. Our objective is to train a neural model, denoted as  $\mathcal{G}$ , which takes a candidate program  $\hat{\mathcal{P}}$ , and the corresponding environmental contexts  $\mathcal{O}$  obtained by executing the program  $\hat{\mathcal{P}}$  as an input. The model generates a refined program by incorporating environmental contexts:  $\mathcal{G}(\hat{\mathcal{P}},\mathcal{O})\rightarrow \mathcal{P}^*$ , such that the mapping of program  $\mathcal{P}^* (I)$  approaches more closely to  $\mathcal{P}(I)$  given input  $I$ . To assess the
Figure 1: The EVAPS architecture proposed involves the preprocessing of the candidate program into a program graph, which can be further utilized for code segment generation and code symbol representation learning. The encoder comprises (a) the partial observation leveraging module that utilizes insights gleaned from partial observations by learning latent environmental context representations, and (b) the code symbol alignment module that brings richer semantic features by aggregating neighbor syntax symbol representations. Eventually, the decoder outputs refined program tokens by incorporating both modules.
generalization ability of the model  $\mathcal{G}$ , we include another set of held-out examples that are not part of the training samples, denoted as  $\{IO\}^{K_{test}}$ , enabling us to measure the equivalence between  $\mathcal{P}^*$  and  $\mathcal{P}$  using  $IO^k \in \{IO\}^K \cup \{IO\}^{K_{test}}$ .
3.1 Leveraging Partial Observations
In the VizDoom domain, the robot perceives its surroundings  $\mathcal{O}$  by collecting an RGB image  $\mathcal{O}_R\in \mathbb{R}^{n_H\times n_W\times n_C}$  and a depth buffer  $\mathcal{O}_D\in \mathbb{R}^{n_H\times n_W\times 1}$  from its visual perspective, denoted as  $\mathcal{O} = \langle \mathcal{O}_R,\mathcal{O}_D\rangle$ . This forms its partial observation of the world. By executing a program, we can obtain a sequence of observation changes  $\mathcal{O}_0\xrightarrow{ac_1}\mathcal{O}_1\xrightarrow{ac_2}\dots\xrightarrow{ac_{n_e}}\mathcal{O}_{n_e}$ , where  $ac_i$  refers to the  $i$ -th executed action,  $n_e$  denotes the number of total execution steps, and  $\mathcal{O}_0$  denotes the initial observed state. As depicted in Figure 1(a), the hidden environment representations can be learned by applying convolutional network layers, and the embedded vector at the  $l$ -th layer can be denoted as:
$$
\mathcal {O} _ {x, y} ^ {[ l ]} = \operatorname {C o n v} \left(\mathcal {O} _ {x, y} ^ {[ l - 1 ]}, \mathbf {K}\right) = g ^ {[ l ]} \left(\sum_ {h = 1} ^ {n _ {H} ^ {[ l ]}} \sum_ {w = 1} ^ {n _ {W} ^ {[ l ]}} \sum_ {c = 1} ^ {n _ {C} ^ {[ l ]}} \mathbf {K} _ {h, w, c} \cdot \mathcal {O} _ {x + h - 1, y + w - 1, c} ^ {[ l - 1 ]}\right), \tag {1}
$$
where  $\mathbf{K}$  represents the convolutional kernel, and  $g^{[l]}$  signifies the activation function at layer  $l$ .
To effectively utilize insights gleaned from partial observations, it is essential that we obtain the antecedent states  $\mathcal{O}_{pre}$  (also known as the precondition) and the potential consequent effects  $\mathcal{O}_{post}$  corresponding to each token of program  $\hat{\mathcal{P}}$ , so that the model can implicitly evaluate whether this token contributes towards completing the task in the current context, and determine whether substitution with another token would heighten the likelihood of task fulfillment.
Given that only those tokens that denote robot actions (e.g., moveForward) have the potential to alter the environment, whereas other tokens function as constituents of a particular statement (e.g., if and while), a straightforward token-by-token modification strategy appears unappealing. Since non-action tokens cannot exist independently and can only render a code block semantically complete when they co-occur with action tokens, we deem it more fitting to process them as a unified segment.
We define a segment as a triplet comprising of  $S = \langle cond, token, body \rangle$ . Here, cond refers to the condition of the segment, and body denotes the associated actions or subsegments. In the case of an action token, the segment form deteriorates into  $S = \langle null, action, null \rangle$ . By recursively parsing the abstract syntax tree of program  $\hat{\mathcal{P}}$ , we can eventually yield a collection consisting of  $T$  program segments, denoted as  $S_T$ . And by executing program segment  $S_t \in S_T$  using the VizDoom interpreter, we can acquire the post-segment effect with  $Execute(S_t, \mathcal{O}_{pre}) \to \mathcal{O}_{post}$ , thereby obtaining the environmental context  $\langle \mathcal{O}_{pre}^t, \mathcal{O}_{post}^t \rangle$  of segment  $S_t$ , presented in Figure 1. The interconnection between the observable world state change and the code segment promotes greater semantic versatility, which, in turn, enables the model to calculate the probability of substituting each token based on the partial environment, resulting in the following formulation:
$$
p _ {\theta} \left(\mathcal {P} _ {m} ^ {*} \mid \mathcal {O} _ {p r e}, \mathcal {O} _ {p o s t}\right) = \prod_ {S _ {t} \in S _ {T}} \left(\prod_ {s = 1} ^ {L _ {t}} p _ {\theta} \left(\epsilon_ {s} ^ {\prime} | \epsilon_ {1}, \dots , \epsilon_ {s - 1}, \epsilon_ {s + 1}, \dots , \epsilon_ {L _ {t}}, \mathcal {O} _ {p r e} ^ {t}, \mathcal {O} _ {p o s t} ^ {t}\right)\right), \tag {2}
$$
where  $L_{t}$  denotes the number of tokens in segment  $S_{t}$ ,  $\epsilon_{s}$  refers to the  $s$ -th token of segment  $S_{t}$ , and  $\epsilon_{s}^{\prime}$  represents the refined program token that substitutes  $\epsilon_{s}$ . To optimize the model parameters, the objective is to minimize the cross-entropy loss, given  $N$  training examples, where  $S_{T}^{i}$  represents the program segment set of the  $i$ -th example, which contains  $T_{i}$  segments:
$$
\mathcal {L} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{T _ {i}} \sum_ {S _ {t} \in S _ {T} ^ {i}} \frac {1}{L _ {t}} \sum_ {s = 1} ^ {L _ {t}} \log \left[ p _ {\theta} \left(\epsilon_ {s} ^ {\prime} | \epsilon_ {1}, \dots , \epsilon_ {s - 1}, \epsilon_ {s + 1}, \dots , \epsilon_ {L _ {t}}, \mathcal {O} _ {p r e} ^ {t}, \mathcal {O} _ {p o s t} ^ {t}\right) \right]. \tag {3}
$$
3.2 Code Symbol Alignment
Now that the environmental contexts have been obtained, they are currently paired separately with their own code segments, and the correlations among environmental contexts are not fully exploited. Relying solely on partial observation is insufficient for the model to comprehend the semantic and syntactic long-range connections of program tokens, making it arduous for the model to consolidate information that traverses through subsequent segments during the program repair process. Furthermore, with regard to program syntax, the inherent nature of a program necessitates the model to jointly reason over code symbols, such as types. For instance, the code while (turnLeft) is syntactically incorrect because turnLeft is an action rather than a perception, which is prohibited to follow while. Therefore, it is desirable to align the code symbols with the partial observations to facilitate a more comprehensive information flow.
To accomplish this, we first construct a program graph  $G \subseteq V \times E$  (presented in Figure 1) to represent each candidate program  $\hat{\mathcal{P}}$ , where  $V$  denotes the set of program token nodes and  $E$  denotes the set of undirected edges. Subsequently, we employ graph attention [6, 50] for each token to capture a contextualized representation. Similar to the definition of a segment, each node within the program graph possesses a leftward pointer that directs towards nodes signifying conditions, and a rightward pointer that indicates a subsequent token node. The node's value encompasses the token index, and token type. Meanwhile, the relationship between tokens and their corresponding segments is stored. In this way, we not only align the code symbols and environmental contexts but also aggregate further semantic and syntactical representations. Additionally, it enables the model to effectively capture the nested structure of a program, thereby bringing more expressiveness.
To elaborate, we begin by obtaining the node representations  $\pmb{h}_i \in \mathbb{R}^d$  for each node  $V_i \in G$  through embedding. Subsequently, a scoring function  $\alpha: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$  is employed to compute the attention score of its neighboring node  $V_j$ . This score reflects the significance of a neighbor node. The attention scores are then normalized across all neighbors  $j \in \mathcal{N}_i$  using softmax, which is defined as:
$$
\alpha \left(\boldsymbol {h} _ {i}, \boldsymbol {h} _ {j}\right) = \frac {\exp \left(\boldsymbol {A} ^ {\top} \text {L e a k y R e L U} \left(\boldsymbol {W} \cdot \left[ \boldsymbol {h} _ {i} \| \boldsymbol {h} _ {j} \right]\right)\right)}{\sum_ {k \in \mathcal {N} _ {i} \cup \{i \}} \exp \left(\boldsymbol {A} ^ {\top} \text {L e a k y R e L U} \left(\boldsymbol {W} \cdot \left[ \boldsymbol {h} _ {i} \| \boldsymbol {h} _ {k} \right]\right)\right)}, \tag {4}
$$
where  $\mathbf{A}$  and  $\mathbf{W}$  are weight matrices that can be learned, and  $\| \cdot$  represents vector concatenation. As a result, node  $V_{i}$  acquires a new representation by calculating the weighted mean of the modified features of its neighboring nodes, as illustrated in Figure 1(b):
$$
\boldsymbol {h} _ {i} ^ {\prime} = \alpha_ {i, i} \cdot \boldsymbol {W h} _ {i} + \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i, j} \cdot \boldsymbol {W h} _ {j}. \tag {5}
$$
Next, we can obtain the corresponding environmental embedding of the token in  $V_{i}$  through  $\mathcal{O}_i' = \mathrm{Conv}(\mathcal{O}_i, \mathbf{K})$  (Eq. (1)). To align the code symbol representations and environmental contexts, we update the representation of node  $V_{i}$  by simply concatenating these embeddings  $\mathbf{x}_i = [h_i' \| \mathcal{O}_i']$ . This leads to a modification to Eq. (2):
$$
p _ {\theta} \left(\mathcal {P} _ {m} ^ {*} \mid \mathcal {O} _ {p r e}, \mathcal {O} _ {p o s t}\right) = \prod_ {S _ {t} \in S _ {T}} \left(\prod_ {s = 1} ^ {L _ {t}} p _ {\theta} \left(\epsilon_ {s} ^ {\prime} | \mathbf {x} _ {s}\right)\right). \tag {6}
$$
The key connection between the proposed approach and observable environments resides in establishing a connection between the program execution context and the partially observable environment. EVAPS takes the execution context of segment  $S$  and the corresponding partially observable environment as a unit for model training. This enables the combination of program syntax and the corresponding partially observable environment to predict a token, thereby enhancing the accuracy of program synthesis. As a result, the model can adeptly resolve the underlying conflict within a program by concurrently capturing well-integrated and highly pertinent semantic and contextual patterns, culminating in a greater practicality of programs.
VizDoom Domain. The programs in this study are structured using Domain-Specific Languages (DSL) in the VizDoom environment [24]. This environment provides a 3D world that closely resembles real-world scenarios, allowing the robot to efficiently perceive, interpret, and learn the 3D realm. In recent times, there has been a burgeoning interest in VizDoom exploring the robot's capabilities to make tactical and strategic determinations [2, 11, 14, 25, 41, 47, 53]. The VizDoom experimental environment consists of 7 action primitives, 6 perception primitives, and the state representation dimension is  $120 \times 160 \times 3$ . The average number of steps required to complete
tasks is 4.6. We contend that the VizDoom environment is more suitable for robot program synthesis research compared to Karel, due to its limited partially observed nature [55], which renders the decision and execution process more realistic and practical for the robot. The DSL comprises actions, perceptions, and control flows, sufficient to encapsulate the fundamental concepts of robot programming. The detailed description is presented in Figure 2.
Dataset. To generate a dataset for learning environmental-context embeddings, we adopt the same approach as previous studies [14, 47], randomly generating 100,000 distinct samples. The dataset is then partitioned into a training set with 70,000 samples, a validation set with 20,000 samples, and a testing set with 10,000 samples. We adhere to the program synthesis conventions [7, 19], where each sample includes a correct program, 5 input-output pairs that act as specifications, and an extra input-output example as the test sample, which is solely employed for evaluation purposes and not for training. During the generation process, we also conduct checks to ensure that all execution branches in the program are covered at least once, thereby representing all aspects of the program's behavior and providing sufficiently challenging tasks.
Program  $m\coloneqq$  def run  $)s$
$$
S t a t e m e n t s := \text {w h i l e} (b): (s) \mid s _ {1}; s _ {2} \mid a \mid \text {r e p e a t} (r): (s) \mid
$$
$$
\mid \text {i f} (b): (s) \mid \text {i f e l s e} (b): (s _ {1}) \text {e l s e}: (s _ {2})
$$
Repetition  $r \coloneqq$  Integer number of repetitions
Condition  $b\coloneqq$  perception  $\gamma$  | not  $^b$
$$
\text {A c t i o n} a := \text {m o v e F o r w a r d} \mid \text {m o v e B a c k w a r d} \mid \text {m o v e L e f t}
$$
$$
| \text {m o v e R i g h t} | \text {t u r n L e f t} | \text {t u r n R i g h t} | \text {a t t a c k}
$$
Perception  $\gamma \coloneqq$  isThere  $\varepsilon$  |inTarget  $\varepsilon$
$$
\text {M o n s t e r} \varepsilon := \text {d e m o n} | \text {h e l l K n i n g t} | \text {r e v e n a n t}
$$
Figure 2: The Domain-Specific Language (DSL) for Vizdoom programs comprises action primitives, perception primitives and control flows.
environment is more suitable for robot program synthesized partially observed nature [55], which renders it safe and practical for the robot. The DSL comprises a set to encapsulate the fundamental concepts of robot design in Figure 2.
Table 1: The average accuracy (with standard deviation) of all methods evaluated on three metrics across Vizdoom tasks, assessed over 5 random seeds. The best results are highlighted in bold.
Methods	Exact Match	Semantic Match	Generalization
Top-1	LGRL [7]	2.18% (0.34%)	33.42% (1.12%)	31.49% (1.26%)
SED [19]	12.69% (0.55%)	43.07% (0.57%)	38.63% (0.51%)
Inferred Trace [43]	14.73% (1.73%)	36.87% (1.81%)	35.09% (1.73%)
Latent Execution [10]	3.53% (1.12%)	44.40% (2.31%)	42.25% (2.22%)
Transformer [49]	15.53% (1.75%)	26.51% (1.96%)	25.93% (1.77%)
EVAPS (Ours)	36.40% (2.52%)	50.29% (2.18%)	48.90% (2.23%)
Top-5	LGRL [7]	4.58% (0.43%)	54.25% (1.01%)	51.56% (1.08%)
SED [19]	25.48% (0.89%)	66.76% (0.92%)	62.25% (0.65%)
Inferred Trace [43]	27.49% (1.87%)	55.13% (0.59%)	52.80% (0.88%)
Latent Execution [10]	4.91% (1.32%)	50.76% (2.07%)	48.62% (1.81%)
Transformer [49]	34.80% (4.23%)	47.96% (2.24%)	46.76% (2.13%)
EVAPS (Ours)	50.62% (2.07%)	69.82% (1.72%)	67.82% (1.81%)
Top-20	LGRL [7]	9.16% (0.76%)	66.22% (0.19%)	63.49% (1.06%)
SED [19]	51.23% (1.01%)	71.42% (3.66%)	67.06% (3.76%)
Inferred Trace [43]	37.75% (2.10%)	66.69% (1.82%)	63.96% (1.54%)
Latent Execution [10]	9.16% (1.21%)	59.93% (1.96%)	57.53% (1.50%)
Transformer [49]	51.53% (3.61%)	64.69% (2.90%)	63.09% (2.77%)
EVAPS (Ours)	62.22% (3.28%)	79.20% (1.08%)	76.84% (1.13%)