SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multiplicative RASMs and equivalence In this work we introduce multiplicative RASMs which need to decrease in expected value by at least some multiplicative factor  $\gamma \in (0,1)$ . We also impose the strict positivity condition outside of the target set  $\mathcal{X}_t$ . Note that any  $(\epsilon, \lambda)$ -additive RASM satisfies this condition with the lower bound  $\epsilon > 0$ .
Definition 3 (Multiplicative reach-avoid supermartingales). Let  $\gamma \in (0,1)$ ,  $\delta > 0$  and  $\lambda > 1$ . A continuous function  $V: \mathcal{X} \to \mathbb{R}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative reach-avoid supermartingale  $((\gamma, \delta, \lambda)$ -multiplicative RASM) with respect to  $\mathcal{X}_t$  and  $\mathcal{X}_u$ , if:
1. Nonnegativity condition.  $V(\mathbf{x})\geq 0$  for each  $\mathbf{x}\in \mathcal{X}$
2. Strict positivity outside  $\mathcal{X}_t$  condition.  $V(\mathbf{x})\geq \delta$  for each  $\mathbf{x}\in \mathcal{X}\backslash \mathcal{X}_t$
3. Initial condition.  $V(\mathbf{x})\leq 1$  for each  $\mathbf{x}\in \mathcal{X}_0$
4. Safety condition.  $V(\mathbf{x})\geq \lambda$  for each  $\mathbf{x}\in \mathcal{X}_u$
5. Multiplicative expected decrease condition. For each  $\mathbf{x} \in \mathcal{X} \setminus \mathcal{X}_t$  at which  $V(\mathbf{x}) \leq \lambda$ , we have  $\gamma \cdot V(\mathbf{x}) \geq \mathbb{E}_{\omega \sim d}[V(f(\mathbf{x}, \pi(\mathbf{x}), \omega))]$ .
We show that a function  $V:\mathcal{X}\to \mathbb{R}$  is an additive RASM if and only if it is a multiplicative RASM.
Theorem 1. [Proof in Appendix C] The following two statements hold:
1. If a continuous function  $V: \mathcal{X} \to \mathbb{R}$  is an  $(\epsilon, \lambda)$ -additive RASM, then it is also a  $(\frac{\lambda - \epsilon}{\lambda}, \min \{\epsilon, \lambda\}, \lambda)$ -multiplicative RASM.
2. If a continuous function  $\dot{V} : \mathcal{X} \to \mathbb{R}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM, then it is also an  $((1 - \gamma) \cdot \delta, \lambda)$ -additive RASM.
Improved bound The following theorem shows that the existence of a multiplicative RASM implies a lower bound on the probability with which the system satisfies the reach-avoid specification.
Theorem 2. [Proof in Appendix D] Let  $\gamma \in (0,1)$ ,  $\delta > 0$  and  $\lambda > 1$ , and suppose that  $V: \mathcal{X} \to \mathbb{R}$  is a  $(\gamma, \delta, \lambda)$ -multiplicative RASM with respect to  $\mathcal{X}_t$  and  $\mathcal{X}_u$ . Suppose furthermore that  $V$  is Lipschitz continuous with a Lipschitz constant  $L_V$ , and that the system under policy  $\pi$  satisfies the bounded step property, i.e. that there exists  $\Delta > 0$  such that  $\| \mathbf{x} - f(x, \pi(\mathbf{x}), \omega) \|_1 \leq \Delta$  holds for each  $\mathbf{x} \in \mathcal{X}$  and  $\omega \in \mathcal{W}$ . Let  $N = \lfloor (\lambda - 1) / (L_V \cdot \Delta) \rfloor$ . Then, for every  $\mathbf{x}_0 \in \mathcal{X}_0$ , we have that
$$
\mathbb {P} _ {\mathbf {x} _ {0}} \left[ \operatorname {R e a c h A v o i d} \left(\mathcal {X} _ {t}, \mathcal {X} _ {u}\right) \right] \geq 1 - \frac {1}{\lambda} \cdot \gamma^ {N}.
$$
Here,  $N = \left\lfloor (\lambda - 1) / (L_V \cdot \Delta) \right\rfloor$  is the smallest number of time steps in which the system could hypothetically reach the unsafe set  $\mathcal{X}_u$ . This is because, for the system to violate safety, by the Initial and the Safety conditions of multiplicative RASMs the value of  $V$  must increase from at most 1 to at least  $\lambda$ . But the value of  $V$  can increase by at most  $L_V \cdot \Delta$  in any single time step since  $\Delta$  is the maximal step size of the system and  $L_V$  is the Lipschitz constant of  $V$ . Hence, the system cannot reach  $\mathcal{X}_u$  in less than  $N = \left\lfloor (\lambda - 1) / (L_V \cdot \Delta) \right\rfloor$  time steps.
We conclude this section by comparing our bound to the bound  $\mathbb{P}_{\mathbf{x}_0}[\mathrm{ReachAvoid}(\mathcal{X}_t,\mathcal{X}_u)]\geq 1 - \frac{1}{\lambda}$  of [68] for additive RASMs. Our bound on the probability of violating reach-avoidance is tighter by a factor of  $\gamma^{N}$ . Notice that this term decays exponentially as  $\lambda$  increases, hence our bound is exponentially tighter in  $\lambda$ . This is particularly relevant if we want to verify reach-avoidance with high probability that is close to 1, as it allows using a much smaller value of  $\lambda$  and significantly relaxing the Safety condition in Definition 3. To evaluate the quality of the bound improvement on an example, in Table 1 we consider the 9-Rooms environment and compare the bounds for multiple values of  $\lambda$  when  $\gamma = 0.99$ . Note that  $\Delta$  is the property of the system and not the RASM. Thus, the value of  $\lambda$  is the key factor for controlling the quality of the bound. Results in Table 1 show that, in order to verify probability  $99.9\%$  reach-avoidance for an edge policy in the 9-Room example, our new bound allows using  $\lambda \approx 100$  whereas with the old bound the algorithm needs to use  $\lambda \approx 1000$ .
Remark 1 (Comparison of RASMs and stochastic barrier functions). Stochastic barrier functions (SBFs) [45, 46] were introduced for proving probabilistic safety in stochastic dynamical systems, i.e. without the additional reachability condition as in reach-avoid specifications. If one is only interested in probabilistic safety, RASMs of [68] reduce to SBFs by letting  $\mathcal{X}_t = \emptyset$  and  $\epsilon = 0$  in Definition 2 for additive RASMs, and  $\mathcal{X}_t = \emptyset$  and  $\gamma = 1$  in Definition 3 for multiplicative RASMs. The bound on the probability of reach-avoidance  $\mathbb{P}_{\mathbf{x}_0}[\text{ReachAvoid}(\emptyset, \mathcal{X}_u)] \geq 1 - \frac{1}{\lambda}$  of [68] for additive RASMs coincides with the bound on the probability of safety implied by SBFs of [45, 46]. Hence, our novel bound in Theorem 2 also provides tighter lower bound on the safety probability guarantees via SBFs.
Exponential stochastic barrier functions (exponential SBFs) have been considered in [49] in order to provide tighter bounds on safety probability guarantees via SBFs for finite time horizon systems, in which the length of the time horizon is fixed and known a priori. Exponential SBFs also consider a multiplicative expected decrease condition, similar to our multiplicative RASMs and provide lower bounds on safety probability which are tighter by a factor which is exponential in the length  $N$  of the time horizon [49, Theorem 2]. However, as the length of the time horizon  $N \to \infty$ , their bound reduces to the bound of [45, 46]. In contrast, Theorem 2 shows that our multiplicative RASMs provide a tighter lower bound on safety (or more generally, reach-avoid) probability even in unbounded (i.e. indefinite) or infinite time horizon systems.
5 Compositional Learning for Probabilistic Specifications
We now present the CLAPS algorithm for learning a control policy that guarantees satisfaction of a probabilistic specification  $(\phi, p)$ , where  $\phi$  is a SPECTRAL formula. The idea behind our algorithm is as follows. The algorithm first translates  $\phi$  into an abstract graph  $G = (V, E, \beta, s, t)$  using the translation discussed in Section 3, in order to decompose the problem into a series of reach-avoid tasks associated to abstract graph edges. The algorithm then solves these reach-avoid tasks by learning policies for abstract graph edges. Each edge policy is learned together with a (multiplicative) RASM which provides formal guarantees on the probability of satisfaction of the reach-avoid specification proved in Theorem 2. Finally, the algorithm combines these edge policies in order to obtain a global policy that guarantees satisfaction of  $(\phi, p)$ . The algorithm pseudocode is presented in Algorithm 1. We assume the POLICY+RASM subprocedure which, given a reach-avoid task and given a probability parameter  $p'$ , learns a control policy together with a RASM that proves that the policy guarantees reach-avoidance with probability at least  $p'$ . Due to the equivalence of additive and multiplicative RASMs that we proved in Theorem 1, for this we can use the procedure of [68] as an off-the-shelf method. For completeness of our presentation, we also provide details behind the POLICY+RASM subprocedure in Appendix E.
Challenges There are two important challenges in designing an algorithm based on the above idea. First, it is not immediately clear how probabilistic reach-avoid guarantees provided by edge policies may be used to deduce the probability with which the global specification  $\phi$  is satisfied. Second, since SPECTRL formulas allow for disjunctive specifications, it might not be necessary to solve reach-avoid tasks associated to each abstract graph edge and naively doing so might lead to inefficiency. Our algorithm solves both of these challenges by first topologically ordering the vertices of the abstract graph and then performing a forward pass in which vertices are processed according to the topological ordering. The forward pass is executed in a way which allows our algorithm to keep track of the cumulative probability with which edge policies that have already been learned might violate the global specification  $\phi$ , therefore also allowing the algorithm to learn edge policies on-demand and to identify abstract graph edges for which solving reach-avoid tasks would be redundant.
Forward pass on the abstract graph Since an abstract graph is a directed acyclic graph (DAG), we perform a topological sort on  $G$  in order to produce an ordering  $s = v_{1}, v_{2}, \ldots, v_{|V|} = t$  of its vertices (line 4 in Algorithm 1). This ordering satisfies the property that each edge  $(v_{i}, v_{j})$  in  $G$  must satisfy  $i < j$ , i.e. any each edge must be a "forward edge" with respect to topological ordering. Algorithm 1 now initializes an empty dictionary  $\mathrm{Prob}$  and sets  $\mathrm{Prob}[s] = 1$  for the source vertex  $s$  (line 5) and performs a forward pass to process the remaining vertices in the abstract graph according to the topological ordering (lines 6-13). The purpose of the dictionary  $\mathrm{Prob}$  is to store lower bounds on the probability with which previously processed abstract graph vertices are reached by following the computed edge policies while satisfying reach-avoid specifications of the traversed abstract graph edges. For each newly processed vertex  $v_{i}$ , Algorithm 1 learns policies for the reach-avoid tasks associated to all abstract graph edges that are incoming to  $v_{i}$ . It then combines probabilities with
Algorithm 1 Compositional Learning for Probabilistic Specifications (CLAPS)
1: Input Dynamics function  $f$ , probability distribution  $d$ , initial set  $\mathcal{X}_0$ ,  
2: SPECTRL specification  $\phi$ , probability threshold  $p \in [0,1]$   
3:  $G = (V,E,\beta,s,t) \gets$  abstract graph for  $\phi$   
4:  $s = v_1, v_2, \ldots, v_{|V|} = t \gets$  topological ordering of vertices in  $G$   
5: Prob  $\leftarrow$  empty hash map, Prob[s]  $\leftarrow$  1  
6: for i = 2, 3, ...|V| do  
7: Prob[v_i]  $\leftarrow$  0  
8:  $N(v_i) \gets \{v \in V \mid (v, v_i) \in E \land \operatorname{Prob}[v] \geq p\}$   
9: for all  $v \in N(v_i)$  do  
10:  $\pi_{(v,v_i)}, p_{(v,v_i)} \gets$  use binary search to find the maximal probability  $p_{(v,v_i)}$  for which POLICY+RASM manages to learn a policy  $\pi_{(v,v_i)}$  that solves the reach-avoid task associated to edge  $(v, v_i)$  with probability at least  $p_{(v,v_i)}$   
11: Prob[v_i]  $\leftarrow$  max{Prob[v_i],  $p_{(v,v_i)} \cdot \operatorname{Prob}[v]$ }  
12: end for  
13: end for  
14: if Prob[t]  $\geq p$  then  
15:  $\pi \gets$  global policy obtained by composing edge policies  
16: return Policy  $\pi$  that guarantees satisfaction of  $(\phi,p)$ .  
17: else  
18: return Policy for the probabilistic specification  $(\phi,p)$  could not be learned.  
19: end if
which the learned edge policies satisfy associated reach-avoid tasks with the probabilities that have been stored for the previously processed vertices in order to compute the lower bound  $\operatorname{Prob}[v_i]$ .
More concretely, for each  $2 \leq i \leq |V|$ , Algorithm 1 first stores  $\mathrm{Prob}[v_i] = 0$  as a trivial lower bound (line 7). It then computes the set  $N(v_i)$  of all predecessors of  $v_i$  for which previously learned edge policies ensure reachability with probability at least  $p$  while satisfying reach-avoid specification of all traversed abstract graph edged (line 8). Recall,  $p$  is the global probability threshold with which the agent needs to satisfy the SPECTRL specification  $\phi$ , and since Algorithm 1 processes vertices according to the topological ordering all vertices in  $N(v_i)$  have already been processed. If  $N(v_i)$  is empty, then the algorithm concludes that  $v_i$  cannot be reached in the abstract graph while satisfying reach-avoid specifications of traversed edges with probability at least  $p$ . In this case, the algorithm does not try to learn edge policies for the edges with the target vertex  $v_i$  and proceeds to processing the next vertex in the topological ordering. Thus, Algorithm 1 ensures that edge policies are learned on-demand. If  $N(v_i)$  is not empty, then for each  $v \in N(v_i)$  the algorithm uses POLICY+RASM together with binary search to find the maximal probability  $p_{(v,v_i)}$  for which POLICY+RASM manages to learn a policy  $\pi_{(v,v_i)}$  that solves the reach-avoid task with probability at least  $p_{(v,v_i)}$  (line 10). We run the binary search only up to a pre-specified precision, which is a hyperparameter of our algorithm. Since the reach-avoid specification of the edge  $(v,v_i)$  is satisfied with probability at least  $p_{(v,v_i)}$  and since  $v$  is reached in the abstract graph while satisfying reach-avoid specification of all traversed edges with probability at least  $\mathrm{Prob}[v]$ , Algorithm 1 updates its current lower bound  $\mathrm{Prob}[v_i]$  for  $v_i$  to  $p_{(v,v_i)} \cdot \mathrm{Prob}[v]$  whenever  $p_{(v,v_i)} \cdot \mathrm{Prob}[v]$  exceeds the stored bound  $\mathrm{Prob}[v_i]$  (line 11). This procedure is repeated for each  $v \in N(v_i)$ .
Composing edge policies into a global policy Once all abstract graph vertices have been processed, Algorithm 1 checks if  $\mathrm{Prob}[t] \geq p$  (line 14). If so, it composes the learned edge policies into a global policy  $\pi$  that guarantees satisfaction of the probabilistic specification  $(\phi, p)$  (line 15) and returns the policy  $\pi$  (line 16). The composition is performed as follows. First, note that for each vertex  $v \neq s$  in the abstract graph for which  $\mathrm{Prob}[v] > 0$ , by lines 7-11 in Algorithm 1 there must exist a vertex  $v' \in N(v)$  such that the algorithm has successfully learned a policy  $\pi_{(v', v)}$  for the edge  $(v', v)$  and such that  $\mathrm{Prob}[v] = p_{(v', v)} \cdot \mathrm{Prob}[v']$ . Hence, repeatedly picking such predecessors yields a finite path  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_k} = t$  in the abstract graph such that, for each  $1 \leq j \leq k$ , the algorithm has successfully learned an edge policy  $\pi_{(v_{j-1}, v_{j})}$  and such that  $\mathrm{Prob}[v_{i_j}] = p_{(v_{i_j-1}, v_{i_j})} \cdot \mathrm{Prob}[v_{i_j-1}]$  holds. The global policy  $\pi$  starts by following the edge policy  $\pi_{(v_{i_0}, v_{i_1})}$  until the target region  $\beta(v_{i_1})$  is reached or until the safety constraint  $\beta(v_{i_0}, v_{i_1})$  associated to the edge is violated. Then,
(a) The objective is to navigate from the initial set in the bottom left (in green) to the target set in the top right room (in violet).
(b) Visualization of trajectories of an end-to-end learned policy. The policy hits the walls and cannot be verified as safe.
(c) Visualization of trajectories of the three verifiable compositional policies with intermediate initial/target states shown in grey.
Figure 1: Stochastic Nine Rooms environment for which an end-to-end learned policy cannot be verified safe while our algorithm is able to decompose the task into three verifiable subtasks.
Figure 2: Decomposition of the Stochastic Nine Rooms task into subtasks. Left: Individual subtasks shown as graph edges. Center: Successfully verifiable subtasks are shown in blue edges, whereas the verification procedure of the gray edges failed. Right: Verified path for solving the complete task.
it follows the edge policy  $\pi_{(v_{i_1}, v_{i_2})}$  until the target region  $\beta(v_{i_2})$  is reached or until the safety constraint  $\beta(v_{i_1}, v_{i_2})$  associated to the edge is violated. This is repeated for each edge along the path  $s = v_{i_0}, v_{i_1}, \ldots, v_{i_k} = t$  until the system reaches a state within the target region  $\beta(t)$ . If at any point the safety constraint is violated prior to reaching the target region associated to the edge, policy  $\pi$  follows the current edge policy indefinitely and no longer updates it as the specification  $\phi$  has been violated. As we prove in Theorem 3, violation of the safety constraint associated to some edge or the system never reaching the target region associated to some edge happens with probability at most  $1 - p$ . If  $\mathrm{Prob}[t] < p$ , then Algorithm 1 returns that it could not learn a policy that guarantees satisfaction of  $(\phi, p)$ . Theorem 3 establishes the correctness of Algorithm 1 and its proof shows that the global policy  $\pi$  obtained as above indeed satisfies the probabilistic specification  $(\phi, p)$ .
Theorem 3. [Proof in Appendix F] Algorithm 1 is compositional, and if it outputs a policy  $\pi$ , then  $\pi$  guarantees the probabilistic specification  $(\phi, p)$ .
We implement a prototype of Algorithm 1 to validate its effectiveness  $^3$ . Motivated by the experiments of [33], we define an environment in which an agent must navigate safely between different rooms. However, different from [33] our Stochastic Nine Rooms environment perturbs the agent by random noise in each step. Concretely, the system is governed by the dynamics function,  $x_{t + 1} = x_t + 0.1\min (\max ((a_t, - 1),1) + \omega_t)$ , where  $\omega_{t}$  is drawn from a triangular noise distribution. The state space of the environment is defined as  $[0,3]\times [0,3]\subset \mathbb{R}^2$ . The set of the initial states is  $[0.4,0.6]\times [0.4,0.6]$ , whereas the target states are defined as  $[2.4,2.6]\times [2.4,2.6]$ . The state space contains unsafe areas that should be avoided, i.e., the red walls shown in Figure 1. Specifically, these unsafe areas cannot be entered by the agent, and coming close to them during the RL training incurs a penalty. Note that this defines a reach-avoid task where the goal is to reach the target region from the initial region while avoiding the unsafe regions defined by red walls.
We first run the POLICY+RASM procedure (i.e. the method of [68]) on this task directly and observe that it is not able to learn a policy with any formal guarantees on probability of reach-avoidance.
3Our code is available at https://github.com/mlech261/neural_martingales
We then translate this task into a SPECTRAL task which decomposes it into a set of simpler reach-avoid tasks by constructing an abstract graph as follows. The vertex set of the abstract graph consists of centers of each of the 9 rooms, and the target regions associated to each room center are  $[0.4 + x, 0.6 + x] \times [0.4 + y, 0.6 + y]$  for  $x, y \in \{0,1,2\}$ . Edges of this abstract graph are shown in Figure 2 left. To each edge, we associate an unsafe region to be avoided by taking red walls incident to rooms in which the edge is contained.
For each reach-avoid edge task, we run the proximal policy optimization (PPO) [50] reinforcement learning algorithm to initialize policy parameters. We then run the POLICY+RASM procedure to learn a policy and a RASM which proves probabilistic reach-avoidance. We set the timeout to 4 hours for each reach-avoid subtask. In Figure 1 b), we visualize the PPO policy that was trained to reach the target states directly from the initial states. As shown, the policy occasionally hits a wall and cannot be verified, i.e., POLICY+RASM procedure times out. In Figure 2 center, we visualize all successfully verified subtasks in blue. The reach-avoid lower bounds for each of the subtasks are listed in Table 2. Note that the simplest decomposition passing over the bottom right room could not be verified, which shows that the systematic decomposition used in our algorithm has advantages over manual task decompositions. Finally, in Figure 2 right, we highlight the path from the initial states to the target states via safe intermediate sets. In Figure 1 c), we visualize the trajectories of these three compositional policies. As shown, the trajectories never reach an unsafe area.
Table 2: Verified lower bounds on reach-avoid probability for each edge policy. Initial and target states are represented by pairs  $(x,y)$  with  $[0.4 + x,0.6 + x]\times [0.4 + y,0.6 + y]$  for  $x,y\in$ $\{0,1,2\}$
Start (x1,y1)	Goal (x2,y2)	Reach-avoidance probability
(0,0)	(1,0)	75.1%
(0,0)	(2,0)	57.7%
(0,0)	(0,1)	72.3%
(1,0)	(2,0)	71.4%
(0,1)	(2,1)	61.5%
(0,1)	(1,1)	72.2%
(1,1)	(1,2)	73.3%
(1,2)	(2,1)	68.5%
(2,1)	(2,2)	74.5%
(0,0)	(2,2)	Fail (method of [68])
(0,0)	(2,2)	33.0% (our method)
We propose CLAPS, a novel method for learning and verifying a composition of neural network policies for stochastic control systems. Our method considers control tasks under specifications in the SPECTRL lanugange, decomposes the task into an abstract graph of reach-avoid tasks and uses reach-avoid supermartingales to provide formal guarantees on the probability of reach-avoidance in each subtask. We also prove that RASMs imply a strictly tighter lower bound on the probability of reach-avoidance compared to prior work. These results provide confidence in the ability of CLAPS to find meaningful guarantees for global compositional policies, as shown by experimental evaluation in the Stochastic Nine Rooms environment. While our approach has significant conceptual, theoretical and algorithmic novelty, it is only applicable to SPECTRL logical specifications and not to the whole of LTL. Furthermore, our algorithm is not guaranteed to return a policy and does not guarantee tight bounds on the probability of satisfying the logical specification. Overcoming these limitations will provide further confidence in deploying RL based solutions for varied applications.
This work was supported in part by the ERC-2020-AdG 101020093 (VAMOS) and the ERC-2020-CoG 863818 (FoRM-SMArt).
[1] Alessandro Abate, Daniele Ahmed, Alec Edwards, Mirco Giacobbe, and Andrea Peruffo. FOSSIL: a software tool for the formal synthesis of lyapunov functions and barrier certificates using neural networks. In Sergiy Bogomolov and Raphaël M. Jungers, editors, HSCC '21: 24th ACM International Conference on Hybrid Systems: Computation and Control, Nashville,
[67] Mojtaba Zarei, Yu Wang, and Miroslav Pajic. Statistical verification of learning-based cyberphysical systems. In Aaron D. Ames, Sanjit A. Seshia, and Jyotirmoy Deshmukh, editors, HSCC '20: 23rd ACM International Conference on Hybrid Systems: Computation and Control, Sydney, New South Wales, Australia, April 21-24, 2020, pages 12:1-12:7. ACM, 2020. doi: 10.1145/3365365.3382209. URL https://doi.org/10.1145/3365365.3382209.
[68] Dorde Zikelic, Mathias Lechner, Thomas A. Henzinger, and Krishnendu Chatterjee. Learning control policies for stochastic systems with reach-avoid guarantees. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 11926-11935. AAAI Press, 2023. doi: 10.1609/aaai.v37i10.26407. URL https://doi.org/10.1609/aaai.v37i10.26407.
A SPECTRL syntax and semantics and the proof of Theorem 4