# Mining Road Traffic Rules with Signal Temporal Logic and Grammar-Based Genetic Programming
## 6 Experimental Evaluation
### Results  
#### 6.3.1 RQ1: Solutions that are Effective  
We ran an experimental campaign with the aforementioned parameters. To evaluate whether our approach is effective, we (1) verify whether it evolved tight formulas, and (2) compare the fitness \(f\) of the best individuals with that of randomly initialized formulas. For the former, Figure 5 plots the histogram for the distribution of \(\rho(\varphi,\mathbf{x},t)\) for the best individuals \(\varphi\) found in each run (i.e., the individual having the lowest fitness \(f\) at the last generation) and for all the car trajectories \(\mathbf{x}\). For the latter, Figure 6 reports median \(\pm\) standard deviation fitness \(f\) over the course of evolution for the best individuals found in each run. Table 1 summarizes (with median \(\pm\) standard deviation) the results in terms of fitness \(f\) and \(|\varphi|\) for the best individual found in each run, as well as evolution time (in seconds).  
From Figure 5, two observations can be made. First, robustness values cluster around \(0.0\). Second, robustness values cluster symmetrically. Table 1 corroborates these observations. Recall from Section 4 that our goal is to mine formulas that are tight since tightness is of fundamental importance when no labels are provided. Formulas are tight when robustness values are as close to zero as possible, in particular \(|\rho(\varphi,x_{i},t_{f})|<\epsilon,\forall i\), with \(\epsilon\) a small quantity, which depends on the system at hand. In other words, formulas must fit the pool of trajectories at hand and in such a way that small perturbations make the robustness value greater than \(\epsilon\) on some trajectory. By looking at Figure 5, we remark that  
Figure 5: Distribution of the robustness \(\rho(\varphi,\mathbf{x},t)\), computed for all the I-80 trajectories \(\mathbf{x}\), for the best individual \(\varphi\) found in each run.  
the mined formulas are indeed tight. Let a value \(e\) be fixed. The best individuals produce robustness values that fit into a segment centered in 0.0 and of length \(2e\); that is, they lie in \([-\epsilon,+\epsilon]\). Choose an adequate value for \(\epsilon\), e.g., \(\epsilon=0.25\). From Figure 5, we notice that the vast majority of values lie in \([-0.25,0.25]\). Recalling our definition of tightness of a formula from Section 4, these results confirm that the best individuals indeed are tight formulas.  
As a further confirmation that our methodology actually learns tight formulas, we also show that learned formulas are significantly better than random formulas. As can be seen from Figure 6, fitness \(f\) progresses over the course of evolution and settles into a (local) optimum. Considering that the initial population is composed entirely of randomly initialized formulas (see Section 5.1), this observation points to the fact that, in terms of fitness \(f\) (and, thus, of tightness), the best individuals are clearly better than random formulas.  
To comment, our methodology succeeds in mining specifications that are effective and tight with respect to the pool of trajectories.