# Digital Twins, Virtual Devices, and Augmentations for Self-Organising Cyber-Physical Collectives +
## 5 Evaluation
### Experiment Results and Discussion  
The results in Figure 6 (mean distance error \(\overline{\delta}\)) and Figure 7 (isolated devices \(I\)) show that the system, augmented with virtual devices, can (i) provide more precise route estimates than the original version (identified with \(R_{v}R=\infty\)) and (ii) improve reachability. In more detail, the system achieves good performance across the board when \(R_{v}R\leq 0.5\), indicating that the generation of virtual devices can indeed provide an increased system performance. Figure 6: System performance measured as distance error for increasing communication range \(R\) (**top** to **bottom**) and different count of explorers \(N_{r}\) (5 in the **left column**, 50 in the **right column**). Implementations that generate virtual devices outperform the base implementation consistently, as far as virtual devices are generated densely enough. In most scenarios, setting \(R_{v}=R2\) achieves a consistently good result. When \(R_{v}\) is larger than \(R\), the system performance degrades very quickly, mostly due to insufficient coverage, which, with few explorers, also causes erratic behaviour. Figure 7: System performance measured as count of isolated devices \(I\) for communication range \(R=10\) m (**left column**) and communication range \(R=100\) m (**right column**), and increasing count of explorers \(N_{r}\) (**top** to **bottom**). As expected, systems that generate virtual devices more densely achieve better coverage. With many explorers, the benefits of a denser generation lower. Even with few explorers, there is little difference between \(R_{v}R=0.25\) and \(R_{v}R=0.5\), confirming the results seen in Figure 6.  
From Figure 6, we see that the system has an initial stabilisation phase: initially, explorers can hardly communicate with each other, and with time, through the generation of virtual devices, the virtual communication network gets closer and closer to the shape of the actual communication routes. We can see that the larger is the count of explorers, the shorter is the time required for the system to stabilise (as more roads get explored more quickly). We note however that in some cases the system error, after a period of reduction, grows out of control. This phenomenon occurs in particular when \(R_{v}R\geq 1\), and when \(R_{v}R=0.75\) and \(R\geq 100\) m, and it is typically coupled with an erratic behaviour of the error. We believe that this effect is due to the logical network no longer mapping the underlying physical infrastructure correctly: virtual devices become too sparse and their connection range too large, the virtual network infrastructure that they build grows further and further different from the curvy structure of the Medieval roads that make the city centre. In some cases, a device placed in an unexplored area may lead to an improvement in the distance error, yet normally there is a high chance that its reification may cause the creation of a non-existing passage between two locations. By observing Figure 7, we can further appreciate how a larger count of explorers helps to reach stability faster. Interestingly, even with reasonably short values for \(R\) (e.g., \(R=100\) m), the system shows a pretty high reachability (the lower the count of isolated devices, the higher the reachability). As expected, for any given value of \(R\), the more virtual devices get generated, the higher is the reachability. Of course, as shown in Figure 8 (cost in computational resources, using the count of virtual devices \(R_{v}\) as a proxy metric) and Figure 9 (communication cost, measured as the sum of messages received per round on all real devices \(\sum_{j\in H}M_{j}\)), it is not a free lunch. In particular, the system is very sensible (both performance-wise and cost-wise) to the way virtual devices are generated, in this case tuned via the parameter \(R_{v}\) (regulated along with the communication range \(R\) by tuning \(R_{v}R\)).