DOI: 10.5281/zenodo.2634272
title: SIMBED - Offline Real-World Wireless Networking Experimentation using ns-3

description: R&D in wireless networking typically depends on experimentation to make realistic evaluations, since simulation is inherently a simplification of the real-world. However, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility.

Real wireless experiments are hardly repeatable. Given the same input they can produce very different output results, since wireless communications are influenced by external random phenomena such as noise, interference, and multipath. Real experiments are also difficult to reproduce: either the original community testbed is unavailable – offline or running other experiments – or the custom testbed used is inaccessible.

Fed4FIRE+ wireless testbeds such as w-iLab.t and NITOS, although deployed in controlled environments, do not fully address the problem. The CONCRETE tool used in such testbeds assures the repeatability and reproducibility of experiments, but ignores executions whose results are also representative of the system operation and often reveal unpredicted behaviour that must be understood.

What if we could make any wireless experiment repeatable and reproducible under the same exact conditions? What if we could share the same Fed4FIRE+ testbed execution conditions among an "infinite" number of users? What if we could run wireless experiments faster than in real time?

INESC TEC has been developing the Offline Experimentation (OE) approach that combines the best of simulation and experimentation to achieve the above-mentioned goals. By relying on Network Simulator 3 (ns-3) and its good simulation capabilities from the MAC to the application layer, we have been exploring how ns-3 can be used to replicate real-world wireless experiments using real traces containing 1) position of nodes and 2) the quality of each radio link.

The <strong>SIMBED </strong>project aimed at running a set of wireless experiments on top of the controlled environments of w-ilab.t and NITOS Fed4FIRE+ testbeds to further validate the OE approach. For that purpose, we configured different fixed and mobile experimental scenarios, representative of Wi-Fi range of operation, and measured the attained network performance using metrics such as throughput and Round-Trip Time (RTT). Then, we repeated each experiment using, both, Pure Simulation (PS) and OE approaches based on ns-3, also measuring the network performance for the same set of executions of experiments for all the different scenarios.

By comparing the performance metrics of each real experiment with its PS and OE counterparts, we were able to measure the relative error of each simulation approach relatively to the real experiments, as well as the accuracy gains introduced by the OE approach when compared to the PS traditional alternative. The main results show that it is possible to repeat and reproduce real experiments in ns-3, using the OE approach, achieving closer to real performance than using the PS approach. For all the experiments performed in SIMBED, using the OE approach resulted in an average accuracy gain of 59% when comparing to the PS approach.  

These results were important for validating a PhD thesis contribution related to the OE approach, as well as for producing two conference papers and one journal paper. The SIMBED results increased our confidence on the accuracy of the OE approach and are envisioned to foster the adoption of the OE approach by the networking community, in complement to the use of real experimentation.

 

The following dataset presents the results of the SIMBED project, organized in different folders, for each subset of experiments carried on:


	<strong><em>SubExp#1: </em></strong><em>Static point-to-point Wi-Fi communications using auto-rate (Minstrel) </em>

	
		<strong><em>SubExp#1.1:</em></strong><em> Using w-iLab.2 (medium to high SNR scenarios)</em>
		<strong><em>SubExp#1.2:</em></strong><em> Using w-iLab.2 (low SNR scenarios)</em>
		<strong><em>SubExp#1.3:</em></strong><em>  Using NITOS</em>
		<strong><em>SubExp#1.4:</em></strong><em> Using w-iLab.1 (datacenter room)</em>
	
	
	<strong><em>SubExp#2: </em></strong><em>Static point-to-point Wi-Fi communications using fixed</em> rate
	<strong><em>SubExp#3: </em></strong><em>Mobile point-to-point Wi-Fi communications using auto-rate (Minstrel)</em>
	<strong><em>SubExp#4: </em></strong><em>Static multiple access Wi-Fi communications using auto-rate (Minstrel) </em>
	
		<strong><em>SubExp#4.1:</em></strong><em> Using w-iLab.2 (bidirectional) (medium to high SNR scenarios)</em>
		<strong><em>SubExp#4.2:</em></strong><em> Using w-iLab.2 (bidirectional) (low SNR scenarios)</em>
		<strong><em>SubExp#4.3:</em></strong><em> Using NITOS (bidirectional)</em>
		<strong><em>SubExp#4.4:</em></strong><em> Using w-iLab.1 (bidirectional)</em>
		<strong><em>SubExp#4.5:</em></strong><em> Using NITOS (2 STAs)</em>
		<strong><em>SubExp#4.6:</em></strong><em> Using w-iLab.2 (2 STAs)</em>
	
	
	<strong><em>SubExpExample</em></strong>: contains raw experimental logs, parsed data and simulation results, to show how data extracted from the nodes is processed to be compatible with the OE approach and comparable with OE and PS simulation results.


Each experiment has an individual folder, named according to the date and time of the experiment and the nodes used. Inside, there’s a folder for the <strong>parsed</strong> experimental results, which contains

This folder contains the details and parsed logs of the experiment, as follows:


	<em>date_time</em><strong>.cfg </strong>– configuration details of the experiment
	<em>date_time_NodeID<sup><strong>[1]</strong></sup>_SenderID<sup><strong>[2]</strong></sup>_ReceiverID<sup><strong>[3]</strong></sup>_FlowType<sup><strong>[4]</strong></sup>_Params<sup><strong>[5]</strong></sup></em><strong>.snr </strong>– logs of the Signal/Noise ratio (1 file per node/flow)  
	<em>date_time_NodeID_SenderID_ReceiverID_FlowType_Params</em><strong>.stats</strong> – logs of the packets received (1 file per node/flow)  
	<em>NodeID</em><strong>.</strong><strong>waypoints</strong> – coordinates of the static nodes
	<em>date_time_MobileNodeID</em><strong>.</strong><strong>waypoints</strong> – waypoints of the mobile nodes (when applicable)


The experiment’s folder also contains a folder for the simulations <strong>output</strong> with the simulations statistics files, for the multiple simulations approaches considered, as follows:


	<em>date_time_NodeID_SenderID_ReceiverID_FlowType_Params</em>.<strong>simstats </strong>– logs of the packets received (simulation)


 

<sub>[1] ID of the node Logging node</sub>

<sub>[2] ID of the Sender node</sub>

<sub>[3] ID of the Receiver node</sub>

<sub>[4] Flow type: Unidirectional, Bidirectional or Unidirectional with Multiple Access</sub>

<sub>[5] Configurable parameters: Sender/Receiver Transmission Power and Data Rate (when applicable)</sub>