DOI: 10.5281/zenodo.3713270
title: SIMBED+ - Replicable Real Wireless Networking Experiments using ns-3

description: Wireless networking R&D depends on experimentation to make realistic evaluations of networking solutions, as simulation is inherently a simplification of the real-world. However, despite more realistic, experimentation is limited in aspects where simulation excels, such as repeatability and reproducibility. Real wireless experiments may be difficult to repeat. For the same input they can produce very different output results, since wireless communications are influenced by external phenomena such as noise, interference, and multipath. Even if repeatable, experiments may still be difficult to reproduce. Namely, other researchers may be unable to reproduce an experiment and confirm previous experimental results right away, because either the testbed is unavailable – offline or running other experiments when using a community testbed –, or inaccessible at all when a custom testbed was originally used. Fed4FIRE+ testbeds such as w-iLab.t, although operating in controlled environment, do not fully address the problem. This is even more evident in testbeds running in non-controlled and very dynamic environments, such as CityLab, where they may suffer from radio interference and competition from existing networks sharing the same radio spectrum. What if we could make any wireless experiment repeatable and reproducible? What if we could share the same Fed4FIRE+ testbed execution conditions among an "infinite" number of users? What if we could run wireless experiments faster than in real time? INESC TEC has developed the Offline Experimentation (OE) approach that combines the best of simulation and experimentation to achieve the above-mentioned goals. By relying on Network Simulator 3 (ns-3) and its good simulation capabilities from the MAC to the Application layer, we have been exploring how ns-3 can be used to replicate real-world wireless experiments using real traces containing 1) position of nodes and 2) the quality of each radio link. The previous <strong>SIMBED </strong>project validated the OE approach for controlled environments and helped identifying some of its limitations. The OE approach was then improved to support experiments using Multiple-In-Multiple-Out (MIMO) and shared radio spectrum with concurrent networks. To further validate the improved OE approach, the <strong>SIMBED+</strong> project aimed at running a set of experiments on top of the controlled and non-controlled environments of w-iLab.t and CityLab Fed4FIRE+ testbeds. For that purpose, we configured different fixed experimental scenarios, representative of Wi-Fi range of operation, subjected to controlled and non-controlled interference from concurrent experiments. For each experiment the achieved network throughput was measured. Then, we repeated each experiment using, both, Pure Simulation (PS) and OE approaches (now with MIMO and channel occupancy information) based on ns-3, also measuring the network throughput for the same set of experiments. To compare the network throughput of the real experiments with their PS and OE counterparts, Cumulative Distribution Function (CDF) curves were used, plotting all 1-second average throughput samples. For all the experiments performed in SIMBED+, using the improved OE approach resulted in network throughput considerably closer to real than using the PS approach. This is even more evident in non-controlled environments using MIMO and shared radio spectrum. These results were important for further validating the OE approach, producing two conference papers and one journal paper. The SIMBED+ results increased our confidence on the OE approach ability of reproducing past experiments, and are envisioned to foster the adoption of the OE approach by the networking community, in complement to the use of real experimentation. The following dataset presents the results of the SIMBED+ project, organized in different folders, for each subset of experiments carried on: <strong><em>SubExp#1: </em></strong><em>Trace-based PHY rate with SISO support (802.11a, BW 20 MHz) </em> <strong><em>SubExp#2: </em></strong><em>Trace-based PHY rate with MIMO support (802.11n/ac, BW 20/40 MHz, MIMO 3x3) </em> <strong><em>SubExp#2.1: </em></strong><em>IEEE 802.</em><em>11n, 20 MHz, MIMO 3x3</em> <strong><em>SubExp#2.2: </em></strong><em>IEEE 802.</em><em>11n, 40 MHz, MIMO 3x3</em> <strong><em>SubExp#2.3: </em></strong><em>IEEE 802.</em><em>11ac, 40 MHz, MIMO 3x3</em> <strong><em>SubExp#3: </em></strong><em>Shared radio spectrum support (occupancy at the sender)</em> <strong><em>SubExp#4: </em></strong><em>Shared radio spectrum support (occupancy at the receiver)</em> <strong><em>SubExp#5:</em></strong> <em>Trace-based PHY rate, MIMO and shared radio spectrum support</em> Each experiment has an individual folder, named according to the date and time of the experiment and the nodes used. Inside, there’s a folder for the <strong>parsed</strong> experimental results, which contains This folder contains the details and parsed logs of the experiment, as follows: <em>date_time</em><strong>.cfg </strong>– configuration details of the experiment <em>date_time_NodeID<sup><strong>[1]</strong></sup>_SenderID<sup><strong>[2]</strong></sup>_ReceiverID<sup><strong>[3]</strong></sup>_FlowType<sup><strong>[4]</strong></sup>_Params<sup><strong>[5]</strong></sup></em><strong>.snr </strong>– logs of the Signal/Noise ratio (1 file per node/flow) <em>date_time_NodeID_SenderID_ReceiverID_FlowType_Params</em><strong>.stats</strong> – logs of the packets received (1 file per node/flow) <em>NodeID</em><strong>.</strong><strong>waypoints</strong> – coordinates of the static nodes <em>date_time_MobileNodeID</em><strong>.</strong><strong>waypoints</strong> – waypoints of the mobile nodes (when applicable) The experiment’s folder also contains a folder for the simulations <strong>output</strong> with the simulations statistics files, for the multiple simulations approaches considered, as follows: <em>date_time_NodeID_SenderID_ReceiverID_FlowType_Params</em>.<strong>simstats </strong>– logs of the packets received (simulation) <sub>[1] ID of the node Logging node</sub> <sub>[2] ID of the Sender node</sub> <sub>[3] ID of the Receiver node</sub> <sub>[4] Flow type: Unidirectional, Bidirectional or Unidirectional with Multiple Access</sub> <sub>[5] Configurable parameters: Sender/Receiver Transmission Power and Data Rate (when applicable)</sub>