###### Abstract  
The performance assessment of routing protocols in vehicular ad hoc networks (VANETs) plays a critical role in testing the efficiency of the routing algorithms before deployment in real conditions. This research introduces the statistical design of experiments (DOE) methodology as an innovative alternative to the one factor at a time (OFAT) approach for the assessment and the modeling of VANET routing protocol performance. In this paper, three design of experiments methods are applied, namely the two-level full factorial method, the Plackett-Burman method and the Taguchi method, and their outcomes are comprehensively compared. The present work considers a case study involving four factors namely: node density, number of connections, black hole and worm hole attacks. Their effects on four measured outputs called responses are simultaneously evaluated: throughput, packet loss ratio, average end-to-end delay and routing overhead of the AODV routing protocol. Further, regression models using the least squares method are generated. First, we compare the main effects of factors resulted from the three DOE methods. Second, we perform analysis of variance (ANOVA) to explore the statistical significance and compare the percentage contributions of each factor. Third, the goodness of fit of regression models is assessed using the adjusted R-squared measure and the fitting plots of measured versus predicted responses. VANET simulations are implemented using the network simulator (NS-3) and the simulator of urban mobility (SUMO). The findings reveal that the design of experiments methodology offers powerful mathematical, graphical and statistical techniques for analyzing and modeling the performance of VANET routing protocols with high accuracy and low costs. The three methods give equivalent results in terms of the main effect and ANOVA analysis. Nonetheless, the Taguchi models show higher predictive accuracy. VANET; NS-3; SUMO; AODV; performance assessment; design of experiments methodology; regression analysis; ANOVA +
Footnote †: journal:  
an essential prerequisite used to identify the way different factors influence the network performance, alongside giving more insights for optimization and improvement prospects. In that regard, various works in the literature have conducted performance analysis studies wherein the conventionally used approach is to study the effect of one factor at a time (OFAT). This approach, which varies only one factor at a time while keeping others constant, is demanding and time consuming [3]. Thus, in order to overcome this problem, we opted for statistically designed experiments that vary several factors simultaneously, known as the design of experiments methodology. The present work focuses on the comparison of the results of three DOE methods namely: the two-level full factorial method (FFM) [4; 5], the Plackett-Burman method (PBM) [6] and the Taguchi method (TM) [7]. The details of each method were presented at international conferences [8; 9; 10]. The effects of four factors were assessed--node density (ND), number of connections (NC), black hole (BKH) and worm hole (WMH) attacks--on the following performance metrics: throughput (TH), packet loss ratio (PLR), average end-to-end delay (AE2 ED) and routing overhead (RO) of the AODV routing protocol. For VANET simulations, we used NS-3 [11] and SUMO [12; 13]. First, the comparison is performed in terms of the main effects of factors on the considered responses that resulted from each DOE method. Secondly, analysis of variance is conducted to determine the factors that are statistically significant and compare their percentage contributions. The adjusted R-squared measure along with the plots of measured versus predicted values of the response are employed to assess the goodness of fit of the regression models. Additionally, only significant estimates are included in the final regression models.