Are MAs profitable to search‐based PLA design?

IET SoftwareVolume 13, Issue 6 p. 587-599 Research ArticleOpen Access Are MAs profitable to search-based PLA design? João Choma Neto, Corresponding Author João Choma Neto joao.choma@gmail.com Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorCristiano da Silva Herculano, Cristiano da Silva Herculano Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorThelma Elita Colanzi, Thelma Elita Colanzi Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorAline Maria Malachini Miotto Amaral, Aline Maria Malachini Miotto Amaral Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this author João Choma Neto, Corresponding Author João Choma Neto joao.choma@gmail.com Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorCristiano da Silva Herculano, Cristiano da Silva Herculano Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorThelma Elita Colanzi, Thelma Elita Colanzi Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this authorAline Maria Malachini Miotto Amaral, Aline Maria Malachini Miotto Amaral Department of Informatics, State University of Maringa, Av. Colombo 5790, Maringa, BrazilSearch for more papers by this author First published: 01 December 2019 https://doi.org/10.1049/iet-sen.2018.5318Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The architectural properties of product-line architecture (PLA) design have been successfully optimised by multi-objective genetic algorithms (GAs). Memetic algorithms (MAs) extend GA by adding a local search after the global search process. MA outperformed GA in the context of class modelling, software testing and the next release problem. However, no studies on the application of MAs for PLA design optimisation were found in the literature. In light of this, the authors performed an exploratory study, where MA was used to apply design patterns for a search-based PLA design (SBPD) and achieved promising results. From the obtained results, they adjusted the MA-based implementation. This study aims at investigating if the MA is more profitable to SBPD than GAs. Two empirical studies that involve four PLA designs were carried out for this task by varying the pair of objective functions. Empirical results show that MA achieved satisfactory solutions, despite being influenced by the original PLA design. 1 Introduction Software product-line (SPL) engineering has been widely adopted by the software industry for ease of reuse. The product-line architecture (PLA) design comprises all the available features of a product and presents the architectural variations in an SPL [1]. It is often attractive and advantageous to interpose a PLA between the feature model and the products constructed from it [2]. The PLA reflects implementation concerns, about recovering traceability links between the feature model and products and is thus one of the most important SPL artefacts [3]. An analysis of its design is important for the SPL life cycle because the PLA has to be generic and flexible enough to support the configuration of many products [1]. When focusing on architectural properties, architects must develop PLA designs taking into account different and conflicting architectural measures, for instance, with high feature modularisation (FM) and low feature scattering. This makes PLA design a people-intensive task for which there is not a single solution owing to the number of factors that influence it. This scenario allows for the modelling of PLA design as a multi-objective optimisation problem [4]. Search-based software engineering (SBSE) techniques have achieved satisfactory results to discover near-optimal solutions for problems such as PLA design [2]. To deal with this problem, an approach called multi-objective approach for PLA design (MOA4PLA) [4] was proposed to optimise PLA design in terms of FM, SPL extensibility and conventional architectural properties that use multi-objective evolutionary algorithms (MOEAs) based on genetic algorithms (GAs). Software metrics to evaluate architectural properties are used in objective functions that guide the search process. Each solution obtained in the search is evaluated and ranked in terms of its objective functions. This means that these functions consist of the objectives that need to be optimised. At the end of the search, several alternative designs may be automatically achieved in accordance with the selected objectives. MOA4PLA is a seminal approach to optimise PLA design using SBSE techniques. The results of MOA4PLA using GA are promising. However, studies demonstrate that memetic algorithms (MAs) present better results in certain domains such as the contexts of class modelling [5], software testing [6-9], next release problem [10] and component deployment problem [11]. MA is an extension of GA, where a local search is carried out after the global search process [7]. The strengths of GAs and the local search have been combined to create MAs, which have shown to be more efficient than GAs and local search separately [11]. Also, to the best of our knowledge, there is no work in the literature about the use of MA for PLA design optimisation. Given this motivation, and taking into account the existence of a search operator that applies design patterns, which is a domain-specific change, and that according to the definition of MA could be promising as a local search operator, we then decided to investigate its adequacy to the PLA design domain. The comparison of MA was made with the original implementation of MOA4PLA – that uses GA – which was considered as the baseline in this work. Given this, we decided to conduct an exploratory study to apply MA to PLA design optimisation [12]. In this paper, the local search applied some Gang of Four (GoF) design patterns [13] to the PLA design to improve some of its properties (further details about the chosen patterns are given in Section 3. The results showed the following: (i) which local search selection approach obtained better results, (ii) the points that needed to be improved in the implementation, (iii) MA found greater diversity of solutions which showed that they are able to explore the solutions space different way from GA and (iv) MA slightly outperformed GA in terms of fitness. Since then, the implementation has been improved, and our MA-based implementation was refined to include a constraint to prevent the global search from harming the changes made by the local search. The experiments should be repeated, based on the new version of our MA so that the results obtained by GA and MA can be compared because the constraint added to the global search can have an impact on the results obtained previously. Experiments replications play a key role in empirical software engineering by allowing the community to build and consolidate knowledge about which results or observations should be held under which conditions [14-17]. Murray Lindsay and Ehrenberg [18] argue that replication 'is needed not merely to validate one's findings, but more importantly, to establish the increasing range of radically different conditions, under which the findings hold and the predictable exceptions'. In light of the context outlined above, the purpose of this paper is two-fold: (i) to replicate a part of the exploratory study and (ii) to increase the body of knowledge about the application of MA in search-based PLA design (SBPD) by performing another experiment that includes objective functions directly related to the changes caused by the design patterns in the optimised PLAs. The first research question investigated in the present work is RQ1 – 'Do MA find better solutions than those obtained by GA in the context of PLA design optimisation using MOA4PLA?'. This study was divided into two experiments. Each experiment included a separate pair of objective functions. The first experiment made use of the conventional metrics (CMs) function to measure conventional architectural properties such as coupling, cohesion and feature-driven metrics (FMs) function to measure FM [4], the same objective functions used in related work [19]. The second experiment used the cohesion of PLA design (COE) and class coupling (ACLASS) functions, which measure relational cohesion and class coupling, respectively. COE and ACLASS were chosen because they are related to the main architectural properties (cohesion and coupling) that are usually improved by the design patterns applied in the local search. The objective functions are described in Section 3. The second experiment aims at providing evidence to answer the second research question: RQ2 – 'Which pair of objective functions is more suitable for evaluating the changes performed by the local search on the PLA design solutions?'. It is expected that COE and ACLASS will obtain better results than CM and FM since the applied design patterns benefit from the metrics of these functions. The results obtained by the experiments were quantitatively analysed using multi-objective quality indicators. After analysing the results and the obtained solutions, we discovered that PLA design with low feature scattering and low feature interlacing improves the performance of MAs. This paper is structured as follows. Sections 2 and 3 describe related work and SBPD, respectively. Section 4 outlines the MA used in SBPD and Section 5 describes the two experiments carried out. Sections 6 and 7 discuss the results and answers the research questions, respectively. Finally, Section 8 concludes this paper and makes suggestions for future work. 2 Related work The automatic generation of software architecture has been successfully carried out in the SBSE field [20]. However, the existing SBSE works [21-25] only address architectures of traditional software. Some of them work at a higher level of abstraction, component–connector level or deployment level [21, 24]. Certain activities in SPL engineering have benefited from SBSE techniques [26, 27]. Most studies focus on the following: automated feature model construction, the configuration of products, feature selection for SPL development and evolution, reverse engineering of feature models and SPL testing [24, 28-35]. Since they usually work with the feature model, their optimisation goal is not PLA design. In light of this, in the previous work, Colanzi et al. [4] introduced an optimisation approach called MOA4PLA. MOA4PLA is a systematic and fully automated approach that uses search-based algorithms to evaluate and improve PLA design. It provides a set of potential solutions with the best trade-off between different and conflicting objectives such as FM, coupling and cohesion. MOA4PLA was implemented employing the non-dominated sorting genetic algorithm II (NSGA-II) algorithm [36], which is based on GAs. Such an approach is described in the next section. MAs have achieved better quality solutions than those obtained using GA [7, 8, 10]. Thus, a systematic mapping (SM) was performed to identify related work involving the use of MAs in SPL and/or software design. This mapping sought to answer two key questions: SMO1 – What are the problems related to software design that can be solved with the aid of MAs? Moreover, SMO2 – How can MAs be used in the context of software product lines?. We searched for works on ACM, IEEE, Scopus and Springer databases and a total of 242 works were found. The studies that were collected underlined the value and versatility of using MA to solve optimisation problems. However, only two papers could respond satisfactorily to the SM questions, and these are as follows. About SMO1, Smith and Simons [5] use MA to optimise class modelling when linked to cohesion and coupling metrics. In this paper, the GA mutation operator is used to exchange methods and attributes between classes and measures how close the new modelling configuration is to the optimal solution. MA is used to speed up the search for solutions. In [9], MA is used in a test case generation and optimisation approach to assess the critical components of software architecture. None of the selected papers addresses problems related to SPL, and thus, it was not possible to answer the SM02 question. Hence, it is clear that there are still no studies about the use of MA in the context of SPL, as well as its architectural design. This fact is evidence that there is a gap in the research since MA has been effective in solving other SBSE problems. In light of this, we performed an exploratory study [12] to apply MA to PLA design optimisation. Four different versions of the MA based on different types of the local search were implemented and tried out. The results showed the best MA version, but also discovered that it is necessary to prevent the application of design patterns from being corrupted when some local search is used together with the search operators of MOA4PLA. For this reason, a constraint was added to the implementation of the operators to prevent this kind of corruption. The best MA version is described in Section 4. 3 Search-based PLA design PLA design is influenced by different factors such as FM, PLA extensibility, PLA reusability etc. Some of these factors may be in conflict, which leads to various modelling possibilities for a specific PLA design. SBSE techniques [2] can be used to optimise PLA to design automatically so that it can provide several near-optimal alternative designs according to the selected objectives. Metrics for evaluating those factors are used as objectives that can be optimised during the search process. In this context, the MOA4PLA [4] approach uses MOEAs with the aim of evaluating and improving PLA design concerning architectural properties such as FM, SPL extensibility, variability, similarity, design elegance, coupling and cohesion. This approach allows the use of different MOEAs in the optimisation of PLAs. The MOA4PLA process consists of four main activities: (i) construction of the PLA representation, (ii) definition of the evaluation model, (iii) multi-objective optimisation and (iv) transformation and selection [4]. Fig. 1 shows the activities as well as their inputs and outputs. In activity (i), the PLA design modelled in an unified modeling language (UML) class diagram is mapped for a symbolic representation based on a meta-model defined for MOA4PLA. Activity (ii) includes an evaluation model that comprises of a set of objective functions. This activity involves selecting a subset of objective functions related to the properties that the architect wants to optimise. The multi-objective optimisation is performed in activity (iii) which relies on particular search operators such as Move Method, Move Attribute, Add Class, Move Operation, Add Package Manager and Feature-Driven Operator [4]. In the last activity, the obtained solutions are converted in a legible view for the architect (a class diagram containing the PLA design), and he/she selects one of them to adopt as the PLA of the SPL. The application of MOA4PLA is automated by OPLA-Tool [37], which implements the NSGA-II [36] and the pareto archived evolution strategy (PAES) [38] algorithms to automate the PLA design optimisation. Fig. 1Open in figure viewerPowerPoint MOA4PLA process [4] The evaluation model of MOA4PLA encompasses several objective functions based on software metrics, which provide indicators about FM, PLA extensibility, variability, design elegance, coupling, cohesion and size. The entire evaluation model is set out by Verdecia et al. [39]. In the next section, we present the four objective functions used in our experiments: CM, FM, COE and ACLASS. CM measures conventional architectural properties by aggregating the coupling, cohesion and size metrics. It aims to achieve solutions with high cohesion and low coupling, formed by the sum of some CMs [40] as expressed in (1), where given a PLA design (pla), c is the number of components, itf is the number of interfaces, cl is the number of classes and f is the number of features of pla: DepIn, Depout, CDepIn, CDepOut and DepPack are coupling metrics; NumOps and H are size and cohesion metrics, respectively (1) (2) (3) (4)FM provides an indicator for FM based on the sum of the metrics for feature scattering (CDAC, CDAO, CDAI), feature interlacing (CIBC, IIBC, OOBC) and feature-driven cohesion (LCC) [39]. It evaluates the FM of the PLA design, which is formed by metrics specific to SPL features [41] according to (2), where given a PLA design pla, c is the number of components and f is the number of its features. The COE objective function evaluates the cohesion of PLA design in terms of the internal relationship of the classes of the PLA design, measured by the H metric (3). ACLASS measures class coupling by the number of architectural elements that depend on other classes of the design (CDepIn), added to the number of elements on which each class depends (CDepOut) according to (4). In the cases of (3) and (4), c is the number of classes. Guizzo et al. [19] conducted a feasibility analysis to determine which GoF design patterns [13] are feasible to be used in the context of MOA4PLA. This meant that a pattern was considered to be feasible if it was possible to automatically identify a suitable scope for its application in the class diagram that represents the PLA design, and the pattern was able to benefit the PLAs in terms of coupling, cohesion, PLA extensibility and feature-driven metrics. As a result, only four GoF patterns were regarded as feasible: Strategy, Bridge, Mediator and Facade. The strategy allows varying the algorithm independently of its users; it thus encapsulates families of algorithms by making them interchangeable. The bridge creates an abstraction for its implementation and allows both to vary independently of each other. Mediator allows a weak coupling and prevents the objects from referring explicitly to each other; it does this by creating an object that encapsulates the iterations of a set of objects so that the interactions can vary independently. Facade defines a unified interface (a facade class) for a set of interfaces in a subsystem. It makes the subsystem easier to use by encapsulating its main operations in a higher-layer interface. Even though facade was considered feasible, its implementation was postponed by the authors for future work. Hence, facade is beyond the scope of this paper. Besides, Guizzo et al. [19] proposed a search operator, called design pattern mutation operator, for the automatic application of the feasible design patterns in PLAs. Such an operator selects one of these patterns at random. After this, it randomly selects a set of classes and interfaces based on the PLA design. This set is called 'scope'. If the scope is suitable for applying the selected pattern, the classes and interfaces are changed to create the design pattern. Methods to determine whether a 'scope' is suitable are described in [19]. Strategy and Bridge patterns influence both the conventional architectural properties, especially coupling and cohesion, and SPL-specific properties such as the SPL extensibility. The main impact made by Mediator is on the coupling. The experiments were carried out by employing the design pattern mutation operator and the native mutation operators of MOA4PLA and achieved satisfactory results. By applying this operator, it was found that the joint usage of native operators and the proposed operator achieved better results than using each in isolation. However, Guizzo et al. [19] have argued that the empirical results showed the overuse of the design pattern mutation operator causes saturation – design patterns overhead – to PLA design and impairs its quality. 4 MAs in SBPD MAs add a local search after the global search process performed by a GA. The MA structure allows the individuals of the global population the possibility of local optimisation [7]. GA can sometimes lose relevant local characteristics, which the global search disregards since it is unable to find neighbouring solutions. This problem is solved through an integrated local search since it is able to explore the neighbourhood. Goldberg et al. [42] state that hybridisation is an extremely effective way of improving the performance and efficiency of GA. The most common form of hybridisation (called MA) is to integrate a local search technique into GAs as a decisive stage in its evolution, and also to incorporate the domain-specific knowledge in the search process. The local search can be characterised as a local refinement within a search space, and for this reason, MA is related to the cultural evolutionary theory, in which individuals must adapt to meet the needs of the problem. In contrast, GA is based on the biological evolution of individuals, so far as the offspring will inherit many of the skills and traits that are found in their progenitors. In this paper, the design pattern mutation operator proposed by Guizzo et al. [19] was applied as a local search operator of the MA, since it incorporates the domain-specific knowledge. The reason for using this kind of operator is that its empirical results are satisfactory [19]. This shows that the joint usage of native operators and the design pattern mutation operator are capable of obtaining better solutions than their isolated usage. As is the case in other situations, we believe that the MA can help to improve the quality of the PLA design solutions [5-7, 9, 10] because a particular design pattern applied in some part of the design could improve some properties of the solution obtained by the global search. In this first study to apply MA for the PLA design optimisation, we are relying on the feasibility analysis performed by Guizzo et al. [19]. In this way, we are using only the patterns whose full-automatic application was considered feasible. In further studies, we intend to investigate the possibility of applying other design patterns in the local search. In this context, the developed MA is shown in Algorithm 1 (Fig. 2). The NSGA-II implemented in OPLA-Tool was adapted to adding the local search, which applies design patterns in the PLA design using the design pattern mutation operator. Two parents ( and ) are selected. The crossover operator obtains two children from the parents (line 5). The children are mutated and, then, the local search is applied for each child. The children resulting from this process form a new population (lines 10 and 11). Fig. 2Open in figure viewerPowerPoint Algorithm 1: MA – adapted from NSGA-II The local search randomly selects the set of classes and interfaces, the scope of which will be analysed to determine if it is feasible for an automatic application of the mediator, bridge or strategy patterns. This means that the local search only makes changes in the PLA design if a suitable scope was selected for applying one of the three design patterns implemented in the operator. Our previous work [12] includes a study aimed at finding out which of four local search selection versions perform best. The four versions were called NoChoice, Bestof2, Bestof12 and UntilBest and are described below: NoChoice adopts the first improvement approach [43]. It returns the current solution (obtained after the local search) which will then proceed to the next generation of the global search. Bestof2 is also based on the first improvement approach [43]. In this version, the solutions obtained in the global and local searches are compared, and the best one is returned. Bestof12 adopts the best improvement approach [44], which creates a neighbourhood of solutions found by the local search and returns the best one. This version of MA was executed in a quad-core processor. Three local searches were instantiated for each core through paging performed by the operational system, resulting in 12 neighbours. This version returns the best solution to this neighbourhood. UntilBest works with the first best improvement approach. The algorithm carries this out by repeating the local search using the same global search solution until it achieves an improvement in the fitness sum, i.e. until it makes the first improvement in the neighbourhood. Once a better solution is found, it will be returned. The Bestof2 version obtained the best quality solutions according to the experimental results [12], and so it was decided to use this version in the experiments carried out in the present work. Its pseudocode is displayed in Algorithm 2 (Fig. 3). The local search saves the original solution in the sOri variable in line 1. In the sequence, it attempts to apply some design pattern (line 2). The solutions sOri and sLoc are evaluated in line 3, and the best one is returned. Fig. 3Open in figure viewerPowerPoint Algorithm 2: Local search Algorithm 3 (Fig. 4) employs the theBest method, which evaluates the best solution. It entails comparing the fitness sums of the solutions provided by the global and local searches (lines 5–8). The solution that has the lowest sum is selected because an optimisation by minimisation is performed. A fitness sum consists of the sum of the values obtained for each objective function. The sum is calculated in lines 1–3. Fig. 4Open in figure viewerPowerPoint Algorithm 3: The best OPLA-Tool encompasses six native mutation operators (Move Method, Move Attribute, Add Class, Move Operation, Add Manager Package and Feature-Driven Operator) which are applied by GA. A constraint was added to these operators to prevent any corruption in the design pattern application; this ensures that they are not able to modify architectural elements where there is some design pattern applied. Hence, the constraint checks whether the class or interface selected for mutation contains an applied design pattern; if so, the mutation process is completed. As mentioned earlier, our exploratory study [12] was the first research study to investigate the application of MA in PLA design optimisation. A replication of the study that uses both the Bestof2 version of MA and the new constraint on GA operators is required to find more evidence for this scheme and supply a body of knowledge that can fill this research gap. With this in mind, the next section describes the empirical study of the MA application for PLA design optimisation that was carried out in this work. 5 Definition of the empirical study In light of the motivating factors discussed in the previous sections, this section describes the empirical study, which seeks to answer the research questions of this work. The study was divided into two experiments: Exp01 and Exp02. As stated earlier, the first experiment used the CM and FM functions (Section 3). Since these functions are composed of metrics aggregation, they cannot be too sensitive for the changes made by the patterns in the PLA design. Metrics such as DepIn, DepOut, DepPack and NumOps (1), aggregated in the CM function, could not undergo variations when the Mediator, Bridge and Strategy patterns were applied. Taking into account that the design patterns applied by the local search improve relational cohesion [metric H – (3)] and class coupling [CDepIn and CDepOut metrics – (4)], the second experiment uses the COE function responsible for measuring the relational cohesion and the ACLASS function, which measures class coupling. COE and ACLASS only consisted of metrics related to the architectural properties affected by the design patterns applied in the local search. For this reason, they tend to capture the changes caused by the patterns more intensely than the CM and FM functions. Besides, the architectural properties, measured by the ACLASS and COE functions (coupling and cohesion), were cited by software engineers as being the most important properties that needed to be evaluated in the PLA design optimisation [45], what corroborates the choice of this pair of objective functions for Experiment 2. Multi-objective quality indicators were used to measure the performance of the algorithms and to provide evidence that could be used to answer . These indicators are described in the next section. We also analysed the design of the obtained solutions to find an answer to . The planning of the experiments and threats to the validity of our empirical study are also addressed in the following sections. 5.1 Multi-objective quality indicators Some of the quality indicators require the true Pareto front. However, it is hard to find the optimal Pareto front, so we defined the reference front (PFtrue) as the set of all non-dominated solutions obtained by all the algorithms in all the runs, which is standard practise [46]. In each run, a multi-objective algorithm generates a set of solutions that is an approximation to PFtrue and is called PFapprox. The front formed by all the non-dominated solutions obtained by each algorithm in all the runs is called PFknown. Hypervolume (HV) [47] measures the area that a Pareto fr