# Evaluating the Effectiveness of Designs for Low-Cost Digital Manufacturing Systems
## 4 Development of the Evaluation Testbed
### Design Effectiveness Measures and Metrics  
To assess the effectiveness of designs, the effort required to design a digital system is measured. Several factors have an impact on the design effort when adopting a reference architecture. These factors can be clustered into three areas, namely system, architecture, and designer. System dependent impact factors include performance requirements, such as latency, and data requirements, such as bandwidth and storage capacity. Architectural impact factors involve the design overhead caused by the architecture, the ease of integration of components and the technology a particular reference architecture is based on. Designers affect the design effort in terms of their skills. Specifically, compared to a novice, a skilled designer may perceive the effort of building a specific digital system as low. The time to develop a digital system may be affected by all three areas. To assess the effectiveness of designs that are directly based on a reference architecture, we focus on architectural impact factors. These factors need to be _measurable_ and _independent_ from a specific system requirement or the designer. While measurability is necessary to quantify the required design effort, independence ensures an objective evaluation and comparison. In particular, design guidelines provided by a reference architecture may not always be strict and thus the required design effort can vary significantly across designs. In what follows, an overview of the architectural impact factors and selected measures and metrics is provided, which is summarised in Table 1. The measures and metrics have been validated through conducting an initial evaluation test. In this test, each selected measure and metric is applied to the three different baseline solution designs described in Section 3.4. If an evaluation measure or metric can be successfully applied and yields meaningful results, it is deemed to be validated. **Architecture overhead. ** The architecture overhead captures the effort of building the architectural elements and tailoring the digital system to connect the system components to those elements. It additionally provides insights on the complexity of the digital system imposed by the reference architecture. The effort of constructing the architecture elements is determined by counting the overall number of those elements for a given system, measuring their size by comparing the LOCs of these elements to the LOCs of the overall digital system and specifying their overlap, which describes how similar these elements are to one another. The tailoring effort to connect components to architecture elements is assessed by calculating the number of additional components, such as an extra HTTP server functioning as an intermediary between a component and an architecture element. The complexity imposed by the reference architecture is measured by counting the number of interactions (i.e., 1's) in the DSM. A key benefit of using the DSM for these modular reference architecture designs is that it requires a clear separation between the different elements of a low-cost digital manufacturing system (e.g., hardware and software components, architecture elements, additional interface components) and the identification of all interactions between those elements. Other design evaluation measures, such as flowcharts, are not subject to such strict rules, which are likely to yield evaluation results that are not comparable across different reference architecture designs.