# Evaluating the Effectiveness of Designs for Low-Cost Digital Manufacturing Systems
## 4 Development of the Evaluation Testbed
### A Case Study Using Reference-Architecture-Based Designs  
Not all design approaches are capable of addressing the various design challenges of low-cost digital systems. The _tree-structured methodology_ yields rigid system designs, which require much effort if design changes need to be made. Conversely, the _platform-_ and _network-based methodologies_ provide a high flexibility for connecting system components, thus enabling a successive development process. As most _architectures_ are based on these methodologies, and they offer similar levels of flexibility. More detailed architectures additionally include standards, technology recommendations and implementation guidelines, such as the development of unified interfaces, which vastly reduces the required effort when designing low-cost digital systems containing disparate components. In contrast, the _aximomatic design approach_ is not suitable for the design of low-cost systems, since its focus lies in large complex systems. However, this study is concerned with small digital systems with few functional requirements and components. Thus, adopting an axiomatic design approach would yield an undesirable design overhead. Finally, an _object-oriented approach_is insufficient for designing low-cost digital manufacturing systems because it cannot be applied to hardware components. This study uses the proposed evaluation methodology for the special case of designs that are directly based on reference architectures. _Reference architectures are models that provide a structured template solution with common terminology and support the design of digital systems through a systematic repeatable framework_[14]. For the structured implementation comparison, we select three candidate reference architectures relevant to low-cost digital systems, namely Shoestring [16], WoT [49] and PROSA/Erlang [50; 51], since they have been compared previously [37]:  
* **Shoestring** proposes two types of elements, namely _building blocks_ and _service modules_. Building blocks provide base functionalities and well-defined interfaces by encapsulating an off-the-shelf technology, whereas service modules join several building blocks with wrappers to provide their functions and data at the network level. * **WoT** abstracts any phyiscal or virtual resource as a web resource. These so called _things_ consist of standardised metadata, which are processed by _consumers_ to interact with those things. * **PROSA** describes the roles and components of manufacturing systems via autonomous cooperating agents, so-called holons [52]. _Resource holons_ contain a physical part representing a system component and an information processing part controlling this component, while _order holons_ define a task in the manufacturing system and coordinate the operation. Figure 1 illustrates how each reference architecture encapsulates system components into their respective architecture elements. For a monitoring system comprising a Raspberry Pi and a sensor, each _architectural element_ (service module, thing, holon) provides an abstract view on these system _components_ and provides their functionalities and data at a network level. The selected _reference architectures are likely to continue to evolve_. The evaluation conducted in this study captures the effectiveness of designs based on each reference architecture at a given moment of time. We acknowledge that the evaluation results may change with future developments. Additionally, using different reference architectures would have a significant impact on the evaluation results. For example, there are many highly abstract digital manufacturing reference architectures with only limited design and implementation guidance, such as RAMI 4.0 or IIRA [14]. These more abstract reference architectures would be much less effective in supporting the design of low-cost digital manufacturing systems compared to Shoestring, WoT or PROSA/Erlang.