# An Open Benchmark Exercise for Model-Based Design Reviews

Victor Romero, Romain Pinquié, Frédéric Noël

► To cite this version:

Victor Romero, Romain Pinquié, Frédéric Noël. An Open Benchmark Exercise for Model-Based Design Reviews. 19th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2022, Grenoble, France. pp.176-185, 10.1007/978-3-031-25182-5_18. hal-03966247

HAL Id: hal-03966247

https://hal.science/hal-03966247v1

Submitted on 19 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# An Open Benchmark Exercise for Model-Based
Design Reviews

Victor Romero(✉), Romain Pinquié, and Frédéric Noël

Univ. Grenoble Alpes, CNRS, Grenoble, INP, G-SCOP, Grenoble, France
{victor.romero,romain.pinquié,frederic.noel}@grenoble-inp.fr

**Abstract.** The design of engineered systems is a process punctuated by design reviews pursuing various goals such as sharing a common understanding of intermediate representations, making sure that the design meets the requirements, and making informed decisions about the future of the project. In a model-based design approach, an issue of the design reviews is the difficulty to establish a common ground since participants have to understand various models that serve as intermediate representations to communicate during meetings. This observation motivates new technology-mediated design situations with no clear evidence of progress. Indeed, a wide variety of design review environments exist; however, there have been a very limited number of comparisons of these environments to date. This is mainly due to the lack of benchmark problems to evaluate candidate design review environments claiming to facilitate the understanding of model-centric designs. This paper proposes an open-science benchmark exercise that includes the definition of the pursued goals, the measures of performance, the sources of a telescope model-centric design from three viewpoints in line with the universal Function-Behaviour-Structure (FBS) ontology, and a systematic experimental protocol. This benchmark problem should enable anyone to provide objective evidence of progress regarding new environments for reviewing model-centric designs.

**Keywords:** Model-based design · Design review · Grounding · Benchmark problem

## 1 Introduction

The increasing complexity of designed systems pushed engineers to give up the document-based design and replace it with a model-based design approach (a.k.a. model-based systems engineering, model-centric design, model-based engineering, integrated model-centric engineering, model-centric engineering, etc.). If the use of models improves design meetings [1], the stakeholders involved in a new product development project still must meet in design reviews.

The establishment of common ground is crucial communication activity to ensure good design [2–4]. Grounding in communication is to “coordinate on content without
assuming a vast amount of shared information or common ground that is, mutual knowl-
edge, mutual beliefs, and mutual assumptions" [5]. In engineering design, as stated
by Burkhardt [6], "The establishment of common ground is a collaborative process in
which the co-designers mutually establish what they know so that design activities can
proceed. Grounding is linked to sharing of information through the representation of
the environment and the artefact, the dialogue, and the supposed "pre-existing" shared
knowledge. This activity ensures inter-comprehension and construction of shared (or
compatible) representations of the current state of the problem, solutions, plans, design
rules and more general design knowledge.". Successive design reviews serve to update
the common ground moment by moment and this is why we will focus on the common
undersdanting of the design of engineered systems.

Today, the most common practice to coordinate content consists in sharing screen-
shots of models in a slide deck [7], which makes the understanding of models and the
identification of relationships between the models difficult [7, 8]. Research studies try to
improve design reviews by proposing new mediation technologies, but they show a lack
of rigour in the validation phase which is also complained about in engineering design [9,
10]. Indeed, the validation of these claims remains weak since most of them demonstrate
that the solution is functional without objective evidence of progress [11–16]. In some
cases, experimentations are performed to demonstrate the efficiency of the solution for
a specific activity [17–19]. Despite empirical validation, results remain incomparable
because of a missing common benchmark problem.

After a literature review of the virtual environments supporting design reviews, Sect. 3 introduces a benchmark problem for evaluating a candidate for model-centric design review environments.

2 Literature Review

As stated in the previous section, with PowerPoint-like presentations, stakeholders of
design projects face difficulties during design reviews to share and understand the models
and their relationships [7, 8].

One way among others to pass by this issue is to visualise these models in specific
environments. Several studies propose tools for the design review process in various
domains such as building architecture [11], civil engineering [12], industrial engineer-
ing design [13, 14], but the most popular applications concern mechanical geometric
definition and space allocation reviews [15–19]. These environments are desktop-based
human-machine interfaces through which the users can interact with the models [11] or
augmented reality devices [13] for a natural capture of the interactions with the environ-
ment. However, the majority of the papers concentrate on virtual reality devices [12],
[14–19] for conducting an immersive design review in a collaborative mode [16] with
3D model interactions to move, scale, and rotate the object [14, 16, 17].

If these studies claim to improve design reviews, the validation of contributions
remains weak. Most of them present a demonstrator [11, 13, 14, 16], sometimes com-
pleted by observation of a few subjects [12, 15], but only a small number of studies
scientifically validate their finding by quantitative and objective experimentations [17–
19]. If presenting a demonstrator is the first step to finding new interactions and visual-
isation techniques, all the more when targeted end-users experience the demonstrator,
the coexistence of too many alternatives hampers the selection of the suitable design review solutions in academia and industry. In engineering design research, a company should rapidly find the leading edge of a desired field of knowledge, that is, what works, for whom, and under which circumstances. However, by proposing experimentations to validate the usefulness of a proposed environment, we are only halfway up because an objective comparison requires evaluating candidate solutions on the same benchmark problems.

# 3 A Benchmark Proposal

To improve the quality of validation, the authors propose the first version of an open benchmark problem to evaluate environments supporting model-centric design reviews.

## 3.1 Introduction

This benchmark problem in engineering design aims at evaluating model-centric design review environments, especially for the activity of grounding that consists in coordinating content by sharing intermediate representations to ensure inter-comprehension of the current design situation.

## 3.2 Goals and Constraints

All candidate solutions evaluated with this benchmark shall intend to improve the user's understanding of a model-based design including three main goals:

- To maximize the understanding of a model-based design, especially the top-down systemic decomposition (vertical traceability) and the relationships linking design objects across different views (horizontal traceability),

- To maximize the usability, and

- To minimize the cognitive load.

Anyone can reuse the go-to telescope dataset that will be introduced later or create a new one under the following constraints:

- The pursued goal is to facilitate the understanding of a model-based design during design reviews with an emphasis on vertical and horizontal traceability,

- The design concerns a multi-engineering system (not only mechanical),

- The design must be model-based (not document-based),

- The data must contain at least three different views (functional, logical, behavioural, structural, safety, quality, manufacturing, etc.).

- The data to be reviewed include engineering data (requirements, functional architectures, geometric definitions, mechanical kinematics, failure modes, system dynamic simulations, reliability block diagrams, etc.) and potentially project data (market studies, financial expectations, Gantt planning, etc.).
## 3.3 Dataset

To measure the efficiency of new candidate solutions, the authors propose a systematic experiment with a telescope as a use case. The model-based design of a simplified fully automated go-to telescope serves as an initial dataset to perform the systematic experiment, but it can be completed with new datasets that will help us to conduct a meta-analysis for drawing more generic conclusions.

The design review takes place during the conceptual design phase of the new product development process, and three models already exist: a system architecture model, a preliminary geometric definition CAD model, and a behaviour model.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Extracts of the SysML-based architecture model of the telescope. All these models can be found in the git page (https://github.com/GIS-S-mart/Benchmark-6_model-based-design-reviews)</figcaption></figure>

We assume that the design process starts with the development of the architecture model in SysML with Papyrus as the editing environment. We begin by defining the environment of the telescope including human and non-human stakeholders that interact (directly) with or influence (indirectly) the system-of-interest (Block Definition Diagram Fig. 1a). Then we define the services (external functions) that the telescope must provide to the stakeholders and the constraints imposed by the environment (Use Case Diagram Fig. 1b). By adding performance intervals to the services, we get the telescope system requirements (Requirement Diagram Fig. 1c). Once the requirements are validated, we make the first design choice at the system level by decomposing the system into sub-systems (Internal Bloc Diagram Fig. 1d). Thus, the telescope is decomposed into a mechanical assembly sub-system, a controller sub-system, and a DC motor sub-system. Then, for each sub-system, we recursively follow the same process by defining its stakeholders, functions, and requirements. Finally, we decompose again these sub-systems into sub-sub-systems until we consider that the system is an elementary component that is acquired according to a buy, make or reuse strategy.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Preliminary structural (left) and behavioural (right) design of the go-to telescope</figcaption></figure>

Two other views of the go-to telescope correspond to the preliminary geometric definition created with PTC Creo (Fig. 2, left) and a first physics-based behavioural model implemented in Modelica with 3D Experience Dynamic Behaviour Modelling (Fig. 2 right). To facilitate data exchange and integration we provide the sources in standardized STEP or OBJ and Modelica formats, respectively.

Figure 3 illustrates the graph-oriented data model with objects, object properties, and relationships giving an overview of the interdependencies between the three views in line with the universal Function-Behaviour-Structure (FBS) ontology: function (architecture model), behaviour (behaviour model), and structure (CAD model). This data model should be reused as it is for comparing competing design review environments, even with new datasets that differ from the telescope, so as to avoid any biases due to changes in the data model that could facilitate the achievement of the tasks and lead to the comparison of incomparable results. The parsing of data is achieved with ad-hoc parsers before being stored in a graph-oriented Neo4j database.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Structure of the graph database used for our dataset</figcaption></figure>
## 3.4 Measures of Performance

Remember that the candidate solutions that will be evaluated on this benchmark problem pursue three goals: 1) to maximize the understanding of a model-based design, especially the top-down systemic decomposition and the digital threads across different views; 2) to maximize the usability, and 3) to minimize the cognitive load. Four measures of performance serve to show how well a solution accomplishes these goals.

* To maximize the understanding of a model-based design

Since the major difficulty during design reviews concerns grounding with a shared understanding of models [7], the questions concentrate on (in)direct relationships between design objects captured in different models and at various systemic levels.

There are two sets of questions, built then to force the user to find a link between elements of different models. For the first set, participants can answer using the design review environment to find the answer to questions 1 to 3. For the second set, participants have 5 min to explore the data and, after closing the environment, they must answer questions 4 to 6 based on what they remember or what they have understood about the system under design.

**Questions with access to the design review environment:**

1. What is the ID of the requirements satisfied by the assembly (from the CAD model) "TelescopeAssembly"?

2. What is the name of the functions needed by the stakeholder "PowerGrid"?

3. What is the name of the stakeholder which needs the requirement "Req Manually pilot inclination position" to be met?

To avoid answering all the questions in a row, they are displayed one by one.

**Questions without access to the design review environment:**

1. What are the stakeholders of the telescope?

2. What are the sub-logical blocks (of the behaviour model) of the telescope?

3. What are the stakeholders of the sub-system "MechanicalSubSystem" (in the architecture model)?

These two sets of questions depend on the telescope dataset. If someone decides to share a new dataset to generalize conclusions based on a meta-analysis, then it would be necessary to adapt the questions by replacing the instances of assemblies (e.g. TelescopeAssembly), stakeholders (e.g. PowerGrid), requirements (e.g. Req Manually pilot inclination position), etc. with instances of the new dataset without fundamentally changing the questions.

Three measures of performance serve to evaluate the level of understanding of the reviewed model-based design:

1. The time spent by each participant to answer the first set of questions 1 to 3. This criterion helps to compare the efficiency (the shorter the better) of exploring data with other design review environments.
2. For each set of questions, the total number of wrong answers.

3. The perceived level of confidence of the user in answering questions 1 to 6. A "confidence index" between 1 (not confident) and 7 (absolutely confident) indicates respectively if participants are sure that an answer is right or wrong. The measure of confidence helps to spot random answers.

* *To maximize the usability*

Although advanced solutions (e.g. eyes tracking) could have been used to evaluate the usability of the candidate model-centric design review environment, the well-known System Usability Scale (SUS) questionnaire [20] is preferred for widespread adoption. The standardized SUS questionnaire gives a usability score between 0 (awful) and 100 (excellent). The SUS questionnaire is independent of the task and can consequently be used to compare candidate environments without any change.

* *To minimize the cognitive load*

The evaluation of the cognitive load relies on the Task Load Index (TLX) measured with the NASA/TLX questionnaire [21]. The workload score varies between 0 (low cognitive load) and 100 (high cognitive load). This test is dependent on the task to achieve and therefore supports the comparison of results obtained with the same questions and dataset.

## 3.5 Participants

The evaluation of the usability with the SUS questionnaire requires 14 participants to have significant results [22], whereas experiments show that 12 subjects may already be significant [23]. Therefore, to be confident in the validity of the results, we select at least 15 subjects and, if possible, with varying profiles (position, age, sex, etc.), especially regarding their level of expertise in Model-Based Design ranging from novice to experts with various domain of expertise (mechanics, systems engineering, etc.).

## 3.6 Systematic Experimental Protocol

The systematic process to experiment is composed of three activities:

### 1. *Answering a demographic questionnaire*

To analyse the impact of participants’ profiles on the results, e.g., a virtual reality expert may overestimate usability compared to an expert in systems engineering. Thus, participants must answer a demographic questionnaire when starting the experiment.

### 2. *Training*

Candidate model-based design review environments will often be new to users. Our goal is not to evaluate the intuitiveness of the solution but its efficiency. Therefore, a training phase is mandatory to reduce experimental bias. During this phase, participants
have unlimited time to learn how to use the environment and get familiar with the visual and interactive metaphors for exploring data. The training relies on the review of an inverted pendulum system which also includes subsystems (a controller sub-assembly, a mechanical sub-assembly and a DC motor sub-assembly) with three viewpoints (function, behaviour and structure), that is, compliant with the data model (Fig. 3). Thus, it is relatively similar to the go-to telescope used for the experiment. During this phase, participants can ask any question about the tool and the engineering data. To validate the training, the participant must rightly answer three questions that are relatively similar to the ones asked during the experiment:

1. What is the name of the requirements associated with the stakeholder "Gravity"?

2. What is the name of the logical block (from the behaviour model) that satisfies the requirement "ReqPilotMotor"?

3. What are the functions fulfilled by the assembly (of the CAD model) "INVERTED-PENDULUM"?

### 3. *Experiment*

Once the participants pass the training questions, the experiment can start with the model-centric design review of the go-to telescope. The time is limited to 20 min. Indeed, by experience, depending on the complexity of the environment, 30 to 45 min are necessary to accomplish the training phase. The total time for the experiment is about 1 h, which is acceptable for convincing the targeted participants.

The first phase of the experiment is the evaluation of how well participants retrieved linked data with the model-based design review environment. In practice, participants read the first set of questions 1 to 3 and try to find the answers. During this phase, the participant can navigate back and forth between the environment and the questions for 20 min, and he does not receive any external feedback such as answers to questions arising during the task. This part of the experiment evaluates both the understanding of the model-based design without requiring memory capabilities and the usefulness of the tool for data exploration. To conclude this first phase, participants estimate a confidence index for each question.

Once the participant considers having answered the questions 1 to 3 or when the 20 min time limit is over, participants have 5 min more in the environment to better understand the model-based design. At the end of the 5 min, participants close the environment before answering the second set of questions 4 to 6 based on their understanding of the system. They evaluate the confidence index too. For this second phase, the time is neither limited nor tracked as we want to evaluate their understanding of the design without considering memorization abilities. To conclude the experiment, participants answer the SUS and NASA/TLX questionnaires.

## 4 Conclusion

Several research studies propose environments which claim to facilitate model-based design reviews, but the lack of focus on validation and common benchmark problems
hampers clear evidence of progress. This paper proposes an open-science benchmark exercise that includes the definition of the pursued goals, measures of performance, sources of a telescope model-centric design from three viewpoints in line with the universal Function-Behaviour-Structure (FBS) ontology, and a systematic experimental protocol to compare such environments. In open access as a GitHub repository<fnref n="1" /> of the open-science S.mart platform [24], this first version of the benchmark problem is open to community-based improvements (addition, removal or change of goals, criteria, datasets, etc.) for enhancing the quality of our research. Future works could focus on the development of new academic and industrial datasets and a meta-analysis to draw more general conclusions on the added value of such mediated design situations.

**Acknowledgements.** This work was supported by the French National Research Agency under the Investments for the Future Programs (PIA) with the grant ANR-21-ESRE-0030 (CONTINUUM project) and by the LabEx Persyval Mine'VR Project.

## References

1. McDermott, T., Hutchison, N., Salado, A., Henderson, K., Clifford, M.: Benchmarking the benefits and current maturity of model-based systems engineering across the enterprise: Results of the MBSE maturity survey. in *Systems Engineering Research Center (SERC)* (2020)

2. D'Astous, P., Détienne, F., Visser, W., Robillard, P.N.: Changing our view on design evaluation meetings methodology: a study of software technical review meetings. Des. Stud. **25**(6), 625–655 (2004). https://doi.org/10.1016/j.destud.2003.12.002

3. Olson, G., Olson, J., Carter, M., Storrosten, M.: Small group design meetings: an analysis of collaboration. Hum.-Comput. Interact. **7**(4), 347–374 (1992). https://doi.org/10.1207/s15327051hci0704_1

4. Stempfle, J., Badke-Schaub, P.: Thinking in design teams - an analysis of team communication. Des. Stud. **23**(5), 473–496 (2002). https://doi.org/10.1016/S0142-694X(02)00004-2

5. Clark, H.H., Brennan, S.E.: Grounding in communication. In: Resnick, L.B., Levine, J.M., Teasley, S.D. (eds.) Perspectives on Socially Shared Cognition. American Psychological Association. Washington, pp. 127–149 (1991). https://doi.org/10.1037/10096-006

6. Burkhardt, J.-M., Détienne, F., Hébert, A.-M., Perron, L., Safin, S., Leclercq, P.: An approach to assess the quality of collaboration in technology-mediated design situations. In: Proceedings of ECCE 2009 European Conference on Cognitive. Ergonomics, pp. 8 (2009)

7. Pinquié, R., Romero, V., Noel, F.: Survey of model-based design reviews: practices & challenges? Proc. Des. Soc. **2**, 1945–1954 (2022). https://doi.org/10.1017/pds.2022.197

8. Baduel, R., Chami, M., Bruel, J.-M., Ober, I.: SysML models verification and validation in an industrial context: challenges and experimentation. In: Pierantonio, A., Trujillo, S. (eds.) ECMFA 2018. LNCS, vol. 10890, pp. 132–146. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92997-2_9

9. Barth, A., Caillaud, E., Rose, B.: How to validate research in engineering design. In: Proceedings of the 18th International Conference on Engineering Design, Copenhagen, p.11 (2011)

<footnote n="1">¹ https://github.com/GIS-S-mart/Benchmark-6_model-based-design-reviews.</footnote>
10. Vermaas, P.E.: Design theories, models and their testing: on the scientific status of design research. In: Chakrabarti, A., Blessing, L.T.M. (eds.) An Anthology of Theories and Models of Design, pp. 47–66. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6338-1_2

11. Shiratuddin, M.F. Thabet, W.: Information-rich virtual environment (VE) for design review, p. 9 (2007)

12. van den Berg, M., Hartmann, T., de Graaf, R.: Supporting design reviews with pre-meeting virtual reality environments. J. Inf. Technol. Constr. **22**, 305–321 (2017)

13. Verlinden, J., Horváth, I., Nam, T.-J.: Recording augmented reality experiences to capture design reviews. Int. J. Interact. Des. Manuf. IJIDeM **3**(3), 189–200 (2009). https://doi.org/10.1007/s12008-009-0074-8

14. Romero, V., Pinquié, R., Noël, F.: An immersive virtual environment for reviewing model-centric designs. Proc. Des. Soc. **1**, 447–456 (2021). https://doi.org/10.1017/pds.2021.45

15. Aromaa, S., Leino, S.-P., Viitaniemi, J., Jokinen, L., Kiviranta, S.: Benefits of the use of virtual environments in product design review meeting. In: Proceedings of DESIGN 2012, the 12th International Design Conference, p. 10. Dubrovnik, Croatia (2012)

16. Adwernat, S., Wolf, M., Gerhard, D.: Optimizing the design review process for cyber-physical systems using virtual reality. Procedia CIRP **91**, 710–715 (2020). https://doi.org/10.1016/j.procir.2020.03.115

17. Wolfartsberger, J.: Analyzing the potential of Virtual Reality for engineering design review. Autom. Constr. **104**, 27–37 (2019). https://doi.org/10.1016/j.autcon.2019.03.018

18. Freeman, I.J., Salmon, J.L., Coburn, J.Q.: CAD integration in virtual reality design reviews for improved engineering model interaction. In: Volume 11: Systems, Design, and Complexity, p. V011T15A006 Phoenix, Arizona, USA (2016). https://doi.org/10.1115/IMECE2016-66948

19. de Casenave, L., Lugo, J.E.: Effects of immersion on virtual reality prototype design reviews of mechanical assemblies. In: Volume 7: 30th International Conference on Design Theory and Methodology, p. V007T06A044. Quebec City, Quebec, Canada (2018). https://doi.org/10.1115/DETC2018-85542

20. Brooke, J.: SUS - A quick and dirty usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I.L., (eds.) Usability Evaluation in Industry, p. 8 Taylor & Francis (1996)

21. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Advances in Psychology, vol. 52, , pp. 139–183. Elsevier (1988). https://doi.org/10.1016/S0166-4115(08)62386-9

22. Tullis, T.S., Stetson, J.N.: A comparison of questionnaires for assessing website usability, p. 12 (2004)

23. Guest, G., Bunce, A., Johnson, L.: How many interviews are enough?: an experiment with data saturation and variability. Field Methods **18**(1), 59–82 (2006). https://doi.org/10.1177/1525822X05279903

24. Pinquié, R., Le Duigou, J., Grimal, L., Roucoules, L.: An open science platform for benchmarking engineering design research. In: Presented at the 32nd Cirp Design Conference 2022 (2022)