# Design System Assessment Tool

S. Gendron, J. Brousseau, A. Elouafi, B. Urli

► To cite this version:

S. Gendron, J. Brousseau, A. Elouafi, B. Urli. Design System Assessment Tool. 9th International Conference on Product Lifecycle Management (PLM), Jul 2012, Montreal, QC, Canada. pp.359-368, 10.1007/978-3-642-35758-9_32. hal-01526135

HAL Id: hal-01526135

https://inria.hal.science/hal-01526135

Submitted on 22 May 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Design system assessment tool

S. Gendron¹, J. Brousseau¹, A. Elouafi¹, B. Urli¹

¹Université du Québec à Rimouski, Rimouski, Canada
{steeve_gendron, jean_brousseau, abderrazak_elouafi, bruno_urli}@uqar.ca

**Abstract.** In 2010, an exploratory survey focused on design methods, tools and techniques, was conducted in order to understand what type of design practices are used in Small and Medium-sized Enterprises (SMES) located in Eastern Quebec. This study reveals that most SMEs declared using a structured design process while very few are regular users of structured Methods, sophisticated Tools, or specialized design Techniques (MTT). Because the results suggest that the design process in SMEs is not as structured as could be expected a new research was launched. Specifically, the research objective consist to identify the most important determinants of design performance and to develop indicators for assessing the design process as well as the other dimensions of design projects. In this model, six dimensions describe what is called the design system: the environmental dimension, the human and techno-scientific axis, management, the design process and the product aspects. The Dimensions are evaluated using an assortment of variables (descriptors) to assess the “design system” of an organization. Each descriptor is related to the following issues: (i) the type of methods, tools and activities used by the organization, (ii) the importance of the descriptor according to the project success, (iii) the performance level of the organization in that matter and (iv) the involvement of the partners. The questionnaire has been tested and refined on a pilot basis by eight partners. In addition to the model and the questionnaire, the paper presents an overview of the survey results and it illustrates how the assessment tool can be used to identify the challenges that a company must face in order to improve its design practices. Finally, it exposes the strengths and weaknesses of the questionnaire and it identifies the improvements to be made before to use it as base of an observatory intended to measure and monitor the evolution of design practices of organizations.

**Keywords:** Engineering Design; Design systems; Design practices; Design assessment; Assessment tools;

## 1 Introduction

In several manufacturing sectors, it is recognized that product design is driven by different motivations [1] derived from a context that imposes adaptations and innovations [2-3]. The companies must not only control the quality, the cost and the time to market, but also, their flexibility and responsiveness [2]. Within this context, companies must focus their attention on the optimization of their manufacturing
processes but also of their design methods [3]. Good design practices are key factors of innovation and competitiveness of companies and those that are investing more in design increase their annual sales and gain a better access to new markets [1, 4].

To better understand the current design practices in SMEs, 31 companies located in Eastern Québec have participated in an exploratory study conducted in 2010 [5]. With regards to the methods of design, there was a gap between the fact of declaring the use of a structured design process (60% of respondents), while on the other hand, the use of design MTT (27% of respondents). Because a design process benefits from being complemented by MTT, these results seem to indicate that the design process used by a company is not as structured as it could or should be [5].

Following this first exploratory research, a new research project was launched, which aims to contribute to the development of a tool for characterizing and evaluating design practices. One of its specific objectives is to develop performance indicators. The proposed study is divided into two stages. The first step, presented in this paper, consists of developing a questionnaire suitable to determine the characteristics of the “design system” of organizations. At the present time, the pilot questionnaire is validated by eight different organizations. When refined, this tool will be used as basic ingredient to conduct the second stage which is a longitudinal study.

The theoretical model of design system and the assessment tool are described in the following sections. In section 2, we more specifically explain the model selected to build the questionnaire, how the descriptors related to the model were determined and what is assessed for each descriptor. In section 3, some graphics show the “design system” of a company and illustrate the usefulness of the questionnaire and how it can be used to identify design deficits. In Section 4, we discuss about the strengths and weaknesses of the assessment tool as well as the improvements to be made as a way to provide the basis for an observatory of design practices.

# 2 Development of the questionnaire

## 2.1 Choice of a conceptual model

The choice of a conceptual model that captures a wide variety of organisations, activities and projects is necessary to build the assessment tool. A synthesis of certain models of project description [2, 6-8] shows that to describe the context in which the design process evolves, one must consider three axes: (i) the human dimension, (ii) scientific and technological knowledge, and (iii) the socio-economic and organizational environment [9]. To take into account the interactions between the axes, it is proposed to consider three intermediate aspects, being: (iv) the product, (v) the process, and (vi) the organization. Figure 1 is adapted from the “design system” model [2]. It was used to build the questionnaire.

In this model [9], the internal and external environment axis represents the company and its environment. The human axis is represented by the human resources involved in the project. The techno-scientific knowledge axis corresponds to the knowledge and the technologies already in place in the company. The product aspect is placed between the environmental and knowledge axes because it is the product
that demonstrates the knowledge and technology and it is by its product that the company performs on the market. The process aspect is found between the knowledge and human axes because it is by the process (phases and activities) that the design team transforms knowledge into a product with such or such characteristic. The organizational (management) aspect is sited between the environmental and the human axes since at the organizational level, the management of a design project reflects in the allocation of resources and by the institution of an organization adapted to the situation. As the figure shows, the design project is at the center of the “design system” and we can see that axes and aspects are influencing each other.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Design System (adapted from Robin V.[2])</figcaption></figure>

## 2.2 Determination of design activities

There are many definitions of the design process [10]. However, it can be defined as a set of activities implemented (in a sequential and/or parallel manner) using the human and material resources to meet the design objectives and lead to the definition of the product [2].

Noting the diversity of ways of representing the design process but also their similarities, we particularly focused the ontology of design activities [11]. This ontology has the advantage to define a terminology based on the classification of design activities from generic sources of design [12-17]. Thus, most of the variables assessed by questionnaire emerged from this synthesis.

The first column of Table 1 shows the activities defined by the ontology while the second column shows the questionnaire items formulated from the first column. The items were formulated either by the association of related activities that were considered to be difficult to differentiate or by subdividing the activities considered to be too abstract. For example, in questionnaire items, “generating solutions” is generic, and it makes no distinction between “abstracting”, “associating” and “composing” that are activities closely linked to “generating” [11]. As another example, in order to help respondent understand the items, Table 1 shows the “decomposing” activity for
which two items were formulated ("decomposition of the problem" and "creating a taxonomy") according to the activity's description [11].

<table><caption>**Table 1. Ontology activities and Questionnaire items**</caption><thead><tr><th>Ontology activities</th><th>Questionnaire items</th><th>Ontology activities</th><th>Questionnaire items</th></tr></thead><tbody><tr><td colspan="2">Design definition activities</td><td colspan="2">Design management activities</td></tr><tr><td>Abstracting<br>Associating<br>Composing<br>Generating</td><td>Solutions generation</td><td>Constraining<br>Identifying</td><td>Feasibility study</td></tr><tr><td>Decomposing</td><td>Decomposition of the problem<br>Creating a nomenclature</td><td>Exploring</td><td>Exploring solutions with<br>the client</td></tr><tr><td>Defining</td><td>Define specifications</td><td>Information gathering</td><td>Technology watch</td></tr><tr><td>Detailing</td><td>Documentation for<br>manufacturing</td><td>Planning</td><td>Task planning<br>Resource planning</td></tr><tr><td>Standardising</td><td>Standardization</td><td>Prioritising</td><td>Objectives prioritization</td></tr><tr><td>Structuring/integrating</td><td>Choice of architecture<br>Benchmarking</td><td>Resolving</td><td>Detecting and resolving<br>conflicts</td></tr><tr><td>Synthesizing</td><td>Final report</td><td>Scheduling</td><td>Scheduling and monitoring</td></tr><tr><td colspan="2">Design evaluation activities</td><td>Selecting</td><td>Use of decision criteria</td></tr><tr><td>Analysing</td><td>Analyzing</td><td>Searching</td><td>Internal search<br>External search</td></tr><tr><td>Decision making<br>Selecting</td><td>Solutions selection</td><td></td><td></td></tr><tr><td>Evaluating</td><td>Performances estimation</td><td></td><td></td></tr><tr><td>Modelling</td><td>CAD<br>Rapid prototyping<br>CAM</td><td></td><td></td></tr><tr><td>Simulating</td><td>Simulating</td><td></td><td></td></tr><tr><td>Testing/experimenting</td><td>Making partial prototype<br>Making complete prototype</td><td></td><td></td></tr></tbody></table>

It should be noted that the presented questionnaire items do not correspond exactly to the questionnaire's statements. To facilitate understanding the questionnaire items, the complete formulations of items include practical examples. Thus, the complete statement of the item "decomposing of problem" is "decomposing of problem (as modules, functions, energies, resources, requirements, etc.)".

## 2.3 Design practice descriptors

The axes and intermediate aspects of the "design system" give the possibility to consider the design practices in a broader perspective instead of considering design only in its process aspect. As several proposed items are derived from the ontology, these are adapted to the model. Thus, the activities coming from the ontology are assigned to the process, the management and techno-scientific aspects.

The descriptors referring to the notion of exploring and understanding the problem are attributed to the process aspect. These activities lead to an increasingly clearer definition of the evolving design until all the details required for production have been obtained [11]. Note that this categorization is not intended to restrict the design process to the definition activities and that other descriptors inspired by the ontology can be integrated to the design process although these characterize other dimensions of the model. The evaluation activities are assigned to the techno-scientific axis. These descriptors seek to analyse and evaluate potential design solutions and require
specialized knowledge in order to select the applicable solutions [11]. The management activities become management descriptors and it was added two descriptors to this dimension ("concurrent engineering" and "Product Data Management (PDM)") in order to take into account the work flow and data management [1].

Some other descriptors were added, in particular for the product aspect and the environmental and human axes. These dimensions include descriptors such as methods (product aspect) and motives (environmental axe) to identify approaches or special orientations of the "design system".

The axes and intermediate aspects and linked variables are illustrated in table 2.

<table><caption>Table 2. Dimensions and descriptors</caption><thead><tr><th>Human axis</th><th>Environnement axis</th><th>Techno-Scientific axis</th></tr></thead><tbody><tr><td>1- Multidisciplinary team<br>2- Skills and expertise<br>3- Personality types<br>4- Promoting collaboration<br>5- Promoting motivation</td><td>1- Customer preferences<br>2- Time to market<br>3- Product performance<br>4- Technology Performance</td><td>1- CAD<br>2- Rapid prototyping<br>3- CAM<br>4- Analyzing<br>5- Simulating<br>6- Performances estimation<br>7- Solutions selection<br>8- Making partial prototype<br>9- Making complete prototype</td></tr><tr><td><strong>Management aspect</strong><br>1- Concurrent engineering<br>2- PDM<br>3- Feasibility study<br>4- Exploring solutions with the client<br>5- Technology watch<br>6- Internal research<br>7- External research<br>8- Task planning<br>9- Resource planning<br>10- Scheduling and monitoring<br>11- Objectives prioritization<br>12- Detecting and resolving conflicts<br>13- Use of decision criteria</td><td>1- Design for Assembling<br>2- Design for Manufacturing<br>3- Design for Maintainability<br>4- Design for Reliability<br>5- Design for Safety<br>6- Design for Serviceability<br>7- Lean design<br>8- Design for Environment<br>9- Design for Quality<br>10- Design for Production<br>11- Design for Testability<br>12- Others Design for X</td><td><strong>Process aspect</strong><br>1- Decomposition of the problem<br>2- Define specifications<br>3- Solutions generation<br>4- Creating a nomenclature<br>5- Choice of architecture<br>6- Standardization<br>7- Benchmarking<br>8- Documentation for manufacturing<br>9- Final report</td></tr></tbody></table>

The Product aspect includes various types of Design for X. These descriptors are oriented to maximize the requirements while reducing costs [18]. The use of DFX gives pertinent information regarding the characteristics of the developed products as well as the orientation of the design to meet market expectations. Descriptors of environmental axis relate to the motives underlying design projects. For North American companies, the main drivers of investing in design are the evolution of the client's preferences and shrinking of delays in marketing, but one might also mention the increase in capacity and performance of products and technology [1]. Descriptors of human axis include the design team organization (multidisciplinary team), its composition (design team recruitment according to skills and expertise, personality types), the arrangements put in place to promote collaborative work and those set up to ensure the motivation of individuals involved in the design project [19].

## 2.4 Issues for descriptors

Each descriptor is assessed by four questions that correspond to four perspectives (facets). For each facet, the evaluation is based on ordered response scales.
The first facet is related to the use of the descriptor in the company's design practices. This facet called Frequency can specify the "design system" of a company. It has to be noted that this facet is also called Project motives into the environmental dimension. Concerning the Frequency of use and Project motives, the choice of answers is: (0) never, (1) rarely, (2) sometimes, (3) often, and (4) always.

It is essential to know which descriptors are relevant for a company. For this reason, it is asked to evaluate their importance regarding to the project success. For this facet, called Importance for success, the choice of answers is: (0) no significant, (1) a bit important, (2) important, (3) very important, and (4) essential.

The third issue is the company's performance on the descriptors. This is required because a practice can be considered very important as the performance level must be improved. For this facet called Performance, the choice of answer is: (0) does not applied, (1) bad, (2) weak, (3) good, and (4) very good.

Because design activities can be assigned or delegated to external partners while internal expertises may remain essential, the fourth facet evaluates the partners' involvement for each descriptor. For Outsourcing and Partnership the answers choice is similar to that of Importance for success: (0) no significant, (1) a bit important, (2) important, (3) very important, and (4) essential.

## 3 Results

The results presented here relate to those obtained from a manufacturing company operating in the sector of metallic structures. This company was chosen to illustrate the results because it has revealed to use most of the descriptors identified in the survey.

Figure 2 shows the overall positioning of the company on the axes and aspects of the "design system". These results are obtained by averaging the scores assigned to the descriptors of each dimension. Averages are only calculated for the descriptors considered significant to the question assessing Importance for success because usually it was no answers to the other facets when Importance was not significant.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Positioning on Design System</figcaption></figure>
According to the evaluator's perception, the positioning on the "design system" shows that this company performs quite well overall except for the human dimension where the Performance curve is clearly below the Importance for success curve. Figure 2 also shows that descriptors are used less frequently in the human dimension. Finally, management aspect, environmental axis and process aspect represent those where certain descriptors are outsourced or delegated to partners.

Figures 3, 4 and 5 show in details the descriptors assessment of each dimension. In these graphs, the numbers correspond to the descriptors listed in Table 2. The graphs mainly show curves of Performance and of Importance for success.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Positioning on Human axis and Management aspect</figcaption></figure>

For the human dimension, with scores of 4.0, "skills and expertise" and "promoting collaboration" are the most important descriptors in a design project context. The descriptors "multidisciplinary team", "personality types", "promoting motivation" with scores of 2.0 are considered less important. The largest differences between the curves of Performance and of Importance for success represent the "promoting collaboration" and "multidisciplinary team" descriptors. Regarding the Frequency of use, it is found that the design team is never chosen on the basis of "personality types" and "promoting motivation" is never considered. These results are surprising since both descriptors were considered Important to a degree 2.

With increasing design performance as an objective, performance deficits are highlighted by the graphs. These concerns are brought out by the differences found between Performance and Importance for success. The idea here is that for a descriptor with a certain score of Importance (as essential = 4), a score at least equivalent should match at Performance (in this case, very good = 4). If not, it's assumed that the descriptor does not optimally contribute to the success. For two descriptors for which the difference found between Performance and Importance remains the same, priority actions should be taken on the descriptor with the highest Importance score. Following this logic, the first concern in human dimension is to increase performance on "promoting collaboration", the second concern is to increase performance on "multidisciplinary team", the third concern is to increase performance level on "skills and expertise" and the latest is on "promoting motivation".
Concerning the management dimension, only the “PDM” descriptor is not considered significant. The curve of outsourcing shows that “exploring solutions with the customer”, “technology watch” and “external research” are outsourced. The curve of Performance surrounds the curve of Importance for success. This pattern suggests a good command of the practice in this dimension. There is a negative gap between Performance and Importance for “feasibility study” and “exploring solutions with the customer”. Concern on “exploring solutions with the customer” should be prioritized while a lower priority concern relates to “feasibility study”.

Figure 4 shows the results for the Environmental axis and the Product aspect.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Positioning on Environmental axis and Product aspect</figcaption></figure>

It appears that for the environmental axis, the single highlight concerns “time to market”. Regarding the product aspect, “design for serviceability” and “design for testability” are excluded and a concern to increase performance on “design for assembling” emerges.

Figure 5 shows the results for Techno-Scientific axis and Process aspect.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Positioning on Techno-Scientific axis and Process aspect</figcaption></figure>

For techno-scientific axis, only “rapid prototyping” is not considered significant. Increasing the performance for “simulating” is the single concern for this dimension. Regarding the process aspect, “creating a nomenclature” and “benchmarking” are
rejected. To increase the performance level, the priority concern should be on "final report" while a second concern targets "solutions generation".

# 4 Discussion

## 4.1 A promising tool

The results show that almost all descriptors were considered relevant by the company' evaluator and those performance deficits can be highlighted whenever the Performance curve lies below Importance for success curve. By acting wisely on targets, a company should meet its objectives more easily and increase chance to success in its design projects. However the prioritization of actions must be nuanced. It belongs to a company to define why it gets its results and to determine its future actions. It is possible that the measures taken to improve performance within a descriptor increase performance on other descriptors. As such, Figure 1 shows the influences between the dimensions of the "design system", indicating that the descriptors are not totally independent. Furthermore, it is important to note that the tool is based on the evaluator's perception. It could be that diagnosis differs within the same firm depending on who is conducting the evaluation and that an accurate evaluation must consider the views of several actors. The fact that respondents might differ in their evaluation of descriptors may be symptomatic of some problems and may impact on the conduct of company projects.

Companies that participated in the survey validation were not compared. However, the second phase of the research will carry a descriptive and relational mandate. Data collected from a large sample should allow, among others, the identification of the most contributing practices to the success for categories of companies, to establish portraits by sector and to compare the pattern of a company with those of its sector. Finally, we believe that although more improvements are needed, the questionnaire can be used in developing comparative performance indicators.

## 4.2 The improvement of the questionnaire

The respondents suggested potential improvements to the survey. Following their comments, the answer option (does not applied) will be added to the questionnaire for all matters relating to the Outsourcing and Partnership and Importance for success. Considering the numerous variables to assess, to mitigate the confusion effect that may occur when responding to a 45-minutes questionnaire, a new version will be put together to allow answering in several stages. Finally, the ordered answer scales will be shown with their corresponding number.

## 4.3 Conclusion

The results presented show that most descriptors were considered by the companies. These variables are suitable to represent the "design system" and can help in
developing indicators. For a company, the highlights on descriptors that could be
improved may help to target actions and raise global performance. According to
respondent's comments, the questionnaire will be improve before it is used for a
longitudinal research entrusted to an observatory whose mandate will be to measure
and monitor the evolution of design practices in organizations.

References

1. DESIGN EXCHANGE en collaboration avec Manufacturiers et exportateurs du Canada et Industrie Canada: L'ÉTAT de la CONCEPTION DE PRODUITS: LE RAPPORT CANADIEN (2010)

2. Robin V. : Évaluation de la performance des systèmes de conception pour la conduite de l'ingénierie des produits ; prototype logiciel d'aide aux acteurs. Doctoral thesis, Université Bordeaux 1, École doctorales des sciences physiques et de l'ingénieur (2005)

3. Tarondeau J.C. : Stratégie Industrielle. Vuibert, Paris (1996)

4. Sisodia R.S.: Competitive Advantage Through Design. Journal of Business Strategy, vol. 13:6, pp. 33-40 (1992)

5. Gendron S., Brousseau J., Elouafi A., Urli B.: Development of a Tool for Assessing the Design Process: A step toward the creation of a Design Observatory. Proceedings of the 2nd Annual CEEA Conference: Memorial University St. John's, Newfoundland (2011)

6. Eder E.: Integration of theories to assist practice. 5th International Conference on Integrated Design and Manufacturing in Mechanical Engineering, IDMME04, Bath (2004)

7. Laurikkala H., Puustinen E., Pajarre E., Tanskanen K.: Reducing complexity of modelling in large delivery projects. 13th International Conference on Engineering Design, ICED01, Glasgow (2001)

8. Marle F. : Modèles d'informations et méthodes pour aider à la prise de décision en management de projet. Thesis, Ecole Centrale de Paris (2002)

9. Topliceanu G., Robin V., Ispas C., Girard P.: Évaluation de la performance des processus collaboratifs en conception de produits. U.P.B. Sci. Bull., Series D, vol. 72:3, pp. 229-245 (2010)

10. Deneux D.: Méthodes et Modèles pour la conception concourante. Habilitation à Diriger des Recherches, Université de Valenciennes et du Hainaut Cambrésis (2002)

11. Sim S.K. , Duffy A.H.B.: Towards an ontology of generic engineering design activities. Res. Eng. Design, vol. 14, pp. 200–223 (2003)

12. Hubka V.: Principles of engineering design. Butterworth Scientific, London (1982)

13. Pahl G., Beitz W.: Engineering Design: A Systematic Approach. Springer, New York (1996)

14. Pugh S.: Total design, integrated methods for successful product engineering. Addison-Wesley, Wokingham (1991)

15. Suh N.P.: The principles of design. Oxford University Press, New York, (1990)

16. Ullman D.: The Mechanical Design Process. McGraw-Hill, Inc., New York (1992)

17. Ulrich K.T., Eppinger S.D.: Product design and development. McGraw-Hill International editions (2000)

18. Huang G.Q. : Design for X : concurrent engineering imperatives. Springer, New York (1996)

19. Wittorski R.: Analyse du travail et production de compétences collectives. Editions L'Harmattan (1997)