The relation between scientific research and industrial design is fraught with controversy. While the number of academic PhD programs on product design grows because the discipline is in its infancy, there is no consolidated method for systematically approaching the generation of knowledge in this domain. This paper aims to review recently published papers from the four top-ranked journals in engineering design in order to analyse the research methods that are frequently used. The research questions consider the aim and contributions of the papers, as well as which experimental design and which sources of data are being used. Frequency tables show the high variety of approaches and aims of the papers, combining both qualitative and quantitative empirical approaches and analytical methods. Most of the papers focus on methodological concerns or on delving into a particular aspect of the design process. Data collection methods are also diverse without a clear relation between the type of method and the objective or strategy of the research. This paper aims to act as a valuable resource for academics, providing definitions related to research methods and referencing examples, and for researchers, shedding light on some of the weaknesses and challenges for current research in the domain of engineering design. Keywords Research methodologies on product design; Engineering design and evaluation 1. Introduction Design as a discipline does not yet have a consolidated research tradition. In 2009, Blessing and Chakrabarti proposed the DRM (Design Research Methodology) motivated by “ frustration about the lack of a common terminology, benchmarked research method and a common research methodology in design” (Blessing & Chakrabarti, 2009, pg 1) . However, the term ‘design research’ does not help to make things clearer, because it is frequently used to refer to approaches and methods for researchers and not for designers (Cash et al., 2016; Creswell & Creswell, 2017). Herbert A. Simon, in his dissertation about The Science of Design, defined design as a search for an optimum solution in a space of alternatives that take into account the specifications and restrictions of a given problem (Simon, 1996). Design could be considered as a research activity, as it requires information to be gathered and proposals to be evaluated with rigour, resulting in innovative and successful products. Nevertheless, to be considered as research, the design activities must generate knowledge. Since the work of Blessing and Chakrabarti, there have been some relevant proposals that have shed light on the activities designers should carry out in order for their work to be considered as research. Koskinen et al., (2011) proposed the term ‘constructive design research’ and presented alternatives to integrate research within the practice of design. Rodgers & Milton (2013) presented a comprehensive catalogue of research methods involved in the design process, pointing out the phases of product design in which they could be applied. Joost et al. (2016) used the term ‘design as research’ in a volume that compiled discourses of experts about questions on design research and its relationship with other disciplines. Vaughan (2017) presented a survey that collected different points of view related to doctoral education in the opinions of design graduates about practice-based research design. Redström (2017) presented an essay about how to develop theory -knowledge- by practice, experimentation and making -design-. These works are a multi-faceted compendium of practical experiences and visions of experts on how to perform activities related to research in the domain of design. All of them agree on the relevance of this theme due to the increasing number of PhD programs on design. In this paper, we present an alternative approach to shed light on the relations between research and design: instead of collecting descuder@infor.uva.es 1 Escuela de Ingenierías Industriales 2 Escuela de Ingeniería Informática 3 Escuela Técnica Superior de Arquitectura Universidad de Valladolid, Plaza San Cruz, 47002 Valladolid, Spain Thanks to: Proyectos de Innovación Docente 2020/21 ID 038 Universidad de Valladolid 2 PID-UVA 038 2021 personal visions of experts, we summarize and classify research papers in terms of methods, approaches and research activities performed by the authors. To this end, we have carried out a systematic review of the literature on research in product design. The overarching research question that drives the review is: What is the current landscape of research methods in product design? There are many possible ways to analyse, categorize or classify research works because there are many dimensions of analysis. Blessing & Chakrabarti, (2009) identified the following ones: Paradigm, that includes empiricism (Solomon, 2007) (Randolph, 2003) and ethno-methodology (Atkinson, 1988); aim, research questions and hypotheses; nature of the study, including observational vs interventional (Thiese, 2014), comparative vs non-comparative; methodologies, theories, views and assumptions (Kothari, 2004); units of analysis; data collection methods including recordings, interview, questionnaires (De Leeuw, 2008); role of the researcher (Fink, 2000); time constraints, duration and continuation of the research process; observed processes including layout drawing, prototype or product; setting referring to laboratory or field research (Paluck & Cialdini, 2014); tasks including type and complexity and nature; number of cases, case size and participants (Diggle et al., 2011); object of analysis distinguishing objects, companies, projects, documents… coding and analysis, analysis and verification methods (Brewer & Crano, 2000); or findings, that is, statement models or conclusions resulting from the study. In this paper, we aim to identify the main goals pursued by researchers in the field, which research approaches they follow, and which data sources and analytical methods are most (and least) commonly used. Our analysis pivots around the division between empirical qualitative and quantitative research and analytical studies. From this germinal division, data-collection methods, strategies and contributions of the studies are reported in cross-analysis tables. The review points to potential gaps and challenges in the development of this relatively young field of research. The structure of the document is the following: First, we present the review method and the categories used to classify the papers. We then present the quantitative results of the number of papers in each of the categories and the cross relations of the different classes, shedding light on the relative weight of each of the qualitative and quantitative approaches and the most frequent data-collection methods used. Next, we discuss the usefulness of the obtained results for academics and professionals interested in research design and the paper ends with the conclusions. Complementary material is provided with a brief description of each of the analysed papers. 2. Method We follow Kitchenham (2004) as a guideline for performing the systematic review. The nature of the research question did not suit a usual search in the databases, as we were interested in analysing the approaches to research published in the field of product design. For this reason, we focused on identifying papers published in relevant journals in the field. The data sources are journal papers in the field of industrial design. There are 100 scientific journals indexed in the Journal of Citation Reports 2020 with the term “Design” in its title, relating to different categories from Electronics and Computer Science to Nuclear Physics or Chemistry. We selected Design Studies (DS), the International Journal of Design (IJD), the Journal of Engineering Design (JED) and Research on Engineering Design (RED) as the journals that satisfy the requirement of being in the category denominated as Engineering Multidisciplinary (listed in Table 1). We excluded the International Journal of Technology and Design Education as it focuses on Education, not research. Papers in special issues were excluded to avoid bias in the analysis. We collected 13 papers from each journal, as 13 is the number of papers available in one of the journals under analysis (IJD) and we chose to use the same number of papers per journal to avoid bias in the study. For the journals with more than 13 papers in the period of analysis, random selection was applied. We focused on the papers published in 2019, which was the most recent available year when this work was started. Every paper was reviewed by at least two of the authors of the Table 1 Journals that are the focus of interest in the study. ENG MULT is ENGINEERING, MULTIDISCIPLINARY – SCIE, ENG MANU is ENGINEERING, MANUFACTURING – SCIE, SS INTER is SOCIAL SCIENCES, INTERDISCIPLINARY – SSCI, ENG IND is ENGINEERING, INDUSTRIAL – SCIE. In the cells, A/B figures mean number of reviewed (A) papers versus total number of papers (B). Special issues are underlined. 3 PID-UVA 038 2021 present work on random assignments. Revisions were done in parallel, and disagreements were resolved in joint meetings. After the round of revisions, the first author of this paper used the data from the analysis to build the final tables and present the results. The overarching research question posed in this review was: RQ: What is the current landscape of research methods in product and engineering design? To answer this research question, we focused on three more concrete aspects, with three sub-questions: RQ1: What are the research goals pursued by the analysed works? RQ2: What are the main experimental approaches found in the reviewed papers? RQ3: What data collection methods are employed in the reviewed works? RQ4: Which instruments are normally used to collect these data? The following aspects were annotated: aims or contribution of the research, experimental approaches or method, and data collection methods and instruments. Next, we present each of these topics, explaining the respective classification categories used. 2.1 Aims and contributions Concerning the aims/contributions of the research (RQ1), we started from an empty list of categories which was enriched as the number of reviewed papers increased. Finally, the following categories were selected: To study or propose a methodology which focuses on papers whose main objective is to study an existing design methodology by analysing its validity in works that propose a new design methodology or that develop a part of it more deeply. To delve into a given aspect of design, which includes papers that focus on exploring an aspect of a design (team communication, sketching, generation of ideas, materials ...) or that explore one area of design which is recognized as challenging (social design, inclusive design, ecological design ...). To design, develop, or test a specific product , which includes those papers that set out the process of creation or development of a specific product or a group of them. Some of these works describe the overall process of creating a product, and others focus on a specific phase of its development (research, ideation, testing and validation). To make recommendations or propose guidelines, which include articles whose main aim is to systematize the results of their research in order to provide advice, either at a methodological level or in the design of new products. Proposing a theory includes those articles that use logical reasoning or mental operations, such as imagination, intuition, abstraction, and deduction, with the aim of enunciating concepts or creating models, explanations or theories about the phenomena under study. Proposing a framework of analysis or a taxonomy that permits concepts or objects to be classified into categories. Papers could be labelled in more than one category. This could be the case of a paper that aims to develop a specific product and ends by proposing guidelines. 2.2 Research approach Concerning the experimental design, we propose the use of the classical distinction between quantitative, qualitative and mixed research, following Creswell (2009). To complement this classification, we have also considered the analytical research method, as specified by (Adrion, 1993) cited in (Glass, 1995). Quantitative empirical studies are those in which a set of numeric values are obtained to analyse the topic of interest (Cash et al., 2016). Qualitative empirical studies are those in which the focus is not on numerical data, but on exploratory research to understand the topic of interest in depth. Mixed-methods studies are those that combine both quantitative and qualitative approaches. Analytical studies are those that use an analytical research method, propose a formal theory or set of axioms, develop a theory, derive results, and, if possible, compare these results with empirical observations. With respect to quantitative empirical studies, we subcategorize them into experiments, quasi-experiments and non-experiments, depending on the subjects of interest of the experimental group with those of an equivalent control group: Experiments : by carrying out a random selection of subjects. Quasi-experiments: with less control of the selection of subjects. Non-experiments: with similar but not equivalent experimental and control groups. When a known qualitative strategy of inquiry is used, it is also annotated. According to the definition proposed by Creswell (2009), strategies of inquiry are types of methods, designs or models that provide specific direction for procedures in a research design. Ethnographic studies are field-oriented studies where the researcher stays on site for a considerable amount of time to analyse practices and behaviours of groups, by observing, interviewing and (sometimes) participating in the process under analysis. Very popular in social sciences, it is also used in architecture and design (Cranz, 2016). 4 PID-UVA 038 2021 In phenomenological research , the researcher identifies the essence of human experiences about a phenomenon as described by participants, while the researcher sets aside his or her own perspective (Wilson, 2015). Grounded theory is a strategy of inquiry in which the researcher derives a general theory grounded in the views of participants, involving the use of multiple stages of data collection (Jørgensen, 2001). Hermeneutics inquiry focuses on disclosing how participants’ interpretations of a phenomenon determine the way they live in the world (Stigliano, 1989). This technique is popular in design and architecture (Perez- Gomez, 1999) . Case study research is an empirical strategy of inquiry that investigates a contemporary phenomenon within its real- life context, especially when the boundaries between phenomenon and context are not clearly evident (Yin, 2003). In case studies, single individuals, groups, programmes, activity, processes, or events are analysed in depth with the goal of explaining or describing them. Action research is an approach to introduce a change through cycles of understanding and change with the involvement of the stakeholders. Very popular in social sciences (Clark et al., 2020; Stringer, 2008), it is also proposed for architecture and design (Herr, 2015; Swann, 2002). This method has been included as part of research- through-design, practice-based-design research or research-by-design proposals, as previously done by other authors (Kennedy-Clark, 2013; Motta-Filho, 2021), but we prefer to include it as a new perspective. Practice-based design research and research-through or by design have recently been presented as alternative approaches to generate knowledge by designers and practitioners of product design (Redström, 2017; Vaughan, 2017). Starting from Herbert Read's book, Education through Art (1948), in 1993 Christopher Frayling established three categories with the intention of systematizing research related to art and design: research for art and design, research into art and design, and research through art and design (Frayling, 1994). Findeli et al. (2008), aiming to conceptually and methodologically clarify the concepts, referring specifically to design, delved into the for-about-through model, which, as stated by (Herrera Batista, 2010), has been gaining popularity in recent years within the academic community. Research for design is normally carried out for the development of a design project, and the final product, therefore, is an object or artifact. This research does not have a scientific purpose and is often not intended to be published or discussed, but rather uses tacit knowledge. Its main objective is to ensure that the different conditioning factors of design (aesthetical, psychological, ergonomic, technological, etc.) have been considered appropriately in the design process. Research about design or research into design is one in which design is taken as a research subject. It is normally carried out from other disciplines that are not properly design, such as anthropology, history, psychology, semiotics, etc. It usually deals with the design of its objects, processes, actors, meaning and importance for society, business, culture, etc. In this approach, therefore, we find a clear separation between design practice and design research (Findeli et al., 2008). Finally, Research through design, or research by design, “focuses on the role of a product prototype as a research instrument” - (Keyson & Bruns, 2009) cited by (Findeli et al., 2008) - . In this approach, the subject and object of the investigation is the design itself. Therefore, it can be understood as an integration of research for design and research about design, taking the interest in improving the practice of design from the first, and the methodological rigor that can be carried out from other disciplines from the second (Findeli et al., 2008). Case study is generally used for exploratory research or for pre-testing some research hypotheses (Blessing & Chakrabarti, 2009). Action research requires a high degree of flexibility and is usually qualitative, data driven, participatory, and makes use of multiple data sources. Case study and action research also appear in the following criteria of classification, following the proposal of (Blessing & Chakrabarti, 2009) referring to data-collection techniques. 2.3 Data-collection techniques In this subsection, we present the list of data-collection techniques we have annotated. We follow the classification proposed by (Blessing & Chakrabarti, 2009), excluding experiments, case study and action research that we prefer to include in the list of inquiry research strategies presented in the previous subsection. Observation is a technique in which the researcher records, in real time, what is happening, either by hand, recording it or using measuring equipment. As (Blessing & Chakrabarti, 2009) explain: ‘The quality of observational data is highly dependent on the skill, training and competency of the observer’ (Blessing and Chakrabarti 2009: 257). Observations are the main source of data in ethnographic studies (see section 2.1), but this strategy is also commonly used in social sciences (Creswell, 2009) and in visual design (Goodwin, 2000), architecture (Cuff, 1992) and design (Wasson, 2000). Simultaneous verbalization refers to the situation in which the participants speak aloud while using a system, with the aim of providing information about the cognitive behaviour of the participants, which may not be obtained through normal observation. Often used to analyse problem-solving behaviour, its most important feature is the real-time aspect. Simultaneous verbalization sessions usually last a few hours and never more than a day, due to the effort required by both the participants and the 5 PID-UVA 038 2021 researchers in their corresponding analysis. Although audio recordings are sometimes used to record simultaneous verbalization, they are understood as inappropriate for a process such as design, which usually involves drawings and gestures, so video recordings are considered more appropriate. Collecting technical documents consists of obtaining technical documents related to a particular project, topic or product, from various sources. Analysis of these documents is often used early in a research project in order to understand the organisation, the background of the project and the experience of the designers. It is commonly employed in most observational studies. However, if it is used as a single source of information, it can result in such limitations as the usual lack of data on the context in which the documents were created, and the reason for their content. It is, therefore, convenient to complement them with other methods such as interviews. Collecting physical objects involves mock-ups, prototypes and other physical models that may be relevant for developing a product or testing it. The model or prototype could refer to a part of the product or the whole product. For traditional engineering research, which focuses, for example, on the analysis of product behaviour, the products are the main source of data (Blessing 2009: 269). In our review, we consider those works that start collecting different objects to carry out a study on their usefulness, or on the behaviour of users, for example. The object is a general term that can refer both to drawings and physical objects. Among the former, we find all those sketches, drawings and diagrams that have emerged throughout the conception of a product or its development, or throughout a research process, which could yield important information to organise ideas and draw conclusions. Questionnaires are used to collect people´s thoughts or opinions about a certain product, process or method. A priori, they seem easier to use than real-time methods, such as observation or simultaneous verbalization, and they are useful to obtain data from a greater number of cases. However, some of its disadvantages, such as the time required by the participants and the potential bias of the results, must also be taken into account. Interviews have the same purpose as the questionnaires, but are carried out face to face. Sometimes they are not carried out individually, but using a group dynamic known as focus group: a group interview that mixes aspects of interviews and observations, as it provides information from the study of the interactions between participants. Focus groups can provide richer information than interviews, but they can have a negative effect on the contribution of specific participants. 2.4 Instruments for the collection of data Data collection methods are supported by instrumentation. This section describes the categories we found to respond to RQ4. Independently of the strategy of inquiry applied, there are a number of instruments that are used to keep records of the observations. These recordings are important to keep evidence and to enable the reproducibility of the analysis. We annotate the classical audio, video and image recordings and the more recent technique of eye tracking (Bergstrom & Schall, 2014). In experiments and case studies, we are also interested in physical measurements that are used to objectify observations. When questionnaires and/or interviews are the data- collection techniques, we annotate who is the attendee, distinguishing between stakeholders , users of products or participants (observed people) in the research and experts or designers. We also found it relevant to annotate when the study uses workshops as a means to obtain information. The last topic of interest that has been annotated is the fact that the research work uses simulation algorithms or tools as a source of information. We use this tag when the simulation tools are a fundamental part of the research, as it provides the information analysed (Behera et al., 2019) in the paper, or because the tool or the algorithm itself is the main contribution (Mathias et al., 2019). 3. Results 3.1 Aims and contributions Table 2 shows the labels assigned to each of the papers analysed. This section summarises the results related to RQ1. As shown in Table 2, most of the works focus on methodologies or on the analysis of a specific aspect of the design processes. The presentation of a product and the building up of knowledge with taxonomies, guidelines, theories, or reviews, are exceptions. Three papers propose a theory: (Comi et al., 2019) presents the concept of shared professional vision; (Benavides & Lara- Rapp, 2019) presents the principle of weaker dependencies in axiomatic design; and (Martinec et al., 2019) introduces the state-transition model (synthesis, analysis, evaluation) in conceptual design. Five works focus on the evaluation of a specific product: a software product in (Takahashi et al., 2018) and (Belkadi et al., 2019); or physical objects in the case of (Roesler et al., 2019), (Hyysalo et al., 2019b) and (McKinnon & Sade, 2019). In the last two cases, the product is involved in a research-through-design process. 6 PID-UVA 038 2021 Concerning the works related to methodologies, we find papers that propose a method based on analytical methods or algorithmic solutions such as those related to axiomatic design (B. Chen et al., 2019; R. Chen et al., 2019); those related to such methods as research-through-design, where the importance of the method followed is prominent in the study (Hanrahan et al., 2019; Hyysalo et al., 2019b; McKinnon & Sade, 2019; Tsai & Hoven, 2018); or methodologies for product development such as (Daalhuizen et al., 2019), with emphasis on different aspects such as work in groups (Gyory et al., 2019) or sustainability (Santolaya et al., 2019). Six papers present frameworks of analysis or classifications with different purposes. (Bresciani, 2019) for classifying visualization dimensions, (McDonald & Michela, 2019) to classify moral goods, (Roy & Warren, 2019) for card sets, (Park-Lee & Person, 2018) identify three practices on briefing, (Vegt et al., 2019) deduce 3 types of invasiveness evoked by the rules in gamified brainstorming, and (Valverde et al., 2019) classify the type of feedback in automotive push buttons. Proposing recommendations is a common result in the analysed research papers, including a variety of themes such as: recommendations on the use of guidelines by new designers (Reimlinger et al., 2019); the use of specific materials (Genç et al., 2018; Pedgley et al., 2018); how to orient future studies on the use of mobile technology by elderly people (Q. Li & Luximon, 2018), or about design and poverty (Jagtap, 2019) or ethnographic studies in developing countries (Wood & Mattson, 2019); appliance design (Selvefors et al., 2018); use of games in brainstorming (Vegt et al., 2019); or specifying requirements (Morkos et al., 2019). The most frequent type of works delve into a particular aspect of product design such as sketching (Self, 2019; Sung et al., 2019), prototyping (Mathias et al., 2019), material (Aktas & Mäkelä, 2019; Barati et al., 2019; Pedgley et al., 2018), interaction (Hobye & Fagerberg Ranten, 2019; Valverde et al., 2019), briefing (Park-Lee & Person, 2018), working in groups (Graeff et al., 2019), or iterations and testing (Piccolo et al., 2019). 7 PID-UVA 038 2021 . Table 2 Classification of the different papers analysed in the categories of aim/results, approach and research method, data-collection technique, instruments and human based information. The papers are assorted by journal. 8 PID-UVA 038 2021 3.2 Strategies of inquiry and methodologies This section summarises the results related to RQ2: qualitative approaches are a majority, but the number of quantitative or mixed-methods studies is also relevant. Other approaches, such as the use of analytical methods, are less frequent. Table 3 shows that, when the goal of the paper is related to proposing or studying a methodology (first column in Table 3), the percentage of pure quantitative papers is lower than in the rest of the cases. Regarding whether there is a tendency towards any methodology depending on the journal; Table 2 shows that the Journal on Engineering Design seems to focus more than the other journals on non- qualitative strategies of inquiry. When quantitative methods are used, experiments are more frequent than quasi-experiments and non-experiments (13 out of the 15 quantitative studies present an experiment). We found 26 experimental studies, with 5 quasi-experiments (Saliminamin et al., 2019a; Santolaya et al., 2019; Self, 2019; Sung et al., 2019; Vegt et al., 2019) and 4 non-experiments (Morkos et al., 2019; Piccolo et al., 2019; Roesler et al., 2019; Selvefors et al., 2018). The use of case studies is pervasive in qualitative research (more than half the studies that classified as qualitative base the research on a case study). Furthermore, many quantitative studies support results from case studies; for example, some analytical studies in which case studies are used as proof of concept of the proposed models (R. Chen et al., 2019; Y. Li et al., 2019; Zhang & Thomson, 2019). Nevertheless, other qualitative methods, such as ethnography, action research and phenomenological studies, are also used. The use of specific methods related to design is scarce (the discussion about this concern is dealt with in detail below). Ethnography is used in three cases (Comi et al., 2019), (Roesler et al., 2019) and (Van der Linden et al., 2019b) – also the annotation as observation in the tables – and one more paper uses ethnography as the study focus (Wood & Mattson, 2019). Action research is used by Pakkanen et al. (2019) to investigate, in combination with case studies, modular systems in industrial environments. The work of (Bresciani, 2019) could be considered an action research study with the goal of building a grounded theory evaluation technique for visual thinking. McKinnon & Sade (2019) align their work in the field of research through design using a set of gadgets to obtain information about environmental home good practices. Research through design is also used by (Genç et al., 2018) to explore new materials and (Tsai & Hoven, 2018) to explore user experience. (Hyysalo et al., 2019a) present the evaluation of a panel following the principles of research through design. Close to this method is that presented by (Barati et al., 2019), who complement their study with workshops where a group of students explores their proposals. 3.3 Data-collection techniques Results regarding RQ3 are summarised in this section. Table 3 shows what the main sources of information are for the Table 3 Number of works in each (sub)category per research approach 9 PID-UVA 038 2021 different types of studies. None of the instruments identified seem to be dominant in the papers studied. Technical documents of a diverse nature are the main source of information used (in 17 out of the 52 papers analysed). Interviewing is also frequent (19 times). Expert and user opinions are both used as sources of information, but neither is a majority (17 and 18 papers, respectively). Observation is mostly used in qualitative studies, where almost half use this technique. Concerning quantitative studies, apart from measurements, expert opinions appear as a frequent resource. This is because it is common to collect the opinions of experts in questionnaires or in evaluation templates that convert opinions into numeric values. Verbalization is used in (Martinec et al., 2019) and (Gyory et al., 2019) for team work analysis and in (Khalaj & Pedgley, 2019), where designers and users had to verbalize impressions. Objects are collected as a data source in a relevant number of studies. Some are the results of students’ work as in (Gralla et al., 2019); brainstorming outputs (Vegt et al., 2019); prototypes (Barati et al., 2019; Feijs & Toeters, 2018a), or commercial products (Roy & Warren, 2019). Sketches are the type of object analysed in (Comi et al., 2019; Genç et al., 2018; Goucher-Lambert & Cagan, 2019; Gyory et al., 2019; Martinec et al., 2019); while for (Sung et al., 2019) and (Q. Li & Luximon, 2018), sketches are the main concern of the research. Questionnaires are less frequently used, and when this happens, they are designed ad-hoc for each study. No standardized questionnaires have been found in the reviewed works. Questionnaires take different formats: Amazon Mechanical Turk is used once (Goucher-Lambert & Cagan, 2019); a Likert scale tool evaluation (Graeff et al., 2019); binary and open questions (Pakkanen et al., 2019); ranking of p