iDesignGPT enhances conceptual design via large language model agentic workflows

Domain: neuroscience
Published: 2026-01-24
Editorial outcome: major_revision

Abstract
Conceptual engineering system design faces challenges from traditional methods and emerging AI tools to fully address its inherently complex, dynamic, and creativity-driven demands. iDesignGPT is a framework that integrates large language models with established design methodologies to enable dynamic multi-agent collaboration for problem refinement, information gathering, design space exploration, and evaluation. By incorporating design metrics such as coverage, diversity, and novelty, iDesignGPT provides quantitative insights for early-stage conceptual design. Performance evaluations across six public design challenges show that iDesignGPT achieves competitive novelty and consistently higher originality and modularity than GPT-4o zero-shot, GPT-4o chain-of-thought and Deepseek-r1, based on metrics and expert assessments. Two controlled user studies show positive reception across profiles and, for novice designers, lower mental demand than human-only design and clearer design flow with iDesignGPT. These results establish iDesignGPT as a practical framework for integrating language-model agents with established engineering design methods, enabling metrics-driven support for conceptual design by both expert and novice designers.

Introduction
Engineering system design plays a pivotal role in technological advancement, serving as the bridge between scientific research and practical application1,2. Conceptual design underpins the earliest phase of development, where fundamental decisions on system functionality, architecture, and feasibility are made3,4. Critically, the “80/20 rule” indicates that up to 80% of a product’s lifecycle cost is determined during conceptual design, with NASA studies showing that poor early decisions led to 73% of total corrective costs in rocket engine programs5,6. Conceptual design demands high levels of creativity, structured reasoning, and the ability to navigate uncertainty under incomplete information7–9.

Additionally, human-related factors, such as unclear requirements10, tight deadlines11, and the varying expertise levels of designers12,13, complicate early-stage conceptual design process, echoing recent discussions on democratizing design education and established engineering design methods15, including approaches such as Quality Function Deployment (QFD)16 and the Theory of Inventive Problem Solving (TRIZ)17, were developed for narrower industrial contexts and face limitations in evolving design spaces as requirements and constraints are revised over time during early-stage design. Thus, these methods often require high domain-specific expertise and struggle to scale to multifaceted, interdisciplinary problems18. Meanwhile, CAD, CAE, and CAM tools mainly support detailed design and manufacturing, leaving conceptual design critically under-supported19. Even in domains such as advanced air mobility, where recent tools improve early-stage modeling and trade studies20,21, exploration remains largely parameter driven and grounded in prior data, limiting discovery of fundamentally novel concepts. Consequently, engineers are still required to manually reconcile diverse multimodal early-stage inputs such as 3D/2D sketching, textual specifications22 and related sources of information. These challenges reduce the quality and potential of conceptual design outcomes, limit the scalability of solutions, and hinder innovation. These issues underscore the need for tools that can operationalize established design processes, methods, and multimodal data into engineering applications, while lowering the expertise barrier, simplifying execution, and dynamically adapting to complex, evolving environments, as illustrated in Fig. 1a.

Recent advancements in large language models (LLMs), such as GPT-4o23 and the newer reasoning model Deepseek-r124, have significantly enhanced reasoning capabilities, enabling them to outperform human experts in specific benchmarks25,26. Beyond these intrinsic strengths, LLMs can serve as the cognitive core of agentic systems that integrate memory, planning, toolkits, and executable workflows27. This integration has demonstrated impact in scientific research28–30 (e.g., material design31, chemical synthesis32) and other domains33. However, its transfer to conceptual engineering system design tasks requires specialized adaptation34,35, which involves definitive evaluation criteria36, vast and unstructured solution spaces37,38, and demands creativity-driven exploration39–41.

Compounding these challenges, as highlighted in Fig. 1b, LLMs lack domain-specific knowledge in engineering design, which prevents them from scientifically guiding the design process or effectively addressing complex engineering objectives. Current tools, such as AutoTriz42 or GPT-powered design assistants43–47, support specific steps like concept generation but are heavily reliant on user input quality, including prompt engineering expertise, which often constrains less experienced users. Moreover, LLMs are prone to hallucinations48,49, generating plausible but incorrect outputs, and frequently misinterpret user intentions50.

To bridge these gaps, we propose iDesignGPT, a framework that harmonizes established design processes, methods, and multimodal data from design studies with LLM-driven agentic workflows51, as depicted in Fig. 1c. iDesignGPT customizes workflows with design processes, including problem definition, information gathering, concept generation, and evaluation. The framework features two operational modes: Copilot mode, enabling natural language-based goal clarification and requirement refinement to address misinterpreted intentions, and Agent mode, where specialized agents automate design tasks such as QFD analysis and idea generation by breaking down complex design methods into manageable sub-tasks. Agents collaborate through predefined flows, perform iterative checks to mitigate hallucination issues, and integrate information from multiple sources.

Quantitative evaluations demonstrate that iDesignGPT systematically refines requirements, expands design spaces, and generates competitive design solutions with improved coverage, diversity, and novelty. Comparative studies against baseline LLM workflows (GPT-4o zero-shot, GPT-4o chain-of-thought, and Deepseek-r1) highlight consistent advantages in originality, modularity, and rationality, supported by both objective measures and expert-based assessments. Two controlled user studies show positive reception across diverse profiles and, for novice designers, lower mental demand than human-only design and clearer design flow with iDesignGPT. Collectively, these results suggest iDesignGPT lowers the expertise barrier and advances the practical integration of AI-assisted conceptual engineering system design.

Results
Core components of iDesignGPT
iDesignGPT, built on the fastGPT platform (Fig. 2), integrates four core components: a. Agents, b. Toolkit, c. Knowledge Base, and d. Web front-end. Designers interact with the system via the Web front-end, submitting requests to initiate the design process. The system utilizes LLM-driven Agents to navigate four key stages: I. problem definition, II. information gathering, III. concept generation, and IV. concept evaluation. During these stages, the system dynamically identifies and refines constraints and integrates diverse design methods to generate innovative solutions. At each stage, an Agent cluster collaborates based on a predefined process. Powered by LLM’s intelligence, the Agents execute tasks by querying the Knowledge Base for local and online access to relevant domain knowledge and by leveraging the Toolkit to address complex tasks, ultimately facilitating the generation of design solutions. This structured workflow operationalizes key design dynamics, including knowledge flow and integration, through LLM-powered agents.

Case study with GoAERO design challenge
To evaluate iDesignGPT’s performance in engineering design challenges and illustrate its interactive usage, we selected the GoAERO Design Challenge (https://www.herox.com/goaero). Sponsored by Boeing and other leading aerospace companies, this initiative seeks groundbreaking solutions for a compact, autonomy-enabled rescue aircraft that is both safe and easy to operate.

iDesignGPT’s design process (Fig. 3) mirrors the conceptual design phase of engineering development procedures52, where user needs are translated into an early-stage Product Design Specification (PDS)53 and a corresponding conceptual design scheme. Initially, the system enters the Copilot mode, involving users in natural language dialogue to clarify design requirements and constraints. By supporting multimodal inputs (e.g., documents and images), iDesignGPT enables refinement of the problem definition, incorporating user feedback to regenerate and prioritize requirements based on their intent. Upon transitioning to the Agent mode, the Analyst agent cluster employed three parallel requirement analysis methods: Maslow’s hierarchy of needs54, multidimensional (Performance, Time, Cost, Quality) analysis, and KANO analysis55. The integrated results produced a comprehensive customer requirements list, with dynamically assigned weights reflecting both user-expressed priorities and system-inferred contextual importance. These weighted priorities directly informed the development of robust evaluation metrics. Subsequently, the system built a QFD15 analysis, mapping customer requirements to engineering characteristics in a matrix. This process quantified the priorities of engineering features and updated evaluation criteria, providing actionable guidance for addressing complex design tasks.

To gather information, the Information Officer agent cluster retrieved domain-specific knowledge from multiple sources, including Google Patents (Patent), Web of Science and Scopus (Articles), and Red Dot Award-winning design repositories (Projects), while additionally recommending targeted journals classified under JCR categories to guide further exploration. Retrieved information was structured into a dedicated literature retrieval report provided to the user for reference. This report informed the construction of the “2. Market Identification” section within the evolving PDS, serving to contextualize external market and technological trends without directly embedding external content into the final conceptual designs, thereby supporting the originality of subsequent outputs. This ensured the integration of high-quality and relevant external information.

During the concept generation stage, the Innovator agent cluster utilized functional decomposition56 validated by autonomous agents to address the hallucination problem, ensuring logical consistency. Designers reviewed and approved agent-validated results before generating functional decomposition graphs that captured material, energy, and information flows. Advanced concept generation methods were then applied sequentially, including morphological analysis57, biomimetic design58, brainstorming59, SCAMPER60, and TRIZ16,61 analysis, which produced modular solutions, with each module presenting three candidates enriched by optimization insights (see Supplementary Note S1 for detailed generation logic and representative outputs from each concept method).

Finally, in the evaluation stage, the Evaluator agent cluster assessed candidate solutions against predefined metrics. The highest-scoring design was selected and summarized in a PDS, including schematic diagrams generated through AI-based visualization (see Supplementary Note S2 for detailed conceptual design scheme and its PDS). To further evaluate the quality of the final output, the conceptual design proposed by iDesignGPT was compared against 22 Phase 1 awarded designs from the GoAERO challenge, enabling a direct benchmark against expert-authored solutions.

Quantitative insights into case study
A quantitative approach was adopted to evaluate iDesignGPT’s performance in the GoAERO challenge, focusing on metrics across different task phases. During the Define Problem stage (Fig. 4a), iDesignGPT initially identified 11 customer requirements, of which 4 were omitted during processing. This omission reflects not only attention-related limitations62, but also the model’s abstraction of task-specific items. For example, A5 (“transport of manikin”) pertains to a competition test case rather than a general design need, while A4, A6, and A7 were likely embedded into broader requirements such as autonomy, modularity, or safety instead of being retained as standalone items. Meanwhile, 10 new requirements were inferred based on frameworks, including Maslow’s hierarchy of needs, multidimensional, and KANO analysis. In total, 17 refined requirements were established, reflecting the model’s ability to both abstract and expand the initial problem space. A complete mapping between the initial task brief and the final requirement set is summarized in Supplementary Tables S1 and S2, detailing both retained/discarded items and the requirement analysis methods origins of newly inferred requirements.

In the Gather Information stage, iDesignGPT demonstrated optimized search strategies tailored to different sources (Fig. 4b). For patents, iDesignGPT employed iterative keyword-based searches. Article searches were conducted with Web of Science-compatible logic, ensuring a balance of precision and scope. Project searches utilized RAG for local database retrieval, generating 20 exhaustive search phrases. Topic modeling using Latent Dirichlet Allocation63 (Fig. 4c) and Principal Component Analysis (PCA)64 (Fig. 4d) clustered the retrieved terms into three primary themes, demonstrating the strategy’s alignment with domain-specific search goals. Topic 1 (62.85%) centered around design subjects such as “aircraft” and “UAV”, Topic 2 (23.73%) emphasized application scenarios like “rescue” and “humanitarian operations”, and Topic 3 (13.42%) highlighted technical details such as “autonomous” and “occupant”. These thematic distributions illustrate iDesignGPT’s capability to mimic human-like domain comprehension while ensuring balanced retrieval.

The Concept Generation stage leveraged semantic analysis with Uniform Manifold Approximation and Projection (UMAP)65 to visualize the evolution of design spaces across methods (Fig. 4e). In the upper panel of Fig. 4f, coverage and diversity metrics were introduced to provide a comprehensive view of the extent and richness of the design space. Coverage, represented by the convex hull area of the design space, quantifies the range of explored design solutions, while diversity, measured by the mean pairwise distance between solutions, reflects their variability. Early-stage methods such as Biomimetic Design and Brainstorm significantly expanded the design coverage, as evidenced by a convex hull area increase from 15.75 to 17.54 (Δ+11.4%). Simultaneously, diversity within modules rose from 0.92 to 1.09 (Δ+18.5%) during these stages, indicating that these methods fostered greater exploration and variability. However, during later stages, methods such as SCAMPER and TRIZ, which focused on refining existing solutions, resulted in a slight decline in diversity (to 1.06), suggesting a shift towards optimization rather than exploration. The calculation methodology for coverage and diversity is detailed in Supplementary Note S2. In the lower panel of Fig. 4f, the novelty of design solutions is evaluated through semantic vector analysis, which retained detailed characteristics without dimensionality reduction. Average Novelty initially increased from 0.50 to 0.62 during early-stage methods (e.g., Brainstorm and Biomimetic Design). Subsequently, a slight decline (0.62 to 0.60) occurred during later-stage methods (e.g., SCAMPER and TRIZ), further corroborating the observed transition from exploration to refinement in diversity. Max Novelty, however, remained stable at approximately 1.06 after the Brainstorm stage, reflecting the task’s inherent constraints on extreme innovation. The detailed methodology for novelty calculations is provided in Supplementary Note S4. These findings collectively highlight how iDesignGPT, supported by diverse design methods, enhances creativity during the Concept Generation stage.

During the Evaluate Concepts stage, iDesignGPT demonstrated reproducibility. In six independent trials of selecting the optimal solution among three candidate solutions, each constructed by selecting one module-level concept per functional module, Plan 1 consistently scored the highest (Fig. 4g). Statistical analysis confirmed significant differentiation among the three plans (p<0.05). This consistent performance highlights the system’s ability to apply predefined criteria in a stable and objective manner across evaluation runs. To further benchmark iDesignGPT’s solution quality, the selected Plan 1 was compared against 22 awarded design submissions from the GoAERO competition. Based on the Customer Requirement Satisfaction Score, a core evaluation criterion developed from the official guidelines, iDesignGPT’s solution ranked within the top 25% (Fig. 4h), indicating practical competitiveness in real-world design contexts. Full comparison data, including requirement-level satisfaction scores for each solution across all customer requirements, are provided in Supplementary Table S3. A structural overview of iDesignGPT’s design configuration within the broader V/STOL landscape is visualized in Supplementary Fig. S2.

Performance evaluation
To assess the design capabilities of iDesignGPT in comparison to established workflows, we adopted a dual-evaluation strategy combining objective metrics and expert-based assessments. The baseline systems included GPT-4o in both zero-shot and chain-of-thought (COT) prompting configurations22,66, as well as the latest reasoning model Deepseek-r123. Evaluation was conducted across six publicly available engineering design challenges sourced from organizations such as NASA and the U.S. Department of Energy. These tasks were selected to ensure diversity in scale, complexity, technical maturity requirements, and degrees of design freedom. Full task descriptions are provided in Supplementary Note S5. All systems received only the competition guidelines as input, and no human intervention was applied during the generation of conceptual design solutions.

Table 1 presents a comparative analysis using five objective metrics, grounded in established engineering frameworks, to evaluate design outcomes across the dimensions of Novelty, Originality, Rationality, Technical Maturity, and Modularity. iDesignGPT consistently demonstrated superior or competitive results. Specifically, it achieved a median Level 3 invention on the TRIZ scale, a 0.4 median patent similarity score, and the lowest component coupling ratio (28%), indicating strong innovation, distinctiveness, and architectural modularity. Deepseek-r1 showed the highest rationality score (93% CR satisfaction), and higher technical maturity, but underperformed in originality, indicating a trade-off between creativity and feasibility. Efficiency, the sixth evaluation dimension, was excluded from Table 1 due to the absence of a widely accepted quantitative metric but was retained in expert evaluation via Fig. 5.

The expert evaluation followed a two-round Delphi method67, involving six domain experts independently scoring each solution across all six dimensions. The expert selection methodology and their profiles are detailed in Supplementary Note S7. The full Delphi protocol, statistical convergence checks, and all raw scores are documented in Supplementary Notes S8 and S9. Inter-rater consistency was quantified using Kendall’s W for both rounds (Supplementary Tables S6 and S14), and all first-round results, significance thresholds, and challenge-specific scoring matrices are visualized in Supplementary Figs. S5–S10 and Tables S8–S13. Re-evaluation outcomes and final convergence results are provided in Supplementary Figs. S11–S17 and Tables S15–S21.

Figure 5a visualizes the final expert ratings across all six dimensions for each design challenge. iDesignGPT consistently outperformed or matched baselines, particularly excelling in Novelty, Originality, and Modularity. These strengths mirror its objective advantages observed in Table 1, reinforcing the validity of the assessment framework. Deepseek-r1 performed well in Rationality and Technical Maturity, reflecting a preference for grounded, optimized designs, which was also captured in Table 1 by its high QFD satisfaction score and TRL. However, it trailed in Novelty and Originality, indicating a trade-off between maturity and creativity. GPT-4o-COT and zero-shot variants both performed poorly on creativity-related but achieved the highest expert scores in Technical Maturity, which was possibly due to their tendency towards safe, conventional solutions. Notably, COT prompting led to the lowest average novelty (2.17) and originality (2.58) across all tasks, suggesting that rigid prompting logic may limit divergent design thinking—a finding inconsistent with its performance in other domains57. While iDesignGPT’s mean score in Technical Maturity was slightly lower (3.03), this is consistent with its design philosophy: enabling exploratory ideation in early-stage conceptual design rather than optimizing for near-term implementation. Interestingly, Efficiency scores across models were relatively close, with GPT-4o-COT slightly leading (3.25).

To further dissect the factors influencing iDesignGPT’s performance, Pearson correlation analysis was conducted between its expert ratings in each dimension and its overall performance scores across the six tasks (Fig. 5b). Rationality exhibited the strongest correlation (r=0.65), followed by Efficiency (0.59), Novelty (0.53) and Originality (0.40), suggesting that experts favored designs that were functionally sound, efficient, and original. In contrast, Modularity and Technical Maturity had near-zero correlations, implying that in early-stage design contexts, structural decomposition and implementation readiness are less determinative of expert judgment. Fig. 5c extended this analysis by examining task-level characteristics and their correlations with iDesignGPT’s overall performance. Tasks with high Design Freedom and moderate Technical Maturity Demand correlated positively with performance, indicating that iDesignGPT thrives in open-ended, exploratory environments. Conversely, tasks with high Complexity and Innovation Demand were associated with lower scores, suggesting that while the system excels in ideation, its performance may diminish under tightly constrained or high novelty-expectation settings.

Overall, the alignment between objective metrics and expert assessments, especially in Novelty, Originality, and Modularity, demonstrates that iDesignGPT delivers consistently high-performing design outcomes that are both innovative and structurally sound. While reasoning-driven models such as Deepseek-r1 outperformed GPT-4o and its COT prompting configuration in most dimensions, only iDesignGPT achieved a robust and balanced performance across all evaluated dimensions. Other than language model capabilities, its advantage lies in the integration of established design processes, validated engineering design methods, and multimodal data from real-world design studies. This synergy enables iDesignGPT’s agentic workflows, currently implemented with GPT-4o and GPT-4o-mini, to operate effectively across conceptual engineering system design contexts where traditional approaches often fall short.

User experience
Design is not solely defined by the outcome but also by the process, particularly the collaborative interactions between designers and iDesignGPT, which shape the overall user experience. To evaluate the perceived value of iDesignGPT in supporting human-AI collaboration, we conducted two controlled experiments. The first compared user experience across different participant profiles, while the second assessed the impact of AI-assisted versus human-only workflows among novice designers. A total of 48 participants were involved, including undergraduate and graduate students, engineers, and academic researchers. All participants had prior exposure to LLMs and initiated their design tasks after a standardized training session with iDesignGPT. Experimental setup and participant details are provided in Supplementary Note S10.

User experience was assessed using three instruments (Supplementary Note S11): the Creativity Support Index (CSI)68, the NASA Task Load Index (NASA-TLX)69, and a custom-designed Workflow-Specific Metric (WSM). To validate the structure of each instrument, we conducted reliability analysis and principal component or factor analysis (Table 2). CSI captures perceptions of cognitive and emotional support during creative tasks, across dimensions like Goal Satisfaction, Exploration, Enjoyment, and Control. NASA-TLX revealed two latent factors corresponding to cognitive workload (Mental Demand, Temporal Demand, Effort) and affective-performance evaluation (Frustration, Performance). For the WSM, two robust factors were extracted: task fluency and process quality (items on design flow, usefulness, and clarity), and emotional impact and satisfaction (items on outcome confidence and decision support). Scree plots and cumulative explained variance (Supplementary Fig. S18) supported the retained factor solutions. All scales showed high internal consistency (Cronbach’s α=0.75–0.85) and sampling adequacy (KMO>0.79), confirming their reliability and suitability for comparative analysis across workflows.

Figure 6a–c presents results from the first experiment (n=11). CSI (Fig. 6a) scores indicate that iDesignGPT was positively received across all user profiles, with graduate students reporting the highest satisfaction, particularly in Goal Satisfaction and Exploration, highlighting iDesignGPT’s value in enhancing adaptive ideation. Undergraduates showed similar enthusiasm, indicating that iDesignGPT lowers the threshold for participation in complex design tasks. Engineers and researchers gave more reserved ratings, especially in the Control dimension, suggesting a transitional phase in integrating new AI tools into established professional workflows. In NASA-TLX (Fig. 6b), iDesignGPT positively impacted various dimensions of workload. Undergraduate and graduate students reported notable reductions in Mental Demand and Frustration, indicating that iDesignGPT effectively alleviates cognitive load and streamlines the design process for less experienced users. Engineers and researchers felt their effort and performance were consistent with (or slightly below) their usual standards. This disparity may stem from their familiarity with design routines and the greater complexity of their self-assigned tasks.

Beyond standard metrics, we developed WSM (Fig. 6c) to delve deeper into user experience. Participants found that the system clarified design flows, uncovered hidden needs, aligned with relevant literature, and supported decision-making. Although the accelerated design timeline (30–40 min) occasionally led to perceptions of information overload, participants particularly valued the system’s process-oriented insights, which enhanced understanding and mitigated concerns about control, as indicated by the neutral responses in the CSI control dimension.

The second experiment focused on first-year undergraduate students (n=37), who were asked to complete the GoAERO design challenge under one of five workflows: human-only, GPT-4o, GPT-4o-COT, Deepseek-r1, or iDesignGPT. Workflow assignment was randomized, and design outcomes were graded as part of course assessment to ensure engagement.

On the CSI (Fig. 6d), scores were broadly comparable across workflows. The only reliable contrast was higher Exploration for iDesignGPT than human-only (p=0.026, d≈1.03). On the NASA-TLX (Fig. 6e), iDesignGPT reduced Mental Demand (p=0.007, d≈1.29) and Effort (p=0.002, d≈1.61) relative to human-only, whereas no reliable differences emerged among AI-assisted workflows. Differences were more selective for WSM (Fig. 6f). iDesignGPT was rated as providing clearer design flow than GPT-4o zero-shot and Deepseek-r1 (both p=0.040), and it outperformed GPT-4o-COT and human-only on Uncovered Ignored Needs, Useful for Concept Generation, and Preferred Process Explanations (all p<0.05). Other items were similar across conditions. These patterns suggest that workflow structure and explanations, rather than underlying model alone, account for iDesignGPT’s advantages on design-specific outcomes. Qualitative feedback (Supplementary Note S12) similarly indicated that hallucination-related issues, such as technically implausible suggestions or inconsistent logic, were reported for GPT-4o-COT and Deepseek-r1 but not for iDesignGPT, which was instead described as more verbose and interface-intensive, consistent with a more structured and controllable workflow.

Taken together, the two experiments show that AI assistance in general lowers workload relative to human-only design, and that iDesignGPT performs comparably to other AI-assisted workflows on global workload and creativity-support indices. The added value of iDesignGPT appears selective and driven by workflow design, with a clearer process, better surfacing of overlooked needs, and explanations that support concept generation and decision making. These benefits appear without broad platform-level advantages on NASA-TLX or CSI, which indicates that the way assistance is integrated and explained matters at least as much as model choice, especially for novice and transitional designers. Qualitative feedback aligns with this pattern and describes lower perceived effort and fewer outputs that are hard to control when using iDesignGPT.

Methods
Framework supporting multiple agents
The implementation framework of iDesignGPT (Fig. 7), built on fastGPT (https://github.com/labring/FastGPT), is built for modular integration of emerging design methodologies and AI advances. The workflow is orchestrated using a flow-based approach, where the overall execution logic primarily adopts a programmed sequence with a small-scale multi-agent design. This balances the orderliness of the design process while striving to harness the emergent capabilities of LLMs. The current implementation serves as one possible realization.

For user inputs, iDesignGPT integrates a safeguard agent that ensures the appropriateness of design intentions. Potentially harmful or socially risky requests are automatically rejected. Once the input is deemed reasonable, the system presents an initial mode selection interface, defaulting to Copilot mode.

In Copilot mode, iDesignGPT operates as a collaborative partner, actively engaging with designers through a cluster of consultant agents. These agents assist in clarifying design objectives by prompting iterative dialogue. Designers specify their requirements, while the consultants probe deeper into details, encouraging exploration and refinement of design problems.

In contrast, Agent mode shifts towards automation, streamlining the design process through four distinct agent clusters: analysts, information officers, innovators, and evaluators. Each cluster corresponds to a key phase of the design process—problem definition, information gathering, concept generation, and evaluation. This structured, automated approach guides the designer step-by-step by an intent classifier and predefined prompts. Transparency is maintained by presenting intermediate analysis results and explaining the reasoning behind decisions. Designers can intervene at any stage to trigger re-execution or proceed to the next step, enabling both interpretability and control.

Workflow enabling design methods
The framework supports different design methods through three primary workflow topologies:
• Star flow is particularly suited for brainstorming scenarios requiring diverse expert perspectives, emphasizing collaborative refinement of design insights. A host agent undertakes organizational and coordination tasks. At the beginning of each topic, the identities of expert agents are defined based on the design problem. During each discussion round, the host provides targeted questions to each expert agent and collects their responses. The expert agents share their conversation history, ensuring that insights are collaboratively refined.
• Loop flow is optimized for tasks requiring systematic validation, such as functional decomposition, where iterative collaboration is adapted to handle logical reasoning and flowchart generation tasks. To ensure accuracy and mitigate hallucination risks, a generator agent collaborates with a checker agent. The checker evaluates the generator’s outputs against predefined criteria, sending feedback for iterative improvement when necessary, until all checks are satisfied. This iterative mechanism ensures robust design outputs while addressing common LLM challenges, such as overgeneralization or hallucinated reasoning72.
• Sequential flow supports stepwise methodologies like TRIZ, where predefined principles are systematically cross-referenced and applied to resolve conflicts. A guider agent first formats the features involved in the design case and sends them to a JavaScript program, which retrieves conflict-resolution principles from TRIZ. These principles are then cross-referenced with a principle database, and the resulting solutions are applied by an analyst agent. This structured workflow simplifies the application of TRIZ principles, offering designers efficient and context-specific recommendations.

These workflow topologies exemplify how iDesignGPT adapts to diverse design methodologies. A detailed description of agent roles and workflows is provided in Supplementary Note S13 and illustrated in Supplementary Figs. S19–S21.

Knowledge base
The knowledge base is composed of two main components: online and local. The online component is designed for large-scaled data retrieval tasks, such as patent and academic literature searches. Patent searches are conducted using the Google API, with filters applied to retrieve only patent-specific content. Academic literature searches are facilitated by the Web of Science and Scopus API, where iDesignGPT can customize search queries and set publication year filters.

The local component focuses on content-based semantic searches, which are critical for task where the content itself is central, such as in design inspiration or biomimetic cases. This component includes databases like the Red Dot-awarded designs from the past three years and selected examples from the Ask Nature platform. Local data is structured using RAG technology, where large text blocks are segmented and indexed using the text-embedding-3-large model73. Search terms are further optimized by GPT-4o-mini to generate related queries. Semantic searches are performed through vector matching, with minimum similarity thresholds and text length constraints. The matched paragraphs are incorporated into the prompts provided to the agent.

Toolkit
Several specialized tools are integrated into the system. During the Web of Science paper retrieval, a Calendar tool automatically captures the current date, ensuring that the API queries are restricted to relevant publications within a specified time range. For TRIZ analysis, a Script Runner tool executes custom code, enabling matrix-based searches and the retrieval of solution principles when features for improvement or deterioration are specified. In functional decomposition, Mermaid generates flowcharts by inputting markdown code formatted to define the structure of the design process. Finally, in the summary phase, key components such as functional modules, solution approaches, and applications scenarios are visualized using DALL-E 3, which generates conceptual diagrams to complement the textual content.

LLM integration
The framework is designed to support diverse LLMs, including state-of-the-art models like GPT-o3 (https://openai.com/index/o3-o4-mini-system-card/) and open-source alternatives such as LLaMA 4. This adaptability ensures that iDesignGPT’s performance can continuously improve alongside advancements in LLM capabilities. The selection of specific models is guided by task requirements, balancing computational efficiency and model capabilities to optimize user experience. In this study, we utilized two versions of OpenAI’s models: GPT-4o and GPT-4o-mini. GPT-4o, with its superior language understanding and generation capabilities, was primarily employed for complex tasks in Copilot mode, such as in-depth interactions with designers. Conversely, GPT-4o-mini was selected for most tasks in Agent mode that required processing large amounts of context at higher speeds, with less stringent logical demands. To balance creativity and consistency in generated responses, we adjusted the temperature parameter based on task requirements. Detailed results are presented in Supplementary Note S14.

At the core of each agent, the LLM is tasked with executing operations based on four key components: Prompt, History, Data, and Question, as illustrated in Fig. 7.

Prompt. As the most crucial element in the task execution, prompts contain the following subcomponents:
Role. Defines the identity and function of the agent within the system. These roles are typically pre-defined at the start of the task, but in certain scenarios, such as during brainstorming sessions, the host agent dynamically defines expert roles from various domains based on the project’s needs.
Background. Provides global variables and related stage results necessary for task execution. This component is especially critical in tasks that involve external API calls and tool usage, as it includes detailed instructions on tool names, descriptions, invocation timing, and operational steps.
Thought Process. Utilizes the chain of thought technique to break down complex tasks into manageable steps, guiding the agent through the problem-solving process to maintain logical consistency and ensure accuracy in task completion.
Output. Defines the structure of the agent’s output to ensure that the generated information can be efficiently processed and transmitted, minimizing information loss or ambiguity during the communication.

History. The History component restricts the number of prior conversation rounds accessible to the model, ensuring that the agent maintains relevant and coherent context when executing the current task.

Data. The data component is populated with information extracted from the knowledge base through retrieval-based methods. This allows the model to access and leverage domain-specific knowledge, overcoming the limitations of the model’s pre-existing training data and compensating for gaps in design-specific knowledge or outdated information. By incorporating information from external APIs, the model ensures access to the most relevant and up-to-date resources.

Question. Refers to the specific inputs provided by the user, enabling real-time interaction between the model and the designer. In Agent mode, the majority of the prompts have been pre-encoded, reducing the dependence on designer input and further streamlining the design process, thereby enhancing the system’s automation capabilities.

Human participant information and ethics
Participants sex and gender information was not collected, and no sex or gender-based analyses were performed. This is because the study focused on comparative evaluation of design workflows and subjective user experience metrics, for which sex and gender were not considered relevant variables. Accordingly, sex and gender were not incorporated into the study design or statistical analyses.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Thanks for the manuscript and the interesting read. The text also appended as attachment. The authors present a Large Language Model based framework (iDesignGPT) developed to address the nature of early phase conceptual design. Evaluated on publicly available design challenge authors used available (published-, and known) metrics based on questionnaires that enable quantitative evaluation of useability and impact when used. Results suggest that the proposed tool/framework, iDesignGPT, performs better than two existing Large Language models used for similar tasks. The set up of the framework is sound. The four core components (Agents, Knowledge Base, Toolkit and Web front- end) to a generic four phase design relevant for a LLM supported design process makes a lot of sense, and the idea to set up the framework with a co-pilot, and an agent- mode, to bridge the gap between the “fuzzy” Design context and the LLM Agent domain is also sound. As such the authors present a solution that advances knowledge in an interesting gap where a theoretical and practical understanding needs to be advanced to benefit in design from the rather new advancements in LLM’s. Another strength is that both the targeted design (noun) is explored together with the process of design (verb), although the level of resolution of the study address engineering systems design rather than engineering design (implying more details in product representation). The single most interesting construct presented is the division into the co-pilot and agent- modes that reflect the nature of the targeted design process phase well. Validity. The nature of design studies, being largely ill-defined and un-structured, is difficult to study rigorously and quantitatively. The choice of using open accessible design challenges makes sense, and the approach clearly a trade off between making a systematic evaluation and a problem that is sufficiently realistic and complex. There are some aspects where the manuscript can be clearer to understand what is in, and out of, scope. - Scope and coverage of Engineering aspects. As the ambition is to address Engineering Design as a discipline, the problems addressed in the “engineering” side are less well positioned as a more general “design” approach. The case and reasoning presented, seems to be before classical engineering design become applicable, and rather address early stakes of engineering systems design, rather than engineering design? As an example, a classic and relevant problem for LLM support is to related to geometrical representations of a product definition, which is not discussed (as resolution stop at the identification of “modules”. In the text, design representation seems to be limited (down-) to shapes and forms (images etc), such can be generated using e.g. Dall-E, but not product geometries necessary to conduct engineering analysis of product models (CAD models e.g.) that is typical for product engineering design. What authors address-, and not-, in the conceptual engineering design studies targeted would need to be more clearly bounded. It becomes important as authors claim (Row 92) to address AI-assisted engineering design by integrating advanced analytical tools; and in engineering design such “advanced analytical tools” is too open for a general claim. What (type of) analytical tools are included and what are not? - The component “knowledge base” (e.g. Row 119) claim “real-time access to domain knowledge … ultimately facilitating the generation of optimized design solutions”. In line with the above, it is not clear to what degree this is actually supported and used in iDesignGPT and not? - The complexity of the problem posed, and the design concepts used, opens for several questions on detail level. As one example, in the derivation of requirements (a part of the design process), it is explained that the down-selection of requirements (row 249) is a part of the process. Some of the initial requirements are omitted, others are created. This makes it difficult to understand the rigor of the evaluation. Normally, requirements must be met for a solution to be valid, and cannot be down-prioritized. The requirements engineering process can be made more clear on this point. Originality and significance. - To the reviewers knowledge the work is original and provides new insights into an important field. Relevant references are included, although it is difficult to cover large knowledge fields, such as classifications of engineering design. The contribution is more to engineering systems design, than engineering design, although occasionally the areas are treated simultaneously. The example methods attributed to engineering design, such as the QFD method origin from Quality Engineering, although similar methods exist in engineering design literature. The reason for this comment, as commented in previous remarks, is that engineering design origin from, and typically address “product” level design. Data & methodology: The overall methodology is clear, and the supplementary material explain sufficiently how the evaluation was done. Some comments and questions: - Using Web of Science only as a source of academic literature is a limitation, especially in engineering design sciences, where also Scopus and other sources have material of relevance. This is not commented on. - The set of metrics used seems interesting from a LLM point of view, as indicators of dimensions of designs, rather than the instances of design. For readability, an example of any of the concepts should be more clearly visualized. At present, these are found in the dense figures of 3 and 4. - The goodness of a design is normally assessed against how well a design (concept) meets its criteria (requirements). In the present text, the evaluation rather evaluates the dimensions of design concepts and the characteristics of the design process. Appropriate use of statistics and treatment of uncertainties: - To the level addressed in the study, there is a balance in how detailed the data is represented and visualized. There is a good discussion onto the interpretation and plausible reasons of the findings, by reference to the sources (the sample set of participants), but the discussion of bias and variation based on the sample set of participants participating and responding is not as clear. Conclusions - There is no explicit conclusion section presented, whereas the discussion section reflect on the conclusions and propose further work. The impact onto design work (row 372 ff) is suggested to lower the threshold for knowledge of a designer, i.e. less educated and experienced people can do more advanced design studies. A critical remark is to what extent the suggested framework resonate with the well known pit falls of LLM’s in general (hallucination and mis-interpretation of intent) – even if mechanisms to improve such phenomena are introduced? To what extent will a designer need to maintain a critical eye to results generated, something that typically requires a high degree of expertise? Ultimately, who is accountable of the results generated? Suggested Improvement - The overall understanding is that the paper predominately address, and contribute, to engineering systems design, rather than- or at least in addition to-, engineering design. As such, works from e.g. De Weck and others are relevant - De Weck, O. L., Roos, D., & Magee, C. L. (2011). Engineering systems: Meeting human needs in a complex technological world. Mit Press. Or - Maier A., et al (2022) Handbook of Engineering Systems Design, Springer, https://doi.org/10.1007/978-3-030-81159-4 Also, an approach to classify different engineering design methods and processes are: - Wynn, D. C., & Clarkson, P. J. (2018). Process models in design and development. Research in engineering design, 29, 161-202. - As mentioned, the QFD method origin predominately from the quality literature. Why relevant in general, there are more typical references for engineering design, (See Wynn et al above) - Its recommended to clarify the conclusions and include also critical implications of the findings (see text above). Clarity and context: The manuscript reads well, is clear and concise. The figures, foremost 3 and 4, are useful, yet too much context and content of importance are condensed in the figures. The paper also lack an explicit conclusion section, although the discussion section is important. The manuscript would benefit from displaying a more clear example of the concepts being generated, to enable the reader to get a better sense onto what level concepts are being generated and evaluated. This is present, but embedded in text and in too small figures. Additional comment As a reviewer I have my base competence in engineering design and systems engineering and currently approach the LLM technologies from their impact onto the design process predominately in my own research. (Remarks on code availability) I have not downloaded and implemented the code. I have read through and included the other supplementary material.", "reviewer_2": "Key Results The paper describes a framework using large language models to support engineering design. The paper claims a number of advances including improving the novelty of solutions and diversity of solutions as well as an ability to uncover hidden requirements and fostering creativity, among others. Validity The paper makes many claims based on the interpretation of results from a series of test cases, for example the Go Aero Design Challenge. In my view the claims are completely unverified by the presented paper and supplementary documents. For example, the paper as presented claims a number of times to be more creative, and to have improved the novelty of solutions. But there are no results from the actual designs to support these claims. In the case of the Go Aero Challenge contestants are asked to design a vehicle that meets a complex set of requirements and cope with multiple mission scenarios. There are no results presented as to what this design looks like with the iDesignGPT framework. Moreover, this kind of vehicle has been developed many times in competitions across the globe, so there is a reasonable data set of existing designs which could be used to demonstrate novelty or improved performance. Additionally, the measure of novelty as presented in supplementary note S2 describes a set of generic equations and makes no reference to the actual data used. This is a critical point. The actual data set, which should include both initial design parameters and corresponding performance metrics must be described and shown to ensure the novelty measure is comparing like with like. The design of an air vehicle is a complex process and typically requires the use of a number of software and analytical tools (a toolchain) to generate the data and provide input to design decisions. The paper makes no mention of any of these tools or whether they are simple spreadsheets for example or even sophisticated simulation packages (e.g. Abaqus FEA, ANSYS, CFD...). Without the details on which package was used and where it was used it is impossible to have confidence in the results of any process. The complete lack of data presented in this submission is a fatal flaw in my view, a renders any conclusions drawn to be invalid. Significance The ability to instantly and autonomously design a complex vehicle or equivalent high value product would be immensely significant. However, noting the points above, this cannot apply to this work as the conclusions cannot be verified. Data and Methodology The presentation of the AI technology at a high level is generally clear and well presented. But the specifics of the iDesignGPT system are not presented. So, although the approach as shown mostly in the early figures appears logical it is not clear at all how this is shaped into a real design process that produces well defined concept designs. The approach and method itself also raise questions. For example, 6 different design challenges are used, and the six people participating each did one. The experts that then review novelty are basing this completely on their experience and judgment. I would have expected an approach where each example is explored by several people so that meaningful measurements and comparison can be drawn for each. In my view the basic methodology for comparing the designs is flawed and a major issue in the work. This coupled with the lack of real data on each design and how they compare is a fatal flaw. Analytical Approach Following on from the data & methodology above the analytical approach is weak because it does not use a comprehensive data set on the actual designs and processes and as so many different examples were studied it reads to me like each has a sample size of one. Again this is a major flaw. Suggested Improvements This is an exciting technical area and one with much potential for the field, and it is important that this field is explored and developed. Some suggestions for improvement are: 1. Significantly reduce the breadth and scope of the paper and focus on only one or two aspects. For example novelty. 2. Include real design data and show how the individual solutions differ in both their design parameters (e.g. Wing span) and their performance measures (e.g. Speed). 3. Develop several solutions to produce at least a 1-sigma study. 4. Clearly define and show how the metrics e.g. Novelty are defined and show how this applied to at least one of the design solutions. 5. In describing how the system works show how all the real engineering calculations are carried out or approximated and show all the various software tools that are used in this process. Clarity and Context The results are all presented as claims of capability in the AI space. The context for the design and the design process are absent and therefore obscures the potential value and capability of the work. The introduction does make many good observations but this should be clearer. For example, TRIZ is mentioned as a design method and appears in the functionality of the framework. However, TRIZ in itself is not a direct tool, although there are some tools there that help. Also, QFD as a method only helps at one part of the design process, and moreover is not universally adopted in engineering design. How do all these tools work together as a toolchain? And how does this fit into the iDesignGPT framework? References There is a good and long list of references many of which are contemporary and therefore up to date. The challenge is to have a set of references that are really relevant to the problem in hand and in this the references could be more comprehensive and more targeted at the specific technical gaps or opportunities being explored in the paper. My Expertise I am a senior academic with substantial industrial experience. I lead a number of major research programmes in engineering design that are using AI tools and addressing challenges in complex engineered systems such as aircraft. I have published extensively in the aircraft design field. (Remarks on code availability)", "reviewer_3": "This papers presents iDesignGPT, an LLM-based design support tool, that is applied in engineering design contexts in order to support designers. The development of iDesignGPT may have a valuable contribution to the field of engineering design, however the manner in which it is reported in the paper makes this not possible to discern. My recommendation is therefore that the paper is rejected and the authors thoroughly re-structure and re-write the paper such that a clear contribution can be evidenced and articulated. Reasons for this conclusion are detailed in the following paragraphs. The paper has a very odd structure, with the results beginning in section 2, immediately after the introduction. As such there is no clear research gap and no formal review of similar work. Whilst LLMs are novel in their application in engineering design, there is some work already in this space (see Ege et al https://doi.org/10.1017/pds.2024.202). In fact, a whole engineering design conference was run on this topic last year (https://www.designsociety.org/1337/NordDesign+2024). Whilst LLMs are relatively new, expert systems, knowledge based systems and design support tools are not, and much exists in this space. Further to this, an established research area is the Democratisation of Design, which seeks to up-skill designers in Engineering Design tasks. iDesignGPT builds on all of these concepts and, as such, an article presenting it needs to consider and review the areas upon which it is built. There is no methodology section. Some methods are mentioned later in the paper but isn’t collated into a clear section that provides cohesive overview of the work undertaken. Some noteworthy omissions from a methodology section are that: 1. there is no clear design research methodology (e.g. DRM) followed; 2. there is no selection of the types of engineering design problems iDesignGPT is applied to - it isn’t possible to evaluate this holistically so some representative areas need to be chosen; 3. there is no clear method of appraising functional success of iDesignGPT - what does success look like?; and, 4. there is no clear method of evaluating user success - how can you evidence it has supported their design process?. Moreover, results presented in the paper are descriptive, and lacking objective measures. I think this is largely due to absence of methodology and means of validating and evaluating the iDesignGPT tool generated. Further to these substantial structural issues in the paper, figures in the paper are too small to be readable (thus they don’t provide valuable results) and the writing style is very verbose. Much of the text feels AI generated. In terms of re-working the paper it strikes me that it should be separated into at least two separate publications. The first would be a technical presentation of the iDesignGPT with validation in 2-3 representative engineering design case studies. This paper could answer the research questions of i) can iDesignGPT provide technically valid solutions to engineering tasks; and, ii) can it outperform a general LLM? The second paper would present a user centred evaluation (with 20-30 participants) of the iDesignGPT demonstrating how it is able to support non experts to improve their design processes. This research question could be about can iDesignGPT support / up-skill designers in engineering design problems? To try and answer both of these questions in a single paper is overly ambitious and I don’t believe both can be covered well. I have not included more in depth comments for individual sections due to the substantial re-work that I believe is required for this manuscript. I wish the authors all the best with the future work, and hope these comments are helpful in reframing the manuscript for future submissions. (Remarks on code availability)", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts. (Remarks on code availability)"}, "author_response": ""}, {"round": 2, "reviewer_comments": {"reviewer_3": "Thanks to the authors for their substantial work in revising the manuscript. They have certainly made many efforts to address the reviewers’ comments. Please see additional comments regarding the revised manuscript. I have some concerns around the additional studies undertaken and the interpretation of the results. Regarding the objective comparative studies reported in Table 1: - It is unclear who conducts the performance evaluation for which results are presented in Table 1. Was it done by the authors or by the experts undertaking the Delphi study? - The metrics require some increased clarification within the paper, if only just to clarify whether a higher value is better for example. Currently the table itself can only tell a reader that your platform is different, not that it is better. - The results demonstrate some objective differences between design outputs generated by different AI platforms and iDesignGPT albeit with a single data point. This needs to be considered as a limitation. Regarding the additional comparative user study undertaken I have some concerns around the methods and interpretation of the results: - It is unclear why 11 participants took part in human-only and iDesignGPT groups but there were only 5 in other groups (as defined in supplementary note S10). What is the reasoning for these different sized groups? It strikes me an odd methodological step to not have balanced group sizes for this study. - The text describing figure 6e appears incorrect. iDesignGPT is stated to reduce mental demand and frustration more than any other system yet Deepseek appears to outperform here? The authors need to review results statements in this section and check they align with what is shown in the figures. - On reviewing Supplementary Tables S27—29 very few statistical differences are actually found. The results presented in figures 6d-f are therefore misleading as they suggest broad differences between AI platforms that simply weren’t found in the study. In fact, the results show: i) broad differences between human only and iDesignGPT conditions n NASA TLX; ii) no differences at all across different AI platforms with TLX; iii) no differences at all between CSI measures between iDesignGPT and any AI platform or human conditions; and, iv) some differences between Comparative workflow specific metrics. Reporting within the paper needs to reflect these results as they are, and not compare aggregate data in such a way that misconstrues the results. Discussion and conclusions need to be updated to make sure the findings are representative. Largely, this means the conclusions need to be softened substantially. - If figures 6d-f are to be kept in the paper, they need to be amended to show distributions (e.g. box plots) as the single point comparisons are misleading. - As an extra note on this, if iDesignGPT is found to perform similarly to extant AI platforms, this is a valid finding and is interesting to comment upon. On review of the abstract having re-read the paper there are a few points that require changing to more accurately reflect the findings in the paper: - The comparative study uses only novice designers so the way ‘diverse participant profiles’ is used seems misleading - I don’t see evidence for how iDesignGPT fosters interdisciplinary collaboration - this isn’t commented on or demonstrated in the results. - It doesn’t lower the expertise barrier vs. other AI tools, but does vs. Human only. - Please make sure all statements in the abstract are specific and the they accurately reflect the findings in the paper. As a further reference for the Democratisation of Design and determining if a reduction in the required skill levels of a designer is achieved this paper may be of interest to the authors - https://doi.org/10.1017/dsj.2023.19. It may be useful for future evaluations of their tool. In summary, I have no issues with the iDesignGPT platform, although I believe it does warrant a more in depth presentation (as per my previous comment #8 in the response document) however the studies undertaken to evaluate it do not produce the conclusive results that are stated in the paper and therefore need to be reviewed and revised accordingly. (Remarks on code availability)", "reviewer_5": "Thank you for the resubmission of the article and for the meticulous recording of the changes. I am satisfied that the changes address the concerns raised. (Remarks on code availability)", "reviewer_6": "The authors present the iDesignGPT framework to support the conceptual design of engineering systems. The current version of the paper is well written. However, a few concerns about the content and presentation need to be addressed. The comments to be addressed are provided below: Major concern: 1. During Stage 2 - Gathering information, patent databases are searched to collect information for concept generation. Given this use of patent information, how do you ensure that the generated concepts do not result in copyright infringements? Minor concerns: 2. Title: Capitalize the first letters of the words in the title. 3. Abstract: i. Please explicitly specify if it is six public challenges or six public design challenges? ii. The term 'baselines' is ambiguous and not clearly defined. Explicitly clarify what the baselines being referred to are. 4. Introduction: i. In line, \"Additionally, human factors, such as unclear requirements....\" Why are these factors referred to as human factors? Please clarify. ii. In line, \"...further complicate the design process,...\" Is it 'conceptual' design? Ensure consistency in the use of specific terms. iii. In line, \"...limitations in addressing complex, multimodal, and evolving design spaces.\" The terms multimodal and evolving are not introduced. This leads to a lack of clarity. Please add a discussion to clarify what these terms mean in this manuscript, in the context of the discussions made regarding conceptual design. iv. In line \"These challenges lead to suboptimal design quality, ...\" Optimization is not a concern during conceptual design. Please clarify your statement in light of the conceptual design. v. Please present all figures close to where they're being referred to for the first time in the manuscript. vi. In Line \"However, its transfer to conceptual engineering system design tasks requires specialized adaptation due to their unique characteristics, which lack definitive evaluation criteria, involve vast and unstructured solution spaces, and demand creativity-driven exploration.\" The claim about the lack of definitive evaluation criteria is not true. Methods for evaluating novelty, coverage, and diversity during conceptual design are well documented in the literature. For example, the Shah metric for novelty evaluation. Please address this concern. vii. What are the baselines? Please clarify in the manuscript. viii. Abbreviations used without expansions. This occurs at multiple locations in the main manuscript and the supplementary information document. For example, FBS and C-K abbreviations in the Figure 1 caption, and IQR in Table 1. Please ensure that all abbreviations are expanded out to their full form in the very first instance they are used. 5. Results: i. Why is the proposed iDesignGPT framework part of the results? Suggest placing the framework as a separate section. ii. Number the core components of the framework in the manuscript. Likewise, in Figure 2. iii. As in Figure 2, number the key stages in the manuscript for clarity. iv. In Line, \"..., ultimately facilitating the generation of optimized design solutions.\" The focus in the conceptual design phase is not on optimizing design solutions. Then, how is this relevant to the discussions? v. Please clarify where/how the solutions generated using iDesignGPT are outputted? vi. Discuss the case study and its results in a new section, with a potential title - Validation using Case study. vii. Please place all webpage links in the manuscript as footnotes. Example: \"....we selected the GoAERO Design Challenge (https://www.herox.com/goaero).\" viii. In Line, \"iDesignGPT’s design process (Fig. 3) mirrors the conceptual design phase of engineering development procedures57, where user needs are translated into Product Design Specification (PDS)58 and a corresponding conceptual design scheme.\" 'Design Specifications' are only specified in the detailed design phase according to the Pahl and Beitz Design process. Are you referring to the design requirements? Please clarify this in the manuscript. ix. Name of agent clusters: Isn't it Analyst agents cluster rather than Analysts agent cluster? Likewise for the other agent cluster names. x. Suggest introducing the different agent clusters and their roles in the framework section, and not in the case study section. xi. What is the Project Design Sheet? It has not been introduced previously. Please describe what it is and its use? xii. In Line, \"During the concept generation phase, the Innovators agent cluster utilized functional decomposition validated by autonomous agents to address the hallucination problem, ensuring logical consistency.\" Ensure consistent usage of terms. Previously, the concept generation phase was referred to as the 'concept generation stage.' Please address all similar issues. Are the autonomous agents part of the Innovators agent cluster? Please clarify. xiii. In the concept generation phase, what is the order in which the advanced concept generation methods are used? Please clarify this in the manuscript and the supplementary information document. xiv. Figure 3 caption: Is it user flow or workflow? Please clarify. xv. How are the percentage increases in convex hull area and diversity within modules computed? What are the baselines used in these computations? xvi. The term 'combination plans' is used in the Evaluate Concepts stage discussion. What are combination plans? Please clarify in the manuscript. xvii. For the Objective metric listed in Table 1, please clarify the nature of these metrics and what they indicate. 6. Discussion: The last line in Paragraph 2 is the same as the last line in Paragraph 1. Address this issue. 7. Methods: i. Consider moving the methods section after the introduction of the framework to ensure continuity. Placing the methods section after the Results isolates it, resulting in a lack of continuity. ii. In the sub-section - Workflow enabling design methods, correct the spelling of methods. 8. Supplementary Information Document: Adding cross-referencing in the supplementary document would be beneficial for readers. (Remarks on code availability)", "reviewer_4": "Reviewer #1:\n“The authors present a Large Language Model based framework (iDesignGPT) developed to\naddress the nature of early phase conceptual design. Evaluated on publicly available design\nchallenge authors used available (published-, and known) metrics based on questionnaires that\nenable quantitative evaluation of useability and impact when used. Results suggest that the\nproposed tool/framework, iDesignGPT, performs better than two existing Large Language models\nused for similar tasks. The set up of the framework is sound. The four core components (Agents,\nKnowledge Base, Toolkit and Web front-end) to a generic four phase design relevant for a LLM\nsupported design process makes a lot of sense, and the idea to set up the framework with a co-pilot,\nand an agent- mode, to bridge the gap between the “fuzzy” Design context and the LLM Agent\ndomain is also sound. As such the authors present a solution that advances knowledge in an\ninteresting gap where a theoretical and practical understanding needs to be advanced to benefit in\ndesign from the rather new advancements in LLM’s. Another strength is that both the targeted\ndesign (noun) is explored together with the process of design (verb), although the level of\nresolution of the study address engineering systems design rather than engineering design\n(implying more details in product representation). The single most interesting construct presented\nis the division into the co-pilot and agent- modes that reflect the nature of the targeted design\nprocess phase well.”\n\nReviewer #2\n“I am a senior academic with substantial industrial experience. I lead a number of major research\nprogrammes in engineering design that are using AI tools and addressing challenges in complex\nengineered systems such as aircraft.”"}, "author_response": "We sincerely appreciate all reviewers for their time and effort in evaluating our manuscript and for providing valuable feedback. We have thoroughly addressed every comment raised by the reviewers and have conducted a comprehensive revision of the manuscript. Overall description of the revision are as follows. (a) Reviewers’ comments are in Black. (b) Authors’ responses immediately following each Reviewer comment and in Blue. Within each response, we use a blue arrow (“→ For complete revised content, please see the Response and Revision – Full Version”) as a locator tag, indicating where the corresponding updated paragraph or section is displayed at the end of the response. (c) Revisions in the main text and Supplementary Information are highlighted in Red, to indicate specific word- or phrase-level edits. (d) For fully rewritten paragraphs or sections, we provide the complete revised text at the end of each corresponding response. These rewritten passages are displayed under a green dashed header to improve readability. As for new and complete insertions, they are not marked in red. (e) To support different reading needs, we have prepared two versions of this response letter: • A concise version (Response and Revision - concise) containing only reviewer comments and responses. • A full version (Response and Revision - full version) that includes all rewritten content in green dashed boxes following each response. We recommend reading the concise version first, then consulting the full version for the exact revised content using the green annotations."}, {"round": 3, "reviewer_comments": {"reviewer_3": "Thank you for the revised manuscript - I appreciate the thorough response to my comments. I have a couple of minor comments, after which I am satisfied with the changes: 1. Thanks for updating supplementary notes S10 in response to my previous comment 4. I think the note of sample sizes needs to be included in the manuscript as well as supplementary material as sample size greatly impacts interpretation of fig. 6. For example: * human-only (n=11), GPT-4o (n=5), GPT-4o-COT (n=5), Deepseek-r1 (n=5), and iDesignGPT (n=11). * If you feel the need to, you can add justification for this in the manuscript 2. Further to this, you state that 48 participants (line 405) took part in the user experience studies, it should be clarified that these are across two studies, with 11 and 37 participants respectively in each one. 3. In the abstract you now state the tool shows positive reception across diverse profiles, but as far as I can tell from the materials, it’s all students (mostly undergrads & PhD students) except for 1 researcher, and 1 engineer. As such I’m not sure it is fair to state this as ‘diverse’. Please remove this from the abstract. 4. In the paper you should state the spread of these participants (e.g. X undergrads, Y postgrads, Z Researchers, A Engineers). This is important as it greatly impacts interpretation of the results.", "reviewer_5": "Thank you for resubmitting the article and for meticulously documenting the changes. I am satisfied that the changes address the concerns raised.", "reviewer_6": "The authors have satisfactorily addressed all the comments/concerns. However, the author(s) are requested to address the following comment. A. Please add the information provided in response to comment 27 regarding the computation of the percentage increase in convex hull area and diversity in the revised manuscript/supplementary material for improved clarity."}, "author_response": "We sincerely appreciate all reviewers for their time and effort in evaluating our manuscript and for providing valuable feedback. We have thoroughly addressed every comment raised by the reviewers and have conducted a comprehensive revision of the manuscript."}]

Decision letter