arrangement of components, the selection of specific components, and the assignment of parameter values to the components. In some cases, design also encompasses the synthesis of new components; however, when no new components are synthesized, the design problem reduces to a configuration design problem [12], which is the focus of this work. Graph -based representations and methodologies like graph grammars are pivotal tools for configuration design, offering a structured approach to explore design spaces [13–16]. Specifically, nodes in a graph can represent components or subsystems while edges represent connectivity between them. A graph grammar provides a ruleset for how these graphs can be manipulated in a formalized manner. More recently, Graph Neural Networks (GNNs) [17] have showcased significant effectiveness in handing graph -based design representations [18–22]. In the realm of configuration design, RL emerges as a promising technique for exploring complex design spaces encapsulated within graph - based representations [4,5,8] . The sequential decision -making capability of RL algorithms aligns with the often iterative nature of design tasks wherein the impact of design choices unfolds over a sequence of steps. Specifically, the algorithms can refine the policy to guide the RL agent towards high -performance regions of the design space in a manner which directs each action towards long-term gains. This synergy can be further enhanced through structured exploration facilitated by domain -specific graph grammars [6]. Furthermore, the scalability of RL algorithms is practically achievable using GNNs as function approximators [6,18,19] . While RL offers numerous possibilities for advancing engineering design, its practical implementation is limited by efficiency challenges, encompassing aspects like resource availability, time utilization, learning stability and convergence behavior [2]. There has been an increasing Symmetry Heuristics for Stable Reinforcement Learning Design Agents 3 interest in addressing these issues in the context of design. For instance, several multi -fidelity RL frameworks employing human -inspired approaches [23], transfer learning [24], and control variate approaches [25] have shown promise in reducing computational time. While these techniques effectively leverage reward approximations, there exists an opportunity to incorporate a broader spectrum of heuristic rewards, relevant across design problems. For instance, symmetry is a commonly exploited heuristic in mechanical design due to its ability to balance loads, reduce vibrations and improve space utilization [26]. These heuristics can be invaluable for improving efficiency in full configuration design problems with large graph -based design spaces [27]. The contribution of this work lies in enrich ing a foundational RL framework for configuration design with a symmetry reward and investigat ing its learning stability in comparison with a naïve approach . The rest of this paper is organized as follows. In Section 2, we brief configuration design and graph -based frameworks, and discuss GNNs and RL in context of design. Further, we discuss the potential of symmetry as a configuration design heuristic including some symmetry detection techniques. Section 3 begins by detailing a graph convolutional neural network -based RL framework for configuration design, setting the stage for the autonomous search of complex design spaces. This is followed by the formulation of a reward component that utilizes a symmetry detection method to bias the agent towards symmetric configurations. In Section 4, a truss design case study is described to demonstrate the effectiveness of the approach. Section 5 presents the results of th is study, including a comparison with a naïve RL agent. Section 6 summarizes the contribution of the paper and proposes directions for future work. 2 Background 2.1 The Synergy Between Configuration Design and Graph -Based Frameworks Configuration design is the systematic process of selecting and arranging predefined components, as well as assigning parameter values to synthesize a functional system that optimizes its objectives under various constraints [12]. Given an exponentially large design space of possible configurations, the choice of representation for the design space is crucial for effective optimization. In this regard, graph -based representations prove particularly ap t [13]. Graphs mode l the inherent structure within a complex system, where vertices symbolize individual components or 4 A. Agrawal and C. McComb subsystems, and edges represent relationships such as spatial connectivity. This intuitive mapping between a design configuration and a graph allows for a formal, structured approach to explore design configurations. Moreover, graphs provide a flexible medium that facilitates the frequent design iterations common in configuration design . From a mathematical standpoint, a configuration design problem can be formulated as an optimization problem over a graph 𝐺=(𝑁,𝐸), where 𝑁 denotes the set of nodes and 𝐸 the set of edges. Each node consists of component or subsystem information, including various continuous and discrete variables like the coordinates, orientation, dimensions, and material. For subsystems, it can capture additional holistic attributes such as component enumerations, and operational parameters like pressure and voltage. Each edge encapsulates a connectivity relationship between two components or sub -systems , encompass ing continuous and discrete variables involving different connectivity types ( like fastening, welding ). The flexibility of graphs is extended by incorporating graph grammars [13–16], which are a set of operations that dictate how the graph can be modified. These operations formalize the steps by which a design can evolve. The simplest types of operations include node addition, node deletion , edge addition, edge deletion and tuning node and edge attributes. These operations can be parameterized to add more context or conditions [28,29] . More complex rules can encapsulate subgraph transformations, wherein a whole cluster of connected vertices and edges can be replaced by another cluster to yield a better configuration [13]. While graph grammar provides a framework for systematically generating a wide array of alternatives in the configuration design space , it presents challenges especially in complex systems [13,29] . Developing an extensive set of rules demands significant expertise, especially for large, diverse design spaces. Further, inadequately defined grammar may restrict design options, potentially resulting in suboptimal outcomes. Addressing these limitations, a grammar derived from GNNs may offer a more effective approach. At the core of GNNs is the ability to capture relational patterns between graph elements [17], which is critical in configuration design where the interplay between components can potentially dictate further design operations. Specifically, by learning these relationships, GNNs can execute complex combinations of elementary operations across the entire graph for a given design configuration. Symmetry Heuristics for Stable Reinforcement Learning Design Agents 5 2.2 Configuration Design with Graph Neural Network Based Reinforcement Learning Agents In context of configuration design, RL algorithms [1] have demonstrate d their applicability in the sequential decision -making task of searching the design space for high -performance configurations [4–9]. Specifically, the state space of the M arkov Decision Process [1] is the set of all possible graphs that represent the design configurations, and the design operations are agent actions. At a specific iteratio n, the agent state is the graph that represents the design configuration at that design step . The agent actions define the set of design operations to be performed on the graph that transforms it to the next configuration represented by another graph. These operations can range from elementary ones to more complex subgraph transformations. Further, the reward function encapsulate s the objectives and constraints of the problem. The transition probability function for configuration design problems is typically deterministic . Lastly, the policy serves as a strategy for design, mapping design configurations to design operations with the goal of optimizing the performance of the configuration over multiple successive design operations. While the sequential decision -making paradigm of RL aligns with configuration design, the high dimensionality and complexity of problems limits the applicability of traditional RL algorithms. In this regard, the incorporation of deep learning with in RL frameworks is emerging as increasingly crucial . Specifically, deep neural networks have proven useful in serving as function approximators for the policy and value function s in RL [3]. In context of the application of RL to configuration design , a GNN based policy autonomously learn s to transform design configuration s for long-term gains, leading to high -performance configurations. Specifically, the reward feedback from evaluating the configurations in simulations informs updates to the GNN, progressively enhancing the policy and subsequently improving the quality of the designs [6]. While deep RL offers numerous possibilities for advancing engineering design, its practical implementation is limited by challenges in achieving efficiency, encompassing aspects like resource availability (computing systems and data needed) , time utilization (speed of learning) , stability (consistency of learning across training runs or episodes) and the resultant convergence behavior [2]. To enhance the learning stability of the RL agent , this work incorporate s a broader spectrum of heuristic rewards that are relevant across design problems , essentially shaping the reward function . It aims to bias the agent exploration in large graph -based design spaces, leading to more stable learning outcomes. Specifically, we enrich a 6 A. Agrawal and C. McComb GNN based RL framework with a reward based on symmetry , a deep - rooted heuristic in engineering design practice. 2.3 Heuristic -Guided R einforcement Learning Heuristic -guided reinforcement learning is an emerging field that combines the strengths of traditional heuristics with the adaptability and learning capabilities of RL . Heuristics encompass strategies that guide problem -solving processes based on empirical evidence, expert knowledge, and theoretical insights which are critical in managing complex problems like those in design [30,31] . The integrat ion of heuristics in the RL algorithm can bias the learning process towards a subset of high-performance solutions , potentially enhancing learning stability . One way to do this is to leverage them in the action space to bias the action selection towards more promising directions [32]. Shaping or regularizing the reward or value functions based on heuristics is another avenue to incorporate prior knowledge [33,34] . For in stance, Cheng et. al [35] start with short -horizon tasks using heuristic rewards and gradually shift to longer horizons with less reliance on these rewards. Symmetry is a pervasive and powerful heuristic that transcends disciplines, from biology, mathematics, ML to engineering. In biology, symmetric structures are favored due to their lower informational encoding requirements, which leads to greater mutation resilience [36]. Symmetry is also a fundamental principle in engineering, employed as a heuristic to address several design challenges [37]. It is particularly applicable for problems with explicit symmetric characteristics (e.g., symmetric boundary and loading conditions in structures [38]), problem s where low sensitivity in behavior is desired (e.g. symmetric ribs in wheels are less prone to vibration while driving [39]), problems where cost is an objective (e.g. symmetric gears are easier to manufacture and assemble [39]), problems seeking aesthetic appeal (e.g. architectural applications [40]), and problems seeking intuitive usability (e.g. user interface s in consumer products [41]). Even in seemingly asymmetrical systems, symmetry could still reign on a local scale, thus showcasing the pervasive nature of symmetry [40]. Furthermore, heuristics like modularity, often intertwine with symmetry, exhibiting how these heuristics could be enveloped within the symmetry framework [42]. This common thread of symmetry illustrates its integral role as a manifestation of underlying physical principles, a natural pattern leveraged by evolution, and a strategic design element in human -engineered systems . Symmetry is mathematically described as an object's attribute of being invariant, when subjected to geometric transformations. It is typically Symmetry Heuristics for Stable Reinforcement Learning Design Agents 7 categorized according to the characteristics of the underlying transformations and the resultant functionality , primarily falling into point group or space group symmetry [26]. Point group s like cyclic and dihedral symmetry groups ensure at least one point remains stationary during transformation . This is crucial for applications like braking systems that requir e rotational or mirror symmetry . In contrast, space group symmetry incorporates translational symmetry and is instrumental in optimizing material use in design layouts . In addition to exploring the significance and applicability of different symmetry types, there has also been extensive research on the automated detection of symmetry. This is pursued both as a theoretical challenge to analyze intricate geometry and as a tool for application s like model compression [40] and capturing design intent [43]. The methodologies for symmetry detection vary depending on factors like the data type (feature tree [44], projected view/drawings [45], images [46], point clouds [47], voxels [48], graphical models [43]), completeness of data (complet e or approximate [49]), type of symmetry transformation (scaling, reflection, rotation, translation or their combinations [50]), scale of symmetry (global [44] and/or local/partial [40]), and precision (exact [44] or approximate [40]). While there is a very diverse range of detection techniques [40,43 – 52], we briefly discuss one for global symmetry detection in graphical data structures. It is based on the Graph Edit Distance ( GED ) [53] between a graph and its symmetric transformation. GED is a measure that finds the minimal set of edit operations needed to transform one graph into another and has been commonly used for graph comparison. The editing operations involved in computing GED include insertions, deletions, and substitutions for both nodes and edges. The detection of symmetry using GED involves the minimization of this distance via an optimization algorithm or picking a symmetry element like a reflection axis from a set of candidates for which this distance is the minimum. 3 Met hods 3.1 Foundational R L Framework for Configuration Design The RL based design agent in this work aims to address the configuration design problem by starting from a seed design graph and iteratively modifying the graph to minimize an objective function, while adhering to the constraints. The interaction between the agent and the design space, particularly as the agent transitions from state 𝑠𝑡 (consisting of the graph 8 A. Agrawal and C. McComb 𝐺𝑡) to state 𝑠𝑡+1 (consisting of the graph 𝐺𝑡+1) by performing actions ( 𝑎𝑡) in the form of operations on the graph, is detailed hereinafter . The action space of the agent is grounded in a graph -structured set of design operations that are derived from a graph convolutional neural network, which serves as a policy function approximator for the RL agent. At each iteration 𝑡, the policy 𝜋 outputs an action graph 𝑎𝑡, wherein each node has an attribute vector. This includes one element corresponding to the first operation of node deletion . This operation also deletes the edges connected to the deleted node. The next set of elements correspond to the tuning of the continuous and integer variables that define the edge attributes . Specifically, the tuning values for edges are obtained through aggregation operations (like a summation) on the vector elements corresponding to nodes that defined the edge. Further, the next set of elements govern the tuning of the continuous and integer variables that define the node attributes . The next element governs the addition of a new node to the node via an edge , thereby expanding the graph. Lastly, the next set of ele ments determine the attributes of the new node if it has been added. These operations (denoted by ⊕) are applied on 𝐺𝑡 to obtain 𝐺𝑡+1. Figure 1 shows an example of the action space for a truss design problem, wherein the action graph consists of seven elements per node. These elements correspond to node deletion (along with the members that it connects), two elements for the editing member type and dimension, two elements to edit the joint co-ordinates, one element to govern joint addition via a member and two elements to determine the co -ordinates of the joint. Fig. 1 Action space of RL agent for a truss design problem Although this example is tailored to trusses, the fundamental nature of these graph -based operations makes them adaptable to any problem that can be modeled with graphs. In addition to the policy function approximator, this work also utilizes a graph convolutional neural network Symmetry Heuristics for Stable Reinforcement Learning Design Agents 9 for approximating the value function as illustrated in Figure 2. Specifically, it utilizes an attention pooling mechanism for an adaptive focus on different parts of the graph. Fig. 2 Value function based on graph convolutional layers and attention pooling Lastly, the agent reward should reflect the quality of the action (𝑎𝑡) that transitions the design from state 𝐺𝑡 to 𝐺𝑡+1. This depends on the amount by which the objective reduce s. Further, when the agent is in the infeasible domain, the change in the amounts by which each of the constraints are violated would guide the agent to navigate to feasible regions. The specific formulations of these reward components are adapted from prior work that utilizes deep RL for skeletal design [23]. Lastly, the graphs that result in invalid designs (i.e. objectives or constraints not computable ) are assigned a poor objective value to steer the agent towards valid configurations . 3.2 Symmetry Reward Formulation In addition to the reward components corresponding solution performance , we introduce a novel symmetry reward component in this work. This component employs a symmetry detection technique that finds the minimum GED between the graph representation of the design and symmetric transformations. These transformations are obtained by mirroring the graph across candidate axes passing through its centroid. Specifically, w e compute the GED corresponding to each axis by preparing a cost matrix of the pairwise distance between nodes and solving a linear sum assignment problem [54]. The goal of this problem is to assign each node in the original graph to a node in the transformed graph such that the sum of all costs is minimized. Based on the solution s of these problem s, we define the degree of symmetry ( 𝜂) as follows : 𝜂 = −min 𝑇𝑖 ∈ 𝒯 𝐺𝐸𝐷 (𝐺,𝑇𝑖(𝐺)) (1) where, 𝒯 is the set of candidate transformations. 10 A. Agrawal and C. McComb Further, we clip the degree of symmetry values below a threshold to ensure that the agent does not need to learn from overly complex patterns amongst highly asymmetric outliers. Figure 3 shows an illustrative example with node attributes as the co -ordinates of truss joints. The GED is evaluated between the graph and its mirror symmetry counterpart about four candidate symmetry axes, the minimum of which determines the degree of symmetry. Fig. 3 Degree of symmetry metric rooted in graph edit distance The degree of symmetry can be further utilized to define a reward component to extend the foundation RL framework in Section 3.1 This reward component denoted by 𝑅𝜂 is defined as the change in the degree of symmetry when the graph transition from 𝐺𝑡 to 𝐺𝑡+1: (𝑅𝜂)𝑡+1=𝑤𝜂 × (𝜂(𝐺𝑡+1)−𝜂(𝐺𝑡)) (2) where, 𝑤𝜂 is the weight associated with this symmetry reward component. Like the other reward components, invalid configurations are assigned a poor value to steer the agent towards valid configurations. 4 CASE STUDY 4.1 Truss Design Problem In this work, a truss design problem serve s as a testbed for evaluat ing the effectivenes s of incorporating symmetry in RL -based configuration design. A truss is a structure which is designed to bear loads in buildings and bridges. The optimization objective in this problem is to minimize the mass of the truss while adhering to a target Factor of Safety (FOS), a measure of the load -bearing capacity of a structure beyond the expected Symmetry Heuristics for Stable Reinforcement Learning Design Agents 11 loads. The specific 2D truss design problem tackled in this work consists of two fixed supports at the bottom corners and a vertical load at a node midway between the m. A target FOS of 1.7 is set for the problem, which is a commonly utilized value for the design of steel members [55]. This problem permits a huge number of potential solutions – up to 1075 unique configurations of trusses with up to 40 joints, without yet accounting for the positioning of joints in a continuous space . This enormity underscores the problem complexity and the challenge it poses for design algorithms. To set the stage for the symmetry -guided RL framework, the truss configurations are represented as graphs, wherein the edges represent the members and nodes represent the joints. Further, the node attributes include the joint co-ordinates , and the edge attributes consists of a scalar size variable . This graph representation defines the agent state. Actions includ e removing a node , editing node and edge attributes and adding a new node . Further, the actions that modify the nodes corresponding to the fixed joints and loads are masked during the transition to the new state. For the symmetry -guided agent, t he reward is computed based on the mass objective, the FOS equality constraint, and the symmetry reward with their weights tuned to yield satisficing designs . For the naïve agent, the same reward function is used without the symmetry component. 4.2 Training and Evaluating the RL agents An online one -step actor -critic [1] algorithm with graph convolutional neural networks as fu nction approximators is used for training the policies of the naïve and symmetry -guided agent . The utilization of an online algorithm improves the memory requirements by only having to store the most recent graph, while also opening the potential for faster learning due to frequent updates. Several random graphs that may or may not be feasible trusses are used as seed designs in the training episodes, each with a maximum length of 20 iterations . To bound the design space and ensure stable learning, the episodes are terminated whenever a configuration with less than 4 or more than 40 nodes is encountered. A common set of seed designs , initial neural networks and hyperparameters are utilized for training the agents . To understand the learning stability of the agents , the policies were periodically frozen during training for further evaluation, the results of which are discussed hereinafter. 5 RESULTS AND DISCUSSION The naïve and symmetry -guided policies were periodically frozen every 200 episodes during training , culminating at 3000 episodes where rewards 12 A. Agrawal and C. McComb plateaued for both agents. After completion of training , each of these polic ies were evaluated to assess its performance in terms of mass, FOS, and the degree of symmetry. This section provides a detailed account of this evaluation data including an analysis of the distribution of solutions and the stability of the learning process across episodes . Firstly, we investigate the distributions of the final solution qualities and visualize a few representative truss designs from this distribution. This quality is measured by the weighted sum of the mass objective and the FOS equality constraint penalty with the same weights that were utilized in the reward formulation. Figure 4 shows these distributions for the naïve and symmetry -guided agent . Both the distributions are multimodal in nature, which implies that the agent s find several local minima in the design space. However, the locations and densities of the peaks are significantly different for both agents. Figure 5 shows some representative truss es corresponding to the labelled peaks in Figure 4. Fig. 4 Distribution of final solution quality We observe that the symmetry -guided agent has a distinct sharp peak of the highest quality (S1), which showcases its ability to find high performance solutions. The corresponding truss in Figure 5 meets the FOS constraint with a moderate mass value. Moreover, this design is symmetric in nature, much like what is observed in trusses designed by human designers [38]. The highest quality design corresponding to the naïve agent’s peak (N1) also meets the FOS constraint but has a higher mass than the design S1. Moreover, it exhibits a lower degree of symmetry compared to design S1. The next two peaks for the symmetry -guided agent (S2 and S3) are also highly symmetric. However, they do not meet the FOS constraint. While these are not immediately usable configurations , Symmetry Heuristics for Stable Reinforcement Learning Design Agents 13 their symmetric structure may still be of value to a human designer for further exploration. Further, for the naïve agent, we observe that the designs N2 and N3 also do not meet the FOS constraint. However, they have higher qualities compared to solutions S2 and S3 respectively, albeit with varying densities. On one hand the symmetry -guided agent is biased towards symmetric solutions, while on the other hand the naïve agent potentially needs to explore a larger portion of the design space to find high performance solutions. Lastly, both agents yield about 20% of infeasible trusses as reflected by the peaks on the far left with a quality of -4.0. The symmetry -guided agent has a lower density of infeasible solutions that further showcases its superiority in exploring the design space. Fig. 5 Representative truss design solutions (red line is detected symmetry axis) To understand the stability of the learning process, we freeze and evaluate the polic ies periodically. The distribution of mass, FOS and the degree of symmetry of this data is visualized in Figures 6, 7 and 8 respectively. Fig. 6 Evolution of mass across evaluation of frozen policies 14 A. Agrawal and C. McComb In Figure 6, the invalid trusses are assigned a mass of 0. At the onset of training, a relatively high number of trusses are invalid for both the agents. As learning progresses, the mass values start to converge to distinct modes for both the agents, much like the peaks that were observed in the overall quality distributions. However, we observe minor inconsistencies across episodes for the naïve agent. For instance, the lowest mode of mass abruptly shifts several kilograms upwards during the last period of evaluation. This is indicative of instability in the learning process. Figure 7 showcases the trend for the FOS. Like the previous plot, invalid trusses are assigned a value of 0 .0. As learning progresses, the values start to converge to distinct modes for both the agents, much like evolution of mass . A fairly high proportion of configurations meet the constraint of 1.7 for both the agents around 2000 episodes. However, the behavior of the agents varies significantly as learning progresses . While the symmetry -guided agent effectively converges with a high density of feasible solutions, the naive agent exhibits high instability in this period. Fig. 7 Evolution of FOS across evaluation of frozen polic ies (red horizontal line is target value) Figure 8 showcases the evolution of the degree of symmetry for both the agents . On the top end, a value of 0.0 corresponds to perfect symmetry , while on the other end a value of -5.0 corresponds to invalid trusses as well as trusses with a degree of symmetry lower than or equal to -5.0. Both the agents exhibit increasing symmetry in the middle of the training around 2000 episodes. This corresponds to the high density of solutions that met the FOS constraint for both the agents. Further, the symmetry -guided agent shows a very high proportion of symmetric designs until the end of learning. However, there is a rapid drop in the case of the naïve age nt as it is not driven by the symmetry reward. This shows that the symmetry reward indeed results in a consistent set of high-performance solutions. Symmetry Heuristics for Stable Reinforcement Learning Design Agents 15 Fig. 8 Evolution of degree of symmetry across evaluation of frozen policies Overall, the evolution of mass, FOS, and symmetry of both the agents highlight the superior stability and convergence behavior of the symmetry - guided agent in comparison to the naïve agent. While it may be valuable to stop the training of the naïve agent early, it is impractical to continually evaluate an RL agent and identify this performance drop while training . Rather, the symmetry -guided technique provides a more stable approach with a tendency to converge without intensive monitoring . 6 CONCLUSION AND FUTURE WORK This research proposes a novel paradigm in the application of deep RL to configuration design, particularly involving the use of symmetry heuristics in a framework based on graph convolutional neural networks . The integrat ion of a reward based on the degree of symmetry of configurations has showcased enhanced stability and convergence in the learning process. Specifically, an empirical investigation is conducted on a truss design problem to substantiate the efficacy of the symmetry -guided RL approach. The results demonstrate that the symmetry -guided agent not only outperforms its naïve counterpart in terms of stability and convergence but also resonates with the natural design inclinations of human experts. While the proposed symmetry -guided approach has proven effective for a symmetrical truss design problem , engineering problems frequently exhibit only partial symmetry [40]. Future work should focus on investigating the effectiveness of the approach in such scenarios. Learning the symmetry characteristics as an integral part of the design configuration, rather than relying on separate detection algorithms, could enhance the framework's adaptability. Furthermore , the computational cost of 16 A. Agrawal and C. McComb calculating the GED for large solutions can be prohibitively high. A potential solution lies in actively learning a surrogate model [56] that can be deployed in a variable fidelity framework [23]. A more extensive exploration of efficiency metrics like time utilization with such algorithmic enhancements remains a valuable pursuit for subsequent studies. Future work could also delve into utilizing the symmetry heuristic within the action space rather than the reward function of the agent [57]. Exploring generalization capabilities of the symmetry -guided approach across a spectrum of truss problems is another promising avenue, potentially incorporating conditional factors for boundary and load -bearing nodes. Beyond trusses, the applicability of this approach to diverse domains, such as the design of drones, which often involve heterogeneous graph representations, could significantly broaden the impact of this research in the field of automated configuration desig n.