## 2 Reinforcement Learning Task for Truss Layout Design
### Sequential Decision Model for Truss Layout Design  
Markov Decision Processes (MDP) is the most basic theoretical model and mathematical expression for Reinforcement Learning problems [24]. The purpose of MDP is to construct a classical formal expression of sequential decision problems, which can construct an interactive learning model between an agent and the environment. The design process of truss layout can be modeled as an MDP model, which contains four components: state, action, transition model, and reward. A state can be expressed as all the information possessed by the current truss layout. When reaching a state that is not terminated, the agent will take a corresponding action by observing the characteristics of the state. The action contains three sequential types, that is, adding nodes, adding bars, and selecting cross-sectional areas of the bars. After taking an action, the transition model indicates the probability distribution of the next state. When all the decision steps have been completed, the agent will calculate a numerical reward based on the objective. The reward can guide the algorithm in the right searching direction towards a better solution. The details of the reward function are given in the pseudo-code Algorithm 1.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Constraints** & **Expression** \\ \hline \(g_{1}\) & Design domain, \(\Omega\) & Check element position in \(\Omega\) \\ \(g_{2}\) & Cross-sectional area & \(A_{min}\leq A_{i}\leq A_{max}\) \\ \(g_{3}\) & Strength & \(-\sigma_{min}\leq\sigma_{i}\leq\sigma_{max}\) \\ \(g_{4}\) & Displacement & \(\delta_{j}\leq\delta_{max}\) \\ \(g_{5}\) & Stability & \(\sigma_{i}^{c}\leq\sigma_{buckle}\) \\ \(g_{6}\) & Stiffness & \(\lambda_{i}\leq\lambda_{max}\) \\ \(g_{7}\) & Bar length & \(l_{min}\leq l_{i}\leq l_{max}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Constraints used in this paper.  
First, the reward function checks the geometric stability of the generated truss. The check process is performed by the function _IsStructure_. The checking process is divided into two steps: firstly, the Degree of Freedom (_DOF_) of the generated truss is calculated according to the Maxwell criterion [25]. If \(DOF>0\), a negative reward -1 will be passed on to the agent. If \(DOF\leq 0\), evaluate the positive definiteness of the stiffness matrix of the truss. If the generated truss is judged as a mechanism by stiffness matrix evaluation, a negative reward of -1 will also be passed on to the agent as a penalty.  
When the generated truss passes the _IsStructure_ function, the algorithm returns a nonnegative reward to the agent according to the iteration number and the structural performance of the current layout. Specifically, if \(iteration<i_{mark}\), the reward is calculated by \(f(obj)_{soft}\). Otherwise, the reward is directly assigned as \(f(obj)_{hard}\). \(f(obj)_{soft}\) is a soft reward model and gives punishment when existing constraint violations. The expression of the soft reward model can be defined as follows:  
\[f(obj)_{soft}=\frac{\lambda}{W(P,E)^{2}*\left(1+\sum penalty_{gi}\right)} \tag{3}\]  
In Equation (3), \(penalty_{gi}\) is the percentage of constraint violations and \(\lambda\) is a parameter to standardize the reward value according to different experiments. \(f(obj)_{hard}\) is a hard reward model and only gives positive reward value to those layouts that pass all the constraints; otherwise, the agent will only receive a zero reward. The expression of the hard reward model can be defined as follows:  
\[f(obj)_{hard}=\frac{\lambda}{W(P,E)^{2}} \tag{4}\]