## 2 Generate Initial Solutions to PFSP Using a Deep Reinforcement Learning Method
### Policy Network Architecture
#### 2.2.2 Graph Encoder  
A complete graph \(G=(V,\ E)\) is introduced to describe the PFSP, as shown in Figure 5. \(V=\left\{x_{1},...,x_{n}\right\}\) is the set of nodes and \(E\) is the set of edges. All elements of the adjacency matrix are equal to one because any two jobs are related to each other, which means that any job can point to all other jobs. Figure 4: The proposed policy network structure (scheduling sequence: 1, 3, 4, 2). Figure 3: The PrNet structure (with attention mechanism). Originally proposed by [34], GNNs extend the existing neural network models to process data from graphs or topological structures, and have achieved good performance in graph node classification, regression problems, and other fields. In the proposed network structure, job context information is obtained by encoding all job nodes through a graph isomorphic network (GIN) [35; 36]. GIN is utilized to learn the context information between one job and other jobs to obtain a high-dimensional embedded representation of each job. The graph encoder learns how messages are passed between jobs. That is, the graph encoder updates the representation (feature vector) of each job using information from the entire problem. Each layer of the improved GIN network is expressed as:  
\[\left\{ \begin{array}{c}x_{i}^{(l)}=\gamma^{(l)}\cdot\omega^{(l)}\cdot x_{i}^ {(l-1)}+\Big{(}1-\gamma^{(l)}\Big{)}\varphi^{(l)}\bigg{(}\frac{1}{|Adj(l)|}\Big{ }\Big{\{}x_{j}^{(l-1)}\Big{\}}_{j\in Adj(l)\cup\{i\}}\bigg{)},\;\forall l\in \{1,\ldots,L\}\\ x_{i}^{(0)}=x_{i}\end{array} \right. \tag{8}\]  
where \(x_{i}^{(l)}\in\mathbb{R}^{d_{l}}\) is the graph-encoded vector of job \(i\) at the \(l\)-th layer with \(l\in\{1,\ldots,L\}\), the eigenvalues of the weight matrix \(\omega^{(l)}\in\mathbb{R}^{d_{l-1}\times d_{l}}\) are regularized using a trainable variable \(\gamma^{(l)}\), \(Adj(i)\) denotes the set of all neighbouring nodes of node \(i\), the information between the two layers is aggregated using the function \(\varphi^{(l)}:\mathbb{R}^{d_{l-1}}\rightarrow\mathbb{R}^{d_{l}}\)[37]. The second equation in Equation (8) represents the first layer of the GIN network, where each node inputs a feature vector (basic information about each job). The first equation indicates that in other layers of the network, each node updates its own features by aggregating them with those of its surrounding neighboring nodes, and the specific aggregation function and weight matrix are trainable. The aggregation function is implemented as a neural network, aggregating information from nodes in the lower layers of the GIN to the next layer. The deeper the layer of the network, the larger the range of neighbor nodes that will be aggregated. The mechanism for embedding aggregation of a node vector in GIN can be represented simply as Figure 6. The figure approximately shows the aggregation and update process of the GIN network when \(L=2\). The information of the one-order and two-order neighbors of \(job1\) will be aggregated. Each update of a node requires a combination of information about the node itself and information about its neighbors, and the aggregation function and the weights of the summation operation are parameters that can be trained. Figure 5: A complete graph representation of the PFSP problem. Since the PFSP graph structure is defined as a complete graph, each layer in the GIN can in turn be represented as:  
\[X^{(l)}=\gamma^{(l)}\cdot\omega^{(l)}\cdot X^{(l-1)}+\left(1-\gamma^{(l)}\right) \cdot\varphi^{(l)}(\frac{X^{(l-1)}}{|Adj(i)|}) \tag{9}\]  
where \(X^{(l)}\in\mathbb{R}^{n\times d_{l}}\), and \(\varphi^{(l)}:\mathbb{R}^{n\times d_{l-1}}\rightarrow\mathbb{R}^{n\times d_{l }}\), \(n\) is the total number of jobs.