## 3 Proposed Blockchain and Digital Twin-Based Heuristic Multi-Cooperation Scheduling Framework
### Blockchain-Based Data Validation
#### 4.3.2 Profit Sharing DRL (with DQN)

The capacities of the sensor and camera systems determine how accurately actions are taken in the smart factory storage DT. For the DRL model to function better, the footage and sensor must guarantee a certain level of precision. However, an increase in the precision of sensors and cameras means an increase in candidate actions (possible actions to be taken by the DRL model), which also significantly correlated with increased accuracy. If a traditional DRL method is used to model the system, initializing the weights will be laborious and challenging to maintain and preset manually. That will have an impact on how effectively the algorithms converge. DQN successfully uses the benefits of neural networks to supplement RL The approach uses the Q-learning target-value function to create the DL tags for creating the loss function to merge these two models efficiently. _DQN_ uses experience replay to address this issue of data correlation. Additionally, two main networks are constructed. To build a loss function, one network creates the current \(\mathbf{Q}\) value (\(\mathbf{QM}\)), and the second generates the targeted \(\mathbf{Q}\) value (\(\mathbf{QT}\)). This is very is crucial for neural network training. Consequently, the following equation shows how the DL loss function may be obtained:

\[\mathbf{DQN}-\mathbf{Loss}\;\mathbf{L}(\mathbf{\theta}t)=(\mathbf{QT}-\mathbf{QM}\;)^{2} \tag{1}\]

Then, the Bellman equation can be used to update the target network (\(\mathbf{QT}\)) as follows:

\[\mathbf{QT}=\mathbf{r}+\mathbf{\gamma}\;\mathbf{max}(\mathbf{a}-\mathbf{r})\;\mathbf{Q}(\mathbf{sr},\;\mathbf{ar},\;\mathbf{\theta}) \tag{2}\]

where the reward amount is denoted by (\(\mathbf{r}\)) and (\(\mathbf{\theta}\)) denotes the iterative process technique, and (\(\mathbf{\gamma}\)) indicates the discount percentage. For our work, we take (\(\mathbf{ar}\)) as the presentlyavailable actions, and we execute an action (\(\mathbf{ar}\)) to go to the next state, which is (\(\mathbf{sr}\)). Despite the addition of DL, which may significantly increase the effectiveness of high-dimensional feature characteristics extraction, the intrinsic flaw in RL persists, necessitating several iterations until convergence. To maximize the advantages of experience replay, a model is supposed to save as many samples as possible. However, when employing DRL to explore the environment, previous data retrieved from replay memory will impact the agent's adaptation to the new environment. On the other hand, profit sharing (PS) may minimize past knowledge's influence on RL by introducing an additional term at the end of the iterative equation. #### 4.3.3 PDQN vs. D.Q.N. Performance Evolution

We made a comparison of various methods, including convergence time (ConvT), average convergence (AveConv), and variance convergence (VarConv), both considering a total of 1000 iterations. Our evaluation begins with varying learning rates (0.1, 0.01, and 0.001). The comparison shows that it considerably impacts convergence stability, as shown in Figure 6. It is worth noting that the simulations with varied parameters are an average of 25 simulation runs 25. The same approach was adopted for the following experiments discussed later in this section. The results demonstrate that a stable operating condition may be maintained during the iterative procedure when the learning rate is 0.01. We conducted tests with classic DQN and PDQN. PDQN requires a shorter convergence time (6.136) than classical DQN (7.528). It performs admirably in terms of stability and dependability and is more stable (10.202) and reliable (68.248) than DQN, which has values of 7.326 and 25.549 for stability and reliability, respectively. For the reliability measure, it indicates the degree of oscillation during learning of the model. The higher the score, the less severe the deviation or variance of operations during the transition between convergence states.