## 3 Methods and Model
### Model  
We utilize a multi-objective DQN as the RL controller to more effectively address the challenge of personalized dose. Since DQN has the ability to handle a number of state spaces and action spaces, its outcome should be improved by more accurately defining the state space. The state of the system is defined as \(s(t)=\{s_{1}(t),s_{2}(t),s_{3}(t)\}\), where \(t\) denotes time. \(s_{1}(t)\), \(s_{2}(t)\), and \(s_{3}(t)\) are given by (31)-(33).  
\[s_{1}(t)=X_{1}(tP)-X_{1d} \tag{31}\]  
\[s_{2}(t)=X_{2}(tP) \tag{32}\]  
\[s_{3}(t)=X_{4}(tP) \tag{33}\]  
where \(P\) is the representation of the sample period. Tumor cells, effector-immune cells, and drug concentration are represented by the symbols \(s_{1}(t)\), \(s_{2}(t)\), and \(s_{3}(t)\), respectively. The tumor cells' ideal value, \(X_{1d}\), can also be thought of as the treatment's final state. In accordance with the [7], \(X_{1d}\) is set to 1, so the ultimate state at the end of treatment stated to be \(s_{1}(t)\) is less than 0 in this paper. For the drug dose \(I(t)\), the restriction was given in Table 1, and the drug dose should be kept within the range of [0,1]. The setting of the action space should be subdivided as much as possible in order to discover the appropriate dose for patients in different states. Hence, in this work, action space \(A(*)=[0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]\), which has satisfied the patient's medicine needs. The adjustment of the drug dose is important because the RL controller's ultimate purpose is to keep the number of effector-immune cells at a high level while treating tumor cells. Because the only external factor that can affect the number of circulating lymphocytes is the drug dose, the number of circulating lymphocytes and the number of effector-immune cells can be regarded as the same goal. The goal of maintaining the number of effector-immune cells in the system is the same as maintaining the number of circulating lymphocytes. The setting for the reward function \(R\) is depicted in (34) and (35).  
\[r_{1}=\frac{X_{1}(tP)-X_{1}((t+1)P)}{\max X_{1}} \tag{34}\]  
\[r_{2}=\frac{X_{2}((t+1)P)-X_{2}(tP)}{\max X_{2}}+loss(t+1) \tag{35}\]  
The rewards from a change in the tumor cell cells and effector-immune cells are represented by \(r_{1}\) and \(r_{2}\). The two reward functions are normalized using parameters \(\max X_{1}\)  
Figure 4: Process of multi-indicator experience replay.  
and max \(X_{2}\), where max \(X_{1}\) represents the maximum value of tumor cells [7] and max \(X_{2}\) represents the maximum value of effector-immune cells [12], respectively. A constraint \(loss(t)\) is added, since our purpose is to prevent large variations of effector-immune cells in neighboring cycles and guarantee that the patient's effector-immune cell count is greater than the lowest threshold that is acceptable to the patient. The \(loss(t)\) was calculated by (36)-(38).  
\[loss(t)=\left\{\begin{array}{ll}z,&X_{2}(tP)\leq X_{2 tsh}\text{ or }gap\geq 20\%\\ 0,&\text{otherwise}\end{array}\right. \tag{36}\]  
\[gap=\frac{X_{2}((t-1)P)-X_{2}(tP)}{X_{2}((t-1)P)} \tag{37}\]  
\[z=\left\{\begin{array}{ll}-10\cdot I(t-1),&X_{2}(tP)\leq X_{2}((t-1)P)\\ 0,&\text{otherwise}\end{array}\right. \tag{38}\]  
where \(X_{2 tsh}\) is lowest threshold. The agent must make difficult choices from a range of rewards based on the problem modeling in this work, and make sure that the patient's cancer therapy is successful while assuring that the number of effector-immune cells declines within a specific level.