# Fine-Tuning a Personalized OpenBioLLM Using Offline Reinforcement Learning
## 3 Methods
### Template Based on Chain of Thought
By simulating the thinking process of human doctors, the model can comprehensively consider the patient's physiological indicators, medical history, and other relevant medical information, so as to conduct a more in-depth and comprehensive analysis, which helps to generate personalized and accurate treatment recommendations. Furthermore, by clearly showing the reasoning process and decision-making basis of the model, it provides valuable reference for medical professionals, helping them make more informed choices in clinical decision making. ### Fine-Tuning Process

This section elaborates on the entire fine-tuning process. The fine-tuning process can be divided into two stages. In the first stage, the DTE module and the InfCTI module are pre-trained to obtain part of the state space and reward function. In the second stage, the OpenBioLLM module and the SAC-CQL module are iteratively trained. In this stage, OpenBioLLM will generate training data based on the output of SAC-CQL, and SAC-CQL will learn the strategy based on the state space generated by OpenBioLLM. The two are iterated repeatedly until the treatment plan generated by OpenBioLLM is consistent with the treatment plan generated by SAC-CQL. The pseudo-code of the entire fine-tuning process is shown in Algorithm 5. ```
1:Initialization:
2:- LLM module: Load OpenBioLLM-8B model
3:- DTE module: Random parameter initialization
4:- InfCTI module: Random parameter initialization
5:- SAC-CQL module: Random parameter initialization
6:Step 1: Training of DTE and InfCTI modules, construction of some datasets:
7:- Use the sorted patient dataset to independently train the DTE module to generate the state_DTE dataset required for the state space
8:- Independently train the InfCTI module to generate the reward dataset
9:- Generate input data for OpenBioLLM using the cot_2 prompt template
10:- OpenBioLLM analyzes patient conditions and generates treatment plans P_treatment
11:- Extract feature tensors from the penultimate layer of OpenBioLLM to construct the state_LLM dataset
12:Step 3: Construction of Markov decision process dataset and training of SAC-CQL module:
13:- Splice state_DTE and state_LLM to form a complete state space
14:- Combine the reward value and action space obtained by InfCTI to train the SAC-CQL module
15:- Compare whether the treatment decision output by the SAC-CQL module is consistent with the treatment decision in p_treatment generated by OpenBioLLM:
16:If consistent, training ends
17:If inconsistent, go to step 4
18:Step 4: Fine-tuning and iterative optimization of OpenBioLLM module:
19:- The patient's condition and the treatment decision of SAC-CQL are formed into input data through the cot_1 prompt template, and the target dataset t_treatment is obtained through OpenBioLLM
20:- The patient's condition is formed into input data through the cot_2 prompt template, output p_treatment, and calculate the loss with t_treatment
21:- Fine-tune OpenBioLLM
22:- Repeat steps 2 and 3 After the fine-tuning process above, we can get a fine-tuned OpenBioLLM and treatment model. OpenBioLLM can receive text information and output text information, and the treatment model can receive the patient's numerical information and generate treatment decisions.