# Fine-Tuning a Personalized OpenBioLLM Using Offline Reinforcement Learning
## 3 Methods
### OpenBioLLM Module  
This module has three key functions: generating fine-tuning data labels, obtaining partial state space, and providing treatment plans. Its core components are the pre-trained medical model OpenBioLLM and two prompt templates based on chain of thought [26] cot_1 and cot_2, which will be described in detail in Section 3.5.  
Generate fine-tuning data labels: Apply the cot_1 prompt template to guide the pre-trained medical model OpenBioLLM to generate a detailed treatment decision analysis based on given patient information, physiological indicators, and treatment decisions, which is used as a supervisory signal for model fine-tuning, denoted as t_treatment.  
Obtain partial state space: Apply the cot_2 prompt template to provide the necessary state representation for the SAC-CQL module. This state representation comes from the penultimate layer of the model, denoted as state_LLM, which contains the medical knowledge inside the OpenBioLLM medical model and provides key information for the CQL algorithm to generate personalized treatment plans. The dimension of the obtained state_LLM is 4096.  
Provide treatment plans: While applying the cot_2 prompt template, the OpenBioLLM medical model provides doctors with a comprehensive analysis of the patient's condition and treatment recommendations through structured text information, denoted as P_treatment, to assist doctors in making more accurate clinical decisions.  
The OpenBioLLM medical language model is fine-tuned based on Llama3-8B, so the model architecture of the network is basically the same as that of Llama3-8B. The model first undergoes word vector embedding, then enters 32 decoder blocks, and then passes through RMS (Root Mean Square) normalization and linear layers. One path connects to the Softmax layer, and the other path connects to the global average pooling layer. Since the size of its word embedding vector is 4096 dimensions, the vector dimension obtained after the global average pooling layer is 4096. In the decoder block, the rotary position encoding is used, which encodes the embedding by adding absolute position information and containing relative position information between tokens, thereby maintaining the order of tokens in the sentence. Grouped attention query is used to reduce the number of parameters that need to be stored, and the SwiGLU activation function copes with negative values. The module structure is shown as Figure 2. The pseudo-code for fine-tuning the OpenBioLLM module is shown in Algorithm 1.  
Figure 1: Fine-tuning of the overall model architecture.  
```
1:Input Pre-trained large language model OpenBioLLM, training dataset \(\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}\), learning rate \(\alpha\), number of fine-tuning rounds \(T\)
2:Output: Fine-tuned language model parameters \(\theta\)
3: Load pre-trained large language model parameters \(\theta_{0}\)
4:for\(t=1\) to \(T\)do
5:for Each training sample \((x_{i},y_{i})\in\mathcal{D}\)do
6: Input sample \(x_{i}\) to the model and calculate the output probability distribution \(p(y|x_{i};\theta_{t})\)
7: Calculate the cross entropy loss function value: \[L(\theta_{t})=-\sum_{i=1}^{N}\log p(y_{i}|x_{i};\theta_{t})\]
8: Back propagation calculates the gradient \(\
abla_{\theta_{t}}L(\theta_{t})\)
9: Update the model parameters according to the learning rate: \[\theta_{t+1}=\theta_{t}-\alpha\
abla_{\theta_{t}}L(\theta_{t})\]
10:endfor
11:endfor
12: Return the fine-tuned language model parameters \(\theta\)
```  
**Algorithm 1** Training OpenBioLLM