# Deep Learning Framework for Advanced De-Identification of Protected Health Information
## 3 Methodology
### Bi-LSTM-CRF Model Architecture, Hyperparameter Tuning, and Optimization  
The Bi-LSTM-CRF model developed for this research integrates a Bi-LSTM network with a Conditional Random Field (CRF) layer to enhance sequential tagging tasks by capturing both contextual dependencies and transition constraints between output tags. The model's key components are as follows:  
* **Embedding Layer**: The input words are transformed into dense vector representations using an embedding layer with a dimensionality range of 50 to 300. Figure 2: Bi-LSTM model. * **Bi-LSTM Layer**: A multi-layer bidirectional LSTM network is employed with a hidden dimension ranging from 50 to 512, to capture both forward and backward contexts for each token. * **CRF Layer**: A transition matrix, initialized randomly, scores transitions between tags, including constraints to prevent invalid transitions to and from designated start and stop tags. **Hyperparameter Tuning**  
Key hyperparameters were fine-tuned to balance performance and computational efficiency:
* **Learning Rate**: Ranges from 0.001 to 0.05, optimizing model convergence while maintaining stability. * **Weight Decay**: Regularization is applied with a weight decay parameter ranging from 1 \(\times 10^{-4}\) to 1 \(\times 10^{-2}\), effectively controlling overfitting. * **Batch Size**: The model processes samples in batches, with a batch size ranging from 8 to 64, enhancing computational efficiency and convergence. * **Epochs**: Training is performed over 300 epochs to ensure sufficient learning while monitoring for convergence. **Optimization**  
The model training utilizes Stochastic Gradient Descent (SGD), selected for its straightforward implementation and effective balance of convergence speed and stability. The design allows for the potential inclusion of gradient clipping to manage exploding gradients, a common consideration in Bi-LSTM models. The Stochastic Gradient Descent (SGD) optimizer in this model is configured with the following key parameters:  
* **Learning Rate**: This is set within the range of 0.001 to 0.05 to balance the learning speed with convergence stability. This range is selected based on empirical observations, providing consistent convergence without overshooting. * **Weight Decay**: This is set within the range of \(1\times 10^{-5}\) to \(1\times 10^{-2}\) as a regularization term, which helps control overfitting by penalizing large weights during training. This Bi-LSTM-CRF architecture thus provides a robust framework for sequence labeling, leveraging the ability of Bi-LSTM to model contextual dependencies and the structured output layer of the CRF to enforce tag transition rules as summarized in Table 2.  
Model Training and Evaluation Metrics  
The dataset consists of 1304 records, which are split into 790 training records and 514 testing records, equivalent to a 60% to 40% split, as presented in Table 3. Each record represents specific medical information about individual patients. In total, there are 269 unique patients, and each patient's data are represented by 3 to 5 files of XML type. The file naming convention follows a pattern, denoted as (XXX-YY.xml), where XXX indicates the patient number and YY represents the record number within that patient's dataset. The dataset structure is designed to efficiently manage and differentiate the medical records associated with each patient, coherently facilitating comprehensive analysis and research. The Bi-LSTM-CRF model is trained using the i2b2 dataset over iterative cycles, referred to as epochs. Each epoch represents a full pass through the training dataset, during which the model's parameters, including weights, are updated based on the input data. After one epoch, the model processes each sample in the training set once, allowing for gradual improvement in its performance through repeated evaluations. This iterative training process enables the model to enhance its data processing capabilities with each cycle, progressively refining its ability to handle complex inputs.