## 3 Electricity Consumption Behavior Classification Based on CLB-BER
### BERT Classification Model  
Pre-trained BERT architectures, renowned for their efficacy, are amenable to both tuning and fine-tuning for diverse linguistic tasks, such as question answering and language identification [26]. BERT is acknowledged as a benchmark model in natural language processing due to its superior performance metrics across various tasks [27]. The adaptation process involves two critical phases: pre-training and fine-tuning. In the pre-training phase, BERT models are trained on a broad array of tasks using unlabeled data, facilitating the extraction of robust features. Subsequently, in the fine-tuning phase, these models are further optimized by adjusting pre-trained parameters using specifically prepared labeled data. Notably, the core architecture remains consistent across both phases, with modifications confined to the output layer to accommodate specific task requirements [28]. In this paper, we adopted the open-source pre-trained DistilBERT model for user electricity consumption behavior classification. DistilBERT is a stripped-down version of BERT with fewer parameters and faster training, but retains most of the performance of BERT. To enhance its classification performance, we embed the electricity consumption data, user ID, and month data into the model. The architecture of DistilBERT is structured into three primary layers: the word embedding layer, the encoding layer, and the classification layer. The first layer integrates token, segmentation, and position embeddings to convert input data into vector representations. Subsequently, the encoding layer, comprising six Transformer units, transforms these embeddings into contextual representations, enhancing the model's grasp of contextual semantics. Finally, the classification layer, which includes a fully connected layer and a Softmax layer, utilizes these contextual representations to classify and predict the appropriate categories. Figure 7 illustrates the DistilBERT classification model. To adapt the numerical electricity consumption data for use with the DistilBERT model, we designed a tailored tokenization and embedding strategy. Since DistilBERT was originally designed for processing text inputs, we first transformed the monthly electricity usage values, user IDs, and month identifiers into a structured pseudo-sentence format using fixed templates. These structured sentences were then tokenized using the standard DistilBERT tokenizer, which mapped the text to token IDs based on pre-trained vocabulary. Figure 7: DistilBERT classification model. After tokenization, the embedded vectors were generated via the DistilBERT embedding layer, which combined token, segment, and positional embeddings. This design ensures that the numerical information is contextually encoded, enabling the model to learn temporal and consumption-related patterns across different user behaviors. Each input word \(w_{i}\) is mapped to a high-dimensional vector space, resulting in the corresponding word embedding vector \(\text{E}_{\text{token}}=\text{TokenEmb}(\text{w}_{\text{i}})\). To distinguish between the two sentences, BERT uses Segment Embedding to mark which sentence each word belongs to. Finally, BERT uses positional encoding to introduce the order information between words. The position encoding is calculated by the following formula, and the encoding of each position in the input sequence is  
\[PosEmb_{(pos,2i)}=sin\bigg{(}\frac{pos}{10000^{\frac{2}{2}}}\bigg{)} \tag{20}\]  
\[PosEmb_{(pos,2i+1)}=cos\bigg{(}\frac{pos}{10000^{\frac{2}{2}}}\bigg{)} \tag{21}\]  
where \(d\) is the dimension of the embedding vector and \(i\) represents the position-encoding dimension index. The three parts of the word embedding layer convert all of the electricity consumption data, user identity data, and month information into vector representation, distinguish different parts of the input data, and mark the relative position in the input sequence. The comprehensive embedding is expressed as in (22).