## 3 Electricity Consumption Behavior Classification Based on CLB-BER
### BERT Classification Model
\[\text{E}=\text{E}_{\text{token}}+\text{E}_{\text{segment}}+\text{E}_{\text{ position}} \tag{22}\]  
The DistilBERT model is based on the Transformer encoder, which is stacked with multiple encoder layers. Each encoder layer is composed of a multi-head self-attention mechanism paired with a feed-forward neural network, and is connected by layer normalization and residual connection. The attention mechanism in the Transformer utilizes scaled dot-product attention, defined as  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\bigg{(}\frac{\mathbf{Q} \mathbf{K}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}}\bigg{)}\mathbf{V} \tag{23}\]  
where \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) represent the query, key, and value matrices, respectively; \(\text{d}_{\text{k}}\) denotes the dimensionality of the keys; and \(\mathbf{w}^{o}\) represents the output transformation matrix. The outputs from all attention heads are concatenated to produce a single multi-head attention output:  
\[\text{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Concat}(h\text{ead}_{1},h\text{ead }_{2},\dots,h\text{ead}_{h})\mathbf{W}^{\text{O}} \tag{24}\]  
\[h\text{ead}_{\text{i}}=\text{Attention}\Big{(}\mathbf{Q}\mathbf{W}^{\text{O}}_{i}, \mathbf{K}\mathbf{W}^{\text{K}}_{i},\mathbf{W}\mathbf{W}^{\text{V}}_{i}\Big{)} \tag{25}\]  
where \(h\) represents the number of headers, \(head_{i}\) represents the output of the head \(i\); and \(\mathbf{W}^{\text{O}}_{i}\), \(\mathbf{W}^{k}_{i}\), and \(\mathbf{W}^{\text{v}}_{i}\) represent the query, key, and value matrix of the head \(i\), respectively. Additionally, the decoder incorporates an extra masked multi-head self-attention layer, which ensures that the model does not access future information during sequence prediction [29]. Consequently, the final output of the decoder is expressed as:  
\[\text{DecoderLayer}(\text{y})=\text{LN}\Big{(}\text{y}+\text{M}_{\text{ MHA}(\text{y})}+\text{MHA}(\text{y},\text{x})+\text{FFN}(\text{y})\Big{)} \tag{26}\]where \(y\) denotes the input sequential data, \(x\) refers to the output sequence from the encoder, MHA signifies the multi-head self-attention layer, FFN represents the feed-forward layer, LN indicates the layer normalization layer, and M\({}_{\text{MHA}}\) denotes the masked multi-head self-attention layer. Based on the feature extraction, we fine-tune the DistilBERT model to adapt to the electricity data classification task. The fine-tuning process consists of adding a classification layer after the feature extraction layer, and the output of the classification layer is the probability distribution of the class to which the user belongs. The classification layer is formulated as  
\[P(y|H)=Softmax(WH+b) \tag{27}\]  
where \(P(y|H)\) is the conditional probability distribution over the classes, and \(W\) and \(b\) are the weights and biases of the classification layer, respectively. In the fine-tuning process, the labeled data is used for supervised training, and the optimization objective is to minimize the cross-entropy loss:  
\[L_{CE}=-\sum_{i=1}^{N}\sum_{c=1}^{C}y_{ic}logP(y_{ic}|H_{i}) \tag{28}\]  
where N represents the number of samples, \(C\) is the number of classes, \(y_{ic}\) represents the true label of the \(i\)-th sample belonging to class C, and \(P(y_{ic}|H_{i})\) is the probability predicted by the model.