SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhengxin Zhang\*§‡, Yucheng Huang\*§‡, Gunaglin Duan\*§‡, Qing Li†‡, Dan Zhao‡ Yong Jiang‡, Lianbo Ma‡, Xi Xiao‡, Hengyang Xu
$^{\S}$ Tsinghua University,  $^{\ddagger}$ Peng Cheng Laboratory,  $^{\flat}$ Northeastern University
Tsinghua Shenzhen International Graduate School,  $\spadesuit$ Tencent
zhang-zx21@mails.tsinghua.edu.cn
Regular expressions (REs) offer one-shot solutions for many networking tasks, e.g., network intrusion detection. However, REs purely rely on expert knowledge and cannot learn from massive ubiquitous network data for automatic management. Today, neural networks (NNs) have shown superior accuracy and flexibility, thanks to their ability to learn from rich labeled data. Nevertheless, NNs are often incompetent in cold-start scenarios and too complex for deployment on network devices. In this paper, we propose Metis, a general framework that converts REs to network device affordable models for superior accuracy and throughput by taking advantage of REs' expert knowledge and NNs' learning ability. In Metis, we convert REs to byte-level recurrent neural networks (BRNNs) without training. The BRNNs preserve expert knowledge from REs and offer adequate accuracy in cold-start scenarios. When rich labeled data is available, the performance of BRNNs can be improved by training. Furthermore, we design a semi-supervised knowledge distillation to transform the BRNNs into pooling soft random forests (PSRFs) that can be deployed on network devices. We collect network traffic data on a large data center for three weeks and evaluate Metis on them. Experimental results show that Metis is more accurate than original REs and other baselines, achieving superior throughput when deployed on network devices.
Symbolic rules are indispensable and widely used in network scenarios (Wang & Zhang, 2021). As one of the most representative forms of symbolic rules, regular expressions (REs) are long-established in natural language processing (NLP) and network tasks, e.g. pattern matching (Hosoya & Pierce, 2001; Zhang & He, 2018), network measurement and anomaly detection (Kumar & Dharmapurikar, 2006; Sherry & Lan, 2015; Hypolite et al., 2020). RE-based systems are constructed based on expert knowledge and do not require labeled data for training. Therefore, RE-based systems are well-suited for zero-shot scenarios. However, the disadvantages of RE-based systems are also obvious. First, RE-based systems rely on experts to construct and update, making them hard to maintain. Second, RE-based systems cannot learn from labeled data, which usually contains underlying implications of the concerned problems beyond experts' subjective perception. Meanwhile, by exploiting labeled training data, many neural networks (NNs) models outperform RE-based systems on various tasks. For example, NNs achieve higher accuracies than RE-based systems in network intrusion detection (Fu et al., 2021) and application classification (Yu et al., 2020). However, in many network scenarios,
† Corresponding author: Qing Li (liq@pcl.ac.cn).
obtaining labeled data is challenging (Mirsky et al., 2018; Li et al., 2022), and the computational capabilities of network devices are limited, making it difficult to deploy NNs.
Recently, Jiang & Zhao (2020) tries to combine the advantages of symbolic rules and neural networks. They transform multi-string matching patterns into a trainable recurrent neural network (RNN), which can benefit from the high accuracy of RNN while being deployable in cold-start scenarios. However, multi-string matching patterns can only match simple strings, which is not sufficient to represent complex text patterns. Regular expression matching is a more versatile tool than multi-string matching for matching complex text patterns, thanks to the representation capacity offered by its syntaxes such as Kleene star (e.g.,  $*$ ), counting constraint (e.g.,  $\{\mathbf{N}, \mathbf{M}\}$ ), character class (e.g., "[a-fA-F0-9]", and hexadecimal numbers (e.g., "\xff"). As such, jointly exploiting the advantages of REs and NNs offers new opportunities for network device affordable solutions with high accuracy and throughput. Nevertheless, two challenges need to be addressed to achieve this high-level goal: 1) how to preserve expert knowledge from REs and develop learning models with adequate accuracy in cold-start scenarios; and 2) how to produce learning models that fit in the network devices while retaining good performance. Since network devices have limited computation and memory resources, deploying complex learning models transformed from regular expressions, e.g., NNs, in these devices is often difficult or even impossible.
In this paper, we propose Metis<sup>1</sup>, an intelligent and general framework that converts RE-based systems into learning models, which offer superior accuracy and throughput when deployed on network devices. Our codes are available at  $\mathsf{Gibth}^2$ . Metis preserves expert knowledge in RE-based systems for adequate cold-start performance and exploits NNs' ability to further improve performance using rich labeled data. First, we extract bytes from network packets as input features and construct byte-level deterministic finite-state automata (DFAs) from network rules, i.e., REs. Second, the DFAs are transformed into byte-level RNNs (BRNNs). Without training data, the BRNNs preserve the accuracy of RE-based systems for adequate performance in cold-start scenarios. When enough labeled data is collected, BRNNs' performance can be further improved by training with data. Third, we train pooling soft random forests (PSRFs) under the instruction of BRNNs using semi-supervised knowledge distillation (SSKD). Through the SSKD, we transfer the superior learning capability of BRNNs to PSRFs, which can be easily deployed on network devices for line-speed processing, i.e., 100 Gbps. To the best of our knowledge, this is the first method to explore the use of model inference as an alternative to RE matching in network scenarios. The key contributions of this paper are as follows: 1) We propose Metis, an intelligent general framework that converts RE-based systems to learning models deployable on network devices. 2) We develop a method for converting complex REs into DFA and subsequently into trainable BRNNs. This method allows us to maintain the accuracy of the RE-based system while leveraging the flexibility and capabilities of NNs. 3) We propose a novel semi-supervised distillation algorithm SSKD, and a student model PSRF for RE matching to achieve lightweight deployment of RE-based systems in network devices. 4) We collect network traffic data from a large data center for three weeks that reflects real network conditions. Experiments on the datasets demonstrate that Metis outperforms original RE-based systems and other baselines.
In this section, we introduce the network rules and modern network devices, highlight the problems of the state-of-the-art in these fields, and discuss the opportunities provided by knowledge distillation.
Network rules. Network rules serve as a fundamental building block for many network security applications, e.g., network intrusion systems (Roesch, 1999; Project, 2022), application identification systems (ntop, 2022), web firewalls (Trustwave, 2022) and some network censorship systems (Hoang & Niaki, 2021). REs are one of the most representative and useful forms of network rules. In many network security applications, such as Snort (Roesch, 1999) and Suricata (Foundation, 2022), REs are used to inspect whether the payload of a packet matches any predefined network rules. With high interpretability and no requirement for a training phase, RE-based systems can be quickly deployed with decent performance in network scenarios. However, the matching speed of regular expressions usually becomes a bottleneck since the pattern matching has to inspect every byte of a packet against
<sup>1</sup>Metis is a goddess of wisdom in Greek mythology.
$^{2}$ https://github.com/YouAreSpecialToMe/Metis.
a set of rules Wang & Zhang (2021). Moreover, RE-based systems cannot boost their accuracy by training on labeled data and thus often underperform learning-based models in data-rich scenarios.
Machine learning in-network. Learning algorithms, especially deep learning ones, have been employed in the network field to leverage their superior learning abilities, e.g., network traffic classification (Barradas & Santos, 2021), malicious traffic detection (Fu & Li, 2021) and flow size prediction (Poupart & Chen, 2016). As the network bandwidth grows dramatically, it has been increasingly difficult for network applications to keep up with the high traffic volume (Cisco, 2021). Unfortunately, existing model-based systems are unable to process high-throughput traffic, due to their processing overhead. Although deploying more servers could achieve higher throughput, doing so would increase the capital costs drastically, which is not symmetric to the rapid growth of network bandwidth and network traffic nowadays.
Modern commodity network hardware devices, e.g., programmable switches (PS) (Barefoot Networks, 2021; Yang et al., 2022) and smart network interface card (NIC) (Lin et al., 2020)) provide hardware programmability. They have comparable power consumption and capital costs as traditional fixed-function network devices, which enables orders of magnitude cost reduction compared to commodity CPU or other hardware alternatives (e.g., GPU and FPGA). However, as programmable network devices only come with limited computational and memory resources, deploying learning models directly on them is often infeasible.
Knowledge distillation. Knowledge distillation (KD) is widely adopted in model compression, transferring the knowledge of an unwieldy teacher model, which is hard to deploy on resource-constrained devices, into a lightweight student model (Gou et al., 2021). According to the definition of knowledge, existing KD methods can be classified into three types: response-based KD (Hinton et al., 2015; Chen et al., 2017), feature-based KD (Romero et al., 2014; Chen & Mei, 2021), and relation-based KD (Yim et al., 2017). Response-based KD methods have been applied in network scenarios since they support training a student model with a different structure from a teacher model to adapt to network devices. The main idea of the response-based KD is to let the student model mimic the prediction of the teacher model. Frosst & Hinton (2017) propose a soft decision tree that is transformed from NNs using response-based KD. Xie et al. (2022) propose Mousika that leverages the response-based KD methods to convert the deep learning models into decision trees, and extract the flow table entries for direct deployment on programmable switches. This opens up new horizons for bringing learning models into network devices. However, existing KD response-based methods rely on abundant labeled data, which is not available in network scenarios.
3 Metis
Figure 1: Metis framework.
In this section, we describe the Metis framework as shown in Figure 1. For the offline training, we first process the network traffic and construct DFAs with bytes from network rules. Then, we convert DFAs into BRNNs. Finally, we propose SSKD to train our PSRF under the instruction of BRNN. For the online inference, we deploy the PSRF on network devices to process traffic in line-speed.
3.1 Rule to DFA
In the field of NLP, the input is usually natural language text. Often, the input text is naturally tokenized by word delimiters, e.g., spaces in English text. Then, by using pre-trained word embeddings, dimension reduction can be easily conducted on features extracted from the tokenized text. However, the inputs in network scenarios are bit streams extracted from packets of network traffic.
In Metis, we utilize byte-level tokenization for network bit streams and network rules. The benefits of byte-level tokenization are two-fold. First, we can reduce the input vocabulary size to  $2^{8} = 256$ . Second, byte-level tokenization facilitates the deployment of the final PSRF in network devices since network devices typically process packets in bytes. For the incoming bit streams, we divide them into bytes and then convert bytes to hexadecimal numbers.
Network security applications, e.g., Snort (Roesch, 1999) and Suricata (Foundation, 2022), usually adopt Perl-compatible regular expression syntax (PCRE) (Philip Hazel, 2022) to construct network rules. PCRE includes more representative features than REs, such as counting constraints, character classes, and hexadecimal numbers. The syntaxes supported in this paper are shown in Appendix A. Note that (Jiang & Zhao, 2020) only supports the first five syntaxes. Taking “\x26cvv\x3d[0-9]{3,4}”, a simple rule in Snort to identify policy, as an example. Hexadecimal numbers will only appear after “\x”. Note that this rule can appear anywhere in a bit stream. So we first convert the rule to“..*\x26cvv\x3d[0-9]{3,4}.*,” where ‘.’ is the wildcard that can match any character. ‘*’ is the Kleene star operator to match the preceding subexpression zero or more times. Characters inside the brackets (e.g., “[0-9]”) form a character class module which ORs the characters included in a character class. To make the rule consistent with the input data format (i.e., bytes), we convert each character in the rule except character classes, wildcard, and counting constraint operators into its corresponding ASCII code, using only one byte per character. So the final converted rule is“..* 0x26 0x63 0x76 0x76 0x3d [0-9]{3,4}.*.”
Next, we construct deterministic finite-state automata (DFA) for the converted rules. Finite-state automata (FA) are mathematical models of computation characterizing transitions among a finite number of states. An RE can be converted into an FA using Thompson's construction algorithm (Thompson, 1968). We can construct a unique FA with a minimum number of states and deterministic transitions (DFA) for an RE by the DFA construction algorithm (Rabin & Scott, 1959) and DFA minimization algorithm (Hopcroft, 1971). Formally, a DFA is defined as a 5-tuple  $\mathcal{A} = (\Sigma, S, T, \alpha_0, \alpha_\infty)$ , whose elements are defined as:  $\Sigma$ : the input vocabulary. In the byte-level input process,  $|\Sigma| = V = 2^8 = 256$ ;  $\mathcal{S}$ : a finite set of states.  $|\mathcal{S}| = K$ ;  $T \in \mathbb{R}^{V \times K \times K}$ : transition weights.  $T[\sigma, s_i, s_j]$  is the weight of transferring  $s_i$  to  $s_j$  according to the input  $\sigma$ . In DFA,  $T[\sigma, s_i, s_j]$  is 1 indicates  $s_i$  can transfer to  $s_j$  otherwise 0;  $\alpha_0 \in \mathbb{R}^K$ : initial weights of  $\mathcal{S}$ .  $\alpha_0[i]$  is the initial weight of  $s_i$  when time  $t = 0$ ;  $\alpha_\infty \in \mathbb{R}^K$ : final weights of  $\mathcal{S}$ .  $\alpha_\infty[i]$  is the final weight of  $s_i$  after reading the whole input. Consider an input sequence  $\mathcal{X} = \{x_1, x_2, \dots, x_N\}$  and a path  $p = \{u_1, u_2, \dots, u_{N+1}\}$ , where  $u_i$  is the index of state considering  $x_i$ . The score  $\mathcal{B}(\mathcal{A}, p)$  of path  $p$  is defined as
$$
\mathcal {B} (\mathcal {A}, p) = \alpha_ {0} [ u _ {1} ] \cdot \left(\prod_ {i = 1} ^ {N} T [ x _ {i}, u _ {i}, u _ {i + 1} ]\right) \cdot \alpha_ {\infty} [ u _ {N + 1} ]. \tag {1}
$$
Let  $\pi (\mathcal{X})$  be the set of all possible paths starting from  $S_0$  and ending at  $S_{\infty}$ , where  $S_0$  is the set of start states and  $S_{\infty}$  is the set of final states. The sum of path scores,  $\mathcal{B}_{fw}(\mathcal{A},\mathcal{X})$ , can be computed by the Forward algorithm Baum & Petrie (1966):
$$
\mathcal {B} _ {\mathrm {f w}} (\mathcal {A}, \mathcal {X}) = \sum_ {\boldsymbol {p} \in \pi (\mathcal {X})} \mathcal {B} (\boldsymbol {A}, \boldsymbol {p}) = \boldsymbol {\alpha} _ {0} ^ {T} \cdot \left(\prod_ {i = 1} ^ {N} \boldsymbol {T} [ x _ {i} ]\right) \cdot \boldsymbol {\alpha} _ {\infty}. \tag {2}
$$
As mentioned above, we can build a DFA using the converted rules. Note that transitions in our DFA consist of bytes. To construct the DFA more concisely, when constructing the transformation matrix, we treat wildcards and character classes as special words in the input vocabulary for quick processing.
3.2 DFA to BRNN
Our DFA is parameterized by  $\Theta = < T, \alpha_0, \alpha_\infty >$ . Let  $h_t \in \mathbb{R}^K$  be the forward score considering the first  $t$  words  $\{x_1, x_2, \ldots, x_t\}$  of  $\mathcal{X}$ . We rewrite the forward score into a recurrent form:
$$
h _ {0} = \alpha_ {0} ^ {T}, \tag {3}
$$
$$
h _ {t} = h _ {t - 1} \cdot T [ x _ {t} ], 1 \leq t \leq N, \tag {4}
$$
$$
\mathcal {B} _ {\mathrm {f w}} (\mathcal {A}, \mathcal {X}) = h _ {N} \cdot \alpha_ {\infty}. \tag {5}
$$
Here, we treat matching on RE as a binary classification task, which can suit most network applications. To this end, we expand  $\mathcal{B}_{\mathrm{fw}}(\mathcal{A},\mathcal{X})$  to a vector  $[1,0]$  when  $\mathcal{B}_{\mathrm{fw}}(\mathcal{A},\mathcal{X})$  is 0, and  $[0,1]$  otherwise. It can be easily generalized to multi-classification tasks as well. Recall that the calculation of the hidden states in the forward propagation of the RNN is formulated by  $h_t = \sigma (Ux^t +Wh^{t - 1} + b)$ , where  $\sigma$  is the activation function. When  $U = 0,b = 0,W = T$ , and  $\sigma$  is the identity function, the forward score calculation of DFA (E.q. 4) is similar to the forward propagation of RNN. Therefore, we convert the DFA with byte-format transitions into an RNN with  $\Theta$ , called BRNN. In practice, fine-grained classification may be desired, which considers the different combinations of final states, i.e., different terminal state combinations correspond to different output categories. In this case, we can use an MLP after BRNN as the aggregation layer. The BRNN converted from DFA can retain the performance of the original RE and be put into production immediately without waiting for data collection. When enough labeled data is collected, the performance of the BRNN can be further improved through training.
3.3 BRNN to PSRF
The BRNN is often difficult to deploy on network devices directly. To promote easy deployment on network devices, we further convert BRNN into pooling soft random forest (PSRF) using semi-supervised knowledge distillation (SSKD).
Existing knowledge distillation approaches (e.g., Mousika) face two major problems when applied in network scenarios. First, labeled data only accounts for a very small proportion of the massive network traffic. It is challenging to distill a student model with high accuracy in such data-scarce scenarios. Second, due to the limited computation resources, memory, and supported operations of network devices, there are stringent restrictions on the selection of student models. Specifically, only tree-based models (e.g., the decision tree and the random forest) can be deployed. However, tree-based models are not suitable for unaligned and sequential features (payloads of network packets), making network tasks challenging, such as RE matching.
In Metis, we propose SSKD and PSRF, which overcome the above-mentioned problems with two key ideas. First, we introduce the semi-supervised learning strategy into the knowledge distillation. Attributing to the cold-start characteristic of our BRNN, the BRNN can effectively preserve the expert knowledge from the RE-based system and hence offers adequate accuracy without training data. As a result, we jointly utilize the ground truth of labeled data and inference results of BRNN on both labeled and unlabeled data to train the student model. Second, we select the PSRF as the student model instead of the soft decision tree. To exploit the context information within the sequence features, we introduce the pooling operation before training. Since the matched pattern in payloads may shift, leveraging context information can alleviate overfitting and boost accuracy.
SSKD aims to train the PSRF, consisting of several pooling soft trees (PST). The training phase of SSKD includes three steps. First, we perform  $t$  times of sampling with a replacement on all data (including both labeled data and unlabeled data) to obtain a sample set  $U = \{U_1, U_2, U_3, \ldots, U_t\}$ , where  $U_i = \{(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)\}$ ,  $i = 1, \dots, t$ , is the sample subset for the  $i$ -th tree,  $t$  is the number of trees in the random forest. Note that for labeled samples,  $y = (1,0)$  or  $y = (0,1)$ , and for unlabeled samples  $y = (-1, -1)$ . Although SSKD focuses on the binary classification task in this paper, it can be easily generalized to multi-classification tasks. The input feature  $x$  is composed of the decimal form of the bytes, i.e. integers ranging from 0 to 255. Second, we calculate the mixed label  $y_i^{mix}$ , and modify each item's label in the sample subset  $U_m$ , e.g.,  $(x_i, y_i)$  to  $(x_i, y_i^{mix})$ , where  $0 \leq m \leq t$  and  $0 \leq i \leq n$ . Specifically, the mixed label is calculated as
$$
y ^ {m i x} = \alpha \cdot y ^ {h a r d} + (1 - \alpha) \cdot y ^ {\text {s o f t}}, \tag {6}
$$
where  $\alpha$  is a hyperparameter ranging from 0 to 1,  $y^{hard}$  is the hard label of a sample (a tuple including true or false), and  $y^{soft}$  represents the soft label (a tuple composed of BRNN output probability). For
unlabeled data,  $y^{mix}$  is effectively the output of the BRNN. For labeled data,  $y^{mix}$  mixes the ground truth and the output of the BRNN, which helps the student model learn the classification ability of the teacher model. Finally, we use each sample subset  $S_{i}$  to train a PST. Our PSRF consists of all the trained PSTs and predicts output by majority vote. We can deploy PSRF on network devices by converting rules to general match action table entries.
Before training, we introduce the max pooling operation to leverage the context information. We do not apply mean pooling since mean pooling leads to an increase in feature complexity. We define the window as  $w$  and the stride as  $s$ . For each incoming feature sequences  $x = [f_1, f_2, f_3, \dots, f_{maxlen}]$ , the feature sequence is transformed to  $[\max(f_1, \dots, f_w), \max(f_{1+s}, \dots, f_{w+s}), \dots]$ , where  $f$  represents the feature of  $x$  and  $maxlen$  is the maximum length of payload sequences. However, the implementation of pooling operation is non-trivial in network devices due to their lack of support for complex float computing. To implement Metis on network devices, we first extract the packet bytes to the packet header vector through the parser. Then in the ingress pipeline, we use the match-action unit to perform a pooling operation on the packet header vector based on the ALU in parallel. Details of technical work can be referred to the technique report. The training process of each PST is similar to the CART decision tree (Breiman et al., 2017) which utilizes purity to split nodes. The purity after splitting the node via feature  $A$  is defined as
$$
P (D, A) = \frac {\left| D _ {l} ^ {A} \right|}{D} \operatorname {G i n i} \left(D _ {l} ^ {A}\right) + \frac {D _ {r} ^ {A}}{D} \operatorname {G i n i} \left(D _ {r} ^ {A}\right), \tag {7}
$$
where  $D$  is the sample set of the parent node, and  $D_{l}^{A}$  and  $D_{r}^{A}$  are the sample sets of the left child and right child split by feature A, respectively. The feature with the smallest  $P(D,A)$  will be used for node split. Note that different from the calculation method of the Gini index for hard labels, we calculate  $Gini(D)$  as
$$
G i n i (D) = 1 - \left(\frac {\sum_ {(x , y) \in D} y ^ {0}}{| D |}\right) ^ {2} - \left(\frac {\sum_ {(x , y) \in D} y ^ {1}}{| D |}\right) ^ {2}, \tag {8}
$$
where  $y = (y^0, y^1)$ . The detailed training process and end conditions of the PST are consistent with the CART decision tree. We mainly consider two hyperparameters: the number of features to consider when looking for the best split (# split features) and the minimum number of samples required to be at a leaf node (minimum samples). To deploy PSRF on network devices efficiently, we heuristically aggregates table entries into clusters based on the similarity among features represented by table entries so that each cluster only requires a more compact table.