Linear Classifier: An Often-Forgotten Baseline for Text Classification
Year: 2023

Abstract:
Large-scale pre-trained language models such as BERT are popular solutions for text classification. Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model. In this opinion paper, we point out that this way may only sometimes get satisfactory results. We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. First, for many text data, linear methods show competitive performance, high efficiency, and robustness. Second, advanced models such as BERT may only achieve the best results if properly applied. Simple baselines help to confirm whether the results of advanced models are acceptable. Our experimental results fully support these points.

heading: 1 Introduction
text: Text classification is an essential topic in natural language processing (NLP). Like the situations in most NLP tasks, nowadays, large-scale pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) have become popular solutions for text classification. Therefore, we have seen that many practitioners directly run pre-trained language models with a fixed number of epochs on their text data. Unfortunately, this way may only sometimes lead to satisfactory results. In this opinion paper, through an intriguing illustration, we argue that for text classification, a simple baseline like linear classifiers on bag-of-words features should be used along with the advanced models for the following reasons. • Training linear classifiers such as linear
SVM (Boser et al., 1992) or logistic regression on bag-of-words features is simple and efficient. This approach may give competitive performance to advanced models for some problems. While various settings of bag-of-words features
such as bi-gram or tri-gram can be considered, we advocate that simple uni-gram TF-IDF features trained by linear classifiers can be a useful baseline to start with for text classification.
• Advanced architectures such as BERT may only achieve the best results if properly used. Linear methods can help us check if advanced methods’ results are reasonable.
In the deep-learning era, the younger generation often thinks that linear classifiers should never be considered. Further, they may be unaware of some variants of linear methods that are particularly useful for text classification (see Section 3.1). Therefore, the paper serves as a reminder of this oftenforgotten technique.
For our illustration, we re-investigate an existing work (Chalkidis et al., 2022) that evaluates both linear SVM and pre-trained language models, but the authors pay more attention to the latter. The linear method is somewhat ignored even though the performance is competitive on some problems. We carefully design experiments to compare the two types of methods. Our results fully demonstrate the usefulness of applying linear methods as simple baselines.
Some recent works (e.g., Yu et al., 2022; Gomes et al., 2021) have shown the usefulness of linear classifiers in the deep-learning era. However, they either consider sophisticated applications or investigate advanced settings in which linear methods are only one component. In contrast, in this paper, we consider the basic scenario of text classification. A more related work (Wahba et al., 2023) has demonstrated the effectiveness of linear classifiers over PLMs on some problems. However, our investigation on linear methods is more comprehensive.
The discussion also reminds us the trade-off between performance gain and the cost including running time, model size, etc. Simple methods are useful to benchmark and justify the usage of advanced methods.
1876
This paper is organized as follows. In Section 2 we take a case study to point out the needs of considering linear methods as a baseline for text classification. We describe the linear and BERT-based methods used for investigation in Section 3. The experimental results and main findings are in Section 4, while Section 5 provides some discussion. Additional details are in Appendix. Programs used for experiments are available at https://github.com/JamesLYC88/ text_classification_baseline_code.

heading: 2 Text Classification These Days: Some Issues in Applying Training Methods
text: Large PLMs have shown dramatic progress on various NLP tasks. In the practical use, people often directly fine-tune PLMs such as BERT on their data for a few epochs. However, for text classification, we show that this way may not always get satisfactory results. Some simple baselines should be considered to know if the obtained PLM model is satisfactory. We illustrate this point by considering the work on legal document classification by Chalkidis et al. (2022), which evaluates the following sets.
• Multi-class classification: SCOTUS, LEDGAR; for this type of sets, each text is associated with a single label.
• Multi-label classification: ECtHR (A), ECtHR (B), EUR-LEX, UNFAIR-ToS; for this type of sets, each text is associated with multiple (or zero) labels.
• Multiple choice QA: CaseHOLD.
We focus on text classification in this work, so CaseHOLD is not considered. For each problem,
training and test sets are available.1
The study in Chalkidis et al. (2022) comprehensively evaluates both BERT-based PLMs and linear SVMs. They use Micro-F1 and Macro-F1 to measure the test performance.2 In Table 1, we present their Micro-F1 results and running time of each model.

heading: 2.1 Linear Models Worth More Investigation
text: The investigation in Chalkidis et al. (2022) focuses on BERT and its variants, even though from Table 1, the performance of BERT-based methods may not differ much. While they did not pay much attention to linear SVM, by a closer look at the results, we get intriguing observations: • Linear SVM is competitive to BERT-based
PLMs on four of the six data sets. For SCOTUS, linear SVM even outperforms others with a clear gap.
• Surprisingly, given linear SVM’s decent performance, its training time was not shown in Chalkidis et al. (2022), nor was the number of parameters; see the “N/A” entries in Table 1.
With the observations, we argue that the results of linear models are worth more investigation.

heading: 3 Settings for Investigation
text: To better understand the performance of linear models and BERT-based PLMs, we simulate how people work on a new data set by training these methods. We consider a text classification package LibMultiLabel3 because it supports both types of train-
1Indeed, training, validation, and test sets are available. See details in Appendix H about how these sets are used.
2Some data instances are not associated with any labels; see Appendix A about how Chalkidis et al. (2022) handle such situations.
3https://github.com/ASUS-AICS/LibMultiLabel
ing methods.

heading: 3.1 Linear Methods for Text Classification
text: To use a linear method, LibMultiLabel first generates uni-gram TF-IDF features (Luhn, 1958; Jones, 1972) according to texts in the training set, and the obtained factors are used to get TF-IDF for the test set. It then provides three classic methods that adopt binary linear SVM and logistic regression for multi-class and multi-label scenarios.4 Here we consider linear SVM as the binary classifier behind these methods.
• One-vs-rest: This method learns a binary linear SVM for each label, so data with/without this label are positive/negative, respectively. Let fℓ(x) be the decision value of the ℓ-th label, where x is the feature vector. For multi-class classification, ŷ = argmaxℓ fℓ(x) is predicted as the single associated label of x. For multi-label classification, all labels ℓ with positive fℓ(x) are considered to be associated with x. This method is also what “TF-IDF+SVM” in Chalkidis et al. (2022) did, though our TF-IDF feature generation is simpler than theirs by considering only uni-gram.5
• Thresholding (Yang, 2001; Lewis et al., 2004; Fan and Lin, 2007): This method extends one-vsrest by modifying the decision value for optimizing Macro-F1. That is, we change the decision value to fℓ(x) + ∆ℓ, where ∆ℓ is a threshold decided by cross validation.
• Cost-sensitive (Parambath et al., 2014): For each binary problem, this method re-weights the losses on positive data. We decide the reweighting factor by cross validation to optimize Micro-F1 or Macro-F1.
These methods basically need no further hyperparameter tuning, so we can directly run them. The last two methods are extensions of one-vs-rest to address the imbalance of each binary problem (i.e., few positives and many negatives). The design relies on the fact that the binary problems are independent, so such approaches cannot be easily applied to deep learning, which considers all labels together in a single network.
4Besides descriptions in this section, some additional details are in Appendix B.
5See details in Appendix C.

heading: 3.2 BERT-based Methods for Text
text: Classification
LibMultiLabel also provides BERT-based methods, which involve several hyper-parameters, such as the learning rate. While practitioners may directly choose hyper-parameters, to seriously compare with linear methods, we run BERT by conducting hyper-parameter selection. More details are in Appendix F.

heading: 4 Experimental Results and Analysis
text: In Table 2, we follow Chalkidis et al. (2022) to report Micro-F1 and Macro-F1 on the test set. The training time is in Table 3.

heading: 4.1 Linear Methods are Good Baselines
text: In Table 2, our one-vs-rest results are slightly worse than the linear SVM results in Chalkidis et al. (2022), which also applies the one-vs-rest strategy. As mentioned in Section 3.1, the difference is mainly due to our use of simple uni-gram TF-IDF features. Anyway, our one-vs-rest is still competitive to BERT results in Chalkidis et al. (2022) on the last four problems.
More importantly, the two extensions of one-vsrest (i.e., thresholding and cost-sensitive) improve almost all situations. For data sets ECtHR (A) and ECtHR (B), where originally one-vs-rest is significantly lower than BERT results in Chalkidis et al. (2022), the gap reduced considerably.
For the training time in Table 3, though the two extensions take more time than the basic one-vsrest strategy, all the linear methods are still hundreds of times faster than BERT. Further, linear methods were run on a CPU (Intel Xeon E5-2690), while for BERT we need a GPU (Nvidia V100). The model sizes listed in Table 4 also show that linear SVM requires a much smaller model than BERT, where details of our calculation are in Appendix D.
The results demonstrate that linear methods are useful baselines. They are extremely simple and efficient, but may yield competitive test performance.

heading: 4.2 Linear Methods can Help to See if Advanced Methods Are Properly Used
text: Surprisingly, our running of LibMultiLabel’s BERT leads to worse test performance than linear methods on almost all data sets. More surprisingly, a comparison between the BERT results by LibMultiLabel and those in Chalkidis et al. (2022) shows
that the former is much worse on data sets ECtHR (A) and ECtHR (B). Interestingly, from Section 4.1, only for these two sets the BERT results in Chalkidis et al. (2022) are much better than linear methods. Thus, our direct run of BERT in LibMultiLabel is a total failure. The training time is much longer than linear methods, but the resulting model is worse.
It is essential to check the discrepancy between the two BERT results. We find that Chalkidis et al. (2022) use some sophisticated settings to run BERT for the first three sets (i.e., ECtHR (A), ECtHR (B), and SCOTUS). They split every document into 64 segments, each of which has no more than 128 tokens, and apply BERT on each segment. Then,
they collect the intermediate results as inputs to an upper-level transformer. After repeating the same process via LibMultiLabel, we can reproduce the results in Chalkidis et al. (2022); see details in Appendices E, F, and G.
We learned that they considered the more sophisticated setting of running BERT because by default, BERT considers only the first 512 tokens. Thus, for long documents, the training process may miss some important information. However, in practice, users may forget to check the document length and are not aware of the need to apply suitable settings. The above experiments demonstrate that BERT can achieve superior results if properly used, but sometimes, a direct run lead to poor outcomes. Linear
methods can serve as efficient and robust baselines to confirm the proper use of an advanced approach.

heading: 5 Discussion and Conclusions
text: In our experiments, we encounter an issue of whether to incorporate the validation set for training the final model, which is used for predicting the test set. For linear methods, we follow the common practice to include the validation set for obtaining the final model. However, for BERT or some other deep learning models, the validation set is often used only for selecting the best epoch and/or the best hyper-parameters. To fully use the available data, we have investigated how to incorporate the validation set for BERT. Experimental results and more details are in Appendix H.
For some text sets evaluated in this work, we have seen that simple linear methods give competitive performance. The reason might be that each document in these sets is not short.6 Then TF-IDF features are sufficiently informative so that linear methods work well. Across all NLP areas, an important issue now is when to use PLMs and when not. We demonstrate that when PLMs may not perform significantly better, traditional methods are much simpler and require fewer resources. However, having a simple quantitative measurement to pre-determine when to use which remains a challenging future research problem. In summary, the study reminds us of the importance of employing simple baselines in NLP applications.
6See details in Appendix Table 6.
Limitations
In this work, we do not propose any new methods because, as an opinion paper, we focus on raising the problems and making vivid demonstrations to readers. The experiments are limited to linear SVM and BERT on data sets in the benchmark LexGLUE. We hope that, within the page limit, our experiments sufficiently convey the points to readers.
Ethics Statement
We ensure that our work complies with the ACL Ethics Policy.

heading: Acknowledgements
text: This work was supported by NSTC of Taiwan grant 110-2221-E-002-115-MY3 and ASUS Intelligent Cloud Services. The authors thank Ming-Wei Chang and reviewers for constructive comments.

heading: B Additional Details of Linear Methods
text: The binary linear SVM is in the following form.
min w
1 2 w⊤w + C
∑
i
ξ(yiw ⊤xi), (1)
where (xi, yi) are data-label pairs in the data set, yi = ±1, w is the parameters of the linear model, and ξ(·) is the loss function. The decision value function is f(x) = w⊤x.
For one-vs-rest, please see descriptions in Section 3.1. We follow the default setting in LibMultiLabel by using C = 1. For more details about thresholding and cost-sensitive, please refer to the explanations in Lin et al. (2022).

heading: C Differences Between Our Implementation of Linear Methods and Chalkidis et al. (2022)
text: We summarize the implementation differences between LibMultiLabel and Chalkidis et al. (2022) in
Table 5.
For the data-preprocessing part, both use scikitlearn for TF-IDF feature generations. The meanings of each parameter are listed as follows.
stop_words: Specify the list of stop words to be removed. For example, Chalkidis et al. (2022) set stop_words to “english,” so tokens that include in the “english” list are filtered. ngram_range: Specify the range of n-grams to be extracted. For example, LibMultiLabel only uses uni-gram, while Chalkidis et al. (2022) set ngram_range to (1, 3), so uni-gram, bi-gram, and tri-gram are extracted into the vocabulary list for a richer representation of the document. min_df: The parameter is used for removing infrequent tokens. Chalkidis et al. (2022) remove tokens that appear in less than five documents, while LibMultiLabel does not remove any tokens. max_features: The parameter decides the number of features to use by term frequency. For example, Chalkidis et al. (2022) consider the top 10,000, 20,000, and 40,000 frequent terms as the search space of the parameter.
For more detailed explanations, please refer to the TfidfVectorizer function in scikit-learn.
The binary classification problem in (1) is referred to as the primal form. The optimization problem can be transferred to the dual form and the optimal solutions of the two forms lead to the same decision function. Thus we can choose to solve the primal or the dual problem; see Table 5. For the model training, they both use the solver provided by LIBLINEAR (Fan et al., 2008).

heading: D Additional Details about Model Size
text: We calculate the model size of linear SVM by multiplying the number of TF-IDF features by the number of labels; see details in Table 6. For BERT, we directly copy the number of parameters from Chalkidis et al. (2022).

heading: E Additional Details about BERT Design in Chalkidis et al. (2022)
text: E.1 Standard BERT for Classification
The setting considers the original implementation in Devlin et al. (2019). They truncate the documents to have at most 512 tokens. We then take a pre-trained BERT appended with an additional linear layer for fine-tuning.
E.2 Document Lengths
In Table 6, we present the document length for each data set in LexGLUE, the benchmark considered in Chalkidis et al. (2022). For ECtHR (A), ECtHR (B), SCOTUS, and EUR-LEX, the document lengths all exceed 512, the length limitation of BERT. Note that the numbers are underestimated because BERT uses a sub-word tokenizer that further tokenizes some words into sub-words.
E.3 Hierarchical BERT
Chalkidis et al. (2022) design a variant of the standard BERT for ECtHR (A), ECtHR (B), and SCOTUS to deal with long document lengths. The detailed steps are as follows. • Each document is split into 64 segments, where
each segment contains at most 128 tokens. • Each segment is then fed into BERT. • The [CLS] tokens generated from each segment
are collected and fed into an upper-level transformer encoder.
• Max pooling is applied to the output of the transformer encoder.
• The pooled results are then fed into a linear layer for the final prediction.

heading: F Differences between the Two BERT Implementations
text: We summarize the implementation differences of BERT between LibMultiLabel and Chalkidis et al. (2022) in Table 7. Here we also try to reproduce results in Chalkidis et al. (2022) by using LibMultiLabel.
For LibMultiLabel, we explain our choices of hyper-parameters as follows. default: This method references the parameters chosen in an example configuration7 from LibMultiLabel. tuned: This method performs a parameter search and is marked as “our BERT” in the main paper; see Table 8 for the search space and the chosen values. reproduced: This method aims to reproduce the BERT results from Chalkidis et al. (2022) using LibMultiLabel. We begin with imposing the same
7https://github.com/ASUS-AICS/LibMultiLabel/ blob/master/example_config/EUR-Lex-57k/bert.yml
weight_decay, learning_rate, and dropout values as Chalkidis et al. (2022) and also the same validation metric. However, for other parameters, which may less affect the results, we use the same values as default and tuned; see Table 7. Except SCOTUS and LEDGAR, we were able to generate similar results to those in Chalkidis et al. (2022). To fully reproduce the results on SCOTUS and LEDGAR, we try to follow every setting did in Chalkidis et al. (2022). Specifically, we replace the PyTorch trainer originally used in LibMultiLabel with the Hugging Face trainer adopted in Chalkidis et al. (2022) and align some of the parameters with the ones used in Chalkidis et al. (2022); see a column in Table 7 for these two sets.
LibMultiLabel supports standard BERT discussed in Appendix E.1. For the “default” and “tuned” settings, we directly run standard BERT. For the “reproduced” method, we follow Chalkidis et al. (2022) to use hierarchical BERT explained in Appendix E.3 for ECtHR (A), ECtHR (B), and SCOTUS and use standard BERT for other data sets.

heading: G Detailed BERT Results
text: In Tables 9 and 10, we respectively present the test performance and the training time. For settings of running LibMultiLabel, see Appendix F. For BERT
in Chalkidis et al. (2022), we present the following two results. paper: Results in the paper by Chalkidis et al. (2022) are directly copied. reproduced: Results from our running of their scripts.8
For ECtHR (A), ECtHR (B), and SCOTUS, because there exist some issues when running the fp16 setting in our environment, we run the code of Chalkidis et al. (2022) by using fp32 instead. This change causes the time difference between the “paper” and “reproduced” settings in Table 10.
Except numbers borrowed from Chalkidis et al. (2022), we run five seeds for all BERT experiments and report the mean test performance over all seeds. Chalkidis et al. (2022) also run five seeds, but their test scores are based on the top three seeds with the best Macro-F1 on validation data.
For the “tuned” setting, because the version of LibMultiLabel that we used does not store the checkpoint after hyper-parameter search, we must conduct the training again using the best hyperparameters. Thus, the total time includes hyperparameter search and the additional training.9
In Appendix I, we give an additional case study to assess the performance of the hierarchical BERT when documents are long.
H Issue of Using Training, Validation, and Test Sets
For each problem in LexGLUE, training, validation, and test sets are available. In our experiments, of course the test set is independent from the training process. However, some issues occur in the use of the training and validation sets.
For linear methods, in contrast to deep learning methods, they do not need a validation set
8https://github.com/coastalcph/lex-glue 9We run five seeds in the part of additional training. Thus, we obtain five values of the total time and report the average.
for the termination of the optimization process or for selecting the iteration that yields the best model. Further, they may internally conduct crossvalidation to select hyper-parameters (e.g., thresholds in the thresholding method). Therefore, we combine training and validation subsets as the new training set used by the linear methods. This is the standard setting in traditional supervised learning.
For BERT training, the validation set is used for selecting the best epoch and/or the best hyperparameters. We follow the common practice to deploy the model achieving the best validation performance for prediction. However, in linear methods, the model used for prediction, regardless of whether internal cross-validation is needed, is always obtained by training on all available data (i.e., the combination of training and validation sets). Therefore, for BERT we may also want to incorporate the validation set for the final model training. We refer to such a setting as the re-training process. Unfortunately, an obstacle is that the optimization process cannot rely on a validation set for terminating the process or selecting the best model in all iterations. Following Goodfellow et al. (2016), we consider the following setting to train the combined set.
1. Record the number of training steps that leads to the best validation Micro-F1 as e∗.
2. Re-train the final model using the combination of training and validation sets for e∗ epochs.
BERT results without/with re-training are shown in Table 11. In general, the re-training process improves the performance, especially for the data sets SCOTUS and EUR-LEX. However, results are slightly worse in both the default and tuned settings for the data set UNFAIR-ToS. Thus the outcome of re-training may be data-dependent.
A comparison between linear methods and BERT with re-training shows that conclusions made earlier remain the same. Because re-training
is not conducted in Chalkidis et al. (2022), in the main paper we report the results without retraining.

heading: I A Case Study of BERT on 20 Newsgroups
text: Wahba et al. (2023) applied BERT for training the data set 20 Newsgroups (Lang, 1995) but did not check the document length. To assess the importance of the document length, we downloaded the 20 Newsgroups set from scikit-learn10 with default
10See https://scikit-learn.org/stable/modules/ generated/sklearn.datasets.fetch_20newsgroups. html for more details. We have checked that the set used in scikit-learn is the same as the “20 News-
parameters. Further, we checked the document length from the word and token levels where the tokens are obtained by the “bert-base-uncased” tokenizer. The data statistics are presented in Table 12. We found that the 20 Newsgroups data set includes a considerable number of documents that exceed 512 tokens. This may be an issue because BERT can only process up to 512 tokens without further design; see Appendix E for more details. To investigate this problem, we conducted experiments using both linear classifiers and BERT. Results are in Table 13. The observations are summarized as follows. • The results of linear classifiers do not improve by
using thresholding and cost-sensitive techniques to handle class imbalance. The reason is that the data set has a small number of labels and a more balanced class distribution. In addition, linear methods are still competitive with BERT.
• The tuned setting of BERT has the best Micro-F1 among all the methods. Thus, for running BERT on this set, parameter selection seems to be important. Interestingly, when we considered the document length using the hierarchical methods in Appendix E.3, the performance was not better than the tuned setting. In conclusion, linear methods are still a simple and efficient solution to this problem. For BERT, we showed that using the hierarchical setting to handle long document length may not always lead to the best performance. The result of applying hierarchical BERT may be data-dependent. Thus a general setting for handling long documents still need to be investigated.
groups sorted by date” set from the original source at http://qwone.com/~jason/20Newsgroups/.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations.
7 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sections Abstract and 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.
3 B1. Did you cite the creators of artifacts you used? Section 3.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix F.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix G.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: However, having a simple quantitative measurement to pre-determine when to use which remains a challenging future research problem. In summary, the study reminds us of the importance of employing simple baselines in NLP applications. 6See details in Appendix Table 6. Limitations
In this work, we do not propose any new methods because, as an opinion paper, we focus on raising the problems and making vivid demonstrations to readers. The experiments are limited to linear SVM and BERT on data sets in the benchmark LexGLUE. We hope that, within the page limit, our experiments sufficiently convey the points to readers