SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For information extraction, we utilize the following models based on the transformer encoder architecture: RoBERTa(chinese-RoBERTa-wwm-ext) [10], Bert(Bert-base-Chinese) [12], ERNIE(ernie-3.0-base-zh) [36], PERT(chinese-pert-base) [11], mBERTa(mbart-large-50) [37], and SBERT(SBERT-base-chinese-nli) [31]. The preliminary screening stage is configured as binary classification, while the fine-grained screening is set up as multi-class classification.
For rewritten sentences, we fine-tune the following models based on the transformer encoder-decoder [42] architecture: mT5 [45], Bert2Bert, Bert2gpt, RoBERTa2gpt, and ERNIE2gpt. These models are initialized with parameters from publicly available models, such as mT5-small [45], Bert-base-Chinese [12], ernie-3.0-base-zh [36], chinese-RoBERTa-wwm-ext [10], and gpt2-base-chinese [29]. These models can be found on Hugging Face's $^2$  model repository. To ensure that the input sequence does not exceed the maximum position allowed by the pre-trained models, we truncate the document by removing the ending part. For the complete code and detailed parameters, please refer to our GitHub repository at https://github.com/EnlightenedAI/CAPP-130.
4.2 Performance Evaluation of the Information Extraction Model
To develop the TCSI-pp information extraction baseline model, we utilized RoBERTa, BERT, ERNIE, PERT, mBERT, and SBERT as the underlying models. Micro-F1 and Macro-F1 were employed as evaluation metrics. During the preliminary screening phase, we fine-tuned the "important-binary" and "risk-binary" classification models based on the "Special marking". For the fine-grained screening model, we conducted fine-tuning using the "Topics". Table 2 displays the evaluation metrics for both the preliminary screening and fine-grained screening models.
In Table 2, it is evident that RoBERTa significantly outperforms other models in the metrics of the "topic" and "important" screening tasks, and its performance in the "risk" screening is nearly on par with the optimal PERT. Therefore, RoBERTa appears to be more suitable for information extraction of CAPP-130. It is worth emphasizing that all models demonstrate a situation where the Micro-F1 is significantly higher than the Macro-F1 in the screening of risk sentence. This is primarily because
$^{2}$ https://huggingface.co/
Table 3: Evaluation metrics for each topic in fine-grained screening models.
Topic	RoBERTa	mBERT
Precision	Recall	F1	Precision	Recall	F1
First Party Collection Information	0.855	0.811	0.832	0.871	0.786	0.826
Permission Acquisition	0.809	0.866	0.836	0.858	0.812	0.834
Third Party Sharing/ Disclosure	0.832	0.847	0.839	0.835	0.863	0.849
Usage	0.644	0.520	0.575	0.760	0.325	0.455
Data Retention	0.848	0.822	0.835	0.807	0.871	0.838
Data Security	0.905	0.676	0.774	0.869	0.641	0.738
Specific Audiences	0.967	0.870	0.916	0.833	0.941	0.883
Edit/Control	0.937	0.877	0.906	0.919	0.893	0.906
Contact Information	0.988	0.921	0.954	0.965	0.921	0.943
Policy Change	0.916	0.890	0.903	0.878	0.890	0.884
Cease Operation	0.824	0.933	0.875	0.824	0.933	0.875
Table 4: Evaluation metrics for the rewritten sentence models.
Model	ROUGE-1	ROUGE-2	ROUGE-L	Bert-score	Bart-score	Carburacy
mT5	0.753	0.609	0.733	0.888	-4.577	0.833
Bert2gpt	0.751	0.574	0.720	0.872	-4.964	0.764
RoBERTa2gpt	0.749	0.577	0.719	0.872	-4.975	0.755
Bert2bert	0.718	0.535	0.689	0.864	-5.020	0.747
ERNIE2gpt	0.623	0.406	0.581	0.809	-5.716	0.715
the percentage of risk-related sentences is exceedingly small, and it tends to be relatively implicit. As a result, it becomes more difficult for the models to extract meaningful semantic information.
In order to delve deeper into the performance disparities of the multi-class fine-grained screening models across data practices, Table 3 showcases the Precision, Recall, and F1 scores for RoBERTa and mBERT within all topics.
As shown in Table 3, both the RoBERTa and mBERT models accomplish an evaluation metric exceeding 0.80 in 11 data practice categories, excluding "Usage". Additionally, their scores surpass 0.90 for numerous topics. This is due to the fact that data usage is a relatively sensitive area, and manufacturers often opt for more specialized vocabulary and intricate expressions when describing these sentences compared to other data practice categories. Based on this finding, we recommend that data regulatory authorities further refine the regulatory requirements related to terms of personal information "usage" in privacy policies. It is important to note that the evaluation metrics of both classification models once again confirm the high quality of CAPP-130.
4.3 Performance Evaluation of the rewritten sentence Model
In this section, rewritten sentences from CAPP-130 were utilized to fine-tune the models introduced in Section 4.1, serving as benchmarks for the rewriting task. Table 4 displays the ROUGE [19], Bert-score [48], Bart-score [46], and Carburacy [23] evaluation metrics for these models:
Table 4 demonstrates that the mT5-based model significantly outperforms other models in terms of all metrics. The ranking is as follows: mT5 >Bert2gpt >RoBERTa2gpt >Bert2bert >ERNIE2gpt.
4.4 Performance of the summarization model
TCSI-pp-zh uses RoBERTa as the information extraction model and mT5 as the sentence rewritten model. To showcase the results, we extracted several paragraphs from privacy policies for validation and demonstration purposes. Figure 3 displays the outcomes of GPT-4 and TCSI-pp-zh in summarizing privacy policies, where text having the same background color represents descriptions of the same part of the privacy policy generated by different algorithms; red text emphasizes incorrect content produced in the summary. The Appendix A.3 contains the originally extracted excerpts as well as summarization from other methods, including the human expert, the large language model
Figure 3: Summarizations of GPT-4 and TCSI-pp.
Table 5: Questionnaire statistics of readability.
Model	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Percentage
TCSI-pp-zh	37	29	29	17	29	25	38	24	24	30	46.23%
GPT-4	20	27	4	39	28	31	19	30	35	24	42.13%
Llama2	4	4	5	3	2	5	4	5	5	5	6.39%
Checkpoint	0	1	23	2	2	0	0	2	2	2	5.25%
ERNIE Bot (文心一言) [32], Tongyi Qianwen (通义千问) [21], the generative model Checkpoint (Randeng-Pegasus-523M-Summary-Chinese) [47], and the extractive model TextRank [22].
As illustrated in Figure 3, GPT-4 has the ability to extract specific details from privacy policies and generate summaries. However, it can also produce false content that is convincingly deceptive, containing information not found in the original text (red font). To optimize its output, GPT-4 requires carefully crafted prompts, a skill that most users typically lack. In contrast, TCSI-pp-zh accurately pinpoints key information without needing additional prompts from the user, and its summaries employ more concise language compared to GPT-4. Moreover, as demonstrated in Appendix A.3, TextRank considers nearly every sentence important based on keyword frequency, resulting in excessive extraction. Due to limitations stemming from the nature and distribution of the initial training corpus, Checkpoint fails to effectively extract crucial details from privacy policies. The combined performance of ERNIE Bot and Tongyi Qianwen falls short of GPT-4; they may also generate non-existent text or make inaccurate judgments due to insufficient understanding. Specifically, ERNIE Bot generated content that did not belong to the original text (red font), while Tongyi Qianwen ignored some key sentences (sentences with yellow and purple backgrounds in the original text). It is important to note that the content produced by language models is random and unpredictable across each generation. The text above represents the best-case scenario achieved through repeated prompt optimization, and some unprofessional prompts might even result in bizarre outputs. In comparison, our approach is capable of summarizing and generalizing the extracted content, offering advantages such as accurate information extraction, strong interpretability, and concise understandability. However, there is still a significant difference between TCSI-PP and human privacy policy experts in terms of induction, such as the ability to summarize across paragraphs.
4.5 Readability Questionnaire Survey
To evaluate the readability of TCSI-pp-zh, we conducted a social survey on  $WJX^3$ . This survey consisted of 10 single-choice questions. Each question included a paragraph from a privacy policy along with four summaries generated by TCSI-pp-zh, CPT-4, Llama2(LLAMA2-Chinese), and Checkpoint (Randeng-Pegasus-523M-Summary-Chinese). We invited interviewees to select the summarization they found most readable and comprehensible. From the survey, We eventually gathered 61 valid responses as shown in Table 5. The detailed statistics of the interviewees are disclosed in Appendix A.4.
As shown in Table 5, among the choices made by the interviewees,  $46\%$  rated the readability of the summaries created by TCSI-pp-zh the highest, followed by GPT-4 at approximately  $42\%$ , Llama2-Chinese at around  $6\%$ , and Checkpoint at around  $5\%$ . TCSI-pp-zh received the most votes in six out of the ten questions, positioning it ahead of GPT-4, which only managed to lead in four
<sup>3</sup>https://www.wjx.cn/
questions. On the other hand, Llama2-Chinese and Checkpoint did not receive the most votes at all. In conclusion, TCSI-pp-zh demonstrates superior readability compared to GPT-4 and other models when summarizing Chinese privacy policies.
This paper addresses the challenge of understanding complex Chinese application privacy policies which are often filled with technobabble and legalese. To overcome this issue, we propose a Topic-Controlled Framework for Summarization and Interpretation of Privacy Policy (TCSI-pp) framework and a fine-grained Chinese Application Privacy Policy (CAPP-130) corpus, which has been carefully annotated and rewritten by legal experts. We fine-tune several transformer models using CAPP-130 as a benchmark for data practice classification, summarization, and sentence rewriting. Among these models, we select the most effective RoBERTa and mT5 to implement the Chinese application privacy policy summary tool (TCSI-pp-zh). We also evaluate the summarization performance of TCSI-pp-zh on real privacy policies. Experimental results show that large language models like GPT-4 and other benchmark models tend to suffer from content extraction errors or generate false content. In contrast, TCSI-pp-zh can accurately extract specific privacy terms and present them in concise and understandable language. We present the limitations and discuss the potential solutions as follows. (1) CAPP-130 and TSCI-pp-zh are currently only applicable to Chinese privacy policies. We are currently creating English corpora and models and have plans to extend our efforts to include privacy policies in various other languages. (2) The data collection of CAPP-130 relies on machines, we are moving towards incorporating more human factors in collecting privacy policies for future versions. (3) Although we have made efforts in data annotation and model selection to deal with the possible "illusion" phenomenon of generative models, we still cannot completely eliminate the possibility of information loss and misunderstanding of terms. Building upon the foundation of CAPP-130, we are exploring the implementation of keyword extraction techniques to provide cues for classification models and establish external legal knowledge bases to provide valuable background knowledge for generative models. Furthermore, we are contemplating allowing users to control the level of summarization or offer the choice to view both the summarized and original privacy policies side by side. In doing so, we hope to alleviate the problem of "information loss and misunderstanding". (4) While users can select specific content from the 11 data practices available in our model, they are unable to access information beyond these 11 data practice categories. In our future work, we intend to explore additional data practice categories to enhance the customization options of our summarization tool. (5) Compared with real privacy policy experts, our model still lacks the ability to summarize across paragraphs effectively. Therefore, the effective summarization of lengthy text-based privacy policies is a future research direction for us. (6) We are actively designing a dashboard that will incorporate visual elements such as icons and symbols, with the intention of aligning the summarization process more closely with users' intuitive perceptions.
The authors would like to thank the anonymous reviewers for their helpful comments. This work was supported in part by the National Key RD Program of China (2021YFB3101100) and NSFC grants (62102352, 62206207).
[1] W. U. Ahmad, J. Chi, Y. Tian, and K.-W. Chang. Policyqa: A reading comprehension dataset for privacy policies. arXiv preprint arXiv:2010.02557, 2020.
[2] H. Alkaissi and S. I. McFarlane. Artificial hallucinations in chatgpt: implications in scientific writing. *Cureus*, 15(2), 2023.
[3] R. Amos, G. Acar, E. Lucherini, M. Kshirsagar, A. Narayanan, and J. Mayer. Privacy policies over time: Curation and analysis of a million-document dataset. In Proceedings of the Web Conference 2021, pages 2165-2176, 2021.
[4] R. Artstein and M. Poesio. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555-596, 2008.
[44] S. Wilson, F. Schaub, A. A. Dara, F. Liu, S. Cherivirala, P. G. Leon, M. S. Andersen, S. Zimmeck, K. M. Sathyendra, N. C. Russell, et al. The creation and analysis of a website privacy policy corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330–1340, 2016.
[45] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.
[46] W. Yuan, G. Neubig, and P. Liu. Bartscore: Evaluating generated text as text generation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 27263-27277. Curran Associates, Inc., 2021.
[47] J. Zhang, R. Gan, J. Wang, Y. Zhang, L. Zhang, P. Yang, X. Gao, Z. Wu, X. Dong, J. He, J. Zhuo, Q. Yang, Y. Huang, X. Li, Y. Wu, J. Lu, X. Zhu, W. Chen, T. Han, K. Pan, R. Wang, H. Wang, X. Wu, Z. Zeng, and C. Chen. Fengshenbang 1.0: Being the foundation of Chinese cognitive intelligence. CoRR, abs/2209.02970, 2022.
[48] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[49] K. Zhao, L. Yu, S. Zhou, J. Li, X. Luo, Y. F. A. Chiu, and Y. Liu. A fine-grained chinese software privacy policy dataset for sequence labeling and regulation compliant identification. arXiv preprint arXiv:2212.04357, 2022.
[50] H. Zhu, M. Zhang, and Y. Lu. An empirical study on privacy policy reading intention of social media users. Journal of the China Society for Scientific and Technical Information, 37(4):362-371, 2018.
[51] S. Zimmeck, P. Story, D. Smullen, A. Ravichander, Z. Wang, J. R. Reidenberg, N. C. Russell, and N. Sadeh. Maps: Scaling privacy compliance analysis to a million apps. Proc. Priv. Enhancing Tech., 2019:66, 2019.