SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 15: The PPL and Accuracy on different training steps
Step	WikiText2 ↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
0	19.09	34.21	57.06	75.68	66.80	59.83	60.94	36.52	40.00	56.69
200	18.10	30.66	64.62	77.20	68.80	63.14	64.31	36.77	39.80	59.24
400	17.69	30.26	63.00	76.66	68.75	63.54	64.39	37.20	40.60	59.16
600	17.69	30.57	66.24	76.28	68.52	63.85	64.48	37.37	41.00	59.68
800	17.64	30.57	65.05	76.22	68.38	63.77	63.64	37.29	40.80	59.31
1000	17.67	30.60	66.39	76.17	68.24	64.17	63.05	37.37	41.60	59.57
1200	17.74	30.75	65.75	76.28	68.28	63.77	63.30	37.63	41.20	59.46
1400	17.88	30.85	64.34	76.28	68.31	63.85	63.47	37.80	41.20	59.32
C.4 Pruning with Large Rates
Additionally, we conducted tests on LLaMA-7B and Vicuna-7B with  $50\%$  parameters pruned. We observe a significant decrease in performance compared to the base model. However, the recovery stage proved to be beneficial, resulting in an improvement of approximately  $7.39\%$ . Pruning a Language Model with such a high pruning rate remains a challenging task.
Table 16: The PPL and Accuracy on LLaMA-7B with  $50\%$  parameters pruned
Pruned Model	Method	WikiText2↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 50% w/o tune	l2	39266.42	48867.85	55.11	53.59	27.03	49.49	26.43	29.01	34.40	39.29
random	3887.90	4337.27	46.79	53.37	27.50	50.59	28.07	27.90	30.00	37.75
Channel	13891.92	16114.91	40.37	52.18	25.72	48.86	25.72	28.24	30.40	35.93
Vector	141.06	236.24	62.17	55.11	27.25	49.88	29.00	25.77	34.00	40.45
Element2	106.07	266.65	52.57	60.45	35.86	49.01	32.83	25.51	34.80	41.58
Element1	112.44	255.38	52.32	59.63	35.64	53.20	33.50	27.22	33.40	42.13
Ratio = 50% w/ tune	Channel	1122.15	1092.26	40.76	54.84	26.94	49.41	27.86	25.43	32.20	36.77
Vector	43.47	68.51	62.11	64.96	40.52	51.54	46.38	28.33	32.40	46.61
Element2	45.70	69.33	61.47	68.82	47.56	55.09	46.46	28.24	35.20	48.98
Element1	38.12	66.35	60.28	69.31	47.06	53.43	45.96	29.18	35.60	48.69
Table 17: The PPL and Accuracy on Vicuna-7B with  ${50}\%$  parameters pruned
Pruned Model	Method	WikiText2 ↓	PTB↓	BoolQ	PIQA	HellaSwag	WinoGrande	ARC-e	ARC-c	OBQA	Average
Ratio = 0%	Vicuna-7B	16.11	61.37	76.57	77.75	70.64	67.40	65.11	41.21	40.80	62.78
Ratio = 50% w/o tune	l2	54516.03	66274.63	45.99	53.48	26.55	47.83	27.53	28.58	30.40	37.19
random	17020.73	13676.54	48.17	53.43	27.31	50.43	26.30	29.78	30.20	37.95
Channel	8360.30	10651.30	38.69	53.10	26.42	50.20	25.97	29.52	29.60	36.22
Vector	189.87	409.75	62.14	55.33	26.99	51.93	27.86	26.02	32.60	40.41
Element2	143.85	427.77	53.76	59.79	34.86	50.28	33.29	27.30	34.60	41.98
Element1	162.38	469.74	52.45	58.65	34.27	49.72	32.41	28.75	35.80	41.72
Ratio = 50% w/ tune	Channel	206.23	526.89	62.11	58.38	32.29	51.22	33.08	24.06	30.20	41.62
Vector	46.11	147.52	62.14	64.91	39.80	50.91	47.77	27.30	32.60	46.49
Element2	42.99	139.91	58.87	69.48	46.38	54.78	46.89	29.01	34.80	48.60
Element1	41.64	143.74	62.08	69.48	46.79	54.54	48.15	29.78	36.20	49.57
D Generations From Compressed Model
Table 18, 19, 20 and 21 show more examples of the models pruned by LLM-Pruner. We present the generation results of both the pruned model with post-training and without post-training. The absence of post-training allows us to better understand the information retained in the model. We include the results of ChatGLM-6B in two languages as it is a bilingual model.
Table 19: Generated Examples from the Compressed Vicuna-5.4B
Model	Generated Sentences
Vicuna w/ Tune	Last night we all danced together in the rain, all our cares and worries forgotten. And as the sun began to rise, I looked into your eyes and knew that everything was going to be okay. That's the way it is now, forever and forever. I love you, my baby, and I always will.
It fills me with such pride and joy to be working along side her, learning from her and sharing our experiences, joys, and challenges as partners. Our bond is growing stronger each day and I can't help but feel grateful for the beautiful life she has brought into my own. To my loving wife, I love you more and more each day. I can't wait to spend the rest of my life with you and raising our future children together projecting the light of Christ into the world. And to all of you reading this, I encourage you to take this seriously and consider your relationship with the one
Meet me at midnight tonight at the Bell Tower in town. Bring a torch, and we'll take the treasure off the church's hands, no matter the cost.' With a deep sense of trepidation, Alaric finds himself agreeing to the plan. He spends the day getting everything he can from the village, preparing for the inevitable attack he'll have to launch. When the day arrives, he and his companions mount their steeds and set off towards the towering keep of the temple. As they draw closer, the wind whispers
Vicuna w/o Tune	Last night we all danced together in the rain. The water felt good on my face. The song played over and over, but I didn't care. I just kept on dancing with you. I don't know what to say. I'm not good with words. But I know that I've never felt this way before. With anyone. Not even myself. And I don't want to let that feeling go. I don't know what you feel, or what you want from me. But I know that this is real. And I don't want to let it go
It fills me with such pride and joy, it's like seeing my baby for the first time, and I just want to hold it close and never let it go. The thought that it might have to be put down one day breaks my heart. I want to do everything in my power to protect it, to keep it safe and give it the best life possible. But it's not mine. It's never been mine to keep. And I know that one day, sooner or later, it will have to be given away. I can't help but feel a sense of responsibility that I don't ...
Meet me at midnight, I promise you'll be the only one there. It's hard for me to believe, but I'm scared. I don't want to be alone tonight, but I know I have to be brave. Hurry down the hill, I'm waiting for you, I'll take you to a place that I know is safe. We won't be seen, nobody'll know. It's almost time for you to go. I don't want you to be late tonight.
Table 20: Generated Examples from the Compressed ChatGLM-5.6B (English)
Model	Generated Sentences
ChatGLM w/ Tune	How to buy a cat? It's important to understand that while it's understandable that you might be looking todnlne a cat for yourself, adoption is a great way to bring home a furry friend without having to shop. Here are some things to consider before adoption: 1. First, make sure you know you're interested in Jr. or Vioness. This determines whether you're looking for a young adult cat or an older cat. 2. research your options: Look up reputable cat owners or organizations that foster cats in your area. This will give you information on cat owners
What is McDonald's? McDonald's is a popular American fast-food restaurant chain that offers a variety of food and beverage items. The company was founded in 1940 by Richard and Maurice Mc Eu Eu in San Francisco, and the chain was originally called "McNugguggets." Today, McDonald's is known for its popular fast-food items such as McDonald's Drive-Thru, McChickens, and McFlurryrry. The company also offers a wide range of other food and beverage items such as Happy Endings, Big Macs, and ...
What is Mean Square Error? The Mean Square Error (MSE) is a measure of the difference between a target value and the best possible estimate of its true value, based on the data. It is calculated by taking the average difference between the actual value and the estimated value over the* number of times the target value is repeated.
ChatGLM w/o Tune	How to buy a cat? It's important to understand that adopting a cat from a shelter or organization is a valuable and responsible adoption option, as well as one that may be safer and more comfortable for you and your family. Additionally, it's important to read carefully before adopting to ensure that you choose a Cat who is suitable for you and your family.
What is McDonald's? McDonald's is a popular fast-food restaurant chain from United States known for its restaurant concepts such as McNugguggets, McFlurryrry, and McRibibs. The chain started in 1955 as a small operation operation called "Speedy McDonald's" in San Antonio, Texas and it quickly expanded and became a widely recognized and popular phenomenon in the country. Today, McDonald's is global icon known around the world.
What is Mean Square Error? Mean Square Error (MSE) is a measure of the accuracy of a function or a measurement when it is repeated and the result is expected to be within a certain range. It is calculated by taking the difference between two sets of measurements and measuring the difference squared. In simple terms, it means the mean sum of the errors over a given period of time or the difference between the actual and expected values of ...
Table 21: Generated Examples from the Compressed ChatGLM-5.6B (Chinese)
Model	Generated Sentences
ChatGLM w/ Tune	请给我推荐两个北京著名的景点 北京著名的景点有很多,以下是两个:1.颐和园:颐和园是北京的一座皇家园林,也是世界文化遗产。它是一座是一座结合了中国传统建筑和现代艺术的杰作,被誉为“万园之园”。颐和园内有很多建筑和景点,包括长廊、排排坐、佛香阁、万寿山等,非常适合漫步和参观。2.故宫:故宫是中国古代皇宫,也是世界文化遗产,是北京的必去之地。它是世界上是世界上现存最大的、最完整的、最完整的、最著名的古代宫殿宫殿建筑之一。故宫内有很多建筑和景点,包括紫禁城
简单来说,相对论指出 相对论是由爱因斯坦提出的一个关于时间和空间如何解释和变化的的理论,被称为狭义相对论。相对论认为,时间和空间并不是绝对的,而是相对的,也就是说,它们的测量值取决于观察者的惯性参照系。根据相对论,当物体以高速运动或以或以某种方式加速时,时间和空间的测量值会发生变化。最著名的是,时间和空间的测量值会“膨胀”,即随着物体运动,它们看起来会变得越来越长或越来越短。
如何健康的生活 要保持健康的生活方式,可以参考以下建议:1.定期锻炼:适量运动可以帮助增强身体健康,减轻压力,改善心情。建议每周至少进行3-5次有氧运动,如慢跑、快走、游泳、瑜伽等。2.饮食均衡:控制饮食摄入量,摄入摄入适量、健康的蛋白质、碳水化合物、脂肪以及维生素和矿物质。3.睡眠充足:充足的睡眠有助于恢复身体和大脑功能,提高身体免疫力。建议每晚睡眠7-8小时。4.勤洗手:避免传染病毒。
ChatGLM w/o Tune	请给我推荐两个北京著名的景点 北京著名的景点太多,以下是两个必去的景点:1.故宫博物院:故宫是世界上是世界上是世界上保存最完整的、最著名的宫殿建筑之一,也是中国历史文化名城和世界文化遗产。游客可以参观紫禁城紫禁城,欣赏到欣赏到大量的历史文物和艺术品。2.长城:长城是中国古代著名的防御工程之一,也是中国历史文化名城和世界文化遗产。游客可以参观长城的各种不同高度和不同用途的客房现代物流设施,还可以在长城上徒步。
简单来说,相对论指出 相对论是由爱因斯坦提出的一组理论和概念,包括两个部分:狭义相对论和广义相对论。狭义相对论是描述在高速运动或缩短物体长度时,时间和空间如何弯曲的理论。它指出,当我们测量一个物体的长度或加速运动时,我们的测量结果观测到的是相对一个观察者观察者静止静止的物体长度或速度的尺子。这个理论解释了为什么光速是恒定不变的不变的。广义无味相对论是描述引力如何加速和改变物体的质量的。它指出,引力是由质量m和引力场引起的,两个物体引力场的中心的引力是
如何健康的生活 想要过健康的生活可以参考以下建议:1.保持规律的规律的充足的睡眠:尽可能每晚每晚睡7-8个小时,保持规律的规律的睡眠时间有助于维持身体健康和精神状态。2.多运动:每天进行适量的运动有助于减轻压力和增强身体免疫力,建议每天进行适量的有氧运动和力量训练。3.饮食均衡:保持均衡的饮食,包括摄入足够的各种营养素和禁止过量摄入垃圾食品和饮料。4.减少不良习惯:尽可能减少不良习惯,如吸烟、酗酒和过度使用电子设备等。