| license: other | |
| # Join our Discord! https://discord.gg/Nbv9pQ88Xb | |
| ## 2500+ members strong 💪 Now with more channels! A hub for users and makers alike! | |
| --- | |
| # Lazarus 2407 100B 🌖 | |
| > To the brave men and women who gave their lives so we could begin again. | |
|  | |
| # Links | |
| - Original: https://huggingface.co/TheDrummer/Lazarus-2407-100B | |
| - GGUF: https://huggingface.co/TheDrummer/Lazarus-2407-100B-GGUF | |
| - iMatrix (Recommended): https://huggingface.co/bartowski/Lazarus-2407-100B | |
| ## A pruned version of Mistral Large 2407 123B | |
| - Theory: https://arxiv.org/abs/2403.17887 | |
| - Practice: https://github.com/arcee-ai/PruneMe | |
| ```yaml | |
| slices: | |
| - sources: | |
| - model: ./largestral | |
| layer_range: [0, 54] | |
| - sources: | |
| - model: ./largestral | |
| layer_range: [70,88] | |
| merge_method: passthrough | |
| dtype: bfloat16 | |
| ``` | |
|  | |
| ``` | |
| block_start,block_end,average_distance | |
| 1,17,0.42018115234375 | |
| 2,18,0.40477392578125 | |
| 3,19,0.3955498046875 | |
| 4,20,0.39202392578125 | |
| 5,21,0.38614453125 | |
| 6,22,0.380365234375 | |
| 7,23,0.37550830078125 | |
| 8,24,0.36995361328125 | |
| 9,25,0.36802099609375 | |
| 10,26,0.36337158203125 | |
| 11,27,0.3570400390625 | |
| 12,28,0.35895654296875 | |
| 13,29,0.360716796875 | |
| 14,30,0.36415234375 | |
| 15,31,0.3656328125 | |
| 16,32,0.3657451171875 | |
| 17,33,0.36321337890625 | |
| 18,34,0.36210791015625 | |
| 19,35,0.3613828125 | |
| 20,36,0.36101806640625 | |
| 21,37,0.36395361328125 | |
| 22,38,0.36294921875 | |
| 23,39,0.36490673828125 | |
| 24,40,0.36392578125 | |
| 25,41,0.3616650390625 | |
| 26,42,0.360994140625 | |
| 27,43,0.357458984375 | |
| 28,44,0.35728759765625 | |
| 29,45,0.35514013671875 | |
| 30,46,0.35244140625 | |
| 31,47,0.34866162109375 | |
| 32,48,0.3489375 | |
| 33,49,0.3480888671875 | |
| 34,50,0.34322607421875 | |
| 35,51,0.3366806640625 | |
| 36,52,0.331208984375 | |
| 37,53,0.31834423828125 | |
| 38,54,0.30483203125 | |
| 39,55,0.29027587890625 | |
| 40,56,0.2789296875 | |
| 41,57,0.263868408203125 | |
| 42,58,0.25043017578125 | |
| 43,59,0.23816162109375 | |
| 44,60,0.223565185546875 | |
| 45,61,0.216619140625 | |
| 46,62,0.212794189453125 | |
| 47,63,0.205733642578125 | |
| 48,64,0.200558837890625 | |
| 49,65,0.196917236328125 | |
| 50,66,0.1951201171875 | |
| 51,67,0.192659423828125 | |
| 52,68,0.191610595703125 | |
| 53,69,0.1910341796875 | |
| 54,70,0.190966064453125 | |
| 55,71,0.1920712890625 | |
| 56,72,0.1935439453125 | |
| 57,73,0.1973837890625 | |
| 58,74,0.200970703125 | |
| 59,75,0.205339111328125 | |
| 60,76,0.216475341796875 | |
| 61,77,0.218605224609375 | |
| 62,78,0.22363916015625 | |
| 63,79,0.229986083984375 | |
| 64,80,0.23776171875 | |
| 65,81,0.24455322265625 | |
| 66,82,0.2542783203125 | |
| 67,83,0.2635498046875 | |
| 68,84,0.27629443359375 | |
| 69,85,0.29324267578125 | |
| 70,86,0.32201220703125 | |
| 71,87,0.36908154296875 | |
| 72,88,0.4007587890625 | |
| ``` |