--- license: apache-2.0 datasets: - neptun-org/neptun.scraper - sequelbox/Titanium - MattCoddity/docker_ps - MattCoddity/dockerNLcommands language: - en - zh base_model: - BlinkDL/rwkv-5-world pipeline_tag: text-generation tags: - Docker - Node.js - GitHub metrics: - code_eval --- # Neptun AI `RWKV-5-World-0.4B-v2-20231113-ctx4096.pth` tuned using [neptun.scraper](https://huggingface.co/datasets/neptun-org/neptun.scraper) and LoRA Finetune. Converted using [convert_ai00.py](https://github.com/Ai00-X/ai00_server/blob/main/assets/scripts/convert_ai00.py). * Precision: bf16 * Context Length: 4096 * Epoch Begin: 0 * Epoch Count: 20 * Epoch Steps: 1000 * Epoch Save: 1000 * Warmup Steps: 0 * Learning Rate Init: 2e-5 * Learning Rate Final: 2e-5 * Micro Batch Size: 1 * Accumulate Gradient Batches: 24 * Beta 1: 0,9 * Beta 2: 0,999 * LoRA R: 64 * LoRA Alpha: 192 * LoRA Dropout: 0,01 * Adam Epsilon: 1e-8 * Gradient Checkpoint: true * Devices: 1 ## Dockerfiles ```bash 1 1.552413 4.7229 0.00002000 2025-01-28 14:47:38.809119 0 2 1.478129 4.3847 0.00002000 2025-01-28 15:37:13.198567 1 3 1.412234 4.1051 0.00002000 2025-01-28 16:26:29.863755 2 4 1.374447 3.9529 0.00002000 2025-01-28 17:16:15.815151 3 5 1.317078 3.7325 0.00002000 2025-01-28 18:06:02.229190 4 6 1.296596 3.6568 0.00002000 2025-01-28 18:55:40.765344 5 7 1.257023 3.5149 0.00002000 2025-01-28 19:45:23.963291 6 8 1.253057 3.5010 0.00002000 2025-01-28 20:34:39.364658 7 9 1.219242 3.3846 0.00002000 2025-01-28 21:23:37.893528 8 10 1.190037 3.2872 0.00002000 2025-01-28 22:12:28.025686 9 11 1.181473 3.2592 0.00002000 2025-01-28 23:02:49.004392 10 12 1.176299 3.2424 0.00002000 2025-01-28 23:52:07.349197 11 13 1.153920 3.1706 0.00002000 2025-01-29 00:42:01.340925 12 14 1.146656 3.1477 0.00002000 2025-01-29 01:30:47.291425 13 15 1.125227 3.0809 0.00002000 2025-01-29 02:19:49.021365 14 16 1.125680 3.0823 0.00002000 2025-01-29 03:08:48.035252 15 17 1.108051 3.0284 0.00002000 2025-01-29 03:57:08.758836 16 18 1.121515 3.0695 0.00002000 2025-01-29 04:45:28.513407 17 19 1.098906 3.0009 0.00002000 2025-01-29 05:33:47.381810 18 20 1.088252 2.9691 0.00002000 2025-01-29 06:22:06.299740 19 ``` ## V1 ```bash 0 1.328085 3.7738 0.00002000 2025-02-24 22:42:19.385036 0 1 1.250953 3.4937 0.00002000 2025-02-24 23:31:09.470548 1 2 1.220156 3.3877 0.00002000 2025-02-25 00:20:28.265176 2 3 1.189695 3.2861 0.00002000 2025-02-25 01:09:00.092662 3 4 1.173590 3.2336 0.00002000 2025-02-25 01:57:14.064583 4 5 1.155840 3.1767 0.00002000 2025-02-25 02:45:25.223964 5 6 1.146094 3.1459 0.00002000 2025-02-25 03:33:37.876671 6 7 1.128762 3.0918 0.00002000 2025-02-25 04:21:51.325201 7 8 1.119391 3.0630 0.00002000 2025-02-25 05:10:04.734031 8 9 1.112895 3.0432 0.00002000 2025-02-25 05:58:17.806360 9 10 1.109004 3.0313 0.00002000 2025-02-25 06:46:33.949976 10 11 1.101230 3.0079 0.00002000 2025-02-25 07:34:52.654477 11 12 1.092156 2.9807 0.00002000 2025-02-25 08:23:09.908699 12 13 1.087285 2.9662 0.00002000 2025-02-25 09:11:25.518814 13 14 1.081148 2.9481 0.00002000 2025-02-25 09:59:37.800531 14 15 1.074223 2.9277 0.00002000 2025-02-25 10:47:51.388452 15 16 1.074324 2.9280 0.00002000 2025-02-25 11:36:07.807562 16 17 1.068234 2.9102 0.00002000 2025-02-25 12:24:22.401156 17 18 1.059605 2.8852 0.00002000 2025-02-25 13:12:37.599124 18 19 1.055234 2.8726 0.00002000 2025-02-25 14:00:52.741654 19 ``` ## V2 ```bash 1 1.432728 4.1901 0.00002000 2025-03-01 18:28:38.262673 0 2 1.295736 3.6537 0.00002000 2025-03-01 19:17:53.931170 1 3 1.277158 3.5864 0.00002000 2025-03-01 20:07:08.693268 2 4 1.266635 3.5489 0.00002000 2025-03-01 20:56:43.106751 3 5 1.232101 3.4284 0.00002000 2025-03-01 21:46:04.793249 4 6 1.197623 3.3122 0.00002000 2025-03-01 22:36:19.047485 5 7 1.198684 3.3157 0.00002000 2025-03-01 23:26:23.852325 6 8 1.153539 3.1694 0.00002000 2025-03-02 00:15:25.319388 7 9 1.190393 3.2884 0.00002000 2025-03-02 01:04:04.914936 8 10 1.151376 3.1625 0.00002000 2025-03-02 01:52:36.610501 9 11 1.142265 3.1339 0.00002000 2025-03-02 02:41:09.421198 10 12 1.125220 3.0809 0.00002000 2025-03-02 03:29:40.905325 11 13 1.108536 3.0299 0.00002000 2025-03-02 04:18:13.008727 12 14 1.152535 3.1662 0.00002000 2025-03-02 05:06:43.459698 13 15 1.137459 3.1188 0.00002000 2025-03-02 05:55:18.549682 14 16 1.107539 3.0269 0.00002000 2025-03-02 06:43:50.482294 15 17 1.137236 3.1181 0.00002000 2025-03-02 07:32:25.326706 16 18 1.119796 3.0642 0.00002000 2025-03-02 08:20:57.625010 17 19 1.109311 3.0323 0.00002000 2025-03-02 09:09:29.684893 18 20 1.098197 2.9988 0.00002000 2025-03-02 09:58:02.055555 19 ```