Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Qwen2.5-14B-Instruct.rkllm filter=lfs diff=lfs merge=lfs -text
+demo_Linux_aarch64/lib/librkllmrt.so filter=lfs diff=lfs merge=lfs -text
+demo_Linux_aarch64/lib/librknnrt.so filter=lfs diff=lfs merge=lfs -text

Qwen2.5-14B-Instruct.rkllm

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebdf5fc0176ec42d6db7fb1debc726471dc234532e6f3ce7809486a8cc9c946b
+size 15595364468

README.md

@@ -0,0 +1,115 @@
+---
+base_model:
+- Qwen/Qwen2.5-14B
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- rknn
+- rkllm
+- chat
+- rk3588
+---
+## 3ib0n's RKLLM Guide
+These models and binaries require an RK3588 board running rknpu driver version 0.9.7 or above
+
+## Steps to reproduce conversion
+```shell
+# Download and setup miniforge3
+curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
+bash Miniforge3-$(uname)-$(uname -m).sh
+
+# activate the base environment
+source ~/miniforge3/bin/activate
+
+# create and activate a python 3.8 environment
+conda create -n rknn-llm-1.1.4 python=3.8
+conda activate rknn-llm-1.1.4
+
+# clone the lastest rknn-llm toolkit
+git clone https://github.com/airockchip/rknn-llm.git
+
+# intstall dependencies for the toolkit
+pip install transformers accelerate torchvision rknn-toolkit2==2.2.1
+pip install --upgrade torch pillow
+
+# install rkllm
+pip install ../../rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
+
+# edit or create a script to export rkllm models
+cd rknn-llm/examples/rkllm_multimodal_demo
+nano export/export_rkllm.py # update input and output paths 
+python export/export_rkllm.py 
+```
+
+Example export_rkllm.py modified from https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_multimodel_demo/export/export_rkllm.py
+```python
+import os
+from rkllm.api import RKLLM
+from datasets import load_dataset
+from transformers import  AutoTokenizer
+from tqdm import tqdm
+import torch
+from torch import nn
+
+modelpath = "~/models/Qwen/Qwen2.5-14B-Instruct/" ## UPDATE HERE
+savepath = './Qwen2.5-14B-Instruct.rkllm' ## UPDATE HERE
+llm = RKLLM()
+
+# Load model
+# Use 'export CUDA_VISIBLE_DEVICES=2' to specify GPU device
+ret = llm.load_huggingface(model=modelpath, device='cpu')
+if ret != 0:
+    print('Load model failed!')
+    exit(ret)
+
+# Build model
+qparams = None
+
+## Do not use the dataset parameter as we are converting a pure text model, not a multimodal
+ret = llm.build(do_quantization=True, optimization_level=1, quantized_dtype='w8a8',
+                quantized_algorithm='normal', target_platform='rk3588', num_npu_core=3, extra_qparams=qparams)
+
+if ret != 0:
+    print('Build model failed!')
+    exit(ret)
+
+# # Export rkllm model
+ret = llm.export_rkllm(savepath)
+if ret != 0:
+    print('Export model failed!')
+    exit(ret)
+```
+
+## Steps to build and run demo
+
+```shell
+# Dwonload the correct toolchain for working with rkllm
+# Documentation here: https://github.com/airockchip/rknn-llm/blob/main/doc/Rockchip_RKLLM_SDK_EN_1.1.0.pdf
+wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
+tar -xz gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
+
+# ensure that the gcc compiler path is set to the location where the toolchain dowloaded earlier is unpacked
+nano deploy/build-linux.sh # update the gcc compiler path
+
+# compile the demo app
+cd delpoy/
+./build-linux.sh 
+```
+
+## Steps to run the app
+More information and original guide: https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_multimodel_demo
+```shell
+# push install dir to device
+adb push ./install/demo_Linux_aarch64 /data
+# push model file to device
+adb push Qwen2.5-14B-Instruct.rkllm /data/models
+
+adb shell
+cd /data/demo_Linux_aarch64
+# export lib path
+export LD_LIBRARY_PATH=./lib
+# soft link models dir
+ln -s /data/models .
+# run llm(Pure Text Example)
+./llm models/Qwen2.5-14B-Instruct.rkllm 128 512
+```

demo_Linux_aarch64/lib/librkllmrt.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cef353105c3bfd31f99c4963fce8498d2fac633d845633c904f523b7c3bcd0a
+size 6239192

demo_Linux_aarch64/lib/librknnrt.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1170e5f99f2db7ed4d3a4c2bdbed941b7363bd090e0c28b4e210f40614327911
+size 7192616