RWKV4neo 
Author: ArEnSc
Pull request: no
Labels: name: New model

### Model description

RWKV - Receptance Weighted Key Value

RWKV is a Sequence to Sequence Model that takes the best features of Generative PreTraining (GPT) and Recurrent Nueral Networks (RNN) that performs Language Modelling (LM). This is used to generate text Auto Regressive manner (AR).

This is a hybrid model.

It has Transformer Level Performance without the quadratic attention mechanism. It borrows ideas from Attention Free Transformers, meaning the attention is a linear in complexity. Allowing for infinite context through the hidden state in RWKV_RNN.

There are two models for RWKV, they are refered to as modes.

RWKV_RNN: This mode is designed for running inference quickly.
RWKV_GPT: This mode is for training or fine tuning your model quickly. 

In the first pass we will be implementing RWKV_RNN Although we can weight share to have RWKV_GPT generate the inital context for RWKV_RNN.


### Open source status

- [X] The model implementation is available
- [X] The model weights are available
- [ ] Scaffolding
- [ ] API Discussion
### Provide useful links for the implementation

More from the Research and Development Repository: https://github.com/BlinkDL/RWKV-LM

Comments:
@ArthurZucker 
@younesbelkada 
https://github.com/huggingface/transformers/issues/17230#issuecomment-1346779059

This is super cool !! 🔥 
Really looking forward to it 
Also as discussed with @leondz , [we'll probably also include a blogpost](https://github.com/huggingface/transformers/issues/17230#issuecomment-1338060393) explaining this new architecture!
Happy to help for the implementation and the blogpost! 

Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D: 

> Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D:

scratch that, it also didn't work, mainly because of my limited harddisk space. Gonna retry mac...

```
Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=d4b123bfb17b8f11ab811da95d253a066a98638c492d76f4d00afa509dfc6d71
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w5ucgfda/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [76 lines of output]
      
      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler
      
          "/usr/bin/cc"
      
        is not able to compile a simple test program.
      
        It fails with the following output:
      
          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-cd2Kpc
      
          Run Build Command(s):/usr/bin/make -f Makefile cmTC_a6349/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      
      
      
      
      
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      
      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler
      
          "/usr/bin/cc"
      
        is not able to compile a simple test program.
      
        It fails with the following output:
      
          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-WGiugk
      
          Run Build Command(s):/usr/bin/make -f Makefile cmTC_3370b/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      
      
      
      
      
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

```
Fails at building wheels for transformers and onnx
I installed onnx manually using pip3 
```
absl-py==1.3.0
aiosignal==1.3.1
appdirs==1.4.4
appnope==0.1.3
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
audioread==3.0.0
autopage==0.5.1
Babel==2.11.0
backcall==0.2.0
backoff==1.11.1
beautifulsoup4==4.11.1
binaryornot==0.4.4
black==22.3.0
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
chardet==5.1.0
charset-normalizer==2.1.1
click==8.1.3
cmaes==0.9.0
cmake==3.25.0
cmd2==2.4.2
colorama==0.4.6
coloredlogs==15.0.1
colorlog==6.7.0
commonmark==0.9.1
contourpy==1.0.6
cycler==0.11.0
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
decorator==5.1.1
dill==0.3.4
distlib==0.3.6
dlinfo==1.2.1
dm-tree==0.1.7
exceptiongroup==1.0.4
execnet==1.9.0
executing==1.2.0
faiss-cpu==1.7.3
fastjsonschema==2.16.2
filelock==3.8.2
fire==0.5.0
flake8==6.0.0
flatbuffers==2.0.7
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
fugashi==1.1.2a6
gast==0.4.0
gitdb==4.0.10
google-pasta==0.2.0
graphql-core==3.2.3
grpcio==1.51.1
h5py==3.7.0
humanfriendly==10.0
hypothesis==6.61.0
idna==3.4
importlib-metadata==4.13.0
iniconfig==1.1.1
ipadic==1.0.0
isodate==0.6.1
isort==5.11.2
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
jupyter_core==5.1.0
kenlm==0.1
keras==2.11.0
keras-nlp==0.3.1
kiwisolver==1.4.4
language-tags==1.1.0
libclang==14.0.6
llvmlite==0.39.1
lxml==4.9.2
Mako==1.2.4
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
mccabe==0.7.0
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nltk==3.8
numba==0.56.4
numpy==1.23.5
oauthlib==3.2.2
onnx==1.13.0
onnxconverter-common==1.13.0
onnxruntime==1.13.1
onnxruntime-tools==1.7.0
opt-einsum==3.3.0
packaging==22.0
pandas==1.5.2
parameterized==0.8.1
parso==0.8.3
pathspec==0.10.3
pbr==5.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
Pint==0.16.1
plac==1.3.5
platformdirs==2.6.0
plotly==5.11.0
pluggy==1.0.0
pooch==1.6.0
portalocker==2.0.0
poyo==0.5.0
prettytable==3.5.0
prompt-toolkit==3.0.36
protobuf==3.19.6
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
py3nvml==0.2.7
pyarrow==10.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.10.0
pycparser==2.21
pyctcdecode==0.4.0
pyflakes==3.0.1
Pygments==2.13.0
pygtrie==2.5.0
pyknp==0.6.1
pylatexenc==2.10
pynvml==11.4.1
pyparsing==3.0.9
pyperclip==1.8.2
pypng==0.20220715.0
pyrsistent==0.19.2
pytest==7.2.0
pytest-timeout==2.1.0
pytest-xdist==3.1.0
python-dateutil==2.8.2
python-slugify==7.0.0
pytz==2022.6
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
ray==2.2.0
rdflib==6.2.0
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
resampy==0.4.2
responses==0.18.0
rfc3986==1.5.0
rich==11.2.0
rjieba==0.1.11
rouge-score==0.1.2
rsa==4.9
sacrebleu==1.5.1
sacremoses==0.0.53
safetensors==0.2.6
scikit-learn==1.2.0
scipy==1.8.1
sentencepiece==0.1.97
six==1.16.0
smmap==5.0.0
sortedcontainers==2.4.0
soundfile==0.11.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.45
stack-data==0.6.2
stevedore==4.1.1
SudachiDict-core==20221021
SudachiPy==0.6.6
sympy==1.11.1
tabulate==0.9.0
tenacity==8.1.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
tensorflow-estimator==2.11.0
tensorflow-metal==0.7.0
tensorstore==0.1.28
termcolor==2.1.1
text-unidecode==1.3
threadpoolctl==3.1.0
timeout-decorator==0.5.0
tokenizers==0.13.2
tomli==2.0.1
toolz==0.12.0
torch==1.13.0
torchaudio==0.13.0
torchvision==0.14.0
tqdm==4.64.1
traitlets==5.7.1
typing_extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
unidic==1.1.0
unidic-lite==1.0.8
uritemplate==4.1.1
urllib3==1.26.13
virtualenv==20.17.1
wasabi==0.10.1
wcwidth==0.2.5
websocket-client==1.4.2
Werkzeug==2.2.2
wrapt==1.14.1
xmltodict==0.13.0
xxhash==3.1.0
yarl==1.8.2
zipp==3.11.0
```
I am on 3.9.11


when running 
 conda install -c conda-forge onnxruntime
it says all the components were installed





hmmm I accepted the xcode license agreement.
It's built transformers I think it fails to build onnx, but whats odd is it uninstalls onnx. Maybe I can do with out it and remove it out of setup.py?
```
Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=e63397c8685e05971af9d4767e872200eb8393ed081e37582f8d24760daaca08
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w75hif9g/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      -- The C compiler identification is AppleClang 14.0.0.14000029
      -- The CXX compiler identification is AppleClang 14.0.0.14000029
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found PythonInterp: /Users/michaelchung/Code/transformers/.env/bin/python3 (found version "3.9.11")
      -- Found PythonLibs: /opt/homebrew/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib (found version "3.9.11")
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

```

Alright dev environment installed, the key here was not to use conda but miniforge

Ok draft created 

Do you know how to convert .pth model to config.json/pytorch_model.bin for RWKV4neo?

I have a conversion script + draft that has consistent logit ordering with the official implementation here:

conversion script: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py
model in torch: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py

I can clean it up and turn it into a PR if that would help?

Sure, there is also #22797 that should be in a pretty good state! I'm about to review it but feel free to add your touch to it if you feel like it! 

Oh cool, that one looks awesome!

> I have a conversion script + draft that has consistent logit ordering with the official implementation here:
> 
> conversion script: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py model in torch: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py
> 
> I can clean it up and turn it into a PR if that would help?

@tensorpro I could really use your scripts but I get 404 when I try to access those links. :/

Ah sorry, the links broke when I changed the branch I was working in. I edited the comment to point to the right branch

That said, you may want to use the code in #22797 since it will be closer to the official HF version and already supports CUDA accelerated WKV.