Package: llm-mlc
llm-mlcLLMplugin for running models usingMLCInstallationInstall this plugin in the same environment asllm.llminstallllm-mlcYou need to install two dependencies manually -mlc-chat-nightlyandmlc-ai-nightly- because the installation process differs from one platform to another in a way that is not yet automated.The steps for this aredescribed in detail on the mlc.ai/packagesite.If you are on an Apple Silicon M1/M2 Mac you can run this command:llmmlcpipinstall--pre--force-reinstall\mlc-ai-nightly\mlc-chat-nightly\-fhttps://mlc.ai/wheelsThellm mlc pipcommand here ensures thatpipwill run in the same virtual environment asllmitself.For other systems,follow the instructions here.Finally, run thellm mlc setupcommand to complete the installation:llmmlcsetupThis will setupgit lfsand use it to download some extra dependencies:Git LFS is not installed. Should I run 'git lfs install' for you?
Install Git LFS? [y/N]: y
Updated Git hooks.
Git LFS initialized.
Downloading prebuilt binaries...
Cloning into '/Users/simon/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib'...
remote: Enumerating objects: 221, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 221 (delta 59), reused 56 (delta 32), pack-reused 135
Receiving objects: 100% (221/221), 52.06 MiB | 9.13 MiB/s, done.
Resolving deltas: 100% (152/152), done.
Updating files: 100% (60/60), done.
Ready to install models in /Users/simon/Library/Application Support/io.datasette.llm/mlcInstalling modelsAfter installation you will need to download a model using thellm mlc download-modelcommand.Here's how to download and install Llama 2:llmmlcdownload-modelLlama-2-7b-chat--aliasllama2This will download around 8GB of content.You can also useLlama-2-13b-chat(about 15.15GB) orLlama-2-70b-chat(extremely big), though these files are a lot larger.The-a/--aliasis optional, but can be used to set a shorter alias for the model. This can then be used withllm -m <alias>instead of the full name.Thedownload-modelcommand also takes a URL to one ofthe MLC repositories on Hugging Face.For example, to installmlc-chat-WizardLM-13B-V1:llmmlcdownload-modelhttps://huggingface.co/mlc-ai/mlc-chat-WizardLM-13B-V1.2-q4f16_1You can see a full list of models you have installed this way using:llmmlcmodelsThis will also show the name of the model you should use to activate it, e.g.:MlcModel: mlc-chat-Llama-2-7b-chat-hf-q4f16_1 (aliases: llama2, Llama-2-7b-chat)Running a prompt through a modelOnce you have downloaded and added a model, you can run a prompt like this:llm-mLlama-2-7b-chat'five names for a cute pet ferret'Great! Here are five cute and creative name suggestions for a pet ferret:Ferbie - a playful and affectionate name for a friendly and outgoing ferret.Mr. Whiskers - a suave and sophisticated name for a well-groomed and dignified ferret.Luna - a celestial and dreamy name for a curious and adventurous ferret.Felix - a cheerful and energetic name for a lively and mischievous ferret.Sprinkles - a fun and playful name for a happy and energetic ferret with a sprinkle of mischief.Remember, the most important thing is to choose a name that you and your ferret will love and enjoy!And to send a follow-up prompt to continue the current conversation, use-c:llm-c'two more'Of course! Here are two more cute name ideas for a pet ferret:Digger - a fun and playful name that suits a pet that loves to dig and burrow, and is also a nod to the ferret's natural instincts as a burrower.Gizmo - a fun and quirky name that suits a pet with a curious and mischievous personality, and is also a nod to the ferret's playful and inventive nature.Model optionsThese options are available for all models. They mostly take a floating point value between 0.0 and 1.0.-o temperature: A higher temperature encourages more diverse outputs, while a lower temperature produces more deterministic outputs.-o top_p: At each step, we select tokens from the minimal set that has a cumulative probability exceeding this value.-o repetition_penalty: Controls the likelihood of the model generating repeated texts.-o max_gen_len: Takes an integer, which controls the maximum length of the generated text.Use them like this:llm-mLlama-2-7b-chat\-otemperature0.5\-otop_p0.9\-orepetition_penalty0.9\-omax_gen_len100\'five names for a cute pet ferret'TheMLC documentationhas more details on these options.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cdllm-mlc
python3-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:pipinstall-e'.[test]'To run the tests:pytest