Chapter 3. Downloading Large Language models Red Hat Enterprise Linux AI allows you to customize or chat with various Large Language Models (LLMs) provided and built by Red Hat and IBM. You can download these models from the Red Hat RHEL AI registry. You can upload any custom model to an S3 bucket. Table 3.1. Red Hat Enterprise Linux AI version 1.4 LLMs Large Language Models (LLMs) Type Size Purpose Model family NVIDIA Accelerator Support granite-7b-starter Base model 12.6 GB Base model for customizing and fine-tuning Granite 2 Not available granite-7b-redhat-lab LAB fine-tuned Granite model 12.6 GB Granite model for inference serving Granite 2 Not available granite-8b-starter-v1 Base model 16.0 GB Base model for customizing and fine-tuning Granite 3 General availability granite-8b-lab-v1 LAB fine-tuned granite model 16.0 GB Granite model for inference serving Granite 3 General availability granite-8b-lab-v2-preview LAB fine-tuned granite model 16.0 GB Preview of the version 2 8b Granite model for inference serving Granite 3 Technology preview granite-3.1-8b-starter-v1 LAB fine-tuned granite model 16.0 GB Version 1 of the Granite 3.1 base model for customizing and fine-tuning Granite 3.1 General availability granite-3.1-8b-lab-v1 LAB fine-tuned granite model 16.0 GB Version 1 of the Granite 3.1 model for inference serving Granite 3.1 General availability granite-8b-code-instruct LAB fine-tuned granite code model 15.0 GB LAB fine-tuned granite code model for inference serving Granite Code models Technology preview granite-8b-code-base Granite fine-tuned code model 15.0 GB Granite code model for inference serving Granite Code models Technology preview mixtral-8x7b-instruct-v0-1 Teacher/critic model 87.0 GB Teacher and critic model for running Synthetic data generation (SDG) Mixtral General availability prometheus-8x7b-v2-0 Evaluation judge model 87.0 GB Judge model for multi-phase training and evaluation Prometheus 2 General availability Important Using the `granite-8b-code-instruct` or `granite-8b-code-base` Large Language models (LLMS) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Models required for customizing the Granite LLM The granite-7b-starter or granite-8b-starter-v1 base LLM depending on your hardware vendor. The mixtral-8x7b-instruct-v0-1 teacher model for SDG. The prometheus-8x7b-v2-0 judge model for training and evaluation. Additional tools required for customizing an LLM The Low-rank adaptation (LoRA) adaptors enhance the efficiency of the Synthetic Data Generation (SDG) process. The skills-adapter-v3 LoRA layered skills adapter for SDG. The knowledge-adapter-v3 LoRA layered knowledge adapter for SDG. Example command for downloading the adaptors Important The LoRA layered adapters do not show up in the output of the ilab model list command. You can see the skills-adapter-v3 and knowledge-adapter-v3 files in the ls ~/.cache/instructlab/models folder. 3.1. Downloading the models from a Red Hat repository You can download the additional optional models created by Red Hat and IBM. Prerequisites You installed RHEL AI with the bootable container image. You initialized InstructLab. You created a Red Hat registry account and logged in on your machine. You have root user access on your machine. Procedure To download the additional LLM models, run the following command: USD ilab model download --repository docker://<repository_and_model> --release <release> where: <repository_and_model> Specifies the repository location of the model as well as the model. You can access the models from the registry.redhat.io/rhelai1/ repository. <release> Specifies the version of the model. Set to 1.4 for the models that are supported on RHEL AI version 1.4. Set to latest for the latest version of the model. Example command USD ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3.1-8b-starter-v1 --release latest Verification You can view all the downloaded models, including the new models after training, on your system with the following command: USD ilab model list Example output You can also list the downloaded models in the ls ~/.cache/instructlab/models folder by running the following command: USD ls ~/.cache/instructlab/models Example output granite-3.1-8b-starter-v1 granite-3.1-8b-lab-v1

Commands:
ilab model download --repository docker://registry.redhat.io/rhelai1/knowledge-adapter-v3 --release latest
ilab model download --repository docker://<repository_and_model> --release <release>
ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3.1-8b-starter-v1 --release latest
ilab model list
+-----------------------------------+---------------------+---------+ | Model Name | Last Modified | Size | +-----------------------------------+---------------------+---------+ | models/prometheus-8x7b-v2-0 | 2024-08-09 13:28:50 | 87.0 GB| | models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 | 87.0 GB| | models/granite-3.1-8b-starter-v1 | 2024-08-09 14:28:40 | 16.6 GB| | models/granite-3.1-8b-lab-v1 | 2024-08-09 14:40:35 | 16.6 GB| +-----------------------------------+---------------------+---------+
ls ~/.cache/instructlab/models
granite-3.1-8b-starter-v1 granite-3.1-8b-lab-v1