Building your RHEL AI environment Red Hat Enterprise Linux AI 1.2 Creating accounts, initalizing RHEL AI, downloading models, and serving/chat customizations Red Hat RHEL AI Documentation Team

Commands:
rhc connect --organization <org id> --activation-key <created key>
sudo mkdir -p /etc/ilab sudo touch /etc/ilab/insights-opt-out
ilab system info
ilab config init
Welcome to InstructLab CLI. This guide will help you to setup your environment. Detecting Hardware We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. Is this correct? [y/N]:
We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. This profile is the best approximation for your system based off of the amount of vRAM. We modified it to match the number of GPUs you have. Is this profile correct? [Y/n]:
Please choose a train profile to use. Train profiles assist with the complexity of configuring specific GPU hardware with the InstructLab Training library. You can still take advantage of hardware acceleration for training even if your hardware is not listed. [0] No profile (CPU, Apple Metal, AMD ROCm) [1] Nvidia A100/H100 x2 (A100_H100_x2.yaml) [2] Nvidia A100/H100 x4 (A100_H100_x4.yaml) [3] Nvidia A100/H100 x8 (A100_H100_x8.yaml) [4] Nvidia L40 x4 (L40_x4.yaml) [5] Nvidia L40 x8 (L40_x8.yaml) [6] Nvidia L4 x8 (L4_x8.yaml) Enter the number of your choice [hit enter for no profile] [0]:
You selected: Nvidia A100/H100 x8 (A100_H100_x8.yaml) Initialization completed successfully, you're ready to start using `ilab`. Enjoy!
ilab config edit
evaluate: base_branch: null base_model: ~/.cache/instructlab/models/granite-7b-starter branch: null gpus: <num-gpus>
serve: backend: vllm chat_template: auto host_port: 127.0.0.1:8000 llama_cpp: gpu_layers: -1 llm_family: '' max_ctx_size: 4096 model_path: ~/.cache/instructlab/models/granite-7b-redhat-lab vllm: llm_family: '' vllm_args: ["--tensor-parallel-size", "<num-gpus>"] gpus: <num-gpus>
rm -rf ~/.local/share/instructlab/taxonomy/ ; git clone https://github.com/RedHatOfficial/rhelai-sample-taxonomy.git ~/.local/share/instructlab/taxonomy/
├─ ~/.cache/instructlab/models/ 1 ├─ ~/.local/share/instructlab/datasets 2 ├─ ~/.local/share/instructlab/taxonomy 3 ├─ ~/.local/share/instructlab/phased/<phase1-or-phase2>/checkpoints/ 4
ilab config show
ilab config edit
ilab config edit
train: max_batch_len=220000 nproc_per_node=8 eval: gpus=1
train: max_batch_len=210000 nproc_per_node=4 eval: gpus=1
train: max_batch_len=200000 nproc_per_node=2 eval: gpus=1
ilab model download --repository docker://<repository_and_model> --release <release>
ilab model download --repository docker://registry.redhat.io/rhelai1/granite-7b-starter --release latest
ilab model list
+-----------------------------------+---------------------+---------+ | Model Name | Last Modified | Size | +-----------------------------------+---------------------+---------+ | models/prometheus-8x7b-v2-0 | 2024-08-09 13:28:50 | 87.0 GB| | models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 | 87.0 GB| | models/granite-7b-redhat-lab | 2024-08-09 14:28:40 | 12.6 GB| | models/granite-7b-starter | 2024-08-09 14:40:35 | 12.6 GB| +-----------------------------------+---------------------+---------+
ls ~/.cache/instructlab/models
granite-7b-starter granite-7b-redhat-lab
ilab model serve
ilab model serve --model-path <model-path>
ilab model serve --model-path ~/.cache/instructlab/models/granite-8b-code-instruct
INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/granite-8b-code-instruct' with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see http://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server.
mkdir -p USDHOME/.config/systemd/user
cat << EOF > USDHOME/.config/systemd/user/ilab-serve.service [Unit] Description=ilab model serve service [Install] WantedBy=multi-user.target default.target 1 [Service] ExecStart=ilab model serve --model-family granite Restart=always EOF
systemctl --user daemon-reload
systemctl --user start ilab-serve.service
systemctl --user status ilab-serve.service
journalctl --user-unit ilab-serve.service
sudo loginctl enable-linger
systemctl --user stop ilab-serve.service
mkdir -p `pwd`/nginx/ssl/
cat > openssl.cnf <<EOL [ req ] default_bits = 2048 distinguished_name = <req-distinguished-name> 1 x509_extensions = v3_req prompt = no [ req_distinguished_name ] C = US ST = California L = San Francisco O = My Company OU = My Division CN = rhelai.redhat.com [ v3_req ] subjectAltName = <alt-names> 2 basicConstraints = critical, CA:true subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer [ alt_names ] DNS.1 = rhelai.redhat.com 3 DNS.2 = www.rhelai.redhat.com 4
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout `pwd`/nginx/ssl/rhelai.redhat.com.key -out `pwd`/nginx/ssl/rhelai.redhat.com.crt -config openssl.cnf
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout
mkdir -p `pwd`/nginx/conf.d echo 'server { listen 8443 ssl; server_name <rhelai.redhat.com> 1 ssl_certificate /etc/nginx/ssl/rhelai.redhat.com.crt; ssl_certificate_key /etc/nginx/ssl/rhelai.redhat.com.key; location / { proxy_pass http://127.0.0.1:8000; proxy_set_header Host USDhost; proxy_set_header X-Real-IP USDremote_addr; proxy_set_header X-Forwarded-For USDproxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto USDscheme; } } ' > `pwd`/nginx/conf.d/rhelai.redhat.com.conf
podman run --net host -v `pwd`/nginx/conf.d:/etc/nginx/conf.d:ro,Z -v `pwd`/nginx/ssl:/etc/nginx/ssl:ro,Z nginx
ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url
curl --location 'https://rhelai.redhat.com:8443/v1' --header 'Content-Type: application/json' --header 'Authorization: Bearer <api-key>' --data '{ "model": "/var/home/cloud-user/.cache/instructlab/models/granite-7b-redhat-lab", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Hello!" } ] }' | jq .
openssl s_client -connect rhelai.redhat.com:8443 </dev/null 2>/dev/null | openssl x509 -outform PEM > server.crt
sudo cp server.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
cat server.crt >> USD(python -m certifi)
ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url https://rhelai.redhat.com:8443/v1
ilab model chat
ilab model chat --model <model-path>
ilab model chat --model ~/.cache/instructlab/models/granite-8b-code-instruct
ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ GRANITE-8B-CODE-INSTRUCT (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default]
export VLLM_API_KEY=USD(python -c 'import secrets; print(secrets.token_urlsafe())')
echo USDVLLM_API_KEY
ilab config edit
serve: vllm: vllm_args: - --api-key - <api-key-string>
ilab model chat
openai.AuthenticationError: Error code: 401 - {'error': 'Unauthorized'}
ilab model chat -m granite-7b-redhat-lab --endpoint-url https://inference.rhelai.com/v1 --api-key USDVLLM_API_KEY
ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ GRANITE-7B-LAB (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default]