Chapter 5. Serving and chatting with your new model You must deploy the model to your machine by serving the model. This deploys the model and makes the model available for interacting and chatting. 5.1. Serving the new model To interact with your new model, you must activate the model in a machine through serving. The ilab model serve command starts a vLLM server that allows you to chat with the model. Prerequisites You installed RHEL AI with the bootable container image. You initialized InstructLab. You customized your taxonomy tree, ran synthetic data generation, trained, and evaluated your new model. You need root user access on your machine. Procedure You can serve the model by running the following command: USD ilab model serve --model-path <path-to-best-performed-checkpoint> where: <path-to-best-performed-checkpoint> Specify the full path to the checkpoint you built after training. Your new model is the best performed checkpoint with its file path displayed after training. Example command: USD ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945/ Important Ensure you have a slash / at the end of your model path. Example output of the ilab model serve command USD ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/<checkpoint> INFO 2024-03-02 02:21:11,352 lab.py:201 Using model /home/example-user/.local/share/instructlab/checkpoints/hf_format/checkpoint_1945 with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see http://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server. 5.2. Chatting with the new model You can chat with your model that has been trained with your data. Prerequisites You installed RHEL AI with the bootable container image. You initialized InstructLab. You customized your taxonomy tree, ran synthetic data generated, trained and evaluated your new model. You served your checkpoint model. You need root user access on your machine. Procedure Since you are serving the model in one terminal window, you must open a new terminal window to chat with the model. To chat with your new model, run the following command: USD ilab model chat --model <path-to-best-performed-checkpoint-file> where: <path-to-best-performed-checkpoint-file> Specify the new model checkpoint file you built after training. Your new model is the best performed checkpoint with its file path displayed after training. Example command: USD ilab model chat --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945 Example output of the InstructLab chatbot USD ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ CHECKPOINT_1945 (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default] Type exit to leave the chatbot.

Commands:
ilab model serve --model-path <path-to-best-performed-checkpoint>
ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945/
ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/<checkpoint> INFO 2024-03-02 02:21:11,352 lab.py:201 Using model /home/example-user/.local/share/instructlab/checkpoints/hf_format/checkpoint_1945 with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see http://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server.
ilab model chat --model <path-to-best-performed-checkpoint-file>
ilab model chat --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945
ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ CHECKPOINT_1945 (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default]