Chapter 3. Red Hat Enterprise Linux AI hardware requirements Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the granite-7b-starter model on Red Hat Enterprise Linux AI. 3.1. Hardware requirements for end-to-end workflow of Granite models The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), training, and evaluating a custom Granite model. 3.1.1. Bare metal Hardware vendor Supported accelerators (GPUs) Aggregate GPU memory Recommended additional disk storage NVIDIA 2xA100 4xA100 8xA100 160 GB 320 GB 640 GB 3 TB NVIDIA 2xH100 4xH100 8xH100 160 GB 320 GB 640 GB 3 TB NVIDIA 4xL40S 8xL40S 192 GB 384 GB 3 TB AMD (Technology preview) 2xMI300X 4xMI300X 8xMI300X 384 GB 768 GB 1536 GB 3 TB 3.1.2. IBM Cloud Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory IBM Cloud Instances Recommended additional disk storage NVIDIA 8xH100 640 GB gx3d-160x1792x8h100 3 TB 3.1.3. Amazon Web Services (AWS) Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory AWS Instances Recommended additional disk storage NVIDIA 8xA100 320 GB p4d.24xlarge 3 TB NVIDIA 8xA100 640 GB p4de.24xlarge 3 TB NVIDIA 8xH100 640 GB p5.48xlarge 3 TB NVIDIA 8xL40S 384 GB g6e.48xlarge 3 TB 3.1.4. Azure Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory Azure Instances Recommended additional disk storage NVIDIA 8xA100 640 GB Standard_ND96amsr_A100_v4 3 TB NVIDIA 4xA100 320 GB Standard_ND96asr_A100_v4 3 TB NVIDIA 8xH100 640 GB Standard_ND96isr_H100_v5 3 TB AMD (Technology preview) 8xMI300X 1536 GB Standard_ND96isr_MI300X_v5 3 TB 3.1.5. Google Cloud Platform (GCP) (Technology preview) Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory GCP Instances Recommended additional disk storage NVIDIA 8xA100 640 GB a2-highgpu-8g 3 TB NVIDIA 8xH100 640 GB a3-highgpu-8g a3-megagpu-8g 3 TB 3.2. Hardware requirements for inference serving Granite models The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI. 3.2.1. Bare metal Hardware vendor Supported accelerators (GPUs) minimum Aggregate GPU memory Recommended additional disk storage NVIDIA A100 80 GB 1 TB NVIDIA H100 80 GB 1 TB NVIDIA L40S 48 GB 1 TB NVIDIA L4 24 GB 1 TB AMD (Technology preview) MI300X 192 GB 1 TB 3.2.2. Amazon Web Services (AWS) Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory AWS Instance family Recommended additional disk storage NVIDIA A100 40 GB P4d series 1 TB NVIDIA H100 80 GB P5 series 1 TB NVIDIA L40S 48 GB G6e series 1 TB NVIDIA L4 24 GB G6 series 1 TB 3.2.3. IBM cloud Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory IBM Cloud Instance family Recommended additional disk storage NVIDIA L40S 48 GB gx series 1 TB NVIDIA L4 24 GB gx series 1 TB 3.2.4. Azure Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory Azure Instance family Recommended additional disk storage NVIDIA A100 80 GB ND series 1 TB NVIDIA H100 80 GB ND series 1 TB AMD (Technology preview) MI300X 192 GB ND series 1 TB 3.2.5. Google Cloud Platform (GCP) (Technology preview) Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory GCP Instance family Recommended additional disk storage NVIDIA A100 40 GB A2 series 1 TB NVIDIA H100 80 GB A3 series 1 TB NVIDIA 4xL4 96 GB G2 series 1 TB

Commands: