Chapter 3. Red Hat Enterprise Linux AI hardware requirements Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the Granite starter model on Red Hat Enterprise Linux AI. 3.1. Hardware requirements for end-to-end workflow of Granite models The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), multi-phase training, and evaluating a custom Granite model. 3.1.1. Bare metal Hardware vendor Supported accelerators (GPUs) Aggregate GPU memory Recommended additional disk storage NVIDIA 2xA100 4xA100 8xA100 160 GB 320 GB 640 GB 3 TB NVIDIA 2xH100 4xH100 8xH100 160 GB 320 GB 640 GB 3 TB NVIDIA 4xL40S 8xL40S 192 GB 384 GB 3 TB AMD (Technology preview) 2xMI300X 4xMI300X 8xMI300X 384 GB 768 GB 1536 GB 3 TB 3.1.2. IBM Cloud Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory IBM Cloud Instances Recommended additional disk storage NVIDIA 8xH100 640 GB gx3d-160x1792x8h100 3 TB 3.1.3. Amazon Web Services (AWS) Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory AWS Instances Recommended additional disk storage NVIDIA 8xA100 320 GB p4d.24xlarge 3 TB NVIDIA 8xA100 640 GB p4de.24xlarge 3 TB NVIDIA 8xH100 640 GB p5.48xlarge 3 TB NVIDIA 8xL40S 384 GB g6e.48xlarge 3 TB 3.1.4. Azure Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory Azure Instances Recommended additional disk storage NVIDIA 8xA100 640 GB Standard_ND96amsr_A100_v4 3 TB NVIDIA 4xA100 320 GB Standard_ND96asr_A100_v4 3 TB NVIDIA 8xH100 640 GB Standard_ND96isr_H100_v5 3 TB AMD (Technology preview) 8xMI300X 1536 GB Standard_ND96isr_MI300X_v5 3 TB 3.1.5. Google Cloud Platform (GCP) Hardware vendor Supported accelerators (GPUs) Aggregate GPU Memory GCP Instances Recommended additional disk storage NVIDIA 8xA100 640 GB a2-highgpu-8g 3 TB NVIDIA 8xH100 640 GB a3-highgpu-8g a3-megagpu-8g 3 TB 3.2. Hardware requirements for inference serving Granite models The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI. 3.2.1. Bare metal Hardware vendor Supported accelerators (GPUs) minimum Aggregate GPU memory Recommended additional disk storage NVIDIA A100 80 GB 1 TB NVIDIA H100 80 GB 1 TB NVIDIA L40S 48 GB 1 TB NVIDIA L4 24 GB 1 TB AMD (Technology preview) MI300X 192 GB 1 TB Intel (Technology preview) Gaudi3 128 GB 1 TB 3.2.2. Amazon Web Services (AWS) Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory AWS Instance family Recommended additional disk storage NVIDIA A100 40 GB P4d series 1 TB NVIDIA H100 80 GB P5 series 1 TB NVIDIA L40S 48 GB G6e series 1 TB NVIDIA L4 24 GB G6 series 1 TB 3.2.3. IBM cloud Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory IBM Cloud Instance family Recommended additional disk storage NVIDIA L40S 48 GB gx series 1 TB NVIDIA L4 24 GB gx series 1 TB 3.2.4. Azure Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory Azure Instance family Recommended additional disk storage NVIDIA A100 80 GB ND series 1 TB NVIDIA H100 80 GB ND sereis 1 TB AMD (Technology preview) MI300X 192 GB ND series 1 TB 3.2.5. Google Cloud Platform (GCP) Hardware vendor Supported accelerators (GPUs) Minimum Aggregate GPU Memory GCP Instance family Recommended additional disk storage NVIDIA A100 40 GB A2 series 1 TB NVIDIA H100 80 GB A3 series 1 TB NVIDIA 4xL4 96 GB G2 series 1 TB

Commands: