GPU Cloud Servers Workstations TensorBook Resources Careers +1 (866) 711-2025 Talk to an engineer Deep Learning GPU Benchmarks GPU training speeds using PyTorch/TensorFlow for computer vision (CV), NLP, text-to-speech (TTS), etc. PyTorch GPU Benchmarks Visualization chart table Metric throughput batch size throughput/watt batch size/watt throughput/$ batch size/$ Precision fp32/tf32 fp16 Number of GPUs 1x 2x 3x 4x 8x All GPUs Model All Models ssd maskrcnn resnet50 gnmt transformerxlbase transformerxllarge bert_base_squad bert_large_squad tacotron2 waveglow ncf Relative Training Throughput w.r.t 1xV100 32GB (All Models) 0.0 0.5 1.0 1.5 2.0 2.5 A100 40GB SXM4 A100 40GB PCIe RTX A6000 Lambda Cloud — RTX A6000 RTX 3090 RTX A40 RTX A5000 V100 32GB RTX 8000 Titan RTX RTX 6000 RTX 3080 RTX A4000 RTX 2080Ti RTX 5000 RTX 3080 Max-Q GTX 1080Ti RTX 3070 RTX 2080 SUPER MAX-Q RTX 2080 MAX-Q RTX 2070 MAX-Q TensorFlow GPU Benchmarks Visualization chart table Metric throughput batch size throughput/watt batch size/watt throughput/$ batch size/$ Precision fp32/tf32 fp16 Number of GPUs 1x 2x 3x 4x 8x All GPUs Model All Models resnet50 resnet152 inception3 inception4 vgg16 alexnet Relative Training Throughput w.r.t 1xV100 32GB (All Models) 0.0 0.5 1.0 1.5 A100 40GB PCIe Lambda Cloud — RTX A6000 RTX A6000 RTX 3090 V100 32GB RTX 3080 RTX 8000 RTX 2080Ti GTX 1080Ti RTX 2080 SUPER MAX-Q RTX 2070 MAX-Q RTX 2080 MAX-Q GPU Benchmark Methodology To measure the relative effectiveness of GPUs when it comes to training neural networks we’ve chosen training throughput as the measuring stick. Training throughput measures the number of samples (e.g. tokens, images, etc...) processed per second by the GPU. Using throughput instead of Floating Point Operations per Second (FLOPS) brings GPU performance into the realm of training neural networks. Training throughput is strongly correlated with time to solution — since with high training throughput, the GPU can run a dataset more quickly through the model and teach it faster. In order to maximize training throughput it’s important to saturate GPU resources with large batch sizes, switch to faster GPUs, or parallelize training with multiple GPUs. Additionally, it’s also important to test throughput using state of the art (SOTA) model implementations across frameworks as it can be affected by model implementation. TensorFlow We are working on new benchmarks using the same software version across all GPUs. Lambda's TensorFlow benchmark code is available here. The RTX A6000 was benchmarked using NGC's TensorFlow 20.10 docker image using Ubuntu 18.04, TensorFlow 1.15.4, CUDA 11.1.0, cuDNN 8.0.4, NVIDIA driver 455.32, and Google's official model implementations. The A100s, RTX 3090, and RTX 3080 were benchmarked using Ubuntu 18.04, TensorFlow 1.15.4, CUDA 11.1.0, cuDNN 8.0.4, NVIDIA driver 455.45.01, and Google's official model implementations. Pre-ampere GPUs were benchmarked using TensorFlow 1.15.3, CUDA 10.0, cuDNN 7.6.5, NVIDIA driver 440.33, and Google's official model implementations. PyTorch We are working on new benchmarks using the same software version across all GPUs. Lambda's PyTorch benchmark code is available here. The RTX A6000, A100s, RTX 3090, and RTX 3080 were benchmarked using NGC's PyTorch 20.10 docker image with Ubuntu 18.04, PyTorch 1.7.0a0+7036e91, CUDA 11.1.0, cuDNN 8.0.4, NVIDIA driver 460.27.04, and NVIDIA's optimized model implementations. Pre-ampere GPUs were benchmarked using NGC's PyTorch 20.01 docker image with Ubuntu 18.04, PyTorch 1.4.0a0+a5b4d78, CUDA 10.2.89, cuDNN 7.6.5, NVIDIA driver 440.33, and NVIDIA's optimized model implementations. © 2021 Lambda Forum About Blog Terms of Sale Careers Contact