SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The performance of ResNet-18 with different batch sizes on the four NVIDIA 1080Ti GPUs, A100, and AGX Orin platforms is illustrated in Figures 6, 7, and 8, respectively. Results for MobileNetV3-small on AGX Orin are presented in Figure 10. A notable observation is that AsyncFGD performance appears largely insensitive to batch size. In contrast, other algorithms typically exhibit poorer performance with smaller batch sizes. Particularly, when the batch size is reduced to 1, these algorithms offer negligible performance improvements over vanilla FGD. Furthermore, the overall
acceleration on a single device is constrained by computational power. For instance, while AsyncFGD achieves a speedup of  $2.84 \times$  on a four GPU cluster, it only delivers a  $2.11 \times$  speedup on a single AGX Orin. Communication also imposes a limit on the overall acceleration, as demonstrated by the superior performance on the A100 in comparison to the four-GPU cluster. This is attributable to the elimination of communication overhead on a single device, except for the sending and receiving operations of CUDA kernels.
Results for MobileNetV3-small with different batch sizes on CPU are depicted in Figure 9. Due to the inherently sequential execution pattern of CPUs, the acceleration is constrained, resulting in only modest speedup and advantage over other algorithms.
(a) Batch size  $= 1$
(b) Batch size  $= 2$
(c) Batch size  $= 3$
(d) Batch size  $= 4$
Figure 6: Acceleration with differernt batch size on ResNet-18, cluster with 4 Nvidia 1080 Ti
I.3 Memory Profiling on Other Basic Units of Convolutional Neural Networks
This section outlines memory profiling for basic units within a Convolutional Neural Network (CNN). Commonly, a CNN layer is coupled with a batch normalization layer and an activation layer using ReLU, so we've combined these elements for our memory testing. We examine the memory consumption against the number of layers and present the results in Figure 11(a). For further examination, we also assess the memory consumption against the number of output channels and batch size, with results shown in Figures 11(b) and 11(c), respectively.
Our findings reveal that implementing forward gradients can significantly reduce memory consumption. Generally, the majority of memory usage in CNNs stems from intermediate results, since CNNs often operate in a 'broadcast then product' pattern (to be specific, they are referred as 'img2col'). Consequently, the additional memory required by the random tangent in AsyncFGD is minimal. As such, the memory consumption appears to be invariant to the number of layers, mainly because in the forward pass we discard almost all the intermediate variables.
(d) Batch size  $= 4$
Figure 7: Acceleration with different batch size on ResNet-18, A100
(c) Batch size  $= 3$
Figure 8: Acceleration with differentert batch size on ResNet-18, AGX Orin
(d) Batch size  $= 4$
(d) Batch size  $= 4$
Figure 9: Acceleration with differenternt batch size on MobileNetV3-small, Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
(c) Batch size  $= 3$
Figure 10: Acceleration with different batch size on MobileNetV3-small, AGX Orin
Table 8: Network architecture for  $ConvS_{ReLU}$  and  $ConvS_{Tanh}$ .  $ConvS_{ReLU}$  denotes using ReLU for the activation functions and  $ConvS_{Tanh}$  denotes using Tanh as activation functions
Layer	Type	Params
conv1	Conv2d	out_channels=32, kernel_size=5, stride=1, padding=2
act1	ReLU/Tanh	N/A
pool1	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
fc1	Linear	out_features=1000
act2	ReLU/Tanh	N/A
fc2	Linear	out_features=10
Table 9: Network architecture for  $FCS_{ReLU}$  and  $FCS_{Tanh}$ .  $FCS_{ReLU}$  denotes using ReLU for the activation functions and  $FCS_{Tanh}$  denotes using Tanh as activation functions
Layer	Type	Params
flatten	Flatten	N/A
fc1	Linear	out_features=1024
ac1	ReLU/Tanh	N/A
fc2	Linear	out_features=512
ac2	ReLU/Tanh	N/A
fc3	Linear	out_features=256
Table 10: Network architecture for  $FCL_{ReLU}$  and  $FCL_{Tanh}$ .  $FCL_{ReLU}$  denotes using ReLU for the activation functions and  $FCL_{Tanh}$  denotes using Tanh as activation functions
Layer	Type	Params
flatten	Flatten	N/A
fc1	Linear	out_features=1024
bn1	BatchNorm1d	N/A
ac1	ReLU/Tanh	N/A
fc2	Linear	out_features=1024
bn2	BatchNorm1d	N/A
ac2	ReLU/Tanh	N/A
fc3	Linear	out_features=1024
bn3	BatchNorm1d	N/A
ac3	ReLU/Tanh	N/A
fc4	Linear	out_features=1024
bn4	BatchNorm1d	N/A
ac4	ReLU/Tanh	N/A
fc5	Linear	out_features=512
bn5	BatchNorm1d	N/A
ac5	ReLU/Tanh	N/A
fc6	Linear	out_features=10
(a) Different number of layers
(b) Different channels
Figure 11: Memory consumption of basic units in convolutional networks, batch size=64, channels=3 and number of layers=18 unless appears as in the x axis
(c) Different batch size
Table 11: Network architecture for  $ConvL_{ReLU}$  and  $ConvL_{Tanh}$ .  $ConvL_{ReLU}$  denotes using ReLU for the activation functions and  $ConvL_{Tanh}$  denotes using Tanh as activation functions
Layer	Type	Params
conv1	Conv2d	out_channels=32, kernel_size=3, stride=1, padding=2
bn1	BatchNorm2d	N/A
act1	ReLU/Tanh	N/A
pool1	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
conv2	Conv2d	out_channels=64, kernel_size=3, stride=1, padding=2
bn2	BatchNorm2d	N/A
act2	ReLU/Tanh	N/A
pool2	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
conv3	Conv2d	out_channels=128, kernel_size=3, stride=1, padding=2
bn3	BatchNorm2d	N/A
act3	ReLU/Tanh	N/A
pool3	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
conv4	Conv2d	out_channels=256, kernel_size=3, stride=1, padding=2
bn4	BatchNorm2d	N/A
act4	ReLU/Tanh	N/A
pool4	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
conv5	Conv2d	out_channels=512, kernel_size=3, stride=1, padding=2
bn5	BatchNorm2d	N/A
act5	ReLU/Tanh	N/A
pool5	Maxpool12d	kernel_size=2, stride=2, padding=0, dilation=1
flatten	Flatten	N/A
fc1	Linear	out_features=10