SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\text {C o m p u t e} = \frac {\text {A d d i t i o n a l c o m p u t e d u e t o G U T}}{\text {T o t a l C o m p u t e}}
$$
The total compute includes the forward pass, backward pass, model updates, gossip averaging, and tracking variable computation flops. We observe that for compact models such as ResNet and MobileNet, the compute overhead is around  $2\%$ . However, for larger models such as VGG-11, the compute overhead shoots up to  $15\%$ .
Table 12: Communication cost and memory requirement per agent during training of CIFAR-10 dataset on various model architectures and graph topologies.
Model	Graph	Graph Size	Method	Comm. Cost (GB)	Memory (MB)
ResNet-20	Ring	16	Gradient Tracking	83.66	129.60
QG-DSGdm	41.83	128.56
QG-GUTm	41.83	130.64
ResNet-20	Ring	32	Gradient Tracking	41.93	129.60
QG-DSGdm	20.97	128.56
QG-GUTm	20.97	130.64
ResNet-20	Torus	32	Gradient Tracking	83.86	129.60
QG-DSGdm	41.93	128.56
QG-GUTm	41.93	130.64
ResNet-20	Dyck	32	Gradient Tracking	62.90	129.60
QG-DSGdm	31.45	128.56
QG-GUTm	31.45	130.64
VGG-11	Ring	16	Gradient Tracking	178.02	31.84
QG-DSGdm	89.01	29.63
QG-GUTm	89.01	34.05
VGG-11	Ring	32	Gradient Tracking	89.22	31.84
QG-DSGdm	44.61	29.63
QG-GUTm	44.61	34.05
Table 13: Communication cost and memory requirement incurred per agent during training of various dataset and model architectures over 16 agents ring topology.
Dataset	Model	Method	Comm. Cost (GB)	Memory (MB)
Fashion MNIST	LeNet-5	Gradient Tracking	11.42	4.86
OG-DSGdm	5.71	4.62
OG-GUTm	5.71	5.10
CIFAR-100	ResNet-20	Gradient Tracking	85.46	129.72
OG-DSGdm	42.73	128.66
OG-GUTm	42.73	130.78
Imagenette	MobileNet-V2	Gradient Tracking	68.86	3774
OG-DSGdm	34.43	3765
OG-GUTm	34.43	3782
Table 14: Communication, memory, and compute overhead incurred per agent during training of various datasets and model architectures for the proposed GUT algorithm. Note that the overheads are independent of the graph topology and graph size.
Dataset	Model	Communication Overhead	Memory Overhead	Compute Overhead
Fashion MNIST	LeNet-5	0.00	0.099	0.275
CIFAR-10	ResNet-20	0.00	0.016	0.021
CIFAR-10	VGG-11	0.00	0.138	0.149
CIFAR-100	ResNet-20	0.00	0.016	0.022
Imagenette	MobileNet-V2	0.00	0.005	0.021