SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10 reports the evaluation time of a configuration when using the real evaluation (compiling using a CPU and executing on a TPU) and using the model prediction (on a CPU) for graphs in the validation set. The real evaluation takes  $94 - 2400\mathrm{x}$  longer than the model prediction. This confirms that it is significantly cheaper to run a learned cost model to estimate the execution time, than to measure the actual execution time (which would require a long compilation). Also note that the graph feature extraction time can be amortized across multiple configurations of the same graph, so the model prediction will be even faster in practice.
Table 8: Prediction errors (\%) on the Tile:XLA collection of different models, where Early and Late refer to the early-join and late-join options of the configuration features. Each line trains three times (with the best hyperparameter configuration per architecture), and chooses the model with the median performance (according to the top-1 error on the entire validation set).
Model / Training	Top-1 Error %	Top-5 Error %	Top-10 Error %
Loss	Type	Val	Test	Val	Test	Val	Test
ListMLE	EarlySAGE	10.5	9.1	3.0	4.2	1.8	2.8
EarlyGCN	11.5	11.4	3.6	4.7	2.4	3.3
LateSAGE	124.8	71.8	53.8	31.7	30.3	19.2
LateGCN	17.2	17.4	6.7	7.3	4.4	4.9
MLP	84.5	52.9	39.7	28.7	25.6	17.0
MSE	EarlySAGE	19.0	22.0	8.2	10.3	5.6	6.5
EarlyGCN	19.4	21.0	8.7	9.9	6.4	7.1
LateSAGE	117.0	67.4	55.2	29.5	33.4	18.6
LateGCN	87.7	67.0	40.0	29.2	27.3	20.6
MLP	117.5	65.3	69.0	32.8	48.7	22.3
Table 9: Per-program prediction errors (%) on the validation set of the Tile:XLA. Each line trains three times (with the best hyperparameter configuration per architecture), and chooses the model with the median performance (according to the top-1 error on the entire validation set). These models are trained with ListMLE. A top-K error of a program is an average of top-K errors of all kernels in the program. The number of kernels per program is in paranethesis.
Program (number of kernels)	MLP	EarlyGCN	EarlySAGE	LateGCN	LateSAGE
Error at top:	1	5	1	5	1	5	1	5	1	5
bert_pretraining.4x4.fp16 (56)	60	25	7	2	6	1	19	5	118	43
inception_v3_batch_128_train (264)	119	53	8	2	6	1	10	3	170	77
mlperf_bert_batch_24_2x2 (75)	34	14	13	5	15	4	23	10	105	33
resnet50.4x4.fp16 (103)	78	33	16	3	15	4	21	8	97	33
resnet_v1_50_official_batch_128 bf16 (108)	75	49	15	6	13	4	23	11	91	50
tf2_bert_pretrainDynamic_batch_size (60)	37	13	8	2	5	2	18	7	60	19
unet_3d.4x4.bf16 (10)	110	53	30	3	48	6	17	0	124	69
Table 10: Real evaluation time vs model prediction time. The 'Batch Inference' column reports time to perform batch inference on 100 configurations divided by 100, while the 'Inference' column reports time to perform inference on a single configuration. The real evaluation takes 94-2400x longer than the model prediction.
Program	Real Evaluation Time (s)	Model Prediction Time (s)
Compilation	Execution	Feature Extraction	Batch Inference	Inference
bert_pretraining.4x4.fp16	45	2.8	0.2	0.02	0.1
inception_v3_batch_128_train	135	3.6	0.3	0.01	0.09
mlperf_bert_batch_24_2x2	104	22	0.6	0.02	0.1
resnet50.4x4.fp16	126	0.9	0.2	0.006	0.08
resnet_v1_50_official_batch_128 bf16	99	1.7	0.1	0.07	0.08
tf2_bert_pretrainDynamic_batch_size	44	2.8	0.4	0.02	0.1
unet_3d.4x4.cf16	475	1.4	0.1	0.004	0.08