SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This metric is a good proxy for the work required by multidimensional backtracking. The majority of the computation cost of the "backtracking" part of the algorithm comes from computing the gradient at the next point (to compute the hypergradient). The overhead due to other operations such as the CUT method are minimal compared to gradient computations. Indeed, beyond the gradient computation, multidimensional backtracking only involves a few vector operations (see Figures 10 and 11). Even solving for the best convex combination numerically to obtain tighter ellipsoids (as in the CUT method in Figure 11) is faster than computing gradients. To illustrate this point, we provide the running times for subsets of a backtracking update from multidimensional backtracking in Table 2.
Figures 15-20 give additional results on small linear and logistic regression problems and large logistic regression problems. Multidimensional backtracking has a consistent performance across problems and does not suffer from the extremely bad conditioning of cpusmall or california-housing (linear regression) or australian, breast-cancer, diabetes and heart (logistic regression).
California-housing
Figure 15: Runs on small linear regression datasets with Bias initialization
Power-plant
Concrete
Figure 16: Runs on small linear regression datasets with Gaussian initialization
California-housing
Power-plant
Figure 17: Runs on small logistic regression datasets with Bias initialization
Breast-cancer
Australian
Figure 18: Runs on small logistic regression datasets with Gaussian initialization
Figure 19: Runs on large logistic regression datasets with Bias initialization
Figure 20: Runs on large logistic regression datasets with Gaussian initialization