SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we compare our proposed beam search method with more heuristic methods from the sparse regression literature. The results are shown in Figure 11. Since many heuristic method implementations do not consider the ridge regression, we set  $\ell_2$  regularization for all algorithms to be 0 for a fair comparison. Since the features are highly correlated, most of the existing heuristic methods do not produce high-quality solutions except the Block Decomposition algorithm. However, our beam search method is significantly faster than the Block Decomposition algorithm.
Moreover, for the purpose of completeness, we also compare with LASSO, the classical sparse learning method. Other researchers have found that LASSO performs poorly in this context. We confirm this claim experimentally. The result is shown in Figure 12.
Training Trajectory Length (s)
Figure 11: Comparison between our method and other heuristic methods on the experiments of discovering sparse differential equations. We obtain better performance on the true positivity rate,  $L_{2}$  coefficient error, and test RMSE. The Block Decomposition algorithm also produces high-quality solutions but is significantly slower than our beam search.
Figure 12: Results on discovering sparse differential equations. Lasso does not do well in this application. Beam search lags behind OKRidge on the MHD system. This shows that beam search alone is not sufficient, and it is necessary to use BnB to obtain the optimal solution.
H.5 Comparison of OKRidge with MIPs Warmstarted by Our Beam Search
A natural question to ask is whether Gurobi would be faster than OKRidge in terms of running speed if we give the high-quality solutions from last subsection as warm starts to Gurobi. Here, we compare with other formulations warmstarted by our beam search solutions. The results are shown in Figure 13. By comparing Figure 10 with Figure 13, we see that our beam search solutions, if used as warm starts, improve Gurobi's speed. However, OKRidge is still significantly faster than all MIP formulations on the high-dimensional MHD system.
Figure 13: Comparison between our method and other MIP formulations solved by Gurobi, which was warmstarted by our beam search solutions on the experiments of discovering sparse differential equations. OKRidge is slower than Gurobi on the Hopf system  $(p = 21)$ , but the running time is only 20 seconds. On the Lorenz and MHD systems, OKRidge is faster than other MIP formulations, especially on the high-dimensional MHD system  $(p = 462)$ .
H.6 Comparison of OKRidge with MOSEK solver, SubsetSelectionCIO, and L0BNB
Lastly, we compare with other MIP solvers/formulations, including the MOSEK solver, SubsetSelectionCIO, and l0bnb. The results are shown in Figure 14.
MOSEK (perspective formulation) produces good solutions on the Hopf dataset, but could not produce optimal solutions on the Lorenz system. MOSEK (optimal formulation) produces suboptimal solutions on the Lorenz and Hopf systems. However, we are using MOSEK to solve the relaxed convex SDP problem. This demonstrates why we need to tackle the NP hard problem instead of the relaxed convex problem. MOSEK runs out of memory on the MHD dataset, so we don't show the results on the MHD dataset.
SubsetSelectionCIO doesn't do well on the Lorenz and MHD systems. On many instances, SubsetSelectionCIO would have errors.
10bnb performs poorly on these datasets. One major reason is that 10bnb could not retrieve the exact support size  $(k = 1,2,3,4,5)$  because the sparsity is controlled implicitly through  $\ell_0$  regularization. It takes a long time to run these differential experiments. 10bnb couldn't find a good  $\ell_0$  regularization which produces the right sparsity level within the time limit.
Please see Appendix G.1 for a detailed discussion of the drawbacks on using MOSEK, SubsetSelectionCIO, and l0bnb.
Figure 14: Comparison between our method and other MIP solvers or formulations on the experiments of discovering sparse differential equations. We obtain better performance on the true positivity rate,  $L_{2}$  coefficient error, and test RMSE.