SlowGuess/NeurIPS_2023_OCR huggingface dataset:
G.4.6 Synthetic 2 Benchmark with  $\lambda_{2} = 10.0$
# of samples	method	upper bound	support size	time(s)	gap(%)
3000	SubsetSelectCIO	-4.92601 · 10^4	10	3915.82	53.90
3000	SubsetSelectCIO + warm start	-3.58868 · 10^4	10	3913.17	111.25
3000	L0BnB	-6.33576 · 10^4	10	1198.54	0.00
3000	MSK persp.	-6.33576 · 10^4	10	3601.44	8.58
3000	MSK persp. + warm start	-6.33576 · 10^4	10	3600.20	8.58
3000	Convex MSK opt. persp.	-6.33576 · 10^4	10	1334.48	9.04
3000	ours	-6.33576 · 10^4	10	3601.41	4.54
4000	SubsetSelectCIO	-7.36488 · 10^4	10	4165.15	41.79
4000	SubsetSelectCIO + warm start	-5.42345 · 10^4	10	3789.71	92.55
4000	L0BnB	-8.63287 · 10^4	10	1579.27	0.00
4000	MSK persp.	-8.63287 · 10^4	10	3600.31	7.64
4000	MSK persp. + warm start	-8.63287 · 10^4	10	3601.55	7.64
4000	Convex MSK opt. persp.	-8.63287 · 10^4	10	1478.13	2.24
4000	ours	-8.63287 · 10^4	10	20.33	0.00
5000	SubsetSelectCIO	-	-	3600.00	-
5000	SubsetSelectCIO + warm start	-6.16010 · 10^4	10	3623.84	112.94
5000	L0BnB	-1.09474 · 10^5	10	1296.29	0.00
5000	MSK persp.	-1.09474 · 10^5	10	3601.04	6.75
5000	MSK persp. + warm start	-1.09474 · 10^5	10	3602.25	6.75
5000	Convex MSK opt. persp.	-1.09474 · 10^5	10	1532.63	1.00
5000	ours	-1.09474 · 10^5	10	20.71	0.00
6000	SubsetSelectCIO	-9.91460 · 10^4	10	3945.11	54.60
6000	SubsetSelectCIO + warm start	-8.28165 · 10^4	10	3702.54	85.08
6000	L0BnB	-1.27542 · 10^5	10	1268.61	0.00
6000	MSK persp.	-1.27542 · 10^5	10	3602.43	6.47
6000	MSK persp. + warm start	-1.27542 · 10^5	10	3607.80	6.47
6000	Convex MSK opt. persp.	-1.27542 · 10^5	10	1314.11	0.61
6000	ours	-1.27542 · 10^5	10	1.69	0.00
7000	SubsetSelectCIO	-1.17343 · 10^5	10	3661.02	53.46
7000	SubsetSelectCIO + warm start	-9.54749 · 10^4	10	3758.42	88.62
7000	L0BnB	-1.50512 · 10^5	10	1063.19	0.00
7000	MSK persp.	-1.50512 · 10^5	10	3600.60	5.68
7000	MSK persp. + warm start	-1.50512 · 10^5	10	3600.79	5.68
7000	Convex MSK opt. persp.	-1.50512 · 10^5	10	1480.64	0.45
7000	ours	-1.50512 · 10^5	10	1.70	0.00
Table 146: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0.1, λ2 = 10.0).
# of samples	method	upper bound	support size	time(s)	gap(%)
3000	SubsetSelectCIO	-1.39974 · 105	10	3764.95	36.22
3000	SubsetSelectCIO + warm start	-1.44268 · 105	10	3937.05	32.16
3000	L0BnB	-1.59409 · 105	10	1563.82	0.00
3000	MSK persp.	-1.59409 · 105	10	3601.01	9.02
3000	MSK persp. + warm start	-1.59409 · 105	10	3600.73	9.02
3000	Convex MSK opt. persp.	-1.59409 · 105	10	1511.63	9.58
3000	ours	-1.59409 · 105	10	3600.62	4.68
4000	SubsetSelectCIO	-1.82403 · 105	10	3763.38	43.97
4000	SubsetSelectCIO + warm start	-2.08718 · 105	10	4023.55	25.82
4000	L0BnB	-2.17472 · 105	10	3126.91	0.00
4000	MSK persp.	-2.17472 · 105	10	3605.75	7.91
4000	MSK persp. + warm start	-2.17472 · 105	10	3601.48	7.91
4000	Convex MSK opt. persp.	-2.17472 · 105	10	1478.15	2.24
4000	ours	-2.17472 · 105	10	86.90	0.00
5000	SubsetSelectCIO	-2.60617 · 105	10	3976.98	25.08
5000	SubsetSelectCIO + warm start	-2.19315 · 105	10	3615.40	48.64
5000	L0BnB	-2.71885 · 105	10	1571.26	0.00
5000	MSK persp.	-2.71885 · 105	10	3602.81	7.26
5000	MSK persp. + warm start	-2.71885 · 105	10	3603.13	7.26
5000	Convex MSK opt. persp.	-2.71885 · 105	10	1509.28	1.01
5000	ours	-2.71885 · 105	10	20.75	0.00
6000	SubsetSelectCIO	-2.62606 · 105	10	3828.76	44.93
6000	SubsetSelectCIO + warm start	-2.84305 · 105	10	3662.07	33.87
6000	L0BnB	-3.16660 · 105	10	3085.01	0.00
6000	MSK persp.	-3.16660 · 105	10	3601.61	6.73
6000	MSK persp. + warm start	-3.16660 · 105	10	3605.65	6.73
6000	Convex MSK opt. persp.	-3.16660 · 105	10	1471.81	0.61
6000	ours	-3.16660 · 105	10	21.05	0.00
7000	SubsetSelectCIO	-2.96322 · 105	10	3700.24	50.35
7000	SubsetSelectCIO + warm start	-2.97673 · 105	10	3747.76	49.67
7000	L0BnB	-3.72509 · 105	10	3125.79	0.00
7000	MSK persp.	-3.72509 · 105	10	3605.45	5.87
7000	MSK persp. + warm start	-3.72509 · 105	10	3601.55	5.87
7000	Convex MSK opt. persp.	-3.72509 · 105	10	1549.05	0.45
7000	ours	-3.72509 · 105	10	1.71	0.00
Table 147: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0.3, λ2 = 10.0).
# of samples	method	upper bound	support size	time(s)	gap(%)
3000	SubsetSelectCIO	-3.12174 · 105	10	3636.16	27.78
3000	SubsetSelectCIO + warm start	-3.27648 · 105	10	3759.74	21.75
3000	L0BnB	-3.27321 · 105	8	7307.65	4.14
3000	MSK persp.	-3.33503 · 105	10	3600.22	9.25
3000	MSK persp. + warm start	-3.33503 · 105	10	3600.89	9.25
3000	Convex MSK opt. persp.	-3.33503 · 105	10	1475.73	9.93
3000	ours	-3.33503 · 105	10	3600.25	6.21
4000	SubsetSelectCIO	-3.96358 · 105	10	3655.16	37.96
4000	SubsetSelectCIO + warm start	-4.44111 · 105	10	3679.12	23.13
4000	L0BnB	-4.53159 · 105	10	4152.36	32.87
4000	MSK persp.	-4.53159 · 105	10	3602.35	8.06
4000	MSK persp. + warm start	-4.53159 · 105	10	3600.09	8.06
4000	Convex MSK opt. persp.	-4.53159 · 105	10	1513.22	2.34
4000	ours	-4.53159 · 105	10	3600.95	0.35
5000	SubsetSelectCIO	-4.93471 · 105	10	3709.44	36.82
5000	SubsetSelectCIO + warm start	-5.27817 · 105	10	3730.92	27.92
5000	L0BnB	-5.62906 · 105	10	3521.21	0.00
5000	MSK persp.	-5.62906 · 105	10	3602.11	7.22
5000	MSK persp. + warm start	-5.62906 · 105	10	3603.85	7.44
5000	Convex MSK opt. persp.	-5.62906 · 105	10	1530.90	1.02
5000	ours	-5.62906 · 105	10	31.47	0.00
6000	SubsetSelectCIO	-5.71812 · 105	10	3620.92	38.27
6000	SubsetSelectCIO + warm start	-6.34165 · 105	10	3675.07	24.67
6000	L0BnB	-6.57698 · 105	10	3313.82	0.00
6000	MSK persp.	-6.57698 · 105	10	3605.13	6.87
6000	MSK persp. + warm start	-6.57698 · 105	10	3608.94	6.87
6000	Convex MSK opt. persp.	-6.57698 · 105	10	1642.87	0.62
6000	ours	-6.57698 · 105	10	21.24	0.00
7000	SubsetSelectCIO	-7.32796 · 105	10	3784.82	26.02
7000	SubsetSelectCIO + warm start	-7.45541 · 105	10	3662.49	23.87
7000	L0BnB	-7.72166 · 105	10	3081.93	0.68
7000	MSK persp.	-7.72166 · 105	10	3604.29	5.98
7000	MSK persp. + warm start	-7.72166 · 105	10	3606.92	5.98
7000	Convex MSK opt. persp.	-7.72166 · 105	10	1526.71	0.45
7000	ours	-7.72166 · 105	10	20.37	0.00
Table 148: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0.5, λ2 = 10.0).
# of samples	method	upper bound	support size	time(s)	gap(%)
3000	SubsetSelectCIO	-6.97357 · 10^5	10	3727.41	27.12
3000	SubsetSelectCIO + warm start	-7.35099 · 10^5	10	3678.19	20.59
3000	L0BnB	-5.52435 · 10^5	1	7287.00	33.44
3000	MSK persp.	-7.41086 · 10^5	10	3602.90	9.56
3000	MSK persp. + warm start	-7.41086 · 10^5	10	3600.62	9.56
3000	Convex MSK opt. persp.	-7.41086 · 10^5	10	1525.68	10.22
3000	ours	-7.41086 · 10^5	10	3600.48	8.32
4000	SubsetSelectCIO	-9.65426 · 10^5	10	3756.83	25.29
4000	SubsetSelectCIO + warm start	-9.86629 · 10^5	10	3731.48	22.60
4000	L0BnB	-7.53327 · 10^5	1	7269.26	31.51
4000	MSK persp.	-1.00277 · 10^6	10	3600.55	8.17
4000	MSK persp. + warm start	-1.00277 · 10^6	10	3601.59	8.17
4000	Convex MSK opt. persp.	-1.00277 · 10^6	10	1555.56	2.60
4000	ours	-1.00277 · 10^6	10	3600.62	1.66
5000	SubsetSelectCIO	-1.14948 · 10^6	10	3626.73	29.45
5000	SubsetSelectCIO + warm start	-1.23402 · 10^6	10	3607.12	20.58
5000	L0BnB	-1.23544 · 10^6	9	7434.60	4.95
5000	MSK persp.	-1.24017 · 10^6	10	3602.18	7.35
5000	MSK persp. + warm start	-1.24017 · 10^6	10	3604.42	7.35
5000	Convex MSK opt. persp.	-1.24017 · 10^6	10	1606.64	1.03
5000	ours	-1.24017 · 10^6	10	386.72	0.00
6000	SubsetSelectCIO	-1.36967 · 10^6	10	3743.26	27.64
6000	SubsetSelectCIO + warm start	-1.43335 · 10^6	10	3802.62	21.97
6000	L0BnB	-1.44849 · 10^6	9	7430.07	4.18
6000	MSK persp.	-1.45406 · 10^6	10	3608.33	6.98
6000	MSK persp. + warm start	-1.45406 · 10^6	10	3604.65	6.98
6000	Convex MSK opt. persp.	-1.45406 · 10^6	10	1665.16	0.62
6000	ours	-1.45406 · 10^6	10	54.13	0.00
7000	SubsetSelectCIO	-1.60283 · 10^6	10	3661.63	27.20
7000	SubsetSelectCIO + warm start	-1.68426 · 10^6	10	3897.62	21.05
7000	L0BnB	-1.69016 · 10^6	8	7280.69	3.21
7000	MSK persp.	-1.70473 · 10^6	10	3603.26	6.07
7000	MSK persp. + warm start	-1.70473 · 10^6	10	3603.98	6.07
7000	Convex MSK opt. persp.	-1.70473 · 10^6	10	1677.45	0.45
7000	ours	-1.70473 · 10^6	10	21.30	0.00
Table 149: Comparison of time and optimality gap on the synthetic datasets (p = 3000, k = 10, ρ = 0.7, λ2 = 10.0).
# of samples	method	upper bound	support size	time(s)	gap(%)
3000	SubsetSelectCIO	-2.76095 · 10^6	10	3644.99	20.66
3000	SubsetSelectCIO + warm start	-2.78489 · 10^6	10	3685.28	19.62
3000	L0BnB	-2.55553 · 10^6	1	7230.61	17.51
3000	MSK persp.	-2.78334 · 10^6	10	3601.82	10.13
3000	MSK persp. + warm start	-2.78334 · 10^6	10	3600.26	10.13
3000	Convex MSK opt. persp.	-2.77971 · 10^6	10	1638.17	10.70
3000	ours	-2.78507 · 10^6	10	3600.78	9.79
4000	SubsetSelectCIO	-3.71695 · 10^6	10	3601.35	21.65
4000	SubsetSelectCIO + warm start	-3.74577 · 10^6	10	3638.03	20.71
4000	L0BnB	-3.45116 · 10^6	1	7514.37	15.03
4000	MSK persp.	-3.74752 · 10^6	10	3602.22	8.70
4000	MSK persp. + warm start	-3.75192 · 10^6	10	3602.31	8.57
4000	Convex MSK opt. persp.	-3.75191 · 10^6	10	1564.01	3.01
4000	ours	-3.75192 · 10^6	10	3600.32	3.23
5000	SubsetSelectCIO	-4.57617 · 10^6	10	3864.92	21.14
5000	SubsetSelectCIO + warm start	-4.60856 · 10^6	10	3764.48	20.29
5000	L0BnB	-4.23012 · 10^6	1	7233.86	13.08
5000	MSK persp.	-4.61866 · 10^6	10	3603.72	7.81
5000	MSK persp. + warm start	-4.61867 · 10^6	10	3603.77	7.90
5000	Convex MSK opt. persp.	-4.61867 · 10^6	10	1492.40	1.19
5000	ours	-4.61867 · 10^6	10	3600.30	0.89
6000	SubsetSelectCIO	-5.31865 · 10^6	10	3718.93	22.96
6000	SubsetSelectCIO + warm start	-5.42880 · 10^6	10	3662.21	20.47
6000	L0BnB	-4.97660 · 10^6	1	7210.21	12.38
6000	MSK persp.	-5.43816 · 10^6	10	3606.21	7.09
6000	MSK persp. + warm start	-5.43816 · 10^6	10	3602.03	7.09
6000	Convex MSK opt. persp.	-5.43816 · 10^6	10	1527.55	0.64
6000	ours	-5.43816 · 10^6	10	3600.22	0.32
7000	SubsetSelectCIO	-6.30345 · 10^6	10	3670.08	20.83
7000	SubsetSelectCIO + warm start	-6.34815 · 10^6	10	3714.08	19.98
7000	L0BnB	-5.82238 · 10^6	1	7236.55	11.40
7000	MSK persp.	-	-	3604.74	-
7000	MSK persp. + warm start	-6.36757 · 10^6	10	3607.94	6.15
7000	Convex MSK opt. persp.	-6.36757 · 10^6	10	1747.23	0.45
7000	ours	-6.36757 · 10^6	10	3600.06	0.10
Table 150: Comparison of time and optimality gap on the synthetic datasets  $(\mathrm{p} = 3000, \mathrm{k} = 10, \rho = 0.9, \lambda_2 = 10.0)$ .
G.5 Experiments on the Case  $p >> n$
# of samples	method	upper bound	support size	time(s)	gap(%)
100	SOS1 + warm start	-2.76902 · 103	10	3600.94	13.02
100	big-M50 + warm start	-2.76902 · 103	10	3600.58	13.02
100	big-M20 + warm start	-2.76902 · 103	10	3600.47	13.02
100	big-M5 + warm start	-2.76902 · 103	10	3600.17	13.02
100	persp. + warm start	-2.76902 · 103	10	3602.22	13.02
100	eig. persp. + warm start	-2.76902 · 103	10	3600.65	13.02
100	Convex opt. persp. + warm start	-2.51480 · 103	10	1706.56	24.44
100	MSK persp. + warm start	-2.71927 · 103	10	3599.92	15.08
100	SubsetSelectCIO + warm start	-1.68157 · 103	10	4350.20	86.11
100	LOBnB	-7.78758 · 102	1	7217.66	255.64
100	ours	-2.76902 · 103	10	3600.22	13.02
200	SOS1 + warm start	-4.15945 · 103	10	3600.08	17.20
200	big-M50 + warm start	-4.15945 · 103	10	3600.05	17.20
200	big-M20 + warm start	-4.15945 · 103	10	3600.11	17.20
200	big-M5 + warm start	-4.15945 · 103	10	3600.22	17.20
200	persp. + warm start	-4.15945 · 103	10	3600.20	17.20
200	eig. persp. + warm start	-4.15945 · 103	10	3600.24	17.20
200	Convex opt. persp. + warm start	-4.11140 · 103	10	1800.00	18.57
200	MSK persp. + warm start	-4.15945 · 103	10	3599.97	17.20
200	SubsetSelectCIO + warm start	-2.74363 · 103	10	3819.10	77.69
200	LOBnB	-1.20299 · 103	1	7208.65	240.10
200	ours	-4.15945 · 103	10	3601.81	17.20
500	SOS1 + warm start	-1.00615 · 104	10	3600.51	20.46
500	big-M50 + warm start	-1.00615 · 104	10	3600.28	20.46
500	big-M20 + warm start	-1.00615 · 104	10	3600.07	20.46
500	big-M5 + warm start	-1.00615 · 104	10	3600.11	20.46
500	persp. + warm start	-1.00615 · 104	10	3600.37	20.46
500	eig. persp. + warm start	-1.00615 · 104	10	3600.10	20.46
500	Convex opt. persp. + warm start	-1.00615 · 104	10	1744.05	20.46
500	MSK persp. + warm start	-1.00615 · 104	10	3600.21	20.46
500	SubsetSelectCIO + warm start	-	-	2.29	-
500	LOBnB	-1.00615 · 104	10	1834.75	0.00
500	ours	-1.00615 · 104	10	3600.23	20.46
Table 151: Comparison of time and optimality gap on the synthetic datasets  $(\mathrm{p} = 3000, \mathrm{k} = 10, \rho = 0.1, \lambda_2 = 0.001)$ . All baselines use our beam search solution as a warm start whenever possible. The results in the case of  $p >> n$  show that OKRidge is competitive with other MIP methods. l0bnb can solve the problem when  $n = 500$  but has difficulty in finding the desired sparsity level in the case of  $n = 100$  and  $n = 200$ . MOSEK sometimes gives the best optimality gap. However, as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and l0bnb are not competitive in discovering differential equations (datasets have  $n \geq p$ ), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case  $p >> n$ .
# of samples	method	upper bound	support size	time(s)	gap(%)
100	SOS1 + warm start	-2.76721 · 103	10	3601.59	13.09
100	big-M50 + warm start	-2.76721 · 103	10	3600.66	13.09
100	big-M20 + warm start	-2.76721 · 103	10	3600.32	13.09
100	big-M5 + warm start	-2.76721 · 103	10	3602.22	13.09
100	persp. + warm start	-2.76721 · 103	10	3600.09	13.09
100	eig. persp. + warm start	-2.76721 · 103	10	3600.03	13.09
100	Convex opt. persp. + warm start	-2.51378 · 103	10	2128.32	24.44
100	MSK persp. + warm start	-2.71018 · 103	10	3599.92	15.36
100	SubsetSelectCIO + warm start	-7.59815 · 102	10	3651.10	311.88
100	LOBnB	-7.78035 · 102	1	7218.43	255.36
100	ours	-2.76721 · 103	10	3600.35	13.08
200	SOS1 + warm start	-4.15848 · 103	10	3600.46	17.23
200	big-M50 + warm start	-4.15848 · 103	10	3600.42	17.23
200	big-M20 + warm start	-4.15848 · 103	10	3600.30	17.23
200	big-M5 + warm start	-4.15848 · 103	10	3600.10	17.23
200	persp. + warm start	-4.15848 · 103	10	3600.12	17.23
200	eig. persp. + warm start	-4.15848 · 103	10	3600.47	17.23
200	Convex opt. persp. + warm start	-4.11040 · 103	10	2159.71	18.55
200	MSK persp. + warm start	-4.15848 · 103	10	3599.93	17.11
200	SubsetSelectCIO + warm start	-3.14066 · 103	10	3632.89	55.23
200	LOBnB	-1.20245 · 103	1	7208.71	237.38
200	ours	-4.15848 · 103	10	3600.13	17.22
500	SOS1 + warm start	-1.00605 · 104	10	3600.08	20.47
500	big-M50 + warm start	-1.00605 · 104	10	3600.08	20.47
500	big-M20 + warm start	-1.00605 · 104	10	3600.80	20.47
500	big-M5 + warm start	-1.00605 · 104	10	3600.52	20.47
500	persp. + warm start	-1.00605 · 104	10	3600.04	20.47
500	eig. persp. + warm start	-1.00605 · 104	10	3600.06	20.47
500	Convex opt. persp. + warm start	-1.00605 · 104	10	2059.28	20.41
500	MSK persp. + warm start	-1.00605 · 104	10	3600.08	20.33
500	SubsetSelectCIO + warm start	-7.47048 · 103	10	3748.65	62.24
500	LOBnB	-1.00605 · 104	10	1835.06	0.00
500	ours	-1.00605 · 104	10	3601.58	20.46
Table 152: Comparison of time and optimality gap on the synthetic datasets  $(\mathrm{p} = 3000, \mathrm{k} = 10, \rho = 0.1, \lambda_2 = 0.1)$ . All baselines use our beam search solution as a warm start whenever possible. The results in the case of  $p >> n$  show that OKRidge is competitive with other MIP methods. l0bnb can solve the problem when  $n = 500$  but has difficulty in finding the desired sparsity level in the case of  $n = 100$  and  $n = 200$ . MOSEK sometimes gives the best optimality gap. However, as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and l0bnb are not competitive in discovering differential equations (datasets have  $n \geq p$ ), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case  $p >> n$ .
# of samples	method	upper bound	support size	time(s)	gap(%)
100	SOS1 + warm start	-2.61800 · 103	10	3600.36	19.31
100	big-M50 + warm start	-2.61800 · 103	10	3600.18	19.31
100	big-M20 + warm start	-2.61800 · 103	10	3600.40	19.31
100	big-M5 + warm start	-2.61800 · 103	10	3602.50	19.31
100	persp. + warm start	-2.61800 · 103	10	3600.47	19.54
100	eig. persp. + warm start	-2.61800 · 103	10	3600.19	19.54
100	Convex opt. persp. + warm start	-2.40846 · 103	10	1968.34	25.24
100	MSK persp. + warm start	-2.61800 · 103	10	3599.93	10.09
100	SubsetSelectCIO + warm start	-1.79729 · 103	10	3646.28	74.12
100	LOBnB	-7.11936 · 102	1	7212.43	236.77
100	ours	-2.61806 · 103	10	3600.78	19.09
200	SOS1 + warm start	-4.06441 · 103	10	3600.07	19.69
200	big-M50 + warm start	-4.06441 · 103	10	3600.92	19.69
200	big-M20 + warm start	-4.06441 · 103	10	3600.61	19.69
200	big-M5 + warm start	-4.06441 · 103	10	3600.07	19.69
200	persp. + warm start	-4.06441 · 103	10	3600.03	11.80
200	eig. persp. + warm start	-4.06441 · 103	10	3600.79	11.80
200	Convex opt. persp. + warm start	-4.01212 · 103	10	2175.61	16.91
200	MSK persp. + warm start	-4.06441 · 103	10	3599.92	9.05
200	SubsetSelectCIO + warm start	-2.68289 · 103	10	3704.01	81.71
200	LOBnB	-1.15079 · 103	1	7207.25	219.55
200	ours	-4.06441 · 103	10	3600.92	19.45
500	SOS1 + warm start	-9.96316 · 103	10	3601.58	21.39
500	big-M50 + warm start	-9.96316 · 103	10	3602.06	21.39
500	big-M20 + warm start	-9.96316 · 103	10	3602.58	21.39
500	big-M5 + warm start	-9.96316 · 103	10	3600.48	21.28
500	persp. + warm start	-9.96316 · 103	10	3600.14	21.65
500	eig. persp. + warm start	-9.96316 · 103	10	3600.12	13.16
500	Convex opt. persp. + warm start	-9.96316 · 103	10	2145.98	16.65
500	MSK persp. + warm start	-9.96316 · 103	10	3600.03	11.45
500	SubsetSelectCIO + warm start	-7.48575 · 103	10	3793.20	61.91
500	LOBnB	-9.96316 · 103	10	1483.07	0.00
500	ours	-9.96316 · 103	10	3600.96	21.14
Table 153: Comparison of time and optimality gap on the synthetic datasets  $(\mathrm{p} = 3000, \mathrm{k} = 10, \rho = 0.1, \lambda_2 = 10.0)$ . All baselines use our beam search solution as a warm start whenever possible. The results in the case of  $p >> n$  show that OKRidge is competitive with other MIP methods. When the number of samples is 100, OKRidge is able to find a slightly better solution through the branch-and-bound process, but other solvers are stuck with the warm start solution. l0bnb can solve the problem when  $n = 500$  but has difficulty in finding the desired sparsity level in the case of  $n = 100$  and  $n = 200$ . MOSEK sometimes gives the best optimality gap. However, as we have shown in the previous subsections, MOSEK is not scalable to high dimensions. Additionally, the results in differential equations show that MOSEK and l0bnb are not competitive in discovering differential equations (datasets have  $n \geq p$ ), which is the main focus of this paper. In the future, we plan to extend OKRidge to develop a more scalable algorithm for the case  $p >> n$ .
G.6 Comparison between Fast Solve and ADMM
In this subsection, we provide the empirical evidence that the ADMM method is more effective than the fast solve method in terms of the computing a tighter lower bound, which helps speed up the certification process.
Figure 7: Comparison of running time (top row) and optimality gap (bottom row) between using fast solve (Equation (16)) alone and using fast solve and ADMM (Equations (19)-(22)) to calculate the lower bounds in the BnB tree with three correlation levels  $\rho = 0.1, 0.5, 0.9$  ( $n = 10000$ ,  $k = 10$ ).
Figure 8: Comparison of running time (top row) and optimality gap (bottom row) between using fast solve (Equation (16)) alone and using fast solve and ADMM (Equations (19)-(22)) to calculate the lower bounds in the BnB tree with three correlation levels  $\rho = 0.1, 0.5, 0.9$  ( $p = 3000$ ,  $k = 10$ ).
H More Experimental Results on Dynamical Systems
H.1 Results Summarization
Comparison with MIOSR We first show a direct comparison between OKRidge and MIOSR (based on SOS1 formulation). The results are in Appendix H.2, where we show OKRidge outperforms MIOSR in terms of both solution quality and running time.
Comparison with Different MIP Formualtions Next, we compare OKRidge with other MIP formulations (SOS1, big-M, perspective, and eigen-perspective) as done on the synthetic benchmarks. These formulations have never been applied to the differential equation experiments. Similar to the synthetic benchmarks, we show that OKRidge obtain better solution quality and smaller running time. The results are shown in Appendix H.3.
Comparison between Beam Search and More Heuristics In addition, we also compare our beam search method with many more heuristic methods from the sparse regression literature. Beam search outperforms all methods in terms of solution qualities in presence of highly correlated features. Block Decomposition algorithm also achieves high-quality solutions, but is much slower than beam search. The results are shown in Appendix H.4
Comparison with Different MIP Formulations Warm Started by Beam Search Due to the superiority of beam search, we give beam search solutions as warm starts to Gurobi with different MIP formulations and repeat the experiments. Our warm start improves the solution quality and running time, but OKRidge still outperforms other MIP formulations in terms of running time. The results are shown in Appendix H.5.
Comparison with Other MIP Solvers and Formulations Lastly, we solve the same problem using the MOSEK solver, 10bnb, and SubsetSelectionCIO. For this application, 10bnb has a time limit of 10 seconds for each  $\ell_0$  regularization and 60 seconds for the total time limit. All other algorithms have a total time limit of 30 seconds as mentioned in the experimental setup subsection. Although they are not as competitive as Gurobi as we have shown on the synthetic benchmarks, we still conduct the experiments for the purpose of thorough comparison. Results in Appendix H.6 show that OKRidge outperforms these MIP solvers/formulations.
H.2 Direct Comparison of OKRidge with MIOSR
In the appendix, we directly compare MIOSR (based on the SOS1 formulation) and our method on the differential equation experiments. The results are shown in Figure 9.
Although MIOSR is faster than OKRidge on the Lorenz system, OKRidge is able to find better solutions. See Appendix H.3 and Figure 10 where the eigen-perspective formulation doesn't have this problem and can produce high-quality solutions.
On the Hopf system, MIOSR is faster than OKRidge. However, the number of features is small  $(p = 21)$ , and the running time is negligible compared to Lorenz and MHD systems.
On the MHD system  $(p = 462)$ , OKRidge outperforms MIOSR in terms of both solution qualities and running times. This shows that OKRidge is much more scalable to high dimensions than MIOSR.
Figure 9: Comparison between our method and MIOSR on the experiments of discovering sparse differential equations. We obtain better performance on the true positivity rate,  $L_{2}$  coefficient error, and test RMSE.
H.3 Comparison of OKRidge with SOS1, Big-M, Perspective, and Eigen-Perspective Formulations
Besides the MIOSR baseline (which is based on the SOS1 formulation), we also compare with other MIP formulations, including Big-M ( $M = 50$ ), perspective, and eigen-perspective formulations. The results are shown in Figure 10. The big-M method produces errors on the Lorenz system, so its result is not shown on the left column. We have discussed about the SOS1 formulation in the last subsection. On the Lorenz system, OKRidge is faster than the perspective and eigen-perspective formulations. On the Hopf system ( $p = 21$ ), OKRidge is slower than Gurobi, but OKRidge takes only 20 seconds. On the high-dimensional MHD system, OKRidge is significantly faster than all MIP formulations.
Figure 10: Comparison between our method and other MIP formulations solved by Gurobi on the experiments of discovering sparse differential equations. We obtain better performance on the true positivity rate,  $L_{2}$  coefficient error, and test RMSE. SOS1 is faster than ours on the Lorenz system, but the solution quality is worse. OKRidge is slower than Gurobi on the Hopf system (p=21), but the time difference is not significant. On the high-dimensional MHD system, OKRidge is significantly faster than Gurobi with all existing MIP formulations.
H.4 Comparison of Our Beam Search with More Heuristic Baselines