SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our model is a graph neural network that closely resembles the model proposed in Gasse et al. [17] and uses some of the modifications from Paulus et al. [35]. The model is sketched in Figure 2. After separately applying batch normalization to both variable and constraint nodes, we embed all nodes in a 64-dimensional space using a multi-layer perceptron with a single hidden layer. This is followed by two bipartite graph convolutions, first from variables to constraints and then from constraints to variables, as in Gasse et al. [17]. Finally, we predict from the convolved variable embedding the parameters  $\theta$  of a probability distribution  $q_{\theta}(x_j|M^{\dagger})$  for each divisible integer variable using another multi-layer perceptron with a single hidden layer. Binary variables are the most common variables in integer programs, and all the instances we considered feature exclusively binary divisible variables. For binary variables,  $q_{\theta}(x_j|M^{\dagger})$  is a Bernoulli distribution and the model outputs the mean parameter  $\theta$ , such that  $\mathbb{P}(x_j = 1) = \theta$ . For general integer variables, we suggest to consider the bitwise representation of the integer domain. For example, the domain of a variable that can assume no more than eight unique integer values can be represented using at most four bits. Each of these bits can be parameterized with its own Bernoulli distribution and the model outputs a mean for each. This approach may be more favorable for diving than for other applications, because slack variables from cutting plane constraints (whose domains are not known initially and may be large) are not divisible. In cases where outputting a fixed-size array of Bernoulli parameters may not be applicable, a variable length array could be outputted by using a recurrent layer, such as an LSTM [21], as is proposed in Nair et al. [33]. Alternatively, a fixed size array could still be used with an additional bit to indicate integer overflow that must be handled appropriately. For example in diving, variables for which the model predicts overflow may be ignored. Further, the frequency with which overflow would be encountered in practice could plausibly be reduced by making predictions relative to the current solution  $\lfloor x_j^*\rfloor$  rather than with respect to the lower variable bound  $\pi_j$ .
Figure 3: Our model is a graph neural network based on Gasse et al. [17] and Paulus et al. [35]. It first embeds variable and constraint nodes using batch normalization and a single-layer feedforward network. It then convolves variable and constraint features with two bi-partite graph convolutions as in Gasse et al. [17]. Finally, for each variable it outputs the parameters of the distribution  $q_{\theta}(x_j)$  using another single-layer feedforward network.
C Solution Augmentation by Counting Optimal Solutions
Many integer and mixed integer linear programs exhibit strong symmetries, particularly those from combinatorial optimization. In these cases multiple optimal solutions may exist, and in particular different integer variable assignments that correspond to an optimal solution. It is possible to identify those by first solving for  $z^{\dagger}$  to then define the auxiliary mixed integer linear program,
$$
\min  _ {x \in P ^ {\mathcal {X}}} c ^ {\intercal} x, \quad P ^ {\mathcal {X}} = \left\{x \in P ^ {\dagger}, c ^ {\intercal} x = z ^ {\dagger} \right\} \tag {9}
$$
Standard solvers, including SCIP 7.0.2, can enumerate the set of feasible solutions  $P^{\mathcal{X}}$  of (9) by adding a constraint handler whenever a new feasible integer variable assignment is found and continuing the solving process. We use this solution augmentation for our experiments on set covering and independent sets where the solver identified multiple optimal solutions in a short period of time.
D Background: Linear Programming
In this section, we briefly review some concepts from linear programming and duality.
Definition 1 (Standard Form). A linear program  $M^{*}$  of the form
$$
\min  c ^ {T} x \quad \text {s u b j e c t} A x = b, \underline {{\pi}} \leq x \leq \bar {\pi} \tag {10}
$$
is said to be in standard form with variable bounds. A basis for this program is a triplet  $(\mathcal{L},\mathcal{B},\mathcal{U})$  where  $x_{\mathcal{B}}$  are the basic variables and  $x_{\mathcal{L}} = \underline{\pi}_{\mathcal{L}}$  and  $x_{\mathcal{U}} = \overline{\pi}_{\mathcal{U}}$ .
Definition 2 (Dual linear program). The dual linear program  $M_D^*$  of the program given in 10 is
$$
\max  _ {y _ {b}, y _ {\underline {{\pi}}}, y _ {\pi}} y _ {b} ^ {\intercal} b + y _ {\underline {{\pi}}} ^ {\intercal} \underline {{\pi}} + y _ {\overline {{\pi}}} ^ {\intercal} \overline {{\pi}} \quad \text {s u b j e c t} A ^ {\intercal} y _ {b} + y _ {\underline {{\pi}}} + y _ {\overline {{\pi}}} = c, y _ {\underline {{\pi}}} \geq 0, y _ {\overline {{\pi}}} \leq 0, \tag {11}
$$
Theorem 1 (Complementary Slackness). Let  $x^{*}$  be a feasible solution to the linear program in (10) and let  $y^{*} := (y_{b}^{*}, y_{\underline{\pi}}^{*}, y_{\overline{\pi}}^{*})$  be a feasible solution to its corresponding dual linear program given in (11). Then,  $x^{*}$  and  $y^{*}$  are optimal solutions for the two respective problems if and only if
$$
\left(x ^ {*} - \underline {{\pi}}\right) \odot y _ {\underline {{\pi}}} ^ {*} = 0 \tag {12}
$$
$$
\left(x ^ {*} - \bar {\pi}\right) \odot y _ {\bar {\pi}} ^ {*} = 0 \tag {13}
$$
where  $\odot$  denotes the Hadamard product. The additional conditions  $y_{b}^{*}\odot (Ax - b) = 0$  and  $(c - A^{\intercal}y_{b} - y_{\overline{\pi}} - y_{\overline{\pi}})\odot x = 0$  are satisfied because both  $x^{*}$  and  $y^{*}$  are feasible.
Proof. The conditions are derived from the definition of the dual program and strong duality in linear programming [see e.g., 10].  $\square$
E.1 Generalization to larger instances
For capacitated facility location, we evaluated the generalization performance of L2Dive. For our experiments in subsection 5.1, we trained and tested on the small instances from Gasse et al. [17] for each problem class. In this additional experiment, we tested the L2Dive model trained on the small capacitated facility location instances and evaluated it along with all standard divers on 100 large capacitated facility location test instances from [17]. We find that L2Dive gracefully generalizes to larger instances of the same problem class (Table 5). It achieves the lowest relative primal gap and outperforms all standard divers on the larger test instances. Overall, the relative primal gap tends to be larger which is expected as the larger instances tend to be more difficult to solve.
E.2 Relative primal gap
In Table 6 we report the relative primal gap for the experiments in section 5.1. The relative primal gap is computed as
$$
\gamma_ {p} ^ {\prime} (\tilde {z}) = \frac {\gamma_ {p} (\tilde {z})}{| z ^ {\dagger} |} \tag {14}
$$
where  $\gamma_p(\tilde{z})$  is as defined in equation (5). It is ill-defined, if the objective value is zero, but this was not the case for any instances considered in our experiments. As the instances within each of the four problem classes tend to have objective values of the same magnitude, the conclusions that can be drawn from the relative primal gap do not differ from those we can draw from the absolute primal gap in Table 1. We included the results here for completeness.
Table 5: L2Dive outperforms standard divers on large test instances from capacitated facility location, even when only trained on small instances. Average primal gap with standard error on test set.
small	large
L2Dive	0.89 (0.05)	1.43 (0.11)
Best heuristic	2.70 (0.04)	2.25 (0.10)
Coefficient	4.25 (0.05)	5.35 (0.17)
Distributional	4.24 (0.05)	5.05 (0.17)
Farkas	2.70 (0.04)	3.16 (0.11)
Fractional	5.91 (0.06)	9.38 (0.20)
Linesearch	5.78 (0.09)	4.56 (0.19)
Pseudocost	2.82 (0.06)	2.25 (0.10)
Vectorlength	4.68 (0.10)	3.09 (0.19)
Random	5.03 (0.08)	4.10 (0.16)
Lower	3.48 (0.05)	4.01 (0.13)
Upper	4.62 (0.08)	3.17 (0.15)
Table 6: We report mean and standard error of the relative primal gap in equation (14) for the experiments in subsection 5.1. The conclusions are the same as those drawn from Table 1.
SET COVER	COMB. AUCTION	FAC. LOCATION	IND. SET
L2Dive	2.86 (0.14)	2.84 (0.09)	0.89 (0.05)	0.23 (0.02)
Best heuristic	5.06 (0.16)	3.27 (0.1)	2.70 (0.04)	0.82 (0.05)
Coefficient	198.02 (2.23)	8.56 (0.14)	4.25 (0.05)	10.91 (0.17)
Distributional	209.29 (1.79)	19.18 (0.15)	4.24 (0.05)	8.70 (0.11)
Farkas	5.57 (0.17)	6.08 (0.12)	2.70 (0.04)	-
Fractional	198.82 (2.26)	8.57 (0.13)	5.91 (0.06)	10.27 (0.16)
Linesearch	69.05 (1.29)	5.96 (0.12)	5.78 (0.09)	3.42 (0.05)
Pseudocost	10.44 (0.43)	3.27 (0.1)	2.82 (0.06)	1.44 (0.08)
Vectorlength	5.06 (0.16)	10.65 (0.26)	4.68 (0.10)	0.82 (0.05)
Random	22.07 (0.6)	8.99 (0.16)	5.03 (0.08)	3.46 (0.11)
Lower	153.59 (2.81)	20.23 (0.14)	3.48 (0.05)	7.59 (0.20)
Upper	12.69 (0.28)	7.81 (0.14)	4.62 (0.08)	2.78 (0.09)
E.3 Execution Times
For completeness, we also report the execution times of all divers considered in the experiments in subsection 5.1 in Table 7. On the instances considered, diving tends to be rapid and differences in execution times are largely negligible (in contrast to differences in the quality of solutions found). Extremely short execution times typically result from diving being aborted early, e.g., because the LP was rendered infeasible.
F Tuning the SCIP solvers for diving
We leveraged expert knowledge and used random search to optimize the use of diving heuristics in SCIP 7.0.2 for our baseline Tuned. The most important parameters to control the standard divers in SCIP are freq and freqofs. For each diving heuristic, these parameters control the depth at which the heuristic may be called or not called. By varying these parameters, diverse diving ensembles can be realized that call different heuristics at different stages of the branch and bound search. We randomly sample solver configurations by setting either freq = -1 (no diving), or freq = [0.5 × freq_default] (double frequency) or freq = freq_default (leave frequency at default) or freq = [2 × freq_default] (halve frequency) with equal probability and setting either freqofs = 0 or freqofs = freqofs_default with equal probability independently for each diving heuristic. We run the solver with the usual time limits for each configuration and each validation instance and pick the configuration with the lowest primal-dual
Table 7: We report mean and standard deviation of the execution time for the experiments in subsection 5.1. Diving on these instances is very fast and differences are mostly negligible. Extremely short execution times tend to indicate that diving was aborted early, e.g., because the LP was rendered infeasible.
SET COVER	IND. SET	COMB. AUCTION	FAC. LOCATION
L2Dive	0.45 (0.05)	0.58 (0.15)	0.38 (0.08)	3.99 (0.76)
Coefficient	0.28 (0.13)	0.04 (0.01)	0.02 (0.01)	3.40 (0.61)
Distributional	0.25 (0.11)	0.04 (0.01)	0.08 (0.02)	3.41 (0.63)
Farkas	0.05 (0.03)	-	0.02 (0.01)	3.66 (0.68)
Fractional	0.29 (0.14)	0.04 (0.01)	0.02 (0.01)	3.83 (0.60)
Linesearch	0.10 (0.04)	0.03 (0.01)	0.03 (0.01)	3.53 (0.79)
Pseudocost	0.02 (0.01)	0.04 (0.02)	0.02 (0.00)	2.77 (0.76)
Vectorlength	0.02 (0.01)	0.02 (0.01)	0.05 (0.01)	1.82 (0.59)
Random	0.08 (0.04)	0.25 (0.10)	0.05 (0.01)	4.42 (1.50)
Lower	0.74 (0.43)	0.66 (0.26)	0.49 (0.07)	4.11 (0.79)
Upper	0.05 (0.02)	0.17 (0.08)	0.04 (0.01)	2.79 (1.02)
integral for server load balancing and with the lowest solving time for neural network verification. For server load balancing, since the original validation dataset was relatively small, we created a new validation dataset of 625 instances from the original training and validation sets. We optimized over 16 random configurations and thus used a budget of 10,000 solver calls for load balancing. For neural network verification, we used the original validation dataset of 505 validation instances. We considered six random configurations and thus used a budget of 3,030 solver calls which is slightly more than what L2Dive used for data collection (2903).
We did not tune any parameters to optimize the use of L2Dive, but this might improve performance.