SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $a_{\alpha} \in \mathcal{C}^{K}$  is a non-zero sufficiently smooth function of dependent variables. If  $\max_{\alpha \in \mathcal{A}} |\alpha| = n$  then we call  $L$  an  $n^{\text{th}}$ -order linear differential operator. If all  $a_{\alpha}$  are constants we say that  $L$  has constant coefficients.
The adjoint of  $L$ , denoted  $L^{\dagger}$ , is a linear differential operator defined as
$$
L ^ {\dagger} u (\boldsymbol {x}) = \sum_ {\boldsymbol {\alpha} \in \mathcal {A}} (- 1) ^ {| \boldsymbol {\alpha} |} \partial^ {\boldsymbol {\alpha}} \left(a _ {\boldsymbol {\alpha}} (\boldsymbol {x}) u (\boldsymbol {x})\right) \tag {20}
$$
Proposition 1 (Variational Formulation of PDEs for linear PDEs). Let  $K \in \mathbb{N}$ . Consider a scalar field  $u: \Omega \to \mathbb{R}$ , such that  $u \in \mathcal{C}^K$ , a  $K^{th}$ -order linear differential operator  $L$ , and a continuous function  $g: \Omega \to \mathbb{R}$ . Let  $\phi: \Omega \to \mathbb{R}$  be a testing function. Then  $u$  satisfies a linear PDE
$$
L [ u (\boldsymbol {x}) ] - g (\boldsymbol {x}) = 0 \forall \boldsymbol {x} \in \Omega \tag {21}
$$
$$
\int_ {\Omega} \left[ u (\boldsymbol {x}) L ^ {\dagger} \phi (\boldsymbol {x}) - g (\boldsymbol {x}) \phi (\boldsymbol {x}) \right] d \boldsymbol {x} = 0 \tag {22}
$$
Note that the integrals are always well-defined as  $\phi$  has a compact support.
B.2 Theorem
Before we prove the Theorem 1, we need the following lemma, which is a particular formulation of the Fundamental lemma of calculus of variations [18]. We also need a generalized version of the divergence theorem [31].
Lemma (Fundamental lemma of calculus of variations). Let  $K \in \mathbb{N}$ ,  $\Omega$  be an open set in  $\mathbb{R}^M$ , and  $u: \Omega \to \mathbb{R}$  be a continuous function. Then  $u$  is equal to 0 on the whole  $\Omega$  if and only if  $\int_{\Omega} u(\pmb{x}) \phi(\pmb{x}) d\pmb{x} = 0$  for all  $\mathcal{C}^K$  functions  $\phi: \Omega \to \mathbb{R}$  with compact support.
Proof. If  $u$  is identically 0 on  $\Omega$  then all the integrals are trivially equal to 0.
We now prove the converse.
Let us assume for contradiction that there exists a point  $\pmb{x}_0 \in \Omega$  such that  $u(\pmb{x}_0) \neq 0$ . Without loss of generality we assume  $u(\pmb{x}_0) = \epsilon > 0$ . As  $u$  is continuous there exists an open ball around  $\pmb{x}_0$  of radius  $\delta$ , denoted  $B_{\pmb{x}_0}^{\delta} = \{ \pmb{x} \in \Omega \mid ||\pmb{x} - \pmb{x}_0||_2 < \delta \}$ , such that  $\forall \pmb{x} \in B_{\pmb{x}_0}^{\delta} u(\pmb{x}) > \epsilon / 2 > 0$ .
Now let  $\phi$  be a  $\mathcal{C}^K$  function that is positive on  $B_{\pmb{x}_0}^\delta$  and 0 elsewhere. Such a function can always be created by appropriately shifting and scaling  $\phi (\pmb {x}) = e^{-1 / (1 - ||\pmb {x}||_2^2)}\cdot \mathbf{1}_{\{|||\pmb {x}||_2 < 1\}}$ . Its support is a closed ball  $\bar{B}_{\pmb{x}_0}^\delta = \{\pmb {x}\in \Omega \mid ||\pmb {x} - \pmb {x}_0||_2\leq \delta \}$  which is compact. Then
$$
\int_ {\Omega} u (\boldsymbol {x}) \phi (\boldsymbol {x}) d \boldsymbol {x} = \int_ {B _ {\boldsymbol {x} _ {0}} ^ {\delta}} u (\boldsymbol {x}) \phi (\boldsymbol {x}) d \boldsymbol {x} > 0 \tag {23}
$$
as both  $u(\pmb{x})$  and  $\phi(\pmb{x})$  are positive on  $B_{\pmb{x}_0}^{\delta}$ . Thus we found a continuous function  $\phi$  with compact support such that:
$$
\int_ {\Omega} u (\boldsymbol {x}) \phi (\boldsymbol {x}) d \boldsymbol {x} \neq 0 \tag {24}
$$
Therefore  $u$  is identically 0 on  $\Omega$ .
To make this section self-contained, we provide the statement of the generalized divergence theorem [31].
Theorem 2 (Divergence theorem). Let  $\Omega$  be an open set in  $\mathbb{R}^M$  and let  $f, g$  be continuous on  $\bar{\Omega} = \Omega \cup \partial \Omega$  and continuously differentiable on  $\Omega$ . Then
$$
\int_ {\Omega} \partial_ {i} ^ {1} [ f (\boldsymbol {x}) ] g (\boldsymbol {x}) d \boldsymbol {x} = - \int_ {\Omega} f (\boldsymbol {x}) \partial_ {i} ^ {1} [ g (\boldsymbol {x}) ] d \boldsymbol {x} + \int_ {\partial \Omega} \nu_ {i} f (\boldsymbol {x}) g (\boldsymbol {x}) d \boldsymbol {x} \tag {25}
$$
where  $\pmb{\nu}$  is a normal unit vector to the boundary  $\partial \Omega$ .
In a 1-dimensional setting, the statement of the theorem reduces to the integration by parts.
We can now prove Theorem 1.
Proof. Let us denote  $\mathcal{E}_p = (\alpha_p, a_p, h_p)$ . Then the PDE in Equation 9 can be written as:
$$
\sum_ {p = 1} ^ {P} a _ {p} (\boldsymbol {x}) \partial^ {\boldsymbol {\alpha} _ {p}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] - g (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) = 0 \forall \boldsymbol {x} \in \Omega \tag {26}
$$
The LHS is continuous as all  $a_{p}$  and  $h_p$  are smooth,  $g$  is continuous,  $u\in \mathcal{C}^K$ , and  $|\alpha_{p}|\leq K\forall p\in [P]$ . Thus we can use the fundamental lemma of calculus of variations to say that the Equation 26 is true if and only if
$$
\int_ {\Omega} \left[ \sum_ {p = 1} ^ {P} a _ {p} (\boldsymbol {x}) \partial^ {\boldsymbol {\alpha} _ {p}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] - g (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] \phi (x) d \boldsymbol {x} = 0 \tag {27}
$$
for all testing functions  $\phi$ . We transform the LHS of Equation 27 using linearity to:
$$
\sum_ {p = 1} ^ {P} \int_ {\Omega} a _ {p} (\boldsymbol {x}) \partial^ {\boldsymbol {\alpha} _ {p}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] \phi (x) - \int_ {\Omega} g (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \phi (\boldsymbol {x}) d \boldsymbol {x} \tag {28}
$$
Let us now focus on
$$
\int_ {\Omega} a _ {p} (\boldsymbol {x}) \partial^ {\boldsymbol {\alpha} _ {p}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] \phi (x) \tag {29}
$$
and let us denote  $\alpha_{p} = (\alpha_{p1},\dots ,\alpha_{pM})$  . Then  $\partial^{\alpha_p} = \partial_1^{\alpha_p1}\ldots \partial_M^{\alpha_pM}$  and the expression can be written as
$$
\int_ {\Omega} \partial_ {1} ^ {\alpha_ {p 1}} \dots \partial_ {M} ^ {\alpha_ {p M}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] a _ {p} (\boldsymbol {x}) \phi (x) d \boldsymbol {x} \tag {30}
$$
Let us denote the support of  $\phi$  as  $\mathcal{B}$ . As  $\phi$  is equal to zero outside of its support, we can write the expression as
$$
\int_ {\mathcal {B}} \partial_ {1} ^ {\alpha_ {p 1}} \dots \partial_ {M} ^ {\alpha_ {p M}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] a _ {p} (\boldsymbol {x}) \phi (x) d \boldsymbol {x} \tag {31}
$$
Without loss of generality, let us assume that  $\alpha_{p1} > 0$ . By the divergence theorem, this can be rewritten as
$$
- \int_ {\mathcal {B}} \partial_ {1} ^ {\alpha_ {p 1} - 1} \dots \partial_ {M} ^ {\alpha_ {p M}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] \partial_ {1} ^ {1} \left[ a _ {p} (\boldsymbol {x}) \phi (x) \right] d \boldsymbol {x} \tag {32}
$$
because the integral over the boundary is equal to 0
$$
\int_ {\partial \mathcal {B}} \nu_ {1} \partial_ {1} ^ {\alpha_ {p 1} - 1} \dots \partial_ {M} ^ {\alpha_ {p M}} [ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) ] a _ {p} (\boldsymbol {x}) \phi (x) d \boldsymbol {x} = 0 \tag {33}
$$
as  $\phi$  has a compact support (and thus vanishes on the boundary). We can perform this operation  $\alpha_{p1}$  times to shift the whole derivative  $\partial_1^{\alpha_{p1}}$  to the second part of the equation and obtain
$$
(- 1) ^ {\alpha_ {p 1}} \int_ {\mathcal {B}} \partial_ {2} ^ {\alpha_ {p 2}} \dots \partial_ {M} ^ {\alpha_ {p M}} \left[ h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \right] \partial_ {1} ^ {\alpha_ {p 1}} \left[ a _ {p} (\boldsymbol {x}) \phi (x) \right] d \boldsymbol {x} \tag {34}
$$
Then we repeat this for other derivatives and we end up with the following expression:
$$
(- 1) ^ {\alpha_ {p 1}} \cdot \dots \cdot (- 1) ^ {\alpha_ {p M}} \int_ {\mathcal {B}} h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \partial_ {1} ^ {\alpha_ {p 1}} \dots \partial_ {M} ^ {\alpha_ {p M}} [ a _ {p} (\boldsymbol {x}) \phi (x) ] d \boldsymbol {x} \tag {35}
$$
As the integrand is zero outside of  $\mathcal{B}$ , this can be rewritten as:
$$
(- 1) ^ {\left| \boldsymbol {\alpha} _ {p} \right|} \int_ {\Omega} h _ {p} (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \partial^ {\boldsymbol {\alpha} _ {p}} \left[ a _ {p} (\boldsymbol {x}) \phi (x) \right] d \boldsymbol {x} \tag {36}
$$
or more compactly, using the functional defined in Definition 3, as:
$$
\mathcal {F} \left(\mathcal {E} _ {p}, \boldsymbol {u}, \phi\right) \tag {37}
$$
Therefore Equation 28 can be written as:
$$
\sum_ {p = 1} ^ {P} \mathcal {F} \left(\mathcal {E} _ {p}, \boldsymbol {u}, \phi\right) - \int_ {\Omega} [ g (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \phi (\boldsymbol {x}) ] d \boldsymbol {x} \tag {38}
$$
Thus, we proved that Equation 26 is true if and only if
$$
\sum_ {p = 1} ^ {P} \mathcal {F} \left(\mathcal {E} _ {p}, \boldsymbol {u}, \phi\right) - \int_ {\Omega} [ g (\boldsymbol {x}, \boldsymbol {u} (\boldsymbol {x})) \phi (\boldsymbol {x}) ] d \boldsymbol {x} = 0 \tag {39}
$$
for all testing functions  $\phi$ .
Table 5: Examples of equations which are Variational-Ready
Name	Equation	Linear	VR
Damped wave eq. with a source	u tt + ρ u t - κ ∇2u = g(x)	✓	✓
Gauss law	∇ · E = ρ/ε0	✓	✓
Burger's equation	u t + uu x - νu xx = 0	✗	✓
Navier-Stokes equations	u t + (u · ∇)u - ν ∇2u = -1/ρ ∇p + g	✗	✓
Korteweg-De Vries equation	u t + u xxx - 6uu x = 0	✗	✓
Kuramoto-Sivashinsky equation	u t + u xx + uxxxx + uu x = 0	✗	✓
Fisher's equation	u t - κu xx = ru(1 - u)	✗	✓
Liouville's equation	u xx + u yy = κeρu	✗	✓
Porous medium equation	u t - ∇2(uκ) = 0	✗	✓
Sine-Gordon equation	u tt - u xx = -sin(u)	✗	✓
B.3 Examples
The examples of VR-PDEs can be found in Table 5.
C D-CIPHER
C.1 Rewrite the inner optimization as a constrained least squares
Let us rewrite the objective in Equation 14.
$$
\sum_ {d = 1} ^ {D} \sum_ {s = 1} ^ {S} \left(\sum_ {p = 1} ^ {P} \mathcal {F} \left(\beta_ {p} \hat {\mathcal {E}} _ {p}, \hat {\boldsymbol {u}} ^ {(d)}, \phi_ {s}\right) - \int_ {\Omega} g (\boldsymbol {x}, \hat {\boldsymbol {u}} ^ {(d)} (\boldsymbol {x})) \phi_ {s} (\boldsymbol {x}) d \boldsymbol {x}\right) ^ {2} \tag {40}
$$
First, let us observe that
$$
\mathcal {F} \left(\beta_ {p} \hat {\mathcal {E}} _ {p}, \hat {\boldsymbol {u}} ^ {(d)}, \phi_ {s}\right) = \int_ {\Omega} h _ {p} (\boldsymbol {x}, \hat {\boldsymbol {u}} ^ {(d)} (\boldsymbol {x})) (- 1) ^ {| \boldsymbol {\alpha} _ {p} |} \partial^ {\boldsymbol {\alpha} _ {p}} \left[ \beta_ {p} a _ {p} (\boldsymbol {x}) \phi_ {s} (\boldsymbol {x}) \right] d \boldsymbol {x} = \beta_ {p} z _ {p} ^ {(d, s)} \tag {41}
$$
if we let  $z_p^{(d,s)} \in \mathbb{R}$  be defined as
$$
z _ {p} ^ {(d, s)} = \int_ {\Omega} h _ {p} (\boldsymbol {x}, \hat {\boldsymbol {u}} ^ {(d)} (\boldsymbol {x})) (- 1) ^ {| \boldsymbol {\alpha} _ {p} |} \partial^ {\boldsymbol {\alpha} _ {p}} \left(a _ {p} (\boldsymbol {x}) \phi_ {s} (\boldsymbol {x})\right) d \boldsymbol {x} \tag {42}
$$
Moreover, if we define  $w^{(d,s)}\in \mathbb{R}$  as
$$
w ^ {(d, s)} = \int_ {\Omega} g (\boldsymbol {x}, \hat {\boldsymbol {u}} ^ {(d)} (\boldsymbol {x})) \phi_ {s} (\boldsymbol {x}) d \boldsymbol {x} \tag {43}
$$
we can rewrite expression 40 as
$$
\sum_ {d = 1} ^ {D} \sum_ {s = 1} ^ {S} \left(\sum_ {p = 1} ^ {P} \beta_ {p} z _ {p} ^ {(d, s)} - w ^ {(d, s)}\right) \tag {44}
$$
Now, the sum over  $p$  can be written as a dot product between  $z^{(d,s)} \in \mathbb{R}^P$  and  $\beta \in \mathbb{R}^P$ . We can also combine the sums over  $d$  and  $s$ . We obtain
$$
\sum_ {(d, s) \in [ D ] \times [ S ]} \left(\boldsymbol {\beta} \cdot \boldsymbol {z} ^ {(d, s)} - w ^ {(d, s)}\right) ^ {2} \tag {45}
$$
which is exactly the same as the objective in Equation 15.
C.2 Pseudocode
The pseudocode of D-CIPHER is presented in Algorithm 1.
Algorithm 1 D-CIPHER  
Input: Observed fields  $\mathcal{D} = \{\pmb{v}^{(d)}\}_{d=1}^{D}$ , grid  $\mathcal{G}$   
Input: Symbolic regression optimization algorithm  $\mathcal{O}$   
Input: Smoothing algorithm  $\mathcal{S}$   
Input: Testing functions  $\{\phi_s\}_{s=1}^S$   
Input: Dictionary  $\mathcal{Q} = \{\hat{\mathcal{E}}_p\}_{p=1}^P$ ,  $\hat{\mathcal{E}}_p = (\alpha_p, a_p, h_p)$   
Output: Target PDE  
 $\hat{\pmb{u}}^{(d)} = \mathcal{S}(\pmb{v}^{(d)}) \forall d \in [D]$   
initialize matrix  $\pmb{Z} \in \mathbb{R}^{D \times S} \times \mathbb{R}^P$ $Z_{p}^{(d,s)} \gets \int_{\Omega} h_{p}(\pmb{x}, \hat{\pmb{u}}^{(d)}(\pmb{x})) (-1)^{|\pmb{\alpha}_p|} \partial^{\pmb{\alpha}_p} (a_p(\pmb{x}) \phi_s(\pmb{x})) d\pmb{x}$   
procedure Loss(g)  
initialize vector  $\pmb{w} \in \mathbb{R}^{D \times S}$ $w^{(d,s)} \gets \int_{\Omega} g(\pmb{x}, \hat{\pmb{u}}^{(d)}(\pmb{x})) \phi_s(\pmb{x}) d\pmb{x}$ $\beta \gets \text{COLLIE}(\pmb{Z}, \pmb{w})$ $L = ||\pmb{Z}\beta - \pmb{w}||_2^2$   
return L  
end procedure  
g = O(Loss)  
initialize vector  $\pmb{w} \in \mathbb{R}^{D \times S}$ $w^{(d,s)} \gets \int_{\Omega} g(\pmb{x}, \hat{\pmb{u}}^{(d)}(\pmb{x})) \phi_s(\pmb{x}) d\pmb{x}$ $\beta \gets \text{COLLIE}(\pmb{Z}, w)$   
return  $\sum_{p=1}^{P} \beta_p \hat{\mathcal{E}}_p[\pmb{u}](\pmb{x}) - g(\pmb{x}, \pmb{u}(\pmb{x})) = 0$
C.3 Testing functions
Testing functions should satisfy the following conditions:
1. Be sufficiently smooth (at least  $\mathcal{C}^K$  for a  $K^{\mathrm{th}}$  order PDE)
2. Compact support
3. Derivatives can be computed analytically
4. Orthonormal
Conditions 1 and 2 follow directly from Definition 3. Condition 3 is necessary because we do not want to estimate the derivatives of the testing functions. Condition 4 follows from the result obtained by [42] that suggests that these functions should be a subset of an orthonormal basis of L2 space.
We use B-Splines [15] as the testing functions in our experiments because we can control their smoothness and the derivatives are easy to compute. We scale and shift them appropriately so that they are orthonormal.
Other testing functions are possible and examining them constitutes an interesting research direction. In particular, piecewise polynomials as defined by [35] or various wavelets. Ideally, we would like to choose wavelets that form an orthonormal basis for the L2 space, such as
- Shannon wavelets - smooth  $(\mathcal{C}^{\infty})$  but not compact
- Meyer wavelets - smooth  $(\mathcal{C}^{\infty})$  but not compact (better rate of decay than Shannon)
- Daubechies wavelets - smooth ( $\mathcal{C}^K$  for a specified  $K$ ) and compact but they do not have a closed-form expression.
Another interesting avenue of research would be to adapt the testing functions to the input data.
D CoLLie
D.1 Lagrangian
The problem that CoLLie is supposed to solve is a constrained least-squares optimization defined as:
$$
\underset {b \rightarrow + \infty} {\text {m i n i m i z e}} \| A z - b \| _ {2} ^ {2} \tag {46}
$$
$$
\text {s u b j e c t} \left. \left| \left| z \right| \right| _ {1} - 1 = 0 \right.
$$
where  $\mathbf{A} \in \mathbb{R}^{m \times n}$  has a full column rank,  $\mathbf{b} \in \mathbb{R}^m$ , and  $\mathbf{z} \in \mathbb{R}^n$  for some  $m, n \in \mathbb{N}$ .
We consider the Lagrangian  $L:\mathbb{R}^n\times \mathbb{R}\to \mathbb{R}$  associated with this problem [8] defined as
$$
L (\boldsymbol {z}, \lambda) = | | \boldsymbol {A} \boldsymbol {z} - \boldsymbol {b} | | _ {2} ^ {2} + \lambda (| | \boldsymbol {z} | | _ {1} - 1) \tag {47}
$$
Now let us define  $\hat{z}:\mathbb{R}\to \mathbb{R}^n$  as
$$
\hat {\boldsymbol {z}} (\lambda) = \underset {\boldsymbol {z} \in \mathbb {R} ^ {n}} {\arg \min } L (\boldsymbol {z}, \lambda) = \underset {\boldsymbol {z} \in \mathbb {R} ^ {n}} {\arg \min } \| \boldsymbol {A} \boldsymbol {z} - \boldsymbol {b} \| _ {2} ^ {2} + \lambda \| \boldsymbol {z} \| _ {1} \tag {48}
$$
The goal of our algorithm is to find  $\lambda^{*} \in \mathbb{R}$  such that  $||\hat{z}(\lambda^{*})||_{1} = 1$ . Let us define a function  $q: \mathbb{R} \to \mathbb{R}$  as
$$
q (\lambda) = \left\| \hat {\boldsymbol {z}} (\lambda) \right\| _ {1} \tag {49}
$$
The goal can be phrased as finding  $\lambda^{*}\in \mathbb{R}$  such that  $q(\lambda^{*}) = 1$
Let us note that  $\hat{z}(0)$  is just a solution to the ordinary least squares (OLS) problem with no constraints and its norm is  $q(0)$ .
D.2 Extending LARS
Case 1.  $q(0)\geq 1$
If we assume that  $\hat{z}$  is continuous then  $q$  is also continuous. From the continuity and the fact that  $\lim_{\lambda \to +\infty} q(\lambda) = 0$  and  $q(0) \geq 1$  we infer that there exists a  $\lambda \geq 0$  such that  $q(\lambda) = 1$ . Moreover, for  $\lambda \geq 0$  the problem in Equation 48 is the same as in LASSO [53]. Therefore we just need to perform LASSO for different  $\lambda$  and choose the one that gives the solution with L1 norm equal to 1.
To do it in practice we use Least Angle Regression (LARS) [17], a popular algorithm used to minimize the LASSO objective. It generates complete solution paths, i.e., a function  $c: \mathbb{R}_+ \to \mathbb{R}^n$  defined as
$$
\boldsymbol {c} (\lambda) = \underset {\boldsymbol {z} \in \mathbb {R} ^ {n}} {\arg \min } \| \boldsymbol {A} \boldsymbol {z} - \boldsymbol {b} \| _ {2} ^ {2} + \lambda \| \boldsymbol {z} \| _ {1} \tag {50}
$$
which is equivalent to  $\hat{z}$  for  $\lambda \geq 0$ . An illustration of LARS solution paths can be seen in Figure 6. Each line corresponds to a function  $c_{i}$  which describes the coefficient for the  $i^{\mathrm{th}}$  covariate. The paths are defined from some  $\lambda_0$  where all  $c_{i}(\lambda_{0}) = 0$  to  $\lambda = 0$  where  $c(0) = \hat{z} (0)$ . In other words, the solution paths cover the whole range of constraints from the strictest, effectively imposing the L1 norm of  $z$  to be 0, up to no constraints, solving the OLS problem.
The solution paths from the LARS algorithm are piecewise linear and the outputs are the values of the coefficients for points  $(\lambda_0 > \ldots > \lambda_n = 0)$  where the slopes change. We calculate the norm at each of these points,  $||c(\lambda_i)||_1$ , and find  $j \in [n]$  such that  $||c(\lambda_{j-1})||_1 < 1 \leq ||c(\lambda_j)||_1$ . As each  $c_i$  is a linear function on  $[\lambda_j, \lambda_{j-1}]$  and we know both  $c(\lambda_{j-1})$  and  $c(\lambda_j)$ , we can effectively search for  $\lambda \in [\lambda_j, \lambda_{j-1}]$  such that  $||c(\lambda)||_1 = 1$ . The search can be performed by any root-finding algorithm. We use Brent's method [9].
Case 2.  $0 < q(0) < 1$ .
This is much more difficult as it corresponds to solving the problem in Equation 48 for  $\lambda < 0$ . The solutions given by the LARS algorithm are too small. In fact, the solution with the biggest norm is  $c(0) = \hat{z}(0)$ , the OLS solution, with norm exactly  $q(0) < 1$ .
To address this challenge, we propose the following heuristic. We extend the solution paths generated by LARS beyond  $\lambda = 0$  for  $\lambda < 0$ . We assume that the paths will continue to be piecewise linear and that they will keep the slope they have in the last interval  $[\lambda_n = 0, \lambda_{n-1}]$ . Let us denote this slope as
$$
\Delta c _ {i} = \frac {c _ {i} (0) - c _ {i} \left(\lambda_ {n - 1}\right)}{0 - \lambda_ {n - 1}} \tag {51}
$$
This is graphically represented in Figure 6. Formally, these extended paths,  $\bar{c}:\mathbb{R}\to \mathbb{R}$  are defined as:
$$
\bar {c} _ {i} (\lambda) = \left\{ \begin{array}{l l} c _ {i} (\lambda), & \lambda \geq 0 \\ c _ {i} (0) + \lambda \Delta c _ {i}, & \lambda <   0 \end{array} \right. \tag {52}
$$
Now, we want to find  $\lambda < 0$  such that  $||\bar{c} (\lambda)||_1 = 1$ . To achieve this in practice, we first make the following observations.
For any  $\lambda < 0$  we say that  $\bar{c}_i(\lambda)$  is on the right side if  $\bar{c}_i(\lambda)\Delta c_i\leq 0$  and we say that  $\bar{c}_i(\lambda)$  is on the wrong side if  $\bar{c}_i(\lambda)\Delta c_i > 0$ . In other words, being on the wrong side just means that the path is yet to cross the x-axis if we keep decreasing  $\lambda$ . We can easily find  $\lambda^\prime$  such that for all  $\lambda < \lambda^{\prime}$  all  $\bar{c}_i(\lambda)$  are on the right side (none of the paths will ever cross the x-axis).
$$
\lambda^ {\prime} = \min  \left\{\frac {0 - c _ {i} (0)}{\Delta c _ {i}} \mid i \in [ n ] \wedge c _ {i} (0) \Delta c _ {i} > 0 \right\} \tag {53}
$$
If  $||\bar{c}(\lambda')||_1 \geq 1$  we just need to search the interval  $[ \lambda', 0 ]$  for  $\lambda$  such that  $||\bar{c}(\lambda')||_1 = 1$ .
If  $||\overline{c}(\lambda')||_1 < 1$  then we need to search  $\lambda < \lambda'$ . However, by definition, for all  $\lambda < \lambda'$ , all  $c_i(\lambda)$  are on the right side. That means  $||\overline{c}(\lambda)||_1$  as a function of  $\lambda$  is just a linear function on the interval  $(-\infty, \lambda')$ . To see that, let us observe that
$$
\left| \left| \bar {c} (\lambda) \right| \right| _ {1} = \sum_ {i = 1} ^ {n} | \bar {c} _ {i} (\lambda) | = \sum_ {i = 1} ^ {n} \operatorname {s i g n} \left(\bar {c} _ {i} (\lambda)\right) \bar {c} _ {i} (\lambda) \tag {54}
$$
Additionally, for  $\lambda < \lambda'$  all  $c_i(\lambda)$  are on the right side, so we have  $\text{sign}(\bar{c}_i(\lambda)) = -\text{sign}(\Delta c_i)$ . We can rewrite  $||\bar{c}(\lambda)||_1$  as:
$$
\begin{array}{l} \left| \left| \bar {c} (\lambda) \right| \right| _ {1} = \sum_ {i = 1} ^ {n} (- s i g n (\Delta c _ {i}) (c _ {i} (0) + \lambda \Delta c _ {i})) \\ = - \sum_ {i = 1} ^ {n} \operatorname {s i g n} \left(\Delta c _ {i}\right) c _ {i} (0) - \left(\sum_ {i = 1} ^ {n} \operatorname {s i g n} \left(\Delta c _ {i}\right) \Delta c _ {i}\right) \lambda \tag {55} \\ = - \sum_ {i = 1} ^ {n} \operatorname {s i g n} \left(\Delta c _ {i}\right) c _ {i} (0) - \left(\sum_ {i = 1} ^ {n} \left| \Delta c _ {i} \right|\right) \lambda \\ \end{array}
$$
Therefore the solution can be found using the following equation
$$
\lambda^ {*} = \lambda^ {\prime} + \frac {1 - | | \bar {c} \left(\lambda^ {\prime}\right) | | _ {1}}{- \sum_ {i = 1} ^ {n} \left| \Delta c _ {i} \right|} \tag {56}
$$
Case 3.  $q(0) = 0$ . In that case, we just return a precomputed solution to the problem
$$
\underset {\text {m i n i m i z e}} {| | \boldsymbol {A} \boldsymbol {z} | | _ {2} ^ {2}} \tag {57}
$$
which we compute by subdividing the problem into  $2^{n}$  quadratic programs and solving each of them separately using CVXOPT algorithm [2] as described in Section 7.
D.3 Comparison
We perform a comparison between CoLLie and an algorithm based on CVXOPT [2] as described in Section 7. CoLLie uses different procedures depending on the problem, thus for a fair comparison we generate an equal number of tests falling under Case 1 and Case 2 (two main cases) as described in D.2. To achieve that, we generate a random  $m \times n$  matrix  $\mathbf{A}$  where each entry is sampled from the standard normal distribution. We use  $m = 1000$  and  $n$  ranging from 2 to 7. We then generate a vector  $\hat{\boldsymbol{z}} \in \mathbb{R}^n$  such that each entry is sampled from a uniform distribution on  $[-0.5, 0.5]$ . Then  $\hat{\boldsymbol{z}}$  is normalized to have L1 norm equal to 1. We sample a number  $l$  from a uniform distribution on  $[-1, 3]$  and multiply  $\hat{\boldsymbol{z}}$  by  $l$  to obtain  $z' = l\hat{\boldsymbol{z}}$ . By this procedure, we are guaranteed that cases  $||z'||_1 < 1$
Figure 6: Panel A shows and example of solution paths calculated by the LARS algorithm. Panel B shows their extended versions as defined in Case 2 in D.2. The x-axis is reversed, so  $\lambda$  decreases as it moves to the right.
and  $||z^{\prime}||_{1}\geq 1$  are equally likely. We then let  $\pmb {b} = \pmb {A}\pmb{z}^{\prime}$  . The task is then to find  $\textbf{z}$  with L1 norm equal to 1 that minimizes  $||\pmb {A}\pmb {z} - \pmb {b}||_2^2$  . As  $q(0) = ||z^{\prime}||_{1} = l$  , Case 1 and Case 2 are equally likely.