SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that in the one-dimensional case, we could have also used the natural parameters, but due to the truncation structure, the parameters we used make the computation simpler, in a way that does not apply to the multidimensional case. Also note that here we are considering a fixed  $x$  and writing  $v$  as a vector. In full generality, we should take  $V = -\Sigma^{-1}W$ , however, this is a simple extension which we omit here for readability. It turns out that density is log-concave in these natural parameters:
Lemma F.1. The log-likelihood function in Eqn (8) is concave in the natural parameter space.
Proof. First, let's write the un-truncated density in terms of these parameters:
$$
\begin{array}{l} f _ {W, \Sigma} (y | x) = \exp \left(- \frac {1}{2} (y - W x) ^ {T} \Sigma^ {- 1} (y - W x) - \frac {1}{2} \log | 2 \pi \Sigma |\right), (47) \\ = \exp \left(- \frac {1}{2} y ^ {T} \Sigma^ {- 1} y + x ^ {T} W ^ {T} \Sigma^ {- 1} y - x ^ {T} W ^ {T} \Sigma^ {- 1} W x - \frac {1}{2} \log | 2 \pi \Sigma |\right) (48) \\ = \exp \left(- \frac {1}{2} y ^ {T} U y - v ^ {T} y - v ^ {T} U ^ {- 1} v - \frac {1}{2} \log \left(\left(2 \pi\right) ^ {n} / | U |\right)\right) (49) \\ \end{array}
$$
Thus, the untruncated conditional density can be written as:
$$
f _ {U, v} (y) = \exp \left(- \frac {1}{2} y ^ {T} U y + y ^ {T} v - A (U, v)\right),
$$
where  $A(U,v)$  is the cumulant function (note this is distinct from the related cumulant generating function). A well known result is that  $A$  is jointly convex in its arguments,  $U$  and  $v$ . Taking logs and using this fact, shows us that  $f_{U,v}(y)$  is log-concave in  $U,v$ .
Our truncated density is simply:
$$
f _ {U, v} (y | x) = \int_ {y _ {S} \leq 0} p _ {U, v} (y | x) d y _ {S},
$$
For any log-concave density  $f(x)$ , integration over a convex subset of the coordinates preserves log-concavity ([9], Example 3.42-3.44). Thus the objective is log-concave.
Then the likelihood function at  $U, v$  can be rewritten as
$$
\begin{array}{l} f _ {U, v} (y) = \log \int_ {t _ {S} \leq 0, t _ {S ^ {\prime}} = y _ {S ^ {\prime}}} \exp \left(- t ^ {T} U t - t ^ {T} v - \frac {v ^ {T} U ^ {- 1} v}{4} + \frac {1}{2} \log | 2 U |\right), \\ = - \frac {v ^ {T} U ^ {- 1} v}{4} + \frac {1}{2} \log | 2 U | + \log \int_ {t _ {S} \leq 0, t _ {S ^ {\prime}} = y _ {S ^ {\prime}}} \exp \left(- t ^ {T} U t - t ^ {T} v\right), \\ \end{array}
$$
Separating the terms corresponding to  $S$  and  $S^{\prime}$  , we get
$$
f _ {U, v} (y) = - \frac {v ^ {T} U ^ {- 1} v}{4} + \frac {1}{2} \log | 2 U | - y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime}} y _ {S ^ {\prime}} - y _ {S ^ {\prime}} ^ {T} v _ {S ^ {\prime}} + \log \int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right).
$$
The last term resembles the log  $\Phi$  term that appears in the univariate case. This resemblance can be made more clear as follows. Let  $r_S = U_{SS'}y_S' + \frac{1}{2} v_S$  and  $M^T M = U_S$ .
$$
\begin{array}{l} = \log \int_ {t _ {S} \leq 0} \exp \left(- \left(t _ {S} ^ {T} M ^ {T} M t _ {S} + 2 t _ {S} ^ {T} r _ {S}\right)\right) \\ = \log \int_ {t _ {S} \leq 0} \exp \left(- \left((M t _ {S}) ^ {T} M t _ {S} + 2 t _ {S} ^ {T} r _ {S} + r _ {S} ^ {T} M ^ {- 1} M ^ {- T} r _ {S} - r _ {S} ^ {T} M ^ {- 1} M ^ {- T} r _ {S}\right)\right) \\ = \log \int_ {t _ {S} \leq 0} \exp \left(- \left\| M t _ {S} + M ^ {- T} r _ {S} \right\| ^ {2} + r _ {S} ^ {T} M ^ {- 1} M ^ {- T} r _ {S}\right) \\ = r _ {S} ^ {T} M ^ {- 1} M ^ {- T} r _ {S} + \log \int_ {t _ {S} \leq 0} \exp \left(- \left\| t _ {S} + U _ {S} ^ {- 1} r _ {S} \right\| _ {U _ {S}} ^ {2}\right) \\ = r _ {S} ^ {T} U _ {s} ^ {- 1} r _ {S} + \log \int_ {t _ {S} \leq 0} \frac {(2 \pi) ^ {d / 2} | U _ {S} ^ {- 1} / 2 | ^ {1 / 2}}{(2 \pi) ^ {d / 2} | U _ {S} ^ {- 1} / 2 | ^ {1 / 2}} \exp \left(- \| t _ {S} + U _ {S} ^ {- 1} r _ {S} \| _ {U _ {S}} ^ {2}\right) \\ = r _ {S} ^ {T} U _ {s} ^ {- 1} r _ {S} - \frac {1}{2} \log | 2 U _ {S} | + \log \int_ {t _ {S} \leq 0} \frac {1}{(2 \pi) ^ {d / 2} | U _ {S} ^ {- 1} / 2 | ^ {1 / 2}} \exp \left(- \frac {1}{2} \| t _ {S} + U _ {S} ^ {- 1} r _ {S} \| _ {2 U _ {S}} ^ {2}\right) + c \\ = r _ {S} ^ {T} U _ {s} ^ {- 1} r _ {S} - \frac {1}{2} \log | 2 U _ {S} | + \log \Phi \left(0; \mu = - U _ {S} ^ {- 1} r _ {S}, \Sigma = \frac {1}{2} U _ {S} ^ {- 1}\right) + c \\ \end{array}
$$
Putting this together,  $f_{U,v}$  can be written as:
$$
\begin{array}{l} f _ {U, v} (y) = - \frac {v ^ {T} U ^ {- 1} v}{4} + \frac {1}{2} \log | U | - y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime}} y _ {S ^ {\prime}} - y _ {S ^ {\prime}} ^ {T} v _ {S ^ {\prime}} + r _ {S} ^ {T} U _ {S} ^ {- 1} r _ {S} - \frac {1}{2} \log | U _ {S} | + | S ^ {\prime} | \frac {\log (2)}{2} \\ + \log \Phi \left(0; \mu = - U _ {S} ^ {- 1} r _ {S}, \Sigma = \frac {1}{2} U _ {S} ^ {- 1}\right) + c \tag {51} \\ \end{array}
$$
Thus, it appears that evaluating the likelihood for even a single sample involves the high-dimensional integral that is the rectangular cdf in equation (51).
F.3 Computing Gradients
F.3.1 One Dimensional Case
In the one-dimensional case, the gradient with respect to  $u$  is easily computed as:
$$
\nabla_ {u} f _ {u, v} (y) = \sum_ {i \in S ^ {\prime}} (v y _ {i} - u \cdot x _ {i}) x _ {i} - \sum_ {i \in S} \frac {1}{m (u \cdot x _ {i})} x _ {i},
$$
where we have previously defined  $m(x)$  as the mills ratio. Furthermore, we have:
$$
\nabla_ {v} f _ {u, v} (y) = \left| S ^ {\prime} \right| \frac {1}{v} - \sum_ {i \in S ^ {\prime}} y _ {i} \left(v y _ {i} - u \cdot x _ {i}\right)
$$
F.3.2 Multidimensional Case
First we consider the non-integral terms in the likelihood. Differentiating each term wrt  $U$ , we get
$$
\begin{array}{l} - \nabla_ {U} \frac {v ^ {T} U ^ {- 1} v}{4} = \frac {1}{4} (U ^ {- 1} v v ^ {T} U ^ {- 1}), \\ \nabla_ {U} \frac {1}{2} \log | 2 U | = \frac {1}{2} U ^ {- 1}, \\ - \nabla_ {U} y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime}} y _ {S ^ {\prime}} = \left( \begin{array}{c c} 0 & 0 \\ 0 & - y _ {S ^ {\prime}} y _ {S ^ {\prime}} ^ {T} \end{array} \right) \\ \end{array}
$$
Differentiating each term wrt  $v$ , we get
$$
\begin{array}{l} - \nabla_ {v} \frac {v ^ {T} U ^ {- 1} v}{4} = - \frac {1}{2} U ^ {- 1} v, \\ - \nabla_ {v} y _ {S ^ {\prime}} ^ {T} v _ {S ^ {\prime}} = \left( \begin{array}{c} 0 \\ - y _ {S ^ {\prime}} \end{array} \right) \\ \end{array}
$$
Now consider the integral term. Differentiating wrt  $U$ , we get
$$
\begin{array}{l} \nabla_ {U} \log \int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right) \\ = \frac {\int_ {t _ {S} \leq 0} \left( \begin{array}{c c} - t _ {S} t _ {S} ^ {T} & - t _ {S} y _ {S ^ {\prime}} ^ {T} \\ & \\ - y _ {S ^ {\prime}} t _ {S} ^ {T} & 0 \end{array} \right) \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right)}{\int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right)} \\ \end{array}
$$
Let  $M$  be a matrix such that
$$
M ^ {T} M = U _ {S}
$$
Then via completion of squares in the exponential term, we get
$$
\begin{array}{l} \nabla_ {U} \log \int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right) (52) \\ = \frac {\int_ {t _ {S} \leq 0} \left( \begin{array}{c c} - t _ {S} t _ {S} ^ {T} & - t _ {S} y _ {S ^ {\prime}} ^ {T} \\ - y _ {S ^ {\prime}} t _ {S} ^ {T} & 0 \end{array} \right) \exp \left(- \| M t _ {S} + (M ^ {- 1}) ^ {T} \left(U _ {S S ^ {\prime}} y _ {S ^ {\prime}} + \frac {v _ {S}}{2}\right) \| ^ {2}\right)}{\int_ {t _ {S} \leq 0} \exp \left(- \| M t _ {S} + (M ^ {- 1}) ^ {T} \left(U _ {S S ^ {\prime}} y _ {S ^ {\prime}} + \frac {v _ {S}}{2}\right) \| ^ {2}\right)} (53) \\ \end{array}
$$
Notice that this density is Gaussian, with mean and covariance:
$$
\mathcal {N} \bigg (- M ^ {- 1} \big (M ^ {- 1} \big) ^ {T} \Big (U _ {S S ^ {\prime}} y _ {S ^ {\prime}} + \frac {v _ {S}}{2} \Big); \frac {U _ {S} ^ {- 1}}{2} \bigg).
$$
And hence, the gradient can be estimated as
$$
\nabla_ {U} \log \int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right) = \underset {t _ {S}} {\mathbb {E}} \left[ \left( \begin{array}{c c} - t _ {S} t _ {S} ^ {T} & - t _ {S} y _ {S ^ {\prime}} ^ {T} \\ - y _ {S ^ {\prime}} t _ {S} ^ {T} & 0 \end{array} \right) \right]
$$
where  $t_S$  is the truncation of
$$
z \sim \mathcal {N} \left(- M ^ {- 1} \left(M ^ {- 1}\right) ^ {T} \left(U _ {S S ^ {\prime}} y _ {S ^ {\prime}} + \frac {v _ {S}}{2}\right); \frac {U _ {S} ^ {- 1}}{2}\right).
$$
to the negative quadrant. A similar calculation gives the gradient for  $v$  as
$$
\nabla_ {v} \log \int_ {t _ {S} \leq 0} \exp \left(- t _ {S} ^ {T} U _ {S} t _ {S} - 2 y _ {S ^ {\prime}} ^ {T} U _ {S ^ {\prime} S} t _ {S} - v _ {S} ^ {T} t _ {S}\right) = \underset {t _ {S}} {\mathbb {E}} \left[ \binom {- t _ {S}} {0} \right]
$$