SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Observe that Theorem 4 includes the case where  $a = 1$ , in which case  $\sigma_{a} = Id$  and  $\mathcal{V} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)) = \overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$ . We note that when  $\sigma_{a}$  is a leaky ReLU-function with parameter  $a > 0$ , Theorem 4 can be applied to compute the inverse of  $\sigma_{a} \circ F_{1}$  that is given by  $F_{1}^{-1} \circ \sigma_{a}^{-1}$ , where  $\sigma_{a}^{-1} = \sigma_{1 / a}$ . Note that the assumption that  $\mathcal{V} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$  makes it possible to apply Theorem 4 in the case when one trains deep neural networks having layers  $\sigma_{a} \circ F_{1}$  and the parameter  $a$  of the leaky ReLU-function is a free parameter which is also trained.
Proof. As the operator  $F_{1}$  can be multiplied by function  $W(x)^{-1}$ , it is sufficient to consider the case when  $W(x) = 1$ .
Below, we use the fact that as  $D \subset \mathbb{R}$ , Sobolev's embedding theorem yields that the embedding  $H^{1}(D) \to C(\overline{D})$  is bounded and there is  $C_S > 0$  such that
$$
\left\| u \right\| _ {C (\overline {{D}})} \leq C _ {S} \| u \| _ {H ^ {1} (D)}. \tag {F.7}
$$
For clarity, we denote below the norm of  $C(\overline{D})$  by  $\| u\|_{L^{\infty}(D)}$ .
Next we consider the Frechet derivatives of  $F_{1}$ . We recall that the 1st Frechet derivative of  $F_{1}$  at  $u_{0}$  is the operator  $A_{u_0}$ . The 2nd Frechet derivative of  $F_{1}$  at  $u_{0}$  to directions  $w_{1}$  and  $w_{2}$  is
$$
\begin{array}{l} {D ^ {2} F _ {1} | _ {u _ {0}} (w _ {1}, w _ {2})} = {\int_ {D} 2 \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) w _ {1} (y) w _ {2} (y) d y + \int_ {D} u _ {0} (y) \frac {\partial k ^ {2}}{\partial u ^ {2}} (x, y, u _ {0} (y)) w _ {1} (y) w _ {2} (y) d y} \\ = \int_ {D} p (x, y) w _ {1} (y) w _ {2} (y) d y, \\ \end{array}
$$
$$
p (x, y) = 2 \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) + u _ {0} (y) \frac {\partial k ^ {2}}{\partial u ^ {2}} (x, y, u _ {0} (y)), \tag {F.8}
$$
$$
\frac {\partial}{\partial x} p (x, y) = 2 \frac {\partial^ {2} k}{\partial u \partial x} (x, y, u _ {0} (y)) + u _ {0} (y) \frac {\partial k ^ {3}}{\partial u ^ {2} \partial x} (x, y, u _ {0} (y)). \tag {F.9}
$$
(F.10)
$$
\| D ^ {2} F _ {1} | _ {u _ {0}} (w _ {1}, w _ {2}) \| _ {H ^ {1} (D)} \leq 3 | D | ^ {1 / 2} \| k \| _ {C ^ {3} (D \times D \times \mathbb {R})} (1 + \| u _ {0} \| _ {L ^ {\infty} (D)}) \| w _ {1} \| _ {L ^ {\infty} (D)} \| w _ {2} \| _ {L ^ {\infty} (D)}.
$$
When we freeze the function  $u$  in kernel  $k$  to be  $u_0$ , we denote
$$
K _ {u _ {0}} v (x) = \int_ {D} k (x, y, u _ {0} (y)) v (y) d y,
$$
Lemma 4. For  $u_0, u_1 \in C(\overline{D})$  we have
$$
\left\| K _ {u _ {1}} - K _ {u _ {0}} \right\| _ {L ^ {2} (D) \rightarrow H ^ {1} (D)} \leq \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} | D | \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)}.
$$
$$
\left\| A _ {u _ {1}} - A _ {u _ {0}} \right\| _ {L ^ {2} (D) \rightarrow H ^ {1} (D)} \leq 2 \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} | D | \left(1 + \left\| u _ {0} \right\| _ {L ^ {\infty} (D)}\right) \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)}. \tag {F.11}
$$
$$
{M _ {u _ {0}} v (x)} = {\int_ {D} u _ {0} (y) \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) v (y) d y,}
$$
$$
{N _ {u _ {1}, u _ {2}} v (x)} = {\int_ {D} u _ {1} (y) \frac {\partial k}{\partial u} (x, y, u _ {2} (y)) v (y) d y.}
$$
$$
M _ {u _ {2}} v - M _ {u _ {1}} v = \left(N _ {u _ {2}, u _ {2}} v - N _ {u _ {2}, u _ {1}} v\right) + \left(N _ {u _ {2}, u _ {1}} v - N _ {u _ {1}, u _ {1}} v\right).
$$
By Schur's test for continuity of integral operators,
$$
\begin{array}{l} \| K _ {u _ {0}} \| _ {L ^ {2} (D) \to L ^ {2} (D)} \leq \left(\sup  _ {x \in D} \int_ {D} | k (x, y, u _ {0} (y)) | d y\right) ^ {1 / 2} \left(\sup  _ {y \in D} \int_ {D} | k (x, y, u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \leq \left\| k \right\| _ {C ^ {0} (D \times D \times \mathbb {R})}, \\ \end{array}
$$
$$
\begin{array}{l} \| M _ {u _ {0}} \| _ {L ^ {2} (D) \to L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | u _ {0} (y) \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) | d y\right) ^ {1 / 2} \left(\sup  _ {y \in D} \int_ {D} | u _ {0} (y) \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \leq \left\| k \right\| _ {C ^ {1} (D \times D \times \mathbb {R})} \| u \| _ {C (\overline {{D}})}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| K _ {u _ {2}} - K _ {u _ {1}} \right\| _ {L ^ {2} (D) \rightarrow L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | k (x, y, u _ {2} (y)) - k (x, y, u _ {1} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\sup  _ {y \in D} \int_ {D} | k (x, y, u _ {2} (y)) - k (x, y, u _ {1} (y)) | d x\right) ^ {1 / 2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left(\| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} \int_ {D} | u _ {2} (y) - u _ {1} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} \sup  _ {y \in D} \int_ {D} | u _ {2} (y) - u _ {1} (y)) | d x\right) ^ {1 / 2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} \left(\int_ {D} | u _ {2} (y) - u _ {1} (y)) | d y\right) ^ {1 / 2} \left(\sup  _ {y \in D} \int_ {D} | u _ {2} (y) - u _ {1} (y)) | d x\right) ^ {1 / 2} \\ \leq \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} \left(| D | ^ {1 / 2} \| u _ {2} - u _ {1} \| _ {L ^ {2} (D)}\right) ^ {1 / 2} \left(| D | \sup  _ {y \in D} | u _ {2} (y) - u _ {1} (y)) |\right) ^ {1 / 2} \\ \leq \left\| k \right\| _ {C ^ {1} (D \times D \times \mathbb {R})} | D | ^ {3 / 4} \| u _ {2} - u _ {1} \| _ {L ^ {2} (D)} ^ {1 / 2} \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)} ^ {1 / 2} \\ \leq \| k \| _ {C ^ {1} \left(D \times D \times \mathbb {R}\right)} | D | \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| N _ {u _ {2}, u _ {2}} - N _ {u _ {2}, u _ {1}} \right\| _ {L ^ {2} (D) \rightarrow L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | u _ {2} (y) k (x, y, u _ {2} (y)) - u _ {2} (y) k (x, y, u _ {1} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\sup  _ {y \in D} \int_ {D} | u _ {2} (y) k (x, y, u _ {2} (y)) - u _ {2} (y) k (x, y, u _ {1} (y)) | d x\right) ^ {1 / 2} \\ \leq \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} | D | ^ {3 / 4} \| u _ {2} \| _ {C ^ {0} (D)} \| u _ {2} - u _ {1} \| _ {L ^ {2} (D)} ^ {1 / 2} \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)} ^ {1 / 2} \\ \leq \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} | D | \| u _ {2} \| _ {C ^ {0} (D)} \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| N _ {u _ {2}, u _ {1}} - N _ {u _ {1}, u _ {1}} \right\| _ {L ^ {2} (D) \to L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | (u _ {2} (y) - u _ {1} (y)) k (x, y, u _ {1} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\sup  _ {y \in D} \int_ {D} | (u _ {2} (y) - u _ {1} (y)) k (x, y, u _ {1} (y)) | d x\right) ^ {1 / 2} \\ \leq \| k \| _ {C ^ {0} (D \times D \times \mathbb {R})} | D | \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| M _ {u _ {2}} - M _ {u _ {1}} \right\| _ {L ^ {2} (D) \rightarrow L ^ {2} (D)} \\ \leq \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})} | D | \left(1 + \| u _ {2} \| _ {C ^ {0} (D)}\right) \| u _ {2} - u _ {1} \| _ {L ^ {\infty} (D)}. \\ \end{array}
$$
Also, when  $D_{x}v = \frac{dv}{dx}$ ,
$$
\begin{array}{l} \left\| D _ {x} \circ K _ {u _ {0}} \right\| _ {L ^ {2} (D) \to L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | D _ {x} k (x, y, u _ {0} (y)) | d y\right) ^ {1 / 2} \left(\sup  _ {y \in D} \int_ {D} | D _ {x} k (x, y, u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \leq \left\| k \right\| _ {C ^ {1} (D \times D \times \mathbb {R})}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| D _ {x} \circ K _ {u _ {1}} - D _ {x} \circ K _ {u _ {0}} \right\| _ {L ^ {2} (D) \rightarrow L ^ {2} (D)} \\ \leq \left(\sup  _ {x \in D} \int_ {D} | D _ {x} k (x, y, u _ {1} (y)) - D _ {x} k (x, y, u _ {0} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\sup  _ {y \in D} \int_ {D} | D _ {x} k (x, y, u _ {1} (y)) - D _ {x} k (x, y, u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left(\| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} \int_ {D} | u _ {1} (y) - u _ {0} (y)) | d y\right) ^ {1 / 2} \\ \times \left(\| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} \sup  _ {y \in D} \int_ {D} | u _ {1} (y) - u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \end{array}
$$
$$
\begin{array}{l} \leq \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} \left(\int_ {D} | u _ {1} (y) - u _ {0} (y)) | d y\right) ^ {1 / 2} \left(\sup  _ {y \in D} \int_ {D} | u _ {1} (y) - u _ {0} (y)) | d x\right) ^ {1 / 2} \\ \leq \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} \left(| D | ^ {1 / 2} \| u _ {1} - u _ {0} \| _ {L ^ {2} (D)}\right) ^ {1 / 2} \left(| D | \sup  _ {y \in D} | u _ {1} (y) - u _ {0} (y)) |\right) ^ {1 / 2} \\ \leq \left\| k \right\| _ {C ^ {2} (D \times D \times \mathbb {R})} | D | ^ {3 / 4} \| u _ {1} - u _ {0} \| _ {L ^ {2} (D)} ^ {1 / 2} \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)} ^ {1 / 2} \\ \leq \| k \| _ {C ^ {2} \left(D \times D \times \mathbb {R}\right)} | D | \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)}. \\ \end{array}
$$
$$
\left\| K _ {u _ {0}} \right\| _ {L ^ {2} (D) \to H ^ {1} (D)} \leq \left\| k \right\| _ {C ^ {1} (D \times D \times \mathbb {R})},
$$
$$
\left\| M _ {u _ {0}} \right\| _ {L ^ {2} (D) \rightarrow H ^ {1} (D)} \leq \left\| u _ {0} \right\| _ {C ^ {0} (D)} \| k \| _ {C ^ {1} (D \times D \times \mathbb {R})},
$$
$$
\| K _ {u _ {1}} - K _ {u _ {0}} \| _ {L ^ {2} (D) \to H ^ {1} (D)} \leq \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} | D | \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)}.
$$
$$
\left\| M _ {u _ {1}} - M _ {u _ {0}} \right\| _ {L ^ {2} (D) \rightarrow H ^ {1} (D)} \leq \| k \| _ {C ^ {2} (D \times D \times \mathbb {R})} | D | \left(1 + \| u _ {2} \| _ {C ^ {0} (D)}\right) \| u _ {1} - u _ {0} \| _ {L ^ {\infty} (D)}.
$$
As  $A_{u_1} = K_{u_1} + M_{u_1}$ , the claim follows.
As the embedding  $H^{1}(D) \to C(\overline{D})$  is bounded and has norm  $C_S$ , Lemma 4 implies that for all  $R > 0$  there is
$$
C _ {L} (R) = 2 \| k \| _ {C ^ {2} \left(D \times D \times \mathbb {R}\right)} | D | \left(1 + C _ {S} R\right),
$$
such that the map,
$$
u _ {0} \mapsto D F _ {1} | _ {u _ {0}}, \quad u _ {0} \in \bar {B} _ {H ^ {1}} (0, R), \tag {F.12}
$$
is Lipschitz map  $\overline{B}_{H^1}(0,R)\to \mathcal{L}(H^1 (D),H^1 (D))$  with a Lipschitz constant  $C_L(R)$  , that is,
$$
\left\| \right. D F _ {1} \left. \right| _ {u _ {1}} - D F _ {1} \left| _ {u _ {2}} \right| _ {H ^ {1} (D) \rightarrow H ^ {1} (D)} \leq C _ {L} (R) \left\| u _ {1} - u _ {2} \right\| _ {H ^ {1} (D)}. \tag {F.13}
$$
As  $u_0 \mapsto A_{u_0} = DF_1|_{u_0}$  is continuous, the inverse  $A_{u_0}^{-1}: H^1(D) \to H^1(D)$  exists for all  $u_0 \in C(\overline{D})$ , and the embedding  $H^1(D) \to C(\overline{D})$  is compact, we have that for all  $R > 0$  there is  $C_B(R) > 0$  such that
$$
\left\| A _ {u _ {0}} ^ {- 1} \right\| _ {H ^ {1} (D) \rightarrow H ^ {1} (D)} \leq C _ {B} (R), \quad \text {f o r a l l} u _ {0} \in \bar {B} _ {H ^ {1}} (0, R). \tag {F.14}
$$
Let  $R_1, R_2 > 0$  be such that  $\mathcal{Y} \subset \overline{B}_{H^1}(0, R_1)$  and  $X = F_1^{-1}(\mathcal{Y}) \subset \overline{B}_{H^1}(0, R_2)$ . Below, we denote  $C_L = C_L(2R_2)$  and  $C_B = C_B(R_2)$ .
Next we consider inverse of  $F_{1}$  in  $\mathcal{V}$ . To this end, let us consider  $\varepsilon_0 > 0$ , which we choose later to be small enough. As  $\mathcal{V} \subset \overline{B}_{H^1}(0,R)$  is compact there are finite number of elements  $g_{j} = F_{1}(v_{j}) \in \mathcal{V}$  where  $v_{j} \in X, j = 1,2,\ldots ,J$  such that
$$
\mathcal {Y} \subset \bigcup_ {j = 1} ^ {J} B _ {H ^ {1} (D)} (g _ {j}, \varepsilon_ {0}).
$$
We observe that for  $u_0, u_1 \in X$ ,
$$
A _ {u _ {1}} ^ {- 1} - A _ {u _ {0}} ^ {- 1} = A _ {u _ {1}} ^ {- 1} \left(A _ {u _ {1}} - A _ {u _ {0}}\right) A _ {u _ {0}} ^ {- 1},
$$
and hence the Lipschitz constant of  $A_{u}^{-1}: u \mapsto A_{u}^{-1}, X \to \mathcal{L}(H^{1}(D), H^{1}(D))$  satisfies
$$
L i p \left(A _ {1} ^ {- 1}\right) \leq C _ {A} = C _ {B} ^ {2} C _ {L}, \tag {F.15}
$$
see (F.11).
Let us consider a fixed  $j$  and  $g_{j}\in \mathcal{V}$ . When  $g$  satisfies
$$
\left\| g - g _ {j} \right\| _ {H ^ {1} (D)} <   2 \varepsilon_ {0}, \tag {F.16}
$$
the equation
$$
F _ {1} (u) = g, \quad u \in X,
$$
is equivalent to the fixed point equation
$$
u = u - A _ {v _ {j}} ^ {- 1} \left(F _ {1} (u) - F _ {1} (v _ {j})\right) + A _ {v _ {j}} ^ {- 1} (g - g _ {j}),
$$
that is equivalent to the fixed point equation
$$
H _ {j} (u) = u,
$$
for the function  $H_{j}:H^{1}(D)\to H^{1}(D)$
$$
H _ {j} (u) = u - A _ {v _ {j}} ^ {- 1} \left(F _ {1} (u) - F _ {1} \left(v _ {j}\right)\right) + A _ {v _ {j}} ^ {- 1} \left(g - g _ {j}\right).
$$
Note that  $H_{j}$  depends on  $g$ , and thus we later denote  $H_{j} = H_{j}^{g}$ . We observe that
$$
H _ {j} \left(v _ {j}\right) = v _ {j} + A _ {v _ {j}} ^ {- 1} \left(g - g _ {j}\right). \tag {F.17}
$$
Let  $u, v \in \overline{B}_{H^1}(0, 2R_2)$ . We have
$$
F _ {1} (u) = F _ {1} (v) + A _ {v} (u - v) + B _ {v} (u - v), \quad \| B _ {v} (u - v) \| \leq C _ {0} \| u - v \| ^ {2},
$$
where, see (F.10),
$$
C _ {0} = 3 | D | ^ {1 / 2} \| k \| _ {C ^ {3} (D \times D \times \mathbb {R})} (1 + 2 C _ {S} R _ {2}) C _ {S} ^ {2},
$$
so that for  $u_{1},u_{2}\in \overline{B}_{H^{1}}(0,2R_{2})$
$$
\begin{array}{l} u _ {1} - u _ {2} - A _ {v _ {j}} ^ {- 1} \left(F _ {1} (u _ {1}) - F _ {1} (u _ {2})\right) \\ = u _ {1} - u _ {2} - A _ {u _ {2}} ^ {- 1} \left(F _ {1} \left(u _ {1}\right) - F _ {1} \left(u _ {2}\right)\right) - \left(A _ {u _ {2}} ^ {- 1} - A _ {v _ {j}} ^ {- 1}\right) \left(F _ {1} \left(u _ {1}\right) - F _ {1} \left(u _ {2}\right)\right), \\ \end{array}
$$
$$
\begin{array}{l} \left\| u _ {1} - u _ {2} - A _ {u _ {2}} ^ {- 1} \left(F _ {1} \left(u _ {1}\right) - F _ {1} \left(u _ {2}\right)\right) \right\| _ {H ^ {1} (D)} \\ = \left\| A _ {u _ {2}} ^ {- 1} \left(B _ {u _ {2}} \left(u _ {1} - u _ {2}\right)\right) \right\| _ {H ^ {1} (D)} \\ \leq \left\| A _ {u _ {2}} ^ {- 1} \right\| _ {H ^ {1} (D) \rightarrow H ^ {1} (D)} \left\| B _ {u _ {2}} \left(u _ {1} - u _ {2}\right)\right\| _ {H ^ {1} (D)} \\ \leq \left\| A _ {u _ {2}} ^ {- 1} \right\| _ {H ^ {1} (D) \rightarrow H ^ {1} (D)} C _ {0} \| u _ {1} - u _ {2} \| _ {H ^ {1} (D)} ^ {2}, \\ \leq C _ {B} C _ {0} \| u _ {1} - u _ {2} \| _ {H ^ {1} (D)} ^ {2}, \\ \end{array}
$$
$$
\begin{array}{l} \left\| \left(A _ {u _ {2}} ^ {- 1} - A _ {v _ {j}} ^ {- 1}\right) \left(F _ {1} (u _ {1}) - F _ {1} (u _ {2})\right) \right\| _ {H ^ {1} (D)} \\ \leq \left\| A _ {u _ {2}} ^ {- 1} - A _ {v _ {j}} ^ {- 1} \right\| _ {H ^ {1} (D) \rightarrow H ^ {1} (D)} \| F _ {1} (u _ {1}) - F _ {1} (u _ {2}) \| _ {H ^ {1} (D)} \\ \leq L i p _ {\overline {{B}} _ {H ^ {1}} (0, 2 R _ {2}) \rightarrow H ^ {1} (D)} (A _ {\cdot} ^ {- 1}) \| u _ {2} - v _ {j} \| L i p _ {\overline {{B}} _ {H ^ {1}} (0, 2 R _ {2}) \rightarrow H ^ {1} (D)} (F _ {1}) \| u _ {2} - u _ {1} \| _ {H ^ {1} (D)} \\ \leq C _ {A} \| u _ {2} - v _ {j} \| \left(C _ {B} + 4 C _ {0} R _ {2}\right) \| u _ {2} - u _ {1} \| _ {H ^ {1} (D)}, \\ \end{array}
$$
see (F.2), and hence, when  $\| u - v_{j} \| \leq r \leq R_{2}$
$$
\begin{array}{l} \left\| H _ {j} \left(u _ {1}\right) - H _ {j} \left(u _ {2}\right) \right\| _ {H ^ {1} (D)} \\ \leq \| u _ {1} - u _ {2} - A _ {v _ {j}} ^ {- 1} \left(F _ {1} (u _ {1}) - F _ {1} (u _ {2})\right) \| _ {H ^ {1} (D)} \\ \leq \| u _ {1} - u _ {2} - A _ {u _ {2}} ^ {- 1} \left(F _ {1} (u _ {1}) - F _ {1} (u _ {2})\right) \| _ {H ^ {1} (D)} + \| \left(A _ {u _ {2}} ^ {- 1} - A _ {v _ {j}} ^ {- 1}\right) \left(F _ {1} (u _ {1}) - F _ {1} (u _ {2})\right) \| _ {H ^ {1} (D)} \\ \leq \left(C _ {B} C _ {0} \left(\| u _ {1} - v _ {j} \| _ {H ^ {1} (D)} + \| u _ {2} - v _ {j} \| _ {H ^ {1} (D)}\right) + C _ {A} \left(C _ {B} + 4 C _ {0} R _ {2}\right) \| u _ {2} - v _ {j} \|\right) \| u _ {2} - u _ {1} \| _ {H ^ {1} (D)} \\ \leq C _ {H} r \| u _ {2} - u _ {1} \| _ {H ^ {1} (D)}, \\ \end{array}
$$
$$
C _ {H} = 2 C _ {B} C _ {0} + C _ {A} \left(C _ {B} + 4 C _ {0} R _ {2}\right).
$$
We now choose
$$
r = \min  (\frac {1}{2 C _ {H}}, R _ {2}).
$$
We consider
$$
\varepsilon_ {0} \leq \frac {1}{8 C _ {B}} \frac {1}{2 C _ {H}}.
$$
$$
r \geq 2 C _ {B} \varepsilon_ {0} / (1 - C _ {H} r).
$$
Then, we have that  $\mathrm{Lip}_{\overline{B}_{H^1}(0,2R_2)\to H^1 (D)}(H_j)\leq a = C_Hr < \frac{1}{2}$ , and
$$
r \geq \| A _ {v _ {j}} ^ {- 1} \| _ {H ^ {1} (D) \to H ^ {1} (D)} \| g - g _ {j} \| _ {H ^ {1} (D)} / (1 - a),
$$
and for all  $u \in \overline{B}_{H^1}(0, R_2)$  such that  $\| u - v_j\| \leq r$ , we have  $\| A_{v_j}^{-1}(g - g_j)\|_{H^1(D)} \leq (1 - a)r$ . Then,
$$
\begin{array}{l} \left\| H _ {j} (u) - v _ {j} \right\| _ {H ^ {1} (D)} \leq \left\| H _ {j} (u) - H _ {j} (v _ {j}) \right\| _ {H ^ {1} (D)} + \left\| H _ {j} (v _ {j}) - v _ {j} \right\| _ {H ^ {1} (D)} \\ \leq a \| u - v _ {j} \| _ {H ^ {1} (D)} + \| v _ {j} + A _ {v _ {j}} ^ {- 1} (g - g _ {j}) - v _ {j} \| _ {H ^ {1} (D)} \\ \leq a r + \left\| A _ {v _ {j}} ^ {- 1} (g - g _ {j}) \right\| _ {H ^ {1} (D)} \leq r, \\ \end{array}
$$
that is,  $H_{j}$  maps  $\overline{B}_{H^{1}(D)}(v_j,r)$  to itself. By Banach fixed point theorem,  $H_{j}:\overline{B}_{H^{1}(D)}(v_{j},r)\to$ $\overline{B}_{H^1 (D)}(v_j,r)$  has a fixed point.
$$
\mathcal {H} _ {j} \left( \begin{array}{c} u \\ g \end{array} \right) = \left( \begin{array}{c} H _ {j} ^ {g} (u) \\ g \end{array} \right) = \left( \begin{array}{c} u - A _ {v _ {j}} ^ {- 1} (F _ {1} (u) - F _ {1} (v _ {j})) + A _ {v _ {j}} ^ {- 1} (g - g _ {j}) \\ g \end{array} \right).
$$
By the above, when we choose  $\varepsilon_0$  to have a value
$$
\varepsilon_ {0} <   \frac {1}{8 C _ {B}} \frac {1}{2 C _ {H}},
$$
the map  $F_{1}$  has a right inverse map  $\mathcal{R}_j$  in  $B_{H^1}(g_j,2\varepsilon_0)$ , that is,
$$
F _ {1} \left(\mathcal {R} _ {j} (g)\right) = g, \quad \text {f o r} g \in B _ {H ^ {1}} \left(g _ {j}, 2 \varepsilon_ {0}\right), \tag {F.18}
$$
it holds that  $\mathcal{R}_j: B_{H^1}(g_j, 2\varepsilon_0) \to \overline{B}_{H^1(D)}(v_j, r)$ , and by Banach fixed point theorem it is given by the limit
$$
\mathcal {R} _ {j} (g) = \lim  _ {m \rightarrow \infty} w _ {j, m}, \quad g \in B _ {H ^ {1}} \left(g _ {j}, 2 \varepsilon_ {0}\right), \tag {F.19}
$$
in  $H^1 (D)$  , where
$$
w _ {j, 0} = v _ {j}, \tag {F.20}
$$
$$
w _ {j, m + 1} = H _ {j} ^ {g} \left(w _ {j, m}\right). \tag {F.21}
$$
We can write for  $g\in B_{H^1}(g_j,2\varepsilon_0)$
$$
\left( \begin{array}{c} \mathcal {R} _ {j} (g) \\ g \end{array} \right) = \lim  _ {m \to \infty} \mathcal {H} _ {j} ^ {\circ m} \left( \begin{array}{c} v _ {j} \\ g \end{array} \right),
$$
where the limit takes space in  $H^{1}(D)^{2}$  and
$$
\mathcal {H} _ {j} ^ {\circ m} = \mathcal {H} _ {j} \circ \mathcal {H} _ {j} \circ \dots \circ \mathcal {H} _ {j}, \tag {F.22}
$$
is the composition of  $m$  operators  $\mathcal{H}_j$ . This implies that  $\mathcal{R}_j$  can be written as a limit of finite iterations of neural operators  $H_j$  (we will consider how the operator  $A_{v_j}^{-1}$  can be written as a neural operator below).
As  $\mathcal{V} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$ , there are finite number of points  $y_{\ell} \in D, \ell = 1,2,\ldots,\ell_0$  and  $\varepsilon_1 > 0$  such that the sets
$$
Z (i _ {1}, i _ {2}, \dots , i _ {\ell_ {0}}) = \{g \in \mathcal {Y}: (i _ {\ell} - \frac {1}{2}) \varepsilon_ {1} \leq g (y _ {\ell}) <   (i _ {\ell} + \frac {1}{2}) \varepsilon_ {1}, \text {f o r a l l} \ell \},
$$
where  $i_1, i_2, \ldots, i_{\ell_0} \in \mathbb{Z}$ , satisfy the condition
(F.23)
If  $(Z(i_{1},i_{2},\ldots ,i_{\ell_{0}})\cap \mathcal{Y})\cap B_{H^{1}(D)}(g_{j},\varepsilon_{0})\neq \emptyset$  then  $Z(i_{1},i_{2},\dots ,i_{\ell_{0}})\cap \mathcal{Y}\subset B_{H^{1}(D)}(g_{j},2\varepsilon_{0})$
To show (F.23), we will below use the mean value theorem for function  $g = \sigma_{a} \circ v \in \mathcal{V}$ , where  $v \in C^{1,\alpha}(\overline{D})$ . First, let us consider the case when the parameter  $a$  of the leaky ReLU function  $\sigma_{a}$  is strictly positive. Without loss of generality, we can assume that  $D = [0,1]$  and  $y_{\ell} = h\ell$ , where
$h = 1 / \ell_0$  and  $\ell = 0,1,\ldots ,\ell_0$ . We consider  $g\in \mathcal{V}\cap Z(i_1,i_2,\dots ,i_{\ell_0})\subset \sigma_a(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$  of the form  $g = \sigma_{a}\circ v$ . As  $a$  is non-zero, the inequality  $(i_{\ell} - \frac{1}{2})\varepsilon \leq g(y_{\ell}) < (i_{\ell} + \frac{1}{2})\varepsilon$  is equivalent to  $\sigma_{1 / a}((i_{\ell} - \frac{1}{2})\varepsilon)\leq v(y_{\ell}) < \sigma_{1 / a}((i_{\ell} + \frac{1}{2})\varepsilon)$ , and thus
$$
\sigma_ {1 / a} \left(i _ {\ell} \varepsilon\right) - A \varepsilon \leq v \left(y _ {\ell}\right) <   \sigma_ {1 / a} \left(i _ {\ell} \varepsilon\right) + A \varepsilon , \tag {F.24}
$$
where  $A = \frac{1}{2}\max (1,a,1 / a)$ , that is, for  $g = \sigma_{a}(v)\in Z(i_{1},i_{2},\ldots ,i_{\ell_{0}})$  the values  $v(y_{\ell})$  are known within small errors. By applying mean value theorem on the interval  $[( \ell -1)h,\ell h]$  for function  $v$  we see that there is  $x^{\prime}\in [( \ell -1)h,\ell h]$  such that
$$
\frac {d v}{d x} (x ^ {\prime}) = \frac {v (\ell h) - v ((\ell - 1) h)}{h},
$$
and thus by (F.24),
$$
\left| \frac {d v}{d x} \left(x ^ {\prime}\right) - d _ {\ell , \vec {i}} \right| \leq 2 A \frac {\varepsilon_ {1}}{h}, \tag {F.25}
$$
$$
d _ {\ell , \vec {i}} = \frac {1}{h} \left(\sigma_ {1 / a} \left(i _ {\ell} \varepsilon_ {1}\right) - \sigma_ {1 / a} \left(\left(i _ {\ell} - 1\right) \varepsilon_ {1}\right)\right), \tag {F.26}
$$
Observe that these estimates are useful when  $\varepsilon_{1}$  is much smaller than  $h$ . As  $g = \sigma_{a} \circ v \in \mathcal{V} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$ , we have  $v \in \overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)$ , so that  $\frac{dv}{dx} \in \overline{B}_{C^{0,\alpha}(\overline{D})}(0,R)$  satisfies (F.25) implies that
$$
\left| \frac {d v}{d x} (x) - d _ {\ell , i} \right| \leq 2 A \frac {\varepsilon_ {1}}{h} + R h ^ {\alpha}, \quad \text {f o r a l l} x \in [ (\ell - 1) h, \ell h ]. \tag {F.27}
$$
Moreover, (F.24) and  $v \in \overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)$  imply
$$
\left| v (x) - \sigma_ {1 / a} \left(i _ {\ell} \varepsilon_ {1}\right) \right| <   A \varepsilon_ {1} + R h, \tag {F.28}
$$
for all  $x\in [(\ell -1)h,\ell h]$
Let  $\varepsilon_{2} = \varepsilon_{0} / A$ . When we first choose  $\ell_0$  to be large enough (so that  $h = 1 / \ell_0$  is small) and then  $\varepsilon_{1}$  to be small enough, we may assume that
$$
\max  \left(2 A \frac {\varepsilon_ {1}}{h} + R h ^ {\alpha}, A \varepsilon_ {1} + R h\right) <   \frac {1}{8} \varepsilon_ {2}. \tag {F.29}
$$
Then for any two functions  $g, g' \in \mathcal{Y} \cap Z(i_1, i_2, \ldots, i_{\ell_0}) \subset \sigma_a(\overline{B}_{C^{1,\alpha}(\overline{D})}(0, R))$  of the form  $g = \sigma_a \circ v, g' = \sigma_a \circ v'$  the inequalities (F.27) and (F.28) imply
$$
\left| \frac {d v}{d x} (x) - \frac {d v ^ {\prime}}{d x} (x) \right| <   \frac {1}{4} \varepsilon_ {2}, \tag {F.30}
$$
$$
\left| v (x) - v ^ {\prime} (x) \right| <   \frac {1}{4} \varepsilon_ {2},
$$
for all  $x\in D$  . As  $v,v^{\prime}\in \overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)$  , this implies
$$
\left\| v - v ^ {\prime} \right\| _ {C ^ {1} (\overline {{D}})} <   \frac {1}{2} \varepsilon_ {2},
$$
As the embedding  $C^1(\overline{D}) \to H^1(D)$  is continuous and has norm less than 2 on the interval  $D = [0,1]$ , we see that
$$
\left\| v - v ^ {\prime} \right\| _ {H ^ {1} (\overline {{D}})} <   \varepsilon_ {2},
$$
$$
\left\| g - g ^ {\prime} \right\| _ {H ^ {1} (\bar {D})} <   A \varepsilon_ {2} = \varepsilon_ {0}. \tag {F.31}
$$