SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Indeed,  $H_{N'} \in NO_{L,N'}(\sigma; D, d_{in}, d_{out})$ ,  $B$  is the linear finite rank operator with  $N'$  rank, and multiplication of two linear finite rank operators with  $N'$  rank is also a linear finite rank operator with  $N'$  rank.
Finally, we estimate for  $a \in K$ ,
$$
\begin{array}{l} \left\| G ^ {+} (a) - G _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \\ = \underbrace {\left\| G ^ {+} (a) - \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}}} _ {(D. 6) \leq \frac {\epsilon}{3}} + \underbrace {\left\| \widetilde {G} (a) - \widetilde {G} _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}}} _ {(D. 7) \leq \frac {\epsilon}{3}} + \left\| \widetilde {G} _ {N ^ {\prime}} (a) - G _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \tag {D.8} \\ \end{array}
$$
Using notation  $(a,\varphi_k)\in \mathbb{R}^{d_{in}}$ , and  $\mathbf{a} = ((a,\varphi_k))_{k\in [N]}\in \mathbb{R}^{Nd_{in}}$ , we further estimate for  $a\in K$
$$
\begin{array}{l} \left\| \widetilde {G} _ {N ^ {\prime}} (a) - G _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (Q) ^ {d _ {o u t}}} = \| \pi_ {d _ {o u t}} H _ {N ^ {\prime}} (a) - B \circ H _ {N ^ {\prime}} (a) \| _ {L ^ {2} (Q) ^ {d _ {o u t}}} \\ = \left\| \pi_ {N ^ {\prime} d _ {o u t}} \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) - \mathbf {B} \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) \right\| _ {2} \\ = \left\| \pi_ {N ^ {\prime} d _ {o u t}} \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) - \pi_ {N ^ {\prime} d _ {o u t}} \circ \mathbf {Q} \circ P _ {\mathbf {V} ^ {\perp}} \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) \right\| _ {2} \\ \leq \left\| \pi_ {N ^ {\prime} d _ {o u t}} \circ \left(P _ {\mathbf {V} _ {0} ^ {\perp}} - P _ {\mathbf {V} ^ {\perp}} + P _ {\mathbf {V} ^ {\perp}}\right) \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) - \pi_ {N ^ {\prime} d _ {o u t}} \circ \mathbf {Q} \circ P _ {\mathbf {V} ^ {\perp}} \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) \right\| _ {2} \tag {D.9} \\ \leq \left\| \pi_ {N ^ {\prime} d _ {o u t}} \circ \left(P _ {\mathbf {V} _ {0} ^ {\perp}} - P _ {\mathbf {V} ^ {\perp}}\right) \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) \right\| _ {2} + \| \pi_ {N ^ {\prime} d _ {o u t}} \circ (I - \mathbf {Q}) \circ P _ {\mathbf {V} ^ {\perp}} \circ \mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) \| _ {2} \\ \leq 5\epsilon_{0}\underbrace{\left\|\mathbf{H}_{N^{\prime}}(\mathbf{a})\right\|_{2}}_{= \| H_{N^{\prime}}(a)\|_{L^{2}(D)^{d_{out}}}<  C_{H}}\leq \frac{\epsilon}{3}, \\ \end{array}
$$
where  $\| \cdot \|_2$  is the Euclidean norm. Combining (D.8) and (D.9), we conclude that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - G _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \frac {\epsilon}{3} + \frac {\epsilon}{3} + \frac {\epsilon}{3} = \epsilon .
$$
E Details in Section 4.1
E.1 Proof of Proposition 3
Proof. Since  $W$  is bijective, and  $\sigma$  is surjective, it is enough to show that  $u \mapsto W u + K(u)$  is surjective. We observe that for  $z \in L^{2}(D)^{n}$ ,
$$
W u + K (u) = z,
$$
$$
H _ {z} (u) := - W ^ {- 1} K (u) + W ^ {- 1} z = u.
$$
We will show that  $H_{z}: L^{2}(D)^{n} \to L^{2}(D)^{n}$  has a fixed point for each  $z \in L^{2}(D)^{n}$ . By the Leray-Schauder theorem, see Gilbarg and Trudinger [2001, Theorem 11.3],  $H: L^{2}(D) \to L^{2}(D)$  has a fixed point if the union  $\bigcup_{0 < \lambda \leq 1} V_{\lambda}$  is bounded, where the sets
$$
\begin{array}{l} V _ {\lambda} := \{u \in L ^ {2} (D): u = \lambda H _ {z} (u) \} \\ = \left\{u \in L ^ {2} (D): \lambda^ {- 1} u = H _ {z} (u) \right\} \\ = \left\{u \in L ^ {2} (D): - \lambda^ {- 1} u = W ^ {- 1} K (u) - W ^ {- 1} z \right\}, \\ \end{array}
$$
are parametrized by  $0 < \lambda \leq 1$ .
As the map  $u \mapsto \alpha u + W^{-1}K(u)$  is coercive, there is an  $r > 0$  such that for  $\| u \|_{L^2(D)^n} > r$ ,
$$
\frac {\left\langle \alpha u + W ^ {- 1} K (u) , u \right\rangle_ {L ^ {2} (D) ^ {n}}}{\| u \| _ {L ^ {2} (D) ^ {n}}} \geq \| W ^ {- 1} z \| _ {L ^ {2} (D) ^ {n}}.
$$
Thus, we have that for  $\| u\|_{L^2 (D)^n} > r$
$$
\begin{array}{l} \frac {\left\langle W ^ {- 1} K (u) - W ^ {- 1} z , u \right\rangle_ {L ^ {2} (D) ^ {n}}}{\| u \| _ {L ^ {2} (D) ^ {n}} ^ {2}} \\ \geq \frac {\left\langle \alpha u + W ^ {- 1} K (u) , u \right\rangle_ {L ^ {2} (D) ^ {n}} - \left\langle \alpha u + W ^ {- 1} z , u \right\rangle_ {L ^ {2} (D) ^ {n}}}{\| u \| _ {L ^ {2} (D) ^ {n}} ^ {2}} \\ \geq \frac {\left\| W ^ {- 1} z \right\| _ {L ^ {2} (D) ^ {n}}}{\left\| u \right\| _ {L ^ {2} (D) ^ {n}}} - \frac {\left\langle W ^ {- 1} z , u \right\rangle_ {L ^ {2} (D) ^ {n}}}{\left\| u \right\| _ {L ^ {2} (D) ^ {n}} ^ {2}} - \alpha \geq - \alpha > - 1, \\ \end{array}
$$
and, hence, for all  $\| u\|_{L^2 (D)} > r_0$  and  $\lambda \in (0,1]$  we have  $u\notin V_{\lambda}$ . Thus
$$
\bigcup_ {\lambda \in (0, 1 ]} V _ {\lambda} \subset B (0, r _ {0}).
$$
Again, by the Leray-Schauder theorem (see Gilbarg and Trudinger [2001, Theorem 11.3]),  $H_{z}$  has a fixed point.
E.2 Examples for Proposition 3
Example 2. We consider the case where  $n = 1$  and  $D \subset \mathbb{R}^d$  is a bounded interval. We consider the non-linear integral operator,
$$
K (u) (x) := \int_ {D} k (x, y, u (x)) u (y) d y, x \in D,
$$
and  $k(x,y,t)$  is bounded, that is, there is  $C_K > 0$  such that
$$
| k (x, y, t) | \leq C _ {K}, x, y \in D, t \in \mathbb {R}.
$$
If  $\left\| W^{-1}\right\|_{\mathrm{op}}$  is small enough such that
$$
1 > \left\| W ^ {- 1} \right\| _ {\mathrm {o p}} C _ {K} | D |,
$$
then, for  $\alpha \in \left(\left\| W^{-1}\right\|_{\mathrm{op}}C_K|D|,1\right)$ ,  $u\mapsto \alpha u + W^{-1}K(u)$  is coercive. Indeed, we have for  $u\in L^{2}(D)$
$$
\begin{array}{l} \frac {\left\langle \alpha u + W ^ {- 1} K (u) , u \right\rangle_ {L ^ {2} (D)}}{\| u \| _ {L ^ {2} (D)}} \\ \geq \alpha \| u \| _ {L ^ {2} (D)} - \left\| W ^ {- 1} \right\| _ {\mathrm {o p}} \| K (u) \| _ {L ^ {2} (D)} \geq \underbrace {\left(\alpha - \left\| W ^ {- 1} \right\| _ {\mathrm {o p}} C _ {K} | D |\right)} _ {> 0} \| u \| _ {L ^ {2} (D)}. \\ \end{array}
$$
For example, we can consider a kernel
$$
k (x, y, t) = \sum_ {j = 1} ^ {J} c _ {j} (x, y) \sigma_ {s} \left(a _ {j} (x, y) t + b _ {j} (x, y)\right),
$$
where  $\sigma_s: \mathbb{R} \to \mathbb{R}$  is the sigmoid function defined by
$$
\sigma_ {s} (t) = \frac {1}{1 + e ^ {- t}}.
$$
$T$  are functions  $a,b,c\in C(\overline{D}\times \overline{D})$  such that
$$
\sum_ {j = 1} ^ {J} \left\| \mathcal {C} _ {j} \right\| _ {L ^ {\infty} (D \times D)} <   \left\| W ^ {- 1} \right\| _ {\mathrm {o p}} ^ {- 1} | D | ^ {- 1}.
$$
Example 3. Again, we consider the case where  $n = 1$  and  $D \subset \mathbb{R}^d$  is a bounded set. We assume that  $W \in C^{1}(\overline{D})$  satisfies  $0 < c_{1} \leq W(x) \leq c_{2}$ . For simplicity, we assume that  $|D| = 1$ . We consider the non-linear integral operator
$$
K (u) (x) := \int_ {D} k (x, y, u (x)) u (y) d y, x \in D, \tag {E.1}
$$
$$
k (x, y, t) = \sum_ {j = 1} ^ {J} c _ {j} (x, y) \sigma_ {\text {w i r e}} \left(a _ {j} (x, y) t + b _ {j} (x, y)\right), \tag {E.2}
$$
in which  $\sigma_{wire}:\mathbb{R}\to \mathbb{R}$  is the wavelet function defined by
$$
\sigma_ {w i r e} (t) = I m \left(e ^ {i \omega t} e ^ {- t ^ {2}}\right),
$$
and  $a_j, b_j, c_j \in C(\overline{D} \times \overline{D})$  are such that the  $a_j(x,y)$  are nowhere vanishing functions, that is,  $a_j(x,y) \neq 0$  for all  $x,y \in \overline{D} \times \overline{D}$ . The and its generalizations (e.g. activation functions which do decay only exponentially).
The next lemma holds for any activation function with exponential decay, including the activation function  $\sigma_{wire}$  and settles the key condition for Proposition 3 to hold.
Lemma 3. Assume that  $|D| = 1$  and the activation function  $\sigma : \mathbb{R} \to \mathbb{R}$  be continuous. Assume that there exists  $M_1, m_0 > 0$  such that
$$
| \sigma (t) | \leq M _ {1} e ^ {- m _ {0} | t |}, \quad t \in \mathbb {R}.
$$
Let  $a_j, b_j, c_j \in C(\overline{D} \times \overline{D})$  be such that  $a_j(x,y)$  are nowhere vanishing functions, that is,  $a_j(x,y) \neq 0$  for all  $x,y \in \overline{D} \times \overline{D}$ . Moreover, let  $K: L^2(D) \to L^2(D)$  be a non-linear integral operator given in (E.1) with a kernel satisfying (E.2). Let  $\alpha > 0$  and  $0 < c_0 \leq W(x) \leq c_1$ . Then function  $F: L^2(D) \to L^2(D)$ ,  $F(u) = \alpha u + W^{-1}K(u)$  is coercive.
Proof. As  $\overline{D}$  is compact, there is  $a_0 > 0$  such that for all  $j = 1,2,\ldots ,J$  we have  $|a_{j}(x,y)|\geq a_{0}$  a.e. and  $|b_{j}(x,y)|\leq b_{0}$  a.e. We point out that  $|\sigma (t)|\leq M_1$ . Next,
$$
\left(\sum_ {j = 1} ^ {J} \left\| W ^ {- 1} c _ {j} \right\| _ {L ^ {\infty} (D \times D)}\right) M _ {1} \varepsilon <   \frac {\alpha}{4}, \tag {E.3}
$$
we consider  $\lambda > 0$  and  $u \in L^{2}(D)$  and the sets
$$
D _ {1} (\lambda) = \{x \in D: | u (x) | \geq \varepsilon \lambda \},
$$
$$
D _ {2} (\lambda) = \{x \in D: | u (x) | <   \varepsilon \lambda \}.
$$
Let  $\varepsilon > 0$  be such that
$$
\left(\sum_ {j = 1} ^ {J} \left\| W ^ {- 1} c _ {j} \right\| _ {L ^ {\infty} (D \times D)}\right) \cdot M _ {1} \varepsilon <   \frac {\alpha}{4}.
$$
Then, for  $x \in D_2(\lambda)$ ,
$$
\begin{array}{l} \sum_ {j = 1} ^ {J} \left\| W ^ {- 1} c _ {j} \right\| _ {L ^ {\infty} (D \times D)} | \sigma \left(a _ {j} (x, y) u (x) + b _ {j} (x, y)\right) u (x) | \\ \leq \sum_ {j = 1} ^ {J} \| W ^ {- 1} c _ {j} \| _ {L ^ {\infty} (D \times D)} M _ {1} \epsilon \lambda_ {\leq (E. 3)} \frac {\alpha}{4} \lambda . \\ \end{array}
$$
After  $\varepsilon$  is chosen as in the above, we choose  $\lambda_0 \geq \max(1, b_0 / (a_0 \varepsilon))$  to be sufficiently large so that for all  $|t| \geq \varepsilon \lambda_0$  it holds
$$
(\sum_ {j = 1} ^ {J} \| W ^ {- 1} c _ {j} \| _ {L ^ {\infty} (D \times D)}) M _ {1} \mathrm {e x p} (- m _ {0} | a _ {0} t - b _ {0} |) t <   \frac {\alpha}{4}.
$$
Here, we observe that, as  $\lambda_0\geq b_0 / (a_0\varepsilon)$ , we have that for all  $|t|\geq \varepsilon \lambda_0$ ,  $a_0|t| - b_0 > 0$ . Then, when  $\lambda \geq \lambda_0$ , we have for  $x\in D_{1}(\lambda)$
$$
\left(\sum_ {j = 1} ^ {J} \left\| W ^ {- 1} c _ {j} \right\| _ {L ^ {\infty} (D \times D)}\right) \Bigg | \sigma \Big (a _ {j} (x, y) u (x) + b _ {j} (x, y) \Big) u (x) \Bigg | \leq \frac {\alpha}{4}.
$$
When  $u\in L^2 (D)$  has the norm  $\| u\|_{L^2 (D)} = \lambda \geq \lambda_0\geq 1$  , we have
$$
\begin{array}{l} \left| \int_ {D} \int_ {D} W (x) ^ {- 1} k (x, y, u (x)) u (x) u (y) d x d y \right| \\ \leq \int_ {D} \left(\int_ {D _ {1}} \left(\sum_ {j = 1} ^ {J} \| W ^ {- 1} c _ {j} \| _ {L ^ {\infty} (D \times D)}\right) M _ {1} \exp \left(- m _ {0} | a _ {0} | u (x) | - b _ {0} |\right) | u (x) | d x\right) | u (y) | d y \\ + \int_ {D} \left(\int_ {D _ {2}} \sum_ {j = 1} ^ {J} \| W ^ {- 1} c _ {j} \| _ {L ^ {\infty} (D \times D)} | \sigma \left(a _ {j} (x, y) u (x) + b _ {j} (x, y)\right) | | u (x) | d x\right) | u (y) | d y \\ \leq \frac {\alpha}{4} \| u \| _ {L ^ {2} (D)} + \frac {\alpha}{4} \lambda \| u \| _ {L ^ {2} (D)} \\ \leq \frac {\alpha}{2} \| u \| _ {L ^ {2} (D)} ^ {2}. \\ \end{array}
$$
$$
\frac {\left\langle \alpha u + W ^ {- 1} K (u) , u \right\rangle_ {L ^ {2} (D)}}{\| u \| _ {L ^ {2} (D)}} \geq \frac {\alpha}{2} \| u \| _ {L ^ {2} (D)},
$$
and the function  $u\to \alpha u + W^{-1}K(u)$  is coercive.
E.3 Proof of Proposition 4
Proof. (Injectivity) Assume that
$$
\sigma \left(W u _ {1} + K (u _ {1}) + b\right) = \sigma \left(W u _ {2} + K (u _ {2}) + b\right).
$$
where  $u_{1}, u_{2} \in L^{2}(D)^{n}$ . Since  $\sigma$  is injective and  $W: L^{2}(D)^{n} \to L^{2}(D)^{n}$  is bounded linear bijective, we have
$$
u _ {1} + W ^ {- 1} K (u _ {1}) = u _ {2} + W ^ {- 1} K (u _ {2}) =: z.
$$
Since the mapping  $u \mapsto z - W^{-1}K(u)$  is contraction (because  $W^{-1}K$  is contraction), by the Banach fixed-point theorem, the mapping  $u \mapsto z - W^{-1}K(u)$  admits a unique fixed-point in  $L^2(D)^n$ , which implies that  $u_1 = u_2$ .
(Surjectivity) Since  $\sigma$  is surjective, it is enough to show that  $u\mapsto Wu + K(u) + b$  is surjective. Let  $z\in L^{2}(D)^{n}$ . Since the mapping  $u\mapsto W^{-1}z - W^{-1}b - W^{-1}K(u)$  is contraction, by Banach fixed-point theorem, there is  $u^{*}\in L^{2}(D)^{n}$  such that
$$
u ^ {*} = W ^ {- 1} z - W ^ {- 1} b - W ^ {- 1} K \left(u ^ {*}\right) \Longleftrightarrow W u ^ {*} + K \left(u ^ {*}\right) + b = z.
$$
E.4 Examples for Proposition 4
Example 4. We consider the case of  $n = 1$ , and  $D \subset [0, \ell]^d$ . We consider Volterra operators
$$
K (u) (x) = \int_ {D} k (x, y, u (x), u (y)) u (y) d y,
$$
where  $x = (x_{1},\ldots ,x_{d})$  and  $y = (y_{1},\dots ,y_{d})$ . We recall that  $K$  is a Volterra operator if
$$
k (x, y, t, s) \neq 0 \Longrightarrow y _ {j} \leq x _ {j} \quad f o r a l l j = 1, 2, \dots , d. \tag {E.4}
$$
In particular, when  $D = (a,b) \subset \mathbb{R}$  is an interval, the Volterra operators are of the form
$$
K (u) (x) = \int_ {a} ^ {x} k (x, y, u (x), u (y)) u (y) d y,
$$
and if  $x$  is considered as a time variable, the Volterra operators are causal in the sense that the value of  $K(u)(x)$  at the time  $x$  depends only on  $u(y)$  at the times  $y \leq x$ .
Assume that  $k(x,y,t,s) \in C(\overline{D} \times \overline{D} \times \mathbb{R} \times \mathbb{R})$  is bounded and uniformly Lipschitz smooth in the  $t$  and  $s$  variables, that is,  $k \in C(\overline{D} \times \overline{D}; C^{0,1}(\mathbb{R} \times \mathbb{R}))$ .
Next, we consider the non-linear operator  $F: L^2(D) \to L^2(D)$ ,
$$
F (u) = u + K (u). \tag {E.5}
$$
Assume that  $u, w \in L^{2}(D)$  are such that  $u + K(u) = w + K(w)$ , so that  $w - u = K(u) - K(w)$ . Next, we will show that then  $u = w$ . We denote and  $D(z_{1}) = D \cap ([0,z_{1}] \times [0,\ell]^{d-1})$ . Then for  $x \in D(z_{1})$  the Volterra property of the kernel implies that
$$
\begin{array}{l} | u (x) - w (x) | \leq \int_ {D} | k (x, y, u (x), u (y)) u (y) - k (x, y, w (x), w (y)) w (y) | d y \\ \leq \int_ {D (z _ {1})} | k (x, y, u (x), u (y)) u (y) - k (x, y, w (x), u (y)) u (y) | d y \\ + \int_ {D (z _ {1})} | k (x, y, w (x), u (y)) u (y) - k (x, y, w (x), w (y)) u (y) | d y \\ + \int_ {D (z _ {1})} | k (x, y, w (x), w (y)) u (y) - k (x, y, w (x), w (y)) w (y) | d y \\ \leq 2 \| k \| _ {C (\overline {{D}} \times \overline {{D}}; C ^ {0, 1} (\mathbb {R} \times \mathbb {R}))} \| u - w \| _ {L ^ {2} (D (z _ {1}))} \| u \| _ {L ^ {2} (D (z _ {1}))} \\ + \| k \| _ {L ^ {\infty} \left(D \times D \times \mathbb {R} \times \mathbb {R}\right)} \| u - w \| _ {L ^ {2} \left(D \left(z _ {1}\right)\right)} \sqrt {\left| D \left(z _ {1}\right) \right|}, \\ \end{array}
$$
so that for all  $0 < z_{1} < \ell$
$$
\begin{array}{l} \left\| u - w \right\| _ {L ^ {2} (D (z _ {1}))} ^ {2} \\ = \int_ {0} ^ {z _ {1}} \left(\int_ {0} ^ {\ell} \dots \int_ {0} ^ {\ell} \mathbf {1} _ {D} (x) | u (x) - w (x) | ^ {2} d x _ {d} d x _ {d - 1} \dots d x _ {2}\right) d x _ {1} \\ \leq z _ {1} \ell^ {d - 1} \left(2 \| k \| _ {C (\overline {{D}} \times \overline {{D}}; C ^ {0, 1} (\mathbb {R} \times \mathbb {R}))} \| u - w \| _ {L ^ {2} (D (z _ {1}))} \| u \| _ {L ^ {2} (D (z _ {1}))} \right. \\ \left. + \| k \| _ {L ^ {\infty} \left(D \times D \times \mathbb {R} \times \mathbb {R}\right)} \| u - w \| _ {L ^ {2} \left(D \left(z _ {1}\right)\right)} \sqrt {| D \left(z _ {1}\right) |}\right) ^ {2} \\ \leq z _ {1} \ell^ {d - 1} \left(\| k \| _ {C (\overline {{D}} \times \overline {{D}}; C ^ {0, 1} (\mathbb {R} \times \mathbb {R}))} \| u \| _ {L ^ {2} (D)} + \| k \| _ {L ^ {\infty} (D \times D \times \mathbb {R} \times \mathbb {R})} \sqrt {| D |}\right) ^ {2} \| u - w \| _ {L ^ {2} (D (z _ {1}))} ^ {2}. \\ \end{array}
$$
Thus, when  $z_{1}$  is so small that
$$
z _ {1} \ell^ {d - 1} \bigg (\| k \| _ {C (\overline {{D}} \times \overline {{D}}; C ^ {0, 1} (\mathbb {R} ^ {n}))} \| u \| _ {L ^ {2} (D)} + \| k \| _ {L \infty (D \times D \times \mathbb {R} \times \mathbb {R})} \sqrt {| D |} \bigg) ^ {2} <   1,
$$
we find that  $\| u - w\|_{L^2 (D(z_1))} = 0$ , that is,  $u(x) - w(x) = 0$  for  $x\in D(z_1)$ . Using the same arguments as above, we see for all  $k\in \mathbb{N}$  that if  $u = w$  in  $D(kz_{1})$  then  $u = w$  in  $D((k + 1)z_{1})$ . Using induction, we see that  $u = w$  in  $D$ . Hence, the operator  $u\mapsto F(u)$  is injective in  $L^2 (D)$ .
Example 5. We consider derivatives of Volterra operators in the domain  $D \subset [0, \ell]^d$ . Let  $K: L^2(D) \to L^2(D)$  be a non-linear operator
$$
K (u) = \int_ {D} k (x, y, u (y)) u (y) d y, \tag {E.6}
$$
where  $k(x,y,t)$  satisfies (E.4), is bounded, and  $k\in C(\overline{D}\times \overline{D};C^{0,1}(\mathbb{R}\times \mathbb{R}))$ . Let  $F_{1}:L^{2}(D)\to L^{2}(D)$  be
$$
F _ {1} (u) = u + K (u). \tag {E.7}
$$
Then the Fréchet derivative of  $K$  at  $u_0 \in L^2(D)$  to the direction  $w \in L^2(D)$  is
$$
\left. D F _ {1} \right| _ {u _ {0}} (w) = w (x) + \int_ {D} k _ {1} (x, y, u _ {0} (y)) w (y) d y, \tag {E.8}
$$
$$
k _ {1} (x, y, u _ {0} (y)) = u _ {0} (y) \frac {\partial}{\partial t} k (x, y, t) \bigg | _ {t = u _ {0} (x)} + k (x, y, u _ {0} (y)) \tag {E.9}
$$
is a Volterra operator satisfying
$$
k _ {1} (x, y, t) \neq 0 \Longrightarrow y _ {j} \leq x _ {j} \quad f o r a l l j = 1, 2, \dots , d. \tag {E.10}
$$
As seen in Example 4, the operator  $DF_{1}|_{u_{0}}: L^{2}(D) \to L^{2}(D)$  is injective.
F Details in Section 4.2
In this appendix, we prove Theorem 3. We recall that in the theorem, we consider the case when  $n = 1$  and  $D \subset \mathbb{R}$  is a bounded interval, and the operator  $F_{1}$  is of the form
$$
\left\| k \right\| _ {C ^ {3} (\bar {D} \times \bar {D} \times \mathbb {R})} \leq c _ {0}, \quad \left\| W \right\| _ {C ^ {1} (\bar {D})} \leq c _ {0}. \tag {F.1}
$$
We recall that the identical embedding  $H^{1}(D) \to L^{\infty}(D)$  is bounded and compact by Sobolev's embedding theorem.
As we will consider kernels  $k(x,y,u_0(y))$ , we will consider the non-linear operator  $F_{1}$  mainly as an operator in a Sobolev space  $H^{1}(D)$ .
The Frechet derivative of  $F_{1}$  at  $u_{0}$  to direction  $w$ , denoted  $A_{u_0}w = DF_1|_{u_0}(w)$  is given by
$$
A _ {u _ {0}} w = W (x) w (x) + \int_ {D} k (x, y, u _ {0} (y)) w (y) d y + \int_ {D} u _ {0} (y) \frac {\partial k}{\partial u} (x, y, u _ {0} (y)) w (y) d y. \tag {F.2}
$$
The condition (F.1) implies that
$$
F _ {1}: H ^ {1} (D) \rightarrow H ^ {1} (D), \tag {F.3}
$$
is a locally Lipschitz smooth function and the operator
$$
A _ {u _ {0}}: H ^ {1} (D) \to H ^ {1} (D),
$$
given in (F.2), is defined for all  $u_0 \in C(\overline{D})$  as a bounded linear operator.
When  $\mathcal{X}$  is a Banach space, let  $B_{\mathcal{X}}(0,R) = \{v\in \mathcal{X}:\| v\|_{\mathcal{X}} < R\}$  and  $\overline{B}_{\mathcal{X}}(0,R) = \{v\in \mathcal{X}:\| v\|_{\mathcal{X}}\leq R\}$  be the open and closed balls in  $\mathcal{X}$ , respectively.
We consider the Hölder spaces  $C^{n,\alpha}(\overline{D})$  and their image in (leaky) ReLU-type functions. Let  $a \geq 0$  and  $\sigma_{a}(s) = \mathrm{ReLU}(s) - a\mathrm{ReLU}(-s)$ . We will consider the image of the closed ball of  $C^{1,\alpha}(\overline{D})$  in the map  $\sigma_{a}$ , that is  $\sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)) = \{\sigma_{a} \circ g \in C(\overline{D}) : \|g\|_{C^{1,\alpha}(\overline{D})} \leq R\}$ .
We will below assume that for all  $u_0 \in C(\overline{D})$  the integral operator
$$
A _ {u _ {0}}: H ^ {1} (D) \rightarrow H ^ {1} (D) \text {i s a n i n j e c t i v e o p e r a t o r .} \tag {F.4}
$$
This condition is valid when  $K(u)$  is a Volterra operator, see Examples 4 and 5. As the integral operators  $A_{u_0}$  are Fredholm operators having index zero, this implies that the operators (F.4) are bijective.
The inverse operator  $A_{u_0}^{-1}: H^1(D) \to H^1(D)$  can be written as
$$
A _ {u _ {0}} ^ {- 1} v (x) = \widetilde {W} (x) v (x) - \int_ {D} \widetilde {k} _ {u _ {0}} (x, y) v (y) d y, \tag {F.5}
$$
where  $\widetilde{k}_{u_0},\partial_x\widetilde{k}_{u_0}\in C(\overline{D}\times \overline{D})$  and  $\widetilde{W}\in C^{1}(\overline{D})$
We will consider the inverse function of the map  $F_{1}$  in a set  $\mathcal{Y} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$  that is a compact subset of the Sobolev space  $H^{1}(D)$ . To this end, we will cover the set  $\mathcal{Y}$  with small balls  $B_{H^1 (D)}(g_j,\varepsilon_0), j = 1,2,\ldots ,J$  of  $H^{1}(D)$ , centred at  $g_{j} = F_{1}(v_{j})$ , where  $v_{j}\in H^{1}(D)$ . We will show below that when  $g\in B_{H^1 (D)}(g_j,2\varepsilon_0)$ , that is,  $g$  is  $2\varepsilon_{1}$ -close to the function  $g_{j}$  in  $H^{1}(D)$ , the inverse map of  $F_{1}$  can be written as a limit  $(F_{1}^{-1}(g),g) = \lim_{m\to \infty}\mathcal{H}_{j}^{\circ m}(v_{j},g)$  in  $H^{1}(D)^{2}$ , where
$$
\mathcal {H} _ {j} \left( \begin{array}{c} u \\ g \end{array} \right) = \left( \begin{array}{c} u - A _ {v _ {j}} ^ {- 1} (F _ {1} (u) - F _ {1} (v _ {j})) + A _ {v _ {j}} ^ {- 1} (g - g _ {j}) \\ g \end{array} \right),
$$
To glue the local inverse maps together, we use a partition of unity in the function space  $\mathcal{V}$  that is given by integral neural operators
$$
\Phi_ {\vec {i}} (v, w) = \pi_ {1} \circ \phi_ {\vec {i}, 1} \circ \phi_ {\vec {i}, 2} \circ \dots \circ \phi_ {\vec {i}, \ell_ {0}} (v, w), \quad \mathrm {w h e r e} \quad \phi_ {\vec {i}, \ell} (v, w) = (F _ {y _ {\ell}, s (\vec {i}, \ell), \epsilon_ {1}} (v, w), w),
$$
where  $\vec{i}$  belongs in some finite index set  $\mathcal{I} \subset \mathbb{Z}^{\ell_0}$ , some  $\epsilon_1 > 0$ , some  $y_{\ell} \in D$  ( $\ell = 1, \dots, \ell_0$ ),  $s(\vec{i}, \ell) = i_{\ell} \epsilon_1 \cdot \pi_1(v, w) = v$  that maps a pair  $(v, w)$  to the first function  $v$  Here,  $F_{z,s,h}(v, w)$  are integral neural operators with distributional kernels
$$
F _ {z, s, h} (v, w) (x) = \int_ {D} k _ {z, s, h} (x, y, v (x), w (y)) d y,
$$
where  $k_{z,s,h}(x,y,v(x),w(y)) = v(x)\mathbf{1}_{[s - \frac{1}{2} h,s + \frac{1}{2} h)}(w(y))\delta (y - z)$ , and  $\mathbf{1}_A$  is the indicator function of a set  $A$  and  $y \mapsto \delta (y - z)$  is the Dirac delta distribution at the point  $z \in D$ . Using these, we can write the inverse of  $F_{1}$  at  $g \in \mathcal{V}$  as
$$
F _ {1} ^ {- 1} (g) = \lim  _ {m \rightarrow \infty} \sum_ {\vec {i} \in \mathcal {I}} \Phi_ {\vec {i}} \mathcal {H} _ {j (\vec {i})} ^ {\circ m} \left(\begin{array}{c}v _ {j (\vec {i})}\\g\end{array}\right), \tag {F.6}
$$
where  $j(\vec{i}) \in \{1,2,\dots,J\}$  are suitably chosen and the limit is taken in the norm topology of  $H^{1}(D)$ . This result is summarized in the following theorem, that is a modified version of Theorem 3 for the inverse operator  $F_{1}^{-1}$  in (F.6) where we have refined the partition of unity  $\Phi_{\vec{i}}$  so that we use indexes  $\vec{i} \in \mathcal{I} \subset \mathbb{Z}^{\ell_0}$  instead of  $j \in \{1,\dots,J\}$ .
This result is summarized in following theorem:
Theorem 4. Assume that  $F_{1}$  satisfies the above assumptions (F.1) and (F.4) and that  $F_{1}:H^{1}(D)\to H^{1}(D)$  is a bijection. Let  $\mathcal{V}\subset \sigma_a(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$  be a compact subset of the Sobolev space  $H^{1}(D)$ , where  $\alpha >0$  and  $a\geq 0$ . Then the inverse of  $F_{1}:H^{1}(D)\to H^{1}(D)$  in  $\mathcal{V}$  can written as a limit (F.6) that is, as a limit of integral neural operators.