SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {C} _ {j} = \left\{ \begin{array}{l l} C _ {j} / A _ {c o l, j}, & A _ {c o l, j} \neq 0 \\ 0, & A _ {c o l, j} = 0 \end{array} , \right. \quad \text {a n d} \quad \beta = \sum_ {j | A _ {c o l, j} = 0} C _ {j} \sigma (b _ {j})
$$
$$
\begin{array}{l} \sup  _ {x \in K} \| f (x) - C \cdot (\sigma \circ (A \cdot x + b)) \| \\ = \sup  _ {x \in K} \left\| f (x) - \sum_ {j = 1} ^ {k} C _ {j} \left(\sigma \left(A _ {j} \cdot x + b _ {j}\right)\right) \right\| \\ = \sup  _ {x \in K} \left\| f (x) - \sum_ {j = 1} ^ {k} \tilde {C} _ {j} \left(\sigma \left(A _ {j} \cdot x + b _ {j}\right) A _ {c o l, j}\right) - \sum_ {j \mid A _ {c o l, j} = 0} C _ {j} \sigma \left(b _ {j}\right) \right\| \\ = \sup  _ {x \in K} \| f (x) - \tilde {C} \cdot \left(\sigma^ {\prime} \circ (A \cdot x + b) \circ A _ {c o l}\right) - \beta \|, \\ \end{array}
$$
Note that  $\tilde{C} \cdot (\sigma' \circ (A \cdot x + b) \circ A_{col}) - \beta = \frac{d}{dx_{col}}\left(\tilde{C} \cdot (\sigma \circ (Ax + b)) - \beta x\right)$ , which is the derivative of a two-layer feedforward integral network.
F Relationship between Number of ProdNets and Model Expressivity
Figure 9: Training MSE for fitting a positive derivative network to  $\sin (x)\cos (y)\sin (z) + 1$
We applied ten summations of positive ProdNets to fit the non-multiplicative-decomposable function  $\sin (x)\cos (y)\sin (z) + 1$ .
Each of these ProdNets consists of three MLP components, each with two hidden layers with 128 dimensions. All models underwent training with a consistent learning rate set at 0.005.
Our results, shown in Figure 9, indicate that increasing the number of ProdNets generally improves the model's performance in fitting the non-decomposable function. The model with the best MSE
uses 10 ProdNets, while the model with 2 ProdNets had the second-worst performance. This result is intuitively sensible, as more linear terms are typically required to express an arbitrary function precisely. However, we observed that employing more ProdNets does not always lead to better performance, as demonstrated by the model with 8 ProdNets.