# Compensating Data Shortages in Manufacturing with Monotonicity Knowledge
## 2 Semi-Infinite Optimization Approach to Monotonic Regression
### Semi-Infinite Optimization Formulation of Monotonic Regression  
In our approach to monotonic regression, polynomial models  
\[\mathbf{x}\mapsto\widehat{y}_{\mathbf{w}}(\mathbf{x})=\sum_{|\alpha|\leq m}w_{\alpha}\mathbf{ x}^{\alpha}\in\mathbb{R} \tag{1}\]  
are used for all input-output relationships \(\mathbf{x}\mapsto y(\mathbf{x})\) to be learned. In the above relation (1), the sum extends over all \(d\)-dimensional multi-indices ([30], Section 1) \(\alpha=(\alpha_{1},\ldots,\alpha_{d})\in\mathbb{N}_{0}^{d}\) with degree \(|\alpha|:=\alpha_{1}+\cdots+\alpha_{d}\) less than or equal to some total degree \(m\in\mathbb{N}\). The terms \(\mathbf{x}^{\alpha}:=x_{1}^{\alpha_{1}}\cdots x_{d}^{\alpha_{d}}\) are the monomials in \(d\) variables of degrees less than or equal to \(m\), and \(w_{\alpha}\) are the corresponding model parameters to be tuned by regression. Since there are exactly  
\[N_{m}=\sum_{k=0}^{m}\binom{k+d-1}{d-1}=\binom{m+d}{m} \tag{2}\]  
\(d\)-dimensional monomials of degrees less than or equal to \(m\), the polynomial regression model (1) can be equivalently written as  
\[\widehat{y}_{\mathbf{w}}(\mathbf{x})=\sum_{i=1}^{N_{m}}w_{i}\phi_{i}(\mathbf{x})=\mathbf{w}^{ \top}\mathbf{\phi}(\mathbf{x}), \tag{3}\]  
where the basis functions \(\phi_{1},\ldots,\phi_{N_{m}}\) constitute any enumeration of the \(d\)-dimensional monomials of degrees less than or equal to \(m\), while \(\mathbf{w}:=(w_{1},\ldots,w_{N_{m}})^{\top}\) and \(\mathbf{\phi}(\mathbf{x}):=(\phi_{1}(\mathbf{x}),\ldots,\phi_{N_{m}}(\mathbf{x}))^{\top}\). Standard polynomial regression without regularization ([31], Section 2.1) is about solving the unconstrained optimization problem:  
\[\min_{\mathbf{w}\in\mathbb{W}}\frac{1}{2}\sum_{l=1}^{N}\big{(}\widehat{y}_{\mathbf{w} }(\mathbf{x}_{l})-t_{l}\big{)}^{2}, \tag{4}\]  
or in other words, about optimally adapting the model parameters \(w_{i}\in[-r,r]\) of the polynomial model (3) to the available dataset \(\mathcal{D}=\{(\mathbf{x}_{l},t_{l}):l\in\{1,\ldots,N\}\}\) containing\(N\) points. In the above relation, the monomial coefficients are allowed to vary in the compact hyperbox  
\[W=\{\mathbf{w}\in\mathbb{R}^{N_{m}}:-r\leq w_{i}\leq r\text{ for all }i\in\{1,\ldots,N_{m}\}\} \tag{5}\]  
with some large but finite \(r>0\). Since \(W\) is compact and non-empty, and since the mean-squared error objective function of (4) is continuous, the standard polynomial regression problem (4) for any given dataset \(\mathcal{D}\) has a solution \(\mathbf{w}\) (which is unique if, for instance, an \(\ell^{2}\)-regularization term is added). In general, however, the resulting model \(\mathbf{x}\mapsto\widehat{y}_{\mathbf{w}}(\mathbf{x})\) does not necessarily exhibit the monotonicity behavior an expert expects for the underlying true physical relationship \(\mathbf{x}\mapsto y(\mathbf{x})\). In order to enforce the expected monotonicity behavior, the following constraints on the signs of the partial derivatives \(\partial_{x_{j}}\widehat{y}_{\mathbf{w}}(\mathbf{x})\) are added to the unconstrained standard regression problem (4):  
\[\sigma_{j}\cdot\partial_{x_{j}}\widehat{y}_{\mathbf{w}}(\mathbf{x})\geq 0\quad\text{ for all }j\in J\text{ and }\mathbf{x}\in X. \tag{6}\]  
The numbers \(\sigma_{j}\in\{-1,0,1\}\) indicate the expected monotonicity behavior for each coordinate direction \(j\in\{1,\ldots,d\}\):  
* \(\sigma_{j}=1\) and \(\sigma_{j}=-1\) indicate that \(\mathbf{x}\mapsto y(\mathbf{x})\) is expected to be, respectively, monotonically increasing or decreasing in the \(j\)th coordinate direction;
* \(\sigma_{j}=0\) indicates that one has no monotonicity knowledge in the \(j\)th coordinate direction. Also, \(J:=\{j\in\{1,\ldots,d\}:\sigma_{j}\
eq 0\}\) is the set of all directions for which a monotonicity constraint is imposed, and the vector  
\[\mathbf{\sigma}:=(\sigma_{1},\ldots,\sigma_{d})\]  
is referred to as the monotonicity signature of the relationship \(\mathbf{x}\mapsto y(\mathbf{x})\).