## 3 Trust-Region Reflective (TRR) Search  
### Trust-Region Method  
The trust-region method is a conceptually simple yet powerful and derivative-dependent algorithm for solving nonlinear least-squares problems in the form of Equation (9), which is a perfect match for the objective function Equation (4). \[F(X)=\min_{X\in[\text{LB,UB}]}\Big{[}f_{1}(X)^{2}+f_{2}(X)^{2}+\dots+f_{N}(X) ^{2}\Big{]} \tag{9}\]  
The essential idea behind the trust-region method is to approximate \(F(X)\) with a simpler quadratic function \(q\)(s) in a small neighborhood \(N_{t}\) around the current point \(X\). This neighborhood is the trust region, whose shape ideally is spherical rather than ellipsoidal. By minimizing the trust-region sub-problem: \(\min\{q(s),\,s\in N_{t}\}\), a trial step \(s\) of current point \(X\) can be obtained, which indicates the direction to reduce \(F(X)\). If \(F(X+s)<F(X)\), the current point \(X\) is updated to be \(X+s\); otherwise, \(X\) remains unchanged while the trust region radius \(\Delta\) is shrunk for the next iteration. Hence, the key steps of the trust-region method [31] are:  
(1) Solve the trust-region sub-problem Equation (10) for determining a trial step \(s\). \[\min\{q(s):\,\|Ds\|\leq\Delta\} \tag{10}\]  
(2) Truncate the trial step \(s\) with Equation (11) to ensure \(X+s\) is strictly feasible. \[\alpha=\min(1,\;\theta\cdot mdis) \tag{11}\]
3. Compute the quadratic approximate function \(q(s)\) with Equation (12), where the first two terms of the Taylor expansion are employed to approximate \([F(X+s)-F(X)]/2\). \[q(s)=g^{T}s+\frac{1}{2}s^{T}(H+C)s\] (12)
4. Adjust the trust-region radius \(\Delta\) according to Equation (13) and update the current point \(X\) to be \(X+s\) if \(F(X+s)<F(X)\). \[\rho=\frac{0.5[F(X+s)-F(X)]+0.5s^{T}Cs}{q(s)}\] (13) where \(\|.\|\) is the Euclidean 2-norm and \(D\) is the diagonal affine scaling matrix defined by Equation (14), which is used to prevent \(s\) directly toward a boundary point. \(\alpha\) is the step size toward the boundary along \(s\), \(\theta\) is defined by Equation (15) representing the correction of \(\Delta\) and implies a potential step-back to ensure \(\Delta\) is strictly feasible, while _mdis_ is defined by Equation (16) and denotes the min-max breakpoint distances to the boundary. \(g\) is the gradient of error function \(f(X)\) and superscript \({}^{T}\) denotes the transpose operation. \(H\) is the Hessian matrix and can be approximated with the symmetric matrix of Equation (17). \(C\) is the affine scaling matrix defined by Equation (18). \(\rho\) measures the agreement between \(F(X)\) and its quadratic approximation \(q(s)\). Since \(F(X)\) is calculated by Equation (9), to match with \(q(s)\) in Equation (12), the coefficient of 0.5 in Equation (13) is necessary while the term of \(0.5s^{T}Cs\) is a natural extension to the bound-constrained problem [31]. As for how to update \(\Delta\) according to \(\rho\), the empirical threshold values are formulated by Equation (19). \[D=\text{diag}(|v|_{-0.5})\] (14) \[\theta=\max(0.95,\;1-||v\cdot g||_{\infty})\] (15) \[mdis=\min\biggl{[}\max\biggl{(}\frac{UB-X}{s},\frac{LB-X}{s}\biggr{)}\biggr{]}\] (16) \[H=J^{T}J\] (17) \[C=D\text{diag}(g)I_{v}D\] (18) \[\Delta=\left\{ \begin{array}{ll}2\Delta,&\text{if }\rho\geq 0.75\text{ and }|| Ds||\geq 0.9\Delta\\ \Delta,&\text{if }0.25<\rho<0.75\\ \min(\Delta/4,\;||Ds||/4),&\text{if }\rho\leq 0.25\end{array} \right.\] (19)  
where \(v\) denotes the distances to the boundary and can be computed by Equation (20). \(J\) denotes the Jacobian matrix estimated usually by the dense forward finite differences of error function \(f(X)\). \(J_{v}\) is a diagonal matrix corresponding to the Jacobian of \(\;1v\,\), with all diagonal components of \(I_{v}\) equal to 0 or \(\pm 1\). If all the components of LB and UB are finite, \(J_{v}=\text{diag(sign(g))}\). In particular, if \(v(i)=0\), the \(i\)th diagonal component of \(I_{v}\) equals 1.