## 5 Discussion
### Numerical Experiment with the Quotient Rule: The Hard Case  
The most interesting results were obtained when the composite objective function \(F\) had the form \(F=\frac{f_{1}}{f_{2}}\), and \(f_{2}\) had a real root near the box constraint \([\ell,u]\). Figures 8-11 clearly show that the calculus-based approach was significantly more efficient and robust than the non-calculus-based approach for these four situations. Note that the composite objective function \(F\) was twice continuously differentiable on the box constraint, but \(F\) was not twice continuously-differentiable on the expanded box constraint (6). The composite objective function \(F\) was also not twice continuously differentiable on \(\ell+\Delta_{\mathrm{s\,max}}\mathbf{1}_{n}\leq xu+\Delta_{\mathrm{s\,max}} \mathbf{1}_{n}\). Using the non-calculus-based approach, it could be the case that \(g^{k}\) or \(H^{k}\) was undefined if one of the sample points landed exactly on a real root of \(f_{2}\). This was very unlikely and it did not happen during the experiment. When using a calculus-based approach, this issue cannot occur, as \(f_{1}\) and \(f_{2}\) are twice continuously differentiable everywhere. This is clearly an advantage of the calculus-based approach. If a sample point is near a root of \(f_{2}\), the value of \(F\) at this point can be a very large number. This could make the accuracy of \(g^{k}\) and \(H^{k}\) very poor when using the non-calculus-based approach. Note that the Lipschitz constants of \(\
abla F\) and \(\
abla^{2}F\) on the box constraint were very large numbers. Hence, the error bounds associated with the approximate gradient \(g^{k}\) and the approximate Hessian \(H^{k}\) were very large numbers (see [59]). This tells us that it is possible to obtain very bad approximations for \(g^{k}\) and \(H^{k}\) when using the non-calculus-based approach. To see how the non-calculus-based approach can fail to provide accurate approximations of the gradient and Hessian, we present Example 2.  
easy to create a test set of problems in which the differences in performance between the calculus-based approach and the non-calculus-based approach are more pronounced than the results presented in this paper (Figures 1-3), which are in favor of the calculus-based approach. However, the gain in performance is not as drastic as the results obtained with the quotient rule in our third experiment (Figures 8-11). Further investigation is needed to determine situations in which the calculus-based approach significantly outperforms the non-calculus-based approach when using the product rule. We remark that our implementation of Algorithm 1 is simple. This is intentional, as the goal is to compare the calculus-based approach with the non-calculus-based approach. Nonetheless, we remark that, recently, [PERSON] et al. found that a DFTR algorithm that used quadratic models built from a forward-finite-difference gradient and a forward-finite-difference Hessian can be surprisingly competitive with state-of-the-art derivative-free algorithms [64]. From our perspective, now that the calculus-based approach has been established as the superior method within a model-based derivative-free algorithm, the next logical step is to use this knowledge to improve current state-of-the-art software (e.g., [65]). Applying a calculus-based approach to real-world optimization problems is, then, the obvious direction of future research. Another direction to explore is inspired by model-based methods for high-dimensional blackbox optimization [66]. In [66], the authors used subspace decomposition to reduce a high-dimensional blackbox optimization problem into a sequence of low-dimensional blackbox optimization problems. At each iteration, the subspace was rotated, requiring new models to be frequently constructed. An examination of whether calculus rules could be adapted to help in this situation would be an interesting direction for future research. This further links to a direction of future research based on _underdetermined gradient/Hessian approximations_.