The temporal-difference methods TD( @math ) and Sarsa( @math ) form a core part of modern reinforcement learning. Their appeal comes from their good performance, low computational cost, and their simple interpretation, given by their forward view. Recently, new versions of these methods were introduced, called true online TD( @math ) and true online Sarsa( @math ), respectively (van Seijen & Sutton, 2014). These new versions maintain an exact equivalence with the forward view at all times, whereas the traditional versions only approximate it for small step-sizes. We hypothesize that these true online methods not only have better theoretical properties, but also dominate the regular methods empirically. In this article, we put this hypothesis to the test by performing an extensive empirical comparison. Specifically, we compare the performance of true online TD( @math ) Sarsa( @math ) with regular TD( @math ) Sarsa( @math ) on random MRPs, a real-world myoelectric prosthetic arm, and a domain from the Arcade Learning Environment. We use linear function approximation with tabular, binary, and non-binary features. Our results suggest that the true online methods indeed dominate the regular methods. Across all domains representations the learning speed of the true online methods are often better, but never worse than that of the regular methods. An additional advantage is that no choice between traces has to be made for the true online methods. Besides the empirical results, we provide an in-depth analysis of the theory behind true online temporal-difference learning. In addition, we show that new true online temporal-difference methods can be derived by making changes to the online forward view and then rewriting the update equations.
Furthermore, @cite_0 derived backward views for a series of increasingly complex forward views. The derivation of the true online TD( @math ) equations in Appendix is similar to those derivations.
@cite_0
2200297111
We provide sharp worst-case evaluation complexity bounds for nonconvex minimization problems with general inexpensive constraints, i.e. problems where the cost of evaluating enforcing of the (possibly nonconvex or even disconnected) constraints, if any, is negligible compared to that of evaluating the objective function. These bounds unify, extend or improve all known upper and lower complexity bounds for unconstrained and convexly-constrained problems. It is shown that, given an accuracy level @math , a degree of highest available Lipschitz continuous derivatives @math and a desired optimality order @math between one and @math , a conceptual regularization algorithm requires no more than @math evaluations of the objective function and its derivatives to compute a suitably approximate @math -th order minimizer. With an appropriate choice of the regularization, a similar result also holds if the @math -th derivative is merely H "older rather than Lipschitz continuous. We provide an example that shows that the above complexity bound is sharp for unconstrained and a wide class of constrained problems, we also give reasons for the optimality of regularization methods from a worst-case complexity point of view, within a large class of algorithms that use the same derivative information. Let us consider the differential equation @math with an @math from @math to @math and suppose that there exists a transformation @math from @math to @math ( @math ) such that @math obeys a differential equation @math with some function @math ; then the first equation is said to be lumpable to the second by @math . Here mainly the case is investigated when the original differential equation has been induced by a complex chemical reaction. We provided a series of necessary and sufficient conditions for the existence of such functions @math and @math ; some of them are formulated in terms of @math and @math only. Beyond these conditions our main concern here is how lumping changes those properties of the solutions which are either interesting from the point of view of the qualitative theory of differential equations or from the point ... We extend the De Giorgi--Nash--Moser theory to a class of kinetic Fokker-Planck equations and deduce new results on the Landau-Coulomb equation. More precisely, we first study the H o lder regularity and establish a Harnack inequality for solutions to a general linear equation of Fokker-Planck type whose coefficients are merely measurable and essentially bounded, i.e. assuming no regularity on the coefficients in order to later derive results for non-linear problems. This general equation has the formal structure of the hypoelliptic equations "of type II" , sometimes also called ultraparabolic equations of Kolmogorov type, but with rough coefficients: it combines a first-order skew-symmetric operator with a second-order elliptic operator involving derivatives along only part of the coordinates and with rough coefficients. These general results are then applied to the non-negative essentially bounded weak solutions of the Landau equation with inverse-power law @math @math [--d, 1] whose mass, energy and entropy density are bounded and mass is bounded away from 0, and we deduce the H o lder regularity of these solutions. We investigate the asymptotic number of elements of size @math in a particular class of closed lambda-terms (so-called @math -terms) which are related to axiom systems of combinatory logic. By deriving a differential equation for the generating function of the counting sequence we obtain a recurrence relation which can be solved asymptotically. We derive differential equations for the generating functions of the counting sequences of other more general classes of terms as well: the class of @math -terms and that of closed lambda-terms. Using elementary arguments we obtain upper and lower estimates for the number of closed lambda-terms of size @math . Moreover, a recurrence relation is derived which allows an efficient computation of the counting sequence. @math -terms are discussed briefly.