least squares temporal difference methods an analysis under general conditions we consider approximate policy evaluation for finite state and action markov de cision processes mdp with the least squares temporal difference lstd algorithm lstd in an exploration enhanced learning context where policy costs are computed from observations of a markov chain different from the one corresponding to the policy under evaluation we establish for the discounted cost criterion that lstd converges almost surely under mild minimal conditions we also analyze other properties of the iterates involved in the algorithm including convergence in mean and boundedness our analysis draws on theories of both finite space markov chains and weak feller markov chains on a topological space our results can be applied to other temporal difference algorithms and mdp models as examples we give a convergence analysis of a td algorithm and extensions to mdp with compact state and action spaces as well as a convergence proof of a new lstd algorithm with state dependent parameters

the boundedness conditions for model free hdp lambda this paper provides the stability analysis for a model free action dependent heuristic dynamic programing hdp approach with an eligibility trace long term prediction parameter lambda hdp lambda learns from more than one future reward eligibility traces have long been popular in q learning this paper proves and demonstrates that they are worthwhile to use with hdp in this paper we prove its uniformly ultimately bounded uub property under certain conditions previous works present a uub proof for traditional hdp hdp lambda 0 but we extend the proof with the lambda parameter by using lyapunov stability we demonstrate the boundedness of the estimated error for the critic and actor neural networks as well as learning rate parameters three case studies demonstrate the effectiveness of hdp lambda the trajectories of the internal reinforcement signal nonlinear system are considered as the first case we compare the results with the performance of hdp and traditional temporal difference td lambda with different lambda values the second case study is a single link inverted pendulum we investigate the performance of the inverted pendulum by comparing hdp lambda with regular hdp with different levels of noise the third case study is a 3 d maze navigation benchmark which is compared with state action reward state action q lambda hdp and hdp lambda all these simulation results illustrate that hdp lambda has a competitive performance thus this contribution is not only uub but also useful in comparison with traditional hdp

the globally weak attraction and global semi stability of lienard system this paper obtains some necessary and sufficient conditions for the globally weak attraction and global semi stability of the zero solution for lienard equation x h y f x y g x

optimal designs for space time linear precoders and decoders we introduce a new paradigm for the design of transmitter space time coding that we refer to as linear precoding it leads to simple closed form solutions for transmission over frequency selective multiple input multiple output mimo channels which are scalable with respect to the number of antennas size of the coding block and transmit average peak power the scheme operates as a block transmission system in which vectors of symbols are encoded and modulated through a linear mapping operating jointly in the space and time dimension the specific designs target minimization of the symbol mean square error and the approximate maximization of the minimum distance between symbol hypotheses under average and peak power constraints the solutions are shown to convert the mimo channel with memory into a set of parallel flat fading subchannels regardless of the design criterion while appropriate power bits loading on the subchannels is the specific signature of the different designs the proposed designs are compared in terms of various performance measures such as information rate ber and symbol mean square error

blow up phenomena of second order nonlinear differential equations abstract this paper studies the blow up phenomena of the second order nonlinear differential equation x f t x x under appropriate conditions it is shown that some solutions of the equation blow up in finite time

stabilization of stochastic iterative methods for singular and nearly singular linear systems we consider linear systems of equations ax b with an emphasis on the case where a is singular under certain conditions necessary as well as sufficient linear deterministic iterative methods generate sequences xk that converge to a solution as long as there exists at least one solution this convergence property can be impaired when these methods are implemented with stochastic simulation as is often done in important classes of large scale problems we introduce additional conditions and novel algorithmic stabilization schemes under which xk converges to a solution when a is singular and may also be used with substantial benefit when a is nearly singular

the continuation of solutions for certain nonlinear differential equations and its applications we give some sufficient conditions for the continuation of solutions for some nonlinear differential equations as an application we obtain a new criterion for the oscillation of solutions of the lienard equation

decaying positive solutions of quasilinear elliptic equations in exterior domains in r2 abstract the existence of decaying positive solutions of quasilinear elliptic equations of the form u x u x u x 2 0 is established in an exterior domain r 2 under suitable smoothness and growth conditions the main result is proved by means of a supersolution subsolution method given by noussair and swanson by using phase plane analysis of a system of lienard type a supersolution and a subsolution of the above equation are found out an extension of the main result to more general case is also attempted finally some examples are attached

asymptotic performance analysis of coded blast architectures with statistical rate and power allocations in this paper we first analyze some mathematical properties of ergodic capacity and outage capacity functions of the layers in bell labs layered space time blast architectures employing successive decoding and interference cancellation we then present statistical rate allocation and power allocation methods that optimize the asymptotic performance of blast architectures since the methods are developed by using ergodic capacity and outage capacity functions of the layers the allocated rates and powers depend only on a given channel statistic finally we prove that the rate allocation yields a better asymptotic performance than the power allocation numerical results show that blast architectures with the rate and power allocation perform better by 4 db and 3 db respectively than a blast architecture with the same rate and power in all layers

optimal transmitter eigen beamforming and space time block coding based on channel mean feedback optimal transmitter designs obeying the water filling principle are well documented they are widely applied when the propagation channel is deterministically known and regularly updated at the transmitter because channel state information is impossible to be known perfectly at the transmitter in practical wireless systems we design in this paper an optimal multiantenna transmitter based on the knowledge of mean values of the underlying channels our optimal transmitter design turns out to be an eigen beamformer with multiple beams pointing to orthogonal directions along the eigenvectors of the correlation matrix of the estimated channel at the transmitter and with proper power loading across beams the optimality pertains to minimizing an upper bound on the symbol error rate which leads to better performance than maximizing the expected signal to noise ratio snr at the receiver coupled with orthogonal space time block codes two directional eigen beamforming emerges as a more attractive choice than conventional one directional beamforming with uniformly improved performance without rate reduction and without essential increase in complexity with multiple receive antennas and reasonably good feedback quality the two directional eigen beamformer is also capable of achieving the best possible performance in a large range of transmit power to noise ratios without a rate penalty

combining beamforming and orthogonal space time block coding multiple transmit and receive antennas can be used in wireless systems to achieve high data rate communication efficient space time codes have been developed that utilize a large portion of the available capacity these codes are designed under the assumption that the transmitter has no knowledge about the channel in this work on the other hand we consider the case when the transmitter has partial but not perfect knowledge about the channel and how to improve a predetermined code so that this fact is taken into account a performance criterion is derived for a frequency nonselective fading channel and then utilized to optimize a linear transformation of the predetermined code the resulting optimization problem turns out to be convex and can thus be efficiently solved using standard methods in addition a particularly efficient solution method is developed for the special case of independently fading channel coefficients the proposed transmission scheme combines the benefits of conventional beamforming with those given by orthogonal space time block coding simulation results for a narrow band system with multiple transmit antennas and one or more receive antennas demonstrate significant gains over conventional methods in a scenario with nonperfect channel knowledge

an emphatic approach to the problem of off policy temporal difference learning in this paper we introduce the idea of improving the performance of parametric temporal difference td learning algorithms by selectively emphasizing or de emphasizing their updates on different time steps in particular we show that varying the emphasis of linear td s updates in a particular way causes its expected update to become stable under off policy training the only prior model free td methods to achieve this with per step computation linear in the number of function approximation parameters are the gradient td family of methods including tdc gtd and gq compared to these methods our emphatic td is simpler and easier to use it has only one learned parameter vector and one step size parameter our treatment includes general state dependent discounting and bootstrapping functions and a way of specifying varying degrees of interest in accurately valuing different states

practical algorithms for a family of waterfilling solutions many engineering problems that can be formulated as constrained optimization problems result in solutions given by a waterfilling structure the classical example is the capacity achieving solution for a frequency selective channel for simple waterfilling solutions with a single waterlevel and a single constraint typically a power constraint some algorithms have been proposed in the literature to compute the solutions numerically however some other optimization problems result in significantly more complicated waterfilling solutions that include multiple waterlevels and multiple constraints for such cases it may still be possible to obtain practical algorithms to evaluate the solutions numerically but only after a painstaking inspection of the specific waterfilling structure in addition a unified view of the different types of waterfilling solutions and the corresponding practical algorithms is missing the purpose of this paper is twofold on the one hand it overviews the waterfilling results existing in the literature from a unified viewpoint on the other hand it bridges the gap between a wide family of waterfilling solutions and their efficient implementation in practice to be more precise it provides a practical algorithm to evaluate numerically a general waterfilling solution which includes the currently existing waterfilling solutions and others that may possibly appear in future problems

policy evaluation with temporal differences a survey and comparison policy evaluation is an essential step in most reinforcement learning approaches it yields a value function the quality assessment of states for a given policy which can be used in a policy improvement step since the late 1980s this research area has been dominated by temporal difference td methods due to their data efficiency however core issues such as stability guarantees in the off policy scenario improved sample efficiency and probabilistic treatment of the uncertainty in the estimates have only been tackled recently which has led to a large number of new approaches this paper aims at making these new developments accessible in a concise overview with foci on underlying cost functions the off policy scenario as well as on regularization in high dimensional feature spaces by presenting the first extensive systematic comparative evaluations comparing td lstd lspe fpkf the residual gradient algorithm bellman residual minimization gtd gtd2 and tdc we shed light on the strengths and weaknesses of the methods moreover we present alternative versions of lstd and lspe with drastically improved off policy performance

minimum ber block precoders for zero forcing equalization we determine the linear precoder that minimizes the bit error rate ber at moderate to high signal to noise ratios snrs for block transmission systems with zero forcing zf equalization and threshold detection the design is developed for the two standard schemes for eliminating inter block interference viz zero padding zp and cyclic prefix cp we show that both the zp minimum ber precoder and the cp minimum ber precoder provide substantially lower error rates than standard block transmission schemes such as orthogonal frequency division multiplexing ofdm the corresponding snr gains can be on the order of several decibels we also show that the cp minimum ber precoder can be obtained by a two stage modification of the water filling discrete multitone modulation dmt scheme in which the diagonal water filling power loading is replaced by a full matrix consisting of a diagonal minimum mean square error power loading matrix post multiplied by a discrete fourier transform dft matrix

comparison of cp based single carrier and ofdm with power allocation existing performance comparisons of single carrier sc and multicarrier transmission are mainly based on simulations in this letter we present a framework to compare the performance of cyclic prefix cp based sc block transmission and orthogonal frequency division multiplexing ofdm in frequency selective fading channels minimum mean squared error equal gain and approximate minimum bit error rate power allocation schemes for cp ofdm are considered performance of these methods is compared analytically and verified by simulations

weak convergence properties of constrained emphatic temporal difference learning with constant and slowly diminishing stepsize we consider the emphatic temporal difference td algorithm etd for learning the value functions of stationary policies in a discounted finite state and action markov decision process the etd algorithm was recently proposed by sutton mahmood and white 2016 to solve a long standing divergence problem of the standard td algorithm when it is applied to off policy training where data from an exploratory policy are used to evaluate other policies of interest the almost sure convergence of etd has been proved in our recent work under general off policy training conditions but for a narrow range of diminishing stepsize in this paper we present convergence results for constrained versions of etd with constant stepsize and with diminishing stepsize from a broad range our results characterize the asymptotic behavior of the trajectory of iterates produced by those algorithms and are derived by combining key properties of etd with powerful convergence theorems from the weak convergence methods in stochastic approximation theory for the case of constant stepsize in addition to analyzing the behavior of the algorithms in the limit as the stepsize parameter approaches zero we also analyze their behavior for a fixed stepsize and bound the deviations of their averaged iterates from the desired solution these results are obtained by exploiting the weak feller property of the markov chains associated with the algorithms and by using ergodic theorems for weak feller markov chains in conjunction with the convergence results we get from the weak convergence methods besides etd our analysis also applies to the off policy td algorithm when the divergence issue is avoided by setting sufficiently large it yields for that case new results on the asymptotic convergence properties of constrained off policy td with constant or slowly diminishing stepsize

homoclinic orbits in generalized lienard systems abstract this paper is devoted to the investigation on the existence of homoclinic orbits of the planar system of lienard type x h y f x y g x here h y is strictly increasing but is not imposed h sufficient conditions are given for a positive orbit of the system starting at a point on the curve h y f x to approach the origin without intersecting the x axis the obtained theorems include previous results as special cases our results are applied to a concrete system and their sharpness are improved

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

approximate minimum ber power allocation for mimo spatial multiplexing systems precoding for multiple input multiple output mimo spatial multiplexing generally requires high feedback overhead and or high complexity processing simultaneous reduction in transmitter complexity and feedback overhead is proposed by imposing a diagonal structural constraint to precoding i e power allocation minimum bit error rate mber is employed as the optimization criterion and an approximate mber amber power allocation algorithm is proposed for a variety of receivers including zero forcing zf successive interference cancellation sic and ordered sic osic while previously proposed precoding schemes either require zf equalization for mber or use a minimum mean squared error mmse criterion we provide a unified mber solution to power allocation for zf sic and osic receiver structures improved error rate performance is shown both analytically and by simulation simulation results also indicate that sic and osic with amber power allocation offer superior performance over previously proposed mber precoding with zf equalization as well as over mmse precoding decoding performance under noisy channels and power feedback is analyzed a modified amber algorithm that mitigates error propagation in interference cancellation is developed compared with existing precoding methods the proposed schemes significantly reduce both transmit processing complexity and feedback overhead and improve error rate performance

online model free n step hdp with stability analysis because of a powerful temporal difference td with lambda td lambda learning method this paper presents a novel n step adaptive dynamic programming adp architecture that combines td lambda with regular td learning for solving optimal control problems with reduced iterations in contrast with a backward view learning of td lambda that is required an extra parameter named eligibility traces to update at the end of each episode offline training the new design in this paper has forward view learning which is updated at each time step online training without needing the eligibility trace parameter in various applications without mathematical models therefore the new design is called the online model free n step action dependent ad heuristic dynamic programming nshdp lambda nshdp lambda has three neural networks the critic network cn with regular one step td td 0 the cn with n step td learning or td lambda and the actor network an because the forward view learning does not require any extra eligibility traces associated with each state the nshdp lambda architecture has low computational costs and is memory efficient furthermore the stability is proven for nshdp lambda under certain conditions by using lyapunov analysis to obtain the uniformly ultimately bounded uub property we compare the results with the performance of hdp and traditional action dependent hdp lambda adhdp lambda with different lambda values moreover a complex nonlinear system and 2 d maze problem are two simulation benchmarks in this paper and the third one is an inverted pendulum simulation benchmark which is presented in the supplemental material part of this paper nshdp lambda performance is examined and compared with other adp methods

global asymptotic stability of nonautonomous systems of lienard type this paper deals with nonautonomous lienard type systems sufficient conditions are given for the zero solution of the systems to be globally asymptotically stable the main result is proved by means of phase plane analysis with a liapunov function examples are included to contrast our theorem with results which were presented by hatvani and cantarelli some global phase portraits are also attached

on global asymptotic stability of systems of lienard type we give new necessary and sufficient conditions under which the zero solution of lienard type systems is globally asymptotically stable to this end we examine i whether all trajectories intersect the vertical isocline or not and ii whether all trajectories tend to the origin or not we also apply our results to a pseudolinear system

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location