on the global convergence of a class of functional differential equations with applications in neural network theory abstract we study a system of retarded functional differential equations which generalise both the hopfield neural network model as well as hybrid network models of the cellular neural network type our main results give sufficient conditions for the global asymptotic stability of such systems and are milder than previously known conditions for the hybrid models when specialised to neural networks our models allow us to consider several different types of activation functions including piecewise linear sigmoids and unbounded activations as well as the usualc smooth sigmoids these issues are vital in the applications we also study neural network models with nonconstant delaysr t

asymptotic enumeration of rna secondary structure based on the new representation of rna secondary structures we obtain the basic relations about secondary structures with a prescribed size m for hairpin loops and minimum stack length l furthermore we make an asymptotic analysis on rna secondary structures with certain additional constrains

moduli spaces and macromolecules techniques from moduli spaces are applied to biological macromolecules the first main result provides new a priori constraints on protein geometry discovered empirically and confirmed computationally the second main result identifies up to homotopy the natural moduli space of several interacting rna molecules with the riemann moduli space of a surface with several boundary components in each fixed genus applications to rna folding prediction are discussed the mathematical and biological frameworks are surveyed and presented from first principles

model free monte carlo like policy evaluation we propose an algorithm for estimating the finite horizon expected return of a closed loop control policy from an a priori given off policy sample of one step transitions it averages cumulated rewards along a set of broken trajectories made of one step transitions selected from the sample on the basis of the control policy under some lipschitz continuity assumptions on the system dynamics reward function and control policy we provide bounds on the bias and variance of the estimator that depend only on the lipschitz constants on the number of broken trajectories used in the estimator and on the sparsity of the sample of one step transitions

empirical dynamic programming we propose empirical dynamic programming algorithms for markov decision processes in these algorithms the exact expectation in the bellman operator in classical value iteration is replaced by an empirical estimate to get empirical value iteration evi policy evaluation and policy improvement in classical policy iteration are also replaced by simulation to get empirical policy iteration epi thus these empirical dynamic programming algorithms involve iteration of a random operator the empirical bellman operator we introduce notions of probabilistic fixed points for such random monotone operators we develop a stochastic dominance framework for convergence analysis of such operators we then use this to give sample complexity bounds for both evi and epi we then provide various variations and extensions to asynchronous empirical dynamic programming the minimax empirical dynamic program and show how this can also be used to solve the dynamic newsvendor problem preliminary experimental results

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

estimate of exponential convergence rate and exponential stability for neural networks estimates of exponential convergence rate and exponential stability are studied for a class of neural networks which includes hopfield neural networks and cellular neural networks both local and global exponential convergence are discussed theorems for estimation of the exponential convergence rate are established and the bounds on the rate of convergence are given the domains of attraction in the case of local exponential convergence are obtained simple conditions are presented for checking exponential stability of the neural networks

enumeration of rna structures by matrix models we enumerate the number of rna contact structures according to their genus i e the topological character of their pseudoknots by using a recently proposed matrix model formulation for the rna folding problem we obtain exact results for the simple case of an rna molecule with an infinitely flexible backbone in which any arbitrary pair of bases is allowed we analyze the distribution of the genus of pseudoknots as a function of the total number of nucleotides along the phosphate sugar backbone

rna matrix models with external interactions and their asymptotic behavior we study a matrix model of rna in which an external perturbation on n nucleotides is introduced in the action of the partition function of the polymer chain the effect of the perturbation appears in the exponential generating function of the partition function as a factor exp 1 n alpha l where alpha is the ratio of strengths of the original to the perturbed term and l is the length of the chain the asymptotic behavior of the genus distribution functions as a function of length for the matrix model with interaction is analyzed numerically for all n l it is found that as n alpha l is increased from 0 to 1 the term 3 sup l in the number of diagrams a sub l g alpha sup at a fixed length l genus g and alpha goes to 2 sup l 3 n alpha l sup l for any n alpha l and the total number of diagrams n sub alpha sup at a fixed length l and alpha but independent of genus g undergoes a change in the factor exp radical l to 1 exp 1 n alpha l radical l for any n alpha l however the exponent l of the dominant length dependent term in a sub l g alpha sup stays unchanged hence the universality is robust to changes in the interaction alpha the distribution functions alsomore exhibit unusual behavior at small lengths less

phase transition in random planar diagrams and rna type matching we study the planar matching problem defined by a symmetric random matrix with independent identically distributed entries taking values zero and one we show that the existence of a perfect planar matching structure is possible only above a certain critical density p c of allowed contacts i e of ones using a formulation of the problem in terms of dyck paths and a matrix model of planar contact structures we provide an analytical estimation for the value of the transition point p c in the thermodynamic limit this estimation is close to the critical value p c 0 379 obtained in numerical simulations based on an exact dynamical programming algorithm we characterize the corresponding critical behavior of the model and discuss the relation of the perfect imperfect matching transition to the known molten glass transition in the context of random rna secondary structure formation in particular we provide strong evidence supporting the conjecture that the molten glass transition at t 0 occurs at p c

an optimal approximate dynamic programming algorithm for concave scalar storage problems with vector valued controls we prove convergence of an approximate dynamic programming algorithm for a class of high dimensional stochastic control problems linked by a scalar storage device given a technical condition our problem is motivated by the problem of optimizing energy flows for a power grid supported by grid level storage the problem is formulated as a stochastic dynamic program where we estimate the value of resources in storage using a piecewise linear value function approximation given the technical condition we provide a rigorous convergence proof for an approximate dynamic programming algorithm which can capture the presence of both the amount of energy held in storage as well as other exogenous variables our algorithm exploits the natural concavity of the problem to avoid any need for explicit exploration policies

the dyck bound in the concave 1 dimensional random assignment model we consider models of assignment for random n blue points and n red points on an interval of length 2n in which the cost for connecting a blue point in x to a red point in y is the concave function x y p for 0 1 where the optimal matching is trivially determined here the optimization is non trivial the purpose of this paper is to introduce a special configuration that we call the dyck matching and to study its statistical properties we compute exactly the average cost in the asymptotic limit of large n together with the first subleading correction the scaling is remarkable it is of order n for p 1 2 and it is universal for a wide class of models we conjecture that the average cost of the dyck matching has the same scaling in n as the cost of the optimal matching and we produce numerical data in support of this conjecture we hope to produce a proof of this claim in future work

neurocomputing with time delay analysis for solving convex quadratic programming problems this paper presents a neural network computational scheme with time delay consideration for solving convex quadratic programming problems based on some known results a delay margin is explicitly determined for the stability of the neural dynamics under which the states of the neural network does not oscillate the configuration of the proposed neural network is provided operational characteristics of the neural network are demonstrated via numerical examples

exact solutions to the transportation problem on the line given distributions w of production and v of consumption on the line the monge kantorovich problem is to decide which producer should supply each consumer in order to minimize the total transportation costs here cost will be assumed to be a strictly concave function of the distance which translates into an economy of scale for longer trips and may encourage cross hauling the resulting solutions display a hierarchical structure that reflects a striking separation into local and global scales also found in the real world moreover this structure can be exploited to reduce the infinite dimensional linear problem to a convex minimization in m variables where 2 m 2 counts the number of times that w v changes sign a combinatorial algorithm is then derived which yields exact solutions by optimizing a certain finite sequence of convex separable network flows

dynamic policy programming in this paper we propose a novel policy iteration method called dynamic policy programming dpp to estimate the optimal policy in the infinite horizon markov decision processes dpp is an incremental algorithm that forces a gradual change in policy update this allows us to prove finite iteration and asymptotic l norm performance loss bounds in the presence of approximation estimation error which depend on the average accumulated error as opposed to the standard bounds which are expressed in terms of the supremum of the errors the dependency on the average error is important in problems with limited number of samples per iteration for which the average of the errors can be significantly smaller in size than the supremum of the errors based on these theoretical results we prove that a sampling based variant of dpp dpp rl asymptotically converges to the optimal policy finally we illustrate numerically the applicability of these results on some benchmark problems and compare the performance of the approximate variants of dpp with some existing reinforcement learning rl methods

stability analysis of optimal adaptive control using value iteration with approximation errors effects of the presence of approximation errors are analyzed on the stability of adaptive optimal control using value iteration initiated from a stabilizing control policy this analysis includes the system operated using any single constant resulting control policy and also using an evolving time varying control policy sufficient conditions on the per iteration approximation errors are obtained for guaranteeing the stability a feature of the presented results is providing estimations of the region of attraction under the approximation errors so that if the initial condition is within this region the whole trajectory will remain inside the training region and hence the function approximation results remain reliable

global exponential convergence of recurrent neural networks with variable delays convergence analysis of recurrent neural networks is an important research direction in the field of neural networks novel methods to study the global exponential convergence of recurrent neural networks with variable delays are proposed a condition for global exponential stability which is independent of the delays is derived by the method of delayed inequalities analysis another condition for global exponential stability which depends on the delays is obtained via the method of constructing a suitable and interesting lyapunov functional

planar diagrams from optimization for concave potentials we propose a toy model of a heteropolymer chain capable of forming planar secondary structures typical for rna molecules in this model the sequential intervals between neighboring monomers along a chain are considered as quenched random variables and energies of nonlocal bonds are assumed to be concave functions of those intervals a few factors are neglected the contribution of loop factors to the partition function the variation in energies of different types of complementary nucleotides the stacking interactions and constraints on the minimal size of loops however the model captures well the formation of folded structures without pseudoknots in an arbitrary sequence of nucleotides using the optimization procedure for a special class of concave type potentials borrowed from optimal transport analysis we derive the local difference equation for the ground state free energy of the chain with the planar rna like architecture of paired links we consider various distribution functions of intervals between neighboring monomers truncated gaussian and scale free and demonstrate the existence of a topological crossover from sequential to essentially nested configurations of paired links

analysis of classification based policy iteration algorithms we introduce a variant of the classification based approach to policy iteration which uses a cost sensitive loss function weighting each classification mistake by its actual regret that is the difference between the action value of the greedy action and of the action chosen by the classifier for this algorithm we provide a full finite sample analysis our results state a performance bound in terms of the number of policy improvement steps the number of rollouts used in each iteration the capacity of the considered policy space classifier and a capacity measure which indicates how well the policy space can approximate policies that are greedy with respect to any of its members the analysis reveals a tradeoff between the estimation and approximation errors in this classification based policy iteration setting furthermore it confirms the intuition that classification based policy iteration algorithms could be favorably compared to value based approaches when the policies can be approximated more easily than their corresponding value functions we also study the consistency of the algorithm when there exists a sequence of policy spaces with increasing capacity

weak convergence properties of constrained emphatic temporal difference learning with constant and slowly diminishing stepsize we consider the emphatic temporal difference td algorithm etd for learning the value functions of stationary policies in a discounted finite state and action markov decision process the etd algorithm was recently proposed by sutton mahmood and white 2016 to solve a long standing divergence problem of the standard td algorithm when it is applied to off policy training where data from an exploratory policy are used to evaluate other policies of interest the almost sure convergence of etd has been proved in our recent work under general off policy training conditions but for a narrow range of diminishing stepsize in this paper we present convergence results for constrained versions of etd with constant stepsize and with diminishing stepsize from a broad range our results characterize the asymptotic behavior of the trajectory of iterates produced by those algorithms and are derived by combining key properties of etd with powerful convergence theorems from the weak convergence methods in stochastic approximation theory for the case of constant stepsize in addition to analyzing the behavior of the algorithms in the limit as the stepsize parameter approaches zero we also analyze their behavior for a fixed stepsize and bound the deviations of their averaged iterates from the desired solution these results are obtained by exploiting the weak feller property of the markov chains associated with the algorithms and by using ergodic theorems for weak feller markov chains in conjunction with the convergence results we get from the weak convergence methods besides etd our analysis also applies to the off policy td algorithm when the divergence issue is avoided by setting sufficiently large it yields for that case new results on the asymptotic convergence properties of constrained off policy td with constant or slowly diminishing stepsize

fundamental study global dynamics for non autonomous reaction diffusion neural networks with time varying delays in this paper a class of non autonomous reaction diffusion neural networks with time varying delays is considered novel methods to study the global dynamical behavior of these systems are proposed employing the properties of diffusion operator and the method of delayed inequalities analysis we investigate global exponential stability positive invariant sets and global attracting sets of the neural networks under consideration furthermore conditions sufficient for the existence and uniqueness of periodic attractors for periodic neural networks are derived and the existence range of the attractors is estimated finally two examples are given to demonstrate the effectiveness of these results

finite time bounds for fitted value iteration in this paper we develop a theoretical analysis of the performance of sampling based fitted value iteration fvi to solve infinite state space discounted reward markovian decision processes mdps under the assumption that a generative model of the environment is available our main results come in the form of finite time bounds on the performance of two versions of sampling based fvi the convergence rate results obtained allow us to show that both versions of fvi are well behaving in the sense that by using a sufficiently large number of samples for a large class of mdps arbitrary good performance can be achieved with high probability an important feature of our proof technique is that it permits the study of weighted lp norm performance bounds as a result our technique applies to a large class of function approximation methods e g neural networks adaptive regression trees kernel machines locally weighted learning and our bounds scale well with the effective horizon of the mdp the bounds show a dependence on the stochastic stability properties of the mdp they scale with the discounted average concentrability of the future state distributions they also depend on a new measure of the approximation power of the function space the inherent bellman residual which reflects how well the function space is aligned with the dynamics and rewards of the mdp the conditions of the main result as well as the concepts introduced in the analysis are extensively discussed and compared to previous theoretical results numerical experiments are used to substantiate the theoretical findings

robust stability for interval hopfield neural networks with time delay the conventional hopfield neural network with time delay is intervalized to consider the bounded effect of deviation of network parameters and perturbations yielding a novel interval dynamic hopfield neural network idhnn model a sufficient condition related to the existence of unique equilibrium point and its robust stability is derived

approximate dynamic programming with a fuzzy parameterization dynamic programming dp is a powerful paradigm for general nonlinear optimal control computing exact dp solutions is in general only possible when the process states and the control actions take values in a small discrete set in practice it is necessary to approximate the solutions therefore we propose an algorithm for approximate dp that relies on a fuzzy partition of the state space and on a discretization of the action space this fuzzy q iteration algorithm works for deterministic processes under the discounted return criterion we prove that fuzzy q iteration asymptotically converges to a solution that lies within a bound of the optimal solution a bound on the suboptimality of the solution obtained in a finite number of iterations is also derived under continuity assumptions on the dynamics and on the reward function we show that fuzzy q iteration is consistent i e that it asymptotically obtains the optimal solution as the approximation accuracy increases these properties hold both when the parameters of the approximator are updated in a synchronous fashion and when they are updated asynchronously the asynchronous algorithm is proven to converge at least as fast as the synchronous one the performance of fuzzy q iteration is illustrated in a two link manipulator control problem

finite state approximations to discounted and average cost constrained markov decision processes in this paper we consider the finite state approximation of a discrete time constrained markov decision process mdp under the discounted and average cost criteria using the linear programming formulation of the constrained discounted cost problem we prove the asymptotic convergence of the optimal value of the finite state model to the optimal value of the original model with further continuity condition on the transition probability we also establish a method to compute approximately optimal policies for the average cost instead of using the finite state linear programming approximation method we use the original problem definition to establish the finite state asymptotic approximation of the constrained problem and compute approximately optimal policies under lipschitz type regularity conditions on the components of the mdp we also obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases

how delays affect neural dynamics and learning we investigate the effects of delays on the dynamics and in particular on the oscillatory properties of simple neural network models we extend previously known results regarding the effects of delays on stability and convergence properties we treat in detail the case of ring networks for which we derive simple conditions for oscillating behavior and several formulas to predict the regions of bifurcation the periods of the limit cycles and the phases of the different neurons these results in turn can readily be applied to more complex and more biologically motivated architectures such as layered networks in general the main result is that delays tend to increase the period of oscillations and broaden the spectrum of possible frequencies in a quantifiable way simulations show that the theoretically predicted values are in excellent agreement with the numerically observed behavior adaptable delays are then proposed as one additional mechanism through which neural systems could tailor their own dynamics accordingly we derive recurrent backpropagation learning formulas for the adjustment of delays and other parameters in networks with delayed interactions and discuss some possible applications

enumeration of chord diagrams on many intervals and their non orientable analogs two types of connected chord diagrams with chord endpoints lying in a collection of ordered and oriented real segments are considered here the real segments may contain additional bivalent vertices in one model but not in the other in the former case we record in a generating function the number of fatgraph boundary cycles containing a fixed number of bivalent vertices while in the latter we instead record the number of boundary cycles of each fixed length second order non linear algebraic partial differential equations are derived which are satisfied by these generating functions in each case giving efficient enumerative schemes moreover these generating functions provide multi parameter families of solutions to the kp hierarchy for each model there is furthermore a non orientable analog and each such model likewise has its own associated differential equation the enumerative problems we solve are interpreted in terms of certain polygon gluings as specific applications we discuss models of several interacting rna molecules we also study a matrix integral which computes numbers of chord diagrams in both orientable and non orientable cases in the model with bivalent vertices and the large n limit is computed using techniques of free probability

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location

absolute exponential stability of a class of recurrent neural networks with multiple and variable delays in this paper we derive some new conditions for absolute exponential stability aest of a class of recurrent neural networks with multiple and variable delays by using the holder s inequality and the young s inequality to estimate the derivatives of the lyapunov functionals we are able to establish more general results than some existing ones the first type of conditions established involves the convex combinations of column sum and row sum dominance of the neural network weight matrices while the second type involves the p norm of the weight matrices with p 1

multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions multistability is a property necessary in neural networks in order to enable certain applications e g decision making where monostable networks can be computationally restrictive this article focuses on the analysis of multistability for a class of recurrent neural networks with unsaturating piecewise linear transfer functions it deals fully with the three basic properties of a multistable network boundedness global attractivity and complete convergence this article makes the following contributions conditions based on local inhibition are derived that guarantee boundedness of some multistable networks conditions are established for global attractivity bounds on global attractive sets are obtained complete convergence conditions for the network are developed using novel energy like functions and simulation examples are employed to illustrate the theory thus developed

strict stationarity of generalized autoregressive processes in this paper we consider the multivariate equation x n 1 a n 1 x n b n 1 with i i d coefficients which have only a logarithmic moment we give a necessary and sufficient condition for existence of a strictly stationary solution independent of the future as an application we characterize the multivariate arma equations with general noise which have such a solution

using randomization to break the curse of dimensionality this paper introduces random versions of successive approximations and multigrid algorithms for computing approximate solutions to a class of finite and infinite horizon markovian decision problems mdps we prove that these algorithms succeed in breaking the curse of dimensionality for a subclass of mdps known as discrete decision processes ddps

rna substructure as a random matrix ensemble combinatorial analysis of a certain abstraction of rna structures has been studied to investigate their statistics our approach regards the backbone of secondary structures as an alternate sequence of paired and unpaired sets of nucleotides which can be described by a random matrix model we obtain the generating function of the structures using the hermitian matrix model with the chebyshev polynomial of the second kind and analyze the statistics with respect to the number of stems to match the experimental findings of the statistical behavior we consider the structures in a grand canonical ensemble and find a fugacity value corresponding to an appropriate number of stems

duality theorems for marginal problems given topological spaces x1 xn with product space x probability measures i on xi together with a real function h on x define a marginal problem as well as a dual problem using an extended version of choquet s theorem on capacities an analogue of the classical duality theorem of linear programming is established imposing only weak conditions on the topology of the spaces xi and the measurability resp boundedness of the function h applications concern among others measures with given support stochastic order and general marginal problems

covering number bounds of certain regularized linear function classes recently sample complexity bounds have been derived for problems involving linear functions such as neural networks and support vector machines in many of these theoretical studies the concept of covering numbers played an important role it is thus useful to study covering numbers for linear function classes in this paper we investigate two closely related methods to derive upper bounds on these covering numbers the first method already employed in some earlier studies relies on the so called maurey s lemma the second method uses techniques from the mistake bound framework in online learning we compare results from these two methods as well as their consequences in some learning formulations

stability in asymmetric hopfield nets with transmission delays sufficient conditions are derived for the delay independent stability of the equilibria in hopfield s graded response networks of the type dxi t dt bixi t nj 1aijfj jxj t ij fi t i 1 2 n when the external inputs fi are held temporally uniform a generalization to continuously distributed delays is briefly indicated several illustrative examples are numerically simulated and the results of simulations are graphically displayed