a quasi monte carlo approach to particle simulation of the heat equation the convergence of the monte carlo method for numerical integration can often be improved by replacing random numbers with more uniformly distributed numbers known as quasi random in this paper the convergence of monte carlo particle simulation is studied when these quasi random sequences are used for the one dimensional heat equation discretized in both space and time convergence is proved for a quasi random simulation using reordering of the particles according to their position experimental results are presented for the spatially continuous heat equation in one and two dimensions the results indicate that a significant improvement in both magnitude of error and convergence rate can be achieved over standard monte carlo simulations for certain low dimensional problems

convergence of controlled models and finite state approximation for discounted continuous time markov decision processes with constraints abstract in this paper we consider the convergence of a sequence m n of the models of discounted continuous time constrained markov decision processes mdp to the limit one denoted by m for the models with denumerable states and unbounded transition rates under reasonably mild conditions we prove that the constrained optimal policies and the optimal values of m n converge to those of m respectively using a technique of occupation measures as an application of the convergence result developed here we show that an optimal policy and the optimal value for countable state continuous time mdp can be approximated by those of finite state continuous time mdp finally we further illustrate such finite state approximation by solving numerically a controlled birth and death system and also give the corresponding error bound of the approximation

the ii p method for estimating multivariate functions from noisy data the method for estimating an underlying smooth function of m variables x l xm using noisy data is based on approximating it by a sum of products of the form m m x m the problem is then reduced to estimating the univariate functions in the products a convergent algorithm is described the method keeps tight control on the degrees of freedom used in the fit many examples are given the quality of fit given by the method is excellent usually only a few products are enough to fit even fairly complicated functions the coding into products of univariate functions allows a relatively understandable interpretation of the multivariate fit

brief wiener hopf design of the optimal decoupling control system with state space formulas the optimal decoupling controller minimizing a given quadratic cost function is derived for the generalized plant model with two degree of freedom controller configuration a minimal set of assumptions for the existence of the optimal controller is presented in both the frequency and the state space domains the optimal controller formula is described in the frequency domain and the corresponding state space formula is derived for computational efficiency

applying experimental design and regression splines to high dimensional continuous state stochastic dynamic programming in stochastic dynamic programming sdp with continuous state and decision variables the future value function is computed at discrete points in the state spac e interpolation can be used to approximate the values of the future value function between these discrete points however for large dimensional problems the number of discrete points required to obtain a good approximation of the future value function can be prohibitively large statistical methods of experimental design and function estimation may be employed to overcome this curse of dimensionality in this paper we describe a method for estimating the future value function by multivariate adaptive regression splines mars fit over a discretization scheme based on orthogonal array oa experimental designs because orthogonal arrays only grow polynomially in the state space dimension our oa mars method is accurately able to solve higher dimensional sdp problems than previously possible to our knowledge the most efficient method published prior to this work employs tensor product cubic splines to approximate the future value function johnson et al 1993 the computational advantages of oa ma rs are demonstrated in comparisons with the method using tensor product cubic splines for applications of an inventory forecasting sdp with up to nine state variables computed on a small workstation in particular the storage of an adequate tensor product cubic spline for six dimensions exceeds the memory of our workstation and the run time for an accurate oa mars sdp solution would be at least an order of magnitude faster than using tensor product cubic splines for higher than six dimensions

improved dynamic programming methods for optimal control of lumped parameter stochastic systems new dynamic programming methods are developed to solve stochastic control problems with a larger number of state variables than previously possible these methods apply accurate interpolation to numerical approximation of continuous cost to go functions greatly reducing the number of discrete states that must be evaluated by efficiently incorporating information on first and second derivatives the approximation reduces computational effort by several orders of magnitude over traditional methods consequently it is practical to apply dynamic programming to complex stochastic problems with a larger number of state variables than traditionally possible results are presented for hypothetical reservoir control problems with up to seven state variables and two random inputs

approximation and bounds in discrete event dynamic programming this paper presents a general dynamic programming algorithm for the solution of optimal stochastic control problems concerning a class of discrete event systems the emphasis is put on the numerical technique used for the approximation of the solution of the dynamic programming equation this approach can be efficiently used for the solution of optimal control problems concerning markov renewal processes this is illustrated on a group preventive replacement model generalizing an earlier work of the authors

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

fitting multivariate regression functions by interaction spline models in nonparametric multivariate regression analysis we seek methods to reduce the dimensionality of the regression function to bypass the difficulty caused by the curse of dimensionality the original additive models approximate a regression function by additive univariate functions which overcome successfully the difficulty caused by the curse of dimensionality and have a clear interpretation but since they ignore any interactions between regressors they are not sufficiently flexible to cope with the case where interactions are of concern

adaptive differential dynamic programming for multiobjective optimal control an efficient numerical solution scheme entitled adaptive differential dynamic programming is developed in this paper for multiobjective optimal control problems with a general separable structure for a multiobjective control problem with a general separable structure the optimal weighting coefficients for various performance indices are time varying as the system evolves along any noninferior trajectory recognizing this prominent feature in multiobjective control the proposed adaptive differential dynamic programming methodology combines a search process to identify an optimal time varying weighting sequence with the solution concept in the conventional differential dynamic programming convergence of the proposed adaptive differential dynamic programming methodology is addressed

a comparison of global and semi local approximation in t stage stochastic optimization the paper presents a comparison between two different flavors of nonlinear models to be used for the approximate solution of t stage stochastic optimization tso problems a typical paradigm of markovian decision processes specifically the well known class of neural networks is compared with a semi local approach based on kernel functions characterized by less demanding computational requirements to this purpose two alternative methods for the numerical solution of tso are considered one corresponding to the classic approximate dynamic programming adp and the other based on a direct optimization of the optimal control functions introduced here for the first time advantages and drawbacks in the tso context of the two classes of approximators are analyzed in terms of computational burden and approximation capabilities then their performances are evaluated through simulations in two important high dimensional tso test cases namely inventory forecasting and water reservoirs management

numerical solution of continuous state dynamic programs using linear and spline interpolation this paper demonstrates that the computational effort required to develop numerical solutions to continuous state dynamic programs can be reduced significantly when cubic piecewise polynomial functions rather than tensor product linear interpolants are used to approximate the value function tensor product cubic splines represented in either piecewise polynomial or b spline form and multivariate hermite polynomials are considered computational savings are possible because of the improved accuracy of higher order functions and because the smoothness of higher order functions allows efficient quasi newton methods to be used to compute optimal decisions the use of the more efficient piecewise polynomial form of the spline was slightly superior to the use of hermite polynomials for the test problem and easier to program in comparison to linear interpolation use of splines in piecewise polynomial form reduced the cpu time to obtain results of equivalent accuracy by a factor of 250 330 for a stochastic

deblurring saturated night image with function form kernel deblurring saturated night images are a challenging problem because such images have low contrast combined with heavy noise and saturated regions unlike the deblurring schemes that discard saturated regions when estimating blur kernels this paper proposes a novel scheme to deduce blur kernels from saturated regions via a novel kernel representation and advanced algorithms our key technical contribution is the proposed function form representation of blur kernels which regularizes existing matrix form kernels using three functional components 1 trajectory 2 intensity and 3 expansion from automatically detected saturated regions their skeleton brightness and width are fitted into the corresponding three functional components of blur kernels such regularization significantly improves the quality of kernels deduced from saturated regions second we propose an energy minimizing algorithm to select and assign the deduced function form kernels to partitioned image regions as the initialization for non uniform deblurring finally we convert the assigned function form kernels into matrix form for more detailed estimation in a multi scale deconvolution experimental results show that our scheme outperforms existing schemes on challenging real examples

color demosaicking via directional linear minimum mean square error estimation digital cameras sample scenes using a color filter array of mosaic pattern e g the bayer pattern the demosaicking of the color samples is critical to the image quality this paper presents a new color demosaicking technique of optimal directional filtering of the green red and green blue difference signals under the assumption that the primary difference signals pds between the green and red blue channels are low pass the missing green samples are adaptively estimated in both horizontal and vertical directions by the linear minimum mean square error estimation lmmse technique these directional estimates are then optimally fused to further improve the green estimates finally guided by the demosaicked full resolution green channel the other two color channels are reconstructed from the lmmse filtered and fused pds the experimental results show that the presented color demosaicking technique outperforms the existing methods both in psnr measure and visual perception

penrose demosaicking the penrose pixel layout an aperiodic pixel layout in rhombus penrose tiling has been shown to substantially outperform the existing square pixel layout in super resolution however it was tested only on grayscale images to study its performance on color images we have to reconstruct regular color images from penrose raw images i e images with only one color component at each penrose pixel resulting in the problem of demosaicking from penrose pixels penrose demosaicking is more difficult than regular demosaicking because none of the color components of the reconstructed regular color images are available therefore most of the traditional demosaicking methods do not apply we develop a sparse representation based method for penrose demosaicking extensive experiments show that penrose pixel layout outperforms regular pixel layouts in terms of both perceptual evaluation and s cielab the penrose pixel layout is unique among all irregular layouts because it is uniformly three colorable and it has only two pixel shapes thick and thin rhombi making its manufacturing relatively easy

error estimates for approximate optimization by the extended ritz method an alternative to the classical ritz method for approximate optimization is investigated in the extended ritz method sets of admissible solutions are approximated by their intersections with sets of linear combinations of all n tuples of functions from a given basis this alternative scheme called variable basis approximation includes functions computable by trigonometric polynomials with free frequencies free node splines neural networks and other nonlinear approximating families estimates of rates of approximate optimization by the extended ritz method are derived upper bounds on rates of convergence of suboptimal solutions to the optimal one are expressed in terms of the degree n of variable basis functions the modulus of continuity of the functional to be minimized the modulus of tikhonov well posedness of the problem and certain norms tailored to the type of basis the results are applied to convex best approximation and to kernel methods in machine learning

technical communique wiener hopf approach to derivation of rational doubly coprime factorizations this paper presents a method to construct a stable rational doubly coprime factorization dcf of a transfer matrix proper or improper from a given polynomial dcf wiener hopf factor polynomials are used to rationalize the polynomial dcf matrices it is shown that the suggested construction procedure results in the stable rational dcf description that includes the normalized matrix fractional description mfd and inner coprime mfd as special cases

multi stage stochastic optimization applied to energy planning this paper presents a methodology for the solution of multistage stochastic optimization problems based on the approximation of the expected cost to go functions of stochastic dynamic programming by piecewise linear functions no state discretization is necessary and the combinatorial explosion with the number of states the well known curse of dimensionality of dynamic programming is avoided the piecewise functions are obtained from the dual solutions of the optimization problem at each stage and correspond to benders cuts in a stochastic multistage decomposition framework a case study of optimal stochastic scheduling for a 39 reservoir system is presented and discussed

learning with kernel smoothing models and low discrepancy sampling this brief presents an analysis of the performance of kernel smoothing models used to estimate an unknown target function addressing the case where the choice of the training set is part of the learning process in particular we consider a choice of the points at which the function is observed based on low discrepancy sequences which is a family of sampling methods commonly employed for efficient numerical integration we prove that under suitable regularity assumptions consistency of the empirical risk minimization is guaranteed with a good rate of convergence of the estimation error as well as the convergence of the approximation error simulation results confirm in practice the good theoretical properties given by the combination of kernel smoothing models with low discrepancy sampling

the parameter iteration method in dynamic programming many practical problems involve making optimal decisions for systems with state characterized by many components these problems lead to dynamic programming problems with a very large number of state variables thus an exact derivation of the optimal policy for such problems is not feasible to solve numerically due to the great amount of computer time and storage involved this paper presents a practical method denoted as the parameter iteration method for obtaining an approximate solution for the above described problem the computational difficulty caused by the tremendously large dimensionality of the state variable is overcome by means of an iterative method which combines simulation and recursive estimation to compute successive approximations of the value function the implementation of the parameter iteration method is illustrated for the problem of optimal replacement policy for a multi item markovian system

low discrepancy points for deterministic assignment of hidden weights in extreme learning machines the traditional extreme learning machine elm approach is based on a random assignment of the hidden weight values while the linear coefficients of the output layer are determined analytically this brief presents an analysis based on geometric properties of the sampling points used to assign the weight values investigating the replacement of random generation of such values with low discrepancy sequences ldss such sequences are a family of sampling methods commonly employed for numerical integration yielding a more efficient covering of multidimensional sets with respect to random sequences without the need for any computationally intensive procedure in particular we prove that the universal approximation property of the elm is guaranteed when ldss are employed and how an efficient covering affects the convergence positively furthermore since ldss are generated deterministically the results do not have a probabilistic nature simulation results confirm in practice the good theoretical properties given by the combination of elm with ldss

simplified formula for a controller in optimal control problems we derive a simplified frequency domain formula for a controller transfer matrix for lq optimal output feedback control of stochastic systems for this purpose we apply a generalization of the wiener hopf method the generalization is characterized by the following three properties 1 we generalize the usual operator cdot on rational matrices in the traditional wiener hopf approach the same algorithms as for computation of cdot are applicable to compute the new operator 2 the matrices of the youla kuccera parametrization do not appear in the optimal controller transfer matrix bf c they appear only in its derivation even if the plant is unstable 3 unlike the traditional wiener hopf method d c youla h a jabr and j j bongiorno jr ieee trans automat control ac 21 1976 pp 319 338 where the spectral factors in the two spectral factorizations are both stable and minimum phase our spectral factors need to be minimum phase only finally three state space applications of the formula are presented

local linear regression for function learning an analysis based on sample discrepancy local linear regression models a kind of nonparametric structures that locally perform a linear estimation of the target function are analyzed in the context of empirical risk minimization erm for function learning the analysis is carried out with emphasis on geometric properties of the available data in particular the discrepancy of the observation points used both to build the local regression models and compute the empirical risk is considered this allows to treat indifferently the case in which the samples come from a random external source and the one in which the input space can be freely explored both consistency of the erm procedure and approximating capabilities of the estimator are analyzed proving conditions to ensure convergence since the theoretical analysis shows that the estimation improves as the discrepancy of the observation points becomes smaller low discrepancy sequences a family of sampling methods commonly employed for efficient numerical integration are also analyzed simulation results involving two different examples of function learning are provided

numerical calculation of the optimal three degree of freedom wiener hopf controller state space algorithms are developed for the optimal three degree of freedom wiener hopf controller without placing any restriction on the plant the algorithms are obtained not only for the regulator problem but for the servo problem as well and with or without feedforward control

automatic design of high sensitivity color filter arrays with panchromatic pixels in most of existing digital cameras color images have to be reconstructed from raw images which only have one color sensed at each pixel as their imaging sensors are covered by color filter arrays cfas at each pixel a cfa usually allows only a portion of the light spectrum to pass through and thereby reduces the light sensitivity of pixels to address this issue previous works have explored adding panchromatic pixels into cfas however almost all existing methods assign panchromatic pixels empirically making the designed cfas prone to aliasing artifacts in this paper based on a mathematical model we propose a fully automatic approach to designing high sensitivity cfas using panchromatic pixels by the frequency structure representation of cfas we formulate high sensitivity cfa design as a continuous multi objective optimization problem where robustness to aliasing artifacts and percentage of panchromatic pixels are simultaneously maximized we analyze the characteristics of our new formulation according to the analysis we develop a new method to propose frequency structure candidates which can produce cfas that reach a desired percentage of panchromatic pixels then for each candidate we optimize parameters to obtain the final cfa which is an appropriately balanced solution to the multi objective optimization problem we formulate the two design procedures as constrained optimization problems and solve them using the alternating direction method extensive experiments confirm the advantage of the proposed method in both low light and normal light conditions

exact and approximate solutions of affine reservoir models we formulate an optimization model of a multiple reservoir water resource system that encompasses interbasin transfers among two or more river basins autocorrelated inflows are modeled with a linear autoregressive stochastic process benefits for each period are assumed to depend separably on storage levels and discharges with the dependence on discharges being linear for the important special case of a single river basin a myopic policy hence computed easily is optimal when the model includes the possibility of interbasin transfers a myopic policy is optimal if the deterministic and stochastic portions of the inflow process are always nonnegative even if this assumption is not valid myopic policies yield useful bounds

joint demosaicing and denoising via learned nonparametric random fields we introduce a machine learning approach to demosaicing the reconstruction of color images from incomplete color filter array samples there are two challenges to overcome by a demosaicing method 1 it needs to model and respect the statistics of natural images in order to reconstruct natural looking images and 2 it should be able to perform well in the presence of noise to facilitate an objective assessment of current methods we introduce a public ground truth data set of natural images suitable for research in image demosaicing and denoising we then use this large data set to develop a machine learning approach to demosaicing our proposed method addresses both demosaicing challenges by learning a statistical model of images and noise from hundreds of natural images the resulting model performs simultaneous demosaicing and denoising we show that the machine learning approach has a number of benefits 1 the model is trained to directly optimize a user specified performance measure such as peak signal to noise ratio psnr or structural similarity 2 we can handle novel color filter array layouts by retraining the model on such layouts and 3 it outperforms the previous state of the art in some setups by 0 7 db psnr faithfully reconstructing edges textures and smooth areas our results demonstrate that in demosaicing and related imaging applications discriminatively trained machine learning models have the potential for peak performance at comparatively low engineering effort

robust stochastic optimization for reservoir operation optimal reservoir operation under uncertainty is a challenging engineering problem application of classic stochastic optimization methods to large scale problems is limited due to computational difficulty moreover classic stochastic methods assume that the estimated distribution function or the sample inflow data accurately represents the true probability distribution which may be invalid and the performance of the algorithms may be undermined in this study we introduce a robust optimization ro approach iterative linear decision rule ildr so as to provide a tractable approximation for a multiperiod hydropower generation problem the proposed approach extends the existing ldr method by accommodating nonlinear objective functions it also provides users with the flexibility of choosing the accuracy of ildr approximations by assigning a desired number of piecewise linear segments to each uncertainty the performance of the ildr is compared with benchmark policies including the sampling stochastic dynamic programming ssdp policy derived from historical data the ildr solves both the single and multireservoir systems efficiently the single reservoir case study results show that the ro method is as good as ssdp when implemented on the original historical inflows and it outperforms ssdp policy when tested on generated inflows with the same mean and covariance matrix as those in history for the multireservoir case study which considers water supply in addition to power generation numerical results show that the proposed approach performs as well as in the single reservoir case study in terms of optimal value and distributional robustness

deterministic learning for maximum likelihood estimation through neural networks in this paper a general method for the numerical solution of maximum likelihood estimation mle problems is presented it adopts the deterministic learning dl approach to find close approximations to ml estimator functions for the unknown parameters of any given density the method relies on the choice of a proper neural network and on the deterministic generation of samples of observations of the likelihood function thus avoiding the problem of generating samples with the unknown density under mild assumptions consistency and convergence with favorable rates to the true ml estimator function can be proved simulation results are provided to show the good behavior of the algorithm compared to the corresponding exact solutions

functional optimization through semilocal approximate minimization an approach based on semilocal approximation is introduced for the solution of a general class of operations research problems such as markovian decision problems multistage optimal control and maximum likelihood estimation because it is extremely hard to derive analytical solutions that minimize the cost in most instances of the problem we must look for approximate solutions here it is shown that good solutions can be obtained with a moderate computational effort by exploiting properties of semilocal approximation through kernel models and efficient sampling of the state space the convergence of the proposed method called semilocal approximate minimization slam is discussed and the consistency of the solution is derived simulation results show the efficiency of slam also through its application to a classic operations research problem i e inventory forecasting

measuring the goodness of orthogonal array discretizations for stochastic programming and stochastic dynamic programming this paper describes a continuous space discretization scheme based on statistical experimental designs generated from orthogonal arrays oas of strength three with index unity chen ruppert and shoemaker oper res 47 1999 pp 38 53 employed this efficient discretization scheme in a numerical solution method for high dimensional continuous state stochastic dynamic programming sdp these oas may be instrumental in reducing the dimensionality of event spaces sdp state spaces and first stage decision spaces in two stage stochastic programming in particular computationally efficient space filling measures for these oas are derived for evaluating how well a specific oa discretization fills the state space comparisons were made with two types of common measures ones which maximize the average or minimum distance between discretization points within the oa and ones which minimize the average or maximum distance between discretization points and nondiscretization points lying on a full grid i e points lying on the full grid that are not contained in the oa discretization oas of strength three were tested by fitting multivariate adaptive regression splines to data from an inventory forecasting continuous state stochastic dynamic program

a geometric method for optimal design of color filter arrays a color filter array cfa used in a digital camera is a mosaic of spectrally selective filters which allows only one color component to be sensed at each pixel the missing two components of each pixel have to be estimated by methods known as demosaicking the demosaicking algorithm and the cfa design are crucial for the quality of the output images in this paper we present a cfa design methodology in the frequency domain the frequency structure which is shown to be just the symbolic dft of the cfa pattern one period of the cfa is introduced to represent images sampled with any rectangular cfas in the frequency domain based on the frequency structure the cfa design involves the solution of a constrained optimization problem that aims at minimizing the demosaicking error to decrease the number of parameters and speed up the parameter searching the optimization problem is reformulated as the selection of geometric points on the boundary of a convex polygon or the surface of a convex polyhedron using our methodology several new cfa patterns are found which outperform the currently commercialized and published ones experiments demonstrate the effectiveness of our cfa design methodology and the superiority of our new cfa patterns

gradient dynamic programming for stochastic optimal control of multidimensional water resources systems a new computational algorithm is presented for the solution of discrete time linearly constrained stochastic optimal control problems decomposable in stages the algorithm designated gradient dynamic programming is a backward moving stagewise optimization the main innovations over conventional discrete dynamic programming ddp are in the functional representation of the cost to go function and the solution of the single stage problem the cost to go function assumed to be of requisite smoothness is approximated within each element defined by the discretization scheme by the lowest order polynomial which preserve its values and the values of its gradient with respect to the state variables at all nodes of the discretization grid the improved accuracy of this hermitian interpolation scheme reduces the effect of discretization error and allows the use of coarser grids which reduces the dimensionality of the problem at each stage the optimal control is determined on each node of the discretized state space using a constrained newton type optimization procedure which has quadratic rate of convergence the set of constraints which act as equalities is determined from an active set strategy which converges under lenient convexity requirements this method of solving the single stage optimization is much more efficient than the conventional way based on enumeration or iterative methods with linear rate of convergence once the optimal control is determined the cost to go function and its gradient with respect to the state variables is calculated to be used at the next stage the proposed technique permits the efficient optimization of stochastic systems whose high dimensionality does not permit solution under the conventional ddp framework and for which successive approximation methods are not directly applicable due to stochasticity results for a four reservoir example are presented the purpose of this paper is to present a new computational algorithm for the stochastic optimization of sequential decision problems one important and extensively studied class of such problems in the area of water resources is the discrete time optimal control of multireservoir systems under stochastic inflows other applications include the optimal design and operation of sewer systems e g mays and wenzel 1976 labadie et al 1980 the optimal conjunctive utilization of surface and groundwater resources e g buras 1972 and the minimum cost water quality maintenance in rivers e g dracup and fogarty 1974 chang and yeh 1973 to mention only a few of the water resources applications and pertinent references an extensive review of dynamic programming applications in water resources can be found in the works by yakowitz 1982 and yeh 1985 before we proceed with the

a feedback theory of two degree of freedom optimal wiener hopf design the design of linear two degree of freedom stabilizing controllers is treated in a quadratic cost setting the class of all such controllers which give finite cost is established and the tradeoff possible between optimum performance tracking cost sensitivity and stability margins is discussed

a class of interpolation algorithms for bounded real functions in this paper a general interpolation formula is presented that yields the real rational solutions when the complex points are interpolated the proposed interpolation formula includes the algorithms of nevanlinna and pick and of dorato and li as special cases the results of this paper are based on the positive real interpolation theories developed by youla and saito

error analysis of conventional discrete and gradient dynamic programming an asymptotic error analysis of the conventional discrete dynamic programming ddp method is presented and upper bounds of the error in the control policy i e the difference of the estimated and true optimal control at each operation period are computed this error is shown to be of the order of the state discretization interval as a result with significant implications in the optimization of multistate systems where the curse of dimensionality restricts the number of states to a relatively small number the error in the optimal cost varies with as 2 the analysis provides useful insights into the effects of state discretization on calculated control and cost functions the comparability of results from different discretizations and criteria about the required number of nodes in an effort to reduce the discretization error in the case of smooth cost functions a new discrete dynamic programming method termed gradient dynamic programming gdp is proposed gdp uses a piecewise hermite interpolation of the cost to go function at each stage which preserves the values of the cost to go function and of its first derivatives at the discretization nodes the error in the control policy is shown to be of the order of as 3 and the error in the cost to vary with as thus as as decreases gdp converges to the true optimum much more rapidly than ddp another major advantage of the new methodology is that it facilitates the use of newton type iterative methods in the solution of the nonlinear optimization problems at each stage the linear convergence of ddp and the superlinear convergence of gdp are illustrated in an example

policy bounds for markov decision processes this paper demonstrates how a markov decision process mdp can be approximated to generate a policy bound i e a function that bounds the optimal policy from below or from above for all states we present sufficient conditions for several computationally attractive approximations to generate rigorous policy bounds these approximations include approximating the optimal value function replacing the original mdp with a separable approximate mdp and approximating a stochastic mdp with its deterministic counterpart an example from the field of fisheries management demonstrates the practical applicability of the results

brief on the two degree of freedom wiener hopf optimal design with tracking and disturbance rejection constraints in this paper the design of the two degree of freedom optimal wiener hopf controller with asymptotic tracking and disturbance rejection constraints is studied based on the solvability conditions and the solution parameterization of diophantine equations over the stable rational fractional ring m s the disturbance rejection problem in the general case is solved a simple cost function is defined to reflect the transient performance of control signal caused by deterministic signals finally the design of the two degree of freedom optimal wiener hopf controller is presented

generalized polynomial approximations in markovian decision processes abstract fitting the value function in a markovian decision process by a linear superposition of m basis functions reduces the problem dimensionality from the number of states down to m with good accuracy retained if the value function is a smooth function of its argument the state vector this paper provides for both the discounted and undiscounted cases three algorithms for computing the coefficients in the linear superposition linear programming policy iteration and least squares

optimization of a large scale water reservoir network by stochastic dynamic programming with efficient state space discretization a numerical solution to a 30 dimensional water reservoir network optimization problem based on stochastic dynamic programming is presented in such problems the amount of water to be released from each reservoir is chosen to minimize a nonlinear cost or maximize benefit function while satisfying proper constraints experimental results show how dimensionality issues given by the large number of basins and realistic modeling of the stochastic inflows can be mitigated by employing neural approximators for the value functions and efficient discretizations of the state space such as orthogonal arrays latin hypercube designs and low discrepancy sequences

an infinite dimensional linear programming algorithm for deterministic semi markov decision processes on borel spaces we devise an algorithm for solving the infinite dimensional linear programs that arise from general deterministic semi markov decision processes on borel spaces the algorithm constructs a sequence of approximate primal dual solutions that converge to an optimal one the innovative idea is to approximate the dual solution with continuous piecewise linear ridge functions that naturally represent functions defined on a high dimensional domain as linear combinations of functions defined on only a single dimension this approximation gives rise to a primal dual pair of semi infinite programs for which we show strong duality in addition we prove various properties of the underlying ridge functions

piecewise affine approximations for the control of a one reservoir hydroelectric system abstract we analyze the computation of optimal and approximately optimal policies for a discrete time model of a single reservoir whose discharges generate hydroelectric power inflows in successive periods are random variables revenue from hydroelectric production is represented by a piecewise linear function we use the special structure of optimal policies together with piecewise affine approximations of the optimal return functions at each stage of dynamic programming to decrease the computational effort by an order of magnitude compared with ordinary value iteration the method is then used to obtain easily computable lower and upper bounds on the value function of an optimal policy and a policy whose value function is between the bounds

wiener hopf design of the general two degree of freedom controllers and the connection to state space formulas h 2 design of the two degree of freedom controller configuration is treated for the generalized plant in both the frequency and state space domains a frequency domain wiener hopf formula is derived first under a set of compact assumptions that gives the entire class of controllers for which the feedback control system is stable and a quadratic cost functional is finite the h 2 design method suggested in this paper allows the reference inputs to include the persistent signals such as step or ramp functions and the class of persistent reference inputs that can be tracked with zero steady state error is characterized explicitly to circumvent the computational difficulties of the wiener hopf formulas the corresponding state space domain formulas are derived by rationalizing the polynomial fractional descriptions it is shown that the well known standard time domain assumptions for the h 2 design problem have the corresponding equivalent frequency domain conditions and this equivalence relationship is show

orthogonal array based latin hypercubes abstract in this article we use orthogonal arrays oa s to construct latin hypercubes besides preserving the univariate stratification properties of latin hypercubes these strength r oa based latin hypercubes also stratify each r dimensional margin therefore such oa based latin hypercubes provide more suitable designs for computer experiments and numerical integration than do general latin hypercubes we prove that when used for integration the sampling scheme with oa based latin hypercubes offers a substantial improvement over latin hypercube sampling

local sparse structure denoising for low light level image sparse and redundant representations perform well in image denoising however sparsity based methods fail to denoise low light level lll images because of heavy and complex noise they consider sparsity on image patches independently and tend to lose the texture structures to suppress noises and maintain textures simultaneously it is necessary to embed noise invariant features into the sparse decomposition process we therefore used a local structure preserving sparse coding lspsc formulation to explore the local sparse structures both the sparsity and local structure in image it was found that with the introduction of spatial local structure constraint into the general sparse coding algorithm lspsc could improve the robustness of sparse representation for patches in serious noise we further used a kernel lspsc k lspsc formulation which extends lspsc into the kernel space to weaken the influence of linear structure constraint in nonlinear data based on the robust lspsc and k lspsc algorithms we constructed a local sparse structure denoising lssd model for lll images which was demonstrated to give high performance in the natural lll images denoising indicating that both the lspsc and k lspsc based lssd models have the stable property of noise inhibition and texture details preservation

a new color filter array with optimal properties for noiseless and noisy color image acquisition digital color cameras acquire color images by means of a sensor on which a color filter array cfa is overlaid the bayer cfa dominates the consumer market but there has recently been a renewed interest for the design of cfas however robustness to noise is often neglected in the design though it is crucial in practice in this paper we present a new 2 3 periodic cfa which provides by construction the optimal tradeoff between robustness to aliasing chrominance noise and luminance noise moreover a simple and efficient linear demosaicking algorithm is described which fully exploits the spectral properties of the cfa practical experiments confirm the superiority of our design both in noiseless and noisy scenarios

state space solutions to standard h sub 2 and h sub infinity control problems simple state space formulas are derived for all controllers solving the following standard h sub infinity problem for a given number gamma 0 find all controllers such that the h sub infinity norm of the closed loop transfer function is strictly less than gamma it is known that a controller exists if and only if the unique stabilizing solutions to two algebraic riccati equations are positive definite and the spectral radius of their product is less than gamma sup 2 under these conditions a parameterization of all controllers solving the problem is given as a linear fractional transformation lft on a contractive stable free parameter the state dimension of the coefficient matrix for the lft constructed using the two riccati solutions equals that of the plant and has a separation structure reminiscent of classical lqg i e h sub 2 theory this paper is intended to be of tutorial value so a standard h sub 2 solution is developed in parallel

tree based reinforcement learning for optimal water reservoir operation 1 although being one of the most popular and extensively studied approaches to design water reservoir operations stochastic dynamic programming is plagued by a dual curse that makes it unsuitable to cope with large water systems the computational requirement grows exponentially with the number of state variables considered curse of dimensionality and an explicit model must be available to describe every system transition and the associated rewards costs curse of modeling a variety of simplifications and approximations have been devised in the past which in many cases make the resulting operating policies inefficient and of scarce relevance in practical contexts in this paper a reinforcement learning approach called fitted q iteration is presented it combines the principle of continuous approximation of the value functions with a process of learning off line from experience to design daily cyclostationary operating policies the continuous approximation performed via tree based regression makes it possible to mitigate the curse of dimensionality by adopting a very coarse discretization grid with respect to the dense grid required to design an equally performing policy via stochastic dynamic programming the learning experience in the form of a data set generated combining historical observations and model simulations allows us to overcome the curse of modeling lake como water system italy is used as study site to infer general guidelines on the appropriate setting for the algorithm parameters and to demonstrate the advantages of the approach in terms of accuracy and computational effectiveness compared to traditional stochastic dynamic programming

approximate stochastic dynamic programming for hydroelectric production planning abstract this paper presents a novel approach for approximate stochastic dynamic programming asdp over a continuous state space when the optimization phase has a near convex structure the approach entails a simplicial partitioning of the state space bounds on the true value function are used to refine the partition we also provide analytic formulae for the computation of the expectation of the value function in the uni basin case where natural inflows are strongly correlated the approach is experimented on several configurations of hydro energy systems it is also tested against actual industrial data

using randomization to break the curse of dimensionality this paper introduces random versions of successive approximations and multigrid algorithms for computing approximate solutions to a class of finite and infinite horizon markovian decision problems mdps we prove that these algorithms succeed in breaking the curse of dimensionality for a subclass of mdps known as discrete decision processes ddps

the calculation of posterior distributions by data augmentation abstract the idea of data augmentation arises naturally in missing value problems as exemplified by the standard ways of filling in missing cells in balanced two way tables thus data augmentation refers to a scheme of augmenting the observed data so as to make it more easy to analyze this device is used to great advantage by the em algorithm dempster laird and rubin 1977 in solving maximum likelihood problems in situations when the likelihood cannot be approximated closely by the normal likelihood maximum likelihood estimates and the associated standard errors cannot be relied upon to make valid inferential statements from the bayesian point of view one must now calculate the posterior distribution of parameters of interest if data augmentation can be used in the calculation of the maximum likelihood estimate then in the same cases one ought to be able to use it in the computation of the posterior distribution it is the purpose of this article to explain how this can be done the basic idea