analysis of an adaptive control scheme for a partially observed controlled markov chain the authors consider an adaptive finite state controlled markov chain with partial state information motivated by a class of replacement problems they present parameter estimation techniques based on the information available after actions that reset the state to a known value are taken it is proved that the parameter estimates converge w p 1 to the true unknown parameter under the feedback structure induced by a certainty equivalent adaptive policy it is shown that the adaptive policy is self optimizing in a long run average sense for any measurable sequence of parameter estimates converging w p 1 to the true parameter

the discrete zak transform application to time frequency analysis and synthesis of nonstationary signals algorithms to compute coefficients of the finite double sum expansion of time varying nonstationary signals and to synthesize them from a finite number of expansion coefficients are presented the algorithms are based on the computation of the discrete zak transform dzt fast algorithms to compute dzt are presented modifications to the algorithms which increase robustness are given

optimal search among false contacts this paper considers rational methods for the design of search patterns when the intended target is among poisson distributed false contacts attention is restricted to situations in which the searcher must immediately decide based on sensed information whether a contact is the intended target and if the answer is yes end the search and commit to the contact discrimination of false contact from target is modeled as reliable only with given probabilities this research seeks to maximize the probability of finding the intended target in a fixed time most of the paper addresses an optimal search trajectory when intended target and false contacts move according to markov processes the searcher s velocity is taken as the control and it is assumed that his speed is limited and his initial position is given it is proved that an optimal control exists in the space l infty a set of necessary conditions for the optimal control is derived a maximum principle in the form of a two point boundary value pr

strong convergence theorems for monotone mappings and relatively weak nonexpansive mappings in this paper we prove strong convergence theorems to a zero of monotone mapping and a fixed point of relatively weak nonexpansive mapping moreover strong convergence theorems to a point which is a fixed point of relatively weak nonexpansive mapping and a solution of a certain variational problem are proved under appropriate conditions

the complexity of the optimal searcher path problem in this note we show that the problem of finding an optimal searcher path that maximizes the probability of detecting a stationary target by the end of a fixed time is np complete we also demonstrate that the problem of finding a path that minimizes mean time to detection is np hard

on solving event based optimization with average reward over infinite stages event based optimization ebo provides a unified framework for problems in which decisions can be made only when certain events occur because the event sequence usually is not markovian the optimal policy could depend on the entire event history which is hard to implement in practice so most existing studies focus on memoryless policies which make decisions only based on the current observable events but it remains open how to find the optimal memoryless policies in general leaving alone to solve the ebo optimally in this technical note we address these two important questions for infinite stage ebos with finite state and action spaces and make the following three major contributions first we extend our previous studies on finite stage ebos and convert infinite stage ebos to partially observable markov decision processes pomdps the belief process of this pomdp is called belief event decision process bedp under certain well known conditions the optimal policies of bedps can be achieved within stationary markov deterministic policies second assuming optimal stationary policies exist the performance difference and derivative formulas are developed potentials of memoryless event based policies are shown to be piecewise linear functions and thus can be efficiently estimated through sample paths third a potential based approximate policy iteration algorithm is developed to obtain near optimal memoryless policies the convergence and performance loss bound of this algorithm are analyzed

the complexity of stochastic games abstract we consider the complexity of stochastic games simple games of chance played by two players we show that the problem of deciding which player has the greatest chance of winning the game is in the class np co np

learning where to attend with deep architectures for image tracking we discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data motivated by theories of perception the model consists of two interacting pathways identity and control intended to mirror the what and where pathways in neuroscience models the identity pathway models object appearance and performs classification using deep factored restricted boltzmann machines at each point in time the observations consist of foveated images with decaying resolution toward the periphery of the gaze the control pathway models the location orientation scale and speed of the attended object the posterior distribution of these states is estimated with particle filtering deeper in the control pathway we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty unlike in our previous work we introduce gaze selection strategies that operate in the presence of partial information and on a continuous action space we show that a straightforward extension of the existing approach to the partial information setting results in poor performance and we propose an alternative method based on modeling the reward surface as a gaussian process this approach gives good performance in the presence of partial information and allows us to expand the action space from a small discrete set of fixation points to a continuous domain

myopic bounds for optimal policy of pomdps an extension of lovejoy s structural results this paper provides a relaxation of the sufficient conditions and an extension of the structural results for partially observed markov decision processes pomdps obtained by lovejoy in 1987 sufficient conditions are provided so that the optimal policy can be upper and lower bounded by judiciously chosen myopic policies these myopic policy bounds are constructed to maximize the volume of belief states where they coincide with the optimal policy numerical examples illustrate these myopic bounds for both continuous and discrete observation sets

gabor transform of time varying systems exact representation and approximation approximate modeling of discrete time linear time varying systems is studied based on a representation of linear systems in the gabor time frequency space the time varying system is assumed to be given in input output or kernel representation this representation has previously received attention because of its applicability in frozen time analysis and design of optimal control for time varying systems but requires a large number of coefficients motivated by its application to signal analysis the gabor transform is considered as a tool for the representation and approximation of linear time variant systems in order to show the main results the class of systems considered is restricted to the one usually considered in the frozen time approach an example is included to illustrate the potential application of the technique

the potential structure of sample paths and performance sensitivities of markov systems we study the structure of sample paths of markov systems by using performance potentials as the fundamental units with a sample path based approach we show that performance sensitivity formulas performance gradients and performance differences of markov systems can be constructed intuitively by first principles with performance potentials or equivalently perturbation realization factors as building blocks in particular we derive sensitivity formulas for two markov chains with possibly different state spaces the proposed approach can be used to obtain flexibly the sensitivity formulas for a wide range of problems including those with partial information these formulas are the basis for performance optimization of discrete event dynamic systems including perturbation analysis markov decision processes and reinforcement learning the approach thus provides insight on on line learning and performance optimization and opens up new research directions sample path based algorithms can be developed

convergence theorems for mixed equilibrium problems variational inequality problem and uniformly quasi ϕ asymptotically nonexpansive mappings abstract we introduce a modified block hybrid projection algorithm for finding a common element of the set of common fixed points of an infinite family of closed and uniformly quasi asymptotically nonexpansive mappings the set of the variational inequality for an inverse strongly monotone operator the set of solutions of the mixed equilibrium problems then we obtain strong convergence theorems for the sequences generated by this process in a 2 uniformly convex and uniformly smooth banach space our results extend and improve ones from several earlier works

strong and weak convergence theorems for equilibrium problems and relatively nonexpansive mappings in banach spaces abstract in this paper we introduce two iterative sequences for finding a common element of the set of fixed points of a relatively nonexpansive mapping and the set of solutions of an equilibrium problem in a banach space then we study the strong and weak convergence of the sequences

finite memory suboptimal design for partially observed markov decision processes we develop bounds on the value function and a suboptimal design for the partially observed markov decision process these bounds and suboptimal design are based on the m most recent observations and actions an a priori measure of the quality of these bounds is given we show that larger m implies tighter bounds an operations count analysis indicates that a z m 1 s multiplications and additions are required per successive approximations iteration of the suboptimal design algorithm where a z and s are the action observation and state spaces respectively suggesting the algorithm is of potential use for problems with large state spaces a preliminary numerical study indicates that the quality of the suboptimal design can be excellent

a two state partially observable markov decision process with three actions abstract a process can be in either a stable or an unstable state interchangeably the true state is unobservable and can only be inferred from observations three actions are available continue with the process con repair the process for a certain fee bring the process to the stable state rep and obtain the state of the process for a cost ins the objective is to maximize the expected discounted value of the total future profits we formulate the problem as a discrete time partially observable markov decision process pomdp we show that the expected profit function is convex and strictly increasing and that the optimal policy has either one or two control limits also we show that dominance in expectation the expected revenue is larger in the stable state than in the unstable state suffices for a control limit structure

a note on the ross taylor theorem abstract in this note the condition used in proving a result due to s m ross and h m taylor are examined these results pertain to the existence of bounded solutions to the average cost optimality equation for controlled markov processes with an average cost criterion in particular we show how the use of commonly found convexity concavity properties of value functions can be employed to verify a seemingly rather restrictive equicontinuity condition in addition we remark that several results in the literature can be viewed as special cases of the result by ross and taylor contrary to claims otherwise

a search game when a search path is given abstract in this paper we investigate a search game in discrete time and space a searcher is given a search path in advance and his look on the path is determined by a randomized look strategy a target selects a path from some options the searcher gains a value on the detection of the target but expends search cost by the look a pay off function of the game for the searcher is the expected reward which is defined as the expected value minus the expected search cost first we show a recursive relation for the conditional optimal look strategy of the searcher given a target path we prove its np completeness though it looks simple and clarify some characteristics of the solution then our original continuous game is converted to a matrix game from these facts we consider a relationship between the game and the one sided optimizing problem and examine some examples

on near optimality of the set of finite state controllers for average cost pomdp we consider the average cost problem for partially observable markov decision processes pomdp with finite state observation and control spaces we prove that there exists an e optimal finite state controller fsc functionally independent of initial distributions for any e 0 under the assumption that the optimal liminf average cost function of the pomdp is constant as part of our proof we establish that if the optimal liminf average cost function is constant then the optimal limsup average cost function is also constant and the two are equal we also discuss the connection between the existence of nearly optimal finite history controllers and two other important issues for average cost pomdp the existence of an average cost that is independent of the initial state distribution and the existence of a bounded solution to the constant average cost optimality equation

the optimal search for a markovian target when the search path is constrained the infinite horizon case a target moves among a finite number of cells according to a discrete time homogeneous markov chain the searcher is subject to constraints on the search path i e the cells available for search in the current epoch is a function of the cell searched in the previous epoch the aim is to identify a search policy that maximizes the infinite horizon total expected reward earned we show the following structural results under the assumption that the target transition matrix is ergodic 1 the optimal search policy is stationary and 2 there exists spl epsi optimal stationary policies which may be constructed by the standard value iteration algorithm in finite time these results are obtained by showing that the dynamic programming operator associated with the search problem is an m stage contraction mapping on a suitably defined space an upper bound of m and the coefficient of contraction spl alpha is given in terms of the transition matrix and other variables pertaining to the search problem these bounds on m and spl alpha may be used to derive bounds on suboptimal search polices constructed

discounted mean bound for the optimal searcher path problem with non uniform travel times we consider an extension of the optimal searcher path problem osp where a searcher moving through a discretised environment may now need to spend a non uniform amount of time travelling from one region to another before being able to search it for the presence of a moving target in constraining not only where but when the search of each cell can take place the problem more appropriately models the search of environments which cannot be easily partitioned into equally sized cells an existing osp bounding method in literature the mean bound is generalised to provide bounds for solving the new problem in a branch and bound framework the main contribution of this paper is an enhancement discounted mean dmean which greatly tightens the bound for the new and existing problems alike with almost no additional computation we test the new algorithm against existing osp bounding methods and show it leads to faster solution times for moving target search problems

solution procedures for partially observed markov decision processes we present three algorithms to solve the infinite horizon expected discounted total reward partially observed markov decision process pomdp each algorithm integrates a successive approximations algorithm for the pomdp due to a smallwood and e sondik with an appropriately generalized numerical technique that has been shown to reduce cpu time until convergence for the completely observed case the first technique is reward revision the second technique is reward revision integrated with modified policy iteration the third is a standard extrapolation a numerical study indicates the potentially significant computational value of these algorithms

efficient learning and planning with compressed predictive states predictive state representations psrs offer an expressive framework for modelling partially observable systems by compactly representing systems as functions of observable quantities the psr learning approach avoids using local minima prone expectation maximization and instead employs a globally optimal moment based algorithm moreover since psrs do not require a predetermined latent state structure as an input they offer an attractive framework for model based reinforcement learning when agents must plan without a priori access to a system model unfortunately the expressiveness of psrs comes with significant computational cost and this cost is a major factor inhibiting the use of psrs in applications in order to alleviate this shortcoming we introduce the notion of compressed psrs cpsrs the cpsr learning approach combines recent advancements in dimensionality reduction incremental matrix decomposition and compressed sensing we show how this approach provides a principled avenue for learning accurate approximations of psrs drastically reducing the computational costs associated with learning while also providing effective regularization going further we propose a planning framework which exploits these learned models and we show that this approach facilitates model learning and planning in large complex partially observable domains a task that is infeasible without the principled use of compression

technical note trading off quick versus slow actions in optimal search we consider the search for a target whose precise location is uncertain the search region is divided into grid cells and the searcher decides which cell to visit next and whether to search it quickly or slowly a quick search of a cell containing the target may damage it resulting in a failed search or it may locate the target safely if the target is not in the cell the search continues over the remaining cells if a slow search is performed on a cell then the search ends in failure with a fixed probability regardless of whether or not the target is in that cell e g because of enemy fire while performing the slow search if the slow search survives this failure possibility then the search ends in success if the target is in that cell otherwise the search continues over the remaining cells we seek to minimize the probability of the search ending in failure and consider two types of rules for visiting cells the unconstrained search in which the searcher may visit cells in any order and the constrained search in which the searcher may only visit adjacent cells e g up down left or right of cells already visited we prove that the optimal policy for the unconstrained search is to search quickly some initial set of cells with the lowest probabilities of containing the target before slowly searching the remaining cells in decreasing order of probabilities for the special case in which a quick search on a cell containing the target damages it with certainty the optimal policy is to search all cells slowly in decreasing order of probabilities we use the optimal solution of the unconstrained search in a branch and bound optimal solution algorithm for the constrained search for larger instances we evaluate heuristics and approximate dynamic programming approaches for finding good solutions

strong planning under partial observability rarely planning domains are fully observable for this reason the ability to deal with partial observability is one of the most important challenges in planning in this paper we tackle the problem of strong planning under partial observability in nondeterministic domains find a conditional plan that will result in a successful state regardless of multiple initial states nondeterministic action effects and partial observability we make the following contributions first we formally define the problem of strong planning within a general framework for modeling partially observable planning domains second we propose an effective planning algorithm based on and or search in the space of beliefs we prove that our algorithm always terminates and is correct and complete in order to achieve additional effectiveness we leverage on a symbolic bdd based representation for the domain and propose several search strategies we provide a thorough experimental evaluation of our approach based on a wide selection of benchmarks we compare the performance of the proposed search strategies and identify a uniform winner that combines heuristic distance measures with mechanisms that reduce runtime uncertainty then we compare our planner mbp with other state of the art systems mbp is able to outperform its competitor systems often by orders of magnitude

partially observed semi markov zero sum games with average payoff we study a zero sum partially observed semi markov game with average payoff on a countable state space under certain ergodicity conditions we show that a saddle point equilibrium exists we achieve this by solving the corresponding average cost optimality equation using a span contraction method the average value is shown to be the unique zero of a lipschitz continuous function a value iteration scheme is developed to compute the value

the complexity of mean payoff games on graphs abstract we study the complexity of finding the values and optimal strategies of mean payoff games on graphs a family of perfect information games introduced by ehrenfeucht and mycielski and considered by gurvich karzanov and khachiyan we describe a pseudo polynomial time algorithm for the solution of such games the decision problem for which is in np conp finally we describe a polynomial reduction from mean payoff games to the simple stochastic games studied by condon these games are also known to be in np conp but no polynomial or pseudo polynomial time algorithm is known for them

optimal control of hidden markov models with binary observations a hidden markov model hmm consisting of a controlled markov chain and a binary output is considered the binary output reports the occurrence of a certain state versus any other state in the markov chain the transition matrix of the markov chain depends on a control vector which is to be chosen to minimize a discounted infinite horizon cost function the information provided for decision making is the entire history of the binary output which presents an incomplete knowledge of the markov chain and as a consequence leads to a stochastic control problem with partial observations the solution to such a problem is a control policy consisting of two components an estimator to derive the posterior probability distribution of the markov chain from the observation set and a nonlinear control law to map this estimate into the control vector this paper introduces a nonlinear state space representation for the estimator develops analytical expressions for the control law and presents numerical methods for efficient computation of the optimal control application of these results to optimal management of a class of inventory systems is discussed and the performance of optimal control for this application is examined by numerical simulations

superposition frames for adaptive time frequency analysis and fast reconstruction in this paper we introduce a broad family of adaptive linear time frequency representations termed superposition frames and show that they admit desirable fast overlap add reconstruction properties akin to standard short time fourier techniques this approach stands in contrast to many adaptive time frequency representations in the existing literature which while more flexible than standard fixed resolution approaches typically fail to provide for efficient reconstruction and often lack the regular structure necessary for precise frame theoretic analysis our main technical contributions come through the development of properties which ensure that our superposition construction provides for a numerically stable invertible signal representation our primary algorithmic contributions come via the introduction and discussion of two signal adaptation schemes based on greedy selection and dynamic programming respectively we conclude with two short enhancement examples that serve to highlight potential applications of our approach

partially observable markov decision processes a geometric technique and analysis this paper presents a novel framework for studying partially observable markov decision processes pomdps with finite state action observation sets and discounted rewards the new framework is solely based on future reward vectors associated with future policies which is more parsimonious than the traditional framework based on belief vectors it reveals the connection between the pomdp problem and two computational geometry problems i e finding the vertices of a convex hull and finding the minkowski sum of convex polytopes which can help solve the pomdp problem more efficiently the new framework can clarify some existing algorithms over both finite and infinite horizons and shed new light on them it also facilitates the comparison of pomdps with respect to their degree of observability as a useful structural result

an algorithm for probabilistic planning abstract we define the probabilistic planning problem in terms of a probability distribution over initial world states a boolean combination of propositions representing the goal a probability threshold and actions whose effects depend on the execution time state of the world and on random chance adopting a probabilistic model complicates the definition of plan success instead of demanding a plan that provably achieves the goal we seek plans whose probability of success exceeds the threshold in this paper we present buridan an implemented least commitment planner that solves problems of this form we prove that the algorithm is both sound and complete we then explore buridan s efficiency by contrasting four algorithms for plan evaluation using a combination of analytic methods and empirical experiments we also describe the interplay between generating plans and evaluating them and discuss the role of search control in probabilistic planning

on the existence of stationary optimal policies for partially observed mdps under the long run average cost criterion abstract this paper studies the problem of the existence of stationary optimal policies for finite state controlled markov chains with compact action space and imperfect observations under the long run average cost criterion it presents sufficient conditions for existence of solutions to the associated dynamic programming equation that strengthen past results there is a detailed discussion comparing the different assumptions commonly found in the literature

an unsupervised texture segmentation algorithm with feature space reduction and knowledge feedback this paper presents an unsupervised texture segmentation algorithm based on feature extraction using multichannel gabor filtering it is shown that feature contrast a criterion derived for gabor filter parameter selection is well suited for feature coordinate weighting in order to reduce the feature space dimension the central idea of the proposed segmentation algorithm is to decompose the actual segmented image into disjunct areas called scrap images and use them after lowpass filtering as additional features for repeated k means clustering and minimum distance classification this yields a classification of texture regions with an improved degree of homogeneity while preserving precise texture boundaries

parallel application of block iterative methods in medical imaging and radiation therapy some row action algorithms which exploit special objective function and constraints structure have proven advantageous for solving huge and sparse feasibility or optimization problems recently developed block iterative versions of such special purpose methods enable parallel computation when the underlying problem is appropriately decomposed this opens the door for parallel computation in image reconstruction problems of computerized tomography and in the inverse problem of radiation therapy treatment planning all in their fully discretized modelling approach since there is more than one way of deriving block iterative versions of any row action method the choice has to be made with reference to the underlying real world problem

discrete gabor transform a feasible algorithm for implementing the gabor expansion the coefficients of which are computed by the discrete gabor transform dgt is presented for a given synthesis window and sampling pattern computing the auxiliary biorthogonal function of the dgt is nothing more than solving a linear system the dgt presented applies for both finite as well as infinite sequences by exploiting the nonuniqueness of the auxiliary biorthogonal function at oversampling an orthogonal like dgt is obtained as the discrete fourier transform dft is a discrete realization of the continuous time fourier transform similarly the dgt introduced provides a feasible vehicle to implement the useful gabor expansion

average cost dynamic programming equations for controlled markov chains with partial observations the value function for the average cost control of a class of partially observed markov chains is derived as the vanishing discount limit in a suitable sense of the value functions for the corresponding discounted cost problems the limiting procedure is justified by bounds derived using a simple coupling argument

remarks on the existence of solutions to the average cost optimality equation in markov decision processes necessary conditions are given for the existence of a bounded solution to the optimality equation arising in markov decision processes under a long run expected average cost criterion the relationships of some of our results to known sufficient conditions are also shown

state of the art a survey of partially observable markov decision processes theory models and algorithms this paper surveys models and algorithms dealing with partially observable markov decision processes a partially observable markov decision process pomdp is a generalization of a markov decision process which permits uncertainty regarding the state of a markov process and allows for state information acquisition a general framework for finite state and action pomdp s is presented next there is a brief discussion of the development of pomdp s and their relationship with other decision processes a wide range of models in such areas as quality control machine maintenance internal auditing learning and optimal stopping are discussed within the pomdp framework lastly algorithms for computing optimal solutions to pomdp s are presented

approximate receding horizon approach for markov decision processes average reward case abstract we consider an approximation scheme for solving markov decision processes mdps with countable state space finite action space and bounded rewards that uses an approximate solution of a fixed finite horizon sub mdp of a given infinite horizon mdp to create a stationary policy which we call approximate receding horizon control we first analyze the performance of the approximate receding horizon control for infinite horizon average reward under an ergodicity assumption which also generalizes the result obtained by white j oper res soc 33 1982 253 259 we then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub mdp the first control policy is based on a finite horizon approximation of howard s policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies along the study we also provide a simple alternative proof on the policy improvement for countable state space we finally discuss practical implementations of these schemes via simulation

a parametric class of discrete gabor expansions the gabor expansion and its discretization have been widely studied and many potential applications have been suggested in various signal processing problems a new approach to the study of the discrete gabor expansion dge is introduced and analyzed in detail using the theory of pseudoframe decompositions a parametric and analytical formula for a class of different gabor analysis sequences is derived it is a simple algebraic formula rather than another abstract system of equations for the first time the structure of analysis sequences is questioned we show that while there is a class of infinite analysis sequences that possess the gabor translation and complex modulation structure there are also infinite analysis sequences of arbitrary forms simulation results are provided to demonstrate the proposed algorithms the study of the dge by means of the theory of pseudoframe decompositions reveals a much broader mathematical perspective on the dge the general algorithm derived provides a feasible platform for optimizations in discrete gabor expansions arising from various applications this is an area that can surely be exploited as algorithms of dges become known and applications become more and more intensive

convergence theorems of common elements for equilibrium problems and fixed point problems in banach spaces the purpose of this paper is to introduce hybrid projection algorithms for finding a common element of the set of common fixed points of two quasi f nonexpansive mappings and the set of solutions of an equilibrium problem in the framework of banach spaces our results improve and extend the corresponding results announced by many others

contingent planning under uncertainty via stochastic satisfiability we describe a new planning technique that efficiently solves probabilistic propositional contingent planning problems by converting them into instances of stochastic satisfiability ssat and solving these problems instead we make fundamental contributions in two areas the solution of ssat problems and the solution of stochastic planning problems this is the first work extending the planning as satisfiability paradigm to stochastic domains our planner zander can solve arbitrary goal oriented finite horizon partially observable markov decision processes pomdps an empirical study comparing zander to seven other leading planners shows that its performance is competitive on a range of problems

strong convergence of shrinking projection methods for quasi φ nonexpansive mappings and equilibrium problems the purpose of this paper is to consider the convergence of a shrinking projection method for a finite family of quasi f nonexpansive mappings and an equilibrium problem strong convergence theorems are established in a uniformly smooth and strictly convex banach space which also enjoys the kadec klee property

a new computational search technique for ai based on the cantor set abstract this paper discusses the development of a new search algorithm for artificial intelligence systems the search technique is based on the mathematical theory of the cantor set the search algorithm will be efficient for specialized search domains where the distribution of the data elements to be searched is approximately normal the approach uses the iterative procedure of deleted middle thirds this facilitates quick pruning of a search space the new search technique has potential applications in computational systems with large input output data handling extensive experiments comparing the cantor set search css to binary search indicate that the new search technique holds good promise

honey pot constrained searching with local sensory information in this paper we investigate the problem of searching for a hidden target in a bounded region of the plane using an autonomous robot which is only able to use local sensory information the problem is naturally formulated in the continuous domain but the solution proposed is based on an aggregation refinement approach in which the continuous search space is partitioned into a finite collection of regions on which we define a discrete search problem a solution to the original problem is obtained through a refinement procedure that lifts the discrete path into a continuous one we show that the discrete optimization is computationally difficult np hard but there are computationally efficient approximation algorithms for solving it the resulting solution to the continuous problem is in general not optimal but one can construct bounds to gauge the cost penalty incurred due to i the discretization of the problem and ii the attempt to approximately solve the np hard problem in polynomial time numerical simulations show that the algorithms proposed behave significantly better than naive approaches such as a random walk or a greedy algorithm c 2006 elsevier ltd all rights reserved

discrete gabor structures and optimal representations the idea of gabor s 1946 signal expansion is to represent a signal in terms of a discrete set of time shifted and frequency modulated signals that are localized in the time frequency or phase space we present detailed descriptions of the block and banded structures for the gabor matrices based on the explicit descriptions of the sparsity of such matrices we can establish the sparse form of the gabor matrix and obtain the dual gabor atom mother wavelet the inverse of the gabor frame operator and carry out the discrete finite gabor transform in a very efficient way some explicit sufficient and also necessary conditions are derived for a gabor atom g to generate a gabor frame with respect to a tf lattice a b

a pseudo polynomial heuristic for path constrained discrete time markovian target search we propose a new heuristic for the single searcher path constrained discrete time markovian target search the algorithm minimizes an approximate instead of exact nondetection probability computed from the conditional probability that reflects the search history over the time windows of a fixed length l having a pseudo polynomial complexity it can solve in reasonable time the instances an order of magnitude larger than those solved in the previous studies by an asymptotic analysis relying on the fast mixing markov chain we show that the relative error of the approximation exponentially diminishes as l increases and the experimental results confirm the analysis the experiment also reveals a correlation very close to 1 between the approximate and exact nondetection probability of a search path this means that the heuristic produces near optimal search paths

moving target search a real time search for changing goals considers the case of heuristic search where the goal may change during the course of the search for example the goal may be a target that actively avoids the problem solver the authors present a moving target search algorithm mts to solve this problem the authors prove that if the average speed of the target is slower than that of the problem solver then the problem solver is guaranteed to eventually reach the target in a connected problem space the original mts algorithm was constructed with the minimum operations necessary to guarantee its completeness and hence is not very efficient to improve its efficiency the authors introduce ideas from the area of resource bounded planning into mts including 1 commitment to goals and 2 deliberation for selecting plans experimental results demonstrate that the improved mts is 10 to 20 times more efficient than the original mts in uncertain situations

robust transient signal detection using the oversampled gabor representation the gabor representation has been investigated previously as an effective and efficient means of detecting transient signals the major problem associated with the most commonly used form of the gabor representation is that it assumes that signals have integer arrival times and modulation frequencies an assumption that if violated causes signal miss match and deterioration in detector performance the present correspondence presents the concept of oversampling the gabor representation which makes the representation more robust to certain types of signal mismatch to show this the performance of an oversampled gabor detector is compared to that of critically sampled gabor detectors gabor coefficients are calculated in the oversampled representation via the least squares method which has been shown to be superior to the more commonly used biorthogonal function method in terms of corresponding detector performance

planning under time constraints in stochastic domains we provide a method based on the theory of markov decision processes for efficient planning in stochastic domains goals are encoded as reward functions expressing the desirability of each world state the planner must find a policy mapping from states to actions that maximizes future rewards standard goals of achievement as well as goals of maintenance and prioritized combinations of goals can be specified in this way an optimal policy can be found using existing methods but these methods require time at best polynomial in the number of states in the domain where the number of states is exponential in the number of propositions or state variables by using information about the starting state the reward function and the transition probabilities of the domain we restrict the planner s attention to a set of world states that are likely to be encountered in satisfying the goal using this restricted set of states the planner can generate more or less complete plans depending on the time it has available our approach employs several iterative refinement routines for solving different aspects of the decision making problem we describe the meta level control problem of em deliberation scheduling allocating computational resources to these routines we provide different models corresponding to optimization problems that capture the different circumstances and computational strategies for decision making under time constraints we consider em precursor models in which all decision making is performed prior to execution and em recurrent models in which decision making is performed in parallel with execution accounting for the states observed during execution and anticipating future states we describe experimental results for both the precursor and recurrent problems that demonstrate planning times that grow slowly as a function of domain size

an optimal branch and bound procedure for the constrained path moving target search problem a searcher and target move among a finite set of cells c 1 2 n in discrete time at the beginning of each time period one cell is searched if the target is in the selected cell j it is detected with probability qj if the target is not in the cell searched it cannot be detected during the current time period after each search a target in cell j moves to cell k with probability pjk the target transition matrix p pjk is known to the searcher the searcher s path is constrained in that if the searcher is currently in cell j the next search cell must be selected from a set of neighboring cells cj the object of the search is to minimize the probability of not detecting the target in t searches

multi task reinforcement learning in partially observable stochastic environments we consider the problem of multi task reinforcement learning mtrl in multiple partially observable stochastic environments we introduce the regionalized policy representation rpr to characterize the agent s behavior in each environment the rpr is a parametric model of the conditional distribution over current actions given the history of past actions and observations the agent s choice of actions is directly based on this conditional distribution without an intervening model to characterize the environment itself we propose off policy batch algorithms to learn the parameters of the rprs using episodic data collected when following a behavior policy and show their linkage to policy iteration we employ the dirichlet process as a nonparametric prior over the rprs across multiple environments the intrinsic clustering property of the dirichlet process imposes sharing of episodes among similar environments which effectively reduces the number of episodes required for learning a good policy in each environment when data sharing is appropriate the number of distinct rprs and the associated clusters the sharing patterns are automatically discovered by exploiting the episodic data as well as the nonparametric nature of the dirichlet process we demonstrate the effectiveness of the proposed rpr as well as the rpr based mtrl framework on various problems including grid world navigation and multi aspect target classification the experimental results show that the rpr is a competitive reinforcement learning algorithm in partially observable domains and the mtrl consistently achieves better performance than single task reinforcement learning

testing probabilistic equivalence through reinforcement learning checking if a given system implementation respects its specification is often done by proving that the two are equivalent the equivalence is chosen in particular for its computability and of course for its meaning that is for its adequacy with what is observable from the two systems implementation and specification trace equivalence is easily testable decidable from interaction but often considered too weak in contrast bisimulation is accepted as the canonical equivalence for interaction but it is not testable richer than an equivalence is a form of distance it is zero between equivalent systems and it provides an estimation of their difference if the systems are not equivalent our main contribution is to define such a distance in a context where 1 the two systems to be compared have a stochastic behavior 2 the model of one of them e g the implementation is unknown hence our only knowledge is obtained by interacting with it 3 consequently the target equivalence observed when distance is zero must be testable to overcome the problem that the model is unknown we use a reinforcement learning approach that provides powerful stochastic algorithms that only need to interact with the model our second main contribution is a new family of testable equivalences called k moment the weakest of them 1 moment equivalence is trace equivalence as k grows k moment equivalences become finer all remaining as well as their limit weaker than bisimulation we propose a framework to define and test a bigger class of testable equivalences test observation equivalences toes and we show how they can be made coarser or not by tuning some parameters

the optimal search for a moving target when the search path is constrained a search is conducted for a target moving in discrete time among a finite number of cells according to a known markov process the searcher must choose one cell in which to search in each time period the set of cells available for search depends upon the cell chosen in the last time period the problem is to find a searcher path i e a sequence of search cells that maximizes the probability of detecting the target in a fixed number of time periods we formulate the problem as a partially observable markov decision process and present a finite time horizon pomdp solution technique which is simpler than the standard linear programming methods

planning and acting in partially observable stochastic domains in this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains we begin by introducing the theory of markov decision processes mdps and partially observable mdps pomdps we then outline a novel algorithm for solving pomdps off line and show how in some cases a finite memory controller can be extracted from the solution to a pomdp we conclude with a discussion of how our approach relates to previous work the complexity of finding exact solutions to pomdps and of some possibilities for finding approximate solutions

optimal infinite horizon undiscounted control of finite probabilistic systems a finite input finite state finite output stochastic control problem with imperfect state observation and classical information pattern is shown to be meaningful as the horizon increases without bound and the discount rate approaches unity the plant model a finite probabilistic system includes the markov decision and partially observed markov decision problems as special cases under conditions resembling controllability and observability in linear systems we show that an optimal strategy exists it may be realized by a stationary policy on the state estimate its performance does not depend on the initial state distribution and convergence rates for its finite horizon and discounted performances are readily established

discrete sequential search with positive switch cost suppose one object is hidden in one of n boxes according to a known probability distribution p1 pn the boxes are to be searched sequentially the probability of overlooking the object if we search box k and if the object is in box k is denoted by qk the cost of a search of box k amounts to ck if we decide to search box k directly after box k we have to pay some extra cost called switch cost tk k the problem is under some reasonable assumptions to construct a strategy with minimal expected cost if t 0 the problem has a well known solution unfortunately there are many reasons which establish the conjecture that there does not exist an easy construction rule for optimal strategies in the general case we give necessary and sufficient conditions for the existence of optimal strategies which are ultimately periodic u p since we state bounds on the length not only of the transient phase but also of the period of optimal u p strategies we may construct these optimal strategies by comparing finitely many strategies if optimal u p strategies exist we prove some other results to reduce the number of strategies which we have to compare at last we present a numerical example

construction of approximation spaces for reinforcement learning linear reinforcement learning rl algorithms like least squares temporal difference learning lstd require basis functions that span approximation spaces of potential value functions this article investigates methods to construct these bases from samples we hypothesize that an ideal approximation spaces should encode diffusion distances and that slow feature analysis sfa constructs such spaces to validate our hypothesis we provide theoretical statements about the lstd value approximation error and induced metric of approximation spaces constructed by sfa and the state of the art methods krylov bases and proto value functions pvf in particular we prove that sfa minimizes the average over all tasks in the same environment bound on the above approximation error compared to other methods sfa is very sensitive to sampling and can sometimes fail to encode the whole state space we derive a novel importance sampling modification to compensate for this effect finally the lstd and least squares policy iteration lspi performance of approximation spaces constructed by krylov bases pvf sfa and pca is compared in benchmark tasks and a visual robot navigation experiment both in a realistic simulation and with a robot the results support our hypothesis and suggest that i sfa provides subspace invariant features for mdps with self adjoint transition operators which allows strong guarantees on the approximation error ii the modified sfa algorithm is best suited for lspi in both discrete and continuous state spaces and iii approximation spaces encoding diffusion distances facilitate lspi performance

optimal stochastic switching under convexity assumptions we address a method of approximate calculation of optimal control policy applicable to a particular class of stochastic control problems whose stochastic dynamics exhibit a certain convexity preserving property problems of this type appear in many applications and encompass important examples arising in the area of optimal stopping and in the framework of control based on partial observations utilizing this specific structure we suggest a numerical method which enjoys a number of desirable properties in particular we work out a remarkably strong method for calculation of the value function within our numerically well trackable approach we show a convergence to the value function of the original problem uniformly on compact sets this issue can be of great advantage particularly for high dimensional control problem where the only competing methods from least squares monte carlo family are able to serve merely l p convergence under several restrictions since the presented algorithm is simpl

strong convergence of a proximal type algorithm in a banach space in this paper we study strong convergence of the proximal point algorithm it is known that the proximal point algorithm converges weakly to a solution of a maximal monotone operator but it fails to converge strongly then in math program 87 2000 pp 189 202 solodov and svaiter introduced the new proximal type algorithm to generate a strongly convergent sequence and established a convergence property for it in hilbert spaces our purpose is to extend solodov and svaiter s result to more general banach spaces using this we consider the problem of finding a minimizer of a convex function

generalized gabor expansions of discrete time signals in l sup 2 z via biorthogonal like sequences in this paper a biorthogonal like sequences bls theory and its application to the generalized gabor expansions equivalently the generalized short time fourier transform filterbank summation are presented a pair of bls s are defined to be two sequences satisfying a biorthogonal like condition blc which is a moment equation and equivalent to a linear difference equation we show that two collections in a hilbert space generated by a pair of bls s in the joint time frequency domain are complete either can be used as an analysis filter and the other can be used as a synthesis filter for a generalized gabor expansion of discrete time signals a sufficient and necessary condition on the existence of bls s based on the moment equation is presented which is simpler to use than frame theory given a filter generating a frame its bls s also generate frames the dual frame is one of them given a fir analysis synthesis filter there is a fir synthesis analysis filter if bls s exist the algorithm to compute fir analysis and synthesis filters based on the linear difference equation is presented in this paper which is simpler than frame operator

an iterative method for total quasi φ asymptotically nonexpansive semi groups and generalized mixed equilibrium problems in banach spaces in this paper we introduce an iterative scheme by the modified halpern mann type for total quasi asymptotically nonexpansive semi groups and prove the strong convergence theorem using our new iterative process which converges strongly to a common element of the set of common fixed points for total quasi asymptotically nonexpansive semi groups and the set of solutions of generalized mixed equilibrium problem in a uniformly smooth and strictly convex banach space with the kadec klee property using the properties of the generalized f projection operator our results extend and improve the corresponding recent results in the literature

the undersampled discrete gabor transform conventional studies on discrete gabor transforms have generally been confined to the cases of critical sampling and oversampling in which the gabor families span the whole signal space in this paper we investigate undersampled discrete gabor transforms for an undersampled gabor triple g a b i e a spl middot b n we show that the associated generalized dual gabor window gdgw function is the same as the one associated with the oversampled g n b n a except for the constant factor ab n computations of undersampled gabor transforms are made possible by applying the methods algorithms developed in oversampled settings the undersampled gdgw is determined then we are able to obtain the best approximation of a signal x by linear combinations of vectors in the gabor family

discrete zak transforms polyphase transforms and applications we consider three different versions of the zak 1967 transform zt for discrete time signals namely the discrete time zt the polyphase transform and a cyclic discrete zt in particular we show that the extension of the discrete time zt to the complex z plane results in the polyphase transform an important and well known concept in multirate signal processing and filter bank theory we discuss fundamental properties relations and transform pairs of the three discrete zt versions and we summarize applications of these transforms in particular the discrete time zt and the cyclic discrete zt are important for discrete time gabor 1946 expansion weyl heisenberg frame theory since they diagonalize the weyl heisenberg frame operator for critical sampling and integer oversampling the polyphase representation plays a fundamental role in the theory of filter banks especially dft filter banks simulation results are presented to demonstrate the application of the discrete zt to the efficient calculation of dual gabor windows tight gabor windows and frame bounds

efficient calculation of finite gabor transforms the gabor transform may be viewed as a collection of localized fourier transforms and as such is useful for analysis of nonstationary signals and images we present a new approach to analyzing the gabor transform and use it to study the various critically sampled discretizations that form the infinite discrete periodic finite discrete and nonperiodic finite discrete versions of the transform in particular we distinguish between the analysis and synthesis forms of the transform and introduce an intermediate operation that decomposes both forms into collections of independent toeplitz operators in the continuous the infinite discrete and the periodic finite discrete cases this decomposition allows us to show that for appropriate windows the analysis and synthesis transforms are inverses of each other in the nonperiodic finite discrete case this relation no longer holds but we are still able to use the decomposition and results on toeplitz matrices to show that both the transform and the inverse transform of p discrete samples are computable in o p log p operations after a setup cost of o plog sup 2 p furthermore we use the decomposition to study in detail the differences between the periodic and nonperiodic versions of the transform and to compare their conditioning

bayesian policy gradient and actor critic algorithms policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate many conventional policy gradient methods use monte carlo techniques to estimate this gradient the policy is improved by adjusting the parameters in the direction of the gradient estimate since monte carlo methods tend to have high variance a large number of samples is required to attain accurate estimates resulting in slow convergence in this paper we first propose a bayesian framework for policy gradient based on modeling the policy gradient as a gaussian process this reduces the number of samples needed to obtain accurate gradient estimates moreover estimates of the natural gradient as well as a measure of the uncertainty in the gradient estimates namely the gradient covariance are provided at little extra cost since the proposed bayesian framework considers system trajectories as its basic observable unit it does not require the dynamics within trajectories to be of any particular form and thus can be easily extended to partially observable problems on the downside it cannot take advantage of the markov property when the system is markovian to address this issue we proceed to supplement our bayesian policy gradient framework with a new actor critic learning model in which a bayesian class of non parametric critics based on gaussian process temporal difference learning is used such critics model the action value function as a gaussian process allowing bayes rule to be used in computing the posterior distribution over action value functions conditioned on the observed data appropriate choices of the policy parameterization and of the prior covariance kernel between action values allow us to obtain closed form expressions for the posterior distribution of the gradient of the expected return with respect to the policy parameters we perform detailed experimental comparisons of the proposed bayesian policy gradient and actor critic algorithms with classic monte carlo based policy gradient methods as well as with each other on a number of reinforcement learning problems

control limits for two state partially observable markov decision processes abstract a production process can be in either a good or a bad state the true state is unknown and can only be inferred from observations if the state is good during one period it may deteriorate and become bad during the next period two actions are available continue or replace for a fixed cost the objective is to maximize the expected discounted value of the total future profits we prove that dominance in expectation the expected profit is larger in the good state than in the bad state suffices for the optimal policy to be of a control limit clt type continue if and only if the good state probability exceeds the clt this condition is weaker than stochastic dominance which has been prevailing we also show that the expected profit function is convex strictly increasing

unsupervised texture segmentation of images using tuned matched gabor filters recent studies have confirmed that the multichannel gabor decomposition represents an excellent tool for image segmentation and boundary detection unfortunately this approach when used for unsupervised image analysis tasks imposes excessive storage requirements due to the nonorthogonality of the basis functions and is computationally highly demanding in this correspondence we propose a novel method for efficient image analysis that uses tuned matched gabor filters the algorithmic determination of the parameters of the gabor filters is based on the analysis of spectral feature contrasts obtained from iterative computation of pyramidal gabor transforms with progressive dyadic decrease of elementary cell sizes the method requires no a priori knowledge of the analyzed image so that the analysis is unsupervised computer simulations applied to different classes of textures illustrate the matching property of the tuned gabor filters derived using our determination algorithm also their capability to extract significant image information and thus enable an easy and efficient low level image analysis will be demonstrated

conformant plans and beyond principles and complexity conformant planning is used to refer to planning for unobservable problems whose solutions like classical planning are linear sequences of operators called linear plans the term conformant is automatically associated with both the unobservable planning model and with linear plans mainly because the only possible solutions for unobservable problems are linear plans in this paper we show that linear plans are not only meaningful for unobservable problems but also for partially observable problems in such case the execution of a linear plan generates observations from the environment which must be collected by the agent during the execution of the plan and used at the end in order to determine whether the goal had been achieved or not this is the typical case in problems of diagnosis in which all the actions are knowledge gathering actions thus there are substantial differences about linear plans for the case of unobservable or fully observable problems and for the case of partially observable problems while linear plans for the former model must conform with properties in state space linear plans for partially observable problems must conform with properties in belief space this differences surface when the problems are allowed to express epistemic goals and conditions using modal logic and place the plan existence decision problem in different complexity classes linear plans is one extreme point in a discrete spectrum of solution forms for planning problems the other extreme point is contingent plans in which there is a branch point for every possible observation at each time step and thus the number of branch points is not bounded a priori in the middle of the spectrum there are plans with a bounded number of branch points thus linear plans are plans with zero branch points and contingent plans are plans with unbounded number of branch points in this work we lay down foundations and principles for the general treatment of linear plans and plans of bounded branching and provide exact complexity results for novel decision problems we also show that linear plans for partially observable problems are not only of theoretical interest since some challenging real life problems can be dealt with them

discrete multiwindow gabor type transforms the discrete finite gabor scheme is generalized by incorporating multiwindows two approaches are presented for the analysis of the multiwindow scheme the signal domain approach and the zak transform domain approach issues related to undersampling critical sampling and oversampling are considered the analysis is based on the concept of frames and on generalized moore penrose inverses the approach based on representing the frame operator as a matrix valued function is far less demanding from a computational complexity viewpoint than a straightforward matrix algebra in various operations such as the computation of the dual frame dft based algorithms including complexity analysis are presented for the calculation of the expansion coefficients and for the reconstruction of the signal in both signal and transform domains the scheme is further generalized and incorporates kernels other than the complex exponential representations other than those based on the dual frame and nonrectangular sampling of the combined space are considered as well an example that illustrates the advantages of the multiwindow scheme over the single window scheme is presented

on hybrid projection methods for asymptotically quasi φ nonexpansive mappings the purpose of this paper is to consider the problem of approximating a common fixed point of two asymptotically quasi f nonexpansive mappings based on hybrid projection methods strong convergence theorems are established in a real banach space

the complexity of decentralized control of markov decision processes we consider decentralized control of markov decision processes and give complexity bounds on the worst case running time for algorithms that find optimal solutions generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described for even two agents the finite horizon problems corresponding to both of these models are hard for nondeterministic exponential time these complexity results illustrate a fundamental difference between centralized and decentralized control of markov decision processes in contrast to the problems involving centralized control the problems we considerprovably do not admit polynomial time algorithms furthermore assuming exp nexp the problems require superexponential time to solve in the worst case

a modified hybrid projection method for solving generalized mixed equilibrium problems and fixed point problems in banach spaces in this paper we introduce a modified new hybrid projection method for finding the set of solutions of the generalized mixed equilibrium problems and the convex feasibility problems for an infinite family of closed and uniformly quasi asymptotically nonexpansive mappings strong convergence theorems are established in a uniformly smooth and strictly convex banach space which also enjoys the kadec klee property our results improve and extend the corresponding results announced by qin et al 2010 and many authors

computationally attractive real gabor transforms we present a gabor transform for real discrete signals and present a computationally attractive method for computing the transform for the critically sampled case we derive a biorthogonal function which is very localized in the time domain therefore truncation of this biorthogonal function allows us to compute approximate expansion coefficients with significantly reduced computational requirements further truncation does not degrade the numerical stability of the transform we present a tight upper bound on the reconstruction error incurred due to use of a truncated biorthogonal function and summarize computational savings for example the expense of transforming a length 2048 signal using length 16 blocks is reduced by a factor of 26 over similar fft based methods with at most 0 04 squared error in the reconstruction

ordered solutions for dynamic programs suppose x indexes a collection of optimization problems in rn and x is an optimal program for the problem indexed by x x sufficient conditions are presented for two other families of optimization problems to possess optimal programs 0x and 0x for the respective problems when x x such that 0x x 0x for all x x this result is then applied to obtain bounds on an optimal policy in the class of dynamic programs which are bounded monotone contractions

a hybrid iteration scheme for equilibrium problems variational inequality problems and common fixed point problems in banach spaces abstract in this paper we introduce an iterative process which converges strongly to a common element of a set of common fixed points of finite family of closed relatively quasi nonexpansive mappings the solution set of generalized equilibrium problem and the solution set of the variational inequality problem for an inverse strongly monotone mapping in banach spaces

search for a randomly moving object after a brief discussion of the operational origin of search problems the mathematical problem is formulated the mathematical quantity of interest is the joint density for location of the object sought and unsuccessful search when the object moves according to a diffusion process this joint density satisfies a parabolic equation after the introduction of scaled variables the search equation can be approximately solved by the ray method the interpretation of the terms in the approximate solution is discussed the case of constant diffusion and drift parameters and piecewise linear searching paths arises often in operational situations this case is considered in detail

optimal strategy of route and look for the path constrained search problem with reward criterion abstract a target is moving among a finite number of cells k 1 k in discrete time t 1 t knowing probability of the target s path selection a searcher is searching for the target with constraints that he can move from cell i to one of the adjacent cells i i the searcher detects the target with the probability pi on the look into cell i in which both the target and the searcher are there he gains a value v t on the detection of the target at time t but expends cost c0 i t on the search in cell i at t the searcher selects his route and determines whether he looks into his current cell on the route or not in this paper our purpose is to find an optimal strategy for the route and the look of the searcher which maximizes the expected reward defined as the expected value minus the expected cost the problem is formulated as an integer programming problem for solving the problem we use a branch and bound procedure with an upper bound estimation

a two state partially observable markov decision process with uniformly distributed observations a controller observes a production system periodically over time if the system is in the good state during one period there is a constant probability that it will deteriorate and be in the bad state during the next period and remains there the true state of the system is unobservable and can only be inferred from observations quality of output two actions are available continue or replace for a fixed cost the objective is to maximize the expected discounted value of the total future income for both the finite and infinite horizon problems the optimal policy is of a control limit clt type continue if the good state probability exceeds the clt and replace otherwise the computation of the clt involves a functional equation an analytical solution for this equation is as yet unknown for uniformly distributed observations we obtain the infinite horizon clt analytically we also show that the finite horizon clts as a function of the time remaining are not necessarily monotone which is counterintuitive