analysis of an adaptive control scheme for a partially observed controlled markov chain the authors consider an adaptive finite state controlled markov chain with partial state information motivated by a class of replacement problems they present parameter estimation techniques based on the information available after actions that reset the state to a known value are taken it is proved that the parameter estimates converge w p 1 to the true unknown parameter under the feedback structure induced by a certainty equivalent adaptive policy it is shown that the adaptive policy is self optimizing in a long run average sense for any measurable sequence of parameter estimates converging w p 1 to the true parameter

the helmholtz machine discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning one fruitful approach is to build a parameterized stochastic generative model independent draws from which are likely to produce the patterns for all but the simplest generative models each pattern can be generated in exponentially many ways it is thus intractable to adjust the parameters to maximize the probability of the observed patterns we describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations our method can be viewed as a form of hierarchical self supervised learning that may relate to the function of bottom up and top down cortical processing pathways

asymptotically efficient adaptive allocation schemes for controlled markov chains finite parameter space the authors consider a controlled markov chain whose transition probabilities and initial distribution are parametrized by an unknown parameter theta belonging to some known parameter space theta there is a one step reward associated with each pair of control and the following state of the process the objective is to maximize the expected value of the sum of one step rewards over an infinite horizon the loss associated with a control scheme at a parameter value is the function of time giving the difference between the maximum reward that could have been achieved if the parameter were known and the reward achieved by the scheme since it is impossible to minimize the loss uniformly for all parameter values the authors define uniformly good adaptive control schemes and restrict attention to these schemes they develop a lower bound on the loss associated with any uniformly good control scheme they construct an adaptive control scheme whose loss equals the lower bound for every parameter value and is therefore asymptotically efficient

transmission of vector quantized data over a noisy channel in the transmission of vector quantized data the vector quantizer and the communication system are usually designed separately with such an approach the channel noise results in significant degradations in the performance of the vector quantizer to solve this problem we should properly create the mapping from the codebook of the quantizer to the channel signal set of the communication system this paper proposes a new approach to construct such a mapping based on the ordering property of the self organizing feature map sofm we use the neighborhood structure of the sofm and the neighborhood structure of the channel signal set to construct the mapping simulation results confirm that the proposed approach is robust with respect to channel noise

yet another algorithm which can generate topography map this paper presents an algorithm to form a topographic map resembling to the self organizing map the idea stems on defining an energy function which reveals the local correlation between neighboring neurons the larger the value of the energy function the higher the correlation of the neighborhood neurons on this account the proposed algorithm is defined as the gradient ascent of this energy function simulations on two dimensional maps are illustrated

optimal control of hidden markov models with binary observations a hidden markov model hmm consisting of a controlled markov chain and a binary output is considered the binary output reports the occurrence of a certain state versus any other state in the markov chain the transition matrix of the markov chain depends on a control vector which is to be chosen to minimize a discounted infinite horizon cost function the information provided for decision making is the entire history of the binary output which presents an incomplete knowledge of the markov chain and as a consequence leads to a stochastic control problem with partial observations the solution to such a problem is a control policy consisting of two components an estimator to derive the posterior probability distribution of the markov chain from the observation set and a nonlinear control law to map this estimate into the control vector this paper introduces a nonlinear state space representation for the estimator develops analytical expressions for the control law and presents numerical methods for efficient computation of the optimal control application of these results to optimal management of a class of inventory systems is discussed and the performance of optimal control for this application is examined by numerical simulations

gibbs sampling methods for stick breaking priors a rich and flexible class of random probability measures which we call stick breaking priors can be constructed using a sequence of independent beta random variables examples of random measures that have this characterization include the dirichlet process its two parameter extension the two parameter poisson dirichlet process finite dimensional dirichlet priors and beta two parameter processes the rich nature of stick breaking priors offers bayesians a useful class of priors for nonparametric problems while the similar construction used in each prior can be exploited to develop a general computational procedure for fitting them in this article we present two general types of gibbs samplers that can be used to fit posteriors of bayesian hierarchical models based on stick breaking priors the first type of gibbs sampler referred to as a polya urn gibbs sampler is a generalized version of a widely used gibbs sampling method currently employed for dirichlet process computing this method applies t

adaptive control of constrained finite markov chains an adaptive control algorithm is presented for constrained finite controlled markov chains with unknown transition probabilities a finite set of algebraic constraints has been considered the lagrange multipliers approach is used to solve this constrained optimization problem this scheme is such that at each time n estimates the control policy on the basis on bush mosteller scheme which is related to stochastic approximation procedures we present the asymptotic properties convergence and order of convergence rate of the algorithm they follow from the law of dependent large numbers martingales theory and lyapunov function analysis approaches

emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces olshausen and field 1996 applied the principle of independence maximization by sparse coding to extract features from natural images this leads to the emergence of oriented linear filters that have simultaneous localization in space and in frequency thus resembling gabor functions and simple cell receptive fields in this article we show that the same principle of independence maximization can explain the emergence of phase and shift invariant features similar to those found in complex cells this new kind of emergence is obtained by maximizing the independence between norms of projections on linear subspaces instead of the independence of simple linear filter outputs the norms of the projections on such independent feature subspaces then indicate the values of invariant features

action time sharing policies for ergodic control of markov chains ergodic control for discrete time controlled markov chains with a locally compact state space and a compact action space is considered under suitable stability irreducibility and feller continuity conditions a flexible family of controls called action time sharing ats policies associated with a given continuous stationary markov control is introduced it is shown that the long term average cost for such a control policy for a broad range of one stage cost functions is the same as that for the associated stationary markov policy in addition ats policies are well suited for a range of estimation information collection and adaptive control goals to illustrate the possibilities we present two examples the first demonstrates a construction of an ats policy that leads to consistent estimators for unknown model parameters while producing the desired long term average cost value the second example considers a setting where the target stationary markov control q is not known but there are sampling schemes available that allow for consistent estimation of q we construct an ats policy which uses dynamic estimators for q for control decisions and show that the associated cost coincides with that for the unknown markov control q

trellis coded vector quantization a vector generalization of trellis coded quantization tcq called trellis coded vector quantization tcvq and experimental results showing its performance for an i i d gaussian source are presented for a given rate tcvq yields a lower distortion that tcq at the cost of an increase in implementation complexity in addition tcvq allows fractional rates which tcq does not

near optimal regret bounds for reinforcement learning for undiscounted reinforcement learning in markov decision processes mdps we consider the total regret of a learning algorithm with respect to an optimal policy in order to describe the transition structure of an mdp we propose a new parameter an mdp has diameter d if for any pair of states s s there is a policy which moves from s to s in at most d steps on average we present a reinforcement learning algorithm with total regret o ds at after t steps for any unknown mdp with s states a actions per state and diameter d a corresponding lower bound of dsat on the total regret of any learning algorithm is given as well these results are complemented by a sample complexity bound on the number of suboptimal steps taken by our algorithm this bound can be used to achieve a gap dependent regret bound that is logarithmic in t finally we also consider a setting where the mdp is allowed to change a fixed number of l times we present a modification of our algorithm that is able to deal with this setting and show a regret bound of o l1 3t2 3ds a

variational message passing bayesian inference is now widely established as one of the principal foundations for machine learning in practice exact inference is rarely possible and so a variety of approximation techniques have been developed one of the most widely used being a deterministic framework called variational inference in this paper we introduce variational message passing vmp a general purpose algorithm for applying variational inference to bayesian networks like belief propagation vmp proceeds by sending messages between nodes in the network and updating posterior beliefs using local operations at each node each such update increases a lower bound on the log evidence unless already at a local maximum in contrast to belief propagation vmp can be applied to a very general class of conjugate exponential models because it uses a factorised variational approximation furthermore by introducing additional variational parameters vmp can be applied to models containing non conjugate distributions the vmp framework also allows the lower bound to be evaluated and this can be used both for model comparison and for detection of convergence variational message passing has been implemented in the form of a general purpose inference engine called vibes variational inference for bayesian networks which allows models to be specified graphically and then solved variationally without recourse to coding

code vector density in topographic mappings scalar case the author derives some new results that build on his earlier work 1989 of combining vector quantization vq theory and topographic mapping tm theory a vq model with a noisy transmission medium is used to model the processes that occur in tms which leads to the standard tm training algorithm albeit with a slight modification to the encoding process to emphasize this difference the model is called a topographic vector quantizer tvq in the continuum limit of the one dimensional scalar tvq it is found that the density of code vectors is proportional to p x sup a alpha 1 3 assuming that the transmission medium introduces additive noise with a zero mean symmetric monotically decreasing probability density this result is dramatically different from the result that is predicted when the standard tm training algorithm is used with a uniform symmetric neighborhood n n and it is noted that this difference arises entirely from using minimum distortion rather than nearest neighbor encoding

a stochastic self organizing map for proximity data we derive an efficient algorithm for topographic mapping of proximity data tmp which can be seen as an extension of kohonen s self organizing map to arbitrary distance measures the tmp cost function is derived in a baysian framework of folded markov chains for the description of autoencoders it incorporates the data by a dissimilarity matrix and the topographic neighborhood by a matrix of transition probabilities from the principle of maximum entropy a nonfactorizing gibbs distribution is obtained which is approximated in a mean field fashion this allows for maximum likelihood estimation using an expectation maximization algorithm in analogy to the transition from topographic vector quantization to the self organizing map we suggest an approximation to tmp that is computationally more efficient in order to prevent convergence to local minima an annealing scheme in the temperature parameter is introduced for which the critical temperature of the first phase transition is calculated in terms o

smoothly distributed fuzzy c means this paper presents a new self organizing map algorithm unlike the well known method of kohonen the new algorithm corresponds to the optimization of an unambiguously defined cost function it consists of a modified version of the widely used fuzzy c means functional where the code vectors are distributed on a regular low dimensional grid and a penalization term is added in order to guarantee a smooth distribution for the values of the code vectors on the grid the mapping properties of the new method similar to those of kohonen s algorithm are illustrated with several data sets computer programs source code and executables and data are available upon request to the authors

asymptotically efficient adaptive choice of control laws incontrolled markov chains we consider a controlled markov chain on a general state space whose transition probabilities are parameterized by an unknown parameter belonging to a compact metric space there is a one step reward associated with each pair of control and the following state of the process given a finite set of stationary control laws under each of which the markov chain is uniformly recurrent an optimal control law is this set is one that maximizes the long run average reward in ignorance of the parameter value we construct an adaptive control rule which uses the optimal control law s at a relative frequency of 1 o n 1 log n and show that this relative frequency gives an asymptotically optimal balance between the control objective and the amount of information needed to learn about the unknown parameter the basic idea underlying this construction is to introduce suitable uncertainty adjustments via sequential testing theory into the certainty equivalence rule thus resolving the apparent dilemma between control and information

a bayesian analysis of self organizing maps in this paper bayesian methods are used to analyze some of the properties of a special type of markov chain the forward transitions through the chain are followed by inverse transitions using bayes theorem backward through a copy of the same chain this will be called a folded markov chain if an appropriately defined euclidean error between the original input and its reconstruction via bayes theorem is minimized with respect to the choice of markov chain transition probabilities then the familiar theories of both vector quantizers and self organizing maps emerge this approach is also used to derive the theory of self supervision in which the higher layers of a multilayer network supervise the lower layers even though overall there is no external teacher

high rate vector quantization for noisy channels with applications to wideband speech spectrum compression this paper considers the high rate performance of source coding for noisy discrete symmetric channels with random index assignment ia accurate analytical models are developed to characterize the expected distortion performance of vector quantization vq for a large class of distortion measures it is shown that when the point density is continuous the distortion can be approximated as the sum of the source quantization distortion and the channel error induced distortion expressions are also derived for the continuous point density that minimizes the expected distortion next for the case of mean squared error distortion a more accurate analytical model for the distortion is derived by allowing the point density to have a singular component the extent of the singularity is also characterized these results provide analytical models for the expected distortion performance of both conventional vq as well as for channel optimized vq as a practical example compression of the linear predictive coding parameters in the wideband speech spectrum is considered with the log spectral distortion as performance metric the theory is able to correctly predict the channel error rate that is permissible for operation at a particular level of distortion

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

on the performance and complexity of channel optimized vector quantizers the performance and complexity of channel optimized vector quantizers are studied for the gauss markov source observations on the geometric structure of these quantizers are made which have an important implication on the encoding complexity for the squared error distortion measure it is shown that an operation equivalent to a euclidean distance measurement with respect to an appropriately defined set of points used to identify the encoding regions can be used to perform the encoding this implies that the encoding complexity is proportional to the number of encoding regions it is then demonstrated that for very noisy channels and a heavily correlated source when the codebook size is large the number of encoding regions is considerably smaller than the codebook size implying a reduction in encoding complexity

hierarchical dirichlet processes we consider problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups we assume that the number of mixture components is unknown a priori and is to be inferred from the data in this setting it is natural to consider sets of dirichlet processes one for each group where the well known clustering property of the dirichlet process provides a nonparametric prior for the number of mixture components within each group given our desire to tie the mixture models in the various groups we consider a hierarchical model specifically one in which the base measure for the child dirichlet processes is itself distributed according to a dirichlet process such a base measure being discrete the child dirichlet processes necessarily share atoms thus as desired the mixture models in the different groups necessarily share mixture components we discuss representations of hierarchical dirichlet processes

a theory for multiresolution signal decomposition the wavelet representation multiresolution representations are effective for analyzing the information content of images the properties of the operator which approximates a signal at a given resolution were studied it is shown that the difference of information between the approximation of a signal at the resolutions 2 sup j 1 and 2 sup j where j is an integer can be extracted by decomposing this signal on a wavelet orthonormal basis of l sup 2 r sup n the vector space of measurable square integrable n dimensional functions in l sup 2 r a wavelet orthonormal basis is a family of functions which is built by dilating and translating a unique function psi x this decomposition defines an orthogonal multiresolution representation called a wavelet representation it is computed with a pyramidal algorithm based on convolutions with quadrature mirror filters wavelet representation lies between the spatial and fourier domains for images the wavelet representation differentiates several spatial orientations the application of this representation to data compression in image coding texture discrimination and fractal analysis is discussed

kullback leibler upper confidence bounds for optimal sequential allocation we consider optimal sequential allocation in the context of the so called stochastic multi armed bandit model we describe a generic index policy in the sense of gittins 1979 based on upper confidence bounds of the arm payoffs computed using the kullback leibler divergence we consider two classes of distributions for which instances of this general idea are analyzed the kl ucb algorithm is designed for one parameter exponential families and the empirical kl ucb algorithm for bounded and finitely supported distributions our main contribution is a unified finite time analysis of the regret of these algorithms that asymptotically matches the lower bounds of lai and robbins 1985 and burnetas and katehakis 1996 respectively we also investigate the behavior of these algorithms when used with general bounded rewards showing in particular that they provide significant improvements over the state of the art

asymptotically efficient adaptive strategies in repeated games part ii asymptotic optimality this paper continues the analysis of a dynamic decision problem modeled as a two person repeated game with random rewards perfect observations and incomplete information on one side the emphasis is on strategies of player 1 the uninformed player which maximize his worst case total reward in a strong non bayesian sense namely for all possible states of nature an asymptotic bound on performance is first established followed by the construction of strategies which achieve this bound the analysis highlights the efficient acquisition of statistical information under conflict conditions and especially the relations between information and payoff which are inherent in this problem

joint trellis coded quantization modulation a joint source channel coding system constructed using trellis coded quantization tcq and trellis coded modulation tcm is described identical trellises are used in the tcq and tcm systems and a straightforward mapping of tcq codewords to tcm symbols is presented which guarantees that euclidean squared distance in the channel is commensurate with quantization mean square error mse hence likely tcm error events of small euclidean distance cause only a small increase in overall source coding mse

normal bandits of unknown means and variances consider the problem of sampling sequentially from a finite number of n 2 populations specified by random variables xki i 1 n and k 1 2 where xki denotes the outcome from population i the kth time it is sampled it is assumed that for each fixed i xki k 1 is a sequence of i i d normal random variables with unknown mean i and unknown variance i2 the objective is to have a policy for deciding from which of the n populations to sample from at any time t 1 2 so as to maximize the expected sum of outcomes of n total samples or equivalently to minimize the regret due to lack on information of the parameters i and i2 in this paper we present a simple inflated sample mean ism index policy that is asymptotically optimal in the sense of theorem 4 below this resolves a standing open problem from burnetas and katehakis 1996b additionally finite horizon regret bounds are given

markov decision processes with imprecise transition probabilities we present new numerical algorithms and bounds for the infinite horizon discrete stage finite state and action markov decision process with imprecise transition probabilities we assume that the transition probability mass vector for each state and action is described by a finite number of linear inequalities this model of imprecision appears to be well suited for describing statistically determined confidence limits and or natural language statements of likelihood the numerical procedures for calculating an optimal max min strategy are based on successive approximations reward revision and modified policy iteration the bounds that are determined are at least as tight as currently available bounds for the case where the transition probabilities are precise

an error control scheme for transmission of vector quantization data over noisy channels this paper presents an error control scheme for transmitting vector quantized data over noisy channels first we review a self organizing feature map sofm based approach to construct the mapping from the codebook of the quantizer to the channel signal set of the communication system this approach is robust with respect to channel noise even if we do not use any error control scheme in the communication system afterwards a new trellis type quantizer namely the trellis coded kohonen 1984 map tckm is presented the design process of the tckm which is based on the neighborhood structure of sofms is simpler than that of conventional trellis coded vector quantizers tcvqs simulation results show that the performance of tckms is comparable with that of tcvqs last we introduce the error control scheme based on the concepts of the above two applications of sofms simulation results show that the proposed error control scheme is more robust with respect to channel noise the advantage of our approaches mentioned above is that the design processes of transmission systems are predefined before the construction of the codebook thus different codebooks with the same neighborhood structure can share the same design

the multi armed bandit problem decomposition and computation this paper is dedicated to our friend and mentor cyrus derman on the occasion of his 60th birthday the multi armed bandit problem arises in sequentially allocating effort to one of n projects and sequentially assigning patients to one of n treatments in clinical trials gittins and jones gittins j c jones d m 1974 a dynamic allocation index for the sequential design of experiments j gani k sarkadi l vince eds progress in statistics european meeting of statisticians 1972 1 north holland amsterdam 241 266 have shown that one optimal policy for the n project problem an n dimensional discounted markov decision chain is determined by the following largest index rule there is an index for each state of each given project that depends only on the data of that project in each period one allocates effort to a project with largest current index the purpose of this paper is to give a short proof of this result and a new characterization of the index of a project in state i viz as the maximum expected present value in state i for the restart in i problem in which in each state and period one either continues allocating effort to the project or immediately restarts the project in state i moreover it is shown that an approximate largest index rule yields an approximately optimal policy these results lead to more efficient methods of computing the indices on line and or for sparse transition matrices in large state spaces than have been suggested heretofore by using a suitable implementation of successive approximations a policy whose expected present value is within 100e of the maximum possible range of values of the indices can be found on line with at most n t 1tm operations where m is the number of operations required to calculate one approximation t is the least integer majorizing the ratio ln e ln a and 0 a 1 is the discount factor

infinite hidden markov models for unusual event detection in video we address the problem of unusual event detection in a video sequence invariant subspace analysis isa is used to extract features from the video and the time evolving properties of these features are modeled via an infinite hidden markov model ihmm which is trained using ldquonormalrdquo ldquotypicalrdquo video the ihmm retains a full posterior density function on all model parameters including the number of underlying hmm states anomalies unusual events are detected subsequently if a low likelihood is observed when associated sequential features are submitted to the trained ihmm a hierarchical dirichlet process framework is employed in the formulation of the ihmm the evaluation of posterior distributions for the ihmm is achieved in two ways via markov chain monte carlo and using a variational bayes formulation comparisons are made to modeling based on conventional maximum likelihood based hmms as well as to dirichlet process based gaussian mixture models

a study of vector quantization for noisy channels several issues related to vector quantization for noisy channels are discussed an algorithm based on simulated annealing is developed for assigning binary codewords to the vector quantizer code vectors it is shown that this algorithm could result in dramatic performance improvements as compared to randomly selected codewords a modification of the simulated annealing algorithm for binary codeword assignment is developed for the case where the bits in the codeword are subjected to unequal error probabilities resulting from unequal levels of error protection an algorithm for the design of an optimal vector quantizer for a noisy channel is briefly discussed and its robustness under channel mismatch conditions is studied numerical results for a stationary first order gauss markov source and a binary symmetric channel are provided it is concluded that the channel optimized vector quantizer design algorithm if used carefully can result in a fairly robust system with no additional delay the case in which the communication channel is nonstationary as in mobile radio channels is studied and some preliminary ideas for quantizer design are presented

image coding using vector quantization a review a review of vector quantization techniques used for encoding digital images is presented first the concept of vector quantization is introduced then its application to digital images is explained spatial predictive transform hybrid binary and subband vector quantizers are reviewed the emphasis is on the usefulness of the vector quantization when it is combined with conventional image coding techniques or when it is used in different domains

pseudo gray coding a pseudo gray code is an assignment of n bit binary indexes to 2 points in a euclidean space so that the hamming distance between two points corresponds closely to the euclidean distance pseudo gray coding provides a redundancy free error protection scheme for vector quantization vq of analog signals when the binary indexes are used as channel symbols on a discrete memoryless channel and the points are signal codevectors binary indexes are assigned to codevectors in a way that reduces the average quantization distortion introduced in the reproduced source vectors when a transmitted index is corrupted by channel noise a globally optimal solution to this problem is generally intractable due to an inherently large computational complexity a locally optimal solution the binary switching algorithm is introduced based on the objective of minimizing a useful upper bound on the average system distortion the algorithm yields a significant reduction in average distortion and converges in reasonable running times the sue of pseudo gray coding is motivated by the increasing need for low bit rate vq based encoding systems that operate on noisy channels such as in mobile radio speech communications

trellis coded quantization of memoryless and gauss markov sources trellis coded quantization tcq is developed and applied to the encoding of memoryless and gauss markov sources the theoretical justification for the approach is alphabet constrained rate distortion theory which is a dual to the channel capacity argument that motivates trellis coded modulation tcm the authors adopt the notions of signal set expansion set partitioning and branch labeling of tcm but modify the techniques to account for the source distribution to design tcq coders of low complexity with excellent mean squared error mse performance for a memoryless uniform source tcq provides an mse within 0 21 db of the distortion rate bound at all positive integral rates the performance is superior to that promised by the coefficient of quantization for all of the best lattices known in dimensions 24 or less for a memoryless gaussian source the tcq performance at rates of 0 5 1 and 2 b sample is superior to all previous results the authors found in the literature the encoding complexity of tcq is very modest tcq is incorporated into a predictive coding structure for the encoding of gauss markov sources simulation results for first second and third order gauss markov sources are presented

asymptotic level density for a class of vector quantization processes it is shown that for a class of vector quantization processes related to neural modeling that the asymptotic density q x of the quantization levels in one dimension in terms of the input signal distribution p x is a power law q x c p x sup alpha where the exponent alpha depends on the number n of neighbors on each side of a unit and is given by alpha 2 3 1 3n sup 2 3 n 1 sup 2 the asymptotic level density is calculated and monte carlo simulations are presented

complexity optimized data clustering by competitive neural networks data clustering is a complex optimization problem with applications ranging from vision and speech processing to data transmission and data storage in technical as well as in biological systems we discuss a clustering strategy that explicitly reflects the tradeoff between simplicity and precision of a data representation the resulting clustering algorithm jointly optimizes distortion errors and complexity costs a maximum entropy estimation of the clustering cost function yields an optimal number of clusters their positions and their cluster probabilities our approach establishes a unifying framework for different clustering methods like k means clustering fuzzy clustering entropy constrained vector quantization or topological feature maps and competitive neural networks

derivation of a class of training algorithms a novel derivation is presented of t kohonen s topographic mapping training algorithm self organization and associative memory 1984 based upon an extension of the linde buzo gray lbg algorithm for vector quantizer design thus a vector quantizer is designed by minimizing an l sub 2 reconstruction distortion measure including an additional contribution from the effect of code noise which corrupts the output of the vector quantizer the neighborhood updating scheme of kohonen s topographic mapping training algorithm emerges as a special case of this code noise model this formulation of kohonen s algorithm is a specific instance of the robust hidden layer principle which stabilizes the internal representations chosen by a network against anticipated noise or distortion processes

online regret bounds for markov decision processes with deterministic transitions we consider an upper confidence bound algorithm for learning in markov decision processes with deterministic transitions for this algorithm we derive upper bounds on the online regret with respect to an e optimal policy that are logarithmic in the number of steps taken we also present a corresponding lower bound as an application multi armed bandits with switching cost are considered

minimax nonredundant channel coding the distortion of a message due to channel noise can be alleviated significantly without redundant error control bits by judicious assignment of binary indices to message symbols the nonredundant coding gain relies only on a notion of distance between symbols in this paper we consider the index assignment problem and adopt a minimax design criterion instead of the usual mean squared error mse measure the problem is found to be np hard and an effective heuristic polynomial time algorithm is presented for computing approximate solutions the minimax criterion yields greatly improved worst case performance while maintaining good average performance in addition the familiar mse criterion is shown likewise to yield an np hard index assignment task the mse problem is a special case of the classical quadratic assignment problem for which computationally and theoretically useful results are available from the discrete mathematics literature

optimal adaptive policies for markov decision processes in this paper we consider the problem of adaptive control for markov decision processes we give the explicit form for a class of adaptive policies that possess optimal increase rate properties for the total expected finite horizon reward under sufficient assumptions of finite state action spaces and irreducibility of the transition law a main feature of the proposed policies is that the choice of actions at each state and time period is based on indices that are inflations of the right hand side of the estimated average reward optimality equations

bayesian adaptive stochastic process termination this paper considers the problem of optimally terminating a number of stochastic processes when the time varying random rewards have distributions belonging to an exponential family with an unknown parameter the problem is formulated as a bayesian adaptive control model with the objective of minimizing the difference between the expected reward and the optimal reward when the parameter is known the paper establishes an asymptotic lower bound on this difference and constructs policies based on a kullback leibler index that obtain this lower bound the results are applied to models of tree harvesting and destructive testing a simulation study shows that these policies are efficient when the number of processes is large