partial information state based optimization of partially observable markov decision processes and the separation principle we propose a partial information state based approach to the optimization of the long run average performance in a partially observable markov decision process pomdp in this approach the information history is summarized at least partially by a or a few statistic s not necessary sufficient called a partial information state and actions depend on the partial information state rather than system states we first propose the single policy based comparison principle under which we derive an hjb type of optimality equation and policy iteration for the optimal policy in the partial information state based policy space we then introduce the q sufficient statistics and show that if the partial information state is q sufficient then the optimal policy in the partial information state based policy space is optimal in the space of all feasible information state based policies we show that with some further conditions the well known separation principle holds the results are obtained by applying the direct comparison based approach initially developed for discrete event dynamic systems

superdiffusive transport due to random velocity fields the motion of a random walk in a medium containing random but spatially correlated velocity fields is discussed this type of disorder generally leads to superdiffusive behavior in which the mean square displacement x2 t grows faster than linearly with time for a two dimensional medium with layers of random velocities in the x direction x2 t is found to increase as t2v with 2v 32 the probability distribution of displacements appears to fit the form p x t t 34 exp x t34 with 1 7 however a lifshitz argument for the tail of the distribution suggests that 43 this discrepancy is yet to be fully resolved we also discuss some intriguing properties in a model with isotropic velocity fields in two dimensions we find v 43 while 3 these values obey the general scaling relation 1 v 1

an adaptive linear approximation algorithm for copositive programs we study linear optimization problems over the cone of copositive matrices these problems appear in nonconvex quadratic and binary optimization for instance the maximum clique problem and other combinatorial problems can be reformulated as such problems we present new polyhedral inner and outer approximations of the copositive cone which we show to be exact in the limit in contrast to previous approximation schemes our approximation is not necessarily uniform for the whole cone but can be guided adaptively through the objective function yielding a good approximation in those parts of the cone that are relevant for the optimization and only a coarse approximation in those parts that are not using these approximations we derive an adaptive linear approximation algorithm for copositive programs numerical experiments show that our algorithm gives very good results for certain nonconvex quadratic problems

sequential detection with mutual information stopping cost this paper formulates and solves a sequential detection problem that involves the mutual information stochastic observability of a gaussian process observed in noise with missing measurements the main result is that the optimal decision is characterized by a monotone policy on the partially ordered set of positive definite covariance matrices this monotone structure implies that numerically efficient algorithms can be designed to estimate and implement monotone parametrized decision policies the sequential detection problem is motivated by applications in radar scheduling where the aim is to maintain the mutual information of all targets within a specified bound we illustrate the problem formulation and performance of monotone parametrized policies via numerical examples in fly by and persistent surveillance applications involving a ground moving target indicator gmti radar

multiresolution community detection for megascale networks by information based replica correlations we use a potts model community detection algorithm to accurately and quantitatively evaluate the hierarchical or multiresolution structure of a graph our multiresolution algorithm calculates correlations among multiple copies replicas of the same graph over a range of resolutions significant multiresolution structures are identified by strongly correlated replicas the average normalized mutual information the variation in information and other measures in principle give a quantitative estimate of the best resolutions and indicate the relative strength of the structures in the graph because the method is based on information comparisons it can in principle be used with any community detection model that can examine multiple resolutions our approach may be extended to other optimization problems as a local measure our potts model avoids the resolution limit that affects other popular models with this model our community detection algorithm has an accuracy that ranks among the best of currently available methods using it we can examine graphs over 40 ifmmode times else texttimes fi 10 6 nodes and more than 1 ifmmode times else texttimes fi 10 9 edges we further report that the multiresolution variant of our algorithm can solve systems of at least 200 text 000 nodes and 10 ifmmode times else texttimes fi 10 6 edges on a single processor with exceptionally high accuracy for typical cases we find a superlinear scaling o l 1 3 for community detection and o l 1 3 text text log text n for the multiresolution algorithm where l is the number of edges and n is the number of nodes in the system

an optimal one way multigrid algorithm for discrete time stochastic control the numerical solution of discrete time stationary infinite horizon discounted stochastic control problems is considered for the case where the state space is continuous and the problem is to be solved approximately within a desired accuracy after a discussion of problem discretization the authors introduce a multigrid version of the successive approximation algorithm that proceeds one way from coarse to fine grids and analyze its computational requirements as a function of the desired accuracy and of the discount factor they also study the effects of a certain mixing ergodicity condition on the algorithm s performance it is shown that the one way multigrid algorithm improves upon the complexity of its single grid variant and is in a certain sense optimal

sequential algorithm for fast clique percolation in complex network research clique percolation introduced by palla der enyi and vicsek nature london 435 814 2005 is a deterministic community detection method which allows for overlapping communities and is purely based on local topological properties of a network here we present a sequential clique percolation algorithm scp to do fast community detection in weighted and unweighted networks for cliques of a chosen size this method is based on sequentially inserting the constituent links to the network and simultaneously keeping track of the emerging community structure unlike existing algorithms the scp method allows for detecting k clique communities at multiple weight thresholds in a single run and can simultaneously produce a dendrogram representation of hierarchical community structure in sparse weighted networks the scp algorithm can also be used for implementing the weighted clique percolation method recently introduced by farkas et al new j phys 9 180 2007 the computational time of the scp algorithm scales linearly with the number of k cliques in the network as an example the method is applied to a product association network revealing its nested community structure

small but slow world how network topology and burstiness slow down spreading while communication networks show the small world property of short paths the spreading dynamics in them turns out slow here the time evolution of information propagation is followed through communication networks by using empirical data on contact sequences and the susceptible infected model introducing null models where event sequences are appropriately shuffled we are able to distinguish between the contributions of different impeding effects the slowing down of spreading is found to be caused mainly by weight topology correlations and the bursty activity patterns of individuals

reduced complexity hmm filtering with stochastic dominance bounds a convex optimization approach this paper uses stochastic dominance principles to construct upper and lower sample path bounds for hidden markov model hmm filters we consider an hmm consisting of an x state markov chain with transition matrix p by using convex optimization methods for nuclear norm minimization with copositive constraints we construct low rank stochastic matrices underline p and bar p so that the optimal filters using underline p bar p provably lower and upper bound with respect to a partially ordered set the true filtered distribution at each time instant since underline p and bar p are low rank say r the computational cost of evaluating the filtering bounds is o x r instead of o x 2 a monte carlo importance sampling filter is presented that exploits these upper and lower bounds to estimate the optimal posterior finally explicit bounds are given on the variational norm between the true posterior and the upper and lower bounds in terms of the dobrushin coefficient

robust control of partially observable failing systems this paper is concerned with optimal maintenance decision making in the presence of model misspecification specifically we are interested in the situation where the decision maker fears that a nominal bayesian model may be miss specified or unrealistic and would like to find policies that work well even when the underlying model is flawed to this end we formulate a robust dynamic optimization model for condition based maintenance in which the decision maker explicitly accounts for distrust in the nominal bayesian model by solving a worst case problem against a second agent nature who has the ability to alter the underlying model distributions in an adversarial manner the primary focus of our analysis is on establishing structural properties and insights that hold in the face of model miss specification in particular we prove i an explicit characterization of nature s optimal response through an analysis of the robust dynamic programming equation ii convexity results for both the robust va

probabilistic planning with clear preferences on missing information for many real world problems environments at the time of planning are only partially known for example robots often have to navigate partially known terrains planes often have to be scheduled under changing weather conditions and car route finders often have to figure out paths with only partial knowledge of traffic congestions while general decision theoretic planning that takes into account the uncertainty about the environment is hard to scale to large problems many such problems exhibit a special property one can clearly identify beforehand the best called clearly preferred values for the variables that represent the unknowns in the environment for example in the robot navigation problem it is always preferred to find out that an initially unknown location is traversable rather than not in the plane scheduling problem it is always preferred for the weather to remain a good flying weather and in route finding problem it is always preferred for the road of interest to be clear of traffic it turns out that the existence of the clear preferences can be used to construct an efficient planner called ppcp probabilistic planning with clear preferences that solves these planning problems by running a series of deterministic low dimensional a like searches in this paper we formally define the notion of clear preferences on missing information present the ppcp algorithm together with its extensive theoretical analysis describe several useful extensions and optimizations of the algorithm and demonstrate the usefulness of ppcp on several applications in robotics the theoretical analysis shows that once converged the plan returned by ppcp is guaranteed to be optimal under certain conditions the experimental analysis shows that running a series of fast low dimensional searches turns out to be much faster than solving the full problem at once since memory requirements are much lower and deterministic searches are orders of magnitude faster than probabilistic planning

detection of community overlap according to belief propagation and conflict abstract most existing methods for detection of community overlap cannot balance efficiency and accuracy for large and densely overlapping networks to quickly identify overlapping communities for such networks we propose a new method that uses belief propagation and conflict pcb to occupy communities we first identify triangles with maximal clustering coefficients as seed nodes and sow a new type of belief to the seed nodes then the beliefs explore their territory by occupying nodes with high assent ability the beliefs propagate their strength along the graph to consolidate their territory and conflict with each other when they encounter the same node simultaneously finally the node membership is judged from the belief vectors the pcb time complexity is nearly linear and its space complexity is linear the algorithm was tested in extensive experiments on three real world social networks and three computer generated artificial graphs the experimental results show that pcb is very fast and highly reliable tests on real and artificial networks give excellent results compared with three newly proposed overlapping community detection algorithms

active classification for pomdps a kalman like state estimator author s zois ds levorato m mitra u abstract 1991 2012 ieee the problem of state tracking with active observation control is considered for a system modeled by a discrete time finite state markov chain observed through conditionally gaussian measurement vectors the measurement model statistics are shaped by the underlying state and an exogenous control input which influence the observations quality exploiting an innovations approach an approximate minimum mean squared error mmse filter is derived to estimate the markov chain system state to optimize the control strategy the associated mean squared error is used as an optimization criterion in a partially observable markov decision process formulation a stochastic dynamic programming algorithm is proposed to solve for the optimal solution to enhance the quality of system state estimates approximate mmse smoothing estimators are also derived finally the performance of the proposed framework is illustrated on the problem of physical activity detection in wireless body sensing networks the power of the proposed framework lies within its ability to accommodate a broad spectrum of active classification applications including sensor management for object classification and tracking estimation of sparse signals and radar scheduling

the simplex and policy iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate we prove that the classic policy iteration method howard r a 1960 dynamic programming and markov processes mit cambridge and the original simplex method with the most negative reduced cost pivoting rule of dantzig are strongly polynomial time algorithms for solving the markov decision problem mdp with a fixed discount rate furthermore the computational complexity of the policy iteration and simplex methods is superior to that of the only known strongly polynomial time interior point algorithm ye y 2005 a new complexity result on solving the markov decision problem math oper res 30 3 733 749 for solving this problem the result is surprising because the simplex method with the same pivoting rule was shown to be exponential for solving a general linear programming problem klee v g j minty 1972 how good is the simplex method technical report o shisha ed inequalities iii academic press new york the simplex method with the smallest index pivoting rule was shown to be exponential for solving an mdp regardless of discount rates melekopoglou m a condon 1994 on the complexity of the policy improvement algorithm for markov decision processes informs j comput 6 2 188 192 and the policy iteration method was recently shown to be exponential for solving undiscounted mdps under the average cost criterion we also extend the result to solving mdps with transient substochastic transition matrices whose spectral radii are uniformly below one

structured threshold policies for dynamic sensor scheduling a partially observed markov decision process approach we consider the optimal sensor scheduling problem formulated as a partially observed markov decision process pomdp due to operational constraints at each time instant the scheduler can dynamically select one out of a finite number of sensors and record a noisy measurement of an underlying markov chain the aim is to compute the optimal measurement scheduling policy so as to minimize a cost function comprising of estimation errors and measurement costs the formulation results in a nonstandard pomdp that is nonlinear in the information state we give sufficient conditions on the cost function dynamics of the markov chain and observation probabilities so that the optimal scheduling policy has a threshold structure with respect to a monotone likelihood ratio mlr ordering as a result the computational complexity of implementing the optimal scheduling policy is inexpensive we then present stochastic approximation algorithms for estimating the best linear mlr order threshold policy

local resolution limit free potts model for community detection we report on an exceptionally accurate spin glass type potts model for community detection with a simple algorithm we find that our approach is at least as accurate as the best currently available algorithms and robust to the effects of noise it is also competitive with the best currently available algorithms in terms of speed and size of solvable systems we find that the computational demand often exhibits superlinear scaling o l 1 3 where l is the number of edges in the system and we have applied the algorithm to synthetic systems as large as 40 ifmmode times else texttimes fi 10 6 nodes and over 1 ifmmode times else texttimes fi 10 9 edges a previous stumbling block encountered by popular community detection methods is the so called resolution limit being a local measure of community structure our potts model is free from this resolution limit effect and it further remains a local measure on weighted and directed graphs we also address the mitigation of resolution limit effects for two other popular potts models

myopic bounds for optimal policy of pomdps an extension of lovejoy s structural results this paper provides a relaxation of the sufficient conditions and an extension of the structural results for partially observed markov decision processes pomdps obtained by lovejoy in 1987 sufficient conditions are provided so that the optimal policy can be upper and lower bounded by judiciously chosen myopic policies these myopic policy bounds are constructed to maximize the volume of belief states where they coincide with the optimal policy numerical examples illustrate these myopic bounds for both continuous and discrete observation sets

bayesian sequential detection with phase distributed change time and nonlinear penalty a pomdp lattice programming approach we show that the optimal decision policy for several types of bayesian sequential detection problems has a threshold switching curve structure on the space of posterior distributions this is established by using lattice programming and stochastic orders in a partially observed markov decision process pomdp framework a stochastic gradient algorithm is presented to estimate the optimal linear approximation to this threshold curve we illustrate these results by first considering quickest time detection with phase type distributed change time and a variance stopping penalty then it is proved that the threshold switching curve also arises in several other bayesian decision problems such as quickest transient detection exponential delay risk sensitive penalties stopping time problems in social learning and multi agent scheduling in a changing world using blackwell dominance it is shown that for dynamic decision making problems the optimal decision policy is lower bounded by a myopic policy finally it is shown how the achievable cost of the optimal decision policy varies with change time distribution by imposing a partial order on transition matrices

the use of field theoretic methods for the study of flow in a heterogeneous porous medium the effects of heterogeneities on the steady state flow of a single fluid in a porous medium are examined it is argued that incomplete knowledge of the permeability requires the use of a stochastic model of the system it is shown that the problem may be written as a field theory which allows a perturbation series to be expressed by diagrammatic means this allows the calculation of effective permeability the mean pressure and the pressure variance the method as well as recovering familiar results gives a formal means of improving the approximation and approaching more complex systems

on the complexity of partially observed markov decision processes abstract in the paper we consider the complexity of constructing optimal policies strategies for some type of partially observed markov decision processes this particular case of the classical problem deals with finite stationary processes and can be represented as constructing optimal strategies to reach target vertices from a starting vertex in a graph with colored vertices and probabilistic deviations from an edge chosen to follow the colors of the visited vertices is the only information available to a strategy the complexity of markov decision in the case of perfect information bijective coloring of vertices is known and briefly surveyed at the beginning of the paper for the unobservable case all the colors are equal we give an improvement of the result of papadimitriou and tsitsiklis namely we show that the problem of constructing even a very weak approximation to an optimal strategy is np hard our main results concern the case of a fixed bound on the multiplicity of coloring that is a case of partially observed processes where some upper bound on the unobservability is supposed we show that the problem of finding an optimal strategy is still np hard but polytime approximations are possible some relations of our results to the max word problem are also indicated

how to schedule measurements of a noisy markov chain in decision making a decision maker records measurements of a finite state markov chain corrupted by noise the goal is to decide when the markov chain hits a specific target state the decision maker can choose from a finite set of sampling intervals to pick the next time to look at the markov chain the aim is to optimize an objective comprising of false alarm delay cost and cumulative measurement sampling cost taking more frequent measurements yields accurate estimates but incurs a higher measurement cost making an erroneous decision too soon incurs a false alarm penalty waiting too long to declare the target state incurs a delay penalty what is the optimal sequential strategy for the decision maker this paper shows that under reasonable conditions the optimal strategy has the following intuitive structure when the bayesian estimate posterior distribution of the markov chain is away from the target state look less frequently while if the posterior is close to the target state look more frequently bounds are derived for the optimal strategy also the achievable optimal cost of the sequential detector as a function of transition dynamics and observation distribution is analyzed the sensitivity of the optimal achievable cost to parameter and strategy variations is bounded in terms of the kullback divergence also structural results are obtained for joint optimal sampling and measurement control active sensing

some monotonicity results for partially observed markov decision processes this paper provides sufficient conditions for the optimal value in a discrete time finite partially observed markov decision process to be monotone on the space of state probability vectors ordered by likelihood ratios the paper also presents sufficient conditions for the optimal policy to be monotone in a simple machine replacement problem and in the general case for the optimal policy to be bounded from below by an easily calculated monotone function

the complexity of markov decision processes we investigate the complexity of the classical problem of optimal policy computation in markov decision processes all three variants of the problem finite horizon infinite horizon discounted and infinite horizon average cost were known to be solvable in polynomial time by dynamic programming finite horizon problems linear programming or successive approximation techniques infinite horizon we show that they are complete for p and therefore most likely cannot be solved by highly parallel algorithms we also show that in contrast the deterministic cases of all three problems can be solved very fast in parallel the version with partially observed states is shown to be pspace complete and thus even less likely to be solved in polynomial time than the np complete problems in fact we show that most likely it is not possible to have an efficient on line implementation involving polynomial time on line computations and memory of an optimal policy even if an arbitrary amount of precomputation is allowed finally the variant of the problem in which there are no observations is shown to be np complete

planning and acting in partially observable stochastic domains in this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains we begin by introducing the theory of markov decision processes mdps and partially observable mdps pomdps we then outline a novel algorithm for solving pomdps off line and show how in some cases a finite memory controller can be extracted from the solution to a pomdp we conclude with a discussion of how our approach relates to previous work the complexity of finding exact solutions to pomdps and of some possibilities for finding approximate solutions

detecting communities in social networks using label propagation with information entropy abstract community detection has become an important and effective methodology to understand the structure and function of real world networks the label propagation algorithm lpa is a near linear time algorithm used to detect non overlapping community however it merely considers the direct neighbor relationship in this paper we propose an algorithm to consider information entropy as the measurement of the relationship between direct neighbors and indirect neighbors in a label update we proposed a new belonging coefficient to describe the weight of the label with the belonging coefficient no less than a threshold each node can keep one or more labels to constitute an overlapping community experimental results on both real world and benchmark networks show that our algorithm also possesses high accuracy on detecting community structure in networks

cooperative sequential spectrum sensing based on level triggered sampling we propose a new framework for cooperative spectrum sensing in cognitive radio networks that is based on a novel class of nonuniform samplers called the event triggered samplers and sequential detection in the proposed scheme each secondary user su computes its local sensing decision statistic based on its own channel output and whenever such decision statistic crosses certain predefined threshold values the secondary user will send one or several bit of information to the fusion center fc the fc asynchronously receives the bits from different sus and updates the global sensing decision statistic to perform a sequential probability ratio test sprt to reach a sensing decision we provide an asymptotic analysis for the above scheme and under different conditions we compare it against the cooperative sensing scheme that is based on traditional uniform sampling and sequential detection simulation results show that the proposed scheme using even 1 bit can outperform its uniform sampling counterpart that uses infinite number of bits under changing target error probabilities snr values and number of sus

convex stochastic dominance in bayesian localization filtering and controlled sensing pomdps this paper provides conditions on the observation probability distribution in bayesian localization and optimal filtering so that the conditional mean estimate satisfies convex stochastic dominance convex dominance allows us to compare the unconditional mean square error between two optimal bayesian state estimators over arbitrary time horizons instead of using brute force monte carlo computations the proof uses two key ideas from microeconomics namely integral precision dominance and aggregation of single crossing the convex dominance result is then used to give sufficient conditions so that the optimal policy of a controlled sensing two state partially observed markov decision process pomdp is lower bounded by a myopic policy numerical examples are presented where the shannon capacity of the observation distribution using one sensor dominates that of another and convex dominance holds but blackwell dominance does not hold these illustrate the usefulness of the main result in localization filtering and controlled sensing applications

active state tracking with sensing costs analysis of two states and methods for n states active state tracking is needed in object classification target tracking medical diagnosis and estimation of sparse signals among other various applications herein active state tracking of a discrete time finite state markov chain is considered noisy gaussian observations are dynamically collected by exerting appropriate control over their information content while incurring a related sensing cost the objective is to devise sensing strategies to optimize the tradeoff between tracking performance and sensing cost a recently proposed kalman like estimator by zois et al is employed for state tracking the associated mean squared error and a generic sensing cost metric are then used in a partially observable markov decision process formulation and the optimal sensing strategy is derived via a dynamic programming recursion the resulting recursion proves to be nonlinear challenging control policy design for two state systems with scalar measurements properties of the related cost functions are derived and sufficient conditions are provided regarding the structure of the optimal control policy enabling characterization of when passive state tracking is optimal to overcome the associated computational burden of the optimal sensing strategy two lower complexity strategies are proposed which apply to n state systems with vector measurements the performance of the proposed strategies is illustrated in a wireless body sensing application where cost savings as high as 70 are demonstrated for a 3 detection error with respect to a static equal allocation sensing strategy

first passage time in a two layer system as a first step in the first passage problem for passive tracer in stratified porous media we consider the case of a two dimensional system consisting of two layers with different convection velocities using a lattice generating function formalism and a variety of analytic and numerical techniques we calculate the asymptotic behavior of the first passage time probability distribution we show analytically that the asymptotic distribution is a simple exponential in time for any choice of the velocities the decay constant is given in terms of the largest eigenvalue of an operator related to a half space green s function for the anti symmetric case of opposite velocities in the layers we show that the decay constant for system lengthl crosses over froml 2 behavior in the diffusive limit tol 1 behavior in the convective regime where the crossover lengthl is given in terms of the velocities we also have formulated a general self consistency relation from which we have developed a recursive approach which is useful for studying the short time behavior

the maximum flow problem is log space complete for p the space complexity of the maximum flow problem is investigated it is shown that the problem is log space complete for deterministic polynomial time thus the maximum flow problem probably has no algorithm which needs only o logk n storage space for any constant k another consequence is that there is probably no fast parallel algorithm for the maximum flow problem

description oriented community detection using exhaustive subgroup discovery communities can intuitively be defined as subsets of nodes of a graph with a dense structure in the corresponding subgraph however for mining such communities usually only structural aspects are taken into account typically no concise nor easily interpretable community description is provided for tackling this issue this paper focuses on description oriented community detection using subgroup discovery in order to provide both structurally valid and interpretable communities we utilize the graph structure as well as additional descriptive features of the graph s nodes a descriptive community pattern built upon these features then describes and identifies a community i e a set of nodes and vice versa essentially we mine patterns in the description space characterizing interesting sets of nodes i e subgroups in the graph space the interestingness of a community is evaluated by a selectable quality measure we aim at identifying communities according to standard community quality measures while providing characteristic descriptions of these communities at the same time for this task we propose several optimistic estimates of standard community quality functions to be used for efficient pruning of the search space in an exhaustive branch and bound algorithm we demonstrate our approach in an evaluation using five real world data sets obtained from three different social media applications

sequential detection and identification of a change in the distribution of a markov modulated random sequence the problem of detection and identification of an unobservable change in the distribution of a random sequence is studied via a hidden markov model hmm approach the formulation is bayesian on line discrete time allowing both single and multiple disorder cases dealing with both independent and identically distributed i i d and dependent observations scenarios allowing for statistical dependencies between the change time and change type in both the observation sequence and the risk structure and allowing for general discrete time disorder distributions several of these factors provide useful new generalizations of the sequential analysis theory for change detection and or hypothesis testing taken individually in this paper a unifying framework is provided that handles each of these considerations not only individually but also concurrently optimality results and optimal decision characterizations are given as well as detailed examples that illustrate the myriad of sequential change detection and identification problems that fall within this new framework