Planning under uncertainty for large-scale problems with applications to wireless networking ; Päätöksenteko epävarmuuden vallitessa suurissa ongelmissa ja sovelluksia langattomaan tiedonsiirtoon

Aalto University, P.O. Box 11000, FI-00076 Aalto www.aalto. fi Author Joni Pajarinen Name of the doctoral dissertation Planning under uncertainty for large-scale problems with applications to wireless networking Publisher School of Science Unit Department of Information and Computer Science Series Aalto University publication series DOCTORAL DISSERTATIONS 20/2013 Field of research Computer and Information Science Manuscript submitted 21 August 2012 Date of the defence 7 February 2013 Permission to publish granted (date) 17 October 2012 Language English Monograph Article dissertation (summary + original articles) Abstract Planning actions into the future is a fundamental task in many real world problems. The uncertain outcome of actions and partial noisy observations often make planning difficult. Specifically, in a wireless network, wireless agents must reason whether to transmit data now or postpone transmission into the future, based only on noisy sensor readings and incomplete information about traffic patterns and the state of other devices. In many settings of this kind, a partially observable Markov decision process (POMDP) defines optimal actions for a single agent and a decentralized POMDP (DEC-POMDP) for multiple co-operative agents. POMDPs and DEC-POMDPs are expressive but computationally demanding models. This thesis presents new efficient POMDP and DEC-POMDP methods, motivated by challenging new wireless networking problems. The first contribution of this thesis is a method for large factored POMDPs that handles larger problems than the comparison methods. The second contribution is the first proposed method for general factored infinite-horizon DEC-POMDPs. The method solves smaller problems with similar accuracy as non-factored methods and it can solve larger problems than the comparison methods. The third contribution is a new kind of controller type for POMDPs and DECPOMDPs, a periodic finite state controller, that allows optimization of larger controllers than previous finite state controller approaches and yields higher performance. The fourth contribution is a POMDP model for a cognitive radio device, which served as motivation for the factored POMDP method. In the model, the cognitive radio transmits on frequency channels occupied by high priority legacy users. The model takes into account varying network traffic burst lengths and reactions of legacy users and performs better than the comparison models. The fifth contribution consists of framing wireless channel access of multiple devices with complicated spatial interference as a factored DEC-POMDP. This allows optimizing over both the spatial and time dimensions and in experiments yields higher performance than the wireless comparison methods. The quality of wireless device decisions depends crucially on the cost and quality of sensor readings. The last contribution is a new spectrum sensing approach, that uses nanotechnology based computations and machine learning for mitigating nanoscale faults and classifying radio signals.Planning actions into the future is a fundamental task in many real world problems. The uncertain outcome of actions and partial noisy observations often make planning difficult. Specifically, in a wireless network, wireless agents must reason whether to transmit data now or postpone transmission into the future, based only on noisy sensor readings and incomplete information about traffic patterns and the state of other devices. In many settings of this kind, a partially observable Markov decision process (POMDP) defines optimal actions for a single agent and a decentralized POMDP (DEC-POMDP) for multiple co-operative agents. POMDPs and DEC-POMDPs are expressive but computationally demanding models. This thesis presents new efficient POMDP and DEC-POMDP methods, motivated by challenging new wireless networking problems. The first contribution of this thesis is a method for large factored POMDPs that handles larger problems than the comparison methods. The second contribution is the first proposed method for general factored infinite-horizon DEC-POMDPs. The method solves smaller problems with similar accuracy as non-factored methods and it can solve larger problems than the comparison methods. The third contribution is a new kind of controller type for POMDPs and DECPOMDPs, a periodic finite state controller, that allows optimization of larger controllers than previous finite state controller approaches and yields higher performance. The fourth contribution is a POMDP model for a cognitive radio device, which served as motivation for the factored POMDP method. In the model, the cognitive radio transmits on frequency channels occupied by high priority legacy users. The model takes into account varying network traffic burst lengths and reactions of legacy users and performs better than the comparison models. The fifth contribution consists of framing wireless channel access of multiple devices with complicated spatial interference as a factored DEC-POMDP. This allows optimizing over both the spatial and time dimensions and in experiments yields higher performance than the wireless comparison methods. The quality of wireless device decisions depends crucially on the cost and quality of sensor readings. The last contribution is a new spectrum sensing approach, that uses nanotechnology based computations and machine learning for mitigating nanoscale faults and classifying radio signals.