The Case for Simulation-Based Evaluation of Ubiquitous Computing Environments

Ubiquitous computing (a.k.a. pervasive computing) is ill-de ned due to its novelty and the community's current emphasis on its technological aspects. This can lead to serious \existential" issues during the evaluation of ubiquitous computing technologies. Use-case driven simulations can address these issues by providing a context for the analysis of ubiquitous computing measurements and metrics. Ubiquitous Computing is an emerging computing discipline which lies at the intersection of desktop computing, networking and embedded systems. Because of its novelty and the community's current emphasis on its technological aspects, ubiquitous computing is ill-de ned. This, in turn, can lead to serious `existential' issues during the evaluation of its underlying technologies. By this we mean that while an evaluation metric may be easily computed and measured, its relevance will come into question. In the end, competing evaluation methodologies will result in confusion and loss of con dence in the possibility for objective evaluation in ubiquitous computing. We have been working to de ne the term \ubiquitous computing" to support our measurement and standardization e orts [1]. Most de nitions appear to focus on the underlying implementation technologies rather than particular user needs. The lack of a user-centric approach complicates evaluation, standardization and measurement e orts because there is little context in which to evaluate emerging technologies, standards and metrics. At the same time, the novelty of ubiquitous computing means that potential users do not have enough experience to clearly articulate their requirements to ubiquitous computing researchers and developers. This may lead to a chicken-and-egg situation where many early attempts to apply ubiquitous computing to real-world problems are likely to fail before accidental successes begin to de ne the eld. Our experience with prototyping a \smart" projector has led us to the conclusion that prototyping is not a panacea for the evaluation of emerging technologies [2, 3]. Implementation decisions must be made early and these decisions can hamper or bias the exploration of potential use cases and their corresponding requirements. Prototyping can also be costly and time consuming. We have begun to explore the use simulation techniques to address these issues. Simulation has been widely touted as an e ective means to explore system engineering issues prior to costly investments of resources [4]. It can be used during the design process to compare several alternatives [5]. Once scenarios of particular interest are found, more resource intensive methods can be used to con rm the validity of the simulated results. The use of simulation requires the proper collection of data and measurements. Because simulation plays an early role in research and development, its use will force research and development teams to properly account for measurement early in the research and development cycle [4]. Simulation plays a signi cant role in traditional scienti c research for similar reasons. Many experiments, for example, in particle physics, require enormous resources. Simulation allows researchers to optimize the use of these limited resources. Research communities are also typically composed of experts whose experiences may lead each to di erent subjective conclusions about an open issue. Simulation can provide a shared context for communication and consensus. [4] For all of these reasons, we are con dent that ubiquitous computing research will also bene t from simulation. Despite these advantages, it has been suggested that