Exploiting predicate structure for efficient reachability detection Partial order (p.o.) reduction techniques are a popular and effective approach for tackling state space explosion in the verification of concurrent systems. These techniques generate a reduced search space that could be exponentially smaller than the complete state space. Their major drawback is that the amount of reduction achieved is highly sensitive to the properties being verified. For the same program, different properties could result in very different amounts of reduction achieved.We present a new approach which combines the benefits of p.o. reduction with the added advantage that the size of the constructed state space is completely independent of the properties being verified. As in p.o. reduction, we use the notion of persistent sets to construct a representative interleaving for each maximal trace of the program. However, we retain concurrency information by assigning vector timestamps to the events in each interleaving. Our approach hinges upon the use of efficient algorithms that parse the encoded concurrency information in the representative interleaving to determine whether a safety violation exists in any interleaving of the corresponding trace. We show that, for some types of predicates, reachability detection can be performed in time that is polynomial in the length of the interleaving. Typically, these predicates exhibit certain characteristics that can be exploited by the detection algorithm.We implemented our algorithms in the popular model checker SPIN, and present experimental results that demonstrate the effectiveness of our techniques. For example, we verified a distributed dining philosophers protocol in 0.03 seconds, using 1.253 MB of memory. SPIN, using traditional p.o. reduction techniques, took 759.71 seconds and 439.116 MB of memory. State space enumeration using interleaving semantics is a highly successful strategy used in the verification of software systems. In this approach, the software system is represented by a labeled transition graph in which every reachable state of the system is enumerated. However, the size of the constructed graph, also known as the state space, quickly becomes exponential in the size of the program description. This is known as state space explosion, and is the fundamental obstacle in the verification of large software systems. Concurrent systems are particularly vulnerable to state space explosion because of the combinatorial number of possible interleavings of a set of concurrent events. For instance, a set of n concurrent events can be interleaved in n! ways, and an exhaustive state space search would explore each of these interleavings.. Partial order (p.o.) reduction strategies [24, 18, 19, 11 ] have emerged as a highly successful, and widely deployed, technique to reduce state space explosion. The widely-used model checker, SPIN, employs p.o. reduction.. Partial order reduction is based on the observation that it is not always necessary to explore all possible interleavings of a set of concurrent events. Sometimes, a given property is satisfied in one interleaving if and only if it is satisfied in all interleavings. Such properties are called equivalence-robust [19] . For the verification of equivalence-robust properties, it is sufficient to explore a single representative interleaving of a set of concurrent events. However, safety properties are rarely equivalence-robust. For example, in a mutual exclusion algorithm, a safety property of interest is that no two processes should be in the critical section simultaneously, that is, ¬EF (incs i ∧ incsj). Consider the (improbably) naive implementation of mutual exclusion in Figure 1 , where each process blindly enters and exits the critical section. The interleaving in Figure 1 (b) satisfies the safety property, whereas a safety violation occurs in Figure 1 (c).. To deal with more general properties, p.o. reduction techniques exploit the notion of invisibility of events. An event is said to be invisible with respect to a predicate only if execution of the event has no effect on the value of the pred-(c) icate. Let each state in an interleaving sequence be labeled by the value of the predicate to be checked. A sequence of adjacent states with the same label is called a stutter.. A labeled interleaving sequence can be collapsed by replacing each stutter by a single state with the same label as the states in the stutter. For example, the labeled sequence abbcccd can be collapsed into the sequence abcd. Two interleaving sequences are said to be stuttering-equivalent iff they can be collapsed into the same labeled sequence. It has been shown [20] that properties expressible in temporal logics without the next-time operator (LT L −X , CT L−X ) cannot distinguish between stuttering-equivalent paths. The reduced state space graph generated by p.o. reduction includes at least one stuttering-equivalent sequence for each interleaving sequence in the full state space graph.. Two distinct sequences can be stuttering-equivalent only if they contain invisible events. So, the effectiveness of p.o. reduction is highly dependent on the number of invisible events in the program. This makes p.o. reduction very sensitive to the predicates being checked [10] . If a large proportion of events change the value of the predicate, then little to no reduction may be achieved in the size of the constructed state space. Experimental results [10] show that the effectiveness of p.o. techniques diminishes rapidly with an increase in the number of visible events.. In this paper, we propose a new approach for detecting safety properties in a concurrent software system. Our approach achieves at least as much state space reduction as p.o. techniques. In addition, the amount of reduction achieved is independent of the properties being checked. Essentially, our approach eliminates the need to consider the visibility of events while constructing the reduced state space.. With p.o. reduction techniques, once a representative interleaving is chosen for inclusion in the reduced graph, we lose all information about the concurrency between events in the interleaving. In our proposed approach, we use p.o. techniques to generate a single representative interleaving for a set of concurrent events, while retaining concurrency information by assigning vector timestamps to the events in an interleaving. The vector timestamps allow us to treat the representative interleaving as a partially ordered set (poset) of events, in which concurrent events are incomparable. Thus, vector timestamps provide a "bridge" between interleaving semantics and a partial order trace semantics as described by Mazurkiewicz [16] .. We also present algorithms that can use the concurrency information carried by the representative interleaving to determine whether any possible interleaving of the corresponding trace could result in a safety violation. We focus on detecting temporal properties of the form EF (ψ). In general, the problem of detecting EF (ψ) on a trace is NP-complete in the number of events in the trace (i.e., in the length of the representative interleaving) [2] . However, for predicate classes that exhibit certain structure (characteristics), we can detect safety violations in time that is polynomial in the number of events in the trace. We introduce some of these predicate classes, together with polynomial-time detection algorithms for them.. Model-checking is rife with intractable problems. It wellknown that model-checking is PSPACE complete [3] . This complexity arises from a combination of factors, including concurrency between processes, local non-deterministic choice within a process, and the range of the variables used in the program. Different approaches attempt to minimize the impact of different factors. Our approach minimizes the impact of concurrency on the size of the constructed state space, and usually results in an exponentially smaller state space representation. In this regard, we are more effective than p.o. reduction techniques.. The most significant contribution of this paper is the notion of performing model checking by exploiting the structure (characteristics) of the properties to be checked. For predicates exhibiting certain characteristics, we can perform reachability checking in time and space that is polynomial in the size of the constructed state space representation. In reality, many properties of interest do exhibit characteristics that make them amenable to polynomial time detection.. To demonstrate the effectiveness of our techniques, we implemented our algorithms in the SPIN model checker [12] . We chose SPIN for our implementation because it is currently the most widely-used software verification tool, and is especially popular for the verification of concurrent software systems. We present experimental results from the verification of three well-known distributed algorithms: a dining philosophers algorithm, a mutual exclusion algorithm, and a leader election algorithm. In experiments, an implementation of our algorithms detected safety violations in a leader election protocol in 53.53 seconds, whereas SPIN with p.o. reduction took 547.41 seconds to detect the same violations.. The rest of this paper is organized as follows. In Section 2, we present necessary background information and definitions. Section 3 formalizes the definition of a trace cover, which is a (possibly infinite) set of traces that contain all the reachable system states. Section 4 presents an algorithm for constructing a (possibly infinite) representative interleaving for each trace in a trace cover. Section 5 describes the mechanism for assigning vector timestamps to the events in an interleaving, in order to obtain the partial order representation of the corresponding trace. In Section 6, we present an algorithm that generates a finite trace cover for finite-state systems. We introduce some tractable classes of predicates in Section 7, and present polynomial-time algorithms to detect such predicates. Experimental results are provided in Section 8. We discuss related work in Section 9, and finish with some concluding remarks in Section 10. In this paper, we proposed a new approach for model checking concurrent and distributed programs. Our approach exploits the compactness of a partial order representation to avoid state space explosion during model generation. Further, we exploit the structure of the predicate to verify safety properties in time that is polynomial in the size of the generated model. We presented such polynomialtime algorithms for two tractable classes of predicates -linear predicates, and 0-1 sum predicates. Our experimental results confirm that our proposed approach results in significant savings in the time and memory required for verification, compared to partial order reduction techniques. For example, we verified a leader election protocol in 75.02 seconds, whereas partial order reduction techniques verified the same protocol in 777.24 seconds.. The algorithms we presented in this paper were limited to the detection of safety properties of the form EF (ϕ). A future direction of research is to develop efficient algorithms for the detection of properties involving additional temporal operators such as AF and EG. Another direction of research is to discover other tractable classes of predicates that allow for polynomial-time detection algorithms. Our approach is a hybrid of an interleaving model such as is used in labeled transition graphs, and a true concurrency model, such as that used in Petri net unfoldings. In our model, we explore all interleavings of events that arise due to local non-deterministic choice within a process, while using a partial order representation to avoid exploring all interleavings of concurrent events. We differ from interleaving-based approaches as we avoid explicit generation of global states by using a partial order representation. Petri nets unfoldings also use a partial order representation of the state space. Unfoldings were originally used for deadlock detection by McMillan in [17] . Esparza later presented an algorithm for using unfoldings to verify safety properties expressed in a limited logic [6] . However, these algorithms were proposed for low-level Petri nets, which are inherently unscalable. Low-level net representations tend to be quite large even for very simple high level programs, making them impractical for use in the verification of (large) real-world programs.

O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Toward a more accurate view of when and how people seek help with computer applications Based on 40 interviews and 11 on-site workplace observations of people using computer applications at work, we confirm that use of printed and on-line help is very low and find that providing greater detail of categories solution methods can present a more realistic picture of users' behaviors. Observed study participants encountered a usability problem on average about once every 75 minutes and typically spent about a minute looking for a solution. Participants consumed much more time when they were unaware of a direct way of doing something and instead used less effective methods.Comparison of results from different data-collection methods suggests that interviews, and probably surveys, provide less reliable views of users' problem-solving behaviors than do participatory evaluation and direct observation. Research in the last ten years has shed new light on when and how people seek help in handling problems they experience in using computer applications. However, there are still big gaps in the picture of people's use of documentation and help systems in practice. These gaps are largely the consequence of known methodological issues, but the methodological difficulties of understanding actual user behavior in the workplace are significant enough that prior studies have used methods that, even if flawed, nevertheless generated otherwise unobtainable results.. One of the focuses of recent research in this field has been users' choice of methods for solving problems they encounter with computer systems. In this paper, we address two issues in particular: lack of detail within categories of solution methods (such as seeking on-line help) and the validity of the methods used in this research to determine how often users employ these various kinds of solution methods. Results of our interviews suggest that studies that categorize people's methods of solving problems when using computer applications can present a clearer picture of people's behaviors if broad categories of solution methods are reported in greater detail. Users report that they consult a help desk slightly more often than they ask a friend or colleague and that they use a search engine nearly half as often as they use local on-line help.. Results of our observations suggest that users of computer applications encounter a usability problem on average about once every 75 minutes and typically spend about a minute looking for a solution. But users consume much more time when they are unaware of a direct way of doing something and so end up slogging through a task with ineffective methods; users typically spend 10 minutes on this sort of task, which probably could have been completed in two to three minutes. Survey and interview methodologies thus likely understate the true impact of frustration with and time lost using computer applications. The interview results also suggest that selfestimates of solution methods, and to a lesser extent self-reports of specific solutions, tend to overstate recourse to help.. From these results, we conclude that it remains a major open issue for research as to why users muddle through instead of seeking help. In some cases, users apparently do not know that there is a better way of doing things. For these users of computer applications, their work-around is, so they believe, the solution. In other cases, the users suspect that a better method exists but do not expend the effort to find it. For example, a participant told us that she thought the application probably had a function for summing numbers, but she used a calculator anyway. As they gain experience with a computer application, why do people settle for asymptotic mediocrity? Are there ways of persuading people to seek help more often?. One odd insight into modern work lives emerging from the observations is that it is difficult to determine, from the transcript alone, the participants' business or occupation. Despite the variety of work settings and computer applications, there is a palpable sameness to the participants' activities-filling out forms, sending e-mail messages, and producing or modifying documents.. Our data also suggest that interviews, and probably surveys, provide less reliable views of users' problem-solving behaviors than do participatory evaluation and direct observation. The sheer level of effort involved in observation studies, both in recruiting subjects and in conducting and analyzing the observations, tends to limit the scope of this approach. With all of these methodologies, the choice of subjects and applications may lead to big differences in results.. Our own observation study was subject to a choice-of-subject consideration. Because we were limited to the subset of interviewed subjects who were willing to then let us observe them at work, it is possible that the observed participants do not fairly represent the larger set of interviewed participants. Likewise, the network of acquaintances through which we recruited subjects may not have led to a sample that fairly represented the general population of people who use computers at work. While we took pains to make clear to potential participants that we were looking at computing in everyday life at work, that we not interested in "computer experts," and that we were evaluating the computer applications rather than the participants, it is possible that fear of embarrassment may have dissuaded people from participating, particularly in the observation phase. The potential discrepancy between interviewed and observed participants could be avoided in future research by studying only participants who agreed at the outset to be observed. The more general problem of the representational validity of the sample remains to be solved; results of such studies can be interpreted in light of their reported distributions of occupations of participants. We now to turn to the fundamental question of how well users' selfestimates of solution methods correlate with their actual behaviors as observed in the workplace. Figure 10 shows the distributions of solution methods as estimated by participants in interviews and as observed in participants' work. (Figures 10 through 12 aggregate solutions across categories because further detail was not provided in some of the studies represented, and categories such as "reboot" or "restart" are omitted.) The correlation between the two distributions is not statistically significant. It appears that there is agreement that people almost never use the printed, but that there is little agreement between the self-estimated and observed distributions as to how often people ask someone else, use on-line help, solve the problem by themselves, or give up. As compared with their mean use of solution approaches in observed work, participants tended to underestimate the extent to which they solved the problem themselves and gave up. In contrast, participants tended to overestimate the extent to which they asked someone else and used on-line help. Participants estimated that they used printed manuals in 5.66 percent of cases, but even that minimal number turned out to be an overestimate relative to actual use. These differences between self-estimated and observed solution methods led us to return to the interview data. We extended our analysis by extracting the reported solution methods for each of the frustration episodes mentioned by the participants, aggregating responses across multiple solution attempts, if present, for an episode. The distribution of participants' reported solution attempts for their frustration episode falls roughly mid-way between those for the interview estimates and the observations, as shown in Figure 11 . We also conducted an extended analysis of the episode data from the Novick and Ward study [16] , and the correlation between these data sets is 0.88. Taken as a whole, the data suggest that, when interviewed about their approaches to solving problems in using computer applications, people tend to overestimate their recourse to help, even for printed manuals. Given the close correspondence between our interview results and those of Novick and Ward [16] , this trend appears reliable. The responses in interviews could reflect a socialdesirability bias or an unusually heightened awareness of documentation and help caused by the experimental design. In either case, we conclude that interviews, even asking for specific frustration episodes, (and, for similar reasons, probably surveys as well) are unreliable indicators of people's actual problem-solving approaches for problems with computer applications. Figure 12 presents the distribution of solution methods for all four studies in which comparable data are obtainable. The current study is represented for interview self-estimates, interview frustration episodes, and for observation. The Novick and Ward study is represented by interview self-estimates and interview frustration episodes. The Mendoza and Novick study is represented by self-reports. And the Ceaparu study is represented by self-reports and observations. The disparities among the distributions likely reflect differences in sampled populations, tasks, and methodology. The one element that all of these disparate results have in common is that users of computer applications rarely use printed documentation. The modestly higher figures in some studies for use of printed documentation likely reflect social desirability bias in the responses. This was probably the case as well for other studies using interview and survey methods (e.g., [18] , [11] ) not represented in Figure 12 .. These results do not mean that interviews are without value. To the contrary, interviews enable researchers to go into greater depth in exploring users' attitudes toward documentation, looking at likes and dislikes, understanding individual frustration episodes, and providing guidance for designers of documentation and help systems. However, interviews appear to be of limited value in determining the overall distribution of users' solution methods.. The comparative value of participative evaluation, as reported by Ceaparu et al. [4] and Mendoza and Novick [12] remains unclear.. As indicated in Figure 13 , the observations and self-reports obtained by Ceaparu et al. matched well; the correlation coefficient is 0.87. In contrast, the self-reports from Mendoza and Novick [12] have virtually no correspondence with the self-reports of Ceaparu et al.. (correlation coefficient = 0.16) nor with the observation results of the current study (correlation coefficient < 0.01). It may be coincidence, but the correlation coefficient between the Ceaparu self-reports and the observations in the present study is 0.93. In interpreting the correlations between studies, it should be born in mind that we collapsed and omitted some solution categories to make possible the comparisons. The differences between the two studies using participative evaluation apparently arise from differences in subject populations (university computer science students vs. middle-school teachers) task (diverse tasks vs. a single task), and environment (performing different tasks independently in a computer lab vs. performing the same task with colleagues nearby). Thus we expect that the availability of colleagues working on exactly the same problem accounts in large part for the disparities between (1) "asked other" and (2) the combination of "solved without help" and "gave up." In what may be a depressing perspective on life in the workplace, our observed participants were even less likely than the computer science students to ask someone else, although they were also less likely to give up. Additionally, the "solved without help" category includes work-arounds, and the middle-school teachers may have been able to reduce the number of work-arounds by finding a colleague who already had the solution. In any case, while the correlation between the distribution of solution methods of university computer science students and those of mostly whitecollar professionals in the workplace may be coincidental, the degree of correlation nevertheless remains striking. Research into how users of computer applications solve usability problems has included self-reports, Web-based surveys, telephone interviews, and in-person interviews. The results vary. Some studies indicate that use of documentation is widespread [17] , [18] , but this was based on a single application, largely among novice users. Other studies indicate that people rarely use on-line help and almost never use printed manuals [4] , [12] . The studies' participants used on-line or printed documentation in only 0 to 4 percent of the occasions they encountered problems with their use of computers. These studies, though, were based on self-reports and peer observations from relatively homogeneous populations, such as computer science students [3] or middle-school teachers using a single application [5] .. A more recent study [14] assessed use of documentation across a much more heterogeneous sample of computer users. This study found that, on average, participants estimated that they used on-line help in about 28 percent of the occasions in which they experienced difficulty and used printed documentation in about 3 percent of these occasions. Indeed, more participants reported that they abandoned a task than used printed documentation. But, as the authors pointed out, this study, too, had methodological weak points. In particular, the interviews to assess solution patterns may have been unreliable, and. some solution categories were probably too broad. Concerns about the interview methodology relate to the validity of participants' accounts of their use of documentation. The issue with the solution categories arose because the category "asked other" could cover asking a colleague, asking someone at an internal or contract help desk, or asking someone at the software publisher's help desk. The category "used on-line help" could cover the help provided with the application, help available from the publisher via the Web, and help available from unofficial sources such as online forums and newsgroups, usually located via a search engine. The interviews suggested that the study's participants considered these to be different sources of information. Similarly, the category "used printed manual" could include both the manual supplied with the software or an "after-market" book.. Our review of related work suggests that methodological issues posed, and continue to pose, problems for research into the incidence of and users' responses to usability problems. These issues include:. • Novice users. Some of the studies [18] , [17] looked primarily at novice users. This approach presents a number of problems. First, novice users are in the learning phase and thus may naturally refer to documentation more often than experienced users. Second, novices tend to encounter different kinds of problems than experienced users [12] . And third, the data reported by [16] may indicate that users refer to printed documentation only when installing a new application. For these reasons, studies focusing on novice users may overestimate the extent to which people use documentation.. • Unrepresentative users or work context. While any user is representative of himself or herself, the studies showing the lowest rates of use of documentation were based on special groups of users or work situations, rather than sampling broadly in the workplace. This likely occurred because of difficulties in obtaining broadly representative subjects or in finding subjects who could be studied longitudinally with a common application. Thus the participants in [4] were computer science students, the participants in [12] were teachers at a middle school, and the participants in [11] were members of the public, including members of underrepresented groups, blind users, and developers and technical support providers who were willing to answer a Web-based questionnaire of about 50 questions with compensation of a $10 gift certificate. The methodological issue is that neither computer science students nor middle school teachers may have the kinds of usability problems and responses that are characteristic of the broader world of work. And participants willing to fill out a long Web form may not represent the general population of users of computers, either. Indeed, the methodological similarities and disparate results of [4] and [12] suggest that subject population and work context plays a large role in the outcome of the study.. • Limited software. Some research examined use of documentation and help systems for a single application. One study [18] focused on a particular word-processing application, and the researchers obtained subjects through a list of purchasers provided by the publisher. Another study [12] looked at use of a single module of a software package for teachers. Until the field has many more such studies, we do not know if the observed phenomena are generally true or rather limited to the circumstances of the particular application.. • Interviews: Whether conducted face to face or by telephone, interviews enable researchers to go into depth but have limited validity. The problem is that people remember things in ways that are systematically skewed. For example, when asked to remember the most recent incident of a certain type, they are likely to remember the most salient rather than the most recent [19] , [2] . Thus studies of use of documentation and help systems that relied on interviews (e.g., [18] , [16] ) may have results that are skewed toward salience rather than recency. And, as we discuss in Section 5, interviews about documentation and help systems may also suffer from other systematic problems, such as social desirability bias and fatalistic acceptance of usability problems with software.. • Participative evaluation. In contrast to conducting post-hoc interviews, some studies [4] , [12] , [13] have relied on contemporaneous reports or diaries from study participants. This technique, called participative evaluation [8] , has fewer problems with recall, but does not permit the researchers to obtain clarification or go into depth because they are not present to ask additional questions of the participants when warranted. Additionally, as we discuss in Section 5, participative evaluation misses cases where participants mistakenly believe that they do not have a usability problem.. • Surveys. Although they may facilitate including larger numbers of participants and may include dozens of questions (cf., [11] ), surveys suffer from the ills of both interviews and of participative evaluation. That is, they are subject to recall effects yet do not allow researchers to follow up answers with clarification.. • Categories with insufficient detail. Since the lead study by Ceaparu et al. [4] , subsequent studies of frustration with computers have tended to use similar categories for the participants' solutions (or non-solutions), which permitted the researchers to compare results. The interview methodology of Novick and Ward [16] enabled going into greater depth when the participants described their attempts to solve usability problems, and this disclosed that at least three solution categories-"asked other," "used on-line help" and "used printed manual"-might be overbroad. For example, using "on-line help" includes both using the help supplied with the software and visiting a Web-based forum.. • Observation. While observation of subjects provides the most direct view of users at work, this advantage comes at considerable cost. Participants and their employers can be reluctant to permit observation in the workplace, particularly if audiovisual recordings are made of the participants at work (The observations reported by Ceaparu et al. [4] were made by student peers). Clarification of behaviors is difficult to obtain because questions from the observer would interfere with and thus change the participants' activities. Observation is time-consuming because in practice frustration episodes turn out to be relative infrequent; a two-hour session may not include any frustration episodes. And the clearer picture of the participants' work brings with it questions not faced in other methods, such as coding of tasks in which the participant uses a work-around but is unaware of the existence of the better technique.. Given these methodological issues, in this study we address these questions:. • Can possibly overbroad solution categories be clarified? • How reliable are interviews as a methodology for assessing users' ways of solving problems with computer applications? In this study, we were specifically interested in clarification of categories of solution approaches and in the nature of the relationship between interview reports and observed behaviors with respect to usability-problem episodes and solutions tried. We expected that the use of self-reports through interviews might lead to systematic biases in the kinds of solution approaches attempted. Thus the study had two phases. The first phase replicated Ward and Novick, conducting interviews with more subjects and with more detail, in order to clarify possibly overbroad solution categories. The second phase returned to a subset of the same subjects, conducting in-person, on-site observation of the subjects at work.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Refining image annotation using contextual relations between words In this paper, we present a probabilistic approach to refine image annotations by incorporating semantic relations between annotation words. Our approach firstly predicts a candidate set of annotation words with confidence scores. This is achieved by the relevance vector machine (RVM), which is a kernel based probabilistic classifier in order to cope with nonlinear classification. Given the candidate annotations, we model semantic relationships between words using a conditional random field (CRF) model where each vertex indicates the final decision (true / false) on a candidate annotation word. The refined annotation is given by inferring the most likely states of these vertexes. In the CRF model, we consider the confidence scores given by the RVM classifiers as local evidences. In addition, we utilise Normalized Google distances (NGD's) between two words as their contextual potential. NGD is a distance function between two words obtained by searching a pair of words using the Google search engine. It has a simple mathematical formulation with a foundation in Kolmogorov theory. We also propose a learning algorithm to tune the weight parameters in the CRF model. These weight parameters control the balance between the local evidence of a single word and the contextual relation between words. Our experiments on the Corel images demonstrate the effect of our approach. With the prevalence of digital imaging devices such as webcams, phone cameras and digital cameras, image data accessible to computer users are now exponentially increased. To that end, an urgent issue is how to browse and retrieve this large volume of images. A possible solution is by content based image retrieval (CBIR) where two images are considered to be similar in semantics if they are close in the space of visual features. This is useful in the case where only image data are available, but limited in any semantic interpretation of images. Another approach is by annotating images with text keywords. With ideal annotations, image retrieval can be solved by well developed techniques in text retrieval. Unfortunately, most of the image data are unannotated or only partially annotated. Manual annotation is impractical for a large number of images. Although in the special case of web images, rough annotation can be extracted from the associated text such as the surrounding web text and tittles, these annotations are very noisy because the associated text are not strongly correlated with the images and they completely ignore the visual content of images. Confronted with these challenges, automatic image annotation (AIA) aims to improve the performance of current image retrieval systems. The basic idea is that given a small dataset of images with annotations, we aim to train an image annotation system that is capable of annotating any new images automatically.. Current techniques for AIA can be summarized into two categories. First, image annotation is formulated as an image classification problem [2, 11, 5] . Specifically, each concept (annotation word) is viewed as a unique class label. For example, in a binary classification setting, we train a single classifier for each concept individually. To annotate new images, these classifiers are applied to a new image one by one and the final annotation results are obtained by ranking their posterior class probabilities [2] . Alternatively, we can train a multi-class classifier altogether. This can reduce the overall complexity of the classifier since there are many common features shared by different concepts [5] . The advantage of the classification approach is that we have various mature machine learning techniques available such as Naive Bayes, SVM, Bayes Point Machines and the 2-D multi-resolution Hidden Markov Model [11] etc. The second approach represents annotation words and images as features in different modalities. Image annotation is then realized by modeling the joint distribution of the visual features and the textual features together on the training data and predicting the missing textual features for a given new image. A common point of these approaches is to decompose an image into a number of sub units such as image regions to generate the similar form of discrete features as that of text. For example, Barnard et al. [10] proposed a translation model for image annotation by viewing the images (a set of blobs) and textual words as two different languages. Other works along this line include the cross-media relevance model (CMRM) [7] , the multiple Bernoulli relevance model (MBRM) [14] , the latent Dirichlet allocation model (LDA) [1] and the continuous relevance model (CRM) [17] .. However, there is a problem remained unsolved, i.e., the semantic gap. All the above approaches try to annotate images based solely on their visual features. On the other hand, our human cognitive understanding of images is far removed from comparing low level visual features. Whilst effort is made to improve image annotation based on visual features alone, a number of works have begun to incorporate semantic knowledge from different sources into image annotation, complementary to the existing visual feature based approaches. To incorporate semantic knowledge into image annotation, there are three major issues: (i) How to represent semantic knowledge? There are mainly two types of representation considered by existing work, we call them hierarchical relationship [6, 15, 19] and flat relationship [9, 18] respectively. In a hierarchical relationship, concepts are related by parent-and-child relation. More general concepts are usually placed in the top level of the hierarchy. For example, the concept scene can be divided into indoor and outdoor. An outdoor scene can be further divided into several categories such as sunset, beach, city etc. The second form, flat relationship, considers pairwise relationships between two concepts. The focus here is the relatedness between two concepts. Two concepts can be related for a number of reasons. For example, sunset is more related with outdoor than with indoor and desk is more related with chair than with lake. (ii) How to build a knowledge database? The most popular method is to extract semantic knowledge from existing ontologies such as WordNet [13, 6, 15, 9] , which is a semantic lexical database providing a number of possible relations between English words. Manual construction [19] or automatic learning [18] of semantic knowledge from external datasets has also been demonstrated. (iii) How to integrate the knowledge database into the annotation process. This is largely dependent on the stages of an annotation process where the knowledge is incorporated. For example, Srikanth et al. [15] introduced ontologies the prepossessing of visual features, i.e., the visual vocabulary to improve the translation model. Wu et al. [19] modelled the causal strength between concepts by a directed acyclic graph (DAG) and the annotation is inferred bases on this graph structure. Jin et al. [9] and Wang et al. [18] integrate ontologies in the post processing stage to refine the initial annotations.. In this paper, we present a novel ontology based image annotation framework. Specifically, we propose a Conditional Random Field (CRF) [8] framework to incorporate the contextual relations between words as a kind of flat relationship extracted automatically from Google search. For a given image, our framework firstly predicts a candidate set of annotation words and then refine them by integrating contextual relationships between concepts. Our work is mostly related to that of Jin et al. [9] and Wang et al. [18] , but differs in a number of ways as follows:. (1) In the approach of Jin et al. [9] , the confidence scores of the initial annotation word are discarded in the following refining stages, while we take the same strategy as that in Wang et al. [18] to keep these confidence scores and integrate them with the contextual knowledge on the annotation words in the refining stage.. (2) The ontologies used in Jin et al. [9] is provided by a specific knowledge database (WordNet) constructed by human experts, while ours is extracted automatically from the world wide web (WWW). The WWW is probably the largest digital text database on the earth, and the latent semantic context information entered by millions of independent users average out to provide a meaningful sense of the contextual relations between words. Wang et al. [18] also attempted to extract the ontologies from the Internet, but our measure of contextual strength is motivated by the normalized Google distance (NGD) proposed by Cilibrasi and Vitanyi [4] . NGD has shown to be able to discover the meaning of words by the extensive experiments [4] .. (3) The approach of integrating ontologies into the annotation in [9] is based on a rule based method which removes the noisy words from the initial annotation, while in [18] they model the annotation refining process as a random walk with restarts. Different from [18] and [9] , we model the assignment decision (true/false) of a word from the initial annotation as a conditional binary random field. The advantage of the CRF approach is that it can provide a coherent probabilistic fusion approach taking into account the individual probabilistic label assignment and the contextual relations between annotation words simultaneously. Moreover, we provide a learning process to tune the balance between the local evidence of each words and the contextual relation between words.. The rest of this paper is organized as follows: section 2 presents an overview of the key components of our automatic image annotation refining framework. Section 3 discusses the probabilistic binary image classification approach based on the bag of visual words model. In section 4 we introduce the normalized Google distance and compare it with the ontologies in WordNet. Section 5 describes the conditional random field which is used to fuse ontologies into image annotation. We present our experimental results in section 6 and conclude in section 7. Fig.1 illustrates the flowchart of our framework of refining image annotation. There are three key components as described in the following, ( In this paper, we presented a probabilistic framework to refine image annotation by incorporation semantic relation between annotation words. Our framework firstly predicts a candidate set of annotation words with confidence scores. This is achieved by the relevance vector machine, which is a probabilistic classifier working in the kernel trick. Given the candidate annotation, we model the context relation between words by a conditional random field model. In the CRF model, each vertex indicate the final decision of a candidate annotation word. Its local evidence is given the the confidence score produced by the RVM classifiers. Its con- text relation is given by the normalized Google distance. The NGD is a distance function between two words obtained by just typing these two words as the search terms in the Google search engine. It has a simple mathematical formulation but has a solid theory foundation. We also propose a learning algorithm to learn the weight parameters in the CRF model. These weight parameters control the balance between the local evidence of a single word and the context relation between words. Our experiment results on the Corel images have shown the effect of our approach. We believe that the semantic knowledge from different sources are complementary to each other. In the future, we will research how to combine together the semantic knowledge from NGD and other sources such as WordNet to achieve a better annotation performance.

O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Using contexts to supervise a collaborative process Abstract-This paper describes a research project that investigated the feasibility of using contextual reasoning to supervise the collaborative work of knowledge workers. In complex projects that require contributions from various experts but whose interaction may be limited to a web-based collaborative tool, proper management of the project is essential to ensure that the project objectives are met. This is typically the job of a project manager. We assert that having situational awareness is likewise essential to managing a project, and we utilize Context-based Reasoning (CxBR) as the tool of choice for implementing situational awareness in an agent that assists project managers. We use rocket design and manufacture as the domain to evaluate our approach. We make use of public domain rocket design software developed by NASA as a guide to the domain. The paper describes the investigation and the related works involved in collaborative design project, as embodied by designing and building a small rocket. Computer Supported Cooperative Works (CSCW) is a field of study that seeks to provide tools and techniques to carefully but effectively manage the process of combining the talents and skills of several workers in a closely-coupled project. The term CSCW was coined by Greif and Farmer in 1984 [1] . CSCW tools have as their objective to facilitate the cooperative project. In this case, cooperative work can defined as work that cannot be done by one entity working on his/her/its own, but must rely on help from other entities to meet its objective.. Much work has been done with regards to CSCW. CSCW is particularly relevant to situations where it is crucial to coordinate, communicate and support the interrelations between several knowledge workers in a complex project. Military operations, team games and managing projects share this common trait. In the context of this project, we focus on the last application, specifically as related to Web-based project collaboration, where awareness of the situation is difficult because of the lack of face-to-face communication.. Several systems have been built to assist project managers (PM) in their tasks. [2] [3] [4] [5] [6] [7] . We note that in the reviewed literature on project management, the situational awareness of the project manager is not directly mentioned. Situational awareness is a term coined by Endsley [8] to reflect an individual's (and hence, an agent's) ability to be fully cognizant of the current situation faced, and the implications it has on one's actions. We base our work described here on the notion that a project manager requires a measure of situational awareness in order to be successful.. We seek to duplicate the function of a project manager. Specifically, we developed an agent (called PM-agent) that does what a competent project manager would do and alert the human project manager to a situation that may have negative implications on the project. This tool aims to help the project manager ensure that the stated objectives of a project are met. These are related to [9] : scope, cost and delivery time. That is, projects must meet the product specification, and be delivered on time and within budget. These can be said to be the duties of a project manager in their most basic elements.. A further distinguishing facet of our work is that we use contextual reasoning to provide situational awareness to the PM-agent. We accomplish this by basing our PM-agent on Context-based Reasoning (CxBR) [10] . CxBR, by its very context-driven nature, facilitates situational awareness in its agents. It has been successfully used to model tactical agents that represent human-controlled forces in a battle. We explain CxBR and the rest of our approach in the next section.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


An information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis A hyperspectral image can be considered as an image cube where the third dimension is the spectral domain represented by hundreds of spectral wavelengths. As a result, a hyperspectral image pixel is actually a column vector with dimension equal to the number of spectral bands and contains valuable spectral information that can be used to account for pixel variability, similarity and discrimination. We present a new hyperspectral measure, the spectral information measure (SIM), to describe spectral variability and two criteria, spectral information divergence and spectral discriminatory probability for spectral similarity and discrimination, respectively. The spectral information measure is an information-theoretic measure which treats each pixel as a random variable using its spectral signature histogram as the desired probability distribution. Spectral information divergence (SID) compares the similarity between two pixels by measuring the probabilistic discrepancy between two corresponding spectral signatures. The spectral discriminatory probability calculates spectral probabilities of a spectral database (library) relative to a pixel to be identified so as to achieve material identification. In order to compare the discriminatory power of one spectral measure relative to another, a criterion is also introduced for performance evaluation, which is based on the power of discriminating one pixel from another relative to a reference pixel. The experimental results demonstrate that the new hyperspectral measure can characterize spectral variability more effectively than the commonly used spectral angle mapper (SAM). The advent of high spectral resolution airborne and satellite sensors improves the capability for collection of ground targets in the fields of agriculture, geology, geography, law enforcement, military and defense, etc. [1] . One major advantage of a hyperspectral sensor over a multispectral sensor is that the former images a scene using as many as 200 contiguous bands as opposed to the latter that only uses tens of discrete bands. Therefore, hyperspectral image data permit the expansion of detection and classification activities to targets previously unresolved in multispectral images. Due to improved spectral resolution and large scene coverage, a hyperspectral pixel is generally a mixture of different materials present in the pixel with various abundance fractions. These materials absorb or reflect within each spectral band. As a consequence, spectral characterization becomes crucial in hyperspectral image analysis. However, because of atmospheric effects, the spectral information of a pixel varies during data acquisition. In order to account for spectral variability, similarity, and discrimination, we present in this correspondence an information-theoretic spectral measure, referred to as Spectral Information Measure (SIM). SIM models the spectral band-to-band variability as a result of uncertainty caused by randomness. It considers each pixel as a random variable and defines the desired probability distribution by normalizing its spectral histogram to unity. With this interpretation, SIM not only can describe the randomness of a pixel, but can generate also high-order statistics of the pixel based on its spectral signature. This is particularly useful for hyperspectral image analysis. Since each hyperspectral image pixel is composed of hundreds of spectral bands, the spectral information provided by this pixel is generally very valuable in material discrimination, detection, classification, and identification. However, this advantage also comes at a price; many unknown signal sources may be also collected by the sensor [2] . SIM is designed to capture such uncertainty and unpredictability. The higher the data dimensionality, the more effective is the SIM. In order to determine the spectral similarity between two pixels, a SIM-based Spectral Information Divergence (SID) is developed, based on the concept of divergence [3] , [4] to measure the discrepancy of probabilistic behaviors between the spectral signatures of the two pixels. Compared to the Spectral Angle Mapper (SAM), which has been widely used in the past [5] , SID is more effective in preserving spectral properties. In many applications, it is required to identify a target in an unknown image scene via a spectral library (database). To meet this need, a criterion is introduced to calculate the spectral discriminatory probabilities of the signatures in the library and to provide the likelihood of identifying the target. Finally, in order to compare the relative discriminatory power between two spectral measures, another criterion is proposed, based on the ratio of the spectral similarities of a pixel to another pixel relative to a reference pixel.. INVESTIGATE statistical target detection in images by methods composed of two suboptimal stages-"transformation" followed by "detection." Transformation is accomplished by operating on the input image with a linear filter that is designed to maximize one of three rational "signal-to-noise-ratio" (SNR) functions. Detection is accomplished by operating on the transformed image with the statistically optimal univariate, or point, detector. This detector correctly accounts for the empirical target and clutter statistical distributions, rather than being based on idealized distribution (e.g., normal) assumptions. The theoretical development is supported by experimental results on infrared (IR) and synthetic-aperture radar (SAR) images, including effectiveness and complexity comparisons with several established image target detectors. A note on the terminology that is used here is worthwhile. I distinguish between three grades of detector optimality: optimal detectors maximize, without constraint, a direct measure of detector effectiveness (e.g., detection probability corresponding to prescribed falsealarm probability); suboptimal detectors also maximize some objective function, but either they only achieve a constrained maximization, or the objective function is only an indirect and imperfect measure of true detector effectiveness; and nonoptimal detectors do not maximize any objective function whatsoever. The transformation filters proposed here are suboptimal in two respects: they are constrained to be linear; and they maximize SNR functions that have a strong, but nevertheless only indirect, influence on detector effectiveness. The detectors proposed here are suboptimal in the sense that they are constrained to be univariate rather than free to be multivariate. The classification scheme used here is not universal, with authors often omitting the suboptimal category, and only distinguishing between optimal and nonoptimal categories. In any credible two-category scheme, both the filters and detectors proposed here would emphatically fit in the optimal category-in no sense are these filters and detectors nonoptimal. Although truly optimal target detectors are theoretically known, for realistic image statistics and manageable image ensemble sizes evaluation of these optimal detectors is an intractable problem, which leaves suboptimal detectors as the most objective of tractable detectors. In this correspondence, we present an information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis. The Spectral Information Measure (SIM) is introduced to model a hyperspectral image pixel as a random variable to capture spectral variability in the pixel. Then the Spectral Information Divergence (SID) was further developed to measure the spectral similarity between two pixels. In order to compare two hyperspectral measures, a new definition of spectral discriminatory power was suggested for performance evaluation. Finally, for the purpose of material identification, the concept of spectral discriminatory probability vector was also developed based on a spectral database or library. The experiments demonstrate that the SIM-based criteria performed more effectively than the traditional spectral measure SAM, widely used in hyperspectral image analysis.

O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I I O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I I O O O O O O B I O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O


Distributed-based massive processing of activity logs for efficient user modeling in a Virtual Campus Abstract This paper reports on a multi-fold approach for the building of user models based on the identification of navigation patterns in a virtual campus, allowing for adapting the campus' usability to the actual learners' needs, thus resulting in a great stimulation of the learning experience. However, user modeling in this context implies a constant processing and analysis of user interaction data during longterm learning activities, which produces huge amounts of valuable data stored typically in server log files. Due to the large or very large size of log files generated daily, the massive processing is a foremost step in extracting useful information. To this end, this work studies, first, the viability of processing large log data files of a real Virtual Campus using different distributed infrastructures. More precisely, we study the time performance of massive processing of daily log files implemented following the master-slave paradigm and evaluated using Cluster Computing and PlanetLab platforms. The study reveals the complexity and challenges of massive processing in the big data era, such as the need to carefully tune the log file processing in terms of chunk log data size to be processed at slave nodes as well as the bottleneck in processing in truly geographically distributed infrastructures due to the overhead caused by the communication time among the master and slave nodes. Then, an application of the massive processing approach resulting in log data processed and stored in a well-structured format is S. Caballé ( ) Open University of Catalonia, Rambla Poblenou, 156, 08018 Barcelona, Spain e-mail: scaballe@uoc.edu F. Xhafa Technical University of Catalonia, c/ Jordi Girona, 1-3, 08034 Barcelona, Spain e-mail: fatos@lsi.upc.edu presented. We show how to extract knowledge from the log data analysis by using the WEKA framework for data mining purposes showing its usefulness to effectively build user models in terms of identifying interesting navigation patters of on-line learners. The study is motivated and conducted in the context of the actual data logs of the Virtual Campus of the Open University of Catalonia. User modeling [3] is a mature research field mostly involved in the information technology context. It is mainly utilized in software systems for inferring the users' goals, skills, knowledge, needs and preferences, thus achieving more adequate adaptation and personalization on the basis of the user activity pattern built. This inference process relies in turn on being able to track the users' actions when interacting with the application such as the users' choice of buttons and menu items [10] .. In this paper, we focus on and are interested in webbased learning applications that support virtual campuses. These applications, due to the high degree of user interaction, take great advantage of the tracking-based techniques of user modeling, such as providing broader and better support for the users of Web-based educational systems [10] . An important consideration in this context is the building of learner models based on the identification of navigation patterns, which allows for adapting the system's usability to the actual learners' needs resulting in a great stimulation of the learning experience [15] . However, the information generated in web-based learning applications can be of a great variety of type and formats [22] . Moreover, these applications are characterized by a high degree of user-user and user-system interaction which stresses the amount of interaction data generated.. The above features can be extracted to a large extent from information kept in log data files of on-line web-based learning applications. As a matter of fact, log file data processing and analysis of the information captured from the actions performed by learners is a core function for the modeling of the learner's behavior during the learning process and of the learning process itself as well [5, 15] . It is clear that, the larger the taxonomy of events captured in log data files, the richer is the information and knowledge extracted due its processing. It is thus of interest to persist in log files all possible information due to the events produced during userto-system and user-to-user interaction. Keeping all possible information, however, comes to the price of (a) dealing with all sorts of data log formats and (b) large amounts of data, which are not human readable nor ready to be analyzed by statistical or data mining techniques [14, 21] .. Therefore, massive processing is a must to efficiently process large amount of log data files. In fact, one can as well use log data files as a basis for real time decision taking system, and thus, high performance data processing using large scale distributed systems are needful. In this paper we present the implementation and evaluation of some massive processing techniques under different distributed infrastructures such as Cluster Computing [9] and PlanetLab [18] . The aim is to identify the limits of the size of the log files for which these infrastructures are efficient solutions to such processing.. The rest of the paper is organized as follows. In Sect. 2 we describe the aims and background of our approach by first presenting the Virtual Campus and its log data file system as the real context of our study and then showing the problem of processing huge amounts of log data coming from the user interaction in the Virtual Campus. In Sect. 3 we describe the research methodology to validate our approach and the evaluation results achieved from massive processing of the log data files using different distributed infrastructures. Finally, in Sect. 4 we present a data mining application that leverages our massive log data processing approach in order to extract useful knowledge in form of navigation patterns. We end the paper in Sect. 5 with some concluding remarks and outlining future directions of research. In this paper we have presented an innovative approach for the building of learner models based on the identification of navigation patterns, allowing for adapting the system's usability to the actual learners' needs resulting in a great stimulation of the learning experience. This approach is based on a massive pre-processing of log files of a Virtual Campus, which generate huge amounts of information of great variety of type and formats. The massive processing is implemented following the master-slave paradigm and evaluated using different distributed infrastructures (Cluster and PlanetLab platforms). The study showed the need to carefully tune the application in terms of chunk data size to be processed at slave nodes as well as the bottleneck in processing in truly geographically distributed infrastructures due to the overhead caused by the communication time among the master and slave nodes. We show then how to extract knowledge from the analysis of the resulting log data from the Virtual Campus preprocessed and stored in a well-structured format ready for data mining purposes aiming to identify the navigation patterns of online users. To this end, we used WEKA framework for mining the data and more specifically, we used the K-means, Apriori and FPGrowth algorithms implemented in WEKA. The proposed approach showed its usefulness in effectively identifying navigation patters of online users of the Virtual Campus.. Ongoing work is adding the Hadoop platform to our study of massive log processing with multiple distributed infrastructures. Apache Hadoop [11] is an open-source framework that allows for scalable, reliable, distributed processing of large data sets across clusters of computers using simple programming models. It implements a computational paradigm named MapReduce [12] , where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster following the master-slave paradigm. Rather than rely on hardware to deliver high-availability, Hadoop is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. Our aim is to compare the log processing results obtained from the above distributed platforms with the results obtained by using Hadoop.. In our future work, we would like to compare the results of the data mining methods employed in this work with that of a bi-clustering algorithm initially studied in [23] . This approach is also useful to extract relevant knowledge about user activity for other purposes beyond navigation patterns, such as identifying activities performed by students as well as to study time parameters related to such activities. Most of the existing approaches in the literature consider a sequential approach mainly due to three reasons: (i) processing for a specific purpose (i.e. limiting the quantity of information needed for that purpose); (ii) processing the information afterwards (i.e. not in real time) and (iii) processing of small data samples, usually for research and testing purposes (i.e. not for real learning needs). Yet, the lack of sufficient computational resources is the main obstacle to process large amounts of data in real time and hence in real situations this processing tends to be done off-line in order to avoid harming the performance of the logging application, but as it takes place after the completion of the learning activity has less impact on it [6, 23] . Recently, distributed infrastructures, such as Cluster Computing [9] , is increasingly being used to reduce the overall, censored time in processing data by taking advantage of its large computing support. The concept of distributed infrastructures has emerged as a way of capturing the vision of a networked computing system that provides broad access not only to massive information resources, but to massive computational resources as well. Thus, in this paper, we show how a distributed infrastructure approach can be used to match the time processing requirements.. Several studies have been conducted at the UOC [5] [6] [7] 16 ] to show that a distributed infrastructure can increase the efficiency of processing a large amount of information from group activity log files [22] . Some of these studies have involved the interaction data collected from the log files of both the BSCW system [2] used at the UOC to support Problem-Based Learning practices in small groups and the own virtual campus of the UOC. The experimental results allowed us to show first the gain provided by the distributed approach in terms of relative processing time and, second, the benefits of using the inherent scalable nature of the approach while the input log files are growing up in both number and large size.. In this paper we report the different experiments carried out at the UOC that first show the feasibility of the Cluster Computing and PlanetLab infrastructures in terms of the complexity and challenges of massive processing in the big data era and then provide guidelines of how to achieve an effective embedding of the extracted knowledge into e-learning practices.

O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O B I O O O O O O O O B I O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O


Learning semantically-annotated routes for context-aware recommendations on map navigation systems Mocholi Agües, JA.; Jaén Martínez, FJ.; Krynicki, KK.; Catalá Bolós, A.; Picón, A.; Cadenas, A. (2012) Modern technology has brought many changes to our everyday lives. Our need to be in constant touch with others has been met with the cellphone, which has become our companion and the convergence point of many technological advances. The combination of capabilities such as browsing the Internet and GPS reception has multiplied the services and applications based on the current location of the user.However, providing the user with these services has certain drawbacks. Although map navigation systems are the most meaningful way of displaying this information, the user still has to manually set up the filter in order to obtain a non-bloated visualization of the map and the available services. To tackle this problem, we present here a semantic multicriteria ant colony algorithm capable of learning the user's routes, including associated context information, and then predicting the most likely route a user is following, given his current location and context data. This knowledge could then be used as the basis for offering services related to his current (or most likely future) context data close to the path he is following. Our experimental results show that our algorithm is capable of obtaining consistent solutions sets even when multiple objective ontological terms are included in the process. The convergence of technological breakthroughs in cellphones and other portable devices allows us to keep in touch with others at all times and has also led to the birth of new services and applications, such as those based on the current user location. A natural and convenient way of providing users with information on the nearby services available can be achieved by means of map navigation systems. However, at the present time users have to manually filter out non-interesting services in order to reduce the amount of information displayed on the map. In this paper we have presented a semantic multicriteria ACO-based algorithm designed to deal with this problem. Our algorithm learns the user's routes, adds the associated context information and then predicts the most likely route the user is following, given his current location and context data. This knowledge could be used to present the user with the nearby available services related to his current (or most likely future) context data. In this way users are not forced to manually filter out services and still obtain a non-bloated map visualization. Besides, SACO is not forced to use a prefixed set of context variables as in other existing works, what allows users to define their own ontologies with individual meaningful context variables that the algorithm will use. This fact also allows users to easily and meaningfully ask for a set of likely routes for several given context data values. We have carried out a comparison in this paper between SACO and two alternative algorithms widely used for graph-based searching: Tabu-Search and Simulated Annealing. The results showed that SACO obtains high quality solutions whose semantic distance to the restriction set is on average as low as those obtained by using SA and TS but with a significantly lower computation cost. It has been also observed that the computational cost of our approach increases linearly as the size of the problem increases which makes it appropriate when dealing with large size problem instances.. We also developed an application to interact with the algorithm and display the returned routes and have performed several experiments that show that multiple terms of the ontology can be used to obtain a set of solutions. The results also show that the algorithm behaved as expected and did not omit suitable routes from the solution space.. Our plans for future work include increasing the expressivity of restriction sets by using temporal logic operators, such as after, until or next between instances of a certain ontology term, also performing consumer tests to assess the usefulness and acceptance of a map navigation system based on the proposed algorithm, and exploring the use of hybrid metaheuristics in order to assess whether results may improve (either the quality of the solutions or the computation time). In particular, our algorithm allows the integration of optional local search heuristics, as stated in section 3.1. Previous works on hybrid metaheuristics such as [15] [32] [33] have shown that this kind of combination may lead to improvements in the obtained results. However, in our case additional experiments would be needed to evaluate the improvement obtained and its trade-off with respect to the required additional computational work because, as we concluded in [16] , the nature of the problem or the design of the approach may reduce the room for improvement while incurring in significant computational costs.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O