Shared-memory performance profiling This paper describes a new approach to jinding pe$orrrrance bottlenecks in shared-mernmy parallel programs and its embodiment in the Paradyn Parallel Performance Tools running with the Blizzard @e-grain distn"buted shared memory system. This approach exploits the underlying system's cache coherence protocol to detect data sharing patterns that indicate potential pe~ormance bottlenecks and presents pe~ormance measurements in a alzta-centric manner. As a demonstration, Parodyn helped us improve the pe~orrnance of a new shared-memory application program by a fmtor of four. Distributed Shared Memory (DSM) alleviates some of the difficulty of programing a parallel computer by hiding the details of communication. The abstraction of a shared address space, while convenient, can also hide serious communication bottlenecks that cripple a program's performance. To effectively use shared memory, programmers need performance tools capable of piercing this abstraction and relating a program's behavior to the underlying hardware's actions. This paper describes a new technique for collecting and displaying shared-memory performance information. The first key aspect of this approach is to detect cache block sharing patterns that indicate potential performance bottlenecks. Our current system detects these patterns at low cost using a modified cache coherence protocol for the Blizzard formance data in a data-centric manner, which relates events on cache blocks to program data structures.. Shared memory programs communicate through references to shared data. The underlying system (hardware, software, or both) responds to a memory reference by communicating, to obtain a copy of data (which is typically cached for subsequent references) and to maintain the global state necessary to implement a coherent programming model. At a load or store, the actual communication depends on the memory system's protocols and the program's dynamic state in ways that can be difficult to understand or predict [9] .. Nevertheless, detecting this hidden communication is essential to isolate and eliminate many shared-memory performance bottlenecks. For example, a cache block's migratory behavior may indicate false sharing, which a programmer can eliminate by aligning and padding data structures. Moreover, recent research has focused on new tools, such as custom protocols and extensible memory systems, that offer programmers far greater control over shared memory systems [3, 28] . For example, a programmer can greatly reduce the cost of producer-consumer sharing by increasing the cache block size or using an invalidate, rather than update, protocol [7] . Again, understanding a program's access pattern is the necessary first step to improving its shared-memory performance.. The first step in memory profiling is to detect access patterns that indicate performance problems. Our pattern detection mechanism is integrated in a shared memory cache-coherence protocol, so it incurs little overhead and requires no changes to programs. More importantly, this mechanism associates addresses with events, so a performance tool can present measurements at the program's level of abstraction,. The performance tool can use these addresses to relate memory accesses to a program's data structures and use conventional profiling techniques to connect accesses with the statements that execute them. We call this process shured-memorypeq$ormance projiling. Memory profiling by itself cannot find all performance problems, so we built memory protiling into the more extensive facilities of the Paradyn performance tool.. Although this work exploits Blizzard's custom protocols, other systems can also provide mechanisms to associate shared memory communication with data. Any hardware shared-memory platform can support a fine-grain DSM system, like Blizzard [25] or Shasta [24] , that exposes a coherence protocol for performance debugging. Since the underlying shared-memory hardware provides fast communication, a DSM system of this sort would incur only moderate overheads. Alternatively, a platform may provide hardware features, such as informing memory operations that trap on cache misses [12] , which can be used to associate a program's memory references with a coherence protocol's actions. This paper illustrates the use of memory profiling through an extended case study of tuning a new sharedmemory protein folding code [13] from researchers in the University of Wisconsin Chemical Engineering Department. With the help of memory profiling and Paradyn, we improved this application's performance by more than a factor of 4 to an efficiency of 80% on 16 nodes.. The paper is organized as follows. Section 2 briefly reviews the features of I%radyn. Section 3 describes our approach to shared-memory profiling and our implementation techniques. Section 4 contains a case study that illustrates how Paradyn and shared memory profiling help identify and eliminate performance problems. Section 5 discusses automating the search for performance bottlenecks. Section 6 describes related work, F@e 12 summarizes our performance tuning. It shows speedups of the different versions, including the original Fortran version on SGI Power Challenge. The initial version scales moderately up to 4 COW nodes, but its efficiency dropped drasticallyy after that. The restructured program initially performed worse on a small number of COW nodes, because of the extra work to track which particle belong to which node. The program sealed better without the sequential bottleneck, but the false sharing still limited the overall speedup. Aligning and padding the shared data structure improved performance considerably, but made the load imbalance into the limiting bottleneck, A simple static load balancing scheme improved performance slightly. Although coherence was no longer a serious performance bottleneck, we tried adding prefetching (version 3c), which did not significantly improve performance.. To see whether performance optimizations can carry over to different platforms, we also ported version 3a of the application back to SGI Power Challenge using the sharedmemory programming model. The new version outperforms the old Fortran version by 43% (see Figure 12 ). More detailed experiments showed that the performance gain is mainly attributable to eliminating the serial bottleneck. Padding and aligning only contribute up to 3% performance improvement. Since the SGI Power Challenge is an SMP with a much faster network, it is less sensitive to false sharing. The speedup curves in Figure 12 show that the best speedup on COW is slightly better than the speedup on the SGI Power Challenge. This result is encouraging. It indicates that with the help of proper performance measurement tools and performance tuning, a distributed shared-memory machine can perform as well as a hardware shared-memory machine.. Other people have also used the Blizzard version of Paradyn to find shared-memory performance bottlenecks. Brian Toonen used the tool to search for and validate performance bottlenecks in the DSM system itself [28] . Satish Chandra used the tool to monitor the performance of the custom protocols he wrote for his HPF compiler for Blizzard [3] . Trishul Chilimbi used the tool to tune the performance of a database storage management system. This paper describes a new approach to shared-memory performance profiling. This approach consists of detecting sharing patterns and data-centric presentation of the memory performance data. Pattern detection reports memory access patterns that indicate the possibility of excessive communication, Our pattern detection is implemented as an augmented coherence protocol, so it incurs little additional overhead. Although our current implementation is Blizzardspecific, the approach of de~ting sharing pattern by observing coherence events is applicable to many systems.. Memory profiling uses a data-centric view of performance to help a programmer find and understand the performance problems with shared-memory accesses. To organize a large volume of information, we present the shared-memory resources in a two-level hierarchy: data structure and cache blocks belong to a data structure. The data structure view helps relate performance problems to high-level programming language constructs. The cache block view helps isolate specific problems, such as false sharing. Moreover, integration of memory profiling with Paradyn's existing control-oriented view provides a complete view of a sharedmemory program's performance. This approach is feasible because Paradyn's dynamic instrumentation reduces the overhead of monitoring by installing instrumentation only when and where it is necessary.. As a test case, we applied these mechanisms to identify and eliminate performance problems in a shared-memory application. Whh Paradyn. we improved the application's performance by more than a factor of four. The case study demonstrates how these mechanisms can help identify shared-memory performance problems. In addition, the example also shows that, with the help of proper performance measurement tool to optimize communication, a distributed shared-memory machine can be as scalable as a hardware shared-memory system. The memory performance tool successful found the performance bottlenecks in this application. Memory profiling helped isolate the shared data structure that incurred the most coherence misses. Pattern detection helped explicate the algorithm's sharing pattern and isolate the false sharing. However, memory profiling by itself would not have found all problems. Pamdyn's other performance measurement facilities helped find the sequential bottlenecks, load imbalance, and excessive barrier time.. Custom protocols offer a powerful, but time-consuming option for improving program performance. One unanticipated benefit of memory profiling is that it can indicate when custom protocols are not necessary to improve performance. In addition to the systems cited previously, several profiling tools address shared memory programs. StormWatch [4] is tool that visualizes coherence protocol actions and links it to a program's behavior. StormWatch provides three linked, graphic views: trace, communication, and source, which reflect the multiple levels of abstmwtion in an application's behavior. There are two major differences between our work and StormWatch. First, Storm Watch is a postmortem tool that analyzes traces. Our mechanisms use dynamic instrumentation, which is less costly and more scalable. Second, StormWateh provides a more detailed, protocol-specific view of a program's execution, while Paradyn is a general purpose performance tool.. Martonosi et al observed that monitoring memory performance in a multiprocessor is similar to enforcing cache coherence [21 ] . They have also noticed the importance of categorizing miss counts according to the code and data structures that incurred the misses. Their work focused on exploiting unused issue slots in protocol processors to monitor performance. We dynamically instrument the protocol code. In addition, their monitoring mechanisms only collected memory statistics, such as cache misses and latencies, and did not detect patterns.. Others have extended cache-coherence protocols to detect and optimize migratory sharing behavior. Cox et al evaluated adaptive protocols for bus-based and directorybased systems [5] . Stenstrom et al presented an adaptive protocol to take advantage of migratory behavior of a shared-memory application [26] . Our pattern detection serves a different role-to detect particular access patterns, but not optimize their communication-which makes our implementation simpler and cheaper. Moreover, our mechanisms detect patterns other than migratoty behavior. Paradyn creates counters or timers to collect performance metrics for a cache block or data structure and uses dynamic instrumentation to insert code to update a counter or timer. Paradyn's data collection facility periodically reads values from these counters or timers and ships them to the Paradyn front end. For example, to measure the time to handle coherence misses, Paradyn inserts code that starts a timer when a fault occurs and stops the timer when the application resumes. To record performance statistics for a cache block, Paradyn allocates a counter or timer for the block, and inserts code into the coherence protocol to update the counter or timer at faults on the block. To monitor a contiguous data structure, Paradyn updates the structure's counter when a fault falls inside the structure's boundary.. A major challenge in memory profiling is handling the large number of cache blocks in shared-memory applications. These blocks form a much larger focus than the other ones that Paradyn typically handles. We reduced the overhead of shipping shared memory information to the Paradyn front end by compressing shared-memory information and batching several messages together.

O B O O B O O O O O O O O O O B O O O O O O O B B B O O O O O O B I O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O B B O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O B O B I O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O B O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O B O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O B O O O O O O O O O O O O O O B B B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O B O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O B O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O


Evaluation of statistical and multiple-hypothesis tracking for video traffic surveillance Abstract. Conventional tracking methods encounter difficulties as the number of objects, clutter, and sensors increase, because of the requirement for data association. Statistical tracking, based on the concept of network tomography, is an alternative that avoids data association. It estimates the number of trips made from one region to another in a scene based on interregion boundary traffic counts accumulated over time. It is not necessary to track an object through a scene to determine when an object crosses a boundary. This paper describes statistical tracing and presents an evaluation based on the estimation of pedestrian and vehicular traffic intensities at an intersection over a period of 1 month. We compare the results with those from a multiple-hypothesis tracker and manually counted ground-truth estimates. Tracking normally refers to the estimation of an object's state trajectory, i.e., the value of an object's state as it changes over time. In this sense, tracking refers to a process that follows an object as it moves around. The need for tracking to interpret video would seem universal. There are numerous examples in video surveillance, including work by Mikic et al. (1998) , Grimson et al. (1998) , Cohen and Medioni (1998) , and Boult et al. (1998) .. The Kalman filter (Gelb 1974) in its simplest form estimates the state of a single object over time from a series of measurements corresponding to that object. Tracking is rarely that simple -there may be multiple objects with measurements for any subset of them, as well as clutter, i.e., measurements that do not correspond to any object. It is first necessary to associate each measurement with the correct object, i.e., solve the data-association problem. The use of multiple sensors further confounds data association. While methods exist to deal with these factors (e.g., Reid 1979; Barr-Shalom and Fortmann 1988; Cox and Hingorani 1996; Isard and Blake 1998), Correspondence to: J.E. Boyd tracking remains a difficult problem in video applications, especially as the number of tracks, amount of clutter, and number of cameras increases. Boyd et al. (1999) showed that, at least for some applications, estimating trajectories is not necessary for analyzing activity if only long-term patterns are required. Their method of statistical tracking is based on network tomography (Vardi 1996) . Network tomography solves the inverse problem of estimating source-destination (SD) network traffic from link statistics alone. If we model a scene under surveillance as a network of connected regions, then the network nodes are physical spaces and the links are the passages between the spaces. Network tomography estimates the number of trips between any two spaces (SD traffic) using counts of traffic between adjacent spaces (link traffic). Statistical tracking simplifies the tracking process because it only requires detection of an object as it moves from one region to an adjacent region. Methods already exist for counting objects crossing a boundary in video sequences; e.g., Albiol et al. (2000) demonstrate a system for counting people entering and exiting a railroad car through a single door. This statistical approach simplifies processing of video sequences and avoids the difficult data-association problem in conventional tracking. In exchange, statistical tracking gives up the ability to estimate an object's trajectory at the time the object is moving -we can only know about the path after statistics have been collected over the entire network for a period of time. The method also gives up some resolution in the identification of individual trips.. While statistical tracking has been demonstrated (Boyd et al. 1999) , it has yet to be evaluated in any empirical sense. The theoretical underpinnings of network tomography are sound and the numerical methods used to solve the tomography problem always provide a set of nonnegative SD traffic estimates. This paper presents an evaluation of statistical tracking when applied to the estimation of pedestrian and vehicle traffic intensities in an intersection on a university campus over a period of 1 month. As part of the evaluation we also look at the performance of a conventional tracker, the multiple-hypothesis tracker (MHT) described by Reid (1979) and Cox and Hingorani (1996) , and ground-truth estimates based on manual tracking. We selected the MHT because it handles clutter, multiple targets, and a variable number of targets, and an im-plementation that is easily adapted to the traffic surveillance problem is available (Cox and Hingorani 1996) . In contrast, the joint probabilistic data-association filter (Bar-Shalom and Fortmann 1988) tracks a fixed number of targets, and condensation (Isard and Blake 1998) tracks only a single target (although it maintains a probability distribution that can represent multiple targets).. The evaluation does not show that statistical tracking is superior to conventional methods. Instead it shows the differences between the approaches and the applications for which each is best suited. The evaluation also indicates what performance can be expected given certain conditions, and the nature of the errors inherent in statistical tracking. With perfect count data, the statistical tracker produced results comparable to those of the MHT. Counting errors due to occlusions, weather, and limited image resolution confound counting and worsen the statistical estimates. Statistical tracking does not exhibit the systematic errors seen in the MHT due to incorrectly dropping the track of an object. Thus, traffic estimates from the statistical tracker can be improved by gathering data over a longer period, and by improving counting with multiple cameras. The ability to easily use multiple cameras with the statistical method is advantageous. It is ideally suited for tasks such as estimating vehicular traffic in a city freeway system or other large areas where data association for conventional tracking is not practical. In order to eliminate the need for data association, statistical tracking gives up the ability to provide current information about a scene. Finally, the ability to track over large areas with multiple cameras has the potential to adversely affect privacy. However, our results show this potential is somewhat diminished by the fact that low-intensity traffic routes are difficult to estimate when sharing a network with much higher traffic. The plots in Figs. 3-5 and the coefficients in Tables 1 and 2 give different views of the data. To compare them, we consider differences inherent in methodology (i.e., statistical vs conventional) separately from differences due to measurement (counting) errors.

O O O O O B O B O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Visual Targeted Advertisement System Based on User Profiling and Content Consumption for Mobile Broadcasting Television Abstract Content personalisation is one of the main aims of the mobile media delivery business models, as a new way to improve the user's experience. In broadcasting networks, the content is sent "one to many", so a complete personalisation where the user may select any content is not possible. But using the mobile bidirectional return channel (e.g. UMTS connection) visual targeted advertising can be performed in a simple way: by off-line storing the advertisement for selectively replacing the normal broadcasted advertisement. In fact, these concepts provide powerful methods to increase the value of the service, mainly in mobile environments. In this article we present a novel intelligent content personalisation system for targeted advertising over mobile broadcasting networks and terminals, based on user profiling and clustering, as a new solution where the use of content personalisation represents the competitive advantage over traditional advertising. Mobile broadcasting networks, such as DVB-H, have opened new possibilities and new challenges to the actors involved in the value chain. But this evolution should be followed by the provision of innovative services which can help users to satisfy their needs and to fulfil their expectations.. In broadcasting networks, the content is sent "one to many" so a complete personalisation where the user may select any content she/he wants is not possible. For mobile content delivery, 3G/4G networks can be used, but on one hand, content providers prefer to deliver content to mobile via these broadcasting networks to take advantage of the Digital Terrestrial Networks [ ], and on the other hand they are costly for distributing the same content with enough quality to large number of terminals, for example, in Uve events.. But with the help of the return channel, a short piece of content can be off-line recorded and played when appropriate, in this case, as a substitution of the regular broadcasted advertisements, thus giving the user the advertisement which is of his/her interest, and the advertiser a more targeted way to deliver advertisement, increasing its effectiveness.. In this paper we present a novel intelligent content personalisation system for targeted advertising over mobile broadcasting networks, based on user profiling and clustering, and taking advantage of the consumption data obtained from the user as well as the information given by user's tastes and behaviour. If we apply content personalisation on advertising environment, both customers and content providers can benefit, because on one side customers receive only the publicity they may be interested in, and on the other side advertisers can reach their potential public with higher efficiency.. The main difference between this system and other recommendation applications is that our system provides a transparent solution for users, by allowing them to receive the content according to their profiles and behaviour when consuming content, without needing their interaction. This is done by linking the advertisement and the user or users, based not only on data extracted about their socio-economic profile, but also in the real behaviour when interacting, their real media consumption obtained using audience measurement technologies, and their social links.. For a commercial service deployment, it is possible to ask the users to register to the targeted advertisement service, receiving premium content and economical advantages because of the subscription to the service.. The system that we describe in this paper can be applied to broadband mobile delivery networks, such as 3G/4G, where the content is delivered using the mobile channel, and used for both broadband and broadcast network. We think that for mobile broadcasting networks, our proposal may have wider impact.. The rest of the paper is organised as follows. Section 2 outlines the state of the art of recommendation and personalisation systems, focused mainly on mobile terminals. In Section 3, the designed system is described in detail. In Section 4 the classification, clustering and targeting is explained. The system operation is widely explained in Section 5. And in Section 6 the system test and evaluation method used for validating the implementation is explained. Finally, conclusions are presented in Section 7. A new system for mobile advertisements personalisation has been presented. This system represents an efficient and useful solution which can produce a new business model for the mobile visual advertising market. Furthermore, it can be applied to other transmission mechanisms that may allow for content customisation, such as IPTV, digital terrestrial, satellite or cable television (using the bidirectional return channel and the Set-top-box middleware). In this sense, the emerging Hybrid BroadcastBroadband TV systems (Hbb-TV) may clearly benefit from our proposal, as they combine broadband and broadcast media delivery.

O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


VIRaL: Visual Image Retrieval and Localization Abstract New applications are emerging every day exploiting the huge data volume in community photo collections. Most focus on popular subsets, e.g., images containing landmarks or associated to Wikipedia articles. In this work we are concerned with the problem of accurately finding the location where a photo is taken without needing any metadata, that is, solely by its visual content. We also recognize landmarks where applicable, automatically linking them to Wikipedia. We show that the time is right for automating the geo-tagging process, and we show how this can work at large scale. In doing so, we do exploit redundancy of content in popular locations-but unlike most existing solutions, we do not restrict to landmarks. In other words, we can compactly represent the visual content of all thousands of images depicting e.g., the Parthenon and still retrieve any single, isolated, non-landmark image like a house or a graffiti on a wall. Starting from an existing, geo-tagged dataset, we cluster images into sets of different views of the same scene. This is a very efficient, scalable, and fully automated mining process. We then align all views Appl (2011) 51:555-592 in a set to one reference image and construct a 2D scene map. Our indexing scheme operates directly on scene maps. We evaluate our solution on a challenging one million urban image dataset and provide public access to our service through our online application, VIRaL. Images in community photo collections have scaled to billions over the last few years. Searching into such huge collections traditionally depends on text and other community generated data; state-of-the-art visual image retrieval has not yet scaled enough. On the other hand, a number of data mining and clustering approaches have emerged that exploit data like location, time, user (photographer) and tags. Such approaches typically focus on popular subsets where visual representation can indeed help, e.g., images containing landmarks or associated to Wikipedia 1 articles. What is more interesting, new applications are emerging, for instance location estimation as in Hayes and Efros [13] , virtual tourism as in Snavely et al. [41] , and landmark recognition as in Zheng et al. [46] . Such applications are becoming part of larger geographical systems, creating a new experience. For instance, Flickr 2 and Panoramio 3 photos can already be seen directly overlaid in panoramas of Bing Maps Streetside Photos 4 and Google Street View, 5 respectively. While matching and alignment can be done automatically with certain accuracy, photos have to be already geo-tagged. History Pin 6 goes further to collect a user-generated archive of the world's historical images and stories. The process here is manual and users need to "pin" photos on Street View by themselves.. GPS-enabled devices can provide geo-tags automatically, but most photos are still uploaded without geo-tags, let alone historical material. Automating the geo-tagging process would be a leap for such applications. Unlike [13] , which only estimates a geolocation probability map, we are interested in location recognition, that is, accurate matching to images of the same scene. This is typically possible in urban scenes, due to their unique structural details. Along with triangulation, it may also lead to exact localization as in Zhang and Kosecka [45] . Unfortunately, current approaches to location recognition either do not scale well, or focus on popular locations like landmarks.. In our recent work (Avrithis et al. [2] ) we have shown how large image clusters of popular places may help in boosting the efficiency of retrieval, while a distortion bound can guarantee that isolated images are still retrieved as in a generic retrieval engine. This has opened the way to a scalable solution that is still able to retrieve non-landmark photos. In this work we present our retrieval framework, we use it to localize a new landmark or non-landmark photo, and recognize landmarks or points of interest when they do appear. We also present our online application VIRaL 7 -Visual Image Retrieval and Localization-that provides public access to all services via an integrated interface.. Given a large set of geo-tagged images, we group them by location first, constructing geographical clusters. The objective here is to identify images that potentially depict views of the same scene. E.g., two images taken 2 km apart are unlikely to depict the same building. We then use sub-linear indexing to compute pairwise visual (dis)similarities efficiently within geographical clusters and group images depicting the same scene into visual clusters. Given a visual cluster, we align all images to a reference image by homography estimation and construct a 2D scene map by grouping together similar local features of all images of the visual cluster. We extend the entire indexing, retrieval, and spatial matching scheme to operate on scene maps rather than images. This not only provides memory savings, but also increases recall significantly.. We experiment on a challenging one million urban image dataset containing images from 22 European cities. The clustering and mining process is very efficient and entirely automatic. It took about two days on a 8-core CPU, while the baseline visual index was already available. At query time, filtering relevant scene maps takes place in the order of milliseconds, whereas verification and re-ranking according to geometry takes a couple of seconds. Given even a single verified match in the dataset, the location of the query image is inferred and displayed on the map along with all similar images found. Finally, locations and text (title, tags) of similar and nearby images are cross-validated with relevant information in Geonames 8 entries and geo-referenced Wikipedia articles. Whenever a known landmark or point of interest appears in the photo, the relevant article is automatically linked to the photo. Different strategies are followed for clustering in existing research works. For instance, Crandall et al. [10] and Li et al. [24] use mean-shift to perform geo-clustering alone and mine high-density locations corresponding to popular places. On the other hand, a second layer of visual clustering follows in other approaches, using different algorithms including k-means [19] and agglomerative clustering [11, 34, 46] . For geoclustering, [19] and [46] use the same algorithm as for visual clustering, whereas [34] and [11] simply quantize locations into overlapping rectangular tiles. There are also [23, 39] and [6] which perform visual clustering alone. Naturally, this does not scale well.. The main drawback of k-means and agglomerative clustering is that there is no control over the maximal intra-cluster distance. This is crucial because it may lead to geo-clusters with photos taken too far apart, or visual clusters with photos that have too few inliers. Note that k-means requires a vector space anyway, so it cannot use the number of inliers as a similarity measure. On the contrary, KVQ controls distortion and works in arbitrary metric spaces.. Mean-shift [5] , used in [10] and [24] , has a similar property of controlling distortion: in this case the upper bound is the bandwidth parameter of the kernel function, or the scale of observation. However, mean-shift needs seeding and e.g., [10] uses spatial bucketing and samples one photo from each bucket as a seed. There is no such need in KVQ and this is fortunate because bucketing also assumes a vector space and would not apply to visual clustering. The fixed tiles of [34] also control scale/distortion in geo-clustering, but KVQ has the advantage of adjusting to data.. A similar use of KVQ in retrieval may be found in Lampert [20] . As a branchand-bound method, [20] relies on visual similarities within the dataset and would reduce to linear search without visual clustering. With our inverted file index on the other hand, we can still work with isolated images in sub-linear time and yet have the advantage of clustering wherever similarities permit.. Finally, the bottleneck of pairwise distance computation in the visual clustering process is typical in most related work. The same problem appears in Quack et al. [34] who use quite small spatial tiles of 200 m because they need to perform exhaustive pairwise homography estimation within each geographic tile. This will fail to capture scenes that extend spatially to more than 200 m, which is quite often. The same quadratic cost appears e.g., in [11, 39, 46] , while for [19] this is a reason for not using local features. We use larger geo-clusters with r g = 700 m, yet achieve a very fast implementation. This implementation is not as fast as the on of Chum [6] , but we do have the advantage of geo-clustering. This lowers the cost and allows one query per image in each geo-cluster. On the other hand, [6] employs hashing with low recall, and is thus limited to popular locations-isolated photos are unlikely to be discovered.. The above formulation bears similarities with several models in different contexts.. To name a few, Lowe [25] performs local feature view clustering by linking similar features that are matched in adjacent views of an object, applying this representation to 3D object recognition. Simon et al. [39] organize matching features of multiple images into tracks, where a track corresponds to a single 3D point of a scene. They use this representation to produce a visual summary of the scene by means of a set of canonical views. Gammeter et al. [11] perform a similar alignment in visual clusters with the objective of isolating bounding boxes of depicted landmarks. In image retrieval, Chum et al. [9] collect the verified images from a query and build a latent model of the scene by averaging term frequency vectors. This model is used on the query side to perform query expansion. Leibe et al. [21] construct a set of spatial occurrence distributions in an implicit shape model for object detection.. Comparing our model to [9] , the latter does not encode feature position and is constructed dynamically on the query side, whereas scene maps reside on the database side and are static. Unlike the object-based approach of [11] we want to keep information from all image regions. Matching features are linked into connected components in [25, 39] , and we need a similar compact representation, that is, more compact than storing features of individual views. However, we also need to control the size of such components, so that components in a scene map behave like features in a single image. One way is to keep a minimal subset S( p) ⊆ F( p) such that no feature in F( p) is too distant from its nearest neighbor in S( p). This justifies our use of KVQ in this case as well.. Our baseline visual indexing scheme is closest to [33] , because it strikes a very good balance between recall performance and memory footprint. On the other hand, our scene map indexing is is beneficial in terms of both. It is analogous to the latent model for query expansion of [9] , but executing offline on the database side. To draw an analogy, recall that scene maps are statically computed in the off-line indexing process and constrained within geo-clusters. On the other hand, a model is built dynamically at query time in [9] , increasing the computational cost. Without any constraint it is prone to drift, especially when iterative. Most importantly, query expansion cannot help at all when relevant images are too few (or just one) and initial query fails. We compare our approach against two query expansion schemes in Section 8.. Different approaches are followed in related work for tag processing and landmark recognition. A learning framework is followed in [10, 24] , where geo-clustering is only used to construct a ground truth set. A classifier is trained on this set using text, visual features, or both. This approach is mostly limited by the dataset being too noisy, and recognition has scaled up to 500 landmarks, while no location recognition is supported. By contrast, our solution supports approximately 8,500 landmarks or points of interest in the current dataset of 23 cities. It is also interesting that [10, 24] only use tags while we have observed that photo titles are typically more reliable.. Quack et al. [34] follow a quite exhaustive offline process: for each visual cluster they extract tags, query Google for related Wikipedia articles, download photos from several such articles, and then match the two photo sets to verify. However, photos within articles are not quite reliable and the mining process is too slow. By using article location and matching to estimated query location, we can narrow down text search so that verification is now achieved at query time. Article location is used during crawling in [11] , which is quite similar to our approach in this sense, although assignment of articles to visual clusters is again offline.. Zheng et al. [46] also assign visual clusters to landmarks in an offline process, and support approximately 5,500 landmarks from 1,300 cities in 144 different countries. They search into a small subset of representative images in each cluster. By contrast, we use scene maps to search efficiently into the entire dataset, can recognize any point of interest within the supported areas and localize any photo, landmark or not.. While mining from user generated content in community photo collections is becoming popular and new applications are emerging, several possibilities are still unexplored. Sub-linear indexing is not typically exploited in landmark recognition applications, while geo-tags are not typically exploited in large scale 3D reconstruction applications. We have combined both, along with a novel scene representation that is directly encoded in our retrieval engine. The result is a considerable increase in retrieval performance, even compared to query expansion methods, at the cost of a slight increase in query time. Memory requirements for the index are also considerably reduced compared to a baseline system. Contrary to landmark recognition applications, we can still retrieve any isolated image from the original database, allowing location recognition at any region where geo-tagged photos are available. Our mining process is even faster than other implementations that employ massive parallelism without exploiting geo-tags. We also recognize landmarks and points of interest by cross-validating location, photo title, frequent tags and geo-referenced Wikipedia article titles in an efficient online process. Our VIRaL application is publicly available online and provides the baseline visual search together with location and landmark recognition.. In the future we would like to investigate more precise methods in measuring dissimilarity of feature appearance during scene map construction. This will enable much more compression of the index, hence increased scalability, as well as more robust matching. Though our visual clustering does not target perceptual summarization or browsing, it may still be the first stage of such a process, exploiting its compact representation and maximum distortion guarantee. Another immediate application can be exact localization i.e., pose detection. Finally, regarding our online VIRaL application, we intend to incorporate our scene maps indexing scheme in the interface. All evaluation results on location and landmark recognition presented in this paper, together with a summary of the proposed approach, can be found online in our project homepage. He is currently a senior researcher at the Image, Video and Multimedia Systems Laboratory (IVML) of NTUA, carrying out research in computer vision, lecturing in NTUA, and supervising research activities in Greek national and European projects. His research interests include visual feature detection, representation of visual appearance and geometry, image matching and registration, content-based indexing and retrieval, object detection and recognition, image classification, spatiotemporal image/video segmentation and tracking, and video summarization. He has been involved in 14 European and nine national research projects, and has published three books, 18 articles in international journals, 87 in conferences and workshops, and seven book chapters in the above areas. He has contributed to the organization of 15 international conferences and workshops, and is a reviewer in 19 conferences and 15 scientific journals. He is an IEEE member, and a member of ACM, EURASIP, the IET and the Technical Chamber of Greece. The mining process that leads to retrieval with scene maps is entirely automated. Geo-clustering on the European Cities 1M dataset takes less than 5 min and generates 1,677 geo-clusters. Visual clustering creates 493,693 visual clusters. Clustering takes approximately 22 min; however, all queries required to compute visual dissimilarity matrices take approximately 52 h, clearly being the most time consuming process. Construction of all scene maps takes another 5 h. It is noteworthy that 351,391 visual clusters are single images, hence do not need scene map construction. Given larger datasets with more cities, the above times would increase linearly, while of course computation can be made parallel. The inverted index of the new retrieval engine requires 1.20 GB of memory instead of 1.61 GB for the baseline, providing a compression of 25%. It is worth mentioning that all experiments are performed with our own C++ implementation on a 2 GHz Quad Core processor with 8 GB of memory. The total number of tags in the entire dataset is 7,764,264 and the codebook contains 188,989 and 181,752 terms, before and after using a stoplist, respectively. A total number of 2,396,926 tags corresponds to the terms removed by the stoplist.. Visual similarity evaluation To evaluate the performance of the proposed method in terms of visual image retrieval, we compute mean average precision (mAP) on the aforementioned European Cities 1M dataset. We compare our scene map retrieval efficiency against a baseline bag of words and two query expansion methods. The first (QE1) is the naive iterative approach, where we re-query using the retrieved images and then merge the results. In our experiments, this expansion was carried out three times iteratively for each query. For the second (QE2) we create a scene map using the initial query's result and re-query once more. All methods use the same spatial re-ranking approach as described in Section 4. The mAP measurements on the 157 ground truth queries for all four methods under varying size of distractor set are depicted in Fig. 14 . Observe that our method using scene maps (SM) outperforms all other methods in terms of mean average precision. As shown in Table 2 , our method does not differ much from the baseline method in terms of speed, which is clearly the fastest. The proposed method offers slightly roughly to two baseline queries and a scene map construction, and QE1 to several baseline queries, resulting to quite impractical query times. The annotated dataset used contains variable sized groups of images depicting the same scene. Small ones usually correspond to non-landmark scenes while large ones to well known landmarks. Achieving high recall scores is challenging when we deal with a large group of similar images. Re-ranking is only performed on the top ranked images and this can lead to missing quite a few images with the baseline method. Figure 15 shows mAP values for each query, against the size of the corresponding annotated group. Observe that scene maps can yield total recall even for scenes containing more than 100 images. For the same scenes, the otherwise powerful QE1 fails to retrieve all the scene instances, since some images were lost from the initial query before the expansion. Furthermore, almost total recall is observed in the small clusters for scene maps, the images of which are usually contained in a very small number of scene maps, usually one or two. Figures 16 and 17 show a query image, of a non-landmark and a landmark respectively and top ranked retrieved and geometrically verified images. Geometrically Tables 3 and 4 contain mAP values for each group of the landmarks and non-landmarks ground truth respectively. Remarkable is the fact that for many groups scene map achieved perfect mAP equal to 1.0 while other methods achieved a worse ranking of the similar images.. Location recognition evaluation All European Cities 1M dataset images are geotagged. Thus, given the outcome of visual retrieval, location recognition is performed as described in Section 6. To evaluate the proposed scheme, we compare the resulting estimation against the hand-picked geographic location information of each annotated group of images in our European Cities 1M. Localization accuracy in comparison to baseline and other methods is shown in Table 5 . As we see, localization percentage is already high even for the baseline method. Still, our method using scene maps reaches the highest percentage.. Samples of query images depicting well known landmarks and the corresponding localization result on the map are presented in Fig. 18 . The first six cases achieve successful recognition. However in the last two cases we present two examples, coming from the evaluation queries, of unsuccessful recognition based on the ground truth geo-tag which is the exact location of the landmark. Final estimation is far from the ground truth location. This is derived from the fact that geo-tags of user images correspond to the location where the photo was taken from. Thus, these are unsuccessful examples of localizing the landmark but successful ones of localizing the photo.. Landmark recognition evaluation Since most photographers are taking pictures of well known landmarks, we can safely assume that some of the annotated groups of Table 6 , where we see the percentage of correctly discovered links. Experiments are carried on 17 of the groups, that is the dataset subset which depict landmarks and has corresponding Wikipedia articles. We regard a query link suggestion as correct, if the ground truth article link is one of those suggested from the landmark recognition process. As the table shows, recognition for landmark queries is really efficient both with the use of scene maps and query expansion. Samples of query images and the corresponding suggested and frequent tags are presented in Fig. 19 . These are examples of successful landmark recognition.

O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I


A Labeled Transition Model A-LTS for History-Based Aspect Weaving and Its Expressive Power , Nonmember, Yoshiaki TAKATA †b) , and Hiroyuki SEKI †c) , Members SUMMARY This paper proposes an event-based transition system called A-LTS. An A-LTS is a simple system consisting of two agents, a basic program and a monitor. The monitor observes the behavior of the basic program and if the behavior matches some pre-defined pattern, then the monitor interrupts the execution of the basic program and possibly triggers the execution of another specific program. An A-LTS models a common feature found in recent software technologies such as Aspect-Oriented Programming (AOP), history-based access control and active database. We investigate the expressive power of A-LTS and show that it is strictly stronger than finite state machines and strictly weaker than pushdown automata (PDA). This implies that the model checking problem for A-LTS is decidable. It is also shown that the expressive power of A-LTS, linear contextfree grammar and deterministic PDA are mutually incomparable. We also discuss the relationship between A-LTS and pointcut/advice in AOP. key words: labeled transition system, pushdown automaton, formal model, aspect-oriented programming, AspectJ Isao Yagi In this paper, we consider a simple system consisting of two agents, namely a basic program and a monitor. The monitor observes the behavior of the basic program and if the behavior matches some pre-determined pattern, such as deviation from a security policy, then the monitor interrupts the execution of the basic program and possibly triggers the execution of a specific program (e.g., an error handler). As described in detail later, this kind of systems can be found in a few fields of computer science, such as history-based access control in system security, aspect-oriented programming (AOP) and active database.. This paper proposes an event-based system called A-LTS for modeling the above-mentioned systems. An A-LTS is a set of finite state machines (FSMs) consisting of a basic program, a monitor and other machines that are inserted (or woven) into the basic program. The aims of the paper are two-fold: One of them is rather theoretical. Since an A-LTS is a transition system, it is natural to consider an A-LTS as a language recognizer that accepts every sequence of events that brings the A-LTS from the initial state to a final state. Thus, it is interesting to compare the expressive power of A-LTS with those of standard models such as FSM Manuscript received August 3, 2006 . Manuscript revised December 1, 2006 . † The authors are with the Graduate School of Information Science, Nara Institute of Science and Technology, Ikoma-shi, 630-0101 Japan.. a) E-mail: isao-y@is.naist.jp b) E-mail: y-takata@is.naist.jp c) E-mail: seki@is.naist.jp DOI: 10.1093/ietisy/e90-d. 5.799 and pushdown automata (PDA). The other is to investigate automatic verification (or model checking [6] ) method for systems that can be modeled as A-LTS. Among a number of types of formal verification, we are interested in formal verification in which a verification property is modeled as a subset of execution histories of a system. This type of formal verification clearly captures behavior of a system, and by appropriate modeling, automatic verification called model checking can be performed. Much research on formal modeling of this kind of systems has been done; however, a simple and clear model suitable for model checking has not been established yet (see Sect. 2.1.2). Furthermore, these two aims are closely related since there is certain relation between the language expressive power and the decidability of model checking. Thus, clarifying the expressive power of a new class of system models may shed light on the decidability and complexity of model checking of the class. We show that the expressive power of A-LTS is strictly stronger than FSM and strictly weaker than PDA under language equivalence, bisimulation, and isomorphism. This result implies that formal verification of a program modeled as an A-LTS is decidable using a model checking method for PDA [9] . Next we compare in detail the expressive power of A-LTS with a few subclasses of PDA (or equivalently of context-free grammars): classes of deterministic PDA and linear grammars.. A-LTS resembles PA (pointcut and advice, cf. Sect. 2.1.1) of AOP languages such as AspectJ [3] , but is simper than PA in the following sense:. 1. An A-LTS is a data-less (or value-free) event-based transition system. In this paper, we proposed a simple formal model A-LTS for history-based aspect weaving. We compared the expressive power of A-LTS with FSM and PDA under language equivalence, bisimulation, and isomorphism. As a result, we showed the relationship among a few subclasses of contextfree languages and the class of the languages of A-LTSs, † Let execute m be an event that represents the beginning of the execution of the body of method m.. shown in Fig. 11 , and FSM ⊂ = A-LTS ⊂ = PDA under language equivalence, bisimulation, and isomorphism. Finally, we stated the relationship between pointcuts in A-LTS and in AspectJ, the most popular implementation of the pointcut and advice mechanism. Since A-LTS is a subclass of PDA, formal verification of a program modeled as an A-LTS is decidable using a model checking method for PDA.. As future work, we will compare the expressive power of A-LTS with other models; e.g., context-free processes [19] and recursive state machines (RSM) [1] , [4] . Moreover, we will discuss a verification of A-LTS using model checking proposed by [1] , [4] , because we conjecture that A-LTS is a subclass of RSM. Benedikt et al. [4] discussed the complexity of the verification of a few subclass of RSM. Following their results, we will try to find an upper and a lower bound of the complexity of the verification of A-LTS. We model a basic program and advices written in AspectJ as follows. Event set of a program is the set of join points and other related actions. For example, call to a method and assignment to a data field are regarded as events. A basic program and advices are defined as processes executing events in a specific order. As stated above, in AspectJ, the execution of the whole body of a method is regarded as a single join point. However, if we consider this join point as an atomic event, then we cannot represent the recursion of method call. Endoh et al. [8] proposed a new join point model that is finer grained than AspectJ. In the model, the start and end points of a method execution are considered as distinct join points. We follow this idea to represent "before call(m)" and "after call(m)." Let call m be an event that represents the call to method m and reception m be an event that represents the return from m. Pointcut "before call(m)" of AspectJ is denoted as a regular language P call m = Σ * call m in A-LTS. Pointcut "after call(m)" is represented as Σ * reception m . We consider pointcut "execute(m)" in a similar way.. Next we consider pointcut "before get( f )." Let get f be an event that represents a reference of the value of data field f . In A-LTS, an advice is invoked just after an event that leads the current state of a pointcut to a final state. That event should not be get f itself because we want to invoke an advice before get f occurs. Thus, we need a prelude event prelude get f that represents just before the reference of f , and we define "before get( f )" as Σ * prelude get f . We can consider pointcut "set( f )" in a similar way.. Finally we consider pointcut "cflow(pc)." Let P pc be a pointcut of A-LTS represented by a regular language that represents pc. Then "after cflow(pointcut)" is represented as P pc + P pc Σ * Σ jp , where Σ jp is the set of all events that correspond to join points. Similarly, "before cflow(pointcut)" is represented as P pc + P pc Σ * Σ b jp where Σ b jp is the set of all call m , all execute m † , and all the prelude events.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I B I B I B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A model for the evolution of concrete deterioration due to reinforcement corrosion One of the most crucial factors affecting the service life of reinforced concrete (RC) structures attacked by aggressive ions is reinforcement corrosion. As the steel corrosion progresses, crack propagation in concrete medium endangers the serviceability and the strength of RC structural members. In this study, a nonlinear mathematical model for determining the displacement and stress fields in RC structures subjected to reinforcement corrosion is introduced. For corrosion products, a nonlinear stress-strain relation which has been previously confirmed by experimental data is incorporated in the present analysis. In formulation of the governing equations for steel-rust-concrete composite, the rational behavior of corrosion products and penetration of rust into the microcracks are considered. An analytical approach as well as an innovative meshless method, gradient reproducing kernel particle method (GRKPM), are employed for solving the nonlinear boundary value problem. A reasonably good agreement between the results of the two methods is achieved. The performance of the proposed model is then investigated through various comparisons of predicted values with experimentally observed data, and again good agreement is obtained. Moreover, the effects of the crucial parameters associated with the mechanical behavior of rust and concrete on time to cover cracking and some measures of deterioration are studied for different values of rust penetration into the microcracks. Reinforcement corrosion due to its role in causing severe damage to reinforced concrete (RC) structures has been increasingly paid attention to by the scientific communities of various disciplines during the past three decades. It is believed that reduction in the pH level in electrolyte, which is present in the concrete pores near the rebar, occurs as the aggressive ion concentration (such as chloride and sulfate ions) reaches the threshold value. This chemical process is followed by removal of the passive film over the rebar [1] [2] [3] . As a result, the anode and cathode will be configured and the corrosion electrochemical reactions will start. The corrosion products occupy more volume than the original steel rebar and thus the corrosion processes accompany expansive pressure on the concrete's inner surface. The increasing trend of rust production to a threshold value results in the initiation of microcracks around the reinforcement. Loss of load transfer between the concrete medium and the rebar as well as the decrease of the concrete's strength are a manifestation of the increase in corrosion products at the rebar-concrete interface and crack growth within the concrete. As the corrosion reactions continue, these two factors become serious threats to the service life of such RC structures. Hence, if one could properly model rust production and concrete damage mechanisms due to rebar corrosion, it would be of great value in realizing failure prevention measures and more accurate service life predictions of the structures.. Bazant [1] presented an analytical model for predicting the time to cover cracking. His theory accounts for the reinforcement diameter, cover thickness, the tensile strength of concrete, the corrosion rate and the corrosion products' density. Dagher and Kulendran [4] considered the formation of smeared cracks caused by the exertion of uniform pressure from the corrosion products at the reinforcement concrete's interface. It was found that a radial expansion of 0.008 mm is sufficient to cause degradation in concrete bridge decks. Molina et al. [5] also used a smeared crack model for the finite element analysis of cover cracking due to reinforcement corrosion. The corrosion of a steel element is modeled by presuming a linear variation of the material properties from those of steel to those of rust. Due to lack of information, they assumed that the mechanical properties of rust nearly resembles that of water which is one of the main constituents of rust. Their analysis was based on the experiments of Andrade et al. [6] where the thickness of the concrete cover was 1.25 and 1.9 times the reinforcement diameter. They found that a corrosion penetration of 0.02 mm was sufficient to initiate a surface crack in the specimens. Noghabai [7] adopted an analytical model for the splitting capacity of a thick-walled concrete ring subjected to an inner pressure based on nonlinear fracture mechanics. Based on field and laboratory data, Morinaga [8] suggested an empirical equation for predicting the time to cover cracking without reference to the evolution of the damage zone. His empirical equation accounts for the time to cover cracking as a function of the corrosion rate, concrete cover thickness and reinforcing diameter. A close scrutiny of the literature [9, 10] reveals that the cover cracking predicted by Morinaga's model is much shorter than the experimentally observed values. Pantazopoulou and Papoulia [11] developed an analytical model to investigate how corrosion products cause concrete cover cracking which endangers the service life of RC structures. They assumed that the rebar and the corrosion products behave like a rigid material. Moreover, the corrosion production is simply modeled by imposing a radial displacement on the inner surface of concrete. They investigated the effects of cover thickness, material properties of concrete, and the rate of rust accumulation on the time of cover cracking. However, further investigations in this field have revealed that the consideration of the appropriate mechanical behavior of corrosion products strongly affects the predicted results [12] . Lundgren [13] , in an effort to study rebar pull-out, presented a reasonable way to model the effect of corrosion on the bond between the corroded reinforcement and concrete. Lundgren employed finite element analysis and assumed that rust behaves like granular materials, in accordance with the experimental tests of Andrade et al. [6] , Al-Sulaimani et al. [14] , Cabrera and Ghoddoussi [15] , and Ghandehari et al. [16] . In another work, Lundgren [17] used the previously developed model of rust ( [13] ) together with a bond mechanism model to explore the effect of uniform and localized corrosion on RC beams with corroded reinforcements. Lundgren pointed out that the model could predict the decrease of the bond when splitting of the concrete occurs. It was emphasized that axisymmetric analysis appears to be a satisfactory level of modeling when a study of concrete cover cracking due to uniform corrosion is of concern. However, three dimensional models should be used if localized corrosion is to be studied. The Young's modulus of iron oxides was measured by Ouglova et al. [18] from both ultrasonic measurements and mechanical tests. The tests were performed on dried iron oxides which were in a powder state. The chemical structure of the laboratory grown oxides were similar to those obtained from 40-year old corroded RC structures. The results showed that iron oxides exhibit a nonlinear behavior with notable plastic strain similar to the powder materials. These results confirm those obtained by Lundgren [13, 17] . Moreover, the elastic modulus of iron oxides increases with the oxide's volume fraction and drying time of the iron oxides. Recently, Bhargava et al. [10, 19, 20 ] presented a mathematical model to predict the time to concrete cover cracking and weight loss of reinforcing bars. However, they assumed that the mechanical properties of corrosion products are the same as those of steel, though reasonably good agreement was obtained between experimental results and the analytical predictions. They showed that tensile strength, initial tangent modulus of concrete, annual mean corrosion rate, and modulus of elasticity of the reinforcement and corrosion significantly influence the predicted time to cover cracking.. In this paper a nonlinear mathematical model is formulated to determine the displacement and stress field in RC structures due to rebar corrosion. The reasonable modeling of the mechanical behavior of rust and the penetration of the corrosion products into the microcracks leads to the nonlinearity of the governing boundary value problem (BVP) of the steel-rust-concrete composite. For this problem, both analytical and numerical solutions are provided. It is proposed to employ an innovative meshless scheme in which the gradient term is incorporated into the reproducing equation of the reproducing kernel particle method (RKPM) [21] . This new approach, presented by Shodja and Hashemian, is called the gradient RKPM (GRKPM) because of the added gradient term [22] . Later, Hashemian and Shodja [23] extended the GRKPM from one to three-dimensional Euclidean space. Application of the GRKPM to the problems of beam-columns and plates shows remarkable results [22, 23] . Furthermore, the high performance and accurate resolution of GRKPM when dealing with the Burgers and Buckley-Leverett equations exhibiting evolutionary high gradients and shocks are well established [24, 25] .. In meshless techniques, the domain of interest is discretized via particles. Therefore, there is no mesh distortion when the medium experiences large deformations. Furthermore, any refinement is possible at a small cost only by inserting additional particles into the desired regions. This remarkable property of meshless methods is particularly attractive for analyzing static and dynamic crack problems as well as problems whose solutions are characterized by steep gradient profiles [26] [27] [28] [29] . More detailed surveying of meshless methods has been given in [30] . It should be noted that to overcome the drawback of the finite element method (FEM) in the simulations where mesh distortion occurs, Liu et al. [31] proposed the smoothed finite element method (SFEM) as an alternative technique. They have pointed out that, SFEM generally achieves more accurate results and a higher convergence rate without increasing the computational effort. The convergence and accuracy of SFEM have also been investigated in some detail by Nguyen-Xuan et al. [32] . They concluded that SFEM attains a higher accuracy and convergence rate than the conventional FEM, particularly for incompressible linear elasticity problems.. In the present work, the performance of GRKPM is investigated through various comparisons of the GRKPM solutions with those of the analytical results and experimental data. Moreover, the effects of the parameters associated with the mechanical behavior of rust and concrete, on the variation of crucial parameters of the embedded rebar under corrosion are explored in some detail for various values of rust penetration into microcracks. A nonlinear mathematical model is developed to determine the displacement and stress fields in RC structures due to rebar corrosion. The mechanical behavior of steel, rust and concrete is assumed to be linear isotropic, power law stress-strain relation and nonlinear anisotropic with postcracking softening, respectively. Presuming the radial strain of a thin film of rust as a nonlinear function of radial stress at the rebar-concrete interface and allowing penetration of corrosion products into the vacant spaces of microcracks as a nonlinear function of crack width openings yield to nonlinearity of the boundary condition of the governing equation for steel-rust-concrete composite media. The mathematical model has been analyzed both numerically and analytically with good correspondence; the minimum and maximum discrepancies in the results are about 0.1% and 3%, correspondingly. The predicted values of the effected parameters have been verified with those of experimentally observed data. The available experimental tests [33, 39] do not provide sufficient data about the chemical composition and physical properties of corrosion products. Hence, the approach of Liu [33] is adopted for choosing a range of values for r m , which is an indication of the chemical composition of rust. Moreover, there is no observed data about the amount of corrosion products which penetrate into the vacant spaces of microcracks during the corrosion processes.. Therefore, the volume fraction of the microcracks filled with the rust (β) is estimated based on the best correspondence between the predicted and experimental data for the amount of consumed rebar. It is observed that for low values of corrosion current density (i corr < 10 µA/cm

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On cores and stable sets for fuzzy games In this paper cores and stable sets for games with fuzzy coalitions are introduced and their relations studied. For convex fuzzy games it turns out that all cores coincide and that the core is the unique stable set. Also relations between cores and stable sets for fuzzy clan games are discussed.MSC: 90D12; 03E72 The purpose of this paper is to study cores and stable sets for fuzzy games. In Section 2, we review definitions of the core, the dominance core, and the stable set (von

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O B I O


Trip destination prediction based on past GPS log using a Hidden Markov Model In this paper, a system based on the generation of a Hidden Markov Model from the past GPS log and current location is presented to predict a user's destination when beginning a new trip. This approach drastically reduces the number of points supplied by the GPS device and it permits a "support-map" to be generated in which the main characteristics of the trips for each user are taken into account. Hence, in contrast with other similar approaches, total independence from a streetmap database is achieved. In developed societies, life is lived in a sedentary and comfortable environment where the displacements of a person within a geographical region usually follow a pattern. Most of these displacements are carried out towards known places such as home, the workplace, relatives' houses, a favourite cinema or fashionable shopping centre. Hence, patterns of these movements can be studied and a prediction system can be designed by allowing all the places where a user could go to be stored in order to anticipate arrival.. It is worth noting that from the elimination of selective availability (that introduced random errors of up to a hundred metres) on May 1, 2000, GPS receptors have evolved from research tools to become consumer goods. The E-911 mandate 1 in the United States and the E112 recommendation in Europe have boosted the development and use of systems based on GPS. Nowadays, millions of civilian users have a GPS navigation system to help them drive to previously unknown locations. However, future location and trip prediction systems are on-going topics in research, and not a software product in the same way as navigation systems. Along these lines, some interesting scenarios have been studied: predictions of meetings between two users and intelligent interruption [1] , location to-do list [9] , and optimization of a hybrid vehicle charge and discharge schedule [4] .. The probabilistic models of destination prediction using GPS was born in 2003 with the work of Ashbrook and Starner [1] . A Markov model was used to find the next most likely destination based on those that had recently been visited from a set of previously clustered candidate destinations. In the same year, a Bayesian model of a traveller moving through an urban environment was presented in [10] , and a year later this development was improved in [8] with a predictive hierarchical model that can learn and infer a user's daily movements using different means of transport. In 2006, the "Predestination" model [7] used Bayesian inference to obtain a probabilistic map of destinations; and a Hidden Markov Model (HMM) is given in [12] to predict the next link road and the final destination of an user. By following similar reasoning, an n th -order Markov model is trained in [6] to probabilistically predict future road segments based on a short sequence of just-driven road segments.. It is important to note that all of the aforementioned work needs external information from the geographical environment. Hence, a street-map is required in [10, 8, 6, 12] since all locations must be within its range. The street-map is modelled as a directed graph whose edges correspond to streets or footpaths and whose vertices correspond to intersections, hence by knowing the current location, the next transitions can be predicted. Furthermore, to adapt the GPS points received from the receptor to the map database, a process called map-matching is developed in these papers. This process is set up to work on a road level but not on a point level, since all approaches previous to this paper are based on partial trips which work with all the roads that the user traverses until the final destination.. Moreover, the street-map database needed in these earlier probabilistic models makes their predictions local to the cities on which they are dependent. All these papers have shown that a common user (not including salespeople, delivery people, etc.) only traverses a reduced percentage of all the existing roads in a city, and hence, in our approach, a Geographical Information System (GIS) is not a constraint of the construction of the user's geographical model.. In this paper, the technique used to extract probable destinations is similar to that given in [10, 8, 6, 12] , that is, to analyze the GPS journeys of users by extracting clustered destination points from these journeys. Furthermore, a Hidden Markov Model is generated in our approach by using a new procedure to extract the significant points which have been used to predict the route and finally the destination of one user when the trip is not yet completed.. The generated HMM 2 is set up to detect an invisible state process by using a visible observation sequence of another process. In this approach, the invisible process is the goal to reach (the destination place) and the visible observations are the sequence of significant points, called support points, that compose a route. Therefore, map-matching problems are avoided in the developed system since it is independent of any street-map.. The paper is structured as follows. In Section 2, the complete process to obtain the data from the GPS logs of different users is described. The HMM for predicting destinations is presented in Section 3. A practical implementation based on this model using a corpus of real driving, running, walking and biking data is provided in Section 4. Conclusions are drawn in the final section. A new approach has been presented to predict destinations given only data of a partial trip by using Hidden Markov Models. Although there are some previous studies related to this approach, they all use a street-map of the city under study and therefore any comparison of results cannot be carried out. It is important to point out that our approach generates its own local street-map and this improves predictions for each user.. In our approach there is a drastic reduction of significant points (ranging from 97.74% to 99.72%) from the first data log information obtained from a user to the resulting set of support points. Hence, the local street-map generated by this approach is simpler than a street-map of the city. Furthermore, the street-map generated is unique for each user.. Due to the reduced number of support points, our "light" HMM can be implemented on mobile devices which include the GPS capability, and can therefore predict destinations accurately in real time, anywhere and without the need of any street-map database. In our experiment, the percentage of correct predictions carried out by the HMM can be considered highly competitive with any other approach.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O