A Statistical Framework for Image Category Search from a Mental Picture Abstract-Starting from a member of an image database designated the "query image," traditional image retrieval techniques, for example, search by visual similarity, allow one to locate additional instances of a target category residing in the database. However, in many cases, the query image or, more generally, the target category, resides only in the mind of the user as a set of subjective visual patterns, psychological impressions, or "mental pictures." Consequently, since image databases available today are often unstructured and lack reliable semantic annotations, it is often not obvious how to initiate a search session; this is the "page zero problem." We propose a new statistical framework based on relevance feedback to locate an instance of a semantic category in an unstructured image database with no semantic annotation. A search session is initiated from a random sample of images. At each retrieval round, the user is asked to select one image from among a set of displayed images-the one that is closest in his opinion to the target class. The matching is then "mental." Performance is measured by the number of iterations necessary to display an image which satisfies the user, at which point standard techniques can be employed to display other instances. Our core contribution is a Bayesian formulation which scales to large databases. The two key components are a response model which accounts for the user's subjective perception of similarity and a display algorithm which seeks to maximize the flow of information. Experiments with real users and two databases of 20,000 and 60,000 images demonstrate the efficiency of the search process. O UR scenario is this: A person has an image concept or category "in mind." This category is essentially semantic and might be represented by various "mental pictures," by an actual object or photograph in hand or merely by subjective impressions. The person wishes to view images in a large database which match this concept. For example, the person is thinking about "old bridges" and has access to the Alinari database; see Fig. 4 . The database is not semantically annotated, for example, by a rich variety of keywords. Even if it were, as pointed out in [1] , there are situations in which the person may have difficulty in expressing his concept in words; he will know it when he sees it. Moreover, it may be more efficient to look over displayed images ". . . and make unconscious 'matches' with the one drawn by imagination. . . " than to rely on text or keywords that may not capture the concept. Our objective is to design a system to accommodate this "user," more specifically to get started by finding a first exemplar. This is the "page zero problem.". Scenarios like this, and the increasing demands of managing the large quantity of existing multimedia documents, have generated a growing interest in content-based retrieval techniques, both from academia and from industry [1] , [2] , [3] , [4] . Query-by-Example (or QBE) is successfully used in many retrieval systems for ranking the elements in a database according to their similarity to a "query image" [1] . This does not solve the page-zero problem since the query image must be available. However, once an exemplar is found, QBE does allow the system to display other images that might even better match the user's concept.. Since a user's concept of similarity is largely semantic, the efficiency of the search process, whether for QBE or mental matching, is adversely affected by the infamous "semantic gap"-the discrepancy between the low-level representations of images and the high-level descriptions meaningful to users [1] , [3] . Indeed, in many cases, images that present similar low-level descriptors may have very different semantic content. A partial solution is provided by relevance feedback (or RF): Divide the search session into several rounds and solicit information from the user at each step, for example by asking the user to declare which displayed images are "relevant" and which are "nonrelevant" with respect to the desired target category [5] . The system iteratively refines a model of the user's target category and uses this model to filter hopefully relevant images from the database.. Due to the page-zero problem, most proposed systems either assume that a starting image has already been identified by the user, or that the query is seeded by keywords. Some simply display randomly sampled pages from the database until the user identifies a suitable starting point. This rapidly becomes impractical for large databases. More direct solutions have been explored as well, such as database categorization [6] and query construction [7] . Other methods, initially directed toward other objectives, such as mental matching for target search [8] , [9] and automatic semantic annotation [10] , [11] , could be adapted to the initialization problem. These connections will be amplified in Section 2.. Our main contribution is a new, iterative approach for discovering an instance from a semantic image category residing in the mind of the user. The search is terminated upon displaying one of these images and performance is measured by the expected number of iterations necessary to achieve this. No semantic annotation is assumed. Also, unlike previous approaches to mental matching, ours extends from target to category search and from small, structured databases to large, unstructured databases. ("Unstructured" means that the images in the database are not labeled by semantic categories.) It could serve either as a stand-alone module in a retrieval system or as a method for initializing another session, such as QBE, to obtain additional examples.. The core of our framework is a new statistical model for relevance feedback by mental matching. A binary random variable is assigned to every image in the database; the value is one if that image belongs to the target class and is zero if it does not. Taken together, these variables determine the category. The relevance feedback session starts with a random screen. At each iteration, the user is asked to choose from among the displayed images the one that is closest to his target category using whatever criteria he desires. The interface used in our experiments is shown in Fig. 12 . Obviously, the target class is not displayed during a search session, ensuring that matching is entirely "mental." Even if exemplars were on display, the decisions are inevitably subjective; indeed, the challenge is to design an "answer model" which accounts for the nature of human decision making, hopefully capturing the gap between the user's "metric" and the one used by the system. Formally, the answer model is the probability distribution for the user's response conditional on the membership status of any given image.. The system maintains a separate, iteration-dependent posterior distribution for each image. Probabilities are updated based on the evidence gathered from the search, i.e., the responses of the user. The evolution of this distribution is depicted in Figs. 15 and 16 for two search sessions, one relatively efficient and one relatively inefficient. Theoretically, the optimal new display would minimize the conditional entropy on the whole family of membership variables conditional on the search history and the new response. As this is computationally intractable, we use an extension of the heuristic proposed in [9] , which is shown to work very well in practice. Moreover, in order to overcome certain problems introduced by the redundancy among images with very similar low-level descriptors, we use an unsupervised categorization of the database into small clusters that are visually highly coherent. The efficiency of the search is illustrated by experiments with real users on two databases of sizes 20,000 (Alinari) and 60,000 (Corel). In both cases, fewer than five iterations are sufficient to locate an instance from a category of order 100 in 50 percent of the searches and fewer than 10 iterations in about 80 percent.. A preliminary version of this system appeared in [12] ; the one presented here is considerably more mature in both practical and theoretical terms (see Section 2).. The paper is organized as follows: Related and motivating work is discussed in Section 2. In Section 3, we formulate the statistical framework for interactive search-a Bayesian relevance feedback model consisting of an update model (Section 3.1), answer model (Section 3.2), and display algorithm (Section 3.3). The lowlevel image descriptors and the clustering algorithm are described in Section 4, followed by an analysis of the "enabling assumption" in Section 5. The parameters are estimated in Section 6 and the whole system is evaluated in Section 7. Finally, we conclude in Section 8 with a discussion of our findings and some speculative remarks. We have presented a Bayesian framework for discovering an instance of a semantic category residing in a large, unstructured database using relevance feedback. Since the category is known only to the user of the system and since we assume no semantic annotation, the feedback is based on mental matching at the image level. Our framework centers on an evolving estimate of the probability that each member of the database belongs to the user's category. A central feature is a new Bayesian model, which includes a pair of positive and negative answer models which are designed to account for subjectivity of the user's choices and their weak correlation with the system metric. The performance of the system is validated on two fairly large databases and very reasonable search times are demonstrated.. It could be argued that semantic annotation of image databases will eventually become feasible and allow for far more efficient text-based search. In particular, searches based on visual similarity will not be necessary. There are at least two holes in this argument. First, "eventually" may be a very long time; progress in automated image interpretation is slow, especially at the level of multipleobject detection and context labeling in unconstrained scenes. Second, even if one is able to automatically annotate image databases with keywords, it is unlikely that this alone will solve the page zero problem for very large databases. (Indeed, some users may not even be able to express in words the nature of their mental pictures.) What is more plausible is that keywords provided by the user will serve to construct, online, a user-specific "prior distribution," effectively filtering a subset of images for analysis based on visual similarity. In any case, for gigantic databases, the number of images remaining as plausible candidates after textual filtering may be on the order of the sizes considered here.. Finally, our statistical framework depends only on the metric between documents; the only input is a distance matrix. Consequently, the system could, in principle, be adapted to other media. For instance, one can imagine trying to find a song in the mind of a user based on choosing among acoustical snippets in a music database, or a system for exploring a database indexed by multimodal descriptors. Marin Ferecatu received the PhD degree in computer science from the University of Versailles/INRIA. He is currently a postdoctoral fellow in the TSI Department at the Institut Telecom, Telecom ParisTech, LTCI CNRS, UMR 5141, Paris, France. His current research interests include machine learning algorithms for multimedia content description and retrieval, more specifically information mining and query personalization in large image repositories using hybrid text and visual descriptors. His scientific interests include machine learning, data clustering and classification, object recognition, and content-based multimedia retrieval. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib. In this section, we describe a number of issues we encountered during our experiments with users of the system. Relevance feedback has matured into an effective method for dealing with the semantic gap in image category search. Assuming a starting image, feedback strategies exploit highlevel information provided by the user in order to discover other images which represent a target class. Recent advances based on kernel methods [13] , [14] avoid restrictive assumptions about the data (e.g., that classes are elliptically shaped in feature space), are flexible, and allow for efficient learning and searching even for large databases [15] , [16] . Other successful approaches include active learning [15] , [17] , manifold learning [18] , [19] , graph Laplacians [20] , and utilizing enhancements of the training set, for example, user logs [21] and query expansions from unlabeled images [22] ; see Zhou and Huang [5] for a review and [1] , [3] for connections with other machine learning and multimedia retrieval methods. The shared aspects with our work are the feedback loop and incremental learning. The key difference is that we seek an initial element of the target class starting from a random display (the page zero problem).. Perhaps the most straightforward solution to the page zero problem, at least for target search (singleton categories), is to ask the user to create the starting image: This is called "query-by-sketch" and was a part of the first image retrieval systems, for example, QBIC [23] . Similarity is then based on shape matching and evidently the results depend on the ability of the user to draw the desired query target. Recent research has focused on elastic matching of images [24] , color [25] , or matching the sketch to an automatically determined relevant subset of regions [26] . Following the same idea, Fauqueur et al. [7] fabricate a query example by composing image patches (regions), utilizing a visual thesaurus composed of many region categories ("sky," "building," "grass," etc.) and logical connectors.. With the page zero problem in mind, Lesaux et al. [6] create a summary of the image database from unsupervised categorization followed by a user-guided refinement of the resulting clusters. Cluster prototypes then provide a summary of the database that can be consulted to find a suitable query point.. For databases which are semantically annotated, a visual search session can, of course, be seeded by keywords provided by the user. Understandably, then, automatic image annotation has generated a lot of interest lately, even if the methods remain far from reliable and the state-of-the-art unsatisfactory. For example, Li and Wang [11] represent semantic concepts by feature-based probability distributions, allowing for models to be updated as the database grows without massive retraining. Carneiro et al. [10] model images as bags of localized feature vectors, estimating a mixture density for each image; the mixtures associated with images with shared annotations are pooled into a density estimate for the corresponding semantic class. Once images are associated with semantic concepts, by whatever method, new queries can be seeded by using natural language or keywords. Even if the annotations are not completely reliable, the user may still find a suitable starting point among the retrieved results.. In the area of category search, but assuming a starting point, Caenen and Pauwels [27] assign to each image in the database a probability that reflects its relevance to the user's intentions. The system is based on a quadratic logistic regression model used to select the next sample of images that will be presented to the user for individual annotation. There is no mental matching. The shared feature with our work is the image-specific distribution and a statistical framework.. A number of probabilistic frameworks for content-based image retrieval have been proposed in the last few years. Vasconcelos and Lippman [28] minimize the probability of retrieval error by combining feature selection and similarity measures into a Bayesian formulation. See also [29] , where the same authors formulate the problem of retrieving images using Bayesian inference; the algorithm relies on belief propagation to account for both positive and negative examples of the user's preferences. Su et al. [30] suppose that the elements of the target class are generated by an underlying Gaussian density and use a Bayesian-classifier reranking of the images after each feedback step. All these results, while not directed toward solving the page zero problem, do establish that probabilistic frameworks, albeit computationally intensive, can provide state-of-the-art results in standard scenarios.. Mental matching seems to have first appeared in the seminal work of Cox et al. [8] on iterative search for a specific image in the database (target search). At every round, the user is asked to choose which of two images displayed by the search engine is "closest" to the target image residing in his mind. The formulation in [8] does not extend to category search because the mechanism gathering information ceases to be computationally feasible. Indeed, one cannot maintain a probability distribution on arbitrary subsets of images, even for small databases. Also, the answer model does not accommodate more complex user behavior associated with displaying multiple images, which is necessary to achieve reasonable search times with large databases.. Fang and Geman [9] and Ferecatu and Geman [12] extended the Bayesian framework introduced in [8] . In the context of target search, an efficient, entropy-based display algorithm was proposed in [9] and applied to mental face retrieval. We shall adapt their display mechanism to our purposes in Section 3.3. Also, unlike in [8] , the answer model is explicitly designed to capture human decision making (through learned parameters). Still, the approach in [9] does not scale to large generic and heterogeneous databases, both computationally and in terms of number of feedback rounds necessary to reach the target. Indeed, the user's notion of similarity is more complex for generic images than for faces and his choices are less likely to be coherent with the feature-based metric employed by the system. Nor does the method in [9] extend to category search and unstructured databases.. The direct precursor of this work is [12] , which adapts [9] to category search. First, we extend that system to handle larger databases (60,000 images), more complex semantic classes (art images, architecture, and history), and userterminated search. Second, we provide theoretical explanations for the main algorithms; in particular, we show that the Voronoi-based display algorithm minimizes the conditional entropy of an ideal user who responds to queries based on a "reference" image in his semantic class. We also provide a quantitative analysis of the enabling assumptions by measuring the degree of semantic unity within sets of images which are close in the system metric and the degree of coherence between the answer statistics of the user and the system. Finally, we provide a behavioral interpretation for the two parameters of the answer model which leads to a highly efficient and statistically rigorous model estimation scheme in the context of two psychovisual experiments.

O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Conversations with a virtual human: Synthetic emotions and human responses To test whether synthetic emotions expressed by a virtual human elicit positive or negative emotions in a human conversation partner and affect satisfaction towards the conversation, an experiment was conducted where the emotions of a virtual human were manipulated during both the listening and speaking phase of the dialogue. Twenty-four participants were recruited and were asked to have a real conversation with the virtual human on six different topics. For each topic the virtual human's emotions in the listening and speaking phase were different, including positive, neutral and negative emotions. The results support our hypotheses that (1) negative compared to positive synthetic emotions expressed by a virtual human can elicit a more negative emotional state in a human conversation partner, (2) synthetic emotions expressed in the speaking phase have more impact on a human conversation partner than emotions expressed in the listening phase, (3) humans with less speaking confidence also experience a conversation with a virtual human as less positive, and (4) random positive or negative emotions of a virtual human have a negative effect on the satisfaction with the conversation. These findings have practical implications for the treatment of social anxiety as they allow therapists to control the anxiety evoking stimuli, i.e. the expressed emotion of a virtual human in a virtual reality exposure environment of a simulated conversation. In addition, these findings may be useful to other virtual applications that include conversations with a virtual human. Humans are social creatures for which conversations with others are an essential part of their everyday life. These conversations allow them to influence each other's behaviour, attitudes and emotions. Conversations are part of complex social interactions, such as learning, negotiation, and coordination. Not surprisingly, people strive to become more comfortable and skilled in conducting conversations. With the introduction of virtual reality and virtual humans, people can experience satisfaction. Several researchers have also studied the impact of positive behaviour of a virtual human on actual humans. De Melo, Carnevale, and Gratch (2012) found that people disliked negotiating with angry virtual humans and tended to treat them as uncooperative and dominant. At the more positive side, Maldonado et al. (2005) found that positive emotions expressed by a co-learner enhanced student's learning gains and enjoyment, even if the co-learner simply existed of a set of photos of human facial expressions. Also Burleson and Picard (2007) showed that systems with a virtual character that provided affective support reduced frustration of less confident users. All these studies show that virtual humans that express emotions may also affect an individual.. Therapists may use this; for example, at an initial stage of an exposure therapy they may use virtual humans expressing positive emotions to limit the amount of anxiety they want to elicit in a patient.. Later on in the exposure they may let the virtual human express negative emotions to again elicit anxiety as the anxiety provoking element of having a conversation with a positive virtual human has worn off. Being able to do this would be beneficial for applications such as VRET. Evidence in the literature supports the idea that positive and negative emotions can be elicited in a conversation with a virtual human, but this evidence is basically indirect in the sense that the literature mainly focused on one-way conversations where a single virtual human or audience listened to a human (Ling, Brinkman, Nefs, Qu, & Heynderickx, 2012; Pertaub et al., 2002; Wong & McGee, 2012) or where a virtual human speak to a human (Baylor, Ryu, & Shen, 2003; Konstantinidis, Hitoglou-Antoniadou, Luneski, Bamidis, & Nikolaidou, 2009; Qiu & Benbasat, 2005) . Here, we systematically examine the effect of emotion expression of a virtual human on its conversational partner in a two-way free-speech dialog. In the context of social anxiety, negative or positive emotion expression refers to expressions of the virtual human from which human conversation partners could deduce that they are The analyses on the data for valence and discussion satisfaction suggest that positive compared to negative synthetic emotions expressed by a talking virtual human can elicit a more positive emotional state in a person, and can create more satisfaction towards the conversation. Therefore, we only found support for the first hypothesis in the speaking behaviour of the virtual human as no significant effect was found for the different emotions expressed by the listening virtual human. This dominance of the speaking phase over the listening phase was also hypothesised by the second hypothesis and confirmed by the data analyses since a larger effect on reported valence and discussion satisfaction was found for the synthetic emotions manipulated in the speaking phase compared to the listening phase of the virtual human. Besides the additional verbal channel to express emotions in the speaking phase, the participants might also have spent less attention to the virtual human when they were talking and the virtual human was listening. In human-human communication, the gaze of a listener is often fixed on the speaker, while the gaze of the speaker is only fixed on the listener when he or she begins or stops talking (D. Morris, 2002b) .

O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O B I O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Cache-Oblivious parallel SIMD Viterbi decoding for sequence search in HMMER Background: HMMER is a commonly used bioinformatics tool based on Hidden Markov Models (HMMs) to analyze and process biological sequences. One of its main homology engines is based on the Viterbi decoding algorithm, which was already highly parallelized and optimized using Farrar's striped processing pattern with Intel SSE2 instruction set extension. Results: A new SIMD vectorization of the Viterbi decoding algorithm is proposed, based on an SSE2 inter-task parallelization approach similar to the DNA alignment algorithm proposed by Rognes. Besides this alternative vectorization scheme, the proposed implementation also introduces a new partitioning of the Markov model that allows a significantly more efficient exploitation of the cache locality. Such optimization, together with an improved loading of the emission scores, allows the achievement of a constant processing throughput, regardless of the innermost-cache size and of the dimension of the considered model.The proposed optimized vectorization of the Viterbi decoding algorithm was extensively evaluated and compared with the HMMER3 decoder to process DNA and protein datasets, proving to be a rather competitive alternative implementation. Being always faster than the already highly optimized ViterbiFilter implementation of HMMER3, the proposed Cache-Oblivious Parallel SIMD Viterbi (COPS) implementation provides a constant throughput and offers a processing speedup as high as two times faster, depending on the model's size. The main insight of the presented approach is based on the observation that current parallel HMM implementations may suffer severe cache penalties when processing long models. To circumvent this limitation, a new vectorization of the Viterbi decoding algorithm is proposed to process arbitrarily sized HMM models.. The presented algorithm is based on a SSE2 intersequence parallelization approach, similar to the DNA alignment algorithm proposed by Rognes [4] . Besides the adopted alternative vectorization approach, the proposed algorithm introduces a new partitioning of the Markov model that allows a significantly more efficient exploitation of the cache locality. Such optimization, together with an improved loading of the emission scores, allows the achievement of a constant processing throughput, regardless of the innermostcache size and of the dimension of the considered model. In what concerns the partitioning, the proposed implementation was based on the observation that the poor cache performance of HMMER is related to the size of the model and to the fact that it is necessary to update all the states in the model for every letter of a query sequence. As a result, large models will force recently computed values out of cache. When this phenomena occurs for every letter in a query, it naturally results in a significant bottleneck.. We speculate that a similar phenomena occurs for the striped pattern of Farrar, in which case our partitioning technique could prove useful. Still, Farrar's algorithm processes one single query at a time, instead of 8. Therefore, the slowdown should only occurs for models 8 times larger, i.e., models of size larger than 10800.. According to the extensive set of assessments and evaluations that were conducted, the proposed vectorized optimization of the Viterbi decoding algorithm proved to be a rather competitive alternative implementation, when compared with the state of the art HMMER3 decoder. Being always faster than the already highly optimized HMMER ViterbiFilter implementation, the proposed implementation provides a constant throughput and proved to offer a processing speedup as high as 2, depending on the considered HMM model size and L1D cache size. Future work may also extend this approach to Intel's recent instruction-set extension AVX2, allowing the processing of twice more vector elements at a time. To conduct a comparative and comprehensive evaluation of the proposed approach, the COPS algorithm was ran against the ViterbiFilter implementation of HMMER 3.1b1, based on Farrar's striped vectorization. For such purpose, a benchmark dataset comprehending both DNA and protein data was adopted, covering a wide spectrum of model lengths, ranging from 50 to 3000 model states, with a step of about 100.. In particular, the DNA data consisted on HMMs sampled from Dfam 1.2 database of Human DNA HMMs [11] , and the human genome retrieved from the NCBI archive.. As The protein data consisted on a mix of 13 small and medium-sized HMMs from Pfam 27.0 [12] and 17 large HMMs created with hmmerbuild tool from Protein Isoforms sampled from Uniprot, and the NRDB90 [13] non-redundant protein database. The short protein models, from Pfam, were the following: M0063-ACT_5, M0400-Alginate_exp, M0800-Patched, M1201-DUF3584, M0101-Bactofilin, M0500-Lant_dehyd_C, M0900-PolC_ DP2, M1301-Orbi_VP1, M0201-Adeno_52K, M0600-Mp p10, M1002-SrfB, M0300-Aldose_epim, M0700-Pox_ VERT_large, M1098-CobN-Mg_che.. The longer models used were generated from the following Uniprot Isoforms: M1400-Q8CGB6, M1800-Q9BYP7, M2203-P27732, M2602-O75369, M1500-Q9V4C8, M1901-Q64487, M2295-Q3UHQ6, M2703-Q8 BTI8, M1600-Q6NZJ6, M2000-Q9NY46, M2403-Q9U GM3, M2802-Q9DER5, M1700-Q3UH06, M2099-Q8NF 50, M2505-O00555, M2898-Q868Z9, M3003-A2AWL7.. The benchmarks were run on two different machines:. • Intel Core i7 3770 K, with an Ivy Bridge architecture, running at 3.50 GHz with a 32 KB L1D cache; • AMD Opteron 6276, with a Bulldozer architecture, running at 2.3 GHz with a 16 KB L1D cache.. All the timings were measured as total walltime, by using the Linux ftime function.

O O O B O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Surjunctivity for cellular automata in Besicovitch spaces The Besicovitch pseudodistance measures the relative size of the set of points where two functions take different values; the quotient space modulo the induced equivalence relation is endowed with a natural metric. We study the behavior of cellular automata in the new topology and show that, under suitable additional hypotheses, they retain certain properties possessed in the usual product topology; in particular, injectivity implies surjectivity. Cellular automata (CA) are transformations of the space C of configurations on a grid that are induced by a finitary rule applied uniformly to each point of the grid. Such mappings are characterized by translational invariance and continuity in the product topology; however, since such topology makes the shift a chaotic map, no translation invariant distance on C can induce it.. To overcome this problem in the unidimensional case, Cattaneo, Formenti, Margara and Mazoyer [2] define a pseudodistance on the space C = {0, 1}. Z by taking the sets of the form U n = [−n, . . . , n], and computing, for each n, the upper limit d B of the densities of the sets of points x ∈ U n where two configurations take distinct values. The quotient space where c 1 and c 2 are identified iff d B (c 1 , c 2 ) = 0 has topological properties similar to those possessed by the space of difference equations-which, as pointed out by Toffoli [8] , are a field of application for CA. Additionally, CA induce transformations on the resulting quotient space which can in turn provide information on several properties of the original CA.. In this paper, we apply the ideas from [2] in the much broader context of finitely generated groups, where CA can still be defined [3, 4, 6] ; we do this by linking pseudodistances to increasing sequences of finite sets which ultimately cover the whole group. This is not just for the sake of generality: we are also trying to shed light on any links between the properties of Z and {U n }, and those of d B . We then address a question asked in [1] : is there a connection between surjectivity of CA and surjectivity of induced map? Finally, we ask whether surjunctivity, i.e., being either surjective or noninjective, is a property of the induced map as well as of the CA.. A summary of answers to these questions is given in having at most length n; let 2 ≤ |Q| < ∞, and let X be the quotient of Q G with respect to the equivalence relation. endowed with the topology induced by the distance. Let A be a cellular automaton over G having set of states Q. Let f, g :. and g(n) f (n). Observe that, if either f or g is a polynomial, the choice β = 1 is always allowed. We indicate the identity of the group G as 1 G . Product and inverse are extended to subsets of G elementwise. If E ⊆ G is finite and nonempty, the closure and boundary of X ⊆ G w.r.t. E are the sets X +E = {g ∈ G : gE ∩ X = ∅} = XE −1 and ∂ E X = X +E \ X, respectively; in general, X ⊆ X +E unless 1 G ∈ E. S ⊆ G is a set of generators if the graph (G, E S ), where E S = {(x, xz) : x ∈ G, z ∈ S∪S −1 }, is connected; a group is finitely generated (briefly, f.g.) if it has a finite set of generators (briefly, f.s.o.g.). The distance between g and h w.r.t. S is their distance in the graph (G, E S ); the length of g ∈ G w.r.t. S is its distance from 1 G . The disk of center g and radius r w.r.t. S will be indicated by D r,S (g); we will omit g if equal to 1 G , and S if irrelevant or clear from the context. Observe that D r (g) = gD r , and that (D n,S ) +DR,S = D n+R,S . For the rest of the paper, we will only consider f.g. infinite groups.. The growth function of G w.r.t. S is γ S (n) = |D n,S |. It is well known [5] . for every n ∈ N and n∈N X n = G. {D n (g)} is an exhaustive sequence. For U ⊆ G, the lower and upper density of U w.r.t. the exhaustive sequence {X n } are, respectively, the lower limit dens inf {Xn} U and the upper limit dens sup {Xn} U of the quantity |U ∩X n |/|X n |. An exhaustive sequence is amenable or a Følner sequence [5, 6, 7] if. for every finite E ⊆ G; a group is amenable if it has an amenable sequence.. Observe that {X n } is amenable iff it satisfies (1) for all the E's in a single exhaustive sequence, or for E = S f.s.o.g. If G is of subexponential growth, then {D n } contains an amenable subsequence, and is itself amenable if G is of polynomial growth (cf. [5] ). If 2 ≤ |Q| < ∞ and G is a f.g. group, the space C = Q G of configurations of G over Q is homeomorphic to the Cantor set. If E ⊆ G is finite, a pattern over Q with support E is a map p ∈ Q E . For c ∈ C, g ∈ G, c g ∈ C is defined by c g (h) = c(gh) for all h ∈ G; transformations of C of the form c → c g for a fixed g ∈ G are called translations. For G = Z and g = +1, the translation. A cellular automaton (briefly, CA) over G is a triple A = Q, N , f , where the set of states Q is finite and has at least two elements, the neighborhood index N ⊆ G is finite and nonempty, and the local evolution function f maps. is the global evolution function of A. Observe that F A is continuous in the product topology and commutes with translations. A is injective, surjective, and so on, if F A is. Two patterns p 1 , p 2 ∈ Q E are mutually erasable (briefly, m.e.) for A if. A is preinjective if does not have two m.e. patterns. A pattern p is a Garden of Eden (briefly, GoE) for A if it does not occur in F A (c) for any c ∈ C. From the compactness of C follows that a CA has a GoE pattern iff it is nonsurjective. By Moore-Myhill's theorem for amenable groups [4] , a CA over an amenable group is surjective iff it is preinjective; in particular, it is surjunctive, i.e., either surjective or noninjective.. A pseudodistance on a set X is a map d : X × X → [0, +∞) satisfying all of the axioms for a distance, except d(x, y) > 0 for every x = y. If d is a pseudodistance on X, then x 1 ∼ x 2 iff d(x 1 , x 2 ) = 0 is an equivalence relation, and d(κ 1 , κ 2 ) = d(x 1 , x 2 ) with x i ∈ κ i is a distance on X/ ∼.. 3 Besicovitch distances and cellular automata. is a pseudodistance on C, and is a distance if and only if G is finite. By an abuse of language, we will also indicate as C B,{Xn} the metric space (C B,{Xn} , d B,{Xn} ). If X n = D n,S for some f.s.o.g. S, we write d B,S instead of d B,{Dn,S } , and speak of Besicovitch distance w.r.t. S; similar nomenclature and notation shall be used in analogous cases. Observe that, if G is infinite, and c k (g) = c(g) if and only if g ∈ X k , then c k → c in the product topology, but d B,{Xn} (c k , c) = 1 for all k ∈ N, so that d B,{Xn} is not continuous in the product topology. Definition 3.3 is an extension of the one given in [2] for the case Q = {0, 1}, G = Z, S = {+1}. In general, the topology of C B,{Xn} is very different from that of C: for example, in the aforementioned case, C B,S is arcwise connected, not locally compact, and infinite-dimensional, while C is totally disconnected, compact, and zero-dimensional. Also, the equivalence classes of ∼ B,{Xn} usually depend on {X n }; for example, if Q = {0, 1}, G = Z, X n = {−n, . . . , n}, X Proof. Let S be a f.s.o.g. for G such that d B,S (c 1 , c 2 ) = 0. Let S ′ be another f.s.o.g. for G: there exist k, n 0 ∈ N and α 1 , α 2 > 0 such that γ S (n) ≤ α 1 n k and. k /α 2 for all n > n 0 . Thus, for all n large enough,. and the rightmost term vanishes for n → ∞. It is proved in [2] that the Besicovitch distance d B,{+1} on {0, 1} Z is invariant by translations; this is not true in the general case. As a counterexample, let S = {a, b} and let G be the free group over S; identify elements of G with reduced words over S ∪ S −1 . Let c 1 (g) = 0 for all g ∈ G, and c 2 (g) = 1 if and only if g begins with a: then c Proof. Let S be a f.s.o.g. for G; it is sufficient to prove that d B,{Xn} (c. n | for all g ∈ E, n ∈ N, so that from the amenability of {X {Xn} (c 1 , c 2 ) . This is true for all c 1 , c 2 ∈ C, g ∈ E, so that, by replacing c i with c g i and g with g −1 , we get the reverse inequality.

B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Automated Brain Structure Segmentation Based on Atlas Registration and Appearance Models Abstract-Accurate automated brain structure segmentation methods facilitate the analysis of large-scale neuroimaging studies. This work describes a novel method for brain structure segmentation in magnetic resonance images that combines information about a structure's location and appearance. The spatial model is implemented by registering multiple atlas images to the target image and creating a spatial probability map. The structure's appearance is modeled by a classifier based on Gaussian scale-space features. These components are combined with a regularization term in a Bayesian framework that is globally optimized using graph cuts. The incorporation of the appearance model enables the method to segment structures with complex intensity distributions and increases its robustness against errors in the spatial model. The method is tested in cross-validation experiments on two datasets acquired with different magnetic resonance sequences, in which the hippocampus and cerebellum were segmented by an expert. Furthermore, the method is compared to two other segmentation techniques that were applied to the same data. Results show that the atlas-and appearance-based method produces accurate results with mean Dice similarity indices of 0.95 for the cerebellum, and 0.87 for the hippocampus. This was comparable to or better than the other methods, whereas the proposed technique is more widely applicable and robust. S EVERAL neurological disorders like Parkinson's disease, schizophrenia, and Alzheimer's disease are believed to cause changes in the shape and size of brain structures. Due to its excellent soft-tissue contrast, magnetic resonance imaging (MRI) has become an important tool for studying the etiology and consequences of these diseases [1] - [3] . Extracting size or shape information from MR images requires accurate and reliable delineation of brain structures. Although manual segmentation is still the gold standard for the analysis of image data, automated segmentation can help decrease the workload and increase reproducibility.. Many methods for automated brain structure segmentation have been introduced in the past decade. Among these, atlasbased segmentation techniques have arguably been the most successful. In this approach a manually labeled atlas image is registered to an unlabeled target image. The resulting deformation field is then used to warp the atlas labels to the target's coordinate system. Especially when registration errors are compensated for by adding a statistical intensity model, this approach can produce very accurate and robust results [4] - [9] .. Many of these atlas-and intensity-based methods use a global intensity model to determine whether a voxel belongs to the foreground or background class [5] - [9] . However, most brain structures have one or more neighbors with similar intensity characteristics, which results in partially overlapping foreground and background distributions. Background voxels with foreground intensities (or vice versa) will, therefore, be mislabeled by the intensity model, unless the atlas information is very strong. As a result atlas-and intensity-based methods are unsuited to segment structures with complex, spatially varying intensity patterns like the cerebellum (Fig. 1) . A large number of background voxels at the interface with the cerebrum and the brainstem will be considered as foreground. This limits the applicability of atlas-and intensity-based techniques. Structures with relatively simple intensity patterns like the hippocampus show limited overlap, and can, therefore, usually be accurately segmented. But when applied to the hippocampus, atlas-and intensity-based methods remain vulnerable to registration errors that push the atlas into background areas like the enthorinal cortex, amygdala, or parahippocampal gyrus (Fig. 1) .. One way to overcome these limitations is by combining atlas registration with a local intensity model that can describe spatially varying intensity distributions [4] , [10] . More recent work has shown the potential of modeling local image appearance of brain structures with high-dimensional feature vectors of Haar filters or Gaussian derivatives at different scales [11] - [16] .. 0278-0062/$26.00 © 2011 IEEE Fig. 1 . Manually segmented T1-weighted images of a cerebellum and a hippocampus (both in sagittal view) together with the intensity distributions sampled from the structure (solid red curves) and background (dashed black curves). The cerebellum distribution overlaps with the intensities of the cerebrum (1) and brainstem (2) . In the hippocampus image background voxels from the amygdala (3) and parahippocampal gyrus (4) In this paper, we present a segmentation method that combines atlas-based segmentation with multi-feature appearance models. The proposed method uses multiple atlas registrations to construct a spatial probability map that models the location of the brain structure in an unlabeled MR image. The appearance of the structure is described by a voxel classifier based on Gaussian scale-space features. The smoothness of the result is controlled with a regularization term. The spatial, appearance, and regularization terms are then combined in a posterior probability function that can be globally maximized using graph cuts [17] , [18] .. This work is closest related to the hybrid discriminative/generative segmentation method presented in [14] , which describes a combination of an appearance model and a statistical shape model. The appearance model is implemented with a boosting technique that uses approximately 5000 image and location features to describe local image information derived from a manually labeled training set. The shape model encourages segmentations that have a similar shape as the training images. In contrast, in our method this role is performed by the spatial probability map, which benefits from the accuracy and robustness of multiple atlas registrations.. The method is evaluated by segmenting the cerebellum and hippocampus in two MRI datasets that were acquired with different scanners and sequences. We determined its accuracy by comparing the results to manual segmentations. The performance of the method was also compared with that of two other techniques based on atlas registration [19] and atlas registration plus an intensity model [7] . The results of I-CRBL showed some cases of over-segmentation at the posterior border with the skull, caused by registration errors in this area. The proposed method is unable to compensate for these errors as high-intensity voxels in the fatty parts of the skull were considered to be foreground by the appearance model. An extreme example can be seen in Fig. 8 .. When segmenting the I-CRBL set with a smaller nine-subject training set we obtained DSI and RV scores of and . The results obtained with the 17-subject training set listed in Table II were and . These differences in DSI and RV were statistically significant at -values of and 0.002, albeit very small. For I-HC a smaller training set resulted in a DSI of versus and an RV of versus . The difference in DSI was statistically significant at , but the difference in RV was not . Fig. 9 shows the average DSI as a function of two model parameters while keeping the other three at the optimal values. For I-CRBL these optimal parameter settings were , , , , . Most parameter settings around these values produced a score of 0.96 or higher, which is close to the accuracy of 0.962 obtained in the leave-one-out experiments. Since the parameter learning for the cerebellum was performed using DSI scores computed over the entire structure, this is a little higher than the score listed in Table I , which was computed over the left and right sides separately. The average maximum DSI for the I-CRBL set was 0.965.. The optimal settings for the I-HC set were , , , , . Although the results are not as stable as for I-CRBL, there are several settings that give a DSI of at least 0.87, which was the score obtained in Table I . For the I-HC set the average maximum SI was 0.878.

O O O B O O B I O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Encoding atlases by randomized classification forests for efficient multi-atlas label propagation We propose a method for multi-atlas label propagation (MALP) based on encoding the individual atlases by randomized classification forests. Most current approaches perform a non-linear registration between all atlases and the target image, followed by a sophisticated fusion scheme. While these approaches can achieve high accuracy, in general they do so at high computational cost. This might negatively affect the scalability to large databases and experimentation. To tackle this issue, we propose to use a small and deep classification forest to encode each atlas individually in reference to an aligned probabilistic atlas, resulting in an Atlas Forest (AF). Our classifier-based encoding differs from current MALP approaches, which represent each point in the atlas either directly as a single image/label value pair, or by a set of corresponding patches. At test time, each AF produces one probabilistic label estimate, and their fusion is done by averaging. Our scheme performs only one registration per target image, achieves good results with a simple fusion scheme, and allows for efficient experimentation. In contrast to standard forest schemes, in which each tree would be trained on all atlases, our approach retains the advantages of the standard MALP framework. The target-specific selection of atlases remains possible, and incorporation of new scans is straightforward without retraining. The evaluation on four different databases shows accuracy within the range of the state of the art at a significantly lower running time. Labeling of healthy human brain anatomy is a crucial prerequisite for many clinical and research applications. Due to the involved effort (a fully manual labeling of a single brain takes 2-3 days [Klein and Tourville, 2012] ), and increasing database sizes (e.g. ADNI, IXI, OASIS), a lot of research has been devoted to develop automatic methods for this task. While brain labeling is a general segmentation task (with a high number of labels), the standard approach for this task is multi-atlas label propagation (MALP) -see [Landman and Warfield, 2012] for an overview of the state of the art. With the atlas denoting a single labeled scan, MALP methods first derive a set of label proposals for the target image, each based on a single atlas, and then combine these proposals into a final estimate.. Currently, there are two main strategies for estimating atlas-specific label proposals. The first and larger group of methods non-linearly aligns each of the atlas images to the target image, and then -assuming one-to-one correspondence at each point -uses the atlas labels directly as label proposals, cf. e.g. [Rohlfing et al., 2004; Warfield et al., 2004; Heckemann et al., 2006] . The second group of patch-based methods has recently enjoyed increased attention [Coupé et al., 2011; Rousseau et al., 2011; Wu et al., 2012] . Here, the label proposal is estimated for each point in the target image by a local similarity-based search in the atlas. Patch-based approaches relax the one-to-one assumption, and aim at reducing the computational times by using linear instead of deformable alignment [Coupé et al., 2011; Rousseau et al., 2011] , resulting in labeling running times of 22-130 minutes per target on the IBSR dataset [Rousseau et al., 2011] . The fusion step, which combines the atlas-specific label proposals into a final estimate, aims to correct for inaccurate registration or labellings. While label fusion is a very active research topic, it is not the focus of this work. Additionally, some approaches perform further refinement, e.g. by learning classifiers for fine-scale class-based correction .. While current state of art techniques can achieve high levels of accuracy, in general they are computationally demanding. This is primarily due to the non-linear registration between all atlases and the target image, combined with the long running times for the best performing registration schemes for the problem [Klein et al., 2009] . Current methods state running times of 2-20 hours per single registration [Landman and Warfield, 2012] . Furthermore, sophisticated fusion schemes can also be computationally expensive. State of the art approaches report fusion running times of 3-5 hours Asman and Landman, 2012a,b] .. While the major drawback of high computational costs is the scalability to large and growing databases, they also A single atlas is encoded by training a corresponding atlas forest on the samples from that atlas only. The labeling of a new target is performed by the testing step on the trained atlas forests, and the following fusion of the probabilistic estimates by averaging. For the entire method, the intensity images are augmented by label priors as further channels, obtained by registering a probabilistic atlas.. limit the amount of possible experimentation during the algorithm development phase.. Our method differs from previous MALP approaches in the way how label proposals for a single atlas are generated, and is designed with the goal of low computational cost at test time and experimentation. In this work, we focus on the question of how a single atlas is encoded. From this point of view, methods assuming one-to-one correspondence represent an atlas directly as an image/labelmap pair, while patch-based methods encode it by a set of localized patch collections. Variations of the patch-based encoding include use of sparsity [Wu et al., 2012] , or use of label-specific kNN search structures [Wang et al., 2013] .. In contrast to previous representations, we encode a single atlas together with its relation to label priors by a small and deep classification forest -which we call an Atlas Forest (AF). Given a target image as input (and an aligned probabilistic atlas), each AF returns a probabilistic label estimate for the target. Label fusion is then performed by averaging the probability estimates obtained from different AFs. Please see Figure 1 for an overview of our method. While patch-based methods use a static representation for each image point (i.e. a patch of fixed size), our encoding is spatially varying. In the training step, our approach learns to describe different image points by differently shaped features, depending on the point's contextual appearance.. Compared to current MALP methods, our approach has the following important characteristics:. 1. Only one registration per target is required. This registration aligns the probabilistic atlas to the target. Since only one registration per target is required, the running time is independent of the database size in this respect. This differs conceptually from patchbased approaches, where the efficiency does not come from reducing the number of registrations, but from using affine instead of non-linear transformations.. 2. Efficient generation of atlas proposals and their fusion. For proposal generation one AF per atlas is evaluated. Due to the inherent efficiency of treebased classifiers at test time, this is significantly more efficient than current approaches.. 3. Efficient Experimentation. A leave-one-out crossvalidation of a standard MALP approach on n atlases requires registration between all images, thus scaling with n 2 . In contrast, the training of the single AFs, which is the most costly component of our approach for experimentation, scales with n (this assumes that generating the probabilistic atlas is not part of experimentation).. Besides being efficient, experiments on 4 databases in Section 3 indicate that our scheme also achieves accuracy within the range of the state of the art.. Being based on discriminative classifiers, our approach is also related to a number of works which employ machine learning techniques. Compared to the use of multi-atlas label propagation techniques discussed above, the use of machine learning for brain labeling is still relatively limited. In [Tu et al., 2008 ], a hybrid model is proposed, which combines a discriminative probabilistic-boosting tree (PBT) classifier [Tu, 2005] with a PCA-based generative shape model of the individual anatomical structures. In [Tu and Bai, 2010] , the Auto-Context framework with the PBT classifier was applied to brain labeling, and shown to outperform [Tu et al., 2008] . Recently, the use of classifiers to correct systematic mistakes of labeling methods in a post-processing step has been shown to improve accuracy [Wang et al., 2011 .. The major difference of these works to our approach is that they use the common scheme in which all available atlases are used for the training of one classifier. This is also true of standard forest schemes (cf. e.g. Iglesias et al., 2011a; Montillo et al., 2011; Zikic et al., 2012] ) which train each tree on data from all training images.. In contrast, the main idea of this paper is to use one classifier to encode a single atlas by training it only on this exemplar. This approach has three advantageous properties for the multi-atlas label propagation setting. Dice mean Dice σ Rousseau (GW-MP) 83. Table 1 : Average mean and standard deviation of Dice score for the variations discussed in Section 3.1. The results of the proposed method with the chosen settings are repeated with a highlighted name for easier comparison. A visual representation of the results is given in Figure 5 .. Next, with fixed s min = 8, we test the influence of the number of trees per atlas forest (Figure 5d ). The performance is stable for different values of this parameter, and we see no large differences between using 1 and 5 trees per atlas forest. This effect is probably due to the use of the deterministic features. We choose to use T = 5 as a conservative setting for subsequent experiments.

O O O O O O O O B I I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Delay tolerant (delto) distributed TAST codes for cooperative wireless networks In a distributed cooperative communication system, as the distances between different relay nodes and the receiving nodes may be different, so the performances of distributed space time codes at receiving nodes may badly be degraded if timing synchronization is not assured. In this article, extending the work of Damen et al. we introduce the design of distributed threaded algebraic space-time (TAST) codes offering resistance to timing delay off-set. We present some new and useful techniques of constructing delay tolerant TAST code for distributed cooperative networks, which, like their brethren codes, are delay tolerant for any delay profile and achieve full diversity for arbitrary number of relays, transmit/receive antennas, and input alphabet size. Our proposed codes with minimum lengths achieve better performances than the existing codes retaining full rate and full diversity with or without use of guard bands. Simulations results confirm our claim of obtaining better performances. Wireless communication systems with multiple antennas have recently attracted considerable interests [1] [2] [3] . Perhaps, the reason is that the performance of a wireless system is often limited by fading and may significantly be improved by exploiting some sort of diversity, for example spatial diversity. But on other hand equipping the pocket size mobile handsets with additional radiofrequency (RF) hardware is not feasible. Therefore, thinking for alternate options, many researchers have proposed different solutions and proposals.. Sendonaris et al. [4] proposed the idea of cooperative diversity which enables the source/destination to use nearby nodes as virtual antennas. In other words, the nearby relay nodes may act as auxiliary receivers/transmitters for the original transmitter/receiver. But, again the main problem with cooperative terminals as indicated by Li and Xia [5, 6] is the asynchronous nature of transmission which forces the traditionally designed space-time codes to lose their diversity and coding gain when used over distributed cooperative networks.. In fact in an unsynchronized cooperative network, the data from different relays reach the destination after different delays. In [7] , it was shown that all the wellknown codes lose their diversity at receiver. Mei et al. [8] proposed a technique of using of guard bands between blocks of symbols. The proposed scheme in [8] could achieve full-diversity but the main drawback of this technique is its limitation in the number of relays (only two are allowed) and a huge rate loss due to the insertion of guard bands.. The proposed delay tolerant codes for asynchronous cooperative network of Li and Xia [5, 6] were further generalized and refined in [9] by including full-diversity delay tolerant space-time trellis codes of minimum constrained length. In [7] , delay tolerant distributed space-time block codes based on threaded algebraic space-time (TAST) codes [10] are designed for unsynchronized cooperative network. The distributed TAST codes of [7] preserve the rank of the space-time codewords under arbitrary delays at the receiver. In a similar way, the authors in [7] further extend their study in [11] by introducing delay tolerant codes with minimum lengths. A lattice-based maximum likelihood detector is used for decoding, which is computationally more complex than the decoupled decoding of orthogonal space-time block codes.. Following the framework of [7, 11] , in this article we introduce some new and useful techniques for designing delay tolerant TAST codes. The proposed codes achieve full diversity and are flexible with respect to signalling constellation, transmission rate, and number of transmit/receive antennas. Our proposed codes achieve better performances over the existing codes particularly at high SNR. For ease to write, hereafter, we will use delto code as short form of delay tolerant code.. The rest of the article is organized as follows. The background and system model are given in Section 2. Some construction techniques of delay tolerant spacetime codes are discussed in Section 3. Multiple threads packing are given in Section 4. Section 5 elaborates the construction methods for delto codes with minimum lengths. Some construction examples are presented in Section 6. Simulation results are given in Section 7 and the conclusion is given in Section 8. Within the same framework developed in [7, 11] , we introduced some easy and useful techniques for the construction of delay tolerant distributed STBC codes having full diversity and full rate. Like their brethren codes, our proposed codes are flexible with respect to constellation size, number of receive/transmit antennas. We introduced two useful techniques for constructing threads codewords matrices. The packing of different threads into a single codeword matrix provides different code structures to be used over cooperative networks with different setup of relays and antennas. In term of error rates, the codes with T >N t developed in (50) to (53) do not outperform the codes introduced by Damen but we hope that their simple structures may reduce the hardware complexity. The codes with T = N t developed in (54) and (68) outperform the existing codes without sacrificing decoding complexity and other nice characteristics. For example, the error performance of the code proposed in (68) is improved by about 2 dB at the BER of 10 -5 when N t = N r = 3, and 0.5 dB at 10 -3 when N t = 2 and N r = 2.. Endnotes a For N t > 4, plus sign is replaced by minus sign.. b A safe prime is a prime number of the form 2p + 1, where p is also a prime. For example, first seven safe prime numbers are [5, 7, 11, 23, 47, 59 ,83].. Author details

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Quality-of-service

in

cloud

computing:

modeling

techniques

and

their

applications

Recent

years

have

seen

the

massive

migration

of

enterprise

applications

to

the

cloud.

One

of

the

challenges

posed

by

cloud

applications

is

Quality-of-Service

(QoS)

management,

which

is

the

problem

of

allocating

resources

to

the

application

to

guarantee

a

service

level

along

dimensions

such

as

performance,

availability

and

reliability.

This

paper

aims

at

supporting

research

in

this

area

by

providing

a

survey

of

the

state

of

the

art

of

QoS

modeling

approaches

suitable

for

cloud

systems.

We

also

review

and

classify

their

early

application

to

some

decision-making

problems

arising

in

cloud

QoS

management.

Cloud

computing

has

grown

in

popularity

in

recent

years

thanks

to

technical

and

economical

benefits

of

the

ondemand

capacity

management

model

Many

cloud

operators

are

now

active

on

the

market,

providing

a

rich

offering,

including

Infrastructure-as-a-Service

(IaaS),

Platform-as-a-Service

(PaaS),

and

Software-as-a-Service

(SaaS)

solutions

The

cloud

technology

stack

has

also

become

mainstream

in

enterprise

data

centers,

where

private

and

hybrid

cloud

architectures

are

increasingly

adopted..

Even

though

the

cloud

has

greatly

simplified

the

capacity

provisioning

process,

it

poses

several

novel

challenges

in

the

area

of

Quality-of-Service

(QoS)

management.

QoS

denotes

the

levels

of

performance,

reliability,

and

availability

offered

by

an

application

and

by

the

platform

or

infrastructure

that

hosts

it

a

QoS

is

fundamental

for

cloud

users,

who

expect

providers

to

deliver

the

advertised

quality

characteristics,

and

for

cloud

providers,

who

need

to

find

the

right

tradeoffs

between

QoS

levels

and

operational

costs.

However,

finding

optimal

tradeoff

is

a

difficult

decision

problem,

often

exacerbated

by

the

presence

of

service

level

agreements

(SLAs)

specifying

QoS

targets

and

economical

penalties

associated

to

SLA

violations

While

QoS

properties

have

received

constant

attention

well

before

the

advent

of

cloud

computing,

performance

heterogeneity

and

resource

isolation

mechanisms

of

cloud

platforms

have

significantly

complicated

QoS

analysis,

prediction,

and

assurance.

This

is

prompting

several

researchers

to

investigate

automated

QoS

management

methods

that

can

leverage

the

high

programmability

of

hardware

and

software

resources

in

the

cloud

This

paper

aims

at

supporting

these

efforts

by

providing

a

survey

of

the

state

of

the

art

of

QoS

modeling

approaches

applicable

to

cloud

computing

and

by

describing

their

initial

application

to

cloud

resource

management..

Scope.

Cloud

computing

is

an

operation

model

that

integrates

many

technological

advancements

of

the

last

decade

such

as

virtualization,

web

services,

and

SLA

management

for

enterprise

applications.

Characterizing

cloud

systems

thus

requires

using

diverse

modeling

techniques

to

cope

with

such

technological

heterogeneity.

Yet,

the

QoS

modeling

literature

is

extensive,

making

it

difficult

to

have

a

comprehensive

view

of

the

available

techniques

and

their

current

applications

to

cloud

computing

problems..

Methodology.

The

aim

of

this

survey

is

to

provide

an

overview

of

early

research

works

in

the

cloud

QoS

modeling

space,

categorizing

contributions

according

to

relevant

areas

and

methods

used.

Our

methodology

attempts

to

maximize

coverage

of

works,

as

opposed

to

reviewing

specific

technical

challenges

or

introducing

readers

to

modeling

techniques.

In

particular,

we

focus

on

recent

modeling

works

published

from

onwards

focusing

on

QoS

in

cloud

systems.

We

also

discuss

some

techniques

originally

developed

for

modeling

and

dynamic

management

in

enterprise

data

centers

that

have

been

successively

applied

in

the

cloud

context.

Furthermore,

the

survey

considers

QoS

modeling

techniques

for

interactive

cloud

services,

such

as

multi-tier

applications.

Works

focusing

on

batch

applications,

such

as

those

based

on

the

MapReduce

paradigm,

are

therefore

not

surveyed..

Survey

Organization.

This

survey

covers

research

efforts

in

workload

modeling,

system

modeling,

and

their

applications

to

QoS

management

in

the

cloud..

Workload

modeling

involves

the

assessment

or

prediction

of

the

arrival

rates

of

requests

and

of

the

demand

for

resources

(e.g.,

CPU

requirements)

placed

by

applications

on

an

infrastructure

or

platform,

and

the

QoS

observed

in

response

to

such

workloads.

We

review

in

Section

cloud

measurement

studies

that

help

characterize

those

properties

for

specific

cloud

systems,

followed

by

a

review

of

workload

characterizations

and

inference

techniques

that

can

be

applied

to

QoS

analysis..

System

modeling

aims

at

evaluating

the

performance

of

a

cloud

system,

either

at

design

time

or

at

runtime.

Models

are

used

to

predict

the

value

of

specific

QoS

metrics

such

as

response

time,

reliability

or

availability.

We

survey

in

Section

formalisms

and

tools

employed

for

these

analyses

and

their

current

applications

to

assess

the

performance

of

cloud

systems..

Applications

of

QoS

models

often

appear

in

relation

to

decision-making

problems

in

system

management.

Techniques

to

determine

optimized

decisions

range

from

simple

heuristics

to

nonlinear

programming

and

meta-heuristics.

We

survey

in

Section

works

on

decision

making

for

capacity

allocation,

load

balancing,

and

admission

control

including

research

works

that

provide

solutions

for

the

management

of

a

cloud

infrastructure

(i.e.,

from

the

cloud

provider

perspective)

and

resource

management

techniques

for

the

infrastructure

user

(e.g.,

an

application

provider

aiming

at

minimizing

operational

expenditure,

while

providing

QoS

level

guarantees

to

the

end

users)..

Section

concludes

the

paper

and

summarizes

the

key

findings.

In

recent

years,

cloud

computing

has

matured

from

an

early-stage

solution

to

a

mainstream

operational

model

for

enterprise

applications.

However,

the

diversity

of

technologies

used

in

cloud

systems

makes

it

difficult

to

analyze

their

QoS

and,

from

the

provider

perspective,

to

offer

service-level

guarantees.

We

have

surveyed

current

approaches

in

workload

and

system

modeling

and

early

applications

to

cloud

QoS

management..

From

this

survey,

a

number

of

insights

arise

on

the

current

state

of

the

art:.

The

number

of

works

that

apply

white-box

system

modeling

techniques

is

quite

limited

in

QoS

management,

albeit

popular

in

the

software

performance

engineering

community.

This

effectively

creates

a

divide

between

the

knowledge

that

can

be

made

available

for

an

application

by

its

designers

and

the

techniques

used

to

manage

it.

A

research

question

is

whether

the

availability

of

more

detailed

information

about

application

internals

can

provide

significant

advantages

in

QoS

management.

Indeed,

a

trade-off

exists

between

available

information,

QoS

model

complexity,

computational

cost

of

decision-making,

and

accuracy

of

predictions.

This

trade-off

requires

further

investigation

by

the

research

community..

Gray-box

models

that

emphasize

resource

consumption

modeling

are

currently

prevalent

in

QoS

management

studies.

However,

description

of

performance

is

often

quite

basic

and

associated

with

mean

resource

requirements

of

the

applications.

However,

the

cloud

measurement

studies

in

Section

have

identified

performance

variability

as

a

major

issue

in

today's

offerings,

calling

for

more

comprehensive

models

that

can

describe

also

the

variability

in

CPU

requirements,

in

addition

to

mean

requirements.

Such

extension

has

been

explored

in

black-box

system

models

(e.g.,

QoS

in

web

services),

but

it

is

far

less

understood

in

white-box

and

gray-box

modeling..

Quite

surprisingly,

we

have

found

a

limited

amount

of

work

specific

to

workload

analysis

and

inference

techniques

in

the

cloud.

Most

of

the

techniques

used

for

traffic

forecasting,

resource

consumption

estimation,

and

anomaly

detection

have

received

little

or

no

validation

in

a

cloud

environment.

As

such,

it

remains

to

establish

the

robustness

of

current

techniques

to

noisy

measurements

typical

of

multi-tenant

cloud

environments..

Another

observation

arising

from

our

survey

is

that

the

literature

is

rich

in

works

focusing

on

IaaS

systems,

often

deployed

on

Amazon

at

present

the

market

leader

in

this

segment..

If

we

consider

the

resource

management

mechanisms

for

applications

QoS

enforcement

provided

by

public

clouds,

they

are

quite

simplistic

if

compared

to

current

research

proposals.

Indeed,

such

mechanisms

are

mainly

reactive

and

are

triggered

by

thresholds

violations

(related

to

response

times,

as

in

Google

App

Engine,

or

CPU

utilization

or

other

low

level

infrastructure

metrics,

as

in

Amazon

Vice

versa,

integrating

workload

characterisation,

system

models

and

resource

management

solutions,

pro-active

systems,

may

help

to

prevent

QoS

degradation.

The

development

of

research

prototypes

that

are

transferable

in

commercial

solutions

seems

to

remain

an

open

point..

Finally,

in

cloud

systems

an

important

role

is

played

by

resource

pricing

models.

There

is

a

growing

interest

towards

understanding

better

cloud

spot

markets,

where

bidding

strategies

are

developed

for

procuring

computing

resources.

Approaches

are

currently

being

proposed

to

automate

dynamic

pricing

and

cloud

resources

selection.

We

expect

that,

in

upcoming

years,

these

models

will

play

a

bigger

role

than

today

in

capacity

allocation

frameworks..

Summarizing,

this

survey

shows

that

the

literature

has

already

a

significant

number

of

works

in

cloud

QoS

management,

but

their

focus

leaves

open

several

research

opportunities

in

the

areas

discussed

above.

O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O