Circle-criterion Based Nonlinear Observer Design for Sensorless Induction Motor Control Abstract: This paper deals with the design of a nonlinear observer for sensorless induction motor control. Based upon the circle criterion approach, a nonlinear observer is designed to estimate pertinent but unmeasurable state variables of the considered induction machine for sensorless control purpose. The observer gain matrices are computed as a solution of linear matrix inequalities (LMI) that ensure the stability conditions of the state observer error dynamics in the sense of Lyapunov concepts. Measured and estimated state variables can be exploited to perform a state feedback control of the machine system. The simulation results are presented to illustrate the effectiveness of the proposed approach for nonlinear observer design. It is well known that induction motor is one of the most widely used machines in industrial applications. This is due to its high reliability, relatively low cost, and modest maintenance requirements. However, induction motor is also known as a complex nonlinear system, in which timevarying parameters entail additional difficulties for machine control, conditions monitoring and fault diagnostic purposes [1] . Furthermore, only a few state variables of the machine are available for on line measurement because of technical and/or economical constraints of the considered application.. In order to perform advanced control techniques, there is a great need of a reliable and accurate estimation of the unmeasurable key state variables of the machine. A state estimator, also called state observer, is a dynamic system that is driven by the inputs and outputs of the considered system, and estimates asymptotically its unmeasurable state variables. It is a "soft sensor" that plays an important role not only in sensorless control techniques but also in conditions monitoring, fault diagnosis, predictive maintenance and fault tolerant control techniques [1−3] . A control literature review shows that nonlinear state observer design approaches can be roughly divided into three classes [4−9] . The first class of approaches attempts to eliminate the system nonlinearities by a technique of linearization [3] . Its drawback is a set of extremely restrictive conditions that can hardly be met by a physical system. The second class of approaches attempts to dominate the system nonlinearities by using a high gain output correction term [4, 5] . Its drawbacks are the block triangular structure, the destabilizing effect of the peaking phenomenon, and the sensitivity against measurement noises. The third class of approaches to design nonlinear observers exploits the system nonlinearity. Lipschitz and sector properties are the main nonlinearity properties that are exploited [6, 10−12] . In order to benefit from some recent advances in nonlinear observer based control developed by the control community, in this paper we focus our attention on the application of one method of the third class, the so-called circlecriterion approach, to design a nonlinear observer for induction motor sensorless control. The circle criterion approach is a new line of research introduced for continuous-time systems by [10] . It permits to handle directly the system nonlinearities with less restriction than linearization and high gain approaches [10, 11] . A detailed proof of the main theorem is given in this paper.. The paper is organized as follows: In the second section, we present the ingredient of nonlinearity satisfying the sector properties and the design of the nonlinear observer. The considered nonlinear induction motor model is presented in the third section. In the fourth section, we present simulation results and comments. A conclusion ends the paper. A circle criterion based nonlinear observer design for induction motor sensorless control has been presented.. The main advantage of the circle criterion approach is that it permits to exploit directly the nonlinearities of the system without attempting to eliminate them. However, it introduces linear matrix inequalities as conditions for the convergence of the observer and the error of state estimation. Resolving the LMI determines the gain matrices of the nonlinear observer.. Simulation results show that the circle criterion based nonlinear observer design can effectively be performed to estimate unmeasurable state variables of the induction machine for possible sensorless control.

B O B I O O O B I O O O O O O O O O O B I O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O


Justification logics, logics of knowledge, and conservativity Abstract Several justification logics have been created, starting with the logic LP, (Artemov, Bull Symbolic Logic 7(1): 2001). These can be thought of as explicit versions of modal logics, or of logics of knowledge or belief, in which the unanalyzed necessity (knowledge, belief) operator has been replaced with a family of explicit justification terms. We begin by sketching the basics of justification logics and their relations with modal logics. Then we move to new material. Modal logics come in various strengths. For their corresponding justification logics, differing strength is reflected in different vocabularies. What we show here is that for justification logics corresponding to modal logics extending T, various familiar extensions are actually conservative with respect to each other. Our method of proof is very simple, and general enough to handle several justification logics not directly corresponding to distinct modal logics. Our methods do not, however, allow us to prove comparable results for justification logics corresponding to modal logics that do not extend T. That is, we are able to handle explicit logics of knowledge, but not explicit logics of belief. This remains open. This paper is partly an advertisement for justification logics, and partly a presentation of some technical results of interest concerning them. Justification logics are a relatively new area within the general field of logics of knowledge and modal logics, so some advertising is appropriate. They provide a natural finer-grained analysis than Hintikka-style logics do, yet relate to them in precise and useful ways. They are becoming appropriate tools for several purposes, and there is much current investigation of them. As to the technical results, we will show conservativity theorems relating the modal logics T, S4, and S5. (We actually show something more general, but this will do for now.) It might be asked what sense conservativity issues make for these logics since they have the same vocabulary. It will be seen that behind the modal operator , or K as it will generally be written here, there is hidden, explicit machinery, supplied by justification logics. When this is taken into account, conservativity results can be easily stated, and demonstrated. The main thing left undone is quite obvious: there is no analog of Theorem 8.1 for logics of belief instead of knowledge-of the form B(S) instead of K(S). The methods of proof used here clearly do not extend to explicit logics of belief. Many of the cases involved in the proof of Proposition 8.3 yield an instance of a Truth Axiom. Without the Truth Axioms, present methods cannot succeed. Nonetheless, either a belief analog of Theorem 8.1 holds, or it does not. A result either way would be of interest. The desirable conjecture is that it holds, and some partial (unpublished) results have been obtained by Artemov.. One other item was left unfinished, but it is of lesser importance. Theorem 9.2 needed the presence of both · and !. Producing a version not needing ! is straightforward, but a bit messy to state. A version without · probably has little intrinsic interest. Also, giving an appropriate version (if there is one) not requiring an injective constant specification is still a problem. But this too is left to others.

O O O O O O B O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Computational indistinguishability: a sample hierarchy We consider the existence of pairs of probability ensembles which may be efficiently distinguished from each other given Computational indistinguishability, introduced by Goldwasser and Micali [7] and defined in full generality by Yao [11] , is a central concept of complexity theory. Two probability ensembles, fX n g n2N and fY n g n2N , where both X n and Y n range over f0; 1g n , are said to be indistinguishable by a complexity class if for every machine M in the class the difference d M n def = jPrMX n = 1 , PrMY n = 1 j is a negligible function in n (i.e., decreases faster than 1=pn for any positive polynomial p).. We stress that, in the definition recalled above, the distinguishing machine (i.e., M) obtains a single sample (from E-mail: oded@wisdom.weizmann.ac.il. Currently visiting the Laboratory for Computer Science of MIT, and partially supported by DARPA grant DABT63-96-C-0018.. y E-mail: madhu@theory.lcs.mit.edu.. either distributions), and casts its "verdict" based on this sample. An important and natural question is what happens when the distinguishing machine is given several samples. It is well known that in several cases (see below), computational indistinguishability is preserved also when many samples are given to the distinguisher. That is, in these cases, if two ensembles are computationally indistinguishable by a single sample then they are also computationally indistinguishable by (polynomially) many samples. Two important cases where this happens are:. 1. When the two probability ensembles are polynomialtime computable, and one considers probabilistic polynomial-time distinguishers. (An ensemble fZ n g n2N is polynomial-time computable if there exists a probabilistic polynomial-time sampling algorithm, S, such that S1 n and Z n are identically distributed.). 2. When one considers computational indistinguishability with respect to the class of non-uniform polynomial-size circuits.. In both cases the proof amounts to using the multi-sample distinguisher to derive a single-sample distinguisher, by incorporating copies of the two ensembles into the single-sample distinguisher (cf., [5, 4] ). This is possible using the fact that the class of distinguishers is able to generate samples from each of the two ensembles.. However, it has been shown that the above may fail in certain other cases (cf., [3, 8, 6] ). Specifically, there exists a pair of (non-efficiently computable) ensembles which, on one hand, are computationally indistinguishable by (uniform) probabilistic polynomial-time algorithms which take a single sample, while on the other hand, can be distinguished in polynomial-time given two samples.. It has been unknown whether separations as above may exists between distinguishability based on, say, 2 samples and 3 samples. Furthermore, it was not known if there is a separation between 2 samples and polynomially many samples.. We show a separation between k samples and k+1, for any polynomially-bounded function k : N 7 ! N. That is, there exist a pair of probability ensembles which are (polynomialtime) indistinguishable based on k samples and yet can be distinguished (in polynomial-time) given k + 1 samples.

B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Generation of multimedia TV news contents for WWW In this paper, we present a system we have developed for automatic TV News video indexing that successfully combines results from the fields of speaker verification, acoustic analysis, very large vocabulary video OCR, content based sampling of video, information retrieval, dialogue systems, and ASF media delivery over IP. The prototype of TV news content processing Web was completed in July 2003. Since then, the system has been up running continuously. Up to the date when this message is written (March 27, 2006), the system records and analyzes the prime time evening news program in Taiwan every day of these years, except a few power failure shutdown. The TV news web is at

O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O


(t, k)-diagnosable system: a generalization of the PMC models Abstract-In this paper, we introduce a new model for diagnosable systems called ðt; kÞ-diagnosable system which guarantees that at least k faulty units (processors) in a system are detected provided that the number of faulty units does not exceed t. This system includes classical one-step diagnosable systems and sequentially diagnosable systems. We prove a necessary and sufficient condition for ðt; kÞ-diagnosable system, and discuss a lower bound for diagnosability. Finally, we deal with a relation between ðt; kÞ-diagnosability and diagnosability of classical basic models. We introduce a notion of ðt; kÞ-diagnosable systems and give a characterization of the system and discuss some relations to one-step and sequentially diagnosable systems. It is expected that there is a trade off relation between diagnosability t and a new parameter k. To solve the dependency of the diagnosability on the parameter k is a future subject. We think that the notion of ðt; kÞ-diagnosis may motivate us to design new diagnosis algorithms.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O


On the interpretation and identification of dynamic Takagi-Sugeno fuzzy models Abstract-Dynamic Takagi-Sugeno fuzzy models are not always easy to interpret, in particular when they are identified from experimental data. Ideally, it is desirable that a dynamic Takagi-Sugeno fuzzy model should give accurate global nonlinear prediction and at the same time that its local models are close approximations to the local linearizations of the nonlinear dynamic system. The latter is important in many applications where the constituent local models are used individually and aids validation and interpretation of the model considerably. This defines a multi-objective identification problem, namely, the construction of a dynamic model that is a good approximation of both local and global dynamics of the underlying system. While these objectives are often conflicting, it is shown that there exists a close relationship between dynamic Takagi-Sugeno fuzzy models and dynamic linearization when using affine local model structures, which suggests that a solution to the multi-objective identification problem exists. However, it is also shown that the affine local model structure is a highly sensitive parameterization when applied in transient operating regimes, i.e., far away from equilibrium. The reason is essentially that the constant term in the affine local model tends to dominate over the linear term during transients. In addition, it is inherently more difficult to design informative experiments in transient regions compared to near-equilibrium regions. Due to the multi-objective nature of the identification problem studied here, special considerations must be made during model structure selection, experiment design, and identification in order to meet both objectives. Some guidelines for experiment design are suggested and some robust nonlinear identification algorithms are studied. These include constrained and regularized identification and locally weighted identification. Their usefulness in the present context is illustrated by examples. We have illustrated fundamental interpretability and identifiability limitations of dynamic Takagi-Sugeno fuzzy models with constituent affine local models. The major problem is that in transient operating regimes (where the nonlinear system has no equilibrium points) the local affine model is poorly identifiable. Further, practical limitations exist that restrict the design of informative experiments for many systems. A theoretical understanding of the problems is developed by relating the dynamic Takagi-Sugeno fuzzy model with local affine models to dynamic linearization. Differences between local constants, linear and affine models are pointed out, the effect of minimizing the number of variables is the premise of the fuzzy rules are investigated and the role of the membership functions are studied. Finally, suggestions are presented, that potentially alleviate or reduce some of these problems through suitable experiment design and use of robust identification methods. In particular, it is shown how constrained and regularized identification methods may improve the interpretability of the constituent local models as local linearizations. Furthermore, locally weighted least squares identification is shown to explicitly address the tradeoff between local and global accuracy of the Takagi-Sugeno fuzzy model.. The practical importance of the problems is illustrated by very simple examples. Still, one may expect that the problems related to interpretability and identifiability will be much more pronounced when more complex higher order and multivariable examples are considered.. The present results are not only relevant to dynamic Takagi-Sugeno fuzzy models, but also related model representation with constituent local models, such as local model networks (e.g., [13] , [25] ) and piecewise linear models (e.g., [2]

O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O


Eliminating read barriers through procrastination and cleanliness Managed languages typically use read barriers to interpret forwarding pointers introduced to keep track of copied objects. For example, in a multicore environment with thread-local heaps and a global, shared heap, an object initially allocated on a local heap may be copied to a shared heap if it becomes the source of a store operation whose target location resides on the shared heap. As part of the copy operation, a forwarding pointer may be established in the original object to point to the copied object. This level of indirection avoids the need to update all of the references to the object that has been copied.In this paper, we consider the design of a managed runtime that eliminates read barriers. Our design is premised on the availability of a sufficient degree of concurrency to stall operations that would otherwise necessitate the copy. Stalled actions are deferred until the next local collection, avoiding exposing forwarding pointers to the mutator. In certain important cases, procrastination is unnecessary -lightweight runtime techniques can sometimes be used to allow objects to be eagerly copied when their set of incoming references is known, or when it can be determined that having multiple copies would not violate program semantics.We evaluate our techniques on 3 platforms: a 16-core AMD64 machine, a 48-core Intel SCC, and an 864-core Azul Vega 3. Experimental results over a range of parallel benchmarks indicate that our approach leads to notable performance gains (20 -32% on average) without incurring any additional complexity. Splitting a program heap among a set of cores is a useful technique to exploit available parallelism on scalable multicore platforms: each core can allocate, collect, and access data locally, moving objects to a global, shared heap only when they are accessed by threads executing on different cores. This design allows local heaps to be collected independently, with coordination required only for global heap collection. In contrast, stop-the-world collectors need a global synchronization for every collection. In order to ensure that cores cannot directly or indirectly access objects on other local heaps, which would complicate the ability to perform independent local heap collection, the following invariants need to be preserved:. • No pointers are allowed from one core's local heap to another.. • No pointers are permitted from the shared heap to the local heap.. Both invariants are necessary to perform independent local collections. The reason for the first is obvious. The second invariant prohibits a local heap from transitively accessing another local heap object via the shared heap. In order to preserve these invariants, the mutator typically executes a write barrier on every store operation. The write barrier ensures that before assigning a local object reference (source) to a shared heap object (target), the local object along with its transitive closure is lifted to the shared heap. We call such writes exporting writes as they export information out of local heaps. The execution of the write barrier creates forwarding pointers in the original location of the lifted objects in the local heap. These point to the new locations of the lifted objects in the shared heap. Since objects can be lifted to the shared heap on potentially any write, the mutator needs to execute a read barrier on potentially every read. The read barrier checks whether the object being read is the actual object or a forwarding pointer, and in the latter case, indirects to the object found on the shared heap. Forwarding pointers are eventually eliminated during local collection. Because the number of reads are likely to far outweigh the number of writes, the aggregate cost of read barriers can be both substantial and vary dramatically based on underlying architecture characteristics [6] . Eliminating read barriers, however, is nontrivial. Abstractly, one can avoid read barriers by eagerly fixing all references that point to forwarded objects at the time the object is lifted to the shared heap, ensuring the mutator will never encounter a forwarded object. Unfortunately, this requires being able to enumerate all the references that point to the lifted object; in general, gathering this information is very expensive as the references to an object might originate from any object in the local heap.. In this paper, we consider an alternative design that completely eliminates the need for read barriers without requiring a full scan of the local heap whenever an object is lifted to the shared heap. The design is based on two observations. First, read barriers can be clearly eliminated if forwarding pointers are never introduced. One way to avoid introducing forwarding pointers is to delay operations that create them until a local garbage collection is triggered. In other words, rather than executing a store operation that would trigger lifting a thread local object to the shared heap, we can simply procrastinate, thereby stalling the thread that needs to perform the store. The garbage collector must simply be informed of the need to lift the object's closure during its next local collection. After collection is complete, the store can take place with the source object lifted, and all extant heap references properly adjusted. As long as there is sufficient concurrency to utilize existing computational resources, in the form of available runnable threads to run other computations, the cost of procrastination is just proportional to the cost of a context switch.. Second, it is not necessary to always stall an operation that involves lifting an object to the shared heap. We consider a new property for objects (and their transitive closures) called cleanliness. A clean object is one that can be safely lifted to the shared heap without introducing forwarding pointers that might be subsequently encountered by the mutator: objects that are immutable, whose elements are only referenced from the stack, or whose set of incoming heap references is known, are obvious examples. The runtime analysis for cleanliness is combined with a specialized write barrier to amortize its cost. Thus, procrastination provides a general technique to eliminate read barriers, while cleanliness serves as an important optimization that avoids stalling threads unnecessarily.. The effectiveness of our approach depends on a programming model in which (a) most objects are clean, (b) the transitive closure of the object being lifted rarely has pointers to it from other heap allocated objects, and (c) there is a sufficient degree of concurrency in the form of runnable threads; this avoids idling available cores whenever a thread is stalled performing an exporting write that involves an unclean object. In this paper, we consider an implementation of these ideas in the context of MultiMLton [17] , a scalable, whole-program optimizing compiler and multicore-aware runtime system for Standard ML [15] , a mostly functional language whose concurrent programs typically enjoy these properties. Our technique does not rely on programmer annotations, static analysis or compiler optimizations to eliminate read barriers, and can be completely implemented as a lightweight runtime technique.. This paper provides the following contributions:. • A garbage collector design that has been tuned for mostly functional languages in which there is typically a surfeit of concurrency (in the form of programmer-specified lightweight threads) available on each core, to realize a memory management system that does not require read barriers.. • A new object property called cleanliness that enables a certain (albeit broad) class of objects to be safely lifted to the shared heap without requiring a full traversal of the local heap to fix existing references to them, reducing the frequency of thread stalls as a result of procrastination.. • An extensive evaluation of the collector performance on three multicore platforms; a 16 core AMD Operton server, Intel's 48 core Single-chip Cloud Computer (SCC), and Azul System's 864 core Vega 3 processor. The results reveal that eliminating read barriers on these platforms can lead to significant performance improvements.. The paper is organized as follows. In the next section, we present additional motivation that quantifies the cost and benefit of read barriers in our system. The overall design and implementation of the collector is provided in Section 3. Section 4 describes our treatment of cleanliness. The modifications to our write barrier to support cleanliness analysis and delayed writes are presented in Section 5. Details about the target platforms we use in our experiments is given in Section 6. Experimental results are presented in Section 7. A comparison to related work is given in Section 8, and Section 9 presents conclusions. The use of read barriers can impose non-trivial overheads in managed languages. In this paper, we consider a design of a runtime system for a thread-aware implementation of Standard ML that completely eliminates the need for read barriers. The design employs a split-heap to allow concurrent local collection, but exploits notions of procrastination and cleanliness to avoid creating forwarding pointers. Procrastination stalls threads about to perform an operation that would otherwise introduce a forwarding pointer, and thus can be used to eliminate read barriers for any exporting write. Cleanliness is an important optimization that helps avoid the cost of stalling by using runtime information to determine when it is safe to copy (rather than move) an object, deferring repair of pointers from the old (local) instance of the object to the new (shared) copy until a later collection. Experimental results on a range of benchmarks and architectural platforms indicate that read barrier elimination contributes to notable performance improvement without significantly complicating the runtime system. In this section, we quantify the cost/benefit of read barriers in our system. The context of our investigation is a programming model that is mostly functional (our benchmarks are written in the asynchronous extension [24] of Concurrent ML [18] ), and that naturally supports large numbers of lightweight user-level threads. We have implemented our garbage collector for MultiMLton [17] , a parallel extension to MLton [16] , that targets scalable, many-core platforms.. In our implementation, lightweight threads are multiplexed over kernel threads, with one kernel thread pinned to every core. Each core has a local heap, and a single shared heap is shared among all of the cores; the runtime system enforces the necessary heap invariants described earlier. In our experiments, we fixed the heap size to 3X the minimum heap size under which the programs would run.. MultiMLton performs a series of optimizations to minimize heap allocation, thus reducing the set of read barriers actually generated. For example, references and arrays that do not escape out of a function are flattened. Combined with aggressive inlining and simplification optimizations enabled by whole-program compilation, object allocation on the heap can be substantially reduced.. The compiler and runtime system ensure that entries on thread stacks never point to a forwarded object. Whenever an object pointer is stored into a register or the stack, a read barrier is executed on the object pointer to get the current location of the object. Immediately after an exporting write or a context switch, the current stack is walked and references to forwarded objects are updated to point to the new location of lifted objects in the shared heap. Additionally, before performing an exporting write, register values are saved on the stack, and reloaded after exit. Thus, as a part of fixing references to forwarding pointers from the stack, references from registers are also fixed. This ensures that the registers never point to forwarded objects either. Hence, no read barriers are required for dereferencing object pointers from the stack or registers. This optimization is analogous to "eager" read barriers as described in [4] . Eager read barrier elimination has marked performance benefits for repeated object accesses, such as array element traversals in a loop, where the read barrier is executed once when the array location is loaded into a register, but all further accesses can elide executing the barrier.. Whenever an object is lifted to the shared heap, the original object's header is set to FORWARDED and the first word of the object is overwritten with the new location of the object in the shared heap. Before an object is read, the mutator checks whether the object has been forwarded, and if it is, returns the new location of the object. Hence, our read barriers are conditional [5, 6] . Figure 1 shows the pseudo-C code for our read barrier. MLton represents non-value carrying constructors of (sum) datatypes us- ing non-pointer values. If such a type additionally happens to have value-carrying constructors that reference heap-allocated objects, the non-pointer value representing the empty constructor will be stored in the object pointer field. Hence, the read barrier must first check whether the presumed pointer does in fact point to a heap object. Otherwise, the original value is returned (line 2). If the given pointer points to a forwarded object, the current location of the object stored is returned. Otherwise, the original value is returned. We evaluated a set of 8 benchmarks (described in Section 7.1) running on a 16 core AMD64, a 48 core Intel SCC and an 864 core Azul Vega 3 machine to measure read barrier overheads. Figure 2 shows these overheads as a percentage of mutator time. Our experiments reveal that, on average, the mutator spends 20.1%, 15.3% and 21.3% of time executing read barriers on the AMD64, SCC and Azul architectures, respectively, for our benchmarks.. Although a Brooks-style unconditional read barrier would have avoided the cost of the second branch in our read barrier implementation, it would necessitate having an additional address length field in the object header for an indirection pointer. Most objects in our system tend to be small. In our benchmarks, we observed that 95% of the objects allocated were less than 3 words in size, including a word-sized header. The addition of an extra word in the object header for an indirection pointer would lead to substantial memory overheads, which in turn leads to additional garbage collection costs. Hence, we choose to encode read barriers conditionally rather than unconditionally.. But, does the utility of the read barrier justify its cost? We measure the number of instances the read barrier is invoked and the number of instances the barrier finds a forwarded object (see Figure 3) . We see that read barriers find forwarded objects in less than one thousands of a percent of the number of instances they are invoked. Thus, in our system, the cost of read barriers is substantial, but only rarely do they have to perform the task of forwarding references. These results motivate our interest in a memory management design that eliminates read barriers altogether. Modern garbage collectors rely on read and write barriers for encapsulating operations to be performed when the mutator reads or writes a reference from or to some heap allocated object. The Baker read barrier [5] was the first to use protection and invariants for mutator accesses. While the Baker read barrier is a conditional read barrier, the Brooks read barrier [8] is an unconditional read barrier, where all loads unconditionally forward a pointer in the object header to get to the object. For objects that are not forwarded, this pointer points to the object itself. The Brooks read barrier eliminates branches but increases the size of objects. Trading branches with loads is not a clear optimization as modern processors allow speculation through multiple branches, especially ones that are infrequent.. Over the years, several local collector designs [1, 9, 21, 22] have been proposed for multithreaded programs. Recently, variations of local collector design have been adopted for multithreaded, functional language runtimes like GHC [14] and Manticore [3] . Doligez et al. [9] proposed a local collector design for ML with threads where all mutable objects are allocated directly on the shared heap, and immutable objects are allocated in the local heap. Similar to our technique, whenever local objects are shared between cores, a copy of the immutable object is made in the shared heap. Although this design avoids the need for read and write barriers, allocating all mutable objects, irrespective of their sharing characteristics can lead to poor performance due to increased number of shared collections, and memory access overhead due to NUMA effects and uncached shared memory as in the case of SCC. It is for this reason we do not treat the shared memory as the oldest generation for our local generation collector unlike other designs [9, 14] .. Several designs utilize static analysis to determine objects that might potentially escape to other threads [13, 22] . Objects that do not escape are allocated locally, while all others are allocated in the shared heap. The usefulness of such techniques depends greatly on the precision of the analysis, as objects that might potentially be shared are allocated on the shared heap. This is undesirable for architectures like the SCC where shared memory accesses are very expensive compared to local accesses. Compared to these techniques, our design only exports objects that are definitely shared between two or more cores. Our technique is also agnostic to the source language, does not require static analysis, and hence can be implemented as a lightweight runtime technique.. Anderson [1] describes a local collector design (TGC) that triggers a local garbage collection on every exporting write of a mutable object, while immutable objects, that do not have any pointers, are copied to the shared heap. This scheme is a limited form of our cleanliness analysis. In our system, object cleanliness neither solely relies on mutability information, nor is it restricted to objects without pointer fields. Moreover, TGC does not exploit delaying exporting writes to avoid local collections. However, the paper proposes several interesting optimizations that are applicable to our system. In order to avoid frequent mutator pauses on exporting writes, TGC's local collection runs concurrently with the mutator. Though running compaction phase concurrently with the mutator would require read barriers, we can enable concurrent marking to minimize pause times. TGC also proposes watermarking scheme for minimizing stack scanning, which can be utilized in our system to reduce the stack scanning overheads during context switches and exporting writes of clean objects.. Marlow et al. [14] propose exporting only part of the transitive closure to the shared heap, with the idea of minimizing the objects that are globalized. The rest of the closure is exported essentially on demand during the next access from another core. This design mandates the need for a read barrier to test whether the object being accessed resides in the local heap of another core. However, since the target language is Haskell, there is an implicit read barrier on every load, to check whether the thunk has already been evaluated to a value. Since our goal is to eliminate read barriers, we choose to export the transitive closure on an exporting write.

O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Outage capacities and optimal power allocation for fading multiple-access channels Abstract-We derive the outage capacity region of an -user fading multiple-access channel (MAC) under the assumption that both the transmitters and the receiver have perfect channel side information (CSI). The outage capacity region is implicitly obtained by deriving the outage probability region for a given rate vector. Given a required rate and average power constraint for each user, we find a successive decoding strategy and a power allocation policy that achieves points on the boundary of the outage probability region. We discuss the scenario where an outage must be declared simultaneously for all users (common outage) and when outages can be declared individually (individual outage) for each user. W IRELESS communication channels vary over time due to user mobility. By applying optimal dynamic power and rate allocation strategies, the Shannon capacities with channel side information (CSI) at both the transmitter and the receiver of a single-user fading channel, a fading multiple-access channel (MAC), and a fading broadcast channel are obtained in [1] , [2] , and [3] , respectively. 1 These results have also been extended to the fading multiple-antenna multiple-access and broadcast channels in [4] , [5] . The Shannon capacity implies no complexity or delay constraints, and is obtained by varying the transmit power and possibly the rate relative to the channel fading conditions such that the average rate is maximized. For delay-constrained applications, Shannon capacity is not a good performance measure, since the transmission delay depends on the channel variation. Thus, a better performance measure for such systems is the zero-outage capacity, defined as the maximum instantaneous mutual information rate that can be maintained under all fading conditions through optimal power control. Under the assumption that CSI is available at both the transmitter and the receiver, the zero-outage capacity regions and the corresponding optimal power allocation schemes are derived for the fading MAC and the fading broadcast channel in [6] and [7] , respectively. 2 Zero-outage capacity is somewhat pessimistic, however, since a constant rate must be maintained under any fading condition. By allowing some transmission outage during severe fading conditions, the maximum mutual information rate that can be kept constant during nonoutage increases. This motivates the investigation of outage channel capacity, defined as the maximum instantaneous information rate that can be maintained under any fading condition during nonoutage such that the allowed average transmission outage probability is satisfied.. Outage capacity is most relevant in a slow-fading environment, where the channel can be assumed to be constant over the duration of a codeword. If only the receiver has CSI in a point-to-point channel, then the transmitter always transmits at a constant rate and cannot use any form of power control. In this scenario, an outage occurs whenever the channel cannot support transmission at the designated constant rate, i.e., whenever the instantaneous mutual information is less than the rate of transmission. Thus, the outage probability is equal to the probability that the channel cannot support the given rate, which is roughly equal to the probability of a decoding error when a channel code designed for a given rate is used on a channel with instantaneous mutual information below this given rate [8] . Alternatively, the outage probability can be viewed as the fraction of time that an incorrect codeword is received.. If both the receiver and the transmitter have CSI, the transmitter can use power control to conserve power by not transmitting at all during designated outage periods and by varying the amount of transmit power during nonoutages such that the instantaneous mutual information is exactly equal to (instead of exceeding) the rate of transmission. Here, the outage probability is described as the probability of not transmitting/receiving a codeword at all, instead of as the probability of decoding error, as is necessarily the case without transmitter CSI. We can also view the outage probability as the fraction of time that no codeword is received, which is relevant to many practical scenarios (e.g., in cellular systems, mobile units outside the service range of a base station are said to be in outage).. For a MAC, the same interpretations for outage probability and capacity hold. With or without transmitter CSI, the outage probability of User is approximately equal to the fraction of time that an incorrect or no codeword is received from User .. A motivating example is an uplink channel in which each transmitter wishes to send constant-rate video to the base station. Severe fading may preclude sufficiently high-rate communication from occurring at all times, but high-rate communication may be possible 95% of the time.. In [8] , the minimum outage probability problem is solved for the single-user fading channel. For an -user fading broadcast channel, under different assumptions about whether the transmission to all users is turned off simultaneously or individually, the optimal power allocation strategy that minimizes the common outage probability or achieves the boundary of the outage probability region of the users under a total average power constraint of all users is derived in [7] . An alternative notion of capacity combining the ideas of outage and ergodic capacity, referred to as the service outage capacity and minimum-rate outage capacity, has also been recently considered [9] - [12] .. In this paper, we derive the outage capacity region and the optimal power allocation policies for an -user fading MAC under similar assumptions about whether the outage declaration from each user is simultaneous or individual. The outage capacity region is explicitly defined in terms of the achievable rates corresponding to the set of all power policies meeting the individual power constraints. Essentially, a user is in outage whenever his power is equal to zero, and is transmitting at the nonoutage rate at all other times. This simple definition of outage capacity gives a unified framework that allows us to easily treat the cases of common and individual outage. Though no explicit coding theorem and converse are given in this paper, operational meaning is given to the definition of the outage capacity region by relating the outage capacity to the zero-outage capacity, for which a rigorous coding theorem and converse exists. Intuitively, the outage capacity is the set of all rates achievable in all nonoutage fading states. Thus, the outage capacity can be related to the zero-outage capacity of the conditional distribution of the fading states, where the conditioning is on a nonoutage event. For the single-user channel, it is intuitively easy to see that the outage states should be the fading states with the smallest amplitude [8] . However, for multiuser channels, no such simple ordering of the joint fading states is possible, and the difficulty remains in determining what set of states should be set as outage states.. The zero-outage capacity of the fading MAC is derived in [6] . Specifically, it is shown in [6] that the zero-outage capacity region is implicitly obtained by determining, for each given rate vector , the set of average transmit powers such that each user can support rate under any fading condition. For the general case where the allowed outage probability of each user is larger than zero, we will show that the outage capacity region is implicitly obtained by determining, for each given rate vector , the set of all common outage probabilities or individual outage probability vectors such that each user can support rate under any nonoutage fading condition while satisfying his given average power constraint. Given the allowed outage probability of each user and a rate vector , we also solve the dual problem of finding the average power region of the users required to support for the given outage probability vector.. For a given rate vector , in order to solve the optimization problem of minimizing the common outage probability or bounding the outage probability region for a given average power constraint on each of the users, we use the Lagrangian method with multiple constraints. Since there is an independent average power constraint for each of the users, Lagrangian multipliers are needed. For each given Lagrangian multiplier vector , the optimization problem is readily solved by applying the techniques developed in [6] and [7] . Standard convex optimization algorithms can be used to find the appropriate Lagrangian multiplier vector for the users such that the average power constraints are satisfied simultaneously.. The remainder of this paper is organized as follows. In Section II, we present the fading MAC model. In Sections III and IV, we give the definitions and notations that will be used in the rest of the paper. In Section V, for the case of simultaneous outage declaration, we derive the minimum common outage probability for a given rate vector and the corresponding optimal power allocation strategy. In addition, the average power region for supporting with a given common outage probability is obtained. As for individual outage declaration, we derive in Section VI the outage probability region boundary for a given rate vector , the corresponding optimal power allocation strategy, and the required average power region for supporting with the given outage probability constraint of each user. In Section VII, we discuss the relationship between outage capacities of the MAC and the broadcast channel. In Section VIII, we present the main difference in the solutions to the above problems when additional peak power constraints are imposed on the users. Our conclusions are given in Section IX. We have obtained the outage capacity region of fading MACs under the assumption that perfect CSI is available both at the transmitters and at the receiver. The capacity region is obtained implicitly by deriving the minimum common outage probability and the individual outage probability region for a given rate vector. Given the average power constraint of each user, we have derived the power allocation policy that minimizes the common outage probability for a given rate vector when transmission to all users is turned off simultaneously.. When an outage can be declared for each user individually, we have derived a power allocation strategy to achieve the outage probability region boundary for the given rate vector.. In both cases, the optimal power allocation policies have been shown to be purely deterministic functions of the fading state, and standard convex optimization algorithms have been used for obtaining the optimal power control parameters. These optimal power allocation policies show that, similar to the zerooutage scenario, successive decoding is optimal and in each fading state, the decoding order is determined by the power prices of the users and their fading gains. By applying these optimal power allocation strategies, we have also obtained the average power regions that can support a rate vector with a given common outage probability or a given outage probability vector for the users. When there are additional peak power constraints, the optimal power allocation for the users in each fading state can be obtained through a greedy algorithm, though in general it cannot be achieved by successive decoding.

O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I I O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O B I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O


Vectorized Matlab codes for linear two-dimensional elasticity A vectorized Matlab implementation for the P 1 finite element is provided for the twodimensional linear elasticity with mixed boundary conditions. Vectorization means that there is no loop over triangles. Numerical experiments show that our implementation is more efficient than the standard implementation with a loop over all triangles. Matlab is nowadays a widely used tool in education, engineering and research and becomes a standard tool in many areas. But Matlab is a matrix language, that is, Matlab is designed for vector and matrix operations. For best performance in extensive problems, one should take advantage of this.. We propose a Matlab implementation of the P 1 finite element for the numerical solutions of two-dimensional linear elasticity problems. Instead of mixing finite element types (e.g. [2, 1] ), we propose a P 1 -triangle vectorized code able to treat medium size meshes in "acceptable" time. Vectorization means that there is no loop over triangles nor nodes. Our implementation needs only Matlab basic distribution functions and can be easily modified and refined.. The paper is organized as follows. The model problem is described in Section 2, followed by a finite element discretization in Section 3. The data representation used in Matlab programs is given in Section 4. The heart of the paper is the assembling functions of the stiffness matrix in Section 5 and the right-hand side in Section 6. Numerical experiments are carried out in Section 7 where post-processing functions are given. Matlab programs used for numerical experiments are given in the appendix.

O B O O O O B O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O


Selecting hierarchical clustering cut points for web person-name disambiguation Hierarchical clustering is often used to cluster person-names referring to the same entities. Since the correct number of clusters for a given person-name is not known a priori, some way of deciding where to cut the resulting dendrogram to balance risks of over-or under-clustering is needed. This paper reports on experiments in which outcome-specific and result-set measures are used to learn a global similarity threshold. Results on the Web People Search (WePS)-2 task indicate that approximately 85% of the optimal F 1 measure can be achieved on held-out data. In the real world, we often face the problem of name disambiguation; this problem becomes more serious in Web IR due to the huge amount of information available on the Web. Web person-name disambiguation has attracted considerable attention, and many approaches have been proposed [1] . Among them, Hierarchical Agglomerative Clustering (HAC) [2] is one of the most widely used. HAC views each Web page containing personnames as a seed cluster, iteratively combining the most similar pair of clusters to form a larger cluster that replaces the pair. Some stopping criterion is used to terminate the agglomeration process in a way that balances the risk of merging references to different people with the risk of failing to merge references to the same person. We can in principle base the stopping point on both global criteria (e.g., the resulting total number of clusters) and on local criteria (e.g., the similarity of items to be merged). We operationalize that approach by using the local criteria to assign a merging score and then thresholding that score based on the global criteria. As Figure 1 illustrates, we can view this process as first creating a dendrogram that represents all possible merging decisions (all the way to one single large cluster), and then selecting an appropriate cut point that partitions the set pages into disjoint clusters. As others have done, we adopt the simplifying "one person per document" heuristic [1] : we treat all mentions of an ambiguous person-name that are found in the same Web page as referring to the same person. This assumption essentially reduces our original problem of clustering individual person-name mentions to the well-studied problem of clustering Web pages. Our results demonstrate that the method we have described chooses an appropriate cut-point threshold reasonably well, which is an important capability when using hierarchical clustering for Web person-name disambiguation. Similar challenges also arise in other applications. For example, search engines seek to cluster news pages from different sources so that pages talking about the same events can be presented together. As it is impossible to know a priori how many events will be described in today's news, the risks of over-or under-clustering must be balanced. Our decomposition of that problem into assigning local merge scores and then choosing a global cut-point threshold is useful, but it prevents us from jointly learning how best to combine local and global evidence. We are, therefore, also interested in exploring applications such as news clustering for which more data might be available (e.g., in the Topic Detection and Tracking collections) to train joint models. What we learn from those applications might then be brought back to inform our future work on the WePS task. Our principal interest in this work is in cut point selection based on global features, so we use well-known similarity-based techniques to generate merging scores. In particular, we define a similarity measure between any pair of pages based on five feature types: all automatically detected named entities, the specific matches to the query name, outlink structure, focal text around mentions, and full text. This page similarity measure is described in detail in [3] . As a cluster merging score we use either the minimum inter-cluster page similarity (for "complete link") or the maximum inter-cluster page similarity (for "single link").. As a basis for selection of a global cut-point on merging scores, we define two cut-point dependent features: (1) the merging score threshold value for a cut-point, and (2) the number of clusters that would result from a cut-point. We also define two additional features that depend on the query but not on the specific cut point that is chosen: (3) the mean across all inter-page similarity values (4) the standard deviation across all inter-page similarity values. Some query result sets exhibit a greater central tendency than others; we model that effect with one additional feature. We design a similarity density d i for page i as:. Where N is the number of pages in the result set (which we set to 100 for training and 150 for testing). We then define a fifth query-dependent (but cut-point-independent) feature: (5) the standard deviation across all pages of the mean inter-page similarity between a page and all other pages (i.e., the standard We model our problem of cut-point selection as binary classification using a Support Vector Machine (SVM) with a radial basis function kernel. For each candidate cut-point threshold (C i ) in a dendrogram, we must decide whether to cut at that point based on our five features and on what we have leaened from the 30 person-name training query result sets that are available in the Web People Search (WePS)-1 test collection. We then test the system on the 30 person-name query result sets in the WePS-2 test collection. 1 Our goal is to optimize macro-averaged F (the balanced harmonic mean of recall and precision, averaged over ground-truth clusters). We therefore define F best as the optimal (post hoc) value of F for any cut-point threshold, and we seek to optimize F/F best by varying our process for SVM training. During training, we calculate F best once for each person-name query result set and we calculate F for each candidate cut-point threshold on that same result set. We define an acceptance region r, a value between 0 and 1 that determines how close to F best a result must be to be used as a positive exemplar during the training process. If F>rF best the cut-point is used. During testing, our SVM may identify more than one suitable cut-point threshold for a given person-name query result set. In such cases, we choose the one that yields a number of clusters closest to the average number of clusters yielded by all such thresholds. Tables 1 and 2 show some results for an experiment in which we fixed r=0.4 and trained on thirty WePS-1 person-name queries. The mean F/F best over 30 WePS-2 queries was 0.84% and 82% for single-link and complete-link clustering, respectively.

O O B O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O


Nonlinear Multilayered Representation of Graph-Signals We propose a nonlinear multiscale decomposition of signals defined on the vertex set of a general weighted graph. This decomposition is inspired by the hierarchical multiscale (BV , L 2 ) decomposition of Tadmor, Nezzar, and Vese (Multiscale Model. Simul. 2(4): 2004). We find the decomposition by iterative regularization using a graph variant of the classical total variation regularization (Rudin et al, Physica D 60(1-4):259-268, 1992). Using tools from convex analysis, and in particular Moreau's identity, we carry out the mathematical study of the proposed method, proving the convergence of the representation and providing an energy decomposition result. The choice of the sequence of scales is also addressed. Our study shows that the initial scale can be related to a discrete version of Meyer's norm (Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations, 2001) which we introduce in the present paper. We propose to use the recent primal-dual algorithm of Chambolle and Pock (J. Math. Imaging Vis. 40:120-145, 2011) in order to compute both the minimizer of the graph total variation and the corresponding dual norm. By applying the graph model to digital images, we investigate the use of nonlocal methods to the multiscale decomposition task. Since the only assumption needed to apply our method is that the input data is living on a graph, we are also able to tackle the task of adaptive multi- Our main concern in this paper is to introduce a multiscale representation for data sets defined on the vertex set of a general undirected graph. Before entering into technical details, we introduce the motivation and background for our work. We have presented a method that allows to generate adaptive multiscale descriptions of general digital data sets that can be supported by a weighted graph structure. The proposed method works by extracting successive layers from the input data in a nonlinear way. Those layers are obtained by successive minimizations of a graph variant of the classical TV regularization approach. The behavior of the method has been studied and quantified. Issues regarding the choice of the scale parameters have been formulated as a convex optimization problem and solved using the recent primal-dual algorithm of [13] . The latter formulation draws connections with previous works concerning the G norm introduced in [28] . We have shown how the proposed method can be applied to generate multiscale representations of digital images. In this setting, we have shown how the incorporation of nonlocal interactions can improve the multiscale description. We have also shown how the proposed method can be applied to nonuniformly sampled data and gave examples for 3-D triangular meshes and point clouds multiscale representation. We present in this section results obtained by applying the proposed multilayered representation to three types of digi-. Decompose v i into u i+1 and v i+1 using model (28)- (29) using λ i+1 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Matching with couples revisited It is well known that a stable matching in a many-to-one matching market with couples need not exist. We introduce a new matching algorithm for such markets and show that for large random markets the algorithm will find a stable matching with high probability. In our model we allow the number of couples to grow at a near-linear rate. Furthermore, truth-telling is an approximated equilibrium in the game induced by the new matching algorithm. Our results are tight: for markets in which the number of couples grows at a linear rate, we show that with constant probability no stable matching exists.

B O O O O O O O O O O B O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O


A framework for computation-memory algorithmic optimization for signal processing Abstract-The heterogeneity of today's computing environment means computation-intensive signal processing algorithms must be optimized for performance in a machine dependent fashion. In this paper, we present a dynamic memory model and associated optimization framework that finds a machine-dependent, near-optimal implementation of an algorithm by exploiting the computation-memory tradeoff. By optimal, we mean an implementation that has the fastest running time given the specification of the machine memory hierarchy. We discuss two instantiations of the framework: fast IP address lookup, and fast nonuniform scalar quantizer and unstructured vector quantizer encoding. Experiments show that both instantiations outperform techniques that ignore this computation-memory tradeoff.

O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Evaluation over thousands of queries Information retrieval evaluation has typically been performed over several dozen queries, each judged to near-completeness. There has been a great deal of recent work on evaluation over much smaller judgment sets: how to select the best set of documents to judge and how to estimate evaluation measures when few judgments are available. In light of this, it should be possible to evaluate over many more queries without much more total judging effort. The Million Query Track at TREC 2007 used two document selection algorithms to acquire relevance judgments for more than 1,800 queries. We present results of the track, along with deeper analysis: investigating tradeoffs between the number of queries and number of judgments shows that, up to a point, evaluation over more queries with fewer judgments is more costeffective and as reliable as fewer queries with more judgments. Total assessor effort can be reduced by 95% with no appreciable increase in evaluation errors. Over the past 40 years, Information Retrieval research has progressed against a background of ever-increasing corpus size. From the 1,400 abstracts in the Cranfield collection, the first portable test collection, to the 3,200 abstracts of the Communications of the ACM (CACM), to the 348,000 Medline abstracts (OHSUMED), to the first TREC collections of millions of documents, to the web-billions of HTML and other documents-IR research has had to address larger and more diverse corpora.. As corpora grow, the assessor effort needed to construct test collections grows in tandem. Sparck Jones and van Rijsbergen introduced the pooling method [17] to deal with the problem of acquiring judgments. Rather than judge every document to every query, the documents ranked by actual retrieval systems could be pooled and judged, thus focusing judging effort on those documents least likely to be nonrelevant. The pooling method has been successful for decades. However, recent work suggests that the growth in the size of corpora is outpacing even the ability of pooling to find and judge enough documents [9] . Rather than trying to keep up simply by judging more documents, there has been interest in focusing judging effort even better and make smarter inferences when few judgments are available.. With fewer judgments available, estimates of evaluation measures will have higher variance. One way to cope with this is to evaluate over more queries. Web search engines, for instance, typically judge very shallow pools for thousands of queries. For the recall-based measures that are ubiquitous in retrieval research, such a shallow pool is not enough.. In this work we describe an evaluation over a corpus of 25 million documents and 10,000 queries, the Million Query Track that ran for the first time at the Text REtrieval Conference (TREC) in 2007 [1] . Using two recent method for selecting documents and evaluating over small collections, we achieve results very similar to an evaluation using 149 queries judged with more depth, with 62% of the assessor effort but 11 times as many queries. But this only establishes an upper bound. An evaluation with high rank correlation can be achieved with 3% of the effort over only 1.34 times as many queries, and using analysis of variance and stability studies, we show that the amount of effort needed to establish that differences between systems are not simply due to random variation in scores is at most 5% of the effort over 1.14 times as many queries.. We begin in Section 2 by describing the methods used to select documents and evaluate for the Million Query Track. In Section 3 we describe the setup of the experiment and the relevance judgments collected. Section 4 presents initial results, comparing the two methods to the baseline set of 149 queries. In Section 5, we approach the question of the number of queries and judgments for each query that is needed to evaluate with minimum effort, and also explore the reusability of such a small test collection. The Million Query Track and subsequent analyses show that we can evaluate retrieval systems with greatly reduced effort, beyond what was done for the track, and down to a few hundred queries with several dozen judgments for each one. Even when this is too few judgments to reliably distinguish between systems, we can still identify the systems that we have the least confidence in and focus on acquiring more judgments for them, thus ensuring future reusability.. The results of our study confirm those found by Sanderson & Zobel [16] , Jensen [15] , and Carterette & Smucker [12] , all of which argued that evaluation over more queries with fewer or noisier judgments is preferable to evaluation over fewer queries with more judgments. There are tradeoffs, of course: failure analysis may be more difficult when judgments are scarce, and there may be limited data for training new algorithms. Exploring and quantifying these tradeoffs are clear directions for future work. In this section, we describe a set of analyses performed on the data collected as described above. Our analyses are of two forms: (1) efficiency studies, aimed at determining how quickly one can arrive at accurate evaluation results and (2) reusability studies, aimed at determining how reusable our evaluation paradigms are in assessing future systems. The 24 runs were evaluated over the TB set using trec_eval and over the 1MQ set using EM AP and statMAP. If TB is representative, we should see that EM AP and statMAP agree with each other as well as TB about the relative ordering of systems. Our expectation is that statMAP will present better estimates of MAP while EM AP is more likely to present a correct ranking of systems.. The left side of Table 2 shows the MAP for our 24 systems over the 149 Terabyte queries, ranked from lowest to highest. The average number of unjudged documents in the top 100 retrieved is also shown. Since some of these systems did not contribute to the Terabyte judgments, they ranked quite a few unjudged documents.. The right side shows EM AP and statMAP over the queries judged for our experiment, in order of increasing MAP over Terabyte queries. It also shows the number of unjudged documents in the top 100. EM AP and statMAP are evaluated over somewhat different sets of queries; statMAP excludes queries judged by MTC and queries for which no relevant documents were found, while EM AP includes all queries, with those that have no relevant documents having some probability that a relevant document may yet be found. Overall, the rankings by EM AP and statMAP are fairly similar, and both are similar to the "gold standard". Figure 1 shows a graphical representation of the two rankings compared to the ranking by Terabyte systems. Figure 2 shows how statMAP, EM AP , and MAP over TB queries correlate. All three methods have identified the same three clusters of systems, separated in Table 2 by horizontal lines; within those clusters there is some variation in the rankings between methods. For statMAP estimates (Figure 2 , left plot), besides the ranking correlation, we note the accuracy in terms of absolute difference with the TB MAP values by the line corresponding to the main diagonal.. Some of the bigger differences between the methods are noted in Table 2 by a * indicating that the run moved four or more ranks from its position in the TB ranking, or a † indicating a difference of four or more ranks between EM AP and statMAP. Both methods presented about the same number of such disagreements, though not on the same systems. The biggest disagreements between EM AP and statMAP were on umelbexp and umelbsim, both of which EM AP ranked five places higher than statMAP. Each method settled on a different "winner": indriDM for the TB queries, JuruSynE for EM AP , and LucSpel0 and LucSyn0 tied by statMAP. However, these systems are all quite close in performance by all three methods.. We also evaluated statistical significance over the TB queries by a one-sided paired t-test at α = 0.05. A run denoted by a ‡ has a MAP significantly less than the next run in the ranking. (Considering the number of unjudged documents, some of these results should be taken with a grain of salt.) Significance is not transitive, so a significant difference between two adjacent runs does not always imply a significant difference between other runs. Both EM AP and statMAP swapped some significant pairs, though they agreed with each other for nearly all such swaps.. An obvious concern about the gold standard is the correlation between the number of unjudged documents and MAP: the tau correlation is −.517, or −.608 when exegyexact (which often retrieved only one document) is excluded. This correlation persists for the number unjudged in the top 10. To ensure that we were not inadvertently ranking systems by the number of judged documents, we selected some of the top-retrieved documents in sparsely-judged systems for additional judgments. A total of 533 additional judgments only discovered 7 new relevant documents for the UAms systems, 4 new relevant documents for the ffind systems, but 58 for umelbexp. The new relevant judgments caused umelbexp to move up one rank. This suggests that while the ranking is fair for most systems, it is likely underestimating umelbexp's performance.. It is interesting that the three evaluations disagree as much as they do in light of work such as Zobel's [21] . There are at least three possible reasons for the disagreement: (1) the gold standard queries represent a different space than the rest; (2) the gold standard queries are incompletely judged; and (3) the assessors did not pick queries truly randomly. The fact that EM AP and statMAP agree with each other more than either agrees with the gold standard suggests to us that the gold standard is most useful as a loose guide to the relative differences between systems, but does not meaningfully reflect "truth" over the larger query sample. But the possibility of biased sampling affects the validity of the other two sets as well: as described above, assessors were allowed to choose from 10 different queries, and it is possible they chose queries that they could decide on clear intents for rather than queries that were unclear. It is difficult to determine how random query selection was. We might hypothesize that, due to order effects, if selection was entirely random we would expect to see the top most query selected most, followed by the second-ranked query, followed by the third, and so on, roughly conforming to a log-normal distribution. This in fact is not what happened; instead, assessors chose the top-ranked query slightly more often than the others (13.9% of all clicks), but the rest were roughly equal (slightly under 10%). But this would only disprove random selection if we could guarantee that presentation bias holds in this situation. Nevertheless, it does lend weight to the idea that query selection was not random.. In an attempt to resolve some of these questions, we evaluated systems over the depth10 set using trec_eval. Evaluation results over this set do not correlate well to any previous set, with rank correlations in the 0.6 − 0.7 range. In particular, the Luc* systems drop from the top tier to the second tier. This is a very interesting result that bears closer investigation, since if queries are a random sample it disagrees with the notion that two samples of queries can be used to evaluate systems reliably. Our current hypothesis (supported by conversations with the assessors) is that assessors selected these queries less randomly than other sets, so that they are not a representative sample. Note in Table 1 that the frequency of relevant documents in this set is significantly lower than any other set.

B O O O B B I B O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O B I I O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O B B O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O B O B I I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O B I I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O B O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O