Reliability in Source Coding With Side Information Abstract-We study error exponents for source coding with side information. Both achievable exponents and converse bounds are obtained for the following two cases: lossless source coding with coded information and lossy source coding with full side information (Wyner-Ziv). These results recover and extend several existing results on source-coding error exponents and are tight in some circumstances. Our bounds have a natural interpretation as a twoplayer game between nature and the code designer, with nature seeking to minimize the exponent and the code designer seeking to maximize it. In the Wyner-Ziv problem, our analysis exposes a tension in the choice of test channel with the optimal test channel balancing two competing error events. The Gaussian and binary-erasure cases are examined in detail. I N a typical lossy data compression problem, a source is to be compressed by an encoder at a prescribed rate so that a decoder may reproduce the source to within some desired fidelity (distortion). Sometimes present, in addition to the data to be compressed, is some correlated information that can be utilized by a second encoder, that is able to send a separate message to the decoder. We refer to this kind of problem as source coding with side information (SCSI). The setup is depicted in Fig. 1 , where a source is compressed by encoder one to a rate with the decoder having access to encoded side information , compressed at rate by encoder two, as well as the compressed version of from the first encoder.. The SCSI scenario arises in a variety of applications. For example, in video applications [1] , can represent a current frame, and a separate correlated frame sent from a second encoder. can even represent the frame(s) preceding the current frame in the stream: while the previous frames are certainly available to the encoder, the encoder's coding scheme can be simplified by not making use of this information and leaving the decoder to exploit the interframe dependence. A second example can be found in communication in networks with relays [2] . A source sends a message to a sink in a network containing a relay. One mode of operation for the relay is "compress and forward," i.e., for the relay to send a compressed version of its observation of the source-sink message to the sink. This compressed message can be used by the sink to further aid its decoding. SCSI appears in applications even beyond communication, for example (with minor changes), it has been proposed as a model for rate-constrained pattern recognition [3] .. For the lossless problem with coded side information (SCCSI), 1 and the lossy problem with full side information (Wyner-Ziv), the "rate region" problem, i.e., determining the rates required to meet a given average distortion constraint, is solved. In this paper, we study these two problems from an error-exponent standpoint. Our motivation for doing so is threefold:. 1) In the applications mentioned previously, the average distortion of a compression scheme is not the only important metric. Indeed, a video compression system with good average performance but that frequently yields poor images, or a communication system that suffers from frequent outages is usually deemed unacceptable. In addition to minimizing the average distortion, one would like to minimize the fraction of time in which the images are poor or the relay is unable to help. 2) In some important cases, there is no rate loss, meaning that there is no difference in the rate-distortion performance between the SCSI problem and the problem in which the side information is available to the encoder as well as the decoder. In particular, it is well known that this is true of both the binary erasure and quadratic Gaussian forms of the Wyner-Ziv problem [6] . This raises the question of whether these two systems are equivalent when performance is measured via error exponents instead of the average distortion. 3) Recently, a connection has been established between error exponents in channel coding and the stabilization of linear systems over noisy channels [7] , and there is a known interdependence between source-and channel-coding error exponents. Thus, new techniques in source-coding error exponents could aid our understanding of problems at the intersection of communication and control [8] . Both theorems can be viewed as a competitive game between two players, nature and the code designer. Nature's goal is to minimize the exponent and the code designer's goal is to maximize it. The structure of the problem determines the parameters and order of the plays. For example, in Theorem 1, nature plays first, choosing a "worst case" side information distribution. Then, knowing nature's choice, the code designer picks the best codebook (via its choice of test channel). Nature plays last, choosing the worst possible consistent joint distribution. Notice that the choices at each step match the "information" available to the players.. A standard application of the change-of-measure argument [10, p. 268] provides the following upper bound on the SCCSI exponent (6) where the is actually a maximum since the objective is either or . It is straightforward to verify that , and so formally provides an improvement upon the standard sphere-packing upper bound. In the game theoretic interpretation, the exponent is obtained by letting nature's play reveal the joint distribution of the source and side information, and then the code designer plays, choosing the best codebook. But in the SCCSI problem, the helper's test channel can only depend on the marginal type of the side information. Thus, our improved upper bound better captures the inherent structure of the problem.. We remark that in this and Section IV, the solutions to the optimization problems in the theorems can be approximated arbitrarily well by searching over a fine grid. We have not studied conditions under which the optimization problems may be solved more efficiently (e.g., using convexity), nor for conditions under which a min-max theorem may simplify the problems. This may be interesting future work.. The optimizations in Theorems 1 and 2 differ in several respects. Foremost, in Theorem 2 the innermost optimization is over , so that , , adhere to the Markov structure, yet in the achievable exponent this Markov constraint is not present. This differing Markov structure is also present in the partial Wyner-Ziv exponent results of Jayaraman and Berger [18] , [32] who attribute the gap between the sphere packing and random exponents (present even at low rates) in the binning exponent problem they studied to this type of difference in the Markov structure. A second difference is the presence of a cardinality bound on the set in Theorem 2. This cardinality bound is established in a slightly nonstandard way, combining the usual Carathéodory argument with a characterization of the optimizing distribution using the KKT conditions. This approach, however, is tailored to the expression in Theorem 2 and does not seem to apply to the analogous quantity in Theorem 1. The other differences between and are the range of the innermost optimization, the presence of the binning term in the achievable exponent and the fact that the choice of test channel is restricted in the upper bound. (This latter difference can be eliminated by adding the restriction to the choice of test channel in the lower bound, which only weakens the result.) Despite these differences, the bounds provided by the theorems do allow us to determine the error exponent exactly in some special cases. When , there is no possibility of encoding the side information. Taking to be constant in both exponents, one recovers the standard point-to-point exponent More generally, if is sufficiently large and is sufficiently close to , then one can show that the achievable exponent (3) coincides with the upper bound in (6) and hence also (4) . The proof of this fact parallels the proof that the random-coding and sphere-packing bounds for channel coding coincide above the critical rate.. As in the SCCSI case, in the Wyner-Ziv case the same gametheoretic interpretation holds, but there are more parameters and the game becomes more elaborate. Nature plays first, choosing the most "difficult" marginal distribution for . The code designer plays next, selecting the "best" test channel for that difficult source. Nature plays again choosing the worst marginal distribution for the side information. Then, knowing everything chosen so far, the code designer chooses the estimation function. Nature has the final play, choosing the worst consistent joint distribution for triple the , , . Once again the choices and order of plays match the problem.. The nature of the optimizations in Theorems 3 and 5 give us some insight into the design of practical coding schemes by revealing a tension, which we examine in detail in Section V for the binary erasure and Gaussian problems. Briefly, we see that the objective functions (respectively, ) contain three cases which correspond to 1) a violation of the distortion constraint even when the codeword is decoded correctly; 2) the use of binning, leading to the potential for decoding the wrong codeword; 3) no possibility for error. A large codebook allows for a cleaner quantization and hence lower chance of the first kind of event. But this large codebook comes with the requirement of binning, leading to the potential for the second kind of event. Thus, these two kinds of errors are in tension.. Theorem 3 allows us to determine a portion of the reliability function for a certain functional source coding problem. If we wish to reproduce a function of the source losslessly at the decoder, who already has , then the rate required is , which follows from the results of Orlitsky and Roche [36] . Setting the distortion measure to be ( is the hamming measure) and evaluating Theorem 3 in the limit as provides an achievable exponent for this problem. This can be seen by always choosing so that and letting the reproduction function be . Using the fact that , one can show that the limit as of the right-hand side of (10) is (16) An upper bound on the error exponent for this problem is given by (17) On account of the fact that both (16) and (17) are optimizations of a continuous function over a compact sets, the is attained. The relationship between these two functions is analogous to the relationship between the sphere-packing and random coding exponents in channel coding [10, Lemma 2.5.4]. Thus, for until some critical rate , the reliability function for the functional source coding problem is given exactly by V. EXAMPLES

O O B I O B I O O B I O B I O B I O O O O O O O O O O O O O O B I O O O O O B I O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O


A Simpler Linear-Time Recognition of Circular-Arc Graphs We give a linear-time recognition algorithm for circular-arc graphs based on the algorithm of Eschen and Spinrad A circular-arc graph (see Fig. 1 ) is an intersection graph of arcs on the circle. That is, every vertex is represented by an arc, such that two vertices are adjacent if and only if the corresponding arcs intersect. The arcs constitute a circular-arc model of the graph. Circular-arc graphs generalize interval graphs which are the intersection graphs of intervals on the line. An extensive overview of circular-arc graphs can be found in the book by Spinrad [18] . Circular-arc graphs can be used to model objects of a circular or a repetitive nature. Recent applications of circular-arc graphs are in modeling ring networks [20] and item graphs of combinatorial auctions [3] . The class of circular-arc graphs and the corresponding recognition problem was first defined by Hadwiger, Debrunner and Klee [7, 11] . The problem of finding a recognition algorithm for circular-arc graphs was solved only after finding a recognition algorithm for interval graphs. Classical recognition algorithms for interval graphs, such as [2] , rely on the fact that any clique in an interval graph corresponds to a nonempty intersection of the corresponding intervals in the interval model. This property of interval graphs is called the Helly property. Due to the Helly property, the number of maximal cliques in an interval graph is linear. In contrast, arcs associated with a clique in a circular-arc graph do not have the Helly property. Furthermore, the number of maximal cliques in a circular-arc graph may be exponential in the size of the graph. Another difference between interval graphs and circular-arc graphs is that two arcs on the circle can intersect from two different directions.. The first characterizations of circular-arc graphs were given by Tucker [21] and by Gavril [6] . These characterizations did not lead to an efficient recognition algorithm.. Tucker [22] gave the first polynomial time recognition algorithm for circular-arc graphs. This algorithm splits into one of two cases according to whetherḠ is bipartite (G is co-bipartite). In caseḠ is not bipartite the algorithm finds an odd length induced cycle inḠ, and further splits into one of two subcases according to whether the cycle it found is of length 3 or of length at least 5. Using Tucker's terminology we refer to the first case where G is co-bipartite as Case I. We refer to the subcase where we found inḠ a cycle of length 3, and therefore we found in G an independent set of size 3, as Subcase IIa. We refer to the subcase where we found inḠ an induced cycle of length at least 5 as Subcase IIb.. Tucker showed how to implement his algorithm in O(n 3 ) time. One of the bottlenecks in Tucker's implementation is a preprocessing phase where he identifies containment relations between the neighborhoods of the vertices. Specifically, for every pair of vertices v and u he determines whether the neighborhood of v is contained in the neighborhood of u or vice versa. Furthermore, Tucker runs his algorithm recursively on particular graphs and this recursive structure also leads to cubic running time.. Spinrad [17] simplified Case I in Tucker's algorithm-the case where G is cobipartite. Spinrad reduced this case to the problem of recognizing two dimensional posets [19] . He constructs the poset using particular relations between the vertices of G. Two vertices are related in the poset if their corresponding arcs are either disjoint, one is contained in the other, or together they cover the circle. The relations between the arcs are determined from the relations between the neighborhoods of the vertices. In case G is a circular-arc graph then from any two total orders that represent the poset he can construct a representation for G. Spinrad showed that this algorithm runs in O(n 3 ) time.. Eschen and Spinrad [4, 5] gave an O(n 2 ) algorithm for recognizing circular-arc graphs by addressing the two bottlenecks in Tucker's implementation. Eschen and Spinrad show how to compute neighborhood containment relations in O(n 2 ) time. Specifically, they construct two graphs such that if G is indeed circular-arc graph then one of the graphs is an interval graph and the other graph is a chordal bipartite graph. These graphs are constructed such that from the neighborhood containment relation between two vertices in these graphs they can determine the neighborhood containment relation between the same vertices in G. The quadratic time bound follows since one can compute neighborhood containment relations in interval graphs and chordal bipartite graphs in quadratic time [4, 5, 13] .. Eschen and Spinrad also showed that in Case I of the algorithm, when G is cobipartite, they can use a reduction similar to the reduction that they used to determine neighborhood containment relations to determine all pair of arcs that can cover circle in a model of G in O(n 2 ) time. Since this was the only bottleneck in Spinrad's algorithm for this case, they obtain an O(n 2 ) implementation of Case I.. To implement Subcase IIa and Subcase IIb in O(n 2 ) time, Eschen and Spinrad changed the recursive structure of Tucker's algorithm. They show how to implement the algorithm such that each recursive call is on a co-bipartite graph (Case I) and therefore does not trigger further recursion. Since the sum of the sizes of the graphs in all recursive calls is proportional to the size of G, the quadratic bound follows. Unfortunately, as we show in this work, the algorithm of Eschen and Spinrad for Subcase IIa has a flaw.. Hsu [8] presented a different recognition algorithm for circular-arc graphs. Hsu's algorithm runs in O(mn) time and reduces the problem to recognition of circle graphs.. Recently, McConnell [14] presented the first recognition algorithm for circulararc graphs that runs in linear time. The algorithm reduced the problem to an interval graph recognition problem where specific intersection types between the intervals are specified. McConnell's algorithm uses the same preprocessing stage of Eschen and Spinrad where it computes neighborhood containment relations. To establish the linear time bound, McConnell tightens the analysis of Eschen and Spinrad's preprocessing stage. He shows that this preprocessing stage can be implemented in linear time since we are interested only in neighborhood containment relations between adjacent vertices, and the associated chordal bipartite graphs cannot be too large.. McConnell's algorithm is quite involved. Its most complicated computation is to find a partition of a graph into a particular kind of modules called modules. These modules are used to turn the input circular-arc graph into an interval graph with specific types of intersections between the intervals, and to find a model for this interval graph. McConnell first presents an implementation that runs in O(m + n log n) time. To get the linear time bound, a more complicated partitioning procedure has to be adapted from the linear-time transitive orientation algorithm of McConnell and Spinrad [15] which is by itself quite involved. This algorithm also uses probe interval graphs to find pairs of arcs that can cover the circle in linear time.. We give a new recognition algorithm for circular-arc graphs. This algorithm is based on the algorithm of Eschen and Spinrad, and runs in linear time. Our algorithm either finds an independent set of size 3, in which case the algorithm of Eschen and Spinrad applies Subcase IIa, or it concludes that the graph has Θ(n 2 ) edges and then the algorithm of Eschen and Spinrad in fact runs in linear time. The fact that a graph that does not have an independent set of size 3 has Θ(n 2 ) edges is an implication of Mantel's Theorem from 1907, which is considered to be the first result in extremal graph theory (see Bollobás [1] ).. Eschen and Spinrad find in Subcase IIa a particular maximal independent set and place the corresponding arcs on the circle. We show that there is a flaw in the way Eschen and Spinrad place the arcs of the independent set on the circle. We present a correction to this flaw, and describe a linear-time implementation of the correct placement algorithm. Our implementation then continues as the implementation of Eschen and Spinrad, but we tighten their analysis to show that the running time is linear. Our main new insight is that each subgraph considered by the algorithm while placing and ordering the arcs on the circle is dense. That is, the number of edges that each subgraph contains is quadratic in the size of its vertex set. Furthermore, the total size of these subgraphs is linear in the size of the input graph.. Our algorithm also performs a preprocessing phase where neighborhood containment relations for adjacent pairs of vertices are computed. As proved by McConnell [14] this can be done in linear time. As all previous algorithms, we also require a postprocessing verification step where we check that the representation we obtain is indeed a representation of G. McConnell [14] gave a straightforward linear-time implementation of this postprocessing step.. We describe a linear-time implementation of Subcase IIa. Subcase IIb can also be implemented in linear time in a similar way. We do not describe it here since we apply Subcase IIb only when we are sure that G has Θ(n 2 ) edges. Our implementation is simpler than McConnell's algorithm.. Hsu [8] defined a normalized model of a circular-arc graph G. A circular-arc model of G is normalized if every arc that contains another arc in some circulararc model of G, contains it also in , and every pair of arcs that covers the circle in some circular-arc model of G, covers the circle also in . The circular-arc recognition algorithms of [4, 5, 8, 17, 22 ] construct a normalized model, but the linear-time algorithm of [14] does not. The model that our algorithm produces is a normalized model. Normalized models of circular-arc graphs take part in the circular-arc graph isomorphism algorithm of Hsu [8] , and in the recent linear-time algorithm for recognizing circular-arc graphs that admit the Helly property of Lin and Szwarcfiter [12] .. An extended abstract of this paper appeared in [10] . We presented a new linear-time algorithm for recognition circular-arc graphs. While our circular-arc algorithm is simpler than the only previously known linear-time algorithm of McConnell [14] , it is still involved. Our algorithm is based on the three stages of Tucker's algorithm [22] , each performed separately. We can find similarities between the three stages, after finding an odd length induced cycle inḠ, all three stages in a sense refine a partition of the vertices of G (they are various forms of partition refinement, see [18] ). It might be possible to unify the three stages, by defining a partition that can be found in a single step. This partition should take the types of intersections between arcs into account, much like the tree that is used for interval matrices [14] and probe interval graphs [16] . The tree cannot deal with a set of arcs that cover the circle, which makes it difficult to derive a simple algorithm for recognizing circular-arc graphs that uses it.

O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Host-to-Host Congestion Control for TCP Abstract-The Transmission Control Protocol (TCP) carries most Internet traffic, so performance of the Internet depends to a great extent on how well TCP works. Performance characteristics of a particular version of TCP are defined by the congestion control algorithm it employs. This paper presents a survey of various congestion control proposals that preserve the original host-to-host idea of TCP-namely, that neither sender nor receiver relies on any explicit notification from the network. The proposed solutions focus on a variety of problems, starting with the basic problem of eliminating the phenomenon of congestion collapse, and also include the problems of effectively using the available network resources in different types of environments (wired, wireless, high-speed, long-delay, etc.) . In a shared, highly distributed, and heterogeneous environment such as the Internet, effective network use depends not only on how well a single TCPbased application can utilize the network capacity, but also on how well it cooperates with other applications transmitting data through the same network. Our survey shows that over the last 20 years many host-to-host techniques have been developed that address several problems with different levels of reliability and precision. There have been enhancements allowing senders to detect fast packet losses and route changes. Other techniques have the ability to estimate the loss rate, the bottleneck buffer size, and level of congestion. The survey describes each congestion control alternative, its strengths and its weaknesses. Additionally, techniques that are in common use or available for testing are described.

O B I O B O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O


System-level exploration tools for MPSoC designs In this paper, we describe the challenges users face for software development on Systems on Chip (SoC) involving multiple cores. We will briefly review the trends and challenges for MPSoC design and will derive from them the resulting requirements. We will then discuss briefly the required exploration tools and close with an analysis of which type of providers will be able to provide appropriate solutions to the MPSoC challenge. Today users are listening to their favorite information channels as pod casts on devices which used to be their cell phone and now have fallen victim to convergence combining a myriad of consumer functions in a single unit. The next wave of convergence will bring even tougher integration challenges. The electronics industry is at risk not being able to deliver the next generation of convergence. The methodologies to develop cheaper, faster, more power efficient and reliable chips that drive consumer electronics are simply running out of steam. Innovative thinking is required to address design and verification across the multiple specializations hardware, software, architecture and verification. If -and only if -System Design Automation becomes reality then the next generation of convergence in consumer devices can be developed with appropriate time to market and reliability.. Those companies that have the best System Design Automation methodology and tools will be able to beat their competitors both in time to market and in product quality Users will be able to look back at today's methodologies and smile while double-checking that their PDA has computed directions correctly using the integrated GPS and being on the phone with their children discussing their homework, all using one integrated device.

O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


FOX-GA: A Genetic Algorithm for Generating and Analyzing Battlefield Courses of Action This paper describes FOX-GA, a genetic algorithm (GA) that generates and evaluates plans in the complex domain of military maneuver planning. FOX-GA's contributions are to demonstrate an effective application of GA technology to a complex real world planning problem, and to provide an understanding of the properties needed in a GA solution to meet the challenges of decision support in complex domains. Previous obstacles to applying GA technology to maneuver planning include the lack of efficient algorithms for determining the fimess of plans. Detailed simulations would ideally be used to evaluate these plans, but most such simulations typically require several hours to assess a single plan. Since a GA needs to quickly generate and evaluate thousands of plans, these methods are too slow. To solve this problem we developed an efficient evaluator (wargamer) that uses course-grained representations of this problem domain to allow appropriate yet intelligent trade-offs between computational efficiency and accuracy. An additional challenge was that users needed a diverse set of significantly different plan options from which to choose. Typical GA's tend to develop a group of "best" solutions that may be very similar (or identical) to each other. This may not provide users with sufficient choice. We addressed this problem by adding a niching strategy to the selection mechanism to insure diversity in the solution set, providing users with a more satisfactory range of choices. FOX-GA's impact will be in providing decision support to time constrained and cognitively overloaded battlestaff to help them rapidly explore options, create plans, and better cope with the information demands of modern warfare.Knowledge-based niching, military reasoning, course of action planning, computer assisted assessment. This paper describes FOX-GA, a decision support tool that applies a genetic algorithm (GA) to generate and evaluate plans in the complex domain of military maneuver planning. These plans are called cozcrsesofacTion (COAs). The GAuses a niching strategy to ensure that a diverse, yet high quality set of plan options are created. FOX-GA's contributions are to demonstrate an effective application of GA technology to a complex real world planning problem, and to provide an understanding of the properties needed in a GA solution to meet the challenges of decision support in complex domains. The name "Fox" refers to the animal, which is known for being smart and capable of "out-foxing" its enemies through clever planning.. Requirements in the maneuver planning domain include the need to 1) assist users in identifylng the "best" of numerous solution options 2) provide diverse yet high quality options from which users can select and 3 ) rapidly generate and evaluate options under tight time constraints. Many of these these challenges are common to decision support tasks in other domains such as medical decision mahng, manufacturing planning, mechanical design, and architectural design.. FOX-GA is implemented in C++. The great majority of the implementation effort went into the design and construction of the knowledge representation and the logic in the wargamer. Jmplenientation of the GA itself required only 5 percent of the effort.. We will begin with overviews of the domain of battlefield reasoning, challenges in constructing this application, and related work. We will follow it with a detailed explanation of FOX-GRs genetic algorithm, bit string representation, wargaming algorithm, and fitness function. Lastly we will discuss two experiments evaluating Fox's performance, and possible future work. Future work includes projects to generalize FOX to allow for any combination of terrain, enemy forces, and friendly forces; and to construct an intelligent human-computer interface to allow users to view trade-offs offered by selected COAs. Such a display would be presented on a three dimensional graph which would simultaneously show the COAIs ability to 1) capture terrain, 2 ) inflict enemy losses, and 3) to preserve the lives of enemy troops. This would allow the battlestaff to quickly visualize trade-offs. We also plan to enhance FOX-GA3 performance by investigating the use of other GA methods. We will conduct a comparative analysis of FOX-GA and various non-GA search methods such as non-GA evolutionary computing, simulated annealing, and linear programming. We have already done a number of usability studies in which feedback from users was used to identify missing functions and awkward interfaces in FOX-GA. Finally, a user evaluation of the effectiveness of FOX-GA for users of a variety of experience levels is currently underway. Early results indicate that users explore a wider variety of COAs, and in more detail, than they do by hand. FOX-GA demonstrates the effective application of a genetic-algorithm for rapidly generating large numbers of offensive COAs (plans), evaluating their fitness, and identifylng a diverse set of distinctly different high quality options which can be presented to users. In designing this system, it was important adapt the GA methods and simulators used by the system so as to produce solutions that made sense in the context of the domain, and to meet users' domain-based needs for speed, solution diversity and quality, control, and flexibility. Many of these challenges are common to decision support tasks in other domains such as medical decision making, manufacturing planning, mechanical design, and architectural design.. A major difficulty in addressing the challenges of this problem was in satisfymg all of the needs simultaneously: diversity, quality and efficiency. The niching strategy used by the GA proved important in producing solutions that were both diverse and high quality within each niche. An important issue in enabling efficient evaluation of solutions was the development of an efficient wargamer to evaluate COAs rapidly. This gain in efficiency was achieved by first developing the BRS Architecture, a set of course-grained representations of domain concepts that allow appropriate yet intelligent trade-offs to be made between computational efficiency and accuracy of evaluation of fitness.. A comparative evaluation of three search techniques (FOX-GA, FOX-MC, and FOX-MCHC) showed that FOX-GA's genetic algorithm and niching strategy substantially outperformed a more traditional monte carlo search (FOX-MC). The reason for FOX-GA's superior performance was that its genetic algorithm allows the search to take advantage of the structure inherent in the solution space, while the monte carlo search can not take advantage of this structure. A second finding from the comparative evaluation was that although FOX-GA's average "fitness" performance was similar to that of the third search strategy which used a mixed monte carlo and hill climbing (FOX-MCHC), FOX-GAls individual solutions tended to be superior and its solution set more appropriate for the application. Although the hill climbing strategy produced a broader variety of solutions, FOX-GA produced a reasonably diverse set of COAs that were higher in quality, and therefore more useful overall for our application which requires both solution diversity and quality. Finally, we found that the diversity of the solutions produced by FOX-GA could be increased by simply increasing the population size. This flexibility inherent in genetic algorithms provides users with flexibility in the way they can choose to explore the problem space. They can adjust the amount of solution diversity produced to suit their preferences. FOX-GAls impact will be to assist battlestaffs, who are both time constrained and cognitively overloaded, to rapidly explore options, plan, and better cope with the information demands of modern warfare. Additionally, since the challenges in this problem are common to a wide range of complex decision support problems, we believe FOX-GAls approach will be useful in a wide range of domains. 'The goals of the Battlefield Reasoning System (BRS) Architecture (Schlabach, 1997) were to develop a comprehensive set of abstractions and to establish a reasoning framework to support effective and efficient battlefield reasoning for a critical set of tasks, one of which WAS COA generation. T h e architecture provides appropriate abstract representations for concepts such as terrain, order of battle, decision points, and military intelligence propositions. It expands _AL,BM's terrain representations into a systematic and comprehensive set of d a t d structures suitable for exploitation by battlefield reasoning applications. T h e BRS Architecture also identifies a set of reasoning processes that comprise a primary "backbone" for battlefield reasoning coupling upstream processes like COA generation with downstream processes like intelligence analysis. FOX-GA is designed to operate in the context of the BRS Architecture, as one of a suite of coordinated intelligent battlefield reasoning agents.. Two GA4 applications in other military domains investigated decision making in the si~nulation of small unit actions (Porto and Fogel, 1997) and control of Naval Surface-to-Air (Kuchinsh, 1985) . However, the domains of these applications provide only limited insights into the use of GAS for COA generation for battlestaffs.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I I I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Efficient web usage mining process for sequential patterns The tremendous growth in volume of web usage data results in the boost of web mining research with focus on discovering potentially useful knowledge from web usage data. This paper presents a new web usage mining process for finding sequential patterns in web usage data which can be used for predicting the possible next move in browsing sessions for web personalization. This process consists of three main stages: preprocessing web access sequences from the web server log, mining preprocessed web log access sequences by a tree-based algorithm, and predicting web access sequences by using a dynamic clustering-based model. It is designed based on the integration of the dynamic clustering-based Markov model with the Pre-Order Linked WAP-Tree Mining (PLWAP) algorithm to enhance mining performance. The proposed mining process is verified by experiments with promising results. Web logs contain not only simple web usage sessions, but also useful information which can be used to trace web usage patterns in relation to browsing behavior and recommending more relevant web pages to users. By mining the web logs using more advanced data mining techniques, the web usage patterns of users can be discovered. This process is called Web Usage Mining (WUM) which aims to discover potential knowledge hidden in the web browsing behavior of users [1] .. Web Usage Mining algorithms can be classified into many categories, such as clustering, classification, association rules, and sequential pattern discovery. There are two major methods of sequential pattern discovery: deterministic techniques (recording the navigational behavior of the user) and stochastic methods (using "the sequence of web pages that have been visited in order to predict subsequent visits") [1] . This paper focuses on the advanced model-based techniques of the stochastic methods for predicting hidden sequential patterns.. Some applications of WUM are clustering web users based on interesting patterns [2] , mining conceptual link hierarchies from web log files for adaptive website navigation [3] , building the frequent web access sequences using a tree algorithm [4] , predicting web navigations using the Markov model or association rules [5] , [6] . Predicting web access patterns using the Markov model is very interesting in web personalization because of its special features, such as modeling a collection of navigation records, modeling user web navigation behavior, or classifying browsing sessions into different categories. According to J. Borges and M. Levene [7] , the Markov model is a powerful and probabilistic model to estimate the probability of visiting web pages. Each web page is referred to as a state in the Markov model. The N-order Markov model enables us to predict the next visited page affected by the previous N-1 visited pages. Although the predictive probability of the N-order Markov model is higher than the lower-order model, the number of its states exponentially increases more. Because the model complexity is measured by the number of states, the complexity of a higher-order Markov model excessively increases when using it to model a huge number of web pages. Some good news for this problem is that there are usually a significant number of unessential web pages which are included in the Markov model in WUM. Effective filtering of such unessential web pages in the web usage data can resolve the aforementioned complexity problem. Some clustering methods have been applied to filter web pages, such as EM and k-means algorithms in [6] .. Hybrid probabilistic predictive models based on the Markov model, such as the dynamic clustering-based Markov model of J. Borges and M. Levene [7] , have shown improved prediction accuracy over the traditional Markov model. However, the complexity problem of the model has not really been resolved as it still mines the web usage data with a large amount of redundant data.. In order to resolve this complexity problem, this paper proposes a new mining algorithm for WUM based on the Markov model from the stochastic approach. The new feature of this algorithm is that it predicts users' web navigation patterns using the frequent web access sequences extracted from the web logs rather than all web pages. As a result, the complexity of the Markov model can significantly decrease because only frequently accessed web pages are used, resulting in a small number of states. The frequent web access sequences fed into the model can be discovered from web log data by using a tree algorithm. Consequently, we have a set of possible cases of interesting web navigation patterns. The frequent web navigation patterns will then be modeled by a higher-order Markov model to predict the next page accessed by a user.. The rest of this paper is organized as follows. Section 2 reviews some existing research related to sequential pattern discovery. Section 3 explains the new web usage mining algorithm. Section 4 analyzes the performance of the new mining algorithm. Section 5 presents the experimental results of the new algorithm against two sets of web logs taken from two real websites. Section 6 concludes the paper and points out the further work. In the mining step, some data mining techniques, such as the Markov model and the PLWAP algorithm are used to discover the data sets. However, the traditional Markov model might fail to model a web log containing numerous web pages with higher prediction accuracy because the number of states will increase significantly in the high-order Markov model. Even though the dynamic clustering-based Markov model tries to control the state space complexity to achieve second-order accuracy by integrating the K-means algorithm to clone the necessary number of states [7] , it is a waste of time and memory to mine a considerable number of unessential web pages from a web log. It will be more efficient to collect only the essential web pages before applying the Markov model. The PLWAP algorithm is one of the promising candidates for generating the required frequent web access sequences because it not only can accomplish the task, but is also verifiable and more effective compared with Apriori-like algorithms [4] . However, the PLWAP algorithm will stop at the same level as the association rules, and cannot usually meet the demand of a user who needs the most relevant web pages to be recommended for the next navigation step in a recommender system. The probabilistic Markov model, on the other hand, can take the frequent patterns from the PLWAP algorithm and further predict the most interesting and relevant web pages to navigate for a user.. The new web usage mining process proposed in this paper is the combination of the PLWAP algorithm and the dynamic clustering-based Markov model. It inherits the advantages of the PLWAP-tree and the dynamic clustering-based Markov model and overcomes their drawbacks by omitting uninteresting web pages. It can predict the most interesting web access patterns from the users' navigation history. It has been tested using the two web log datasets taken from real websites. The testing results show that the new mining process can considerably improve the drawbacks of Markov model and enhances the performance of PLWAP algorithm.. Another important contribution of this paper is that the resultant web page link graph not only presents all possible links of the web sites, but also predicts the frequently visited links or the frequent web navigations. This makes the new mining process be very 20% 60% useful for web personalization systems using recommender systems. In contrast, a web usage mining process only using the Markov model does not highlight frequent web navigation or user interests.. Our future work will apply the new process to mine the web usage data from an e-government service website to discover the user navigation patterns in order to predict users' interests and preferences in that specific service domain. Sequential pattern mining is considered as an efficient method in web usage mining because of its tolerance of crawling attacks which create fake profiles [8] . In the last decade, many sequence mining algorithms have been proposed in the literature. Ezeife and Lu [4] proposed the PLWAP (tree) algorithm which "builds the frequent header node links of the original WAP-tree in a pre-order fashion and uses the position code of each node to identify the ancestor/descendant relationships between nodes of the tree". This algorithm was successful in extracting the set of all frequent patterns from the web access sequences. Their experiment results show a huge performance boost by predicting frequent navigation patterns of web users.. Regarding the modeling of user web navigation behavior, José Borges [7] proposed a dynamic clustering-based method to increase accuracy of a Markov model in representing a collection of user web navigation sessions. The novelties of this approach are to use the state cloning concept to duplicate states in a way that separates the in-links whose corresponding second-order probabilities diverge, and to use a clustering technique which determines an efficient way to assign in-links with similar secondorder probabilities to the same clone. The algorithm only clones inaccurate states whose first and second-order probabilities diverge or the difference between these probabilities is greater than a certain threshold. The results show that the number of additional states induced by the dynamic clustering method can be controlled through a threshold parameter, so the state space does not increase significantly, and the performance of the method has linear time regarding the size of the model. Hybrid methods might be more robust than standard data mining algorithms such as the Markov models, clustering algorithms, and association rules in current trustworthy recommender systems [9] . Specially, some research has shown that the adaptive mixture of Markov models achieves higher performance for web personalization.. Some approaches to adaptive mixture of Markov models are the combination of the Markov model with clustering techniques or association rules. Liu et al. [5] proposed a clustering method based on a mixture of Markov models to cluster users and capture the sequential relationships hidden in user web navigation histories. The performance of this method is higher than the traditional Markov models, the association rules, or clustering methods. Jalali et al. [10] proposed a clustering approach based on "the graph partitioning for modeling user navigation patterns". Their experimental results could improve the quality of clustering user navigation patterns in WUM systems. Khalil [6] combined three techniques of clustering, association rules and a low-order Markov model. This combination obtains better accuracy of web page access prediction, less state space complexity and fewer generated rules than the single methods.. All of these works attempt to build a more accurate and efficient model for mining the users' web navigation patterns from the web usage data. However, most of them are not concerned with the datasets in the mining process. This may lead to mining unessential or uninteresting data. Considering the methods proposed in [4] and [7] , while mined patterns seem to be separated and have not yet presented relationships between access points in the PLWAP-tree [4] , the Markov model could fail to mine a huge dataset including necessary and unnecessary information [7] .. Compared with the existing work, our work presents an innovative idea of combining the tree algorithm and the Markov model to build a novel mining process. This process can predict users' web navigation patterns more effectively by only using the interesting web access patterns and can resolve the drawbacks of the existing methods, i.e., the complexity problem.

O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Performance of Affordable Neural Network for Back Propagation Learning Cell assembly is one of explanations of information processing in the brain, in which an information is represented by a firing space pattern of a group of plural neurons. On the other hand, effectiveness of neural network has been confirmed in pattern recognition, system control, signal processing, and so on, since the back propagation learning was proposed. In this study, we propose a new network structure with affordable neurons in the hidden layer of the feedforward neural network. Computer simulated results show that the proposed network exhibits a good performance for the back propagation learning. Furthermore, we confirm the proposed network has a good generalization ability. Recently, studies on the brain have been carried out actively on various levels. On the system level, many researchers investigate how the brain operates to realize higher functions such as learning, memory, emotion, and so on. Cell assembly [1] - [4] is one of explanations of information processing in the brain, in which an information is represented by a firing space pattern of a group of plural neurons. This mechanism has been proposed by Hebb who is a psychologist in 1949. Cell assembly could achieve multiple information processing carried out at the same time and parallel dispersion processing of the brain by forming a functional circuit with some neurons according to the necessary processing. Although this concept is impressive, we consider that more complex mechanism is realized in the brain.. On the other hand, Back Propagation (BP) learning [5] is one of engineering applications of neural networks. The BP learning operates with a feedforward neural network which is composed of an input layer, a hidden layer and an output layer, and the effectiveness of the BP learning has been confirmed in pattern recognition, system control, signal processing, and so on [6] - [8] . When the BP learning is applied, the hidden layer plays the essential role to decide the performance of the neural network. Hence, for example, there have been a lot of reports on the number of the neurons in the hidden layer. However, there have not been many studies on changing the structure of the hidden layer to improve the performance on convergence speed or learning efficiency.. In this study, we propose a new network model which is a feedforward neural network with affordable neurons in the hidden layer for more efficient BP learning. In this network, we prepare some extra neurons in the hidden layer. When the network executes the BP learning, all of the neurons in the hidden layer are not used at every updating. Namely, some of the neurons are selected for the learning and the rest of the neurons are deactivated. Computer simulated results show that the proposed network exhibits a good performance for the BP learning on both convergence speed and learning efficiency. Furthermore, we investigate a generalization ability of the proposed network by inputting an unknown pattern after learning. We are interested in such a generalization ability of artificial neural networks, because generalization should be one of the important mechanisms of higher functions in the brain. In this study, we have proposed a new network model which was a feedforward neural network with affordable neurons in the hidden layer for more efficient BP learning. Computer simulated results showed that the proposed network with affordable neurons gained a good performance for the BP learning on both convergence speed and learning efficiency. Furthermore, we confirmed the affordable neural network has a good generalization ability.. In future work, we investigate various kinds of methods to select the affordable neurons in the hidden layer. because chaotic selection methods might be efficient, This is because, we have confirmed that chaotic behavior can obtain good performance to escape out of local minima for combinatorial optimization problems [9] - [11] .

O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Energy-efficient instruction set synthesis for application-specific processors Several techniques have been proposed to enhance the energy-efficiency of ASIPS (Application-Specific Instruction set Processors). While those techniques can reduce the energy consumption with a minimal change in the instruction set (IS), they fail to exploit the opportunity of designing the entire IS from the energy-efficiency perspective. In this paper, we present an energy-efficient IS synthesis technique that can comprehensively reduce the energy-delay product (EDP) of ASIPs through optimal instruction encoding, considering both the instruction bitwidth and the dynamic instruction count. Experimental results with a typical embedded RISC processor show that our technique can generate application-specific IS's that are up to 40% more energy-efficient over the native IS for several application benchmarks. It is well known that CISC IS's (Instruction Sets) are more energyefficient than RISC IS's for the same microarchitecture [I]. However, it is not well known how we can utilize this observation to "This research was conducted while the first two authors were visiting UC Irvine, and supported in part by grants from NSF (CCR-0203813 and CCR-02057 12) and Hitachi Ltd. We also thank members of the UCI EXPRESS compiler team for their assistance.. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission andor a fee. ISLPED'O3, August 25-27, 2003, Seoul, Korea.. Copyright 2003 ACM 1-581 13-682-X/03/0008 ... $5.00. generate more energy-efficient IS's, especially when we are given the freedom to modify an IS on an application (or application domain) basis. With the recent development in soft IP's (Intellectual Properties) and configurable processors, IS customization has become possible and even necessary to make differentiation in today's competitive markets. Nonetheless, previous work on lowpower ASIPs (Application-Specific Instruction Set Processors) has not been so ambitious as to fully exploit the flexibility of ASIPs and redesign the IS from the energy-efficiency perspective.. In this paper, we present an energy-efficient IS synthesis approach for application-specific processors. We optimize IS's under given microarchitectural constraints, as the design change in the datapath may incur significant engineering cost and thus not be desirable. With a fixed microarchitecture, specialization can be made in such areas as instruction encoding, the number of instructions, and the instruction bitwidth, all of which can be considered as instruction encoding in a broad sense. Thus, our objective is to find the best instruction encoding (manifested by RISC vs. CISC) that leads to the maximal energy-efficiency through fewer number of instructions fetched (reducing the instruction memory energy) or fewer number of execution cycles (reducing the processor core energy) or a balance of the two.. One of the critical elements of the proposed energy-efficient IS synthesis is reducing the instruction fetch energy through multiple dimensions of the code volume (the number of instructions fetched multiplied by the instruction bitwidth). While some previous lowpower techniques can also have similar effects of reducing the code volume, only one dimension has typically been considered! Our technique, on the contrary, addresses the multiple factors of the code volume and provides a comprehensive optimization framework for energy-efficient IS's. Also, our scheme generates a single IS as opposed to dual IS's (compressed, uncompressed) as the code compression techniques do; thus, it avoids the problems of dual IS's such as requiring an instruction decompressor (or re-map table) and, for some techniques, having to take care of the changes in the branch target addresses. It should be noted, however, that our IS customization framework assumes minor architectural changes in the datapath, allowed by the given architectural constraints, such as inserting additional muxes in front of functional units, as well as changing the instruction decoder logic. Our experimental re-'For example, low-power instruction compression schemes [2] [3] try to reduce the code volume by focusing only on the bitwidth of frequently occurring binary instruction patterns (thus not changing the number of instructions whether static or dynamic). Likewise, code size reduction techniques [4] aim to reduce the static code size, thus may not be effective for reducing the dynamic instruction count. sults show that our technique can generate application-specific IS's that outperform the native IS of a typical embedded RISC processor, not only in performance but also in energy and the EDP, up to about 40% with each metric.. The rest of the paper is organized as follows. In Section 2 we briefly discuss the previous work for energy-efficient ASIPs and in Section 3 we highlight the key elements of the encoding-oriented IS synthesis for ASIP customization. In Section 4 we derive the contribution of each instruction to the overall energy-efficiency based on an ASIP energy consumption model. We present our experimental results in Section 5 and conclude the paper. from the IS has been suggested [9] . We have presented an energy-efficient IS synthesis technique that is based on an encoding-oriented IS synthesis framework. To comprehensively reduce the code volume, our technique optimizes the instruction encoding, considering both the instruction bitwidth and the dynamic instruction count. To apply the IS synthesis framework for energy-efficiency optimization, we formulated the EDP change due to IS customization and derived the contribution of each complex instruction. The experimental results show that our technique can generate application-specific IS's that outperform the native IS of a typical embedded RISC processor, in performance, energy, and the EDP, up to about 40% with each metric.. Our methodology to improve the IS of a processor from the energy-efficiency perspective focuses on the flexibility of ASIPs at the instruction level. In the future, we plan to investigate customization techniques at higher-levels (e.g., loops, functions), to exploit more opportunity available in future complex systems-onchips. While these techniques may fit well where only minimal changes can be made in the architecture, they are too conservative when a more aggressive IS redesign is preferred to seize the opportunity afforded by configurable processors. Contrastingly, we consider redesigning the IS from the energy-efficiency perspective to better exploit the flexibility of configurable processors. In [lo] , an IS synthesis technique is proposed considering instruction encoding, which is, however, developed for performance improvement and does not consider energy-efficiency. Our energy-efficient IS synthesis is based on this technique, but significantly extends it by considering various optimization goals and their efficacy. For our experiments, we used the MIPS microprocessor architecture [l l] , from which we defined a basic IS annotated with resource and timing information. For deriving complex instruction sets, we used a number of realistic benchmark applications covering multimedia (e.g., h.263 decoder, JPEG encoder), controlintensive (e.g., ADPCM codeddecoder), and cryptography (e.g., DES) domains. The benchmark applications were preprocessed using the EXPRESS retargetable compiler [12] targeting the basic IS to generate preliminary assembly code, which was used for the rest of the IS synthesis process.. While the MIPS architecture has 32-bit instructions, the native IS [ 113 of the MIPS uses the code space of only about 230, meaning that the native IS essentially uses only 30 bits, reserving the rest of the code space for future versions. Since the basic IS for the MIPS was defined with the code space of about 1.42 x IC?', we used the code space of 2" -1.42 x lo8 for the complex IS synthesis.. After the IS synthesis process, the application was recompiled using the same retargetable compiler targeted for the synthesized IS. The execution cycle counts and the instruction fetch counts were obtained through cycle-accurate simulation (at the basic block level) and profiling. We assume that the ID energy is the same for all Is's, as we compare the synthesized IS's with the native IS, which also has many complex instructions.. For the energy consumption estimation, we made the following assumptions on the architecture. We assumed that the instruction memory is large enough to fit each application so that there is no off-chip memory access for instruction fetch. To further simplify, we assumed that there is no instruction cache; thus, instruction memory energy consumption only depends on the number of instruction fetch. For the energy consumption ratio between IF, ID, and EX stages, we assumed that the ratio between EfF + E L and EEX is I: 1 for all applications! Note that this assumption also allows for an easy comparison of the EDP values for different Is's; that is,. where REDP, the cycle count values (respectively) of two Is's.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On the power of Ambainis lower bounds The polynomial method and the Ambainis lower bound (or Alb, for short) method are two main quantum lower bound techniques. While recently Ambainis showed that the polynomial method is not tight, the present paper aims at studying the power and limitation of Alb's. We first use known Alb's to derive (n 1.5 ) lower bounds for BIPARTITENESS, BIPARTITENESS MATCHING and GRAPH MATCHING, in which the lower bound for BIPARTITENESS improves the previous (n) one. We then show that all the three known Ambainis lower bounds have a limitation √ N min{C 0 (f ), C 1 (f )}, Quantum computing has received a great deal of attention in the last decade because of the potentially high speedup over classical computation. Among others, the query model is extensively studied, partly because it is a natural quantum analog of classical decision tree complexity, and partly because many known quantum algorithms fall into this framework [13, 14, 16, 18, 19, 28, 29, 32] . In the query model, the input is accessed by querying an oracle, and the goal is to minimize the number of queries made. We are most interested in doubleside bounded-error computation, where the output is correct with probability at least 2/3 for all inputs. We use Q 2 (f ) to denote minimal number of queries for computing f with double sided bound-error. For more details on the quantum query model, we refer to [6, 15] as excellent surveys.. Two main lower bound techniques for Q 2 (f ) are the polynomial method by Beals et al. [11] and Ambainis lower bounds [4, 5] , the latter of which is also called quantum adversary method. Many lower bounds have recently been proven by applying the polynomial method [1, 11, 22, 24, 27] and Ambainis lower bounds [2, 4, 5, 17, 31] . Recently, Aaronson even uses Ambainis lower bound technique to achieve lower bounds for some classical problems [2] . Given the usefulness of the two methods, it is interesting to know how tight they are. In a recent work [5] , Ambainis proves that polynomial method is not tight, by showing a function with polynomial degree M and quantum query complexity (M 1.321... ). So a natural question is the power of Ambainis lower bounds. We show that all known Ambainis lower bounds are not tight either, among other results.. There are several known versions of Ambainis lower bounds, among which the three Ambainis theorems are widely used partly because they have simple forms and are thus easy to use. The first two Alb's are given in [4] as follows. In the open problems part of [5] , Ambainis mentions the √ C 0 (f )C 1 (f ) limitation of Alb 1 , and asks for new quantum lower bound techniques higher than. is not shown in [5] whether Alb 2 and Alb 3 are also bounded by the. limitation for total function f , and actually even whether. was still open at the time, according to a private communication between Ambainis and us.. Recently Spalek and Szegedy independently show in [30] that the all quantum adversary methods, including Alb 3 by Ambainis [5] , Alb 4 in an earlier version of the present paper [33] , and another quantum adversary method proposed in [10] , are actually equivalent. Using this fact, they gave a simple proof that all of them cannot prove quantum lower bounds better than ( N · C − (f )) for general function and not better than (. is also derived by Laplante and Magniez by using Kolmogorov complexity in [20] .And the (n 1.5 ) lower bound for Matching is independently obtained by Berzina, Dubrovsky, Freivalds, Lace and Scegulnaja in [12] , and the same lower bound for Bipartiteness is independently obtained by Durr (cited in [20] ).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Robust Learning with Missing Data Abstract. This paper introduces a new method, called the robust Bayesian estimator (RBE), to learn conditional probability distributions from incomplete data sets. The intuition behind the RBE is that, when no information about the pattern of missing data is available, an incomplete database constrains the set of all possible estimates and this paper provides a characterization of these constraints. An experimental comparison with two popular methods to estimate conditional probability distributions from incomplete data-Gibbs sampling and the EM algorithm-shows a gain in robustness. An application of the RBE to quantify a naive Bayesian classifier from an incomplete data set illustrates its practical relevance. The Bayesian estimation of conditional probabilities from a data is a task relevant to a variety of machine learning applications, such as classification (Langley, Iba, & Thompson, 1992) and clustering (Cheeseman & Stutz, 1996) . When no entry is missing in the database, these conditional probabilities can be efficiently estimated using standard Bayesian analysis (Good, 1968) . Unfortunately, when the database is incomplete, i.e., some entries are reported as unknown, the simplicity and efficiency of this analysis are lost. Exact Bayesian analysis requires that one estimates the conditional probability distributions in each database that can be completed by replacing the missing entries with some value and then computes their average estimate. As the number of the completed databases increases exponentially with the number of missing entries, this exact analysis is computationally intractable.. During the past few years, several methods have been proposed for learning conditional probabilities from incomplete data sets. The two most popular methods are the expectation maximization algorithm (Dempster, Laird, & Rubin, 1977) and Gibbs sampling (Geman & Geman, 1984) . Both methods make the simplifying assumption that data are missing at random (Rubin, 1976) . Under this assumption, the probability that an entry is not reported is independent of the missing entries in the data set and, in this situation, the missing values can be inferred from the available data. However, there is no way to verify this assumption on a database and, when this assumption is violated, all these methods can suffer of a dramatic decrease in accuracy (Spiegelhalter & Cowell, 1992) . This situation motivated the recent development of a deterministic method, called Bound and Collapse (Ramoni & Sebastiani, 1998) , that does not rely, per se, on a particular assumption about the missing data mechanism but allows the user to specify one, including the missing at random assumption. However, it still requires the specification of a particular missing data pattern, and this information may not be readily available. Approximate methods and simplifying assumptions make the task of learning from incomplete data feasible, but they prompt one to question the reliability of the estimates obtained in this way.. A typical solution to this problem is to measure this reliability by estimating the conditional probabilities under different assumptions about the missing data mechanism and by assessing the sensitivity of the estimates to these assumptions. A drawback of this approach is that each missing data mechanism explored requires a new estimation process and the choice of the mechanisms to consider in this sensitivity analysis is left entirely to the analyst. The rationale behind the approach presented in this paper is closely related to this idea and it can be regarded as an automated method for sensitivity analysis.. This paper introduces the robust Bayesian estimator (RBE) to learn conditional probability distributions from incomplete data sets without making any assumption about the missing data mechanism. The major feature of the RBE is to produce probability estimates that are robust with respect to different types of missing data. This robustness is achieved by providing probability intervals containing the estimates that can be learned from all completed data sets. The width of these intervals is a monotonically increasing function of the information available in the data set and thus provides a measure of the information conveyed by the data. We will focus on Bayesian networks, although the RBE can be used for the general task of learning conditional probabilities. During the past few years, there has been an increasing interest in algorithms that propagate probability intervals during inference in Bayesian networks (Fertig & Breese, 1993; Ramoni, 1995) but, to our knowledge, no effort has been made to apply the same interval-based approach to the task of learning such networks. The method presented in this paper can be therefore regarded as the learning counterpart of this research on reasoning methods, and we will show how the networks learned with the RBE can be used for classification and inference by means of these propagation methods.. The reminder of this paper describes our approach. Section 2 establishes some notation and reviews the background and motivation of the research. Section 3 describes the theoretical framework of the RBE and the use of probability intervals for inference, while Section 4 outlines the algorithms required for the implementation. Section 5 compares the RBE to expectation and maximization and Gibbs sampling in a controlled experiment, and Section 6 applies the RBE to an incomplete data set in a classification task. Real-world data sets are often incomplete, and machine learning methods must be able to handle incomplete data before they can be widely applied. A key issue for such learning methods is the reliability of the knowledge they generate. This paper introduced a robust method, the RBE, to estimate conditional probabilities. Compared to traditional Bayesian estimation methods, the RBE relies on a new strategy: rather than guessing the value of missing data on the basis of the available information, it bounds the set of all estimates consistent with the data. The estimates computed by the RBE are, therefore, probability intervals containing all possible estimates that could be computed from the consistent completions of the database and, as such, they are robust with respect to the distribution of missing data. Hence, one feature of the RBE is to provide an automated method to analytically perform sensitivity analysis, at a low computational cost, with respect to different assumptions on the missing data mechanism.. Efficient sensitivity analysis is not, however, the only feature of the RBE. The probability intervals it computes can be used, as they are, to draw robust inferences. Furthermore, these intervals provide a measure of the information conveyed by the data and can be used to assess the reliability of the point-valued estimates computed by other learning methods, like EM and Gibbs sampling, that are based on specific assumptions about the missing data mechanism. The experimental evaluation presented in this paper showed that when data are missing at random or completely at random, both EM and Gibbs sampling can exploit the information provided by the observed data to return more precise inferences, although the predictive accuracy of the RBE appears to be superior. However, when the deletion process produces data that are informatively missing, the RBE is more reliable in terms of both accuracy and precision.. The application described in Section 6 showed another potential use of the method introduced in this paper. Interval-based classification breaks the database in two sets: the set of those cases that can be classified independently of any assumption made about the missing data mechanism, and the set of cases that can be classified only by assuming a particular mechanism. One can then use the accuracy and coverage of the interval-based classification to derive an estimate of the classification accuracy obtained under the assumed missing data mechanism and therefore evaluate the impact of this assumption on the overall classification task. Interval-based classification seems to be a promising area of application of the RBE, and a systematic investigation of this problem could lead to the development of a robust Bayes classifier, able to couple the computational efficiency of standard naive Bayes with the ability to handle incomplete databases with no assumption about the missing data mechanism. A more ambitious avenue would involve applying the same interval-based approach to other machine learning tasks, such as clustering, dependency discovery, and the identification of hidden variables.

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O