Conditional Value-at-Risk in Stochastic Programs with Mixed-Integer Recourse Abstract. In classical two-stage stochastic programming the expected value of the total costs is minimized. Recently, mean-risk models -studied in mathematical finance for several decades -have attracted attention in stochastic programming. We consider Conditional Value-at-Risk as risk measure in the framework of two-stage stochastic integer programming. The paper addresses structure, stability, and algorithms for this class of models. In particular, we study continuity properties of the objective function, both with respect to the first-stage decisions and the integrating probability measure. Further, we present an explicit mixed-integer linear programming formulation of the problem when the probability distribution is discrete and finite. Finally, a solution algorithm based on Lagrangean relaxation of nonanticipativity is proposed. Most real-world problems in operations research involve uncertain data. Thus, finding optimal decisions turns into selecting a "best" random variable. Then the question comes up which criteria to use for the selection. Quickly, the matter of risk aversion becomes an issue. Further, for realistic modeling integer variables are often helpful and sometimes inevitable. This paper suggests a way of how to make such decisions by using Conditional Value-at-Risk as risk measure in the framework of recourse stochastic integer programming.. Throughout the paper, we impose a cost minimization framework. Consider a family of real random cost variables {Z(x, ω)} x∈X ⊆ Z on a probability space (Ω, A, IP ). We want to decide on the variable x and so on the corresponding random variable by comparing certain scalar characteristics of the random variables, namely by so-called mean-risk models min x∈X IE{Z(x, ω)} + ρR{Z(x, ω)}. where IE : Z −→ R denotes the expected value, R : Z −→ R a risk measure, Z the space of all real random cost variables Z : Ω −→ R satisfying IE{|Z(ω)|} < ∞, and ρ > 0 a suitable weight factor. This paper deals with a risk measure called Conditional Value-at-Risk applied to random variables occurring in recourse stochastic integer programming. These random variables are essentially defined by value functions of mixed-integer linear programs being discontinuous and nonconvex, such that, in particular, convexity of the objectives in the mean-risk model is not given. Therefore, for the sake of applicability to real-world problems, it is inevitable to choose a risk measure such that, despite the poor properties of the random variables, the resulting stochastic integer programs are structurally sound and amenable to algorithmic treatment. In the following we will show that Conditional Value-at-Risk is a risk measure that satisfies these requirements.. The mean-risk model (1) aims at minimizing the weighted sum of two competing objectives. Viewed from a more general perspective, it is a scalarization of the multiobjective optimization problem min x∈X IE{Z(x, ω)}, R{Z(x, ω)} .. For an introduction to multiobjective optimization we refer to [7, 20] . An accepted notion of optimality in multiobjective optimization is efficiency. A point x ∈ X is called efficient for (2) if there is no other point x ∈ X such that IE{Z(x, ω)} IE{Z(x, ω)} and R{Z(x, ω)} R{Z(x, ω)}, with at least one strict inequality. The set of all efficient points is named efficient frontier. Every optimal solution to the mean-risk model (1) with a weight factor ρ > 0 is an efficient point, a so-called supported efficient point. Due to the lacking convexity of our objective functions, not all efficient points are supported, and thus cannot be computed by solving scalarizations (1) . However, solving the mean-risk model (1) for various values of ρ > 0 has the capability to trace the supported part of the efficient frontier. In Section 5 a discrete tracing method is described and carried out for a real-life optimization problem.. The relation of stochastic dominance, one of the fundamental concepts in decision theory, introduces a partial order in the space of real random variables. This provides a basis for selecting "best" members from families of random variables. Ogryczak and Ruszczyński have studied mean-risk models and their consistency with the multiobjective criteria induced by stochastic dominance, see [23, 24] . Artzner et al., cf. [4] , proposed axiomatic properties of risk measures, calling risk measures fulfilling these properties coherent. Conditional Value-atRisk is a coherent risk measure, cf. [2, 4, 30] , and the corresponding mean-risk model is consistent with the second degree stochastic dominance relation, cf. [24] .. For controlling the variability of the costs, the first risk measure that comes into mind is the variance, cf. [21] . However, the variance has several drawbacks. In general, it is neither consistent with the stochastic dominance relation nor coherent. Further, the square and the mentioned discontinuities may lead to an objective which is not even lower semicontinuous, such that (1) , with compact X, may have a finite infimum which is not attained, cf. [33] .. In [33] another risk measure, the Excess Probability, see Section 2, has been analyzed. It is consistent with the first degree stochastic dominance relation. The present paper shall be seen as a continuation of that research, analyzing a risk measure with a higher degree of consistency with the stochastic dominance relation.. Section 2 gives an introduction to Conditional Value-at-Risk. In Section 3 we extend the traditional modeling in two-stage stochastic integer programming towards risk aversion and formulate a mean-risk model with Conditional Valueat-Risk. Section 4 analyzes structure and stability of the risk measure on the space of the special random variables in question, defined essentially by valuefunctions of mixed-integer programs with random right-hand sides. An explicit mixed-integer linear programming formulation of the model when the probability distribution is discrete and finite is presented in Section 5. Moreover, a solution algorithm is proposed, and finally, we report on some first numerical experiments.

B I O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Empirical analysis of database server scalability using an N-tier benchmark with read-intensive workload The performance evaluation of database servers in N-tier applications is a serious challenge due to requirements such as non-stationary complex workloads and global consistency management when replicating database servers. We conducted an experimental evaluation of database server scalability and bottleneck identification in N-tier applications using the RUBBoS benchmark. Our experiments are comprised of a full scale-out mesh with up to nine database servers and three application servers. Additionally, the fourtier system was run in a variety of configurations, including two database management systems (MySQL and PostgreSQL), two hardware node types (normal and low-cost), and two database replication techniques (C-JDBC and MySQL Cluster). In this paper we present the analysis of results generated with a read-intensive interaction pattern (browse-only workload) in the client emulator. These empirical data can be divided into two kinds. First, for a relatively small number of servers, we find simple hardware resource bottlenecks. Consequently, system throughput increases with an increasing number of database (and application) servers. Second, when sufficient hardware resources are available, non-obvious database related bottlenecks have been found that limit system throughput. While the first kind of bottlenecks shows that there are similarities between database and application/web server scalability, the second kind of bottlenecks shows that database servers have significantly higher sophistication and complexity that require in-depth evaluation and analysis. Scalability of N-tier applications in general and database servers in particular present serious challenges to both academia and industry. This is largely due to requirements such as non-stationary workloads and dependencies created by requests that are passed between web servers, application servers, and database servers. Some of these challenges have been previously referred to as "gaps between theory and practice" [10] in database replication. We attempt to start bridging these gaps with a large-scale experimental evaluation of database server scalability using a N-tier application benchmark (RUBBoS [6] ). For our study we have collected more than 500GB of data in over 6,000 experiments. These experiments cover scale-out scenarios with up to nine database servers and three application servers. The configurations were varied using two relational database management systems (MySQL and PostgreSQL), two database replication techniques (C-JDBC and MySQL Cluster), workloads ranging from 1,000 to 13,000 concurrent users, and two different hardware node types.. The initial analysis of our extensive data produced several interesting findings. First, we documented detailed nodelevel bottleneck migration patterns among the database, application, and clustering middleware tiers when workload (a) Software setup. (c) Sample C-JDBC topology (1/2/1/2L). and number of servers increased gradually. These bottlenecks were correlated with the overall system performance and used to explain the observed characteristics. Second, the explicit comparison of performance differences between varying system configurations yielded concrete configuration planning insights, which have escaped analytical approaches so far. Third, the identification of non-obvious bottlenecks showed "migratory bottleneck phenomena" that can arise in real systems and result in surprising performance effects. The paper makes two main contributions. First, using an automated experiment creation and management system, we collected a significant amount of data on database server performance in N-tier applications. We present an experimental evaluation of database server scalability in a read-intensive scenario (i.e., setup, description, GBs of trace data, and runtime statistics). Second, our analysis of this data set shows interesting bottlenecks and their characteristics as mentioned above, which are potentially widely applicable. The detailed bottleneck migration patterns and explicit performance comparisons, for example, can be useful for datacenter administrators that manage N-tier systems by providing measured results on concrete N-tier system performance and resource utilization.. The remainder of this paper is structured as follows. Section 2 outlines experimental setup and methods. Section 3 presents the analysis of our data on database scale-out through database replication. In Section 4 we detail an interesting scenario that shows how our methodology can be used to uncover unexpected bottlenecks with surprising performance effects. Related work is summarized in Section 5, and Section 6 concludes the paper. We used techniques and tools developed for automated experiment management, including automatically generated scripts for running experiments and collecting measurement data, to study the RUBBoS N-tier application benchmark with read-intensive workload. The data collected over a wide range of settings and system configurations was analyzed using statistical tools such as kernel density estimation, identifying hardware resource bottlenecks, and producing detailed bottleneck migration patterns as the number of servers and workload increase gradually. When there are sufficient hardware nodes, non-obvious bottlenecks were found through careful hypothesis generation, followed by confirmation through data analysis. An example is the finding of temporary bottlenecks caused by PostgreSQL's differentiated processing of SQL queries with more than nine times self-joins, which seems to take several orders of magnitude more CPU.. Our results suggest that further study (e.g., through refined analysis of our data or new experimental data) may bring new understanding of database server scale-out in Ntier applications. In addition, techniques used in the "tuning" of database management systems to improve performance may be useful in the identification of internal bottlenecks that can take a critical role in achieving service level objectives in mission critical applications.. Our current work involves a similar analysis as presented in this paper on data generated with read/write workload. Our midterm goal is to compare the presented middlewarebased replication strategy to DBMS-internal clustering. Cloud computing, data center provisioning, server consolidation, and virtualization have become ubiquitous terminology in the times of ever-growing complexity in large-scale computer systems. However, database replication continues to fall short of real world user needs, which leads to continuously emerging new solutions and real world database clusters to be small (i.e., less than 5 replicas) [10] . In this paper we address this shortcoming explicitly. This work extends our previous approaches to empirical system analysis and bottleneck detection. Initially, we have analyzed our data for complex bottleneck phenomena. We have introduced terminology that explicitly distinguishes single-bottlenecks and multi-bottlenecks [19] . We have also investigates different ways of evaluating the economical implications of largescale empirical system analysis. With the help of this investigation we have developed a configuration planning tool (i.e., CloudXplor) that is able to facilitate iterative and interactive configuration planning in clouds [18] .. The work that is closest to this paper follows a similar observational approach to system analysis and evaluation [17] . While the context and framing of the two publications are closely related, the contributions are fundamentally different. More concretely, the evaluated datasets are completely disjunct. Where this paper focuses solely on read-only workload, our previous work [17] was exclusively concerned with workload that exhibits read/write access patterns to the database. Consequently, the findings, conclusions, and contributions are independent of each other.. There exist many popular solutions to database replica-tion (i.e., IBM DB2 DataPropagator, replication in Microsoft SQL Server, Sybase Replication Server, Oracle Streams, replication in MySQL). Among others there are academic prototypes that use master-slave replication such as Ganymed [21] and Slony-I [2] . In this work we focus on multi-master architectures, which are used in commercial products such as MySQL Cluster and DB2 Integrated Cluster. Academic multi-master prototypes are C-JDBC [11] , which is used in this work or Tashkent [13] . Further efforts focus on adaptive middleware to achieve performance adaptation in case of workload variations [20] . The evaluation of database replication techniques often falls short of real representability. There are two common approaches to evaluation with benchmarks. Performance is either tested through microbenchmarks or through webbased distributed benchmark applications such as TPC-W, RUBiS, and RUBBoS [8] . These benchmark applications are modeled on real applications (e.g., Slashdot or Ebay.com), offering real production system significance. Therefore this approach has found a growing group of advocates ( [11-13, 21, 22] ).

O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Test Data Compression Using Selective Encoding of Scan Slices Abstract-We present a selective encoding method that reduces test data volume and test application time for scan testing of Intellectual Property (IP) cores. This method encodes the slices of test data that are fed to the scan chains in every clock cycle. To drive scan chains, we use only tester channels, where = log 2 ( +1) +2. In the best case, we can achieve compression by a factor of using only one tester clock cycle per slice. We derive a sufficient condition on the distribution of care bits that allows us to achieve the best-case compression. We also derive a probabilistic lower bound on the compression for a given care-bit density. Unlike popular compression methods such as Embedded Deterministic Test (EDT), the proposed approach is suitable for IP cores because it does not require structural information for fault simulation, dynamic compaction, or interleaved test generation. The on-chip decoder is small, independent of the circuit under test and the test set, and it can be shared between different circuits. We present compression results for a number of industrial circuits and compare our results to other recent compression methods targeted at IP cores. T EST DATA volume is now recognized as a major contributor to the cost of manufacturing testing of integrated circuits (ICs) [1] - [4] . Recent growth in design complexity and the integration of embedded cores in system-on-chip (SoC) ICs has led to a tremendous growth in test data volume; industry experts predict that this trend will continue over the next few years [5] . For example, the 2005 ITRS document predicted that the test data volume for integrated circuits will be as much as 30 times larger in 2010 than in 2005 [6] .. High test data volume leads to an increase in testing time. In addition, high test data volume may also exceed the limited memory depth of automatic test equipment (ATE). Multiple ATE reloads are time consuming because data transfers from a workstation to the ATE hard disk, or from the ATE hard disk to ATE channels are relatively slow; the upload time ranges from tens of minutes to hours [7] . Test application time for scan Manuscript testing can be reduced by using a large number of internal scan chains. However, the number of ATE channels that can directly drive scan chains is limited due to pin count constraints. Logic built-in self-test (LBIST) [8] has been proposed as a solution for alleviating these problems. LBIST reduces dependencies on expensive ATEs and allows precomputed test sets to be embedded in test sequences generated by BIST hardware to target random pattern resistant faults. However, the memory required to store the top-up patterns for LBIST can exceed 30% of the memory used in a conventional automatic test pattern generation (ATPG) approach [8] . With increasing circuit complexity, the storage of an extensive set of ATPG patterns on-chip becomes prohibitive [1] . Moreover, BIST can be applied directly to SoC designs only if the embedded cores are BIST-ready; considerable redesign may be necessary for incorporating BIST in cores that are not BIST-ready.. Test data compression offers a promising solution to the problem of increasing test data volume. A test set for the circuit under test (CUT) is compressed to a much smaller data set , which is stored in ATE memory. An on-chip decoder is used to generate from during test application. A popular class of compression schemes relies on the use of a linear decompressor. These techniques are based on LFSR reseeding [9] - [12] and combinational linear expansion networks consisting of XOR gates [13] - [15] , and they have been implemented in commercial tools such as TestKompress from Mentor Graphics [1] , SmartBIST from IBM/Cadence [3] , and DBIST from Synopsys [16] . These compression schemes exploit the fact that scan test vectors typically contain a large fraction of unspecified bits even after compaction. However, the on-chip decoders for these techniques are specific to the test set, which necessitates decompressor redesign if the test set changes during design iterations. Finally, in order to achieve the best compression, these methods resort to fault simulation and test generation. As a result, they are less suitable for test reuse in SoC designs based on Intellectual Property (IP) cores.. Another category of compression methods uses statistical coding, variants of run-length coding, dictionary-based coding, and hybrid techniques [17] - [22] . These methods exploit the regularity inherent in test data to achieve high compression. However, most of these schemes target single scan chains and they require synchronization between the ATE and CUT.. We present a selective encoding method that reduces test data volume and test application time for the scan testing of IP cores. This method encodes the slices of test data that are fed to the scan chains in every clock cycle. Unlike many prior methods, the proposed method does not encode all the specified (0 s and 1 s) and unspecified (don't care) bits in a slice. For example, if a slice contains more 1's than 0's, only the 0's are encoded and all don't cares are mapped to 1. We use only tester channels, where , to drive scan chains. The logarithmic reduction in the number of tester channels allows us to use a large number of internal scan chains, thereby reducing test application time significantly. In the best case, we can achieve compression by a factor of using only one tester clock cycle per slice. We derive a sufficient condition on the distribution of care bits that allows us to achieve the best-case compression. We also present a probabilistic lower bound on the compression for a given care bit density.. The proposed technique does not require dedicated test pins for each core in an SoC. If cores are tested sequentially, only one common test interface is needed. If some cores are tested in parallel, then they can together be viewed as a larger core with more scan chains. For example, if five cores are tested in parallel, and each core has 255 scan chains, it is equivalent to one core with 255 5 scan chains. The proposed technique will therefore only require 13 pins. Meanwhile, the on-chip decoder must be modified to handle the scan-in and capture for the five different cores.. The pattern decompression is of the continuous-flow type because no complex handshakes are required between the tester and the chip, and there is no need to introduce tester stall cycles. Unlike popular compression methods, such as Embedded Deterministic Test (EDT) [1] , the proposed approach is suitable for IP cores because it does not require structural information for fault simulation, dynamic compaction, or interleaved test generation. The on-chip decoder is small, independent of the circuit under test and the test set. We present compression results for a number of industrial circuits, and compare our results to other recent compression methods targeted at IP cores.. The steady increase in clock frequencies over recent years has led to designs with a small number of gates between latches, or between latches and input/output (I/O) pins [23] . As a result, logic circuits today have very short combinational logic depth, and many logic cones with very little overlap. This is in contrast to older circuits such as the ISCAS'85 benchmarks that tend to have a smaller number of overlapping logic cones. A consequence of the shallow logic depth is that test patterns in present day circuits contain many don't care bits; e.g., it has been reported recently that test sets for industrial circuits contain only 1%-5% care bits [24] . After a desired stuck-at coverage is obtained (using methods such as dynamic compaction and fault grading to reduce pattern count), a commercial test pattern generator typically uses random fill to increase the likelihood of surreptitious detection of unmodeled faults. However, if the test sets for the cores are delivered with the don't care bits to the system integrator, an appropriate compression method can be used at the system level to reduce test data volume and testing time. This imposes no additional burden on the core vendor. Unmodeled faults can still be detected if the compression method does not arbitrarily map all don't cares to either 1's or 0's.. We do not address the problem of output compaction in this paper. The proposed input compression method can be used with recent output compaction methods such as X-compact [2] , convolutional compaction [25] , and i-compact [26] to further reduce test data volume.. The rest of this paper is organized as follows. The details of the proposed approach are described in Section II. Section III presents the decompression architecture and Section IV presents compression results for industrial circuits. II. PROPOSED APPROACH As shown in Fig. 1 , the proposed approach encodes the slices of test data (scan slices) that are fed to the internal scan chains. The on-chip decoder contains an -bit buffer, and it manipulates the contents of the buffer according to the compressed data that it receives. After all the compressed data for a single slice is received, the data in the buffer is delivered to the scan chains.. Each slice is encoded as a series of -bit slice-codes, where , , and is the number of internal scan chains in the CUT. The number of slice codes needed to encode a given slice depends on the distribution of 1's, 0's, and don't cares in the slice. As shown in Fig. 2 , the first two bits of a slice-code form the control-code that determines how the following bits, referred to as the data-code, are interpreted.. As described in Section I, the proposed approach only encodes a subset of the specified bits in a slice. First, the encoding procedure examines the slice and determines the number of 0-and 1-valued bits. If there are more 1's (0's) than 0's (1's), then all X's in this slice are mapped to 1 (0), and only 0's (1's) are encoded. The 0's (1's) are referred to as target-symbols and are encoded into data-codes in two modes: 1) single-bit-mode and 2) group-copy-mode.. In the single-bit-mode, each bit in a slice is indexed from 0 to . A target-symbol is represented by a data-code that takes the value of its index. For example, to encode the slice XXX10000, the X's are mapped to 0 and the only target symbol of 1 at bit position 3 is encoded as 0011. In this mode, each target symbol in a slice is encoded as a single slice code. Obviously, if there are many target symbols that are adjacent or near to each other, it is inefficient to encode each of them using separate slice codes. Hence, the group-copy mode has been designed to increase compression efficiency.. In the group-copy mode, an -bit slice is divided into groups, and each group (with the possible exception of the last group) is -bits wide, as shown in Fig. 3 . The groups are numbered from 0 to , with the first group (referred to as group 0) containing bits 0 to and so on. If a given group contains more than one target symbol, then the group-copy mode is used and the entire group is copied to a data code. Two data codes are needed to encode a group. The first data code specifies the index of the first bit of the group, and the second data code contains the actual data. In the group- copy mode, don't cares can be randomly filled instead of being mapped to 0 or 1 by the compression scheme.. For example, let and , i.e., each slice is 31-bits wide and consists of six 5-bit groups and one 1-bit group. To encode the slice X1110_00000_XX00X_XXXX0_00000_00XXX_0 (the slice is shown separated by underscores into 5-bit groups), only the three 1's in group 0 are encoded. Since group 0 has more than one target symbol, the group-copy mode is used, yielding two data codes 00000 and X1110. The first data code refers to bit 0 (first bit of group 0), and the second data code carries the content of group 0.. The concept of a group is relevant only for the group-copymode. If only the single-bit mode is used, we do not need to consider groups at all. The underlying idea for the proposed method is: 1) only encode target symbols; 2) the bit-index of the target symbol is encoded in the slice code for proper decompression; 3) to further improve encoding efficiency, the group-copy mode is introduced, in which the scan slice is divided into groups.. Since data codes are used in both modes, control codes are needed to avoid ambiguity. Control codes 00, 01, and 10 indicate that the current slice code is a single-bit mode slice code, and control code 11 indicates the current slice code is a groupcopy mode slice code. The first slice code for any scan slice is always a single-bit mode code, and its control code can only be 00 or 01. Hence, control codes 00 and 01 are referred to as initial-control codes and they indicate the start of a new slice. Control code 00 (01) indicates that all X's in the current scan slice should be mapped to 0 (1), and the target symbol for the whole slice is 1 (0). Except the first slice code, all other single-bit mode slice codes for the same scan slice can only have the control code 10, i.e., simply setting the bit specified by the associated data code to the target symbol of this slice. Table I shows a complete example to further illustrate how scan slices are encoded into slice codes. In this example, and . In Case 1, the slice contains only one target symbol (1), and is encoded into one slice-code using the single-bit mode. In Case 2, the slice contains no target symbols.. To handle this special case, we introduce a dummy data code whose value is . Since the bits in the scan slice is indexed from 0 to , a data code with value implies that no bit should be set to the target symbol.. In Case 3 and Case 4, the same slice is encoded with the group-copy mode turned off and on, respectively. In Case 3, since the slice contains six target symbols, six single-bit mode slice codes are required to encode it, resulting in negative compression, which is undesirable. In Case 4, however, only four slice codes are needed: three group-copy mode slice codes to encode groups 0 and 1, and one single-bit mode slice code to encode the 1 at bit 30.. As discussed before, two group-copy mode slice codes are needed to encode a given group, one to indicate the starting bit of the group, and the other to carry the content of the group. However, to further improve compression efficiency, if adjacent groups are to be encoded using the group-copy mode, they can be merged together. This optimization procedure is referred to as group merging. As shown in Case 4, since groups 0 and 1 are adjacent, instead of using four slice codes 1100000, 11X1100, 1100101, and 1101101 to encode the two groups, only three slice codes are needed. The code that indicates the starting bit of group 1 (1100101) is omitted. Therefore, to encode adjacent groups, group-copy mode slice codes are needed, with the first slice code indicating the starting bit of the first group, and the other group-copy mode slice codes carrying the contents of the groups. With group merging, the number of consecutive group-copy mode slice codes is no longer fixed. To avoid ambiguity, two series of consecutive group-copy mode slice codes that encode two series of adjacent groups must be interleaved by one single-bit mode slice code.. Cases 3 and 4 also show that the group-copy mode should be used whenever possible. Since two group-copy mode slice codes are sufficient to encode any -bit group, if a group contains more than one target symbol, it should be encoded using the group-copy mode.. The encoding procedure is summarized in Fig. 4 . In Step 1), each test vector is divided into a series of slices. We then encode each slice as a series of slice codes. In Steps 3)-7), the numbers of 0's and 1's are calculated, and the target symbol as well as the control code of the first slice code are set. The first slice code of each slice must contain an initial-control code (00 or 01).. Steps 8)-14) encode all the groups of a slice. For each group of a slice, if it contains more than one target symbol, it is encoded using the group-copy mode; otherwise, it is encoded using the single-bit mode.. Once all groups have been encoded, the slice code generation step (Step 15) becomes straightforward. It first merges group-copy mode slice codes that correspond to adjacent groups, and then appropriately interleaves group-copy mode slice codes with single-bit mode slice codes. It also ensures that the first slice code is a single-bit mode code.. As can be seen from Fig. 4 , the encoding procedure consists of two nested loops: the outer loop is for scan slices, and the inner loop is for groups of a given slice. Hence, its time complexity is , where is the total data volume of the test set in bits.

B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Incremental incrementally compacting garbage collection Abstract : A mixed-strategy garbage collection algorithm is presented, which combines mark-and-sweep an d copy collection . The intent is to benefit from the compacting and linearizing properties of copy collectio n without losing computational use of half the memory . The stop-and-collect version of the algorithm is a simpl e and cheap technique to fight memory fragmentation . The collection strategy may be dynamically adapted t o minimize the cost of collection, according to the amount of memory actually accessed by the computing process . The parallel version of the algorithm is to our knowledge the only parallel compacting collector for varisized cells, that leaves most (more than half) of the memory available for the computing process .The purpose of techniques known as garbage collectio n or scavenging is to provide automatic reclamation o f memory cells that have become useless to a running program because it can no longer access them . Two families o f garbage collection algorithms have been developed . Th e first one is based on an on-the-fly reference count scheme Permission to copy without fee all or part of this material is granted provide d that the copies are not made or distributed for direct commercial advantage , the ACM copyright notice and the title of the publication and its date appear , and notice is given that copying is by permission of the Association fo r Computing Machinery . To copy otherwise, or to republish, requires a fee and / or specific permission .©1987 ACM 0-89791-235-7/87/0006/0253 751In practice specific aspects have to be carefully worked out , especially to avoid explicit synchronization for parallel algorithms, or to ensure real-time behavior by locking the pace of the collector and the mutator (the user's program in th e terminology of [DijLMSS 78]) . The issues we wish to address concern single space garbage collection and compaction, either in stop-and-collect or in non-stop mode . We briefly mention relations with generation collectors [LieH 83, Moo 84, Ung 84], but w e ignore problems related to distributed garbage collection [Rud 86] .In this section we show how all tracing algorithms (i .e . the second family mentionned in the introduction) may b e derived from a unique abstract description . The description of the new algorithm proposed in this paper will then b e based on the structural uniformity of existing algorithms .The purpose of a tracing collection algorithm is t o identify all rnemory cells that are not accessible by th e mutator (the user's program) at some point in its execution , so as to make them available for the satisfaction of futur e memory allocation requests . An accessible cell is a cell tha t may be reached through some linked chain of cells from a collection of root pointers which are approximately th e mutator's pointer variables, either global or allocated on th e execution stack of the mutator . It is assumed that there i s some available function to identify all the root pointers, an d that one can also determine all the pointers contained in a n accessible cell .The abstract collection algorithm is based on the consideration of three sets of memory cells (see figure 1 ) : n~o 253 The purpose of techniques known as garbage collectio n or scavenging is to provide automatic reclamation o f memory cells that have become useless to a running program because it can no longer access them . Two families o f garbage collection algorithms have been developed . Th e first one is based on an on-the-fly reference count scheme Permission to copy without fee all or part of this material is granted provide d that the copies are not made or distributed for direct commercial advantage , the ACM copyright notice and the title of the publication and its date appear , and notice is given that copying is by permission of the Association fo r Computing Machinery . To copy otherwise, or to republish, requires a fee and / or specific permission .. ©1987 ACM 0-89791-235-7/87/0006/0253 751. In practice specific aspects have to be carefully worked out , especially to avoid explicit synchronization for parallel algorithms, or to ensure real-time behavior by locking the pace of the collector and the mutator (the user's program in th e terminology of [DijLMSS 78]) . The issues we wish to address concern single space garbage collection and compaction, either in stop-and-collect or in non-stop mode . We briefly mention relations with generation collectors [LieH 83, Moo 84, Ung 84], but w e ignore problems related to distributed garbage collection [Rud 86] . In stop-and-collect mode our algorithm is mostly usefu l because it is simpler and/or cheaper than other compactin g algorithms, using less passes and/or memory . Its adaptiveness in a continuum from pure mark-and-sweep to pur e copy collection minimizes collection costs according to th e amount of memory actually used . It can also adapt th e choice of the copy-collected area according to the framentation profile of the memory .. As a non-stop collector, this algorithm can compac t varisized cells without restricting effective mutator use o f the memory to half its actual size . This property is particularly important for real-time collectors since, because of th e time cost and unpredictability of paging, large virtua l memory may not always be acceptable for real-time applications [Bak 78], and real memory is usually relatively small . E . Spir is currently implementing the algorithm i n stop-and-collect mode for the coming version of the Le_Lis p system [ChaDH 84] .

B O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O B O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O B O O B B O O B O O O B O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O


Mixed Memory Markov Models: Decomposing Complex Stochastic Processes as Mixtures of Simpler Ones We study Markov models whose state spaces arise from the Cartesian product of two or more discrete random variables. We show how to parameterize the transition matrices of these models as a convex combinationor mixture-of simpler dynamical models. The parameters in these models admit a simple probabilistic interpretation and can be fitted iteratively by an Expectation-Maximization (EM) procedure. We derive a set of generalized Baum-Welch updates for factorial hidden Markov models that make use of this parameterization. We also describe a simple iterative procedure for approximately computing the statistics of the hidden states. Throughout, we give examples where mixed memory models provide a useful representation of complex stochastic processes. The modeling of discrete time series is a fundamental problem in machine learning, with widespread applications. These include speech recognition (Rabiner, 1989) , natural language processing (Nadas, 1984) , protein modeling (Haussler et al., 1993) , musical analysis/ synthesis (Dirst & Weigend, 1993) , and numerous others.. Probabilistic models of discrete time series typically start from some form of Markov assumption-namely, that the future is independent of the past given the present. For the purpose of statistical estimation, problems arise if either: (i) the system possesses a large number of degrees of freedom, or (ii) the window of present knowledge required to predict the future extends over several time steps. In these cases, the number of parameters to specify the Markov model can overwhelm the amount of available data. In particular, for a system with n possible states and memory length k, the number of free parameters scales exponentially as n k+1 . The difficulties are compounded for latent variable models in which the Markov assumption applies to the hidden state space. In this case, it may be computationally intractable to infer values for the hidden states. For example, in first-order hidden Markov models (HMMs), computing the likelihood of a sequence of observations scales as n 2 , where n is the number of hidden states (Rabiner, 1989) . In practice, exact probabilistic inference is therefore limited to HMMs with relatively small (or tightly constrained) state spaces.. In this technical note, we propose a principled way to investigate Markov models with large state spaces. This is done by representing the transition matrix as a convex combination-or mixture-of simpler dynamical models. We refer to the resulting models as mixed memory Markov models. While the use of mixture distributions to parameterize higher-order Markov models is well known (Raftery, 1985; Ney, Essen, & Kneser, 1994; MacDonald & Zucchini, 1997) , here we apply this methodology more broadly to factorial models-models in which large state spaces are represented via the Cartesian product of smaller ones.. Our note builds on earlier work describing factorial HMMs (Ghahramani & Jordan, 1997 ) and dynamic probabilistic networks (Binder et al., 1997) . These papers show that complex stochastic processes can be graphically represented by sets of Markov chains connected (via directed links) to a common set of observable nodes. Such models arise naturally in the study of coupled time series, where the observations have an a priori decomposition as the Cartesian product of two or more random variables. Factorial HMMs aim to combine the power of latent, distributed representations with the richness of probabilistic semantics (Williams & Hinton, 1990) . Capturing this type of probabilistic reasoning is a fundamental problem in artificial intelligence (Dean & Kanazawa, 1989) .. We believe that mixed memory Markov models have several advantages for representing complex stochastic processes and learning from examples. The parameters in these models admit a simple probabilistic interpretation and can be fitted iteratively by an ExpectationMaximization (EM) procedure (Dempster, Laird, & Rubin, 1977) . The EM algorithm has several desirable properties, including monotone convergence in log-likelihood, lack of step size parameters, and naturalness at handling probabilistic constraints. In many situations, it provides a compelling alternative to gradient-based learning methods (Baldi & Chauvin, 1996; Binder et al., 1997) .. Mixed memory models can also express a rich set of probabilistic dependencies, making them appropriate for modeling complex stochastic processes. Applied to factorial HMMs, they generalize the work by Ghahramani and Jordan (1997) in two important directions: by introducing coupled dynamics, and by considering non-Gaussian observations. They also give rise to a simple iterative procedure for making inferences about the hidden states. We describe this procedure not only for its practical value, but also because it very cleanly illustrates the idea of exploiting tractable substructures in intractable probabilistic networks (Saul & Jordan, 1997) .. The main significance of this work lies in its application to factorial HMMs and the modeling of coupled time series. In principle, though, mixed memory Markov models can be applied wherever large state spaces arise as the Cartesian product of two or more random variables. We will take advantage of this generality to present mixed memory models in a number of different settings. In doing this, our goal is to build up-in a gradual way-the somewhat involved notation needed to describe factorial HMMs.. The organization of this note is therefore as follows. In order of increasing complexity, we consider: (1) higher-order Markov models, where large state spaces arise as the Cartesian product of several time slices; (2) factorial Markov models, where the dynamics are first order but the observations have a componential structure; and (3) factorial HMMs, where the Markov dynamics apply to hidden states, as opposed to the observations themselves. We conclude that mixed memory models provide a valuable tool for understanding complex dynamical systems. Many parameterizations have been proposed for probabilistic models of time series. The mixed memory models in this note have three distinguishing features. First, they can express a rich set of probabilistic dependencies, including coupled dynamics in factorial models. Second, they can be fitted by EM algorithms, thus avoiding potential drawbacks of gradient descent. Third, they are compact and easy to interpret; notably, as in ordinary Markov models, every parameter defines a simple conditional probability. All these features should enable researchers to build more sophisticated models of dynamical systems.

O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O


Non-approximability results for scheduling problems with minsum criteria We provide several non-approximability results for deterministic scheduling problems whose objective is to minimize the total job completion time. Unless P = N P, none of the problems under consideration can be approximated in polynomial time within arbitrarily good precision. Most of our results are derived by APX-hardness proofs.We show that, whereas scheduling on unrelated machines with unit weights is polynomially solvable, the problem becomes APX-hard if release dates or weights are added. We further show APX-hardness for scheduling in flow shops, job shops, and open shops. We also investigate the problems of scheduling on parallel machines with precedence constraints and unit processing times, and two variants of the latter problem with unit communication delays; for these problems we provide lower bounds on the worst-case behavior of any polynomial-time approximation algorithm through the gap reduction technique. Since the early 1970s, the algorithms and optimization community has put much effort into identifying the computational complexity of various combinatorial optimization problems. Nowadays it is common knowledge that, when dealing with an N P-hard optimization problem, one should not expect to find a polynomial-time solution algorithm that solves each instance of the problem to optimality. This insight motivates the search for approximation algorithms that output provably good solutions in polynomial time. It also immediately raises the question of how well we can approximate a specific optimization problem in polynomial time.. We say that a polynomial-time approximation algorithm for some optimization problem has a performance guarantee or worst-case ratio ρ, if it outputs a feasible solution with cost at most ρ times the optimum value for all instances of the problem; such an algorithm is also called a polynomial-time ρ-approximation algorithm. Now, given an N P-hard optimization problem, for which values of ρ does there exist and for which values of ρ does there not exist a polynomial-time ρ-approximation algorithm? In this paper we focus on 'negative' results for scheduling problems in this area, i.e., we demonstrate for several scheduling problems that they do not have polynomial-time ρ-approximation algorithms with ρ arbitrarily close to 1, unless P = N P.. Until now, the literature only contains a small number of non-approximability results for scheduling problems. Most of the known non-approximability results have been derived for the objective of minimizing the makespan; they are also valid for the more general problem of minimizing total weighted completion time subject to precedence constraints. As far as we know, the only other non-approximability results for scheduling problems with minsum objective are presented in the papers of Kellerer, Tautenhahn & Woeginger [1996] and Leonardi & Raz [1997] . In the first paper, the authors prove that the problem of minimizing the total flow time on a single machine subject to release dates cannot be approximated in polynomial time within any constant factor. In the second one, the authors derive a similar result for the total flow time problem subject to release dates on parallel machines. Below, we give a short list with the most important results for makespan minimization, all under the assumption that P = N P; for more detailed information on the scheduling problems mentioned, see the first paragraphs of Sections 3 through 5, or see the survey article by Lawler, Lenstra, Rinnooy Kan & Shmoys [1993] . We refer the reader interested in more results on approximability and nonapproximability in scheduling to Lenstra & Shmoys [1995] . Throughout this paper, we use the notation introduced by Graham, Lawler, Lenstra & Rinnooy Kan [1979] to denote scheduling problems.. • Lenstra & Rinnooy Kan [1978] prove that P | prec, p j = 1 | C max (makespan minimization on parallel machines with precedence constraints and unit processing times) does not have a polynomial-time approximation algorithm with performance guarantee strictly better than 4/3.. • Lenstra, Shmoys & Tardos [1990] show that R | | C max (makespan minimization on unrelated machines) does not have a polynomial-time approximation algorithm with performance guarantee strictly better than 3/2.. • Hoogeveen, Lenstra & Veltman [1994] prove that P | prec, c = 1, p j = 1 | C max and P ∞ | prec, c = 1, p j = 1 | C max (two variants of makespan minimization on parallel machines with unit processing times and unit communication delays) cannot be approximated with worst-case ratios better than 5/4 and 7/6, respectively.. • Williamson et al. [1997] prove that O | | C max , J | | C max , and F | | C max (makespan minimization in open shops, job shops and flow shops, respectively) cannot be approximated in polynomial time with performance guarantees better than 5/4.. All the listed non-approximability results have been derived via the so-called gap technique, i.e., via N P-hardness reductions that create gaps in the cost function of the constructed instances.. More precisely, such a reduction transforms the YES-instances of some N P-complete problem into scheduling instances with objective value at most c * , and it transforms the NO-instances into scheduling instances with objective value at least g · c * , where g > 1 is some fixed real number. Then a polynomial-time approximation algorithm for the scheduling problem with performance guarantee strictly better than g (i.e., with guarantee g − ε where ε > 0) would be able to separate the YES-instances from the NO-instances, thus yielding a polynomial-time solution algorithm for an N P-complete problem. Consequently, unless P = N P, the scheduling problem cannot have a polynomial-time ρ-approximation algorithm with ρ < g. In this paper, we present non-approximability results for the corresponding minsum versions of the makespan minimization problems listed above. We prove that none of the scheduling problems. can be approximated in polynomial time within arbitrarily good precision, unless P = N P. Our main contribution is the non-approximability result for the problem R | r j | C j , which answers an open problem posed by Skutella [1997] . Interestingly, we do not prove this nonapproximability result by applying the standard gap technique that we sketched above, but by establishing APX-hardness of this problem (see Section 2 for some information on APXhardness). In the APX-hardness proof for R | r j | C j we are recycling some of the ideas that Lenstra, Shmoys & Tardos [1990] used in their gap reduction for R | | C max ; the APX-hardness proof of R | | w j C j follows along the lines of the APX-hardness proof for R | r j | C j . Also the non-approximability results for F | | C j and O | | C j are established via APX-hardness proofs; part of these arguments stem from the paper by Williamson et al. [1997] . The nonapproximability results for P | prec, p j = 1 | C j , for P | prec, c = 1, p j = 1 | C j , and for P ∞ | prec, c = 1, p j = 1 | C j follow from the well-known gap reductions for the corresponding makespan minimization problems P | prec, p j = 1 | C max , P | prec, c = 1, p j = 1 | C max , and P ∞ | prec, c = 1, p j = 1 | C max in a straightforward way.. The paper is organized as follows. In Section 2, we summarize some useful information on approximation schemes, non-approximability, APX-hardness and APX-completeness, and L-reductions, as we need it in the remainder of the paper. The non-approximability results are presented in Sections 3 through 5: Section 3 deals with scheduling unrelated machines; Section 4 considers flow shops and open shops; and Section 5 presents the results on parallel machine scheduling with precedence constraints and with communication delays. The paper is concluded with a short discussion in Section 6. In this paper, we have derived a number of non-approximability results for scheduling problems with total job completion time objective. Since we wrote the first version of this paper back in fall 1997, tremendous progress has been made in deriving positive approximation results for minsum scheduling problems on parallel machines. First, Skutella & Woeginger [1999] derived a PTAS for P | | w j C j . This was followed by a breakthrough paper by Afrati, Bampis, Chekuri, Karger, Kenyon, Khanna, Milis, Queyranne, Skutella, Stein & Sviridenko [1999] , that gave a PTAS for P | r j | w j C j and for Rm | r j | w j C j . Since in this paper we have proved the non-approximability of R | r j | C j and of R | | w j C j , we now know the exact borderline between minsum problems with good approximability behavior (arbitrary number of identical machines with release dates and weights; fixed number of unrelated machines with release dates and weights) and minsum problems with bad approximability behavior (arbitrary number of unrelated machines with release dates or weights).. Minsum scheduling problems with precedence constraints still are not well-understood, even on a single machine: Hall, Schulz, Shmoys & Wein [1997] give a polynomial time 2-approximation algorithm for 1 | prec | C j . Schulz & Skutella [1997] give a polynomial time approximation algorithm for 1 | prec, r j | w j C j whose performance guarantee can be made arbitrarily close to the Euler constant e ≈ 2.718. Munier, Queyranne & Schulz [1998] give a polynomial time 4-approximation algorithm for P | prec, r j | w j C j . However, not a single negative result has been proved for all these problems. We conjecture that 1 | prec | C j does not have a PTAS; in fact, we even conjecture that 1 | prec | C j does not possess a polynomial time (2 − ε)-approximation algorithm (unless P = N P).. Optimization at Chorin (Germany) in September 1999. We also thank two unknown referees for a careful reading of this paper and for many helpful comments.

B O O B O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O B I O O B I O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


There is more to IXPs than meets the eye Internet eXchange Points (IXPs) are generally considered to be the successors of the four Network Access Points (NAPs) that were mandated as part of the decommissioning of the National Science Foundation Network (NSFNET) in 1994/95 to facilitate the transition from the NSFNET to the "public Internet" as we know it today. While this popular view does not tell the whole story behind the early beginnings of IXPs, what is true is that since around 1994, the number of operational IXPs worldwide has grown to more than 300 (as of May 2013 1 ), with the largest IXPs handling daily traffic volumes comparable to those carried by the largest Tier-1 ISPs. However, IXPs have never really attracted much attention from the networking research community. At first glance, this lack of interest seems understandable as IXPs have apparently little to do with current "hot" topic areas such as data centers and cloud services or Software Defined Networking (SDN) and mobile communication. However, we argue in this article that, in fact, IXPs are all about data centers and cloud services and even SDN and mobile communication and should be of great interest to networking researchers interested in understanding the current and future Internet ecosystem. To this end, we survey the existing but largely fragmented sources of publicly available information about IXPs to describe their basic technical and operational aspects and highlight the critical differences among the various IXPs in the different regions of the world, especially in Europe and North America. More importantly, we illustrate the important role that IXPs play in today's Internet ecosystem and discuss how IXP-driven innovation in Europe is shaping and redefining the Internet marketplace, not only in Europe but increasingly so around the world. Recent findings and events concerning different aspects of the Internet ecosystem have been instrumental in bringing Internet eXchange Points (IXPs) to the attention of networking researchers, have heightened the concern among network operators with respect to the IXPs' proper operations, and have raised awareness about IXPs among the public in general. First, for networking researchers, the 2012 study by Ager et al. [55] that analyzed sFlow data from a large European IXP and revealed the presence of some 50K+ actively used peering links in this single IXP was an eyeopener. To put the discovery of such a rich peering fabric supported at this single IXP into proper perspective, recall that for many years, the networking research community has been studying AS-level maps of the global Internet that had far fewer peering links in total than was observed in just this single location. In the process, the authors of [55] also highlighted the increasingly important role that such large IXPs play especially in Europe, where their public peering-related daily traffic volumes are comparable to those seen by the largest global Tier-1 ISPs, where their constantly growing list of member ASes reads like the "who is who in Internet commerce" and where their service offerings rival and compete head-on with those traditionally provided by the incumbent ISPs.. Second, the critical role of network operators to ensure "bestin-class" operations of IXPs and a clear need for networking researchers to know more about these IXPs' basic mode of operations and underlying business models have come to the forefront in the recent discussions in the trade press [10, 21, 26] and technology blogs [11, 45, 47, 49] concerning the highly publicized Distributed Denial of Service (DDoS) attacks in late March 2013 featuring players like Spamhaus, Stophaus, CloudFlare, and IXPs [40] . In short, in an attempt to dilute the massive DDoS attacks launched by Stophaus against Spamhaus, Spamhaus apparently started to rely on CloudFlare's services and remained accessible. To counter Spamhaus' tactic, Stophaus allegedly redirected its effort and started to DDoS the network components that are at the core of CloudFlare's technology: the IXPs where CloudFlare connects to its upstream providers for the purpose of exchanging traffic. Irrespective of the motives of the involved parties or the accuracy of the reported actions, these and similar incidents demonstrate that understanding the end-to-end flow of traffic in today's Internet has to include as key component the IXPs and the networks that peer at those IXPs.. Lastly, given the widely acknowledged differences between the European IXP scene and the IXP marketplace in North America (see Section 2.3), very recently announced intentions [35, 64, 81] to revive the relatively stagnant North-American IXP marketplace by trying to gain a foothold for the European exchange model in the US and Canada could have far-reaching implications for a wide range of Internet stakeholders. Most importantly, if successful, such an attempt would address the scarcity of public peering opportunities in North America in comparison to Europe and would clearly challenge the dominant commercial (i.e., for-profit) IXP business model in the US. In view of such a potential business model "import" from Europe to North America, it will be important to understand the key reasons for the enormous attraction that these large IXPs have in Europe and examine the differences in economic conditions that may or may not favor a status quo over a complete overhaul for the North American IXP scene.. Motivated by these and other examples, this article serves on the one hand as a short tutorial where we describe the basic operations of a generic IXP and the typical set of service offerings that networks can use once they become a participant of such an IXP. Given the significant differences in the various IXP marketplaces, we also explain the key aspects that distinguish the European IXP scene from the North American one or from IXPs in the Asian-Pacific, African, or Central/South-American regions. All the information provided in this article is publicly available and may be well-known within the NANOG/network operator community. However, the critical role that IXPs are playing in the overall Internet ecosystem has until recently gone unnoticed by large parts of the networking research community, and much of the provided background material may be essentially unknown to many networking researchers whose focus has traditionally been US-centric, meaning that their emphasis to date has been typically on large ISPs, content distribution networks (CDNs), and large content providers and not on what appears at first glance relatively unimportant components of the Internet infrastructure in the form of US-based IXPs. 2 In this sense, this tutorial is also intended to be a survey of the existing literature on IXP-related and relevant works that have not necessarily appeared in mainstream networking research venues.. At the same time, we also view this article as an opportunity to showcase IXPs as a rich source for interesting networking research problems that have largely gone unnoticed in the past. For one, IXPs are essential for studying the Internet ecosystem, including the increasingly sophisticated peering strategies deployed by many of today's Internet players and the increasingly complex nature of the traffic flow across the network that results from these peering arrangements. In particular, having a solid grasp of the role that IXPs play in the existing Internet ecosystem is critical for understanding how content is distributed in today's Internet and how the different parties (e.g., content providers, CDNs, ISPs) are adapting to the constantly changing nature of content distribution. Moreover, IXPs are bound to play a similarly important if not more critical role for the emerging cast of cloud providers and their attempts to bring their services as close to the end users as possible. Finally, from a more IXP-centric perspective, the sustained rapid growth in IXP members as well as IXP traffic volumes in some of the most successful IXPs combined with the demand for and need to properly manage increasingly complex IXP-specific traffic engineering solutions raises serious scalability issues with respect to IXPs' switching architectures and network operations and may provide concrete application domains for Software Defined Networking (SDN) and also inspire new research problems in that space.. Finally, while this article is intended for the general networking research community, it is not meant to and cannot subsume the ex-positions on a wide range of IXP-related issues by W. Norton (see, for instance, [70] ). On the contrary, we encourage the readers of this article to use it as a stepping stone towards becoming more familiar with the related topics discussed by Norton and considering some of his more economics-focused arguments into future studies of the Internet ecosystem. While IXPs have occasionally featured in networking research papers (see, for example, [60, 85, 65, 56, 76, 59, 78, 63] , to mention but a few), they are typically the focal point of Norton's white papers on various aspects related to peering (e.g., [69, 71, 72] ) and of reports by organizations such as Euro-IX [24, 77] , Internet Society [66, 67] , or governing or regulatory bodies such as OECD [29, 83] , EU [41] , FCC [51, 54] , and ITU [52, 53] . Our hope is that this article will raise awareness among networking researchers about the importance and relevance of IXPs for today's Internet ecosystem by connecting ongoing or planned research efforts in this area more tightly with the expertise that exists within the network operator/provider communities on IXPs and IXP-related issues but is often ignored or not understood by the more research-oriented networking community. In the past, IXPs have received little or no attention from the networking researchers who considered them either just as "add-ons" to an Internet dominated by large Tier-1 ISPs and large content providers/distributors or as components of the Internet's infrastructure that are largely void of interesting research problems (after all, an IXP is just a switch ...). We argue in this article that once the research community has a better understanding of the architectures, operations, service offerings, and different business models of the hundreds of operational IXPs in today's Internet, it will likely revise its low opinion of IXPs as a research topic and realize their potential for being a rich source for very interesting and certainly very important problems dealing with a combination of technological and economic issues facing the current and future Internet ecosystem. It is our hope that this article spurs the interest of networking researchers to look into IXPs, entices them to examine in more detail the surveyed literature to learn about the in's and out's of IXPs that this article could not cover, and ultimately convinces the community that there is indeed more to IXPs than meets the (uninformed) eye and that the IXPs when viewed as critical components of the Internet deserve to be taken seriously.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


An Approach for Constructing Parsimonious Generalized Gaussian Kernel Regression Models The paper proposes a novel construction algorithm for generalized Gaussian kernel regression models. Each kernel regressor in the generalized Gaussian kernel regression model has an individual diagonal covariance matrix, which is determined by maximizing the correlation between the training data and the regressor using a repeated guided random search based on boosting optimization. The standard orthogonal least squares algorithm is then used to select a sparse generalized kernel regression model from the resulting full regression matrix. Experimental results involving two real data sets demonstrate the effectiveness of the proposed regression modeling approach. A fundamental principle in practical nonlinear data modeling is the parsimonious principle of ensuring the smallest possible model that explains the training data. Forward selection using the orthogonal least squares (OLS) algorithm [1] - [5] is a simple and efficient construction method that is capable of producing parsimonious linear-in-the-weights nonlinear models with excellent generalization performance. Alternatively, the state-of-art sparse kernel modeling techniques, such as the support vector machine and relevant vector machine [6] - [9] , have been gaining popularity in data modeling applications. These existing sparse regression modeling techniques typically place the kernel centers or mean vectors at the training input data and use a fixed common kernel variance for all the regressor kernels. The value of this common kernel variance obviously has a critical influence on the sparsity and generalization capability of the resulting model, and it has to be determined via some sort of cross validation. For example, in [3] a genetic algorithm is applied to determine the appropriate common kernel variance through optimizing the model generalization performance.. In this paper, we consider a generalized Gaussian kernel model, in which each kernel regressor has an individually tuned diagonal covariance matrix. Such a generalized kernel regression model has the potential of improving modeling capability and producing sparser final models, compared with the standard approach of single fixed common variance. The difficult issue is then how to determine these kernel covariance matrices. Since the correlation function between a kernel regressor and the training data defines the "similarity" between the regressor and the training data, it can be used to "shape" the regressor by adjusting the associated kernel covariance matrix in order to maximize the absolute value of this correlation function. A weighted optimization algorithm, which has its root from boosting [10] - [12] , is considered to perform the associated optimization task. This weighted optimization algorithm is a guided random search method and the solution obtained may depend on the initial choice of population. To provide a robust optimization and guarantee stable solutions regardless of the initial choice of population, the algorithm is augmented into a repeated weighted optimization method.. The determination of kernel covariance matrices essentially provides the full bank of regressors or the full regression matrix, and this allows the application of the standard OLS algorithm [1] , [2] to select a parsimonious subset model. The outline of the paper is as follows. Section 2 gives the generalized Gaussian kernel regression model to be considered. Section 3 derives the correlation criterion to be used for determining the kernel covariance matrices and presents a repeated boosting search optimization algorithm for performing the corresponding optimization tasks. Section 4 briefly summarizes the standard OLS algorithm used to select a sparse kernel regression model, while Section 5 describes our modeling experiments. Finally, Section 6 offers our conclusions. . is a priori unknown system mapping, and 2. is an unknown parameter vector associated with the appropriate, but yet to be determined, model structure. The system model (1) is to be identified from an b -sample system observational. , using some suitable functions which can approximate ¥ a § p Y a 3 with arbitrary accuracy.. We will model the unknown dynamical process (1) by using the following generalized Gaussian kernel regression model. where q F ¡ denotes the model output given the input B ¡ , w v are the model weight parameters, and y @ v p Y a 3 are the kernel regressors. We allow the regressor function to be chosen as the general Gaussian function. where the kernel covariance matrix takes the form of. As is in a standard kernel regression model, the kernel mean vectors are placed at the training input data points. If all the diagonal covariance matrices are set to the identical form of diag . by defining. The objective of sparse modeling is to construct a subset model consisting of b § § ¦ b 3 significant regressors only from the full set of regressors defined in (9). A novel construction algorithm has been developed for the generalized Gaussian kernel model.. Each kernel regressor in the pool of candidate regressors has an individual diagonal covariance matrix, which is determined by maximizing the absolute value of the correlation between the regressor and the training data using a repeated weighted search optimization. The standard orthogonal least squares algorithm is then applied to select a parsimonious model from the full regression matrix. Compared with the existing kernel regression modeling approaches which adopt a single common kernel variance for all the regressors, the proposed method has the advantages of improving modeling capability and producing sparser models. These advantages have been demonstrated by the experimental results involving two real data sets. Table 3 : Subset model generated for the gas furnace data set by the OLS algorithm with a generalized Gaussian kernel model. The kernel covariance matrices are determined by maximizing the correlation criterion using the repeated weighted optimization algorithm.

O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O B I I O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O