VarioML framework for comprehensive variation data representation and exchange Background: Sharing of data about variation and the associated phenotypes is a critical need, yet variant information can be arbitrarily complex, making a single standard vocabulary elusive and re-formatting difficult. Complex standards have proven too time-consuming to implement. Results: The GEN2PHEN project addressed these difficulties by developing a comprehensive data model for capturing biomedical observations, Observ-OM, and building the VarioML format around it. VarioML pairs a simplified open specification for describing variants, with a toolkit for adapting the specification into one's own research workflow. Straightforward variant data can be captured, federated, and exchanged with no overhead; more complex data can be described, without loss of compatibility. The open specification enables push-button submission to gene variant databases (LSDBs) e.g., the Leiden Open Variation Database, using the Cafe Variome data publishing service, while VarioML bidirectionally transforms data between XML and web-application code formats, opening up new possibilities for open source web applications building on shared data. A Java implementation toolkit makes VarioML easily integrated into biomedical applications. VarioML is designed primarily for LSDB data submission and transfer scenarios, but can also be used as a standard variation data format for JSON and XML document databases and user interface components. Conclusions: VarioML is a set of tools and practices improving the availability, quality, and comprehensibility of human variation information. It enables researchers, diagnostic laboratories, and clinics to share that information with ease, clarity, and without ambiguity. VarioML enables researchers, diagnostic laboratories, and clinics to improve the quality of human variation information, and to share that information with ease, clarity, and without ambiguity. VarioML resolves the inherent tendency of variation data to diverge in format and meaning through a modular design that lives in an open collaboration framework, composed of two linked community forums.. With this open collaboration framework, the variome community itself closely binds the evolution of the annotation format and its tooling to the science of the study of human mutations. For example, as new configurations and extensions of the format are developed by various implementers, they can be discussed and improved at the GEN2PHEN Knowledge Center, alongside submissions of relevant data made through Cafe Variome. As community consensus emerges, this agreement translates to changes in the schema and tooling in the common repository. At each step, the provenance of even small contributions are captured and can be used as microattributions [42] .. For such bottom-up, self-organizing management of a common variation standard to work, teams working at critical junctions in the variation pipeline must translate a passion for the vision of the unified variome, into both implementation and development of the shared standard. To date, VarioML has been implemented in three applications (the Leiden Open Variation Database [24] , GenSearch [36] , and BC|SNPMax [37] ), and is currently being tested in a fourth (Alamut [38] ). In each case, VarioML is used to enable push-button submission of data through the Cafe Variome service [28] .. With consensus on a minimal standard, implementation is the remaining bottleneck. Users, from research teams to commercial software producers, need to focus their software-related activity to those tasks in which their resource costs are proportionally smaller than the added value afforded by adopting new tools and data models. VarioML has been designed to minimize the effort required for both implementation and extension, framing the specification itself with Java and JSON APIs on the one hand, and an open collaboration framework on the other. We hope this approach proves useful throughout the variation science community, as it meets the challenge and potential of next generation sequencing, and quickens to open the path from discovery to delivery. MB VarioML has been designed to immediately serve data exchange needs for LSDB's, focusing on curated variant entries produced at the end of data production pipelines. However, as an end-to-end variation pipeline comes together, for a common specification to be truly useful, it must be extensible beyond the immediate LSDB usecase. Next-generation sequencing pipelines make possible exon-capture scenarios in which tens to hundreds of patients are sequenced in one or more genes, presenting new challenges in variant calling, annotation, and data sharing [70] . To meet the holistic data integration challenge and realize the grand variation pipeline, we need to harmonize the data models, data standards, and content specifications in use at each step, to encompass the descriptive needs of all LSDBs, standardizing their quality and accuracy, and enabling more comprehensive and high quality data curation [26] . A number of projects have previously attempted to fill these requirements and provide a single multipurpose implementation format for LSDBs, yet have come up against difficulties at multiple levels of design and implementation [11] . Variation data can be arbitrarily complex, making a single standard specification elusive. LSDB use cases vary a great deal in the depth of detail and structure needed for data capture. Complex standards have proven too time-consuming to implement. Solutions designed in one format cannot be readily transferred to another. Further hampering progress towards a common specification are the multiple strong motivations which labs have to keep variant data private [71] .. As work on a unified variome progresses, genetics research faces a paradox: another attempt at a variation standard will not be enough to surmount these obstacles. No matter how comprehensive our current efforts, new standards will inevitably follow. Our understanding of genomic variation is rapidly evolving; multiple and often conflicting forms of variant annotation seem to be required to serve differing use cases, implementations, and viewpoints. Attempts to comprehensively integrate all such descriptions in a single standard can, at this point, be expected to produce either unmanageable complexity, or inaccurate oversimplification. Furthermore, looking ahead, we can be fairly certain that new discoveries and technologies will arise that cannot be presently designed for.. In designing VarioML, we therefore turned away from seeking a top-down monolithic solution, choosing instead to make a lightweight framework for composing interoperable, use-case specific 'micro-standards' around the generalized concepts of observation targets, ontology terms, and observations, adapted from the Observ-OM specification [16] . The core set of VarioML schema elements can be used as building blocks, addressing use cases from the most minimal towards the more complex, while maintaining the underlying interoperability of the data. Implementations can use as many or as few of these blocks as needed, and new elements can be added into the specification as needed. However, with this extensibility also comes the danger of the fragmenting specifications into incompatible versions. While elements are utilized in an increasing number of new representations and schema, at the same time, they are also converging all variation data into a unified variome pipeline. Yet equalizing these needs for divergence and convergence is a task that cannot be planned by a committee. As next generation sequencing continues accelerating both the scale and complexity of the data produced, all producers of variation data have a stake in decreasing the gap between a general variation annotation standard, and the community it serves. Accordingly, VarioML is intended less as a 'completed' specification, more as a nucleation centre around which new specifications can be developed. All variation data producers are called upon to develop this specification collaboratively.. To this end, VarioML development has been turned over to the community. The specification lives inside an open collaboration framework, tightly binding new variation reporting structures to the common schema and tooling, maintaining consistent application generation capability and backwards compatibility with earlier applications and data [72] . We chose two forums to realize this collaboration framework: the VarioML forum at the science-centered GEN2PHEN Knowledge Center [73] , where format details are discussed alongside immediate access to a unified catalog of LSDBs and other tools for variation data integration; and VarioML's GitHub repository [74] , where the schema and XML, JSON, Java, and RDF tools are available, in addition to UML documentation that clarifies the relationships between specification and implementations [75] . Modified or new compositions using schema elements must be reported in either of these forums and discussed openly, enabling the collaborative extension of the format without breaking existing implementations.. The open-ended nature of the VarioML specification means there should continuously be elements under active redefinition and modification by the community.. These features should not be implemented in applications until consensus on usage is reached. For example, the Variant element allows recursive sub-elements, for cases where the reporting variant is composed of other variants located on the same or a sister chromosome. Yet this and other features (see Additional XML elements section) are not currently implemented in applications. The study of disease-causing and benign variations in the human genome is progressing rapidly. Whole genome and exome sequencing continues to expand, and improved tools for variant calling are becoming available [1] [2] [3] . Cost-effective sequencing, paired with variant discovery, promises to make early detection and intervention accessible for the millions of individuals with genetic diseases.. However, realizing this potential is blocked by the problem of integrating and coordinating the steps towards "a pipeline leading from discovery to delivery" [4] . The GEN2PHEN project was initiated in 2008 to unify human and model organism genetic variation databases, and remove the obstacles to translation of variant data from laboratory to clinic to public [5] . This has involved attempting to unify the divergent data representations of various database communities.. The focus of this effort is the locus-specific database (LSDB) [6] . LSDBs describe the variants discovered on a single gene, a gene family or a group of genes involved in the similar diseases or traits. As of this writing, 4, 111 LSDBs can be easily searched online [7] . LSDBs are curated by experts on their respective loci, and as such are typically the best resources of gene variant information available [8] . A comprehensive 2010 analysis of 1,188 LSDBs provides a useful overview of the domain, providing encouraging results, such as finding only 5.4% to be outdated [9] . However, the study also found that only 8% provided detailed disease and phenotypic descriptions. LSDBs also vary widely in format, diverging to satisfy the immediate requirements of numerous use cases, making comprehensive, global analysis of data concerning a given variant difficult, if not impossible [10] . LSDBs are also typically incomplete, either from a lack of capacity on the part of the data submitters or curators to include all pertinent data, or from the original data lacking key elements altogether [9, 11] . It is well recognised that the data will often be incomplete if you ask too much of the submitters [12] . Into this situation, next-generation sequencing pipelines are rapidly increasing the scale and complexity of data to be managed [13] . XML remains the reference platform of choice, providing a mature specification, and advanced tools such as schema definition languages [57] . Our use of extensible XML elements encourages implementers to collaborate closely, since extending the format requires formulating a common development strategy. However, we realized that, as the XML schema is extended, a lacuna could easily open between the data model and its implementations. Adapting changing XML schema into applications has tended to be laborious. We reasoned that absolving application developers of the need to reinvent the wheel of data translation across formats was fundamental to easing the effort and cost of adopting the VarioML standard. We further reasoned that, in biomedicine as well as in other scientific domains, the era of big data likely makes it no longer feasible to develop formats separately from the tooling that transports them bidirectionally across the required data languages (recalling the computer science maxim, 'Data = code' [58] [59] ).. In practice, this meant that VarioML schema elements have to work transparently as XML, JSON, and possibly in future as RDF, without incurring a cost of translation to the implementer or user. Providing support for bidirectional translation to JSON was a clear way in which we could enable schema extensions to much more quickly and inexpensively be reflected in applications. To this end, VarioML comes with Java and JSON APIs (application programming interfaces), which developers can plug into their applications to handle conversion and publication.. JSON is the common data serialization format now recognized as the lingua franca for data exchange over the web (while we find no academic reference to this fact, it is a commonplace in the web application domain. e.g. [60] ), proven to be faster and consume fewer resources than XML [61] . Serializing to JSON facilitates applying programming techniques to data, to create interactive content, user interface components, etc. in formats native to the web, simplifying the provision of data access [62] . VarioML provides a JSON implementation, currently defined using JAXB [63] and Jackson [64] annotations. This JSON implementation is made available as a VarioML Java library [65] , which can be used to read and write XML and JSON versions of the format. An API is auto-generated from XML instances, providing Java object representations for all VarioML objects. This API will be kept synchronized with the format, and can be used as a helper tool in Java applications. A JSON example of the source element is shown in Figure 4 .. In addition to JSON support, the VarioML Java API supports EXI, a binary compression of XML. EXI support leverages the VarioML XML schema, reducing file sizes and the time required for data processing operations by factors of three to ten [66] . While VarioML is primarily focused on curated variant entries produced at the end of HTP pipelines, the use of EXI makes it feasible to use VarioML for earlier stages of production pipelines.. The JSON tooling provided with VarioML makes it possible for implementers to develop dynamic user interfaces with substantially less effort and cost [67] , expanding on the possibilities demonstrated by the 'Web Analysis of the Variome' project [68, 69] . A logical next step to contribute to the end-to-end variome pipeline would be to build a variant annotator widget, usable with different database implementations.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The Empirical Distribution of Good Codes Abstract-Let the kth-order empirical distribution of a code be defined as the proportion of k-strings anywhere in the codebook equal to every given k-string. We show that for any fixed k, the kth-order empirical distribution of any good code (i.e., a code approaching capacity with vanishing probability of error) converges in the sense of divergence to the set of input distributions that maximize the input/output mutual information of k channel uses. This statement is proved for discrete memoryless channels as well as a large class of channels with memory. If k grows logarithmically (or faster) with blocklength, the result no longer holds for certain good codes, whereas for other good codes, the result can be shown for k growing as fast as a certain fraction of blocklength.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Automated processing of shoeprint images based on the Fourier transform for use in forensic science Abstract-The development of a system for automatically sorting a database of shoeprint images based on the outsole pattern in response to a reference shoeprint image is presented. The database images are sorted so that those from the same pattern group as the reference shoeprint are likely to be at the start of the list. A database of 476 complete shoeprint images belonging to 140 pattern groups was established with each group containing two or more examples. A panel of human observers performed the grouping of the images into pattern categories. Tests of the system using the database showed that the first-ranked database image belongs to the same pattern category as the reference image 65 percent of the time and that a correct match appears within the first 5 percent of the sorted images 87 percent of the time. The system has translational and rotational invariance so that the spatial positioning of the reference shoeprint images does not have to correspond with the spatial positioning of the shoeprint images of the database. The performance of the system for matching partial-prints was also determined. S HOE marks-the mark made by the outside surface of the sole of a shoe (the outsole)-are distinctive patterns that are often found at crime scenes. Shoe marks can be broadly broken into two classes: 1) shoe impressions which contain 3-dimensional information (e.g., shoe impression at the beach) and 2) shoeprints which contain 2-dimensional information (e.g., shoeprint on a floor). Shoe marks are common at crime scenes and are believed to be present more frequently than fingerprints [1] . In [2] , a study of several jurisdictions in Switzerland revealed that 35 percent of crime scenes had shoeprints usable in forensic science, while in [3] , Girod found that 30 percent of all burglaries provide usable shoeprints.. Shoe marks provide valuable forensic evidence. In many instances, shoe marks can be positively identified as having been made by a specific shoe to the exclusion of all other shoes. The identification is based on the physical match of random individual characteristics the shoe has acquired during its life. The evidence provided by a positively identified shoe mark is as strong as the evidence from fingerprints, tool marks, and typewritten impressions [1] .. In other instances, the detail retained in a shoe mark may not be sufficient to uniquely identify an individual shoe but nevertheless it is still very valuable. Due to the wide variety of shoes available on the market, with most having distinctive outsole patterns, this implies that any specific model of shoe will be owned by a very small fraction of the general population [1] . If the model of a shoe can be determined from its mark, then this can significantly narrow the search for a particular suspect.. An image of a shoe mark can be obtained using photography, gel, or electrostatic lifting or by making a cast when the impression is in soil. Subsequently, in the forensic laboratory, the image of the shoe mark is compared with the shoeprints and shoe impressions of known shoe samples. A process of detection and recovery of footwear impression evidence and of comparison of the impressions with suspect shoes is described in [1] .. The photograph of the impression or of the lifted impression or cast can be subsequently scanned and a digital image produced. Forensic analysis requires comparison of this image against specific databases. These databases include:. 1. Database of marks made by shoes currently and previously available on the market and 2. Database of shoe marks found at other crime scenes. Comparing crime scene shoe mark images to databases is currently a laborious task and it is commonly manually conducted by searching paper catalogues or computer databases. Due to the time consuming nature of the task, shoe mark evidence is not used as frequently as it could be. For example, in 1993, only 500 of 14,000 recovered prints in The Netherlands were identified [4] . Thus, computer-based methods that reduce the operator effort for this task offer great benefit to forensic scientists.. A number of semiautomatic methods have been proposed to assist forensic laboratories making these database comparisons [3] , [5] , [6] . In all methods, an operator uses a palette of basic shapes to build a model of a shoeprint pattern. The basic shapes model visible characteristics of the outsole, such as wavy patterns, concentric circles, logos, etc. This information is then used to match the print against a database of shoeprints that have been similarly modeled.. Disadvantages of this method are that modern shoes have increasingly more intricate outsole patterns which are difficult and tedious to describe with a few basic shapes and that the output of the system is dependent on the operator. Different operators may select different geometric patterns when coding the same shoeprint and this may lead to difficulties in determining a match at a later stage.. While a number of schemes exist for semiautomatic shoeprint processing systems, work in the area of automatic shoeprint recognition has not been reported widely. Geradts and Keijzer [4] describe an approach to an automated system. As above, the outsole pattern is described by a series of geometric shapes that are either entered by an expert or are generated automatically using a number of primitive "erosion" and "dilation" operators applied to a digital image of the shoeprint. The spatial position of these shapes is recorded. For each type of geometric shape, the twodimensional Discrete Fourier transform (DFT) was used to determine the spatial frequency of the shapes. The advantage of the DFT is that it is invariant to translations. Matching was achieved with a neural network processing the Fourier transform coefficients. Bouridane et al. [7] and Alexander et al. [8] utilize fractals to represent the shoeprints and a mean square noise error method for determining the final match. The performance of our system is compared to Bouridane et al.'s system in Section 7. This study developed a system that automatically sorts a database of shoeprint pattern categories in response to a reference shoeprint to accelerate the human observer identification of the tread pattern of the reference shoeprint image. Tests of the system have shown that when a reference image of a full-shoeprint is presented to the system, the first sorted image correctly matched the reference image 65 percent of the time. There is an 87 percent probability that a matching image will be found within the first 5 percent of the sorted database patterns. The system was also able to successfully sort database images in response to reference partial-prints. The system has translational and rotational invariance so that the spatial positioning of the reference shoeprint image and the database shoeprint images need not correspond. A flow diagram of the shoeprint analysis system developed in this study is shown in Fig. 4 . The system processes a reference shoeprint image and produces a transformed image. The transformed reference image is then compared to each transformed database image and a measure of similarity calculated. Next, the similarity of the reference image to each pattern category is determined. On the basis of the similarity measures, the database categories are sorted so that categories that most likely have an identical pattern to the reference shoeprint appear at the start of the sorted list. The forensic scientist then reviews the sorted categories and identifies an occurrence (if one exists) of a database category that has the same pattern as the reference shoeprint image. If a matching database category is identified, then the associated database fields can provide shoe manufacturer style information of the input shoeprint. The frequency of the pattern category can also be established and the database queried to determine the average age of the typical wearer.. The Fourier transformation (FT) is a key element within the sorting algorithm, due to having translation and rotation properties as follows [9] .. Let the image f 2 be a rotated and translated replica of the image f 1 with rotation and translation x 0 ; y 0 ð Þ, then According to the Fourier translation and rotation properties, the corresponding FT, F 1 , and F 2 (with frequency variables and ) will be related by. The power spectral density (PSD) function can be calculated by squaring the absolute value of the FT. For images f 1 and f 2 , the corresponding PSD functions M 1 and M 2 will be related by. Thus, the PSD is invariant to translation and the rotation of an image results in the same rotation of the PSD. Experiments were conducted to optimize the system using the full-prints. Once optimized, an assessment of the performance of the system using the full and partial-prints was conducted.

O O O B O O O O B I O O O B I O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Least-cost design of water distribution systems under demand uncertainty: the robust counterpart approach In this study, a non-probabilistic robust counterpart (RC) approach is demonstrated and applied to the least-cost design/rehabilitation problem of water distribution systems (WDSs). The uncertainty of the information is described by a deterministic user-defined ellipsoidal uncertainty set that implies the level of risk. The advantages of the RC approach on previous modelling attempts to include uncertainty are in making no assumptions about the probability density functions of the uncertain parameters and their interdependencies, having no requirements on the construction of a representative sample of scenarios, and the deterministic equivalent problem preserves the same size (i.e. computational complexity) as the original problem. The RC is coupled with the cross-entropy heuristic optimization technique for seeking robust solutions. The methodology is demonstrated on an illustrative example and on the Hanoi network. The results show considerable promise of the proposed approach to incorporate uncertainty in the least-cost design problem of WDSs. Further research is warranted to extend the model for more complex WDSs, incorporate extended period simulations, and develop RC schemes for other WDSs related management problems. In this study, an RC approach was suggested for least-cost design of WDSs. The results show considerable promise of the robust approach, which exhibits several advantages that need to be further investigated: (1) in contrast to stochastic optimization, the RC does not assume the parameter uncertainty is stochastic with underlying PDF; (2) it results in a deterministic equivalent, which is especially appealing due to the number of required simulations. This becomes even more evident when the problem is solved using simulationbased optimization techniques; (3) it can provide a trade-off between reliability and cost as a function of safety parameter γ set by the engineer; and (4) its implementation is rather straightforward and flexible.. It is important to note some of the drawbacks of the RC approach, particularly in the current formulation: (1) it is a min-max oriented approach, thus the solution may be conservative;. (2) the current formulation of the RC to WDS results in a degenerate case, Equation (14) (3) the current formulation does not explicitly relate the uncertainty to the reliability, i.e. γ to R, and the reliability is computed after the problem is solved. The RC application to WDS design should be further investigated addressing all the above-mentioned issues including application to more complex networks.

B I O B I I O O B O O O O O O O O O O O O O O O O O O O O O O O B I I O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Growth optimal investment in horse race markets with costs Abstract-We formulate the problem of growth optimal investment in horse race markets with proportional costs and study growth optimal strategies both for stochastic horse races as well as races where one does not make any distributional assumptions. Our results extend all known results for frictionless horse race markets to their natural analog in markets with costs.Index Terms-Growth optimality, proportional transaction costs, universal investment. In this work we formulate the problem of growth optimal investment in horse race markets with proportional transaction costs. These markets are, as the name suggests, very much like horse races, i.e., in every market period one of the assets pays off and all the other assets pay nothing. In fact, the wealth invested in these assets is lost completely. Horse race markets, also called erodible asset markets, are very special cases of general markets and are in a sense the extreme points for the distribution of asset returns [1] , [2] .. The objective of growth optimal investment is to maximize the long-run interest rate. Growth optimal investment in independent and identically distributed (i.i.d.) horse race markets with no transaction costs was introduced by Kelly [3] . Kelly showed that log-optimum investment, where the investor maximizes the conditional expected logarithm of his one-step return, maximizes the growth rate of the cumulative wealth. Kelly established a duality between the entropy rate and the growth rate by showing that they sum to a constant. Breiman [4] extended this framework to investment in i.i.d. markets with general asset returns. Algoet and Cover [5] showed that conditionally log-optimum investment is, indeed, growth optimal for all frictionless stationary ergodic markets with general asset returns. This work (see also [6] ) extended the duality between information Manuscript received August 9, 1999 . This work was supported in part by NSF under Grant CCR-9973134 and MURI#DAAD19-99-1-0252.. G. N. Iyengar is with the Department of IEOR, Columbia University, New York, NY 10027 USA (e-mail: garud@ieor.columbia.edu).. T. M. Cover is with the Departments of Electrical Engineering and Statistics, Stanford University, Stanford, CA 94305 USA (e-mail:cover@isl.stanford.edu).. Communicated by I.Csiszár, Associate Editor for Shannon Theory.. Publisher Item Identifier S 0018-9448(00)09670-X.. rate and growth rate to stationary ergodic markets. Subsequently, Cover [7] introduced universal investment where one does not make any distributional assumptions on the sequence of asset returns and constructed a sequential policy that does as well in growth rate as any other constant rebalanced policy, even those chosen in hindsight. The universal result was extended to an individual sequence result by Ordentlich and Cover [2] , [8] .. The preceding work assumes that transactions in the market do not incur any costs. Unfortunately, the policies designed for frictionless markets do not perform well in a market with costs. In fact, in a continuous time market such policies lead to immediate bankruptcy [9] . The situation in discrete time markets is not as severe [10] , but one can do substantially better by incorporating the costs into the model. Growth optimal investment in markets with costs was introduced by Taksar et al. [11] in the context of geometric Wiener markets with one risky asset and cash. Several related works, notably Davis et al. [9] and Akian et al. [12] , study investment policies that maximize the discounted utility of the consumption stream in markets with costs. Iyengar [13] and Akian et al. [14] have recently extended continuous time growth optimal investment to the general case of several risky assets.. The outline of this correspondence is as follows. In Section II we introduce our model for a market with proportional transaction costs and then formulate the problem of growth optimal investment in discrete-time stochastic horse race markets. We show that the conditionally log-optimum policy is growth-optimal. We then discuss some properties of the log-optimum policy and its sensitivity to the information structure. Although the results in this section are specific to the special case of horse race markets, we believe that the intuition developed here will generalize to general discrete-time markets [15] . Section III studies universal investment in horse race markets with proportional transaction costs. The last section includes some concluding remarks, connection to general markets, and some open issues. Growth optimal investment in horse race or erodible asset markets with proportional transactions costs is similar to the problem without costs. For the stochastic horse race markets we show that the conditionally log-optimum policy still remains growth-optimal. At time n, if z n = e i , the optimal policy invests all the wealth in the portfolio b n E E E log(b t Xn X n01 1 = p (n01) :. The optimal policy in a horse race market with transactions costs has the following simple characterization. where j = (1 0 i )=(1 + j ). Thus the intent is always the same-place sell and buy orders as if there were no transactions costs ( [5] , [3] ) and accept whatever you get as a result.. The maximum achievable growth rate g in a stationary, ergodic market with transaction costs is given by

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I B I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Supporting the annotation of chronic obstructive pulmonary disease (COPD) phenotypes with text mining workflows Background: Chronic obstructive pulmonary disease (COPD) is a life-threatening lung disorder whose recent prevalence has led to an increasing burden on public healthcare. Phenotypic information in electronic clinical records is essential in providing suitable personalised treatment to patients with COPD. However, as phenotypes are often "hidden" within free text in clinical records, clinicians could benefit from text mining systems that facilitate their prompt recognition. This paper reports on a semi-automatic methodology for producing a corpus that can ultimately support the development of text mining tools that, in turn, will expedite the process of identifying groups of COPD patients. In this paper, we elucidate our proposed text miningassisted methodology for the gold-standard annotation of COPD phenotypes in a corpus of full-text scientific articles. We demonstrate with the proposed scheme that the annotation task can be kept simple for curators whilst producing expressive and computable annotations. By constructing a semi-automatic annotation workflow in Argo, we seamlessly integrate and take advantage of several automatic NLP tools for the task. Furthermore, we are providing the domain experts with a highly intuitive interface for creating and manipulating annotations. The comparison of annotations automatically generated by the workflow against manually validated ones (i.e., gold standard) reveals an F-score of 45.70% using relaxed matching. New concept recognisers trained on these gold standard annotations demonstrate dramatically better performance (i.e., with a 20-to 30-percentage point margin in terms of F-scores) over the off-the-shelf components used in the Argo workflow.. Manual expert validation of the text mining-generated annotations on the remaining 21 papers in the corpus is still ongoing. In the meantime, we are enhancing our ontology concept linkers, which, once ready, will be applied on the gold standard concepts to enrich our corpus with computable annotations. Our expert collaborators are also working hard on obtaining a subset of clinical records from their hospital, which will then be used to augment our corpus. With the resulting resource, which will be made publicly available upon completion, we aim to support the development and evaluation of text mining systems that can ultimately be applied to evidence-based healthcare and clinical decision support systems. Performance is compared with that of the components utilised in the text mining-assisted workflow. Table 6 Results of evaluation using a fixed split over 381 paragraphs (training set: 75% or 286 paragraphs; held-out set: 25% or 95 paragraphs), using exact matching Various corpora have been constructed to support the development of clinical natural language processing (NLP) methods. Some contain annotations formed on the basis of document-level tags indicating the specific diseases that clinical reports pertain to. In the 2007 Computational Medicine Challenge data set [9] , radiology reports were assigned codes from the ninth revision of the International Classification of DiseasesClinical Modification (ICD-9-CM) terminology [10] . In similar corpora, chest X-ray reports were manually labelled with any of four pneumonia-related concepts [11] whilst any of 80 possible disease names were assigned to documents in another collection of clinical records [12] with the assistance of automatic tools MetaMap Transfer (MMTx) [13] for concept recognition and NegEx [14] for negation detection. Whilst suitable for evaluating information retrieval methods, such document-level annotations cannot sufficiently support the extraction of phenotypic concepts which are described in clinical records in largely variable ways, making it necessary for automated methods to perform analysis by looking at their actual mentions within text.. Several other clinical corpora were thus enriched with text-bound annotations, which serve as indicators of specific locations of phenotypic concept mentions within text. For instance, all mentions of signs or symptoms, medications and procedures relevant to inflammatory bowel disease were marked up in the corpus developed by South et al [15] . Specific mentions of diseases and signs or symptoms were similarly annotated under the ShARe scheme [16, 17] and additionally linked to terms in the SNOMED Clinical Terms vocabulary [18] . Whilst the scheme developed by [19] had similar specifications, it is unique in terms of its employment of an automatic tool to accelerate the annotation process. One difficulty encountered by annotators following such scheme, however, is with manually mapping mentions of phenotypic concepts to vocabulary terms, owing to the high degree of variability in which these concepts are expressed in text. For instance, many signs or symptoms (e.g., gradual progressive breathlessness), cannot be fully mapped to any of the existing terms in vocabularies.. Alleviating this issue are schemes which were designed to enrich corpora with finer-grained text-bound annotations. The Clinical e-Science Framework (CLEF) annotation scheme [20] which defined several clinical concept types and relationships, required the decomposition of phrases into their constituent concepts which were then individually assigned concept type labels and linked using any of their defined relationships. Also based on a fine-grained annotation approach is the work by Mungall et al. [21] on the ontology-driven annotation of inter-species phenotypic information based on the EQ model [22] . Although their work was carried out with the help of the Phenote software [23] for storing, managing and visualising annotations, the entire curation process was done manually, i.e., without the support of any NLP tools. The effort we have undertaken, in contrast, can be considered as a step towards automating such EQ model-based fine-grained annotation of phenotypic information.. In this regard, our work is unique amongst annotation efforts within the clinical NLP community, but shares similarities with some phenotype curation pipelines employed in the domain of biological systematics. Curators of the Phenoscape project [24] manually link EQ-encoded phenotypes of fishes to the Zebrafish Model Organism Database using Phenex [25] which is a tool for managing character-by-taxon matrices, a formal approach used by evolutionary biologists. To accelerate this process, Phenex has been recently enhanced with NLP capabilities [26] upon the integration of a text analytic known as CharaParser [27] . Based on a combination of bootstrapping and syntactic parsing approaches [28] , CharaParser can automatically annotate structured characteristics of organisms (i.e., phenotypes) in text, but currently does not have full support for linking concepts to ontologies [29] . Also facilitating the semi-automatic curation of systematics literature is GoldenGATE [30] , a stand-alone application modelled after the GATE framework [31] , which allows for the combination of various NLP tools into text processing pipelines. It is functionally similar to our Webbased annotation platform Argo [32] in terms of its support for NLP workflow management and manual validation of automatically generated annotations. However, the latter fosters interoperability to a higher degree by conforming to the industry-supported Unstructured Management Information Architecture [33] and allowing workflows to be invoked as Web services [34] .. By producing our proposed fine-grained phenotype annotations which are linked to ontological concepts, we are representing them in a computable form thus making them suitable for computational applications such as inferencing and semantic search. The Phenomizer tool [35] , for instance, has demonstrated the benefits of encoding phenotypic information in a computable format. Leveraging the Human Phenotype Ontology (HPO) [36] whose terms are linked to diseases in the Online Mendelian Inheritance in Man (OMIM) vocabulary [37] , it supports clinicians in making diagnoses by semantically searching for the medical condition that best matches the HPO signs or symptoms given in a query. We envisage that such an application, when integrated with a repository of phenotypes and corresponding clinical recommendations, e.g., Phenotype Portal [38] and the Phenotype KnowledgeBase [39] , can ultimately assist point-of-care clinicians in more confidently providing personalised treatment to patients. Our work on the annotation of COPD phenotypes aims to support the development of similar applications in the future. An umbrella term for a range of lung abnormalities, chronic obstructive pulmonary disease (COPD) pertains to medical conditions in which airflow from the lungs is repeatedly impeded. This life-threatening disease, known to be primarily caused by tobacco smoke, is not completely reversible and is incurable. COPD was ranked by the World Health Organization as the fifth leading cause of death worldwide in 2002, and is predicted to become the third by year 2030. Estimates have also shown that the mortality rate for COPD could escalate by at least 30% within the next decade if preventive measures are not implemented [1] .. The disease and clinical manifestations of COPD are heterogeneous and widely vary from one patient to another. As such, its treatment needs to be highly personalised in order to ensure that the most suitable therapy is provided to a patient. COPD phenotyping allows for well-defined grouping of patients according to their prognostic and therapeutic characteristics, and thus informs the development and provision of personalised therapy [2] .. The primary approach to recording phenotypic information is by means of electronic clinical records [3] . However, as clinicians at the point of care use free text in describing phenotypes, such information can easily become obscured and inaccessible [4] . In order to expedite the process of identifying a given patient's COPD group, the phenotypic information locked away within these records needs to be automatically extracted and distilled for the clinicians' perusal.. Capable of automatically distilling information expressed in natural language within documents, text mining can be applied on clinical records in order to efficiently extract COPD phenotypes of interest. However, the development of sophisticated text mining tools is reliant on the availability of gold standard annotated corpora, which serve as evaluation data as well as provide samples for training machine learning-based approaches.. This paper presents our ongoing efforts on the annotation of COPD phenotypes in a collection of scientific papers. In our previous publication [5] on which this work is built upon, we proposed to form a corpus of clinical records from the Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC II) Clinical Database [6, 7] . However, our UK-based expert collaborators (i.e., stakeholders who will incorporate our text mining technology into their systems in the near future) recently pointed out that there are substantial discrepancies between the hospital system in the US (on which MIMIC II is focussed) and that in the UK. After considering their advice, we decided to utilise scientific articles from various COPD-relevant journals, rather than build a corpus of clinical records which are highly US-specific. As previous work demonstrated techniques which successfully extracted information from unseen data even if the training/development data used was of a different document type [8] , we believe that a gold standard corpus of full scientific articles should still allow for the development of phenotype extraction tools for clinical records. Nevertheless, our collaborators are still currently working on obtaining a subset of clinical records from their own hospital, which will also be annotated to become part of an augmented version of our corpus.. In embarking on this effort, we are building a resource that will support the development of text mining methods for the automatic extraction of COPD phenotypes from free text. We envisage that such methods will ultimately foster the development of applications which will enable point-of-care clinicians to more easily and confidently identify a given COPD patient's group, potentially leading to the provision of the most appropriate personalised treatment. Furthermore, text mining methods can be employed in order to facilitate the linking of COPD phenotypes with genotypic information contained in published scientific literature.. In the remainder of this paper, we firstly provide a review of the state of the art (Related Work). We proceed to describing our methods for corpus development (Methods), including our strategy for document selection followed by our proposed annotation scheme. A discussion of our text mining-assisted annotation workflow is also provided. We then share the results and analysis of our evaluation (Results and Discussion). Lastly, we conclude the paper with a summary of our contributions and an overview of ongoing and future work. We describe in this section our strategies for collecting documents for the corpus and our proposed annotation scheme. We also elaborate on the technology behind our text mining-assisted annotation methodology.

O O O O B I I I O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


AGGREGATE CERTAINTY ESTIMATORS Selecting an effective method for combining the votes of base inducers in a multiclassifier system can have a significant impact on the system's overall classification accuracy. Some methods cannot even achieve as high a classification accuracy as the most accurate base classifier. To address this issue, we present the strategy of aggregate certainty estimators, which uses multiple measures to estimate a classifier's certainty in its predictions on an instance-by-instance basis. Use of these certainty estimators for vote-weighting allows the system to achieve a higher overall average in classification accuracy than the most accurate base classifier. Weighting with these aggregate measures also results in higher average classification accuracy than weighting with single certainty estimates. Aggregate certainty estimators outperform three baseline strategies, as well as the methods of modified stacking and arbitration, in terms of average accuracy over 36 data sets. In the late eighteenth century, the Marquis de Condorcet composed the Essay on the Application of Analysis to the Probability of Majority Decisions. It outlined the concept now known as the "Condorcet Jury Theorem," the idea that, if each member of a group has a greater than 50% chance of making a correct decision, the probability that a plurality of voters will make the correct decision increases as voters are added. The essay was originally intended to provide a theoretical argument for the benefits of democracy. However, the concept also has application in the field of supervised learning. In theory, a group of classifiers should be better suited to the task of classification. The base classifiers need not even be highly accurate. As long as each classifier can exhibit a better-than-random performance, an ensemble of these classifiers should be able to take advantage of the expertise of each to assign more accurate classifications. Schapire (1990) demonstrated how a collection of "weak" classifiers-ones that perform only slightly better than random guessing-can be combined to produce a classifier with arbitrarily high accuracy. This provided the theoretical basis for his wellknown AdaBoost algorithm, (Freund and Schapire 1996; Schapire and Singer 1998) . Other researchers have proposed similar ensemble-creation strategies, ranging from the simple strategy of bagging (Breiman 1996) to random forests (Breiman 2001) and Bayesian model averaging (Hoeting et al. 1999) . Rokach (2010) outlines four basic building blocks of an ensemble: a labeled training set, a base inducer, a diversity generator, and a combiner. A number of strategies can be used in selecting the base inducers and generating diversity. For example, with the adaptive mixture of local experts strategy (Jacobs et al. 1991) , a gating network determines the probability of selecting the output of one of the base inducers. Delegating (Ferri, Flach, and Hernandez-Orallo 2004) is an approach where a base inducer assigns the final class label to a given instance only if it has highcertainty in that particular class. If it is less confident, the instance is delegated to another base inducer. Diversity can be generated by selecting different subsets of the training set for use in training each of the base inducers.. With bagging (Breiman 1996) , instances are drawn randomly with replacement from the original training set to create a new set with which to train the base inducers. The AdaBoost algorithm (Freund and Schapire 1996; Schapire and Singer 1998) takes into account which instances were misclassified by previously constructed base inducers when selecting data for training subsequent ones. Multiclassifier systems address the problem of diversity generation by using different algorithms for training their base inducers. For example, Ho, Hull, and Srihari (1994) use four different algorithmic approaches to character recognition and discuss methods of combining their outputs.. This work focuses on the fourth aspect outlined by Rokach: combining the outputs of the base inducers. It investigates the possibility that the same principle of combining weak classifiers to produce a strong one can also be applied to the weighting strategies used when combining the votes of those classifiers. While, as in the case of Naïve Bayes, there is often a standard method for estimating confidence in a classifier's predictions, certainty in classification can be estimated in a variety of ways. If each of these methods can demonstrate even a slight tendency for assigning higher certainly values to correctly classified instances as opposed to incorrectly classified instances, they could theoretically be combined to create a more accurate measure of certainty, much as weak inducers in an ensemble can be combine to form a stronger classifier.. These multiple measures are incorporated into the strategy of aggregate certainty estimators. With this technique, the votes of classifiers are weighted by certainty as determined on an instance-by-instance basis. Each classifier is trained using a different algorithm on the same training data set. Then, each instance in the test set is assigned a class value and an overall certainty rating for that classification by each of n classifiers. Multiple factors are taken into consideration when determining this overall certainty rating. For example, six different certainty estimators are used to calculate certainty in the prediction of a decision tree classifier. A given instance would receive six certainty ratings, reflecting properties such as the purity of the leaf node in which it was classified and the number of instances classified at that leaf. These six numbers are then aggregated to produce an overall certainty rating for the decision tree's classification of this particular instance. A similar method is used to calculate an overall certainty rating for each of the classifiers. The class label assigned to the instance is then calculated by summing the weights for each possible label and selecting the class label with the maximum total.. The technique of aggregate certainty estimators is shown to achieve higher average classification accuracy over 36 data sets than the standard combination strategies employed by Bagging and Boosting as well as the SelectBest strategy of allowing the most accurate classifier in the system to make all the classifications. It also outperforms Arbitration (Ortega, Koppel, and Argamon 2001) and the Stacking algorithm presented by Dzeroski and Zenko (2004) in terms of average classification accuracy.. Section 2 of this work provides an overview of related research. Section 3 outlines the aggregate certainty estimators algorithm. Section 4 presents certainty estimators for five common classification algorithms. Section 5 provides results comparing aggregate certainty estimators with standard voting, voting by accuracy, the SelectBest strategy, Arbitration, and Modified Stacking. Section 6 outlines conclusions and suggests options for further research. This work presents a viable new method of combining the outputs of base inducers in a multiclassifier system using multiple certainty estimators to predict certainty in the classification of a given instance. A number of certainty estimators designed for this task are proposed for each of five different types of classifiers. Aggregate measures are shown to be more highly correlated with whether an instance is correctly classified than any of the individual measures. The strategy of aggregate certainty estimators, which employs all of the certainty estimators presented, is shown to achieve a higher average classification accuracy over 36 data sets than five alternate ensemble strategies.. The certainty estimators presented in this work explore some of the strengths and weaknesses of a given classifier on a given data set. This information could result in the development of new algorithms. For example, a new instance-based classifier might be developed in which only instances that were correctly classified in hold-one-out cross-validation would be allowed to vote on the classification of an unseen instance. The probabilities output by a Naïve Bayes classifier might be altered slightly based on information gained through certainty estimators like the ones presented here. Insights gained by observing the behavior of the certainty estimators on various data sets may help target areas of improvement to increase classification accuracy of individual classifiers. Aggregate certainty estimators are able to achieve higher average classification accuracy than any of three standard baseline strategies over the 36 data sets studied. A comparison between Tables 7 and 8 shows that using single certainty estimators in weighting the votes of an ensemble can allow the ensemble to make improvements in average predictive accuracy. Two of the three single certainty estimator ensembles can achieve higher average classification accuracy than a baseline strategy of standard voting. However, the use of these single certainty estimator values is not sufficient to create an ensemble that can produce a higher average predictive accuracy on a level that is statistically significant, so investigation into additional certainty estimators is warranted.. The higher average accuracy of aggregate certainty estimators does come with a higher cost of computation, but for two-thirds of the certainty estimators, the increase in computational complexity is only linear in regards to the size of the data set. The other one-third of the certainty estimators requires a cross-validation strategy in the training set. The computational complexity for these certainty estimators could be reduced substantially by reducing the number of folds used in the calculations. To motivate the need for multiple certainty estimators, the accuracies of different ensembles created with single certainty estimators are tested. Three different options are given for selecting single certainty estimators. The first alternate ensemble is created by using certainty estimators traditionally used in predicting certainty in a classification: * Decision Tree: Purity of classification. * Rule-based classifier: Purity of classification. * Instance-based classifier: Neighbors in agreement. * Naïve Bayes classifier: Probability of class value. * Multilayer perceptron: Activation output.. The next ensemble is also constructed using single certainty estimators to predict certainty. However, in this case, an attempt is made to select more effective certainty estimators. For this ensemble, each algorithm uses the measure most highly correlated with whether or not an instance was correctly classified as the method for predicting certainty: * Decision tree: Correctly classified instances. * Rule-based classifier: Instance is covered by rule. * Instance-based classifier: Highest minus second. * Naïve Bayes classifier: Correctly classified neighbors. * Multilayer perceptron: Correctly classified neighbors.. In addition, the Weka source code provides a way of calculating a probability distribution over possible classes for each instance classification. In most cases, Weka's method of calculating the probability distribution is similar to the first certainty estimator for each classifier presented in this work. Weka makes a few modifications and refinements to these measures. For a third comparison ensemble, the probability distributions predicted by Weka are used to weight the votes of each of the five classifiers.. All these techniques are then compared to the aggregate certainty estimators' strategy of using a larger set of measures to predict certainty. The resulting predictive accuracies, as shown in Table 7 , demonstrate the utility of using more certainty estimators.. An application of the Friedman test reports significant differences in accuracy among the classifiers. (14.19 ∼ χ 2 , D F = 3, p <= 0.003). The Bonferroni-Dunn post-hoc test reveals that the differences in average ranks between aggregate certainty estimators and two of the three other methods exceed the critical difference for significance at a certainty level of 95% (adjusted α = 0.05 4−1 , critical difference = 0.89, mean rank differences: 1.01, 0.85, 0.92). An algorithm-by-algorithm comparison between the various strategies using the Wilcoxon signed-rank test shows that aggregate certainty estimators outperform the other three algorithms at a certainty level of 99% (p values<=0.0001, 0.001, 0.001).. In the next set of experiments, aggregate certainty estimators are compared with several different baseline methods. Once again, overall accuracy for each method is calculated by using 10-fold cross-validation and averaging accuracy over each of the 10-folds. The first is a standard voting method where each classifier in an ensemble votes on the classification of an instance and the votes are weighted equally. The second baseline method weights the votes by the overall accuracy of the classifier on the training data for a given fold of the experiments. The third baseline method, identified here as the SelectBest method, chooses the classifier in the ensemble that achieved the highest accuracy on the training data and uses that classifier alone on the test data.. Aggregate certainty estimators is also compared to the method of Stacking found to be most effective by Dzeroski and Zenko (2004) . In this method, identified as Modified Stacking in the following analyses, the output probabilities of each of the component classifiers are given as input to a set of model trees. Each tree is designed to make a binary decision about a given possible output class, and the ensemble assigns a value to the instance according to which model tree has the highest positive certainty in its prediction. Table 8 shows the results of these comparisons. Again, the Friedman test reports significant differences in accuracy among the classifiers. (15.51 ∼ χ 2 , D F = 5, p <= 0.008). An application of the Bonferroni-Dunn post-hoc test reveals that the differences in average ranks between aggregate certainty estimators and four of the remaining five methods exceed the critical difference for significance at a certainty level of 95% (adjusted α = 0.05 6−1 , critical difference = 1.14, mean rank differences: 1.21, 0.88, 1.24, 1.31, 1.43). An algorithm-by-algorithm comparison between the various strategies using the Wilcoxon signed-rank test shows that aggregate certainty estimators outperform the other five algorithms at a certainty level of 99% (p values< = 0.001, 0.003, 0.014, 0.003, 0.008). One common method of combining votes of base classifiers on an instance-by-instance basis is to use posterior class probabilities (Rokach 2010) . However, most traditional classifiers are not naturally designed to output these probabilities. Thus, statistical or heuristic estimates of how certain a given classifier should be in its classification of an instance can prove useful.. For example, Provost and Domingos (2003) found that using a simple Laplace correction improves probability-based rankings. Specific pruning strategies and bagging techniques also resulted in ranking improvement. Ferri, Flach, and Hernandez-Orallo (2003) found that the performance of these trees could further be improved by using different splitting criteria and a new smoothing technique that considers all the nodes along the classification path from root to leaf. These types of techniques have also been applied to rule-based classifiers. For example, Dzeroski, Cestnik, and Petrovski (1993) used the m-estimate to smooth probabilities and make predictions more effective.. While Naïve Bayes classifiers naturally produce a probability distribution for all class values, Domingos and Pazzani (1997) found that the power of the Naïve Bayes classifier lies more in the ordering of the classes than the actual probabilities predicted. They found that the classifier performs surprisingly well, even when the prior assumption of feature independence is clearly not met. Other researchers, He and Xiaoqing (2007) , introduce a number of smoothing methods that can improve these probability estimates, resulting in more accurate and stable estimates than those that can be achieved with Laplace smoothing.. Unlike Bayesian models, multilayer perceptrons do not calculate class probabilities explicitly, but Ruck et al. (1990) provide a proof that the activation outputs of a multilayer perceptron approximate these probabilities. Richard and Lippman (1991) also found that the accuracy of these probability estimates depended on the network complexity, the amount of training data, and how well the training data represented true a priori class probabilities.. In addition, while combining votes of base inducers according to class probabilities may be an intuitive method, other methods benefit from taking additional factors into consideration. For example, Dolev, Leshem, and Yagel (2010) focus on attributes of the data set when determining how to weight votes. Their algorithm assumes a percentage of corrupted data, and they statistically analyze the data and attempt to remove corrupted data by identifying outliers in the distributions. The estimated distribution parameters are also used to determine the likelihood of feature values in the training set and a corresponding certainty level for leaf nodes in a decision tree where these training set instances are classified. Carney, Cunningham, and Bhagwan (1999) focus on the variances among the distributions of base classifier output when determining how votes should be combined. Ali and Pazzani (1996) compare simple voting to three other evidence combination methods. "Bayesian Combination" approximates the optimal Bayes approach, taking into account both accuracy on training data set and posterior probabilities when weighting rule output. The "Distribution Summation" method takes into account the number of training instances covered by a given rule. "Likelihood Combination" takes both coverage and training set accuracy into account when assigning weights.. In most cases in literature, estimates of certainty are calculated using a single measure, and improvements are aimed at finding ways to smooth and improve this single measure's accuracy. Certainty measures may take into account variables such as the inherent properties of a classifier, how classifiers behave in an overall system, or distributions of attributes in a data set. However, given the variety of things that may be considered when estimating certainty of classification, it stands to reason that an algorithm could greatly benefit from taking multiple variables into account. The certainty estimators incorporated in this work include traditional class probability estimators, but also consider other features discussed by researchers such as data set coverage and distributions of base classifier outputs.. To demonstrate the usefulness of our proposed weighting system, we compare the technique of aggregate certainty estimators to other methods that weight votes of base classifiers on an instance-by-instance basis. Arbitration (Ortega et al. 2001 ) creates a "referee" to determine the certainty that a learning model has in its classification of the various subdomains of a given problem. Information about both the misclassification of instances and the classifiers themselves is used in the development of the metalearner referees. Stacking (Wolpert 1992 ) makes use of a metalevel learning algorithm that discovers the best way to combine outputs from the base-level classifiers. Dzeroski and Zenko (2004) found that the accuracy of an ensemble over a data set is often no better than the accuracy of one of the classifiers contributing to the ensemble. To justify the overhead of creating an ensemble, the ensemble should be able to perform better than a strategy of simply selecting the best classifier by cross-validation. In the algorithms Dzeroski and Zenko explore, only their Modified Stacking strategy was able to consistently achieve this level of performance. We demonstrate that our strategy of aggregate certainty estimators also tends to outperform the single most accurate base classifier in a given ensemble.

O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O B I O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O


Asymptotics for the number of walks in a Weyl chamber of type B Abstract. We consider lattice walks in R k confined to the region 0 < x 1 < x 2 ... < x k with fixed (but arbitrary) starting and end points. The walks are required to be "reflectable", that is, we assume that the number of paths can be counted using the reflection principle. The main results are asymptotic formulas for the total number of walks of length n with either a fixed or a free end point for a general class of walks as n tends to infinity. As applications, we find the asymptotics for the number of k-non-crossing tangled diagrams on the set {1, 2, ..., n} as n tends to infinity, and asymptotics for the number of k-vicious walkers subject to a wall restriction in the random turns model as well as in the lock step model. Asymptotics for all of these objects were either known only for certain special cases, or have only been partially determined or were completely unknown. Lattice paths are well-studied objects in combinatorics as well as in probability theory. A typical problem that is often encountered is the determination of the number of lattice paths that stay within a certain fixed region. In many situations, this region can be identified with a Weyl chamber corresponding to some reflection group. In this paper, the region is a Weyl chamber of type B, and, more precisely, it is given by 0 < x 1 < · · · < x k . (Here, x j refers to the j-th coordinate in R k .) Under certain assumptions on the set of allowed steps and on the underlying lattice, the total number of paths as described above can be counted using the reflection principle as formulated by Gessel and Zeilberger [9] . This reflection principle is a generalisation of a reflection argument, which is often attributed to André [1] , to the context of general finite reflection groups (for details on reflection groups, see [14] ).. A necessary and sufficient condition on the set of steps for ensuring the applicability of the reflection principle as formulated by Gessel and Zeilberger [9] has been given by Grabiner and Magyar [12] . In their paper, Grabiner and Magyar also stated a precise list of steps that satisfy these conditions.. In a recent paper that attracted the author's interest, and that was also the main initial motivation for this work, Chen et al. [ 5, Obervations 1 and 2] gave lattice path descriptions for combinatorial objects called k-non-crossing tangled diagrams. In their work, they determined the order of asymptotic growth of these objects, but they did not succeed in determining precise asymptotics. Interestingly, the sets of steps appearing in this description do not satisfy Grabiner and Magyar's condition. Nevertheless, a slightly generalised reflection principle turns out to be applicable because the steps can be interpreted as sequences of certain atomic steps, where these atomic steps satisfy Grabiner and Magyar's condition. In this manuscript, we state a generalised reflection principle that applies to walks consisting of steps that are sequences of such atomic steps (see Lemma 2.1 below).. Our main results are asymptotic formulas for the total number of walks as the number of steps tends to infinity that stay within the region 0 < x 1 < · · · < x k , with either a fixed end point or a free end point (see Theorem 5.1 and Theorem 6.2, respectively). The starting point of our walks may be chosen anywhere within the allowed region. The proofs of the main results can be roughly summarised as follows. Using a generating function approach, we are able to express the number of walks that we are interested in as a certain coefficient in a specific Laurent polynomial. We then express this coefficient as a Cauchy integral and extract asymptotics with the help of saddle point techniques. Of course, there are some technical problems in between that we have to overcome. The most significant comes from the fact that we have to determine asymptotics for a determinant. The problem here is the large number of cancellations of asymptotically leading terms. It is surmounted by means of a general technique that is presented in Section 4. As a corollary to our main results, we obtain precise asymptotics for k-non-crossing tangled diagrams with and without isolated points (for details, see Section 7). Moreover, we find asymptotics for the number of vicious walks with a wall restriction in the lock step model as well as asymptotics for the number of vicious walks with a wall restriction in the random turns model. Special instances of our asymptotic formula for the total number of vicious walks in the lock step model have been established by Krattenthaler et al. [16, 17] and Rubey [21] . The growth order for the number of vicious walks in the lock step model with a free and point, and for the number of k-non-crossing tangled diagrams has been determined by Grabiner [11] and Chen et al. [4] , respectively. To the author's best knowledge, the asymptotics for the number of vicious walks in the random turns model seem to be new. In some sense, one of the achievements of the present work is that it shows how to overcome a technical difficulty put to the fore in [24] . In order to explain this remark, we recall that Tate and Zelditch [24] determined asymptotics of multiplicities of weights in tensor powers, which are related to reflectable lattice paths in a Weyl chamber (for details, we refer to [12, Theorem 2] ). For the so-called central limit region of irreducible multiplicities (for definition, we directly refer to [24] ) they did not manage to determine the asymptotic behaviour of these multiplicities, and, therefore, had to resort to a result of Biane [2, Théorème 2.2]. More precisely, although they were able to obtain the (indeed correct) dominant asymptotic term in a formal manner, they were not able to actually prove its validity by establishing a sufficient bound on the error term. For a detailed elaboration on this problem we refer to the paragraph after [24, Theorem 8] . The techniques applied in [24] are in fact quite similar to those applied in this manuscript (namely, the Weyl character formula/reflection principle and saddle point techniques). However, it is the above mentioned technique presented in Section 4 which forms the key to resolve the problem by providing sufficiently small error bounds in situations as the one described by Tate and Zelditch [24] .. The paper is organised as follows. In the next section, we give the basic definitions and precise description of the lattice walk model underlying this work. We also state and prove a slightly generalised reflection principle (see Lemma 2.1 below) that can be used to count the number of lattice walks in our model. At the end of this section, we prove an exact integral formula for this number. In Section 3, we determine the possible step sets the walks in our model may consist of. Additionally, we state and prove some technical results. This allows us to give the proofs of our main results in a more accessible manner. Section 4 presents a factorisation technique for certain functions defined by determinants. These results are crucial to our proofs since they enable us to determine precise asymptotics for these functions. Our main results, namely asymptotics for total number of random walks with a fixed end point and with a free end point, are the content of Section 5 and Section 6, respectively. The last section presents applications of our main results, namely Theorem 5.1 and Theorem 6.2. Here we determine asymptotics for the number of vicious walks with a wall restriction in the lock step model as well as asymptotics for the number of vicious walks with a wall restriction in the random turns model. Furthermore, we determine precise asymptotics for the number of k-non-crossing tangled diagrams on the set {1, 2, . . . , n} as n tends to infinity. This generalises results by Krattenthaler et al. [16, 17] and Rubey [21] . Additionally, we provide precise asymptotic formulas for counting problems for which only the asymptotic growth order has been established. In particular, we give precise asymptotics for the total number of vicious walkers with wall restriction and free end point, as well as precise asymptotics for the number of k-non-crossing tangled diagrams with and without isolated points. (The growth order for the former objects has been established by Grabiner [11] , whereas the growth order for the latter objects has been determined by Chen et al. [5] .)

B O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O