Ensemble pruning via individual contribution ordering An ensemble is a set of learned models that make decisions collectively. Although an ensemble is usually more accurate than a single learner, existing ensemble methods often tend to construct unnecessarily large ensembles, which increases the memory consumption and computational cost. Ensemble pruning tackles this problem by selecting a subset of ensemble members to form subensembles that are subject to less resource consumption and response time with accuracy that is similar to or better than the original ensemble. In this paper, we analyze the accuracy/diversity trade-off and prove that classifiers that are more accurate and make more predictions in the minority group are more important for subensemble construction. Based on the gained insights, a heuristic metric that considers both accuracy and diversity is proposed to explicitly evaluate each individual classifier's contribution to the whole ensemble. By incorporating ensemble members in decreasing order of their contributions, subensembles are formed such that users can select the top p percent of ensemble members, depending on their resource availability and tolerable waiting time, for predictions. Experimental results on 26 UCI data sets show that subensembles formed by the proposed EPIC (Ensemble Pruning via Individual Contribution ordering) algorithm outperform the original ensemble and a state-ofthe-art ensemble pruning method, Orientation Ordering (OO) [16] . The construction of classifier ensembles is an active research field in machine learning and data mining communities [3] . Rather Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. than relying on a single classifier, an ensemble is a set of classifiers that make decisions collectively. It is well accepted that an ensemble usually generalizes better than a single classifier [22] . Dietterich stated [8] "A necessary and sufficient condition for an ensemble of classifiers to be more accurate than any of its individual members is if the classifiers are accurate and diverse." Since the diversity of the ensemble decreases with the increase in accuracies of ensemble members, the key to the success of any ensemble learning method is the appropriate handling of the trade-off between accuracy and diversity.. Many approaches have been proposed to create accurate and diverse ensembles. Examples include bagging [4] , boosting [11] , random forests [5] , the random subspace method [12] and random decision trees [10] . In most ensemble methods, the diversity and accuracy are acquired by manipulating subsets of data points or features. One problem with these ensembling approaches is that they tend to construct unnecessarily large ensembles, which requires a large amount of memory to store the trained classifiers and decreases the response time for prediction. Ensemble pruning, or selective ensembles, is a technique that tackles this problem by choosing a subset of individual classifiers from a trained ensemble to form a subensemble for prediction. The classifiers in the subensemble need to be carefully chosen so that it is small enough to reduce the memory requirement and response time with predictive accuracy that is similar to or better than the original ensemble.. Given an ensemble of size M , the problem of finding the subset of ensemble members with the optimal generalization ability involves searching the space of 2 M − 1 non-empty subensembles, which was proved to be an NP-complete problem [19] . Like other approaches in ensemble learning, the performance gain of the ensemble pruning methods stems from the balanced accuracy/diversity trade-off, where choosing only the most accurate individual classifiers to form the subensemble is theoretically unsound. As an informative example, Zhang et al. [23] has demonstrated that "an ensemble of three identical classifiers with 95% accuracy is worse than an ensemble of three classifiers with 67% accuracy and least pairwise correlated error (which is perfect!)." At the other extreme, as a strategy that only considers diversity for pruning, Kappa pruning [14] was shown to have "the poorest overall performance on the data sets investigated" of the ensemble pruning methods compared in [15] .. Many approaches [15] exist for selecting good subensembles to achieve the accuracy gain and reduce the computational cost, where existing efforts roughly fall into the following two categories: (1) regarding ensemble pruning as a mathematical programming and optimization problem [23] [25] ; and (2) reordering the original ensemble members based on some predefined criteria [16] , such as the classifiers' prediction accuracies, and selecting a subset of ensem-ble members from the sorted list. For solutions in the first category, the selected subensembles are unscalable to the users' request on the subensemble size, whereas for solutions in the second category, they fail to effectively integrate both accuracy and diversity, which are two critical measures for ensemble pruning. Indeed, while the correlation between diversity and ensemble learning have been observed for many years [13] , the emphasis has been traditionally on the characterization of the diversity and the creation of the diverse classifiers. We currently lack effective pruning principles to assess the amount of "contribution" that each ensemble member can bring to the ensemble. Existing approaches fall short in answering some fundamental questions, such as, given two ensemble members, (1) what are the differences between their impact to the ensemble? (2) which one is better for ensemble learning if we have to prune out one of them? and (3) what are the effective criteria if both accuracy and diversity are considered for ensemble pruning?. To address the above concerns, we report, in this paper, our recent work on individual contribution ordering for scalable ensemble pruning. We argue that although obtaining the optimal ensemble pruning is NP-complete, two general properties can still be observed: (1) when the accuracies of individual members of two ensembles are similar, the more diverse ensemble should perform better than the other one; and (2) when two ensembles are similarly diverse, the one with more accurate individual members should perform better. To assert that these two properties are genuinely true for ensemble construction, we limit our analysis, in the paper, to a disagreement based diversity measure [12] and two-class learning problems and carry out a theoretical study to prove the validity of the first property. Because a similar proof can also be obtained for the second property, our theoretical analysis concludes that an individual classifier makes a more important contribution to the ensemble not only when it is more accurate, but also when its predictions more often disagree with the majority vote of the ensemble.. Based on this insight, a heuristic metric for evaluating contribution of each ensemble member that explicitly considers both accuracy and diversity is proposed in this paper. By reordering the ensemble members according to the proposed metric, an algorithm named Ensemble Pruning via Individual Contribution ordering (EPIC) that incorporates ensemble members with decreasing order of individual contributions into subensembles is introduced. Experimental results on 26 UCI data sets show that subensembles formed by EPIC are significantly more accurate than the original ensembles. In all experiments, bagging is chosen to be the method for constructing the original ensemble because it has been shown to be a safe and robust method [7] . To compare EPIC with peer ensemble pruning methods, Orientation Ordering (OO) [16] is chosen to be the control case, because it was shown to exhibit good performance with a low computational cost in [15] . Our experiments show that EPIC outperforms OO with a similar pruning time, which is O(M log(M )) for an ensemble with size M .. The remainder of the paper is structured as follows. Section 2 reviews related work. Section 3 carries out theoretical study on the accuracy/diversity trade-off, proposes a metric for evaluating individual contribution, and introduces the EPIC algorithm. Section 4 reports the experimental settings and results, and we conclude the paper in Section 5. In this paper, an effective ensemble pruning method termed Ensemble Pruning via Individual Contribution ordering (EPIC) is introduced, which orders individual classifiers in an ensemble in terms of their importance to subensemble construction. We argued that although obtaining the optimal ensemble pruning solution is NPcomplete, some general properties can still be derived for ensemble pruning. To achieve the goal, we limited the scope of our analysis to pair-wise diversity and two-class learning problems and have proved some general principles for ensemble pruning. Based on the derived principles, the proposed EPIC algorithm can sort ensemble members into a list, through which a subensemble with arbitrary size can be formed by selecting the top p percent of ensemble members. Experimental comparisons on 26 UCI data sets showed that EPIC outperforms both the original ensemble and a state-of-the-art ensemble pruning method, OO [16] .. Although the original goal of ensemble pruning methods is to reduce the memory consumption and the computational cost with minimum impact on predictive accuracy, subensembles constructed with recent pruning methods such as OO and the method introduced in this paper are consistently more accurate than whole ensembles, which suggests that it is practically unnecessary, even without the resource consumption concerns, to keep all ensemble members for predictions.. In the paper, we limited our scope to specific diversity measures and two-class learning problems, and used Bagging to generate the initial pool of classifiers. But the general principles derived in the paper are valid for broad diversity measures, multi-class learning problems, and any other ensemble methods. Future work will involve testing EPIC's performance when working with other ensemble methods, designing better individual contribution measures and automating the selection of the sizes of the subensembles. Ensemble methods usually have two stages [16] : (1) training a set of accurate and diverse classifiers; and (2) combining individual classifiers, using strategies such as majority voting [4] , weighted voting [11] or stacking [21] , for predictions. Ensemble pruning can be viewed as a special case of the weighted voting approach, where each individual member is associated with a binary number indicating whether or not it should be included in the predicting process [23] . The major difference between ensemble pruning and other ensemble methods is that its main focus is to reduce memory and computational costs by choosing small subensembles for predictions, as well as maintaining or increasing the predictive accuracy.. In 1997, Margineantu and Dietterich [14] observed that although ensemble methods can achieve better generalization accuracy than a single learner, the resulting ensembles are often unnecessarily large. Sometimes the memory for storing the ensemble members is even larger than the size of the training set. A question was then raised: are all classifiers generated essential for the ensemble's performance? Through investigation of five pruning heuristics, they concluded that the size of the ensemble can be substantially reduced without serious impact on performance, and sometimes the pruned ensemble can even outperform the original ensemble. Tamon and Xiang [19] proved the problem of pruning boosting ensembles to be NP-complete.. In the work carried out by Zhou et al. [25] , analysis showed that ensembling a subset of classifiers could be better than using all the classifiers. Further, a genetic algorithm (GA) based algorithm called GASEN was proposed to evolve the voting weights of individual classifiers in an ensemble of neural networks. Only the classifiers with weights above a certain threshold were chosen to form the subensembles for prediction. Experimental results of GASEN confirmed the conclusion that ensembles can be pruned to yield similar or better performance. A similar approach was used to prune ensembles of decision trees [24] .. Zhang et al. [23] formulated the ensemble pruning problem as a quadratic integer programming problem, which is NP-hard. By applying semi-definite programming techniques, approximate solutions can be obtained in polynomial time for this NP-hard problem. Another work [6] first obtained an extensive library of more than 2000 classifiers trained by existing methods such as C4.5 [18] with different parameters, and then ensembles were constructed by selecting classifiers using metrics such as accuracy and ROC area.. The approach this paper takes belongs to a family of ensemble pruning methods based on ordered aggregation, where the original ensembles are reordered according to some criteria, and subensembles are constructed according to this order. In ensemble learning methods, the generalization error generally monotonically decreases with the increase of the ensemble size. Exceptions are when applying algorithms such as boosting to noisy data sets, the generalization error might increase because of overfitting. By reordering ensemble members, highest accuracy can be obtained for subensembles with an intermediate number of classifiers. A representative of this approach is [16] , in which classifiers were reordered by increasing values of angles between the signature vectors (binary vectors that indicate each classifier's correctness in predicting each data point) and a reference vector. The algorithm introduced here differs from this work in that EPIC explicitly considers both accuracy and diversity.. In [17] , Partalas et al. identified that an ensemble member's prediction on one data point can be divided into four subsets: (1) the subset in which the individual classifier predicts correctly and is in the minority group; (2) the subset in which the individual classifier predicts correctly and is in the majority group; (3) the subset in which the individual classifier predicts incorrectly and is in the minority group; and (4) the subset in which the individual classifier predicts incorrectly and is in the majority group. The paper concluded that considering all four cases is crucial to designing ensemble diversity measures and ensemble selection techniques.. Although the designing of EPIC is also based on the four cases, it differs from [17] in two ways. Firstly, while both algorithms agree that considering the four cases is crucial, a more important question is understanding the relationship among the four cases and using it for pruning. [17] made an intuitive observation and concluded that ensemble members near to change status (correct/incorrect classification) are more important, which did not show the intrinsic relationship among the four cases. In this paper however, our theoretical analysis shows an unintuitive conclusion that ensemble members with more correct predictions in the minority contribute more for subensemble construction, and ensemble members with more incorrect predictions in the minority are less harmful for a subensemble's predictions. Thus EPIC is different from [17] because we show for the first time a strict order of importance for the four cases. Secondly, [17] is an iterative algorithm with a polynomial pruning time, while EPIC's pruning time is O(M log(M )) for an ensemble with size M .. [2] is another work that considers the four cases. Like [17] , [2] did not show the relationship among the four cases, and its pruning time is polynomial.. Recent work [15] gives a review and comparative study of the ensemble pruning literature.

B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O


A rate control scheme for adaptive real-time applications in IP networks with lossy links and long round trip times Abstract-Currently there is no control for real-time traffic sources in IP networks. This is a serious problem because real-time traffic can not only congest the network but can also cause unfairness and starvation of TCP traffic. However, it is not possible to apply current solutions for Internet to the networks with high bandwidth-delay products and high bit error rates. The channel errors may result in inaccurate congestion control decisions and unnecessary rate throttles leading to severe performance degradation. This problem is amplified in the links with high bandwidth-delay products, since the link is inefficiently utilized for a very long time until the unnecessary rate throttle is recovered. In this paper, a new Rate Control Scheme, RCS, is introduced for real-time interactive applications in networks with high bandwidth-delay products and high bit error rates. RCS is based on the concept of using dummy packets to probe the availability of network resources. Dummy packets are treated as low priority packets and consequently they do not affect the throughput of actual data traffic. Therefore, RCS requires all the routers in the connection path to support some priority policy. A new algorithm is also proposed to improve the robustness of the RCS to temporal signal loss conditions. The delay-bound considerations for real-time traffic sources using RCS rate control scheme are also investigated. Simulation experiments show that in environments with high bandwidth-delay products and high bit error rates, RCS achieves high throughput performance without penalizing TCP connections.Index Terms-Flow control, high bandwidth-delay products, high bit error rates, real-time protocols. In this paper we introduced RCS, a new rate control scheme for real-time traffic in networks with high bandwidth-delay products and high link error rates. RCS improves throughput using low priority dummy packets to probe available network resources. Therefore, RCS requires the routers along the connection to implement some priority mechanism. 3 The main feature of RCS is that it is an end-to-end protocol, i.e., it needs to be implemented only at the source and destination. RCS is a TCP-friendly rate control scheme, which halves its transmission rate in case of packet loss. RCS can then resume its original rate very rapidly in case of packet losses due to link errors. If the packet loss is due to congestion, RCS follows TCP-friendly behavior rules and increases transmission rate additively after rate halving. For temporal signal loss situations, RCS avoids unnecessary rate throttles and resumes its original transmission rate very rapidly after signal is back. Simulation results show that RCS achieves high performance in terms of throughput and fairness for real-time multimedia applications in networks with high bandwidth-delay product and high link error rate while providing TCP-friendly behavior. In this section, we report the related work provided in the literature; i.e., rate control schemes proposed for multimedia traffic, transport layer solutions proposed to address the performance degradation due to high link error rate, and the existing link probing mechanisms.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B I I I O O O O O O O O O O O B I I B I I I B I O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Throughput-Delay Tradeoff for Hierarchical Cooperation in Ad Hoc Wireless Networks Abstract-Hierarchical cooperation has recently been shown to achieve better throughput scaling than classical multihop schemes under certain assumptions on the channel model in static wireless networks. However, the end-to-end delay of this scheme turns out to be significantly larger than those of multihop schemes. A modification of the scheme is proposed here that achieves a throughput-delay tradeoff D(n) = (log n) 2 T(n) for T(n) between 2( p n=log n) and 2(n= log n), where D(n) and T(n) are respectively the average delay per bit and the aggregate throughput in a network of n nodes. This tradeoff complements the previous results of El Gamal et al., which show that the throughput-delay tradeoff for multihop schemes is given by D(n) = T(n) where T(n) lies between 2(1) and 2( p n).Meanwhile, the present paper considers the network multipleaccess problem, which may be of interest in its own right. The present paper shows that hierarchical cooperation not only can lead to higher throughput in ad hoc networks, but also to reasonable end-to-end delay, given that some extra care is taken in setting up cooperation at the lower levels and scheduling communications. Meanwhile, we have discussed the network multiple-access problem in the present paper, which is of interest in its own right.

B I O B I O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Detection of event-related potentials in individual subjects using support vector machines Abstract Event-related potentials (ERPs) are tiny electrical brain responses in the human electroencephalogram that are typically not detectable until they are isolated by a process of signal averaging. Owing to the extremely small size of ERP components (ranging from less than 1 lV to tens of lV), compared to background brain rhythms, statistical analyses of ERPs are predominantly carried out in groups of subjects. This limitation is a barrier to the translation of ERP-based neuroscience to applications such as medical diagnostics. We show here that support vector machines (SVMs) are a useful method to detect ERP components in individual subjects with a small set of electrodes and a small number of trials for a mismatch negativity (MMN) ERP component. Such a reduced experiment setup is important for clinical applications. One hundred healthy individuals were presented with an auditory pattern containing pattern-violating deviants to evoke the MMN. Two-class SVMs were then trained to classify averaged ERP waveforms in response to the standard tone (tones that match the pattern) and deviant tone stimuli (tones that violate the pattern). The influence of kernel type, number of epochs, electrode selection, and temporal window size in the averaged waveform were explored. When using all electrodes, averages of all available epochs, and a temporal window from 0 to 900-ms post-stimulus, a linear SVM achieved 94.5 % accuracy. Further analyses using SVMs trained with narrower, sliding temporal windows confirmed the sensitivity of the SVM to data in the latency range associated with the MMN. Event-related potential (ERP) components are specific deflections in the averaged waveform, which are considered to be tied to discrete stages of neural processing. They are extracted from trials or epochs of electroencephalogram (EEG) data that are temporally locked to a repeated event, either a stimulus or a behavioural response, which are segmented from the continuous EEG recording and then averaged together. This averaging process reduces the amplitude of brain activity that is unrelated to the event, thus presumed to be random with respect to it in time, and retains brain responses that are temporally related to the event (see [1] for a full description of ERP derivation). The most commonly used definition of an ERP component was outlined by Näätänen and Picton in 1987 [2] . Their definition emphasizes that an ERP component is described not only by the brain regions that contribute to its production, but also by the experimental parameters which must be satisfied in order to observe the response. ERPs can be evoked by stimulation in any sensory modality and can also occur in response to motoric and cognitive events.. In the auditory domain, ERPs can be used to index processing all the way from early sensory processing in the brainstem to cortical language processing. The auditory brainstem response (ABR) has a long history of use in both neurology and audiology [3] . Cortical ERPs have yet to become a routine part of clinical practice, even in audiology (reviewed in [4] ) and cognitive assessment of higher functions like language [5] [6] [7] where their potential utility is very clear.. In large part, the failure of cortical ERPs to be translated into clinical practice is due to the requirements of the signal-averaging process. Many repetitions of the evoking stimulus or event must be presented in order to isolate ERPs, which can lead to long testing times. Another important consideration for the application of ERPs for both clinical and research purposes is the need to statistically verify the presence of the response. Given the low signal-to-noise ratio of these responses versus background EEG, the typical approach in ERP research is to perform univariate statistics across groups, or across experimental conditions, using amplitudes that are measured in the averaged waveform [8] .. Many groups have proposed solutions to verify the presence of ERPs in individual subjects, and even single epochs, by applying techniques such as wavelet analysis [9] [10] [11] , independent component analysis [12] , integrated waveforms [13] , and nonparametric analyses [14] . In some very specific clinical situations, the approach of repeating stimulation until a statistical threshold is reached [15] , or using basic t-scores to evaluate the presence or the absence of a particular component [16, 17] , has been examined.. In particular, the field of Brain-Machine Interfaces (BMIs) has yielded much progress in the single-subject detection of responses like ERPs. This area seeks to use brain imaging data, mostly noninvasive technology, such as EEG, to identify brain waves in continuous EEG recordings that can be interpreted as commands by a computer [18] . The progress that has been seen in this area is mainly due to advances in machine learning techniques. While results in BMI research show that signals can be extracted from single trials, here we investigate specifically the use of machine learning techniques for the detection of ERPs in individual subjects for the purpose of clinical evaluations. Support vector machines (SVMs) are evaluated for their applicability as part of the protocol known as the Halifax Consciousness Scanner [19] . Our approach is hence mediating between a classical clinical approach and methods of BMIs.. The SVMs are a form of supervised statistical machine learning (for a primer, see [20] ) and were first described by Vapnik and colleagues in 1992 [21] . When classified cases are used to train an SVM, it derives an optimal function based on the features that separate the two classes. Unlike other statistical classification methods, such as discriminant function analysis and linear regression, SVMs derive their decision function from only a subset of the data-the cases that are difficult to classify. These cases are referred to as support vectors. The goal of an SVM is to maximize the distance between the decision function's boundary and the support vectors. They can operate in either a linear or a nonlinear fashion and are notable among machine learning techniques in view of their ability to produce generalizable models from small datasets [22] .. The SVMs have demonstrated very high accuracy for discriminating between experimental conditions [23] , and moderately high accuracy for the prediction of clinical group membership [8] based on ERP features. The present work arose from the development of a novel method for evaluating brain function using ERPs, known as the Halifax Consciousness Scanner (HCS; described in [19] ). The initial intent of the HCS is for applications in brain-injured populations. This context presents very different challenges from BMIs because patients may be unable to comply with task instructions [24] . It therefore uses stimulation parameters that do not require effortful task completion. This approach is also beneficial for clinical populations that may seek to influence their test results to conceal, augment, or even fake an injury [25] . Yet another important feature that distinguishes clinical applications from typical SVM applications in BMIs is the need for rapid ERP detection, without any kind of training period for the classifier. For this reason, we apply SVM training across subjects, rather than in the within-subject fashion that is typical for BMIs.. To evaluate the ability of an SVM to verify ERPs in individual subjects, the present study uses an ERP known as the mismatch negativity (MMN). First discovered by Näätänen and colleagues in 1978, the auditory MMN is an ERP that indexes the detection of change in a sequence of sounds [26] . The MMN is linked to a level of auditory memory that is believed to form the basis of conscious perception [27] , and is an important part of the HCS method [19] , particularly due to its well-established utility for coma monitoring [15, [28] [29] [30] [31] [32] . Owing to its small size, the MMN presents a challenge for detection at the individual subject level.. The MMN is commonly evoked using an experimental design known as an oddball paradigm. Although it is well understood that the oddball paradigm frequently causes an undesirable overlap of other ERPs with the MMN [33] , it provides a simple framework for understanding how the MMN is obtained. In the oddball paradigm, a sequence of identical ''standard'' stimuli is interrupted at random and unpredictable times by ''deviant'' stimuli that possess some changed features, such as tonal frequency, intensity, or duration. The MMN is observed as an enhancement in negative voltage that occurs approximately 100-250 ms from deviant stimulus onset. It rests atop other ERP components that are associated with sensory processing, such as the ''obligatory'' P1-N1-P2 complex, and is isolated from these responses through a process of subtraction in which the averaged waveform to the standard stimuli is subtracted point-by-point from the averaged waveform to the deviant stimuli. This produces the so-called ''difference wave,'' in which the MMN appears as a negative deflection in the 100-250 ms range. Although the amplitude of the MMN can be enhanced by experimental parameters such as reducing the silent period between stimuli (the interstimulus interval, or ISI), or increasing the difference between the standard and deviant tones [34] , it is often quite small in amplitude, being around 1 lV.. The MMN is not restricted to simple oddball deviance and can also be evoked by more complex violations of concrete or abstract rules [35] . Under these circumstances, a pure MMN should be obtained, which is free of overlap from other ERP components. In the present study, the MMN is evoked in response to violations of a simple pattern of two alternating tones. Two types of SVM analyses are performed: one to achieve high classification accuracy (SVM acc ), and the other to help one clarify whether the waveform features that are associated with the MMN are indeed used by the SVM (SVM time ). The SVM acc analysis explores a variety of factors which could impact classifier accuracy: number of epochs entered into the averaged waveforms, scalp electrode selection, kernel type, and temporal window selection.. It is generally accepted that, as the number of epochs entered into the ERP waveform increases, the SNR of the ERP improves versus background EEG [1] . For the MMN, the number of epochs that is generally recommended for averaging is at least 150 [36] , although robust MMNs have been observed with approximately 60 [34] . For this reason, the first SVM analysis explores the influence of number of epochs entered into the average on the accuracy of the classifier.. The typical scalp distribution for the MMN is maximal at frontal sites, and inverted at sites below the Sylvian fissure when a nose reference is used. This distribution has been attributed to bilateral, vertically oriented cortical generators on the supratemporal plane, as well as a frontal source [37] . Thus, a classifier that is limited to frontal sites might achieve the highest performance. However, for clinical populations such as patients with acquired brain injuries, atypical scalp distributions are expected [38] . For this reason, the value of using scalp electrode sites separately, or combined, is evaluated.. The so-called kernel trick permits SVMs to operate in a nonlinear fashion by projecting low-dimensional data into higher-dimensional space [20] . This may provide a better fit to complex datasets, but this improved accuracy may come at the cost of overfitting-a lack of generalizability of the results to new datasets [20] . For this reason, a linear classifier may be preferable, but the values of different kernel types are examined to determine whether significant improvements in performance can be achieved over the linear solution.. The MMN is expected to occur approximately 100-250 ms from stimulus onset [36] ; however, other information in the ERP waveforms may be useful for classification. The portion of the waveform that is entered into the feature vectors is therefore varied, to determine what temporal window produces the maximum performance. A difficulty in the interpretation of SVMs, however, is that they derive their rules using a complex combination of factors. Any waveform features that are within the selected temporal window, be they part of the MMN or not, can contribute to the decision function. Thus, the SVM time analysis is performed, not to achieve high accuracy, but to provide finer detail regarding which temporal regions of the waveforms contain detectable information for classification. The tone sequence evoked a moderate-sized, frontally maximal MMN component that inverted in polarity at the mastoid site. This scalp distribution, though observed on a reduced array of sensors, conforms with the classic 3 Grand average waveforms for standard tones (black), repetition deviants (gray), and the deviant minus standard difference wave (dotted). All amplitudes are in microvolts and all time values are in milliseconds. A sharp hardware-related deflection was observed at stimulus onsets (i.e., at 0 ms and every 265 ms thereafter). The MMN is indicated at the Frontal and Mastoid sites by an arrow. In the standard stimulus waveforms, obligatory auditory responses including P1, N1, and P2 were apparent. Because of the rapid rate of stimulus delivery, these took on the appearance of a steady state response Detecting ERPs using SVMs 7 morphology of the MMN and is identical to that observed using similar pattern paradigms [40] . The SVM acc analysis demonstrated that standard tone waveforms and deviant tone waveforms could be classified with 94.5 % accuracy (as shown in Fig. 6 for the all sites feature vector at window size 900 ms). These results bode well for the use of SVMs to detect ERPs for clinical applications, such as the ERP test battery approach that has been developed by our group, known as the Halifax Consciousness Scanner (described in [19] , and a companion paper, [14] .. The performance of the SVM classifier was affected by a number of factors. First, the number of epochs used in the standard and deviant tone averaged waveforms was explored using a linear SVM. The accuracy of the classifier increased directly as a function of the number of epochs entered into the average. This result is entirely consistent with the general belief that the averaging process increases the SNRs of ERPs [1] . There was a consistent trend in which accuracy increased as the number of epochs entered into the average was increased from 2 to 40. The effect of Random Trials Fig. 4 Accuracy of a linear SVM as a function of the number of standard and deviant tone epochs entered into the average (using all scalp sites). Accuracy increased as the number of epochs entered into the average was increased from 2 to 40, but increased much more steeply in the serial-than the random-selection condition, and reached a higher maximum when 40 epochs were used (78.13 %, SD = 0.02 % for serial selection, versus 65 %, SD = 0.043 % for random selection). number of epochs on the accuracy of the SVM was much more dramatic for serially than randomly selected epochs, and achieved a higher accuracy when using 40 epochs in the serial condition. These results suggest that sequential effects that emerge over the stimulation protocol may have contributed to the accuracy of the classifier.. The effect of averaging up to 67 epochs (the full available set) could not be explored across the group, due to the availability of less than 67 deviant epochs in some subjects as a result of artifact rejection. Nevertheless, the full set of available epochs was used for each subject in the final SVM solution, which achieved an accuracy of 94.5 %. Although the recommended number of epochs for MMN derivation is 150, with a typical recording time of 6-12 min [36] , the very high accuracy that was observed here with only 67 deviants, presented in an optimized 2.5-min sequence, is promising for clinical applications. While optimal recording conditions may be difficult to achieve, particularly in a clinical environment, the relatively small amount of data required for classification using this method represents a significant reduction in testing time versus conventional stimulation parameters. Short testing times are necessary in a clinical context because they interfere minimally with routine care procedures and avoid fluctuations in vigilance, attention, and fatigue which can cause false negatives on ERP-based tests [42] .. Other factors that influenced SVM accuracy were the type of kernel (linear, RBF, cubic, or quadratic), the temporal window used, and the electrode site used. When all kernel types were explored across different temporal windows, using a feature vector with data from all sites, the linear SVM clearly outperformed other kernel types. This effect was particularly noticeable as the temporal window size approached 200 ms-the period in which the MMN begins to emerge. At the maximum window size, 900 ms, Detecting ERPs using SVMs 9 the linear SVM performed significantly better than all other kernel types, particularly the RBF, which never rose above 59 %. It remains possible that optimizing the parameters of each SVM kernel type could improve performance for the nonlinear classifiers. More complex kernel types, however, can lead to overfitting, and a lack of generalizability [20] . For this reason, the effect of parameter optimization was not explored for the nonlinear classifiers. The electrode site that was entered into the feature vector also had a significant impact on the accuracy of the classifier. In line with the known scalp distribution of the MMN [37] , feature vectors from the Frontal and Central site produced significantly better results than ones from the Parietal site. Similarly, feature vectors containing data from all three scalp sites produced very high performance, which was not significantly different from the classifiers that were trained exclusively with data from the Frontal and Central sites. This approach may be preferable in clinical populations whose scalp distributions can vary from that typically observed in healthy individuals.. For the linear SVM, increasing the width of the temporal window entered into the feature vector had a direct impact on the accuracy of the classifier, in which increasing the width of the window systematically improved performance. The most dramatic improvement in accuracy occurred around 200 ms, near the peak of the MMN in the grand average (193 ms). Maximum accuracy was observed when the full post-stimulus epoch was utilized (900 ms).. One drawback of SVMs for classification, as opposed to other statistical classifiers like discriminant function analysis, is that their decision functions use complex combinations of information that can make their interpretation difficult. Thus, it is unknown which ERPs or brain oscillations that lie in the employed temporal window are contributing to the performance of the classifier.. To gain insight into the waveform features that contribute most to classification accuracy, a secondary analysis (SVM time ) was performed. Of the 46 SVMs trained in nonoverlapping 20-ms (5 sample) windows, those in the 160-260 and 720-820 ms range achieved the highest accuracy (first period maximum = 86.5 %, SD = 0.1 %, second period maximum = 83.5 %, SD = 0.058 %). The first period corresponds well with the expected latency of the MMN to the pattern deviant, which peaked on average at 193 ms. The second period corresponds with a region of positivity in the deviant tone response, which could reflect a P3-type response. While both MMN and P3 indicate that deviance detection has occurred, P3 is linked to subsequent cognitive operations like attention switching and working memory updating [43] . Regardless of the true component structure of this positive response, it is clear that a classifier which uses this portion of the waveform does not detect the MMN per se, but rather uses a larger variety of waveform features to perform the task of classifying waveforms associated with deviance detection.. In conclusion, linear SVMs proved to be effective at classifying the MMN. This method could be extremely valuable for clinical applications of the MMN, and other ERPs, by both reducing testing time versus conventional stimulation parameters and permitting the application of ERPs at the single subject level. Among the potential applications for the MMN is monitoring the status of severely brain-injured patients [15, 19, 28, 32] . Further development for use in this patient population may require additional testing to determine efficacy in the context of the severely altered EEG patterns that are commonly observed in this group [38] . Results must also be interpreted with care when applied in the absence of more concrete indicators of neurological status [44] [45] [46] [47] .

O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Applying query structuring in cross-language retrieval We will explore various ways to apply query structuring in cross-language information retrieval (CLIR). In the fist test, English queries were translated into Finnish using an electronic dictionary, and were run in a Finnish newspaper database of 55, 000 articles. Queries were structured by combining the Finnish translation equivalents of the same English query key using the syn-operator of the InQuery retrieval system. Structured queries performed markedly better than unstructured queries. Second, the effects of compound-based structuring using a proximity operator for the translation equivalents of source language compound components were tested. The method was not useful in syn-based queries but resulted in decrease in retrieval success. Proper names are often non-identical spelling variants in different languages. This allows n-gram based translation of names not included in a dictionary. In the third test, a query structuring method where the Boolean and-operator was used to assign more weight to keys translated through n-gram matching gave good results. Cross-language information retrieval (CLIR) refers to an information retrieval task where the language of queries is other than that of the retrieved documents. A user may present a query in his or her native language, and in response the system retrieves documents in another language. For an overview of CLIR approaches, see (Hull and Greffenstette, 1996; Oard and Diekema, 1998; Pirkola et al., 2001 ). In dictionary-based cross-language retrieval, queries are translated by means of electronic dictionaries by replacing source language query keys with their target language equivalents.. In this paper we will study three different conditions in which query structuring could be applied in CLIR to improve retrieval effectiveness. The basic idea of query structuring is to group query keys and to use query operators in such a way that more weight is being assigned to important or correct keys than the other keys. To implement this, there has to be some indicator available to show the probability of a key being a good or bad key. Two factors implicitly present in dictionaries are assumed to indicate key goodness. First, it is assumed that important keys often have 1-2 translation equivalents while less important have several equivalents. Second, proper names often are important keys but general dictionaries do not generally include proper names. Therefore the absence or presence of a key in a dictionary is regarded as another indicator. Pirkola (1998) and Pirkola et al. (2002) studied the effects of query structure on CLIR performance in cross-language retrieval where the source language was Finnish and the target language English (Fin-Eng CLIR). Queries were structured by grouping together the English equivalents of the same Finnish query key by means of the syn-operator of the InQuery retrieval system. The syn-operator PAGE treats its operand query keys as synonyms, i.e., instances of the same query key. Structuring caused significant improvements in retrieval performance for long queries (formed on the basis of the title and description fields of TREC topics) and modest improvements for short queries (title fields) with respect to the performance of unstructured (undisambiguated) queries. The relative performance improvements were 107% for long queries and 18% for short queries (Pirkola et al., 2002) . These figures show that in Fin-Eng CLIR syn-based structuring is very effective for queries characterized by the abundance of mistranslated and other bad keys and helpful for queries which contain few bad keys.. The query structuring technique shown to be effective in Fin-Eng text retrieval could be implemented readily in operational retrieval systems. However, it is possible that the specific linguistic features of Finnish as a source language or English as a target language contributed to the good performance of the structured queries. Thus, the effectiveness of the method may depend on the languages of a CLIR system as well as the direction of translation. In this paper we will explore whether the method is useful also in Eng -Fin text retrieval, i.e., the case where translations are done in an opposite direction with respect to those done by Pirkola (1998) and Pirkola et al. (2002) . Thus, we will seek evidence that the syn-structuring technique holds generally, irrespective of a language pair used in a CLIR study or a system.. It is known that languages differ considerably in linguistic features, such as the frequency of lexical ambiguity (Ullman, 1967) and morphological features (Greenberg, 1960; Pirkola, 2001 ). English and Finnish are different types of languages, particularly in morphology. In English grammatical relations are indicated mainly by prepositions while Finnish typically uses a grammatical case. In Finnish, there are altogether 14 features in the category of case (Karlsson, 1987) . Therefore, the number of word forms that a Finnish noun may take is very high, theoretically 2200 forms (Karlsson, 1983) . Inflection has depressing effect on CLIR effectiveness, but it is hard to estimate in which case, Fin-Eng or Eng -Fin, the problem is more severe. In Fin-Eng retrieval, inflection causes difficulties especially in query processing whereas in Eng -Fin retrieval troubles occur in indexing.. In Finnish multiword expressions are typically compound words, in English they are often phrases 1 . Thus, a Finnish compound word is often translated as a noun phrase in English. From the IR and CLIR perspectives, a compound word is a more convenient type of expression than a phrase, because compound decomposition is easier than phrase identification. In this respect Fin-Eng retrieval may be easier than Eng -Fin retrieval. Finnish compounds can be split effectively into component words by a dictionary-based morphological analyzer. The English equivalents of the components can be combined by a proximity operator in CLIR queries(the final target language queries).. Second, we will test the effects of compound-based structuring in CLIR. It refers to a query structuring method where a proximity operator (Inquery's uwn-operator) is used to combine the translation equivalents that correspond to the first part of a source language compound to the equivalents that correspond to the second part of the compound. This method has been used in some studies in which the effects of the syn-operator have been tested (Gollins, 2000; Hedlund et al., 2001; Pirkola, 1998) . However, the effects of the uwn-operator have not been tested; it is not clear what is the contribution of compound-based structuring to the effectiveness of syn-based queries.. Therefore we will test in this study whether syn-structured queries benefit from using the proximity operator for the translation equivalents of compound components.. Third, we will examine query structuring in combination with n-gram based translation (matching) of words not included in a dictionary. General translation dictionaries may include some proper names, such as the names of capital cities and countries. The vast majority, however, is not covered. Particularly, translation dictionaries do not contain personal names. Typically untranslatable keys are used in their original form in a CLIR query. Proper names in different languages often are nonidentical spelling variants, e.g., Brussels (in English) and Bryssel (in many languages). In these cases a source language name does not match its variant form in the target database index, causing loss of retrieval effectiveness.. However, the fact that proper names often are spelling variants of each other allows the use of ngram matching or some other fuzzy matching (approximate string matching) technique for proper name translation. N-gram matching is a language independent means to recognize words whose character strings resemble each other (Angell et al., 1983; Hall and Dowling, 1980; Robertson and Willett, 1998) . Query keys and the words of documents are decomposed into n-grams, i.e., into substrings of length n. The degree of similarity between query keys and index terms can then be computed by comparing their n-gram sets. N-gram matching has been reported to be an effective method to find monolingual name spelling variants and spelling error forms (Pfeifer et al., 1996; Robertson and Willett, 1998; Zobel et al., 1995) as well as cross-lingual name spelling variants (Pirkola et al., 2002) . In this paper, we will investigate whether query structuring in combination with n-gram translation will improve retrieval effectiveness. We found in this paper that, as in Fin-Eng text retrieval, syn-based query structuring is effective in Eng -Fin text retrieval, causing substantial improvements in retrieval performance. This finding provides further evidence that the syn-queries are effective irrespective of languages of a CLIR system. Characteristic linguistic features of specific languages may, however, affect the relative effectiveness of the technique.. Syn/uwn queries are not as effective as syn-queries. Thus, compound-based structuring is not helpful in syn-based queries. However, there are different types of compound words regarding semantic transparency. If a compound is sematically opaque it seems clear translating the component words separately and applying a proximity operator for the equivalents does not yield good results. The question of semantic transparency needs further investigation.. Many proper names are untranslatable due to limited coverage of translation dictionaries. They may also stay unnormalized in database index construction or query processing because of not being included in the lexicon for morphological analysis. For these cases, n-gram translation is helpful. N-gram matching is a language independent means to recognize translation equivalents and is suited for many language pairs used in CLIR. N-gram keys are often proper names and the most important keys of queries. We showed in this study that assigning them more weight than the other keys of a query will result in the improvement in retrieval success. We regard this as an interesting finding that needs to be studied more.

O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Strong converse for identification via quantum channels In this paper we present a simple proof of the strong converse for identification via discrete memoryless quantum channels, based on a novel covering lemma. The new method is a generalization to quantum communication channels of Ahlswede's recently discovered appoach to classical channels. It involves a development of explicit large deviation estimates to the case of random variables taking values in selfadjoint operators on a Hilbert space. This theory is presented separately in an appendix, and we illustrate it by showing its application to quantum generalizations of classical hypergraph covering problems. Ahlswede and Dueck [4] found the identification capacity of a discrete memoryless channel by establishing the optimal (second order) rate via a so-called soft converse. Subsequently, the strong converse, conjectured by them, was proved by Han and Verdú [11] . Even their second, simplified proof [12] uses rather involved arguments.. In [3] it is shown how simple ideas regarding coverings of hypergraphs (formalized in lemma 8) can be used to obtain the approximations of output statistics needed in the converse.. Formally, we investigate the following situation: consider a discrete memoryless channel W n : X n → Y n (n ≥ 1), i.e. for x n = x 1 . . . x n ∈ X n , y n = y 1 . . . y n ∈ Y n W n (y n |x n ) = W (y 1 |x 1 ) · · · W (y n |x n ), with a channel W : X → Y which we identify with the DMC. It is well known [23] that the transmission capacity if this channel (with the strong converse proven by Wolfowitz [28] ) is C(W ) = max P p.d. on X I(P ; W ).. Here I(P ; W ) = H(P W ) − H(W |P ) is Shannon's mutual information, where P W = x∈X P (x)W (·|x) is the output distribution on Y, and H(W |P ) = x∈X P (x)H(W (·|x)) is the conditional entropy of the channel for the input distribution P . Ahlwede and Dueck, considering not the problem that the receiver wants to recover a message (transmission problem), but wants to decide whether or not the sent message is identical to an arbitrarily chosen one (identification problem), defined a (n, N, λ 1 , λ 2 ) identification (ID) code to be a collection of pairs {(P i , D i ) : i = 1, . . . , N }, with probability distributions P i on X n and D i ⊂ Y n , such that the error probabilities of first resp. second kind satisfy. for all i, j = 1, . . . , N , i = j. Here D c i = Y n \ D i is the set complement of D i in Y n , and W n (A|x n ) = y n ∈A W n (y n |x n ) is a convenient shortcut for the probability of an event A ⊂ Y n conditional on x n . Define N (n, λ 1 , λ 2 ) to be the maximal N such that a (n, N, λ 1 , λ 2 ) ID code exists.. With these definitions one has Theorem 1 (Ahlswede, Dueck [4] ) For every λ 1 , λ 2 > 0 and δ > 0, and for every sufficiently large n N (n, λ 1 , λ 2 ) ≥ exp(exp(n(C(W ) − δ))).. The work [3] is devoted to a comparably short, and conceptually simple proof of Theorem 2: Let λ 1 , λ 2 > 0 such that λ 1 + λ 2 < 1. Then for every δ > 0 and every sufficiently large n N (n, λ 1 , λ 2 ) ≤ exp(exp(n(C(W ) + δ))).. Note that for λ 1 + λ 2 ≥ 1 no upper bound on N (n, λ 1 , λ 2 ) can hold: a successful strategy would be that the receiver ignores the actual signal, and to identify i guesses YES with probability 1 − λ 1 , NO with probability λ 1 ≥ 1 − λ 2 .. The first proof of theorem 2 was given in [11] , the method to be further extended in [12] . In [3] it is returned to the very first idea from [4] , essentially to replace the distributions P i by uniform distributions on "small" subsets of X n , namely with cardinality slightly above exp(nC(W )).. Löber [18] began the study of identification via quantum channels. Following his work, and after Holevo [14] , we define a (discrete memoryless) classical-quantum channel (quantum channel for short) to be a map. with X a finite set, as before, and S(H) the set of quantum states of the complex Hilbert space H, which we assume to be finite dimensional. In the sequel, we shall use a = |X | and d = dim H. We identify S(H), as usual, with the set of density operators, i.e. the selfadjoint, positive semidefinite, linear operators on H with unit trace 1 :. In the sequel we will write W x for the images W (x) of the channel map.. Associated to W is the channel map on n-blocks W n : X n −→ S(H ⊗n ), with W n x n = W x1 ⊗ · · · ⊗ W xn . One can use quantum channels to transmit classical information, and Holevo [16] showed that the capacity is . The strong converse for this situation was proved (independently) in [20] and [25] .. Quantum channels are a generalization of classical channels in the following sense: choose any orthonormal basis (e y : y ∈ Y) of the |Y|-dimensional Hilbert space H, and define for the classical channel W : X → Y the corresponding quantum channel W : X → S(H) by. Obviously for another channel V one has V ×W = V ⊗ W .. Regarding the decoding sets let D ⊂ Y, then the corresponding operator D = y∈D |e y e y | satisfies for all x W (D|x) = Tr( W x D).. Observe that by this translation rule a partition of Y corresponds to a projection valued measure (PVM) on H, i.e. a collection of mutually orthogonal projectors which sum to ½. Conversely, given any operator D on H with 0 ≤ D ≤ ½, define the function δ : Y → [0, 1] by δ(y) = e y |D|e y . Then for all x [7] for the mathematics to describe quantum systems.. which implies that every quantum observation, i.e. a positive operator valued measure (POVM), of the states W x can be simulated by a classical randomized decision rule on Y. One consequence of this is that the transmission capacities of W and of W are equal: C(W ) = C( W ). Equally, also the identification capacities (whose definition in the quantum case is given below) coincide. For randomization at the decoder cannot improve either minimum error probability.. Abstractly, just given the states W x , this situation occurs if they pairwise commute: for then they are simultaneously diagonalizable, hence the orthonormal basis (e y : y ∈ Y) arises.. According to [18] a (n, N, λ 1 , λ 2 ) quantum identification (QID) code is a collection of pairs. with probability distributions P i on X n , and operators D i on H ⊗n satisfying 0 ≤ D i ≤ ½, such that the error probabilities of first resp. second kind satisfy. for all i, j = 1, . . . , N , i = j. Again, define N (n, λ 1 , λ 2 ) to be the maximal N such that a (n, N, λ 1 , λ 2 ) QID code exists.. This definition has a subtle problem: since the D i need not commute, it is possible that identifying for a message i prohibits identification for j, as the corresponding POVMs (D i , ½ − D i ) and (D j , ½ − D j ) may be incompatible. To allow simultaneous identification of all messages we have to assue that the D i have a common refinement, i.e. there exists a POVM (E k : k = 1, . . . , K) and subsets I i of {1, . . . , K} such that. for all i. In this case the QID code is called simultaneous, and N sim (n, λ 1 , λ 2 ) is the maximal N such that a simultaneous (n, N, λ 1 , λ 2 ) quantum identification code exists. Clearly N sim (n, λ 1 , λ 2 ) ≤ N (n, λ 1 , λ 2 ).. In analogy to the above theorems it was proved: Theorem 3 (Löber [18] ) For every λ 1 , λ 2 > 0 and δ > 0, and for every sufficiently large n N sim (n, λ 1 , λ 2 ) ≥ exp(exp(n(C(W ) − δ))).. On the other hand, let λ 1 , λ 2 > 0 such that λ 1 + λ 2 < 1. Then for every δ > 0 and every sufficiently large n N sim (n, λ 1 , λ 2 ) ≤ exp(exp(n(C(W ) + δ))).. Looking at the examples given in [4] the simultaneity condition seems completely natural. But this need not always be the case.. Example 4: Modify the "sailors' wives" situation (Example 1 form [4] ) as follows: the N sailors are not married each to one wife but instead are all in love with a single girl. One day in a storm one sailor drowns, and his identity should be communicated home. The girl however is capricious to the degree that it is impossible to predict who is her sweetheart at a given moment: when the message about the drowned sailor arrives, she will only ask for her present sweetheart, and only she will ask.. With our present approach we can get rid of the simultaneity condition in the converse (whereas by the above theorem identification codes approaching the capacity can be designed to be simultaneous -namely, by [4] for any sort of channel and a transmission code of rate R for it, one can construct an ID code "on top" of the transmission code, and with identification rate R, asymptotically):. Theorem 5: Let λ 1 , λ 2 > 0 such that λ 1 + λ 2 < 1. Then for every δ > 0 and every sufficiently large n N (n, λ 1 , λ 2 ) ≤ exp(exp(n(C(W ) + δ))). The rest of the paper is divided into two major blocks: first, after a short review of the ideas from [3] in section II, the rest of the main text will be devoted to the proof of theorem 5 (as explained, this contains theorem 2 indeed as a special case), in section III.. The other block is the appendix, containing the fundamentals of a theory of (selfadjoint) operator valued random variables. There the large deviation bounds to be used in the main text are derived. The theory of real random variables provides the framework of much of modern probability theory, such as laws of large numbers, limit theorems, and probability estimates for 'deviations', when sums of independent random variables are involved. However several authors have started to develop analogous theories for the case that the algebraic structure of the reals is substituted by more general structures such as groups, vector spaces, etc., see for example [10] .. In the present work we focus on a structure that has vital interest in quantum probability theory, namely the algebra of operators on a (complex) Hilbert space, and in particular the real vector space of selfadjoint operators therein which can be regarded as a partially ordered generalization of the reals (as embedded in the complex numbers). In particular it makes sense to discuss probability estimates as the Markov and Chebyshev inequality (subsection C), and in fact one can even generalize the exponentially good estimates for large deviations by the so-called Bernstein trick which yield the famous Chernoff bounds (subsection D).. Otherwise the plan of this appendix is as follows: subsection B collects basic definitions and notation we employ, and some facts from the theory of operator and trace inequalities, after the central subsections C and D we collect a number of plausible conjectures (subsection E), and close with an application to the noncommutative generalization of the covering problem for hypergraphs, in subsection F.

O O O B O B I O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O