Visual Pattern Recognition in the Years Ahead

Conventional classification algorithms have already reached a plateau at the trade-off imposed by the bias due to the structure of the classifier and the variance due to the limited size of the training set. The latter may be alleviated by exploiting known constraints, including class and style priors, language models, statistical correlations between spatially proximate patterns, statistical dependence due to isogeny (common source) of patterns, and even information-theoretic properties of the representations that have evolved for symbolic patterns intended for communication. Another development that may lead to new applications of pattern recognition is more effective human intervention. The interplay of human and machine abilities requires models that are both human and computer accessible. 1. Beyond representative training sets The fundamental assumption behind most trainable classifiers is that the test set is statistically representative of the training set. How representative is representative? The test set and the training set should be as similar as if they had been randomly chosen from a single corpus of patterns. In practice this assumption is often unworkable because drawing a random training sample requires prior access to the population to be classified. Even if enough representative training patterns were available, it would be too costly to label them. We discuss two approaches to circumventing these limitations. The first is constrained recognition, and the second is explicit design for optimal human intervention. I will first discuss constrained recognition for high-volume tasks like OCR, where automatic classification is essential. Then I will discuss interactive recognition for low-throughput or sporadic applications, where accuracy is paramount. However, the two approaches are not incompatible: human intervention plays a part even in OCR. Note that the word "visual" in the title specifically excludes speech recognition and other signal processing applications, not because these ideas are inapplicable to them, but only to delimit the scope of discussion. 2. Constrained recognition In addition to architectural differences among the data structures used to store and access decision boundaries, classifiers differ from one another by the type of information used to adjust their parameters. There has been much research, and significant recent progress, in "minimalist" pattern recognition, where the objective is to extract new information from sets of patterns represented by feature vectors (unsupervised learning [1,2]), from labeled vectors (supervised learning [3]), or from both labeled and unlabeled samples (semi-supervised learning [4]). The ultimate goal has been the development of universal classification algorithms based on arbitrary collections of feature vectors. In contrast, the goal of constrained recognition is explicit and formal algorithmic application of whatever domain-specific information is available. The classifier should make use of all available contexts, much as a human would. Instead of recognizing patterns in isolation, it should recognize interrelated groups of patterns (patterns of patterns). Furthermore, we expect dynamic classifier design to foster learning during the entire operational lifetime of the classifier by making use of feedback from downstream use of its output [5]. Examples of pertinent constraints, starting with some that are already widely exploited, are listed below. • The number and relative population of classes and subclasses (class and style priors) assume decisive importance with highly skewed demographics. Priors can be dynamically re-estimated according to strata defined by already extracted information, as in advanced form-processing systems [6]. Invited plenary presentation, Procs. ICPR04, VOl. II, pp. 311-314, Cambridge UK, August 2004 • Statistical dependence among the class labels of adjacent patterns (linguistic context) has been modeled at the morphological, lexical, and syntactic levels [7,8], Many Hidden Markov Models make use of language models. The underlying transition probabilities need not be estimated from the same training set as the OCR features, because linguistic content is seldom correlated with typeface or handwriting. • Statistical dependence among the features of adjacent patterns (due, for instance, to ligatures) occurs in handwriting (allographs), and in printed Arabic and Indic text. • Class similarities may be predictable: the digit "1" is likely to be misrecognized as the letter "l". Confusion matrices are fairly stable over feature sets, typefaces, and writers. This constraint is occasionally applied to reduce alphanumeric confusions in forms reading. • We have had some success in exploiting common-source constraints that give rise to statistical dependence among the features of (not necessarily adjacent) patterns in isogenous fields (style context) [9,10,11,12]. How Anne writes "4" may suggest how she writes "9". A more general notion of visual and structural style is presented in [13]. • Restrictions on the maximum difference between the class-means of test patterns and the class-means of the training samples lead to decision-directed approximation [14,15,16,17]. This constraint expresses the confidence that the classifier will recognize most of the patterns in the test set, even though they are statistically different from the training set. If so, the classifier can be re-trained with the newly labeled patterns. When does "unsupervised" adaptation work? While some sufficient conditions have been developed, they are much too restrictive for pattern distributions observed in practice. Necessary conditions have not been found. • A common property of OCR feature spaces is that no linear combination of the mean vectors of a set of classes is located near the mean vector of any other class. Furthermore, the distribution of patterns about the class means is asymmetric: deviations away from the means of other classes (and from the mean of all patterns) tend to be larger than deviations towards other patterns. This affects outlier detection and may be the consequence of the evolution of scripts according to information-theoretic principles that preserve class-distinctions in the presence of noise [18]. It is less likely to hold for natural objects, such as multispectral crop signatures in remote sensing. • The correlation between a pair of features depends much more on the chosen pair of features than on the class or style of the samples over which the correlation is computed. This constraint justifies various covariance matrix regularization schemes that are useful when there are not enough samples to accurately estimate class-covariances, but enough to estimate the overall covariance matrix. • The order of the extracted features is preserved under elastic deformation of a sequence of patterns, as when handwriting is squeezed on approaching the right margin. Sequence-preserving inter-pattern relations are exploited in Symbolic Indirect Correlation (SIC). Nearest Neighbor classifiers are not directly applicable to unsegmented pattern sequences, but SIC is. Because such classifiers have high variance, they need many reference patterns. The development of algorithms that can perform the required comparison of bipartite graphs is only at a preliminary stage [19]. 3. Interactive recognition All operational pattern recognition systems make some use of human intervention. But except in Image Database, human interaction is seldom mentioned in research publications. We usually report only that the classifier was trained and tested on a "ground-truthed" database of so many patterns, and that 90% or 99% of the test patterns were correctly classified, without mentioning how much time was expended on producing the ground truth, or what is to be done with the 10% or 1% errors. Even in operational systems, human intervention usually takes place only at the beginning and at the end. In OCR, for example, the scanned pages are usually inspected, and problematic pages are either set aside for manual data entry or rescanned with different settings. The segmentation of the document into text and non-text fields may also be inspected and corrected. At the other end of the pipeline, rejected words or fields are manually re-entered. In critical applications, either the entire output or a subset thereof (possibly selected by automated triage [20]) may be proofread in order to catch and correct OCR errors. Research on several other aspects of human interaction, listed below, would pay rich dividends. • Who is in charge? It is tempting to think of the human as providing on-call assistance to a fallible automated classifier. It is more productive, however, to focus on how a computer can help human recognition. If human accuracy is desired, we cannot depend on the machine's assessment of when it is likely to be wrong. Confidence measures based on the tails of probability distributions are inherently untrustworthy. Therefore it is essential to let the human retain the initiative throughout the entire classification process. The role of the machine is only to save time by performing routine tasks under close supervision. Invited plenary presentation, Procs. ICPR04, VOl. II, pp. 311-314, Cambridge UK, August 2004 • How can we take advantage of the differences between human and machine cognitive abilities? Humans apply to recognition a rich set of contextual constraints and superior noise filtering abilities to excel in gestalt tasks, like object-background separation. Humans are also good at judging whether two images represent the same class. But computers can store thousands of images and associations between them, never forget a name or a label, and compute geometric moments and conditional probabilities. These differences suggest that a system that combines human and machine abilities can, in some situations, outperform both. (And have also inspired