Report on theWorkshop on Operational Text Classiﬁcation Systems(OTC-01)
David D. Lewis
Independent Consultant
Chicago, US
E-mail: <dave@daviddlewis.com>

1

Fabrizio Sebastiani
Consiglio Nazionale delle Ricerche
Pisa, ITE-mail: <fabrizio@iei.pi.cnr.it>

Goal of the Workshop

The Workshop on Operational Text Classiﬁcation (OTC-01), occurred September 13, 2001 in New
Orleans, Louisiana, US. It was co-located with ACM SIGIR 2002 and brought together researchers,
practitioners, and system designers interested in building and ﬁelding operational text classiﬁcationsystems. The workshop organizers were David Lewis (chair), Susan Dumais, Ronen Feldman, andFabrizio Sebastiani.
Text classiﬁcation research and practice has exploded in the past decade. This work has
been pursued under a variety of headings (text categorization, automated indexing, document
clustering, text mining, topic detection and tracking, etc.). Both the automated assignmentof textual data to classes, and the automated discovery of such classes (by techniques such asclustering) have been of intense interest. A variety of practical applications have been ﬁelded, inareas such as indexing of documents for retrieval, classiﬁcation of Web sites under hierarchicaldirectories, alerting, routing of news, creation of customized information products, enforcement ofinformation security, ﬁltering of unsuitable content, help desk automation, knowledge discovery intextual and partially textual databases, and many others.
However, while experiments on text classiﬁcation data sets have been widely presented ina variety of forums, the details of text classiﬁcation in actual use have rarely been discussed.
The workshop therefore encouraged contributions discussing issues such as cost eﬀectiveness, userneeds, personnel and training, resource demands, maintaining eﬀectiveness over time, and integration of classiﬁcation with existing processes and software.

2

Format of the Workshop

Prospective speakers submitted a total of 18 extended abstracts, from which the program committee selected 9 for presentation. A call for demos was later issued among prospective attendees.
Only 1 demo proposal was submitted, and it was accepted for presentation. About 45 participants
attended the workshop.
The selected abstracts, along with brief interest paragraphs from attendees, were included in aninformal notebook, whose distribution was limited to participants to encourage frank discussion.
A subset of authors agreed to have their extended abstracts and slides posted publicly. Theseappear at http://www.DavidDLewis.com/events/otc2001.

3Presentations

We brieﬂy summarize here the presentations from the workshop, giving our personal interpretations of these detail-rich talks. As requested by the speakers, we do not include certain details,
such as personnel costs, that came up in the discussions.
Two talks, by Richard Tong (Tarragon Consulting) and Mark Krellenstein (Northern LightTechnology) were canceled due to travel interruptions after the September 11 terrorist attacks
in the US. A third talk, by Mark Shewhart (LexisNexis), was presented by his colleague MarkWasson. The schedule of presentations was compressed to allow for the rapidly changing travelplans of participants, with a single extended discussion session after the presentations.
Mark Wasson (LexisNexis) presented two talks on TTI (Term-based Topic Identiﬁcation
System), a text categorization system which has been in operation at LexisNexis for many years.
It classiﬁes tens of thousands of news articles, legal cases, Web sites, company reports and otherdocuments on a daily basis. Over 70,000 categories are used, including both subject categories
and named entities. Categories can be included in Boolean searches by end users. They are also
used to create “specialized libraries”, allowing a user search to run against a virtual collection ofmaterial on a speciﬁc topic. The set of categories has evolved over time, and customer requests
to supply particular categorizations (which may or may not ﬁt well in the existing taxonomy)
present ongoing challenges. Demands for accuracy and consistency are high, as the user baseincludes many information professionals.
Classiﬁers are built manually in an iterative fashion. Pure supervised machine learning approaches were not felt to be eﬀective across the wide range of source types to be handled. However,supervised learning in the form of chi-square tests and stepwise linear regression is used to aid
manual selection and weighting of words and groups of words. Similarly, while glossaries, namevariant generation, and other “knowledge-based” approaches are used, the outputs of all of these
are reviewed and often overridden based on human expertise. Techniques that would limit theclassiﬁer builders’ ﬂexibility (punctuation and stop word removal, stemming, etc.) are avoided.
Building of a classiﬁer may require anywhere from ﬁve minutes (for, say, a simple company namedetector) to eight hours (for categories corresponding to complex topics) of staﬀ time.
The diversity of material to be classiﬁed is partly dealt with by source speciﬁcation ﬁles.
These map from the ﬁelds of each document to a canonical ﬁelds that are referenced by rules (e.g.
mapping Title from one source and Heading from another to a canonical title ﬁeld). Even so,
properties of speciﬁc sources must sometimes be manually accounted for in classiﬁers.
Mark reported precision and recall rates higher than 90–95% for most topics and sources, withclassiﬁcation speeds on the order of 100,000 characters/sec. Precision, recall, and similar measures are used in quality assurance and management, and considerable resources are devoted tobenchmarking on real data. Eﬀectiveness data from the research literature or text classiﬁcation
vendors have not always been found to be good predictors of eﬀectiveness on LexisNexis data.
Max Copperman and Scott Waterman (Kanisa) reported on their experience with using
text categorization to support interactive, self-service help systems. Documents consist of FAQs,
troubleshooting guides, product information, and the like, and need to be classiﬁed under a varietyof dimensions: this means that multiple taxonomies coexist (up to 150 taxonomies, with up to 2,000categories per taxonomy), addressing orthogonal concerns (e.g. a Product taxonomy, a Symptoms
taxonomy, etc.). User queries are also classiﬁed under the same taxonomies, and matching on
categories drives the response to each query.
The large number of closely related categories (e.g. Creating Forms vs. Editing Form Fields) andshortage of data on many categories (as well as labeling costs) argued against a pure supervised
learning approach. At least one commercial classiﬁcation tool was found wanting in internal tests.
Kanisa’s current approach relies heavily on manual deﬁnition and choice of features, followed bysome use of training data to set weights. As with LexisNexis, the classiﬁer environment allows
manual overriding of defaults for issues such as stemming and phrase matching. Also as with LexisNexis, source deﬁnitions are used to control which parts of documents are used in classiﬁcation.

Kanisa found that improving classiﬁer eﬀectiveness did not always lead to better help dialogues,
which is the goal of their system. Testing therefore emphasizes dialogue quality as evaluated bycustomers and by semi-automated full system testing.
Thomas Montgomery (Ford Motor Company) reported progress toward a system for classifying patents to support competitive intelligence. Currently, manual classiﬁcation of patents into
technology categories is done to support detection of trends and levels of activity by competitors.
Automating this classiﬁcation promises to speed up the availability of such information, allow
greater coverage, and allow reclassiﬁcation as categories change to reﬂect new technologies.
Commercial text classiﬁcation tools, research software, and internally built tools were all evaluated for this application. Tests were carried out using a sample of 136,063 manually labeled
patents produced by the current manual process. A taxonomy of 4,000 categories were used, with
documents allowed to belong to multiple categories. Preliminary results on high level categories
showed support vectors machines (SVMs) outperforming a nearest neighbor approach (k-NN), andthat in turn outperforming a naive Bayes approach. Evaluation is ongoing. Lower level categories
were harder to assigned than higher level ones, and some collapsing together of lower level categories with too few training examples was necessary. Tom indicated that the choice of a ﬁnal
approach will depend not just on eﬀectiveness, but on issues such as vendor size and stability,
maintenance eﬀort, cost, etc.
Bryan Goodman (Ford Motor Company) reported work to develop a personalized information delivery systems. Here, there was no existing classiﬁed data or taxonomy, so the two werecreated together. A taxonomy of 57 Ford-speciﬁc categories (plus “Other”) was created, guided
by existing taxonomies from other applications as well as self-organizing map (SOM) software. A
total of 2600 papers were labeled, with bootstrapping from initially labeled data used to aid thelabeling (and correction of labeling) of further documents. As with the project reported by Tom
Montgomery, a variety of machine learning approaches were tested. The vast majority of work on
the project involved writing software to connect tools together, reformat data for each piece ofsoftware, etc. Hope was expressed for standardization as text classiﬁcation software matures.
Tamas Doszkocs (National Library of Medicine) reported on preliminary work on text classiﬁcation to improve search of a document base on hazardous substances. Shallow parsing andmatching of strings against UMLS (Uniﬁed Medical Language System) have been investigated,
and a test collection has been built to support future studies of machine learning approaches.
Khalid Al-Kofahi (Thomson Legal & Regulatory) described a “case routing program” (CARP)
for legal publishing in use at Thomson, which classiﬁes a weekly feed of case law summaries (CLSs)
under a proprietary classiﬁcation scheme, called American Law Reports (ALR). ALR is actuallya set of about 13,800 articles (“annotations”) that provide in-depth analysis of narrow points oflaw. The task of CARP is to suggest court cases that should be cited by each ALR article.
CARP has been operational since January 2001. It processes about 12,000 headnotes per week
(each case has on the order of ﬁve headnotes associated with it), and makes on the order of 1,600
suggestions to link particular headnotes with ALR articles. Roughly 900 of these suggestions areaccepted, 170 rejected as inappropriate, and 530 considered on topic but redundant or too general.
This compares with the previous purely manual process which produced about 700 new citations
per week. Supervised learning was used to train classiﬁers, which take the form of a weighted
committee of individual classiﬁers. Inputs to classiﬁers include words, phrases, and previously
assigned categories from other taxonomies. (Khalid mentioned that over 200 separate taxonomies
are in use for diﬀerent purposes in Thomson.)
Finally, Ronen Feldman (Clearforest) gave a demonstration of the ClearCat categorization
system. The system comprises a suite of tools, including diﬀerent learning (SVM, perceptron,Rocchio, naive Bayes, kNN), and indexing (e.g. bag of words, noun phrases, etc.) methods, withoptions for manual feature selection.

4Discussion

A wide-ranging discussion followed the presentations, lasting about four hours. There was aparticular emphasis on issues that were common among the talks, but which have received littleattention in the research literature.
For instance, while training on labeled data is widely used, it is rarely used in the “data in/ classiﬁer out” model common in experimental work. Manual feature construction and feature
selection before training, and/or modiﬁcation of classiﬁers after the fact, appear common in operational systems. Reasons include both the value provided by human domain knowledge (particularly
in constructing features that would not be found by a learning systems), as well as problems oflack of data to label, cost of labeling, and inaccuracies in labeling. The important role of domain
knowledge sometimes leads to learning approaches or classiﬁer forms that would not be optimal
in a pure supervised learning setting, but which make manual intervention easier. For software
vendors, being able to quickly provide manual input during a demo was another argument for this.
Eﬀectiveness measures such as recall and precision are sometimes used. Customers on ﬁrst
learning of them often have unrealistic expectations (e.g. 100% precision and recall). Education
about the limits of technology and the subjectiveness of classiﬁcation, as well as looking into theactual needs and context of an application, helps.
However, these measures do not capture everything one would want. In particular, the notionof degrees of correctness of category assignments came up several times. Some mistakes are worse
than others in terms of end user perception. Several attendees mentioned tuning of thresholds
to avoid blatant false positives, especially in systems which do not involve human checking ofautomated output. The question is not merely one of avoiding false positives, however. Assigninga category which is wrong, but closely related to a correct category, is viewed as less bad by users
than assigning a completely unrelated one. This raised the question of whether we need new
eﬀectiveness measures that take such relationships into account.
Some techniques that have excited interest or controversy in the research community werediscussed. Multiword phrases were mentioned several times, but often in the context of manualfeature construction, so their role in a purely automated setting is unclear. Using hierarchicalrelationships among categories was found useful in tests by some groups and useless by others.
None of the talks discussed using links between hypertext documents to aid classiﬁcation, andthere was speculation on the diﬀerence in linkage structure between the Web and, for instance,intranets.
There was, not surprisingly given the workshop topic, considerable enthusiasm for automatedclassiﬁcation. Many participants felt that automated systems, or automation plus human checking,can reduce both costs and increase consistency.

5Future activities

Participants overwhelmingly indicated that the workshop was successful, and discussions werelively. Many expressed the theme that insights from the operational world will shake up theimplicit assumptions of text classiﬁcation research, with beneﬁts to both.
There was strong support for a follow-up workshop. Not surprisingly given the setting, a straw
poll favored SIGIR-02 in Tampere as the location, but KDD-02 in Edmonton was also discussed.
David Lewis also announced his intention to edit a book-length collection on operational textclassiﬁcation, with papers to be solicited widely. Look for announcements of these and otheractivities on the DDLBETA text categorization mailing list and other forums.