Invited Talk
Date/time Friday, December 9, 2005 / 9:15-9:45"Cross Language Evaluation Forum: CLEF 2005"
Gareth Jones and Carol Peters

Abstract
This presentation will report the activities of the CLEF 2005 evaluation campaign. CLEF 2005
consisted of 8 tracks focusing on topics in multilingual information retrieval. An assessment of theresults will be given with particular focus on two important tracks: multilingual ad-hoc retrieval andcross-language search in image collections. The multilingual task this year had two objectives: toexplore reuse of test collections to measure an improvement in system performance over time and toexamine in depth the problem of merging results from a multilingual search over different
collections and languages. The ImageCLEF track was very successful with several different tasks
and a large participation. A main aim was to investigate how visual and textual features can best be
combined for effective retrieval. The presentation will conclude with a brief description of theagenda for CLEF 2006.

Speaker Bio
Gareth Jones is Senior Lecturer in Computing at Dublin City University, Ireland where he is amember of the Centre for Digital Video Processing and the National Centre for Language
Technology. He was previously a Lecturer at the University of Exeter, U.K. and a post-doctoral
Research Associate at the University of Cambridge. He was awarded B.Eng and PhD degrees by theUniversity of Bristol. From 1997-1998 he was a Toshiba Fellow at the Toshiba Corporation Researchand Development Center in Kawasaki, and in 2002 a JSPS Visiting Fellow at the National Instituteof Informatics, Tokyo. His .principal research interests are in information retrieval, applied natural
language processing and multimedia information management. He has been active in the EU
Cross-Language Evaluation Forum (CLEF) since its formation in 2000, and is currently investigatoron a China-Ireland research collaboration project exploring English-Chinese Cross-Language
Question Answering.

Invited Talk
Date/time Friday, December 9, 2005 / 9:45-10:15"Evaluation and Applications of Automatic Text Summarization"
Chin-Yew LIN

Abstract
Automated text summarization aims to create concise restatements of the topic and main ideas of
input documents. It has been one major topic in natural language processing research. However, until
recently there are no common, convenient, and repeatable evaluation methods that can be easily
applied to support fast turn-around system development and just-in-time comparison among
different summarization methods.

The recent adoption of the ROUGE automatic evaluation method by the text summarization
community is a step toward addressing the problem. However, despite the success of ROUGE, it stillworks strictly on lexical level, relies on simple matching heuristics and is weak in theoreticalframework. In this talk, I will introduce an information-theoretic approach to automatic evaluation ofsummarization that measures the quality of a summary using the Jensen-Shannon (JS) divergence
between the summary and a set of reference summaries. Several variants of the approach are also
considered and compared. The initial results indicate that JS divergence-based evaluation method
achieves comparable performance with the current de facto automatic evaluation method ROUGE.

Speaker Bio
Chin-Yew LIN is a senior research scientist at the Information Sciences Institute of the University ofSouthern California. His research interests are text summarization, question answering, opinion
extraction, and machine translation. He was the ACL 2005 area chair of text summarization andquestion answering. He built two state-of-the-art text summarization systems: SUMMARIST andNeATS and was the chief architect of ISI's Webclopedia question answering system. He also
developed the automatic summarization evaluation package ROUGE and BE that have been used in
the DUC evaluations. He has co-chaired several text summarization and question answering
workshops in ACL, NAACL, HLT, and COLING.