TREC, TAC, takeoffs,tacks, tasks, andtitillations for 2009
Ian Soboroff, NIST
ian.soboroff@nist.gov

Agenda• TREC 2008
• (some) reflections on TREC
• TAC, a new evaluation conference for NLP• TREC 2009 preview

TREC Goals• To increase research in information retrieval basedon large-scale collections• To provide an open forum for exchange ofresearch ideas to increase communication among
academia, industry, and government• To facilitate technology transfer between researchlabs and commercial products• To improve evaluation methodologies andmeasures for information retrieval• To create a series of test collections covering
different aspects of information retrieval

TREC 2008 Program Committee
Ellen Voorhees, chair

David Lewis

James AllanJohn Prager

Chris Buckley

Steve Robertson

Gord Cormack

Mark Sanderson

Sue Dumais

Ian Soboroff

Donna Harman

Richard Tong

Bill Hersh

Ross Wilkinson

TREC 2008 Participants
Beijing Univ. of Posts &Telecommunications

Korea University

Brown University

Max-Planck-Institut Informatik University of Glasgow

Carnegie Mellon University

Nat’l Univ. of Ireland, Galway

Univ. of Illinois, Chicago

Chinese Acad. of Sciences

Northeastern UniversityU. Illinois, Urbana-Champaign

Clearwell Systems, Inc.
CNIPA ICT Lab
Dalian U. of Technology
Dublin City University

Open Text Corporation
Pohang Univ Science & Tech
RMIT University
Sabir Research

University of Iowa (2)University of Lugano
Univ. Maryland, College Park
University of Massachusetts

Fondazione Ugo Bordoni

SEBIR

Univ. of Missouri-Kansas City

Fudan University
H5

St. Petersburg State Univ.
SUNY Buffalo

University of Neuchatel
University of Pittsburgh

Heilongjiang Inst. of Tech.

TNO ICT

University of Texas at Dallas

Hong Kong Polytechnic U.
IBM Research Lab

Tsinghua University
Universidade do Porto

University of Twente
University of Waterloo (2)Indian Inst Tech, Kharagpur

University College, London

Ursinus College

Indiana UniversityINRIA

Univ. of Alaska, Fairbanks
University of Amsterdam (2)Wuhan University
York University

Kobe University

Univ. of Arkansas, Little Rock

University of Avignon

Tracks
blog
enterpriselegalmillion query
relevance feedback

Craig Macdonald, Iadh Ounis,
Ian SoboroffPeter Bailey, Nick Craswell,
Arjen de Vries, Ian Soboroff,
Paul Thomas
Jason Baron, Bruce Hedin,
Doug Oard, Stephen
Tomlinson
James Allan, Jay Aslam
Chris Buckley, Stephen
Robertson

TREC 2008
• TREC 2008: November 18-21(we are between the conference and thefinal proceedings)• But here are some things to look for...

TREC 2008
• Evaluation challenges• continue exploring alternatives to traditional
pooling for test collection building• sampling methods in MQ, rel fdbk, legal tracks• new samples entail new evaluation measurecomputations• revisit impact of variability in relevance judgments• Contextualizing search• enterprise, legal, blog tasks target specific use cases

Blog TrackTasks:
1. Finding blog posts that contain opinions
about the topic
2. Ranking positive and negative blog posts
3. (A separate baseline task to just find blog
posts relevant to the topic)
4. Finding blogs that have a principal, recurringinterest in the topic

Enterprise Track• Enterprise: CSIRO• Topics taken from CSIRO Enquiries(they get the “contact us” emails)• Tasks:
1. Find key pages which answer the enquiry
2. Find people who are topic experts thatmight help answer the enquiry

Legal Track• Legal discovery search task
• Topics divided among several complaints.
topic includes a request, a Boolean query• Each(with negotiation), and more...
• Relevance feedback task
• Interactive task
to find as many responsive documents as• Goal:possible for any of three topics
group could use 10 hours of time with a• Eachdomain expert lawyer

Million Query Track• 10,000 queries• Gov2 collection (25M web pages, 425 GB)• Queries divided among long/short, many/fewclicks• ~800 queries judged by NIST assessors using
two sampling strategies• “Minimal test collections” method(Carterette et al, SIGIR 2006)• “statAP” method (Aslam et al, SIGIR 2006)

Relevance Feedback Track• Goal: look again at relevance feedback, inmodern collections and with modernmethods• 264 topics run on the Gov2 collection• 50 terabyte topics + 214 MQ topics• All queries included in this year’s MQ set• A range of feedback conditions

TREC 2008
• Results are still preliminary...
• So I won’t show them here.
• (Think of this as an invitation to participate)• Final papers due in February.
• Proceedings in the spring (hopefully).

Reflections• TREC 2009 will be our 18th year
• 2 GB → 426 GB• 50 topics → 1,800 topics• tasks: ad hoc, filtering, novelty, question
answering, known-item search ...

• multiple languages, media, document types• multiple domains: legal, genomics,
enterpriseThe TREC Tracks
Personal documents

BlogSpamRetrieval in a domain

Legal
GenomeAnswers, not docs

Novelty
Q&AWeb searching, size

EnterpriseTerabyte
WebVLC

Beyond text

VideoSpeech
OCR

Beyond just English

X→{X,Y,Z}
ChineseSpanishHuman-in-the-loop

Interactive, HARD, fdbk

Filtering
Routing

Streamed text

Million query
Ad Hoc, Robust2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

Static text• The Text Analysis Conference is a new
NIST evaluation forum.

• TAC focuses on natural language processingtasks.

Why TAC?
TREC

SemEvalOpen MT
ACE
DUC

CoNLL
RTE

Why TAC?
TREC - QA

DUC

RTE

Why TAC?
TREC - QA

DUC

RTE

Features of TAC• Component evaluations situated within context ofend-user tasks (e.g., summarization, QA)• opportunity to test components in end-user tasks• Test common techniques across tracks• Small number of tracks• critical mass of participants per track• sufficient resources per track (data, assessing,technical support)• Leverage shared resources across tracks(organizational infrastructure, data, assessing, tools)

TAC 2008 Tracks
RTE: systems recognize when one piece of text entails orcontradicts another
QA: systems return a precise answer in response to aquestion, focusing on opinion questions asked over blogs
Summarization: systems return a fluent summary of
documents focused by a narrative or set of questions
1. Update: summarize new information in newswire
articles for a user who has already read an earlier setof articles
2. Opinion pilot: summarize blog documents containing
answers to opinion question(s) -- joint with QA

Recognizing Textual Entailment (RTE)• Goal: recognize when one piece of text is entailed byanother• Classification Task: given T(ext) and H(ypothesis)• H is entailed by T
• H is not entailed by T
• H contradicts T• H neither contradicts nor is entailed by T
• T/H pairs from IR, IE, QA, and summarization
contexts.

RTE Pairs from QA Setting• H: generated from questions and candidate answer
terms returned by QA systems searching the WebBaldwin is Antigua's Prime Minister.

• T: candidate answer passages returned by QA
systems

The opposition Antigua Labour Party (ALP) has
blasted that country's prime minister, BaldwinSpencer, for publicly advocating that Cuba's Fidel
Castro be awarded the Order of the Community(OCC) - the Community's highest honour.

Update Summarization Task
• Given a topic and 2 chronologically orderedclusters of news articles, A and B, where A
documents precede B documents• Create two brief (<=100 words), fluent

summaries that contribute to satisfying theinformation need expressed in the topic
statement:• Initial summary (A): summary of cluster A• Update summary (B): summary of cluster
B, assuming reader has read cluster A

Pipelined Opinion QA/Summarization Task

Pipelined Opinion QA/Summarization Task

Why
don’t people like
Trader Joe’s?

Pipelined Opinion QA/Summarization Task
filthyloved it!

yummy snacks
parking nightmare

Yuk!
innovative
service could have been better
unhelpful clerk

Why
don’t people like
Trader Joe’s?

Pipelined Opinion QA/Summarization Task
filthyloved it!

yummy snacks
parking nightmare

Yuk!
innovative
service could have been better
unhelpful clerk

Why
don’t people like
Trader Joe’s?

Pipelined Opinion QA/Summarization Task
filthyloved it!

yummy snacks
parking nightmare

Yuk!
innovative
service could have been better
unhelpful clerk

Why
don’t people like
Trader Joe’s?

hy,tlfis
is
Joe’r
e
andd,
aec
Triv
serroop
ing
haskr
ap
is are.
amthnig

Opinion QA
TARGET:"MythBusters"

1018.1 RIGID LIST

Who likes Mythbusterʼs?

1018.2 SQUISHY LIST Why do people like Mythbusterʼs?

1018.3 RIGID LIST

Who do people like on Mythbusterʼs?

Opinion QA
TARGET:"MythBusters"

1018.1 RIGID LIST
BLOG06-3334
BLOG06-8580
BLOG06-3982

Who likes Mythbusterʼs?
CAPS_CHAMP
Jon
Zonk

1018.2 SQUISHY LIST Why do people like Mythbusterʼs?
BLOG06-6706
The Mythbusters chicas are purdy .
BLOG06-5962
It's geek, period. And a lot of fun. I like thatthey have women on their team who are also into mechanical stuff
and applied science.
1018.3 RIGID LISTBLOG06-3187
BLOG06-4849
BLOG06-6570

Who do people like on Mythbusterʼs?
Kari Byron
scottie
Jamie Hyneman

Opinion Summarization• Input• Target, 1-2 squishy list questions• Documents known to have answers• Optional answer-snippets in each document• Output• Single fluent summary of the answers to all
the questions• Opinion polarity classification (positive vsnegative) may help fluency

Proposed TAC 2009 Tracks
1. RTE
2. Summarization:• Update ?
• Opinion ?
• Meeting/Speech ?
3. Information Extraction for Knowledge BasePopulation

New for TREC 2009
• Chemical IR track• New blog track collection• New web collection supporting fourtracks(two new, two old)

• Legal, RF, and MQ tracks will continue• Enterprise track endingChemical IR track
John Tait (IRF), Jianhan Zhu (Open U), Jimmy• Coordinators:
Huang (York U), Mihai Lupu (IRF)• Document collection (to be provided by IRF)
• ~100,000 patents (XML formatted)
journal articles from the UK Royal Society of• ~45,000Chemistry• Tasks:patent claim search: given claims, find the patent• Chemicalreferences
hoc search, with topics developed and judged by• Ad
domain experts

Blog track• Coordinators: Iadh Ounis, Craid Macdonald (UGlasgow), Ian Soboroff (NIST)• New Blog08 collection• spans a full year
• 600k feeds, 40 million blog posts• permalinks, feeds, and blog homepages• Task changes• faceted blog distillation (authority, expertise)• news + blog tracking pilot

New web collection• 1 billion web pages (~ 25 TB uncompressed)• Collected by CMU (with advice from major searchengines)• Spans the 10 most prominent languages on the web• Available on a set of hard drives for nominal cost
• We are exploring making the collection available
from one or more clusters• And establishing subsets of the collection• RF, MQ, and two new tracks are planning to use thiscollection

Web Track• Coordinators: Nick Craswell (MSR), CharlesClarke (U Waterloo)• A reborn web track• Focus on web search tasks• navigational, topic distillation• diversity ranking• failed queries• spammed queries• Driven by query logs and click data

Entity Track• An entity is something with a homepage – people,
products, organizations...

• Coordinators: Krisztian Balog (UvA), Paul Thomas(CSIRO), Arjen de Vries (CWI), Thijs Westerveld(Teezir)• Task: related entity finding• given a homepage, the entity type, and a narrative• return related entity homepages• ex: find studios that Tom Cruise worked with
• Other tasks under discussion

TREC 2009
• Blog• Chemical• Legal• Web• Entity• RelevanceFeedback• Million Query• The call for participation is out now:
http://trec.nist.gov/• CfP includes addresses of track mailing lists
and other track detailssseelm
Sha lug!
PICWSM 2009

• The International Conference on Weblogging andSocial Media has a data challenge• (chairs: Ian Soboroff and Akshay Java)• Data: 40 million blog posts covering 8 weeks (Aug- Oct 2008) (provided by Spinn3r.com)• The data is free to get, free to use• Conference papers due January 21st
• Data workshop papers due March 1st• http://icwsm.org/