WORKSHOP REPORTDIR 2011: the eleventh Dutch-BelgianInformation Retrieval Workshop
Corrado Boscarino1
c.boscarino@cwi.nl
Edgar Meij2edgar.meij@uva.nl
1

Katja Hofmann2
k.hofmann@uva.nl
Maarten de Rijke2
derijke@uva.nl

Valentin Jijkoun2
jijkoun@uva.nl
Wouter Weerkamp2
w.weerkamp@uva.nl

Centrum Wiskunde & Informatica, The Netherlands
2
University of Amsterdam, The Netherlands
Abstract

The 11th edition of the annual Dutch-Belgian Information Retrieval workshop (DIR 2011)
took place on February 4 in Amsterdam. It was organized by the University of Amsterdam
and the Centrum Wiskunde & Informatica. The focus of this year’s workshop was on interaction, with the goal of facilitating and increasing interaction, especially within the local
research community, and between industry and academia. The scientific program included
demos, research papers, and compressed contributions. The keynotes by Nick Belkin andGabriella Kazai provided intriguing outlooks on the future of IR evaluation.

1Introduction

The Dutch-Belgian Information Retrieval workshop (DIR) is an annual event that promotes
exchange between IR researchers in the region. This year’s workshop took place on February
4 in Amsterdam. As in previous years, DIR 2011 reflected a growing IR community in theNetherlands and Belgium. With more than 110 attendees, this was the largest DIR to date.
Some new groups were also present besides the regional community, including international
attendees, participants from industry, and researchers from related fields such as machinelearning. DIR 2011 focused particularly on interaction. The organizers reached out to amore varied audience, and created a program that allowed space for interaction around avaried and high-quality scientific program.
Below we highlight some aspects of the program. The full program and the proceedings
are available on the workshop’s website: http://ilps.science.uva.nl/dir2011.

2Keynotes

The two keynotes provided insights into how IR evaluation could change in the future, inparticular with regarsd to human aspects of evaluation.

The first keynote was given by Nick Belkin of Rutgers university. He argued that the evaluation paradigm that focuses on recall and precision—developed with a view of information
seeking in library systems more than 50 years ago—is not sufficient for evaluating many of
today’s retrieval systems. Instead, he argued for “usefulness” as a criterion for IR evaluation.
Belkin demonstrated how this measure could be used to evaluate interactive systems, evenover complete information seeking episodes.
The second keynote was given by Gabriella Kazai of Microsoft Research Cambridge, andfocused on crowdsourcing. Crowdsourcing has become increasingly popular for collecting
data where human input is required. Work is split up into small tasks that are completed byworkers across the world, who are compensated for their work with small amounts of money
per completed task. Crowdsourcing is an interesting way of obtaining human relevance labels
for search evaluation quickly and relatively inexpensively. Kazai discussed her recent andongoing work concerning quality assurance of the data that can be obtained in this way.

3

Research Contributions

An important aspect of DIR is to provide junior researchers with a forum for presenting
new research ideas and discussing work in progress. Five original research contributions wereaccepted.
Carter, Tsagkias, and Weerkamp addressed the problem of language identification in microblogs. They found that state-of-the-art language identifiers show suboptimal performancein this setting, and that performance can be improved using blogger-based and link-basedpriors.
Dadvar, Hauff, and de Jong addressed the problem of negation detection for analyzing thepolarity of movie reviews. They find that traditional negation detection approaches cannot
capture implicitly expressed negations that often occur in the studied domain and suggest
further study of the lexical features that are used for implicit sentiment expressions.
Hinne, van der Heijden, Verberne, and Kraaij presented a new model for characterizing
search intent. The authors suggest a multi-dimensional model that takes into account action,
form, and mode of the result, and provide examples of how this classification can be used toanalyze a search log.
Lubell-Doughtie and Hofmann study the use of Probabilistic Latent Semantic Analysis
for result diversification. They find that interpolating pLSA-based clusters and document
relevance can improve result diversity, but that the topics induces by pLSI are often suboptimal.
Tigelaar and Hiemstra study caching as a way to reduce delays in peer-to-peer IR networks. In realistic simulations based on live traffic, the authors find that caching may be aneffective way of reducing delays, with an estimated cache hits ratio of 30% or more.
In future editions of DIR, we would like to encourage more junior researchers in the IRcommunity to submit their work.

4

Compressed Contributions

Compressed contributions were first introduced at DIR 2010. These summaries of papers
that were published at international conferences in in international journals allow attendants

to become familiar with some of the most interesting IR research that has recently been
completed in the area. For DIR 2011, 14 submissions were accepted.
The accepted contributions covered a broad range of recent work in IR that had beenpreviously published in JASIS&T and at SIGIR’10, CIKM’10, ACL’10, ECIR’10, SAC’11,and AWIC’11. Contributions addressed the use of anchor text or tags for retrieval, entity
retrieval, sentiment analysis, graph measures for query similarity, annotating web pages,
diversification, language models and word clouds, transaction log analysis, analysis of news
comments, and retrieval in context.
Particularly intriguing was the paper by Yilmaz, Kanoulas, Robertson, and Aslam, who
combined several of their recent publications on constructing collections for learning to rank.
Assuming that a fixed amount of resources was available for building a collection, the authors
found that obtaining shallow judgments for a large number of queries leads to better testcollections, deep judgments for a smaller set of queries. The authors also discussed which
method was most appropriate for building such a collection and that a learning algorithm
should be optimized for more informative measures, such as average precision or gradedaverage precision.

5DemosThe demo session was particularly successful, with 10 high-quality submissions. Both participants from industry and from research groups presented and explained their most recentdevelopments. The demos included fuzzy faceted search, browsing topic structure for videos,an IR system for children, analysis of Dutch-language news, annotation-based IR for multimedia, natural language search, semantic search for recruiters, associative catalogue search,a term extractor for dutch, and online sentiment monitoring.

6Conclusion

DIR 2011 was a very successful event, with the largest attendance yet. The high-quality
program enabled interaction between researchers from the local and international community,
as well as industry and government representatives with an interest in IR. The proceedingsof the workshop can be downloaded from the workshop website. The 12th edition of DIR
will take place in Ghent, Belgium.

Acknowledgements
We would like to thank everyone who made DIR 2011 possible. In particular, we thank theprogram committee, who put a lot of time and effort into preparing detailed reviews thatprovided valuable feedback for contributing authors.
DIR 2011 was organized by the Intelligent Systems Lab of the University of Amsterdam,
supported by the Centrum Wiskunde and Informatica. The event was organized under theauspices of SIKS and WGI, and generously sponsored by TomTom, Textkernel, Q-Go, theSTEVIN programme, Teezir, Google, NWO and the University of Amsterdam.