EVENT REPORT

Report on the 21st Dutch-Belgian Information
Retrieval Workshop (DIR 2023)
Avishek AnandMaria Soledad Pera

Maria Heuss

WIS - TU Delft
Netherlands

WIS - TU Delft
Netherlands

University of Amsterdam
Netherlands

avishek.anand@tudelft.nl

m.s.pera@tudelft.nl

m.c.heuss@uva.nl

Venktesh V

Matteo Corsi

TU Delft
Netherlands

TU Delft
Netherlands

v.viswanathan-1@tudelft.nl

m.corsi@tudelft.nl

Abstract
This year, the 21st edition of the Dutch-Belgian Information Retrieval Workshop (DIR), washosted by TU Delft. Following a series of online interactions due to the COVID-19 pandemic,
this edition returned to its traditional in-person format. In this report, we provide a summaryof the program and discuss key insights and lessons learned gleaned from the event.
Date: 27 November 2023.
Website: https://dir2023.github.io/DIR2023/.

1Overview

The Dutch-Belgian Information Retrieval (DIR) Workshop, has an established tradition datingback to the year 2000.1 Historically, DIR has been a forum for academic researchers (at different
stages of their careers, from bachelor’s to Professors) and industry practitioners in InformationRetrieval (IR) come together to connect, exchange ideas, and discuss the latest developments in IRand related research areas taking place at different academic institutions and other organizations
across the Netherlands (NL) and Belgium (BE).
This year, in its 21st edition, DIR took place at TU Delft–specifically at the Mondai House ofAI–and was co-organized by Avishek Anand and Sole Pera (both IR “oldies” but relatively new
in the Netherlands’ IR landscape), with the support of PhD students Matteo Corsi (MultimediaComputing Group - TU Delft), and Maria Heuss (IRLab - University of Amsterdam), in addition
to post-doctoral researchers Venktesh Viswanathan (Web Information Systems - TU Delft).
We invited quality research contributions addressing relevant challenges in IR from both doctoral students and early-stage researchers. Contributions ranged from theoretical work to descriptions of applied research and real-world systems. We also elicited contributions focused on
1Earlier DIR editions can be found here https://www.informatiewetenschap.org/dir-history.html

ACM SIGIR Forum

1Vol. 57 No. 2 – December 2023

IR resources, such as datasets and ongoing European projects, to expand the visibility of theseresources among the DIR community. Our main aim for this edition of DIR was to bring together
the IR community in NL and BE (and beyond) by: offering a platform to present important ideas and results in IR in the last two years,
 allowing early-stage researchers to present and get feedback on their research vision, discussing challenges related to diversity and equality in the IR community, and fostering interaction and networking among researchers and practitioners.

Participants from several industry and academic institutions across NL and BE joined DIR
2023 for a day of discussion, inspired by presentations, a keynote address, and a session addressing
the importance of diversity within the IR community. In the rest of this manuscript, we sharesome of the main highlights of the day and lessons learned.

2The Latest on IR from NL and BE

The program included 11 presentations [Knyazev and Oosterhuis, 2023; Heuss et al., 2023; Menget al., 2023; Joko and Hasibi, 2022; Rostami et al., 2023; Oosterhuis, 2023; Deffayet et al., 2023;
Askari et al., 2023; Vrijenhoek et al., 2023; Gupta et al., 2023; Granitzer et al., 2023], which
spotlighted the various areas of IR in which researchers and industry practitioners are focused on–
from recommender systems and evaluation to Learning to Rank, Large Language Models (LLMs),and infrastructure for web search.
During the first session, attendees were exposed to a variety of timely IR topics–from uncertainty measurement and calibration in ranking and recommender systems to normative aspects
impacting IR design. This session also highlighted the wide variety of applications for which IR
plays a lead role, such as health-aware food recommender systems and conversational search. Theworks also highlighted the importance of classical approaches for problems like entity linking, tomitigate hallucinations in the era of LLMs.
The second session started with a call to action to reclaim search by Arjen de Vries2 , and how
can we envision personal search engines of the future. This was followed by several presentations
that highlighted machine learning methods applied to ranking tasks that aimed to improve click
estimation and modeling in Learning to Rank. The talks also highlighted the importance of riskaware ranking approaches for safe deployment. The session also covered approaches and artifacts
related to classic text ranking tasks, including query performance prediction and query expansions.
LLMs were also featured in this session when we learned about more effective prompting strategies
and expansion techniques using LLMs for text ranking.

3A Glimpse into the Future of IR

This year’s keynote was delivered by Suzan Verberne, a professor of Natural Language Processing(NLP) at the Leiden Institute of Advanced Computer Science at Leiden University. Her research
focus is to advance NLP “beyond the benchmark”, addressing challenging problems in specific2

https://openwebsearch.eu/

ACM SIGIR Forum

2Vol. 57 No. 2 – December 2023

domains. This involves the development of LLMs for low-resource settings, and addressing issues
related to misinformation, transparency, and bias.
Her keynote address was on the topic of “Information Retrieval in the Age of LLMs”. She
highlighted the advantages of approaches from IR as opposed to the direct usage of LLMs for IRtasks. She presented recent research that leverages LLMs to improve IR systems, for example forthe enrichment of sparse data. Prof. Verberne also discussed how IR techniques can be applied
more directly to use LLMs for retrieval. Additionally, she reflected on how IR approaches likeRetrieval Augmented Generation (RAG) can help ground LLM outputs and mitigate pitfalls likehallucinations. She also invited the audience to critically consider the challenges and potential
risks of LLMs in these contexts. Prof. Verberne also highlighted the need for open-source LLMs
to encourage accessibility and reproducibility. In summary, the talk offered a balanced view ofhow LLMs and techniques in IR can be employed together to further progress in diverse tasks.
The discussion that followed her keynote address made apparent that there are several open
questions concerning the capacity of LLMs and the risks of using these models for IR tasks. There
were also discussions about the need for robust and holistic evaluation of LLMs applied to IRtasks. Prominent was also the need to avoid ‘re-inventing the wheel’ when considering how to use
LLMs for tasks like retrieval. While the IR community has accumulated experience and a great
number of techniques for these tasks, the exploding field of LLMs seems to often be unaware ofadvances in our field. This was attributed to the (relatively small) size of our field, as well as theswift pace at which research on LLMs has been advancing.

4Diversity in IR

To bring attention to diversity–a topic of interest not only to the IR community in NL and BE, but
worldwide–we invited Dr. Cynthia C. S. Liem MMus, an Associate Professor in the Multimedia
Computing Group at TU Delft, and pianist of the Magma Duo. Her research interests are intrustworthy and responsible AI; she especially focuses on techniques that make people discover
new interests and content that would not trivially be retrieved and questions of validation andvalidity in data-driven decision-making.
Dr. Liem discussed her academic trajectory and difficulties in finding acceptance and support
for her ideas and projects. She described the difficulties of conducting research that may not becommercially viable or considered highly influential in society. She also shared her experienceswith inter- and trans-disciplinary research and the associated challenges, all the while highlighting
the impact she made as an educator and communicator who bridged engineering expertise with
policy-making. Dr. Liem concluded her talk by providing some recommendations to the audienceon actively creating a more diverse and inclusive academic community–one that encourages diverse
perspectives and ideas.

5Lessons Learned

To wrap up, we summarize organizational lessons learned during the planning and execution of
the DIR conference, which could be instrumental for future editions. We also share some general
impressions emerging from the event.
ACM SIGIR Forum3Vol. 57 No. 2 – December 2023

1. Scheduling. Selecting an appropriate time slot is crucial for maximizing participation. We
strategically scheduled DIR 2023 towards the end of November, positioning it two weeks
after the Dutch-Belgian Conference on AI and ML (BNAIC)3 and two weeks before theDutch-Belgian Workshop on Recommender Systems(DBWRS)4 . This timing places DIR at
the culmination of the conference cycle, following CIKM. However, a concurrent overlap
with SIGIR-AP was an oversight. Future planning should consider such potential conflict.
2. Participant Estimation & Support. DIR 2023 received generous support from SIGIR
and a support statement from SIKS. A major challenge was estimating the actual number of participants until the event’s conclusion. Notably, there was a surge in last-minuteregistrations, aligning with our upper-limit expectations.
3. Inclusivity & Registration Costs. We consciously decided to keep registration costslow (15 euros for students and 25 euros for senior researchers) to enhance event inclusivity.
Despite efforts to extend our reach beyond NL and BE to include Germany and Austria, theparticipation from these countries was below our expectations. For future editions, it may
be beneficial to directly engage with researchers in these regions and consider offering travel
grants to support broader participation.
4. Promotion & Outreach. To promote DIR, we relied on a short yet effective social media
advertising campaign, utilizing platforms like X and LinkedIn, in addition to traditional
mailing lists. This approach was particularly important in re-establishing DIR’s presence
post-COVID, contributing significantly to its visibility and attendee engagement.
5. Interaction & Networking. To foster interchanges among participants, presenters wereasked to keep their presentations relatively short (up to 10 minutes), leaving 5 minutes for
Q&A. Moreover, we scheduled extended coffee and lunch breaks, to facilitate networking
and mingling among attendees.
Diversity manifests in various forms. Examining attendance, we see that from among 60
registrations, 70% students were students (Masters and PhDs) and 30% senior scientists. Onlyaround 10% of DIR 2023 participants had an industry affiliation; an imbalance was also observed
among attendees from NL vs. BE. Future organizers should consider addressing these industryacademia and geographic imbalances. However, there was a less noticeable gender imbalanceamong attendees, with nearly 30% identifying as females, a ratio deemed acceptable considering thefield. Dr. Liem’s presentation also sparked discussions on diversity, particularly on the challenges
related to hiring interdisciplinary research teams and advising policy-makers. Further, young
researchers had the opportunity to ask for advice on achieving success in their pursuits, evenwhen their interests and research areas of focus deviate from ‘traditional’ expectations.
Discussions throughout the day kept coming back to “What is the future of (IR) topic ‘X’ inthe era of LLMs?” (where X= name entity linking, answer generation, query expansion, etc.). It
was indeed refreshing that the conversation did not center only around maximizing performance–a
notion typically emphasized in IR-related conference proceedings and journals. Instead, during
DIR 2023, attendees revisited perspectives on how the state-of-the-art can be shifted to increase
fairness, enable predictions with confidence values, or balance performance with ancillary objectives such as fairness or normative aspects, among others.
34

https://bnaic2023.tudelft.nl/
https://www.dbwrs23.be/conference

ACM SIGIR Forum

4Vol. 57 No. 2 – December 2023

Acknowledgments
We want to thank past DIR organizers for sharing their insights and lessons learned. We extend
our appreciation to the event sponsors: ACM SIGIR and SIKS. We also acknowledge the TU Delft
AI Initiative for the publicity support and help around communications. We would be remiss if
we did not thank Claudia Hauff, Emiliana Murgia, and Lorenzo Corti for their help with the event
logo and branding; along with Lijun Lyu and Jurek Leonhardt for their on-site support.

References
Arian Askari, Mohammad Aliannejadi, Evangelos Kanoulas, and Suzan Verberne. A test collectionof synthetic documents for training rankers: Chatgpt vs. human experts. In Proc. of ACM
CIKM, pages 5311–5315, 2023.
Romain Deffayet, Philipp Hager, Jean-Michel Renders, and Maarten de Rijke. An offline metricfor the debiasedness of click models. In Proc. of ACM SIGIR, pages 558–568, 2023.
Michael Granitzer, Stefan Voigt, Noor Afshan Fathima, Martin Golasowski, Christian Guetl,Tobias Hecking, Gijs Hendriksen, Djoerd Hiemstra, Jan Martinovič, Jelena Mitrović, et al.
Impact and development of an open web index for open web search. JASIST, 2023.
Shashank Gupta, Harrie Oosterhuis, and Maarten de Rijke. Safe deployment for counterfactual
learning to rank with exposure-based risk minimization. In Proc. of ACM SIGIR, pages 249–258,2023.
Maria Heuss, Daniel Cohen, Masoud Mansoury, Maarten de Rijke, and Carsten Eickhoff. Predictive uncertainty-based bias mitigation in ranking. In Proc. of ACM CIKM, pages 762–772,2023.
Hideaki Joko and Faegheh Hasibi. Personal entity, concept, and named entity linking in conversations. In Proc. of ACM CIKM, pages 4099–4103, 2022.
Norman Knyazev and Harrie Oosterhuis. A lightweight method for modeling confidence in recommendations with learned beta distributions. In Proc. of ACM RecSys, pages 306–317, 2023.
Chuan Meng, Negar Arabzadeh, Mohammad Aliannejadi, and Maarten de Rijke. Query performance prediction: From ad-hoc to conversational search. In Proc. of ACM SIGIR, pages
2583–2593, 2023.
Harrie Oosterhuis. Doubly robust estimation for correcting position bias in click feedback forunbiased learning to rank. ACM TOIS, 41(3):1–33, 2023.
Mehrdad Rostami, Mohammad Aliannejadi, and Mourad Oussalah. Towards health-aware fairness
in food recipe recommendation. In Proc. of ACM RecSys, pages 1184–1189, 2023.
Sanne Vrijenhoek, Lien Michiels, Johannes Kruse, Alain Starke, Nava Tintarev, and Jordi Viader Guerrero. Normalize: The first workshop on normative design and evaluation of recommender systems. In Proc. of ACM RecSys, pages 1252–1254, 2023.
ACM SIGIR Forum5Vol. 57 No. 2 – December 2023