EVENT REPORT

Report on the 1st Training School on Domain
Specific Systems for Information Extraction and
Retrieval (DoSSIER 2022)
Michail Salampasis

Florina Piroi

Allan Hanbury

International Hellenic University
Thessaloniki, Greece

TU WienVienna, Austria

TU WienVienna, Austria

msa@ihu.gr

florina.piroi@tuwien.ac.atallan.hanbury@tuwien.ac.at

Abstract
The DoSSIER project, an European Training Network, was kicked-off in late 2019. The first
PhD candidate recruiting activities took place at the beginning of the Covid-19 pandemic
and the first years of the project lived in the on-line universe. Naturally, this posed certain
challenges to the cohesion of our network. With some delay due to the lock-downs and travel
restrictions, our first Training School was successfully organized in September 2022 and it
constituted one of the few occasions we had where the members of DoSSIER network met inperson. This report gives an account on the first Summer School organized by DoSSIER, for
the DoSSIER Early Stage Researchers, a school set in the picturesque village of Olympiada,
which gave amply opportunity of interaction between students and senior researchers in the
Information Retrieval domain.
Date: 25–30 September, 2022.
Website: https://dossier-project.eu/1st-dossier-training-school.

1

Introduction

DoSSIER1 is an EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval. DoSSIER will elucidate, model, and address the different information needsof professional users.
DoSSIER groups its research sactivities into three general areas (Applications, Methods, andModels), pertaining to three domains: Innovation, Health, and Legal/Law. These feed into eachother to generate new hypotheses, identify new experimental procedures, and bring about a better
understanding of knowledge and information needs, and the processes by which the two interact.
The programme is implemented via 15 Early Stage Researcher (ESR) projects. Eight of the 15
projects clearly identify one of the three application areas while the remaining seven projects
provide horizontal links across domains and contribute to the harmonization of observations,1

https://dossier-project.eu/

ACM SIGIR Forum

1

Vol. 56 No. 2 December 2022

cross-fertilisation of ideas, and identification of synergies between research groups, domains, andtopics.
The results of the DoSSIER research will provide the vital know-how and tools to the professional search industry. DoSSIER aims to make an impact on professional search processes andsystems through research in all three areas. An additional key objective is the education andtraining of Early Stage Researchers. Specifically, within the DoSSIER network we train a newgeneration of scientifically-principled, creative, entrepreneurial and innovative researchers withthe academic and industrial experience necessary to make a significant impact on professional
search in Europe, and hence on the European economy. DoSSIER fosters excellence by structuring research and doctoral training to lead to a professional certification of the competences:
research knowledge and intellectual abilities; research personal effectiveness; research governanceand organization (including ethics and sustainability); and researcher engagement, influence andimpact.
In this context, from 25 to 30 of September 2022, in collaboration with the Intelligent Systems Laboratory of the International Hellenic University, DoSSIER has organized in Olympiada(Chalkidiki, Greece) the 1st DoSSIER Training School as an internal training event. The small seaside village of Olympiada is a popular summer destination in the larger area known as Chalkidiki
and mount Athos. Only half a kilometre from the current settlement of Olympiada is the locationof Ancient Stageira, birthplace of Aristotle.
The school was intended for PhD students working in the 15 DoSSIER projects. The project
partners collaborated on designing the most appropriate set of lectures answering both to thestudents’ expressed requests for specific training topics and to the DoSSIER project aims forresearcher training. The 1st DoSSIER Training School was planned as a week-long event consistingof a series of lectures and seminars. The aim of the school was to give grounding in core research
topics (e.g. IR Experimentation), but also to provide training in other subjects relevant to theirresearch activities (e.g. Licensing practices, Search in Industry settings).

2

Lectures

The program started with two lectures focusing on the main topics of IR experimentation. Then it
moved to User Studies, Information Interaction, Interfaces and Evaluation, and Search Systems forClinical Decision Support. The next days contained lectures focusing on Challenges of Enterprise
Search, Reproducibility in IR and Code Development in Industry settings. Finally, the school
was completed with lectures on paths to a productionised service and licensing practices. Each of
the twelve lecturers had two to three hours for presenting their topics. A short description of all
lectures follows:• Modelling and Analyzing Information Retrieval System Performance: FromFoundations to Measurement Issues.
by Nicola Ferro, University of Padua, Department of Information Engineering
Starting with a brief background on the foundations of Information Retrieval Evaluation,
Nicola introduced advanced IR topics for retrieval systems evaluations, including the notionof hypothesis testing and types of errors, as well as approaches on how to deal with multiACM SIGIR Forum2

Vol. 56 No. 2 December 2022

ple comparisons and Family-wise Error Rate (FWER). The lecture also included modellingsystem effects using Analysis of Variance (ANOVA), i.e. one-way ANOVA, and a detailed
explanation on how to incorporate system and topic effects (two-way ANOVA) to obtain amore reliable comparison across different retrieval systems.
• Towards Better IR Experimentation: Validity and Reproducibility.
by Norbert Fuhr, Universitat Duisburg-Essen, Informatik und Angewandte Kognitionswissenschaft
Norbert’s talk pointed out common mistakes found in Information Retrieval publicationsin terms of validity and reproducibility. He explained that validity could be placed into two
categories: (1) internal validity: the degree to which a study minimizes systematic error,and (2) external validity: the degree to which the results of a study can be generalized.
Reproducibility means the results are reproducible independently. In other words, an experimental result is not fully established unless it can be independently reproduced. Based on
his presentation, a summary of common mistakes in IR are: hypotheses are often formulated
after the experiment, significance tests frequently ignore the multiple comparisons problem,
MRR and ERR evaluation metrics violate basic requirements for a metric, MAP is basedon unrealistic assumptions, effect sizes are ignored, reproducibility of the experiments might
be nearly impossible, and sometimes authors claim proof by experimentation while proofs
are about universally valid statements. Finally, Norbert outlined the possible solutions forovercoming the common mistakes such as Bonferroni correction for multiple comparisons,computing Effect Size, adding Metadata into the raw data, defining workflows in terms ofrules, and Containerization with Docker.
• User Studies.
by Katriina Bystrom, OlsoMet, Faculty of Social Sciences.
Katriina’s presentation covered three main parts: “who is a user”, “how are user’s information needs and their information sources related in a workplace”, and “why do a user
study in a workplace”. Katriina discussed some central concepts for a user study in a workplace setting, such as user, context, information source, and information activities (e.g.,
information needs, information searching, information seeking). In small groups, the audience discussed how to define the information needs for different types of human users.
According to Katriina, information need(ing) is seen as an activity alongside information
seeking and searching. Katriina ended the presentation by providing a user study pre-design
guide where she shows three study modes: Experimental (pseudo-contextual; often systemcentred), Semi-naturalistic (semi-contextual; often person-centred), and Naturalistic (fully
contextual; often ethnographic). She explained the aims, goals, methods, impact, challenges,
and ethics for each mode.
• Code Development in Industrial Settings.
by Antonis Makropoulos, contextflow

ACM SIGIR Forum

3

Vol. 56 No. 2 December 2022

Antonis’ lecture presented the challenges of developing a search system for clinical decision support. He presented the example of contextflow, a startup working with radiology
images to identify diseased areas in the lungs. He emphasised the need for such systems inthe daily work of a radiologist. The presentation, then, focused on the steps necessary tomove from the research ideas and results to a commercial product from the perspective of
R&D. The first stages of each project should identify available models, datasets and performance metrics on which one could evaluate their solutions. Method development andtesting should involve thorough optimisation and documentation and testing for both single
performance metrics and user satisfaction of the end-to-end system. He also emphasised theimportance of adhering to good coding practices like unit tests, code reviews, and following
coding standards (e.g. Python’s PEP-8). Each idea in the presentation was supported byexamples from the development of the contextflow SEARCH Lung CT software.
• Information Interaction, Interfaces and Evaluation.
by Elaine Toms, University of Sheffield, Management School.
This lecture was a continuation of Katriina Bystrom’s lecture in which Elaine defined theconceptual foundations underpinning information search/retrieval. The lecture had twoparts. In the first part, Elaine discussed information interaction, interfaces and integrating
search. She talked about the context in which knowledge workers operate and defined how
to integrate information interaction into knowledge work. Elaine further discussed the importance of graphical user interfaces and provided concrete design principles to designing
an interface which take into consideration cognitive tasks and tools. The second part ofthe lecture focused on the evaluation of the graphical user interfaces. Elaine discussed thelandscape of interface evaluation and different aspects of an interface that we can evaluate
and then talked about the different elements of interface evaluation. Finally, all the steps
involved in designing a solid user study from the data collection to the evaluation were outlined. The talk was then closed with the audience having a discussion about how they would
apply the theory they learned to their own research projects.
• Challenges of Enterprise Search.
by Udo Kruschwitz, Universitat Regensburg, Information Science.
This lecture started with a brief introduction to various types of searches such as desktop
search, site search, blog search, database search, expertise search, etc. Various enterprisesearch examples and problems related to it were discussed. The lecture continued to mentiondifferences between enterprise and web search, consisting mainly of differences with respect
to the source of data, to smaller search scale, to lack of interconnected documents. Amongthe challenges of designing an Enterprise Search system Udo referred to the fact that in nonenterprise settings content is often created for dissemination purposes and not necessarily
for answering possible future information needs. Other challenges include the handling of
documents with different access restrictions (which a retrieval system may find but not beable to display in the result list), or the handling of redundant, outdated, or trivial data.

ACM SIGIR Forum

4

Vol. 56 No. 2 December 2022

• A New Generation of Neural Search and Knowledge Discovery Tools.
by Jakub Zavrel, Zeta Alpha, Amsterdam.
In his presentation, Jakub made the claim that large pretrained language models will augment human cognition for making decisions. He presented how Zeta Alpha brings thisphilosophy into reality with their knowledge discovery platform for the scientific domain. In
this platform they leverage state-of-the-art neural search to create a contextual knowledge
discovery system. He presented use cases of the Zeta Alpha platform where these novel
methods are used to overcome known problems of traditional retrieval models, like lexical
gap, bridging them with semantic understanding and handling of complex, relational andmultilingual queries. Further, he presented their research for domain adaptation of neuralsearch models with unsupervised data generation.
• Designing Search @Spinque.
by Roberto Cornacchia, Spinque.
“It is time to design your own search engine.” That initial quote in Roberto’s lecture
outlines, in a few words, Spinque’s primary mission. Spinque is a spin-off company thatleverages clients’ unique expertise to create the best search solutions for them. In his presentation, Roberto describes how they achieve their goal by analysing the different productdesign phases. By integrating data from multiple sources, they represent implicit information in a knowledge graph. As outlined in several parts of the lecture, an essential
part of product design is dedicated to understanding clients’ needs and search tasks before
designing the necessary search solutions. The latter involves relevance estimation and is
implemented using their state-of-the-art column store database technology. Having outlined
these components, Roberto presented two search applications related to cultural heritageand e-commerce, along with a hands-on session that allowed him to provide further detailsof the product design. Concluding his lecture, he mentioned the importance of the humanfactor in designing tailored search solutions for complex search problems.
• Reproducibility in IR for Evolving Document Corpora.
by Andreas Rauber, Technische Universitat Wien, Institute of Information Systems Engineering.
The subject of the lecture given by Andreas was the need for reproducibility in IR. Indeed, even if a study has a very high scientific value, it is difficult to integrate it in thelarger context of research if it is not reproducible. For instance, if documents are changed ordeleted, it would affect the ranking list, even if the documents are not present in the ranking.
As for user studies, it should be possible to reproduce the experiments with different users.
The solution would be to create algorithms that would be robust on changing data or having a different group of participants. One of the solutions would be dynamic data citation,
which consists of working the data which contains time of access information, enhancing themeans of access to the data (queries) with time stamping and hashing the value computed
over the data for local storage. If the data changes, it is necessary to identify the queries
ACM SIGIR Forum

5

Vol. 56 No. 2 December 2022

which are affected. Also, column store systems allow tracking of changes in the documents,
along with their insertion to and removal from the collections.
• Licensing Practices in Open Science and LCT tool.
by Marianna Katrakazi, Legal associate, ATHENA RC.
Marianna talked about copyright and intellectual property licensing in the Open Science
context. She introduced the Licence Clearance Tool, developed within the NI4OS project,that aims to assist researchers in selecting the appropriate licences for sharing their codeor data. During this talk, the audience also discussed the researcher’s interests on project
outcome licensing, and elucidated how to approach intellectual property licensing schemes
from the resource creator roles. Given the rapid and wide distribution of new knowledge
published openly and the adopted approaches for sharing outcomes of science, it is important to be aware of the legal implications of this practice. Legal boundaries allow us toprotect our work and define the limits of the usage for outcomes of our research. We should
ask ourselves what we want to share and how much others could and should use what weshare. Since, in general, access to this specific information is limited, Marianna presented
the License Clearance Tool, which could help by bringing awareness of, for example, theimplications of choosing a specific licence and how licences are compatible between them.
• From an Experiment to Production.
by Petr Knoth, Open University, Big Scientific Data and Text Analytics GroupIn his lecture, Petr introduced CORE.uk which emerged from his PhD work and is now
one of the world’s largest aggregator of Open Access research papers retrieved from Open
Access repositories, with a team of 12 people actively maintaining it. CORE.uk’s mission is
to aggregate all Open Access research worldwide and deliver unrestricted access for all. In theprocess of aggregating content, CORE.uk relies on open protocols, such as OAI-PMH, andhas a 6-stage pipeline to create a final index of Open Access resources. The pipeline’s firstcomponent downloads the metadata of Open Access articles. According to Petr, CORE.uk
now scans over 10,000 data providers in over 90 languages which gives CORE.uk over 218
million metadata records, of which around 28 million include full-text Open Access publications, stored by CORE.uk. CORE.uk has around 30 million monthly active users thataccess the repository in 4 different uses: 1) content search and discovery, recommendations;2) access provision to open access content in a machine readable format doing all the heavy
lifting with the help of AI, which involves harmonizing metadata, cleaning and creating a
structured representation of scholarly data, accessible through CORE.uk’s ”Raw Data Services; 3) own repository management, through a specially designed dashboard, available
to academic subscribers, with overviews of repository harvesting status; 4) OIA resolutionof metadata record identifiers, harvested and automatically registered as globally uniquepersistent identifiers (PIDs)

ACM SIGIR Forum

6

Vol. 56 No. 2 December 2022

Figure 1: School participants before the panel discussion in Ancient Stageira, Olympiada,
Chalkidiki.

3

Conclusions

Thirteen PhD students attended the school and also discussed and coordinated their efforts todevelop a search engine prototype for domain-specific tasks in a scheduled session. The organization of the panel on the topic of “Do we need academia in AI?” in the Agora of Ancient Stageira,birthplace of Aristotle, has been an inspiring moment to bridge ideas on ethics and research fromthe times of the ancient Greeks to the present society.
The lectures took place in the local municipal cultural center. There were interesting discussions during and after each lecture, and intense individual interactions between trainees andlecturers, which led to ideas for further research collaborations and research internships. The
trainees had the opportunity to get to know experienced researchers working on the same, similar
or complementary research areas.
Participants and lecturers made very positive comments regarding the location and organization of the training school. The location, a small village, offered an ideal environment where all
the participants were focused on the training events. During the lectures the students participatedactively by asking questions, which led to extensive off-line discussions with the lecturers during
the coffee breaks, lunches, or dinner time. The organized social events allowed all participants to
get to know each other and establish friendships.
As the summer school reached its end, after five days of intense attendance and tutoring, thegeneral feeling was very positive and pleasant, making the 1st DoSSIER Training School a very
successful event.
ACM SIGIR Forum

7

Vol. 56 No. 2 December 2022

Acknowledgements
This event was supported by the DoSSIER project, an EU Horizon 2020 project under the Marie
Sklodowska-Curie grant agreement No 860721. Many thanks also to the mayor of Olympiada foroffering the local cultural center as the venue to host the lectures and to the local community forsupporting the event in general. Finally, thanks must go to the DoSSIER ESRs for contributing
the paragraphs with their views and comments about the lectures presented.

ACM SIGIR Forum

8

Vol. 56 No. 2 December 2022