WORKSHOP REPORTPROMISE Technology Transfer Day: Spreading the Word on
Information Access Evaluation at an Industrial Event
Pamela Forner
CELCT, Italy
forner@celct.it
Martin Braschler
Zurich University ofApplied Sciences,
Switzerland
bram@zhaw.ch
Nicola Ferro
University of Padua,Italyferro@dei.unipd.it
Jussi Karlgren
Gavagai, Sweden
jussi@gavagai.se

Luisa Bentivogli
FBK, Italy
bentivo@fbk.eu
Khalid Choukri
ELDA, France,
choukri@elda.org

Allan Hanbury
Vienna University ofTechnology, Austria
hanbury@ifs.tuwien.ac.at
Henning Müller
University of Applied
Sciences Western
Switzerland, Switzerland
henning.mueller@hevs.ch

Abstract
The Technology Transfer Day was held at CeBIT 2013 fromMarch 5 to March 9, at the Deutsche Messe in Hannover,Germany. PROMISE presented three events at CeBIT: apanel in the CeBIT Global Conference (CGC) - Power Stage,a one-day workshop hosted in the CeBIT Convention Center,and a stand "EU Language & Big Data Projects" in Hall 9.
The whole program included 4 panelists, 12 invited talks, anddiscussions among the speakers and with the public. This
report overviews the aims and contents of the events andoutlines the major outcomes.

1

Introduction

PROMISE1 is a European Union (EU) funded research project for advancing the evaluation and benchmarkingof multimedia and multilingual information systems. One of the key activities is to make industrial
stakeholders aware of the merits of systematic evaluation activities. Since web-based information access ismore and more becoming a commodity of predictable quality and transparency, enterprise systems are pressedto provide similar user experiences. However, the quality of systems in terms of target notions such as
relevance, enjoyability, or effectiveness is seldom evaluated by industrial stakeholders. To this end, PROMISE
1

http://www.promise-noe.eu

ACM SIGIR Forum

53

Vol. 47 No. 1 June 2013has prepared a report on Best Practices [3] and organized a Technology Transfer Day2 at CeBIT 20133 fromMarch 5 to March 9, at the Deutsche Messe in Hannover, Germany.
CeBIT is the world's largest IT and communications trade fair and an international forum where professionals
come to evaluate industry trends, network with peers, and discover the latest technology products andsolutions.
This year, CeBIT consisted of four application-oriented platforms - pro, gov, lab, and life - that focus on
information technology, telecommunications, electronics and online media. As well as being an exhibition,
CeBIT is the world’s most important and comprehensive congress for the digital industries, featuring lectures,
seminars and workshops. This year, more than 100 top speakers from the ICT industry appeared on three
separate stages as part of a program spanning over 75 hours.
PROMISE presented three events at CeBIT which were jointly organized by CELCT (Center for the
Evaluation of Language and Communication Technologies) 4 , the University of Padua 5 , the University ofGothenburg6, the Zurich University of Applied Sciences7, and ELDA (Evaluations and Language Resources
Distribution Agency)8.

1.1 PROMISE @ CeBIT 2013
PROMISE featured three events at CeBIT:(i) Tuesday, March 5, 2013 from 02.00 p.m. until 02.45 p.m.: a 45 minute slot in the main program of the
CeBIT Global Conference (CGC) - Power Stage. The title of this session, organized as a panel discussion,
was: Is your Search Engine Making you Miss Business Opportunities? How Lessons learned in Research
about Information Access Evaluation can Help Industry, focused on the challenges in search, the ability ofsearch engines to find relevant information and to make it accessible to customers and business partners.

(ii) Wednesday, March 6, 2013: a whole day event in the middle of CeBIT exhibition grounds at theConvention Center targeting three different kinds of stakeholders in separate sessions during the day: decision
makers of consumer companies, decision makers of producer companies, and technology people. The aim wasto put researchers, software developers and users together in order to facilitate two-way communication ofresearch advances and results by researchers, as well as the reverse flow of requirements and ideas from users
and developers. The final goal was to stimulate the discussion on Information Access technologies and provide
a forum in which invited speakers can share their experiences, identify common strategies and needs, anddetect future challenges in this field.
(iii) For the entire period of the exhibition (5-9 March) we had a booth in Hall 9, Stand D25 with the title “EU
Language & Big Data Projects”. The stand was shared with other European projects including Khresmoi9.

2

Panel Discussion in the CeBIT Global Conferences

The web has given us a global information economy in many forms and many languages. Business successdepends more and more on the ability to find relevant information and make it accessible to customers andbusiness partners. Finding what is most relevant for a user requires search and access tools for multilingual
multimedia, and those tools need to be aware of the many ways in which human beings want to read, use and2

http://www.promise-noe.eu/events/technology-transfer-day
http://www.cebit.de/home4
http://www.celct.it/
5
http://www.unipd.it/en/6
http://www.gu.se/7
http://www.zhaw.ch/en.html
8
http://www.elda.org/
9
http://www.khresmoi.eu/
3

ACM SIGIR Forum

54

Vol. 47 No. 1 June 2013

enjoy information. This is a great market opportunity for new technologies: the future tools are being
researched, developed, and built in labs today.
The session was held in the form of a panel involving Stephen E. Arnold (ArnoldIT.com, USA), Nicola Ferro(University of Padua, Italy), Martin Braschler (Zurich University of Applied Sciences, Switzerland), Jussi
Karlgren (Gavagai, Sweden). They talked about the challenges of search, namely the problem of working withads supported software, the analytics and the use of big data, the pressure from cost control, social media, andthe need to index videos.
The following issues were also addressed: how do you know whether a search engine is good enough for yourbusiness? With search filling a communication and integration role, how do you know whether the search isreally delivering the right content?
During the discussion it emerged that new technologies coming from Labs need to be properly benchmarked
and validated to make sure they not only achieve technological excellence but also meet known requirementsfrom real world use cases. With search becoming a commodity, we need to understand its effects on an
information access process as a whole, to develop measures beyond precision and recall, and to look beyond
evaluating systems and components.
The panelists also introduced methodologies for the systematic evaluation of search engines, and examined
future trends and requirements for the next generation of information access systems.
This event was published on the CeBIT website 10 , in the printed program, social media channels (e.g.
Facebook, Twitter, LinkedIn) as well as in mailshots, newsletters and other marketing publications. The slot
achieved a good participation.

3

Whole Day Workshop

Twelve high-quality, interesting and inspiring talks from speakers coming from different European countries
and also from Australia, were given. The selected speakers were key representatives both from industry andacademia who were asked to share ideas, innovations and insights with their peers in an informal, interactive
setting. The event turned out to be an opportunity for brainstorming and discussion for the speakers, who could
bring their own perspective in the field about some relevant issues such as: multimedia and multilingual
information access, domain specific, e.g. pharmacovigilance, features and issues for information access; useroriented and usability-oriented evaluation of information access systems, lab-style evaluation of information
access systems, how to evaluate running enterprise applications from the outside, and so on.
The first session of the day “Getting out the most from your application” was focused on hearing about theusers’ experience.
Opening the day was Emma Bayne who gave an insight into some of the challenges involved in dealing withsearching the interface to the 20 million records in The National Archives (UK), and the role that taxonomies,
personas and user-centred design play in that process.
This was followed by Célia Boyer, who focused on the main problems in using the Internet for health, andshowed how the Health On the Net Foundation is working on providing certifications for health websites. She
also presented the user requirements even though at the moment there is no real business model for quality
health information.
Problems and challenges were the theme of Ian Kegel’s presentation which gave us an insight on how BT, theBritish multinational telecommunications services company headquartered in London, is trying to use thecustomer behavior to improve the TV experience. It is often difficult for service providers to justify a strategic
approach to collecting customer behavior data, with the result that tactical solutions are hard to retro-fit.

10

http://www.cebit.de/en/about-the-trade-show/programme/cebit-global-conferences/program

ACM SIGIR Forum

55

Vol. 47 No. 1 June 2013

The session was completed by Martin Braschler who described a methodology to perform an applicationcentric evaluation of operational information access systems. The application is treated as a black-box and themethodology can evaluate aspects of functionality which typical users can actually access and experience.
After the break different strategies to make information searchable and findable were presented in the session“Pushing your application to real needs”. Henrik Strindberg, Director of Research at Findwise, talked about
Search as a Service. They create search-driven solutions for intranets, web, e-commerce and applications andmake sure these are implemented to best support the goals and strategies of each organisation. Findability is allabout meeting the various information access needs and maximising business value from search technology
investments.
This theme was continued by Tony Russell-Rose, UXLABS, who was suggesting that there are four
dimensions of search experience: user, task, context, and mode. He also pointed out the differences betweensite search and enterprise search, i.e. they have common modes, but different patterns.
David Hawking, Funnelback, argued that efficient access to information can increase productivity and qualityof work in most organisations, and can reduce frustration in both working and private lives. Access to
information can be impeded by many factors, e.g.: poorly designed or inaccessible human-computer interfaces,lack of assistance in formulating appropriate requests, poor matching of requests to information objects, poor
ranking of objects retrieved, and poor presentation of the results of searches. An enterprise search company
such as Funnelback is constantly striving to remove or reduce these impediments across a broad range of
information access scenarios.
Gregory Grefenstette, Chief Technology Officer at Exalead, in his talk Multimedia Applications and Applied
Semantics, described a platform to index and process multimedia objects.
Martin Braschler presented the best practice recommendations for information retrieval system developers,
application implementers and application maintainers, covering the main aspects of IR systems andapplications, as well as recommendations for the user interface and evaluation.
Christoph Goller presented a demo of IntraFind technologies for full-text search, (optionally) enhanced withmodules for linguistic, similarity, semantic, thesaurus and taxonomy-based search, entity recognition, text
clustering, text classification, and knowledge maps. The products for information access to structured andunstructured data are based on the open-source products Lucene and Solr.
Philipp Daumke, provided an account of Averbis solutions, thanks to which information can be extracted fromtext collections and databases. Averbis provides software solutions for semantic text analysis, tailor-made text
searches and intelligent knowledge management. They possess particular know-how in the health sector, thepharmaceutical industry, libraries and in publishing.
Closing the session and also the day was Nicola Ferro, who tackled a topic at the very heart of search:evaluation. The need for an evaluation infrastructure is increasing in the community worldwide. He explained
the motivations for an evaluation infrastructure and illustrated its architecture and outcomes.

4Stand "EU Language & Big Data Projects"

The booth was located in Hall 9, Stand D25 in the CeBIT lab platform. Hall 9 was devoted to Research & New
Technologies. Here, decision-makers from scientific institutions, industrial companies, universities, start-up
and spin-off enterprises – as well as product and business developers from all sectors – come to present theirvisionary concepts, research findings and prototypes.
A large monitor on the back wall of the stand showed a loop of slides and videos either about PROMISE andthe other projects sharing the stand. Four round tables at the front were used to show live project demos.
A great amount of dissemination material was prepared and distributed to visitors, i.e.: the PROMISE leaflet,
PROMISE posters, pens, folders, bags, two PROMISE deliverables, the PROMISE Retreat report, USB key,
notebooks, and T-shirts.
ACM SIGIR Forum

56

Vol. 47 No. 1 June 2013

5Conclusions and Reflections

By all accounts, the event on the whole was a success, with quite a good number of people visiting the boothand providing positive feedback to the best practices and the black-box evaluation.
The panel and the workshop were really beneficial with great high-quality presentations from skilled speakers
and inspiring debates from across the floor.
In conclusion, it could be said that the event was successful on three counts. First, it delivered a lot of content
in a short period of time on issues related to Information Access and was a useful lens for examining thecurrent situation providing the points of view of both academia and industry.
The speakers on both days did an admirable job in pointing out the ever changing search technology
challenges, the methodology to evaluate them in an attempt to detect future trends and requirements for the
next generation of information access systems.
Second, to appreciate is the ambition of the event: the fact of having the Technology Transfer Day at the
CeBIT was something of an experiment; this is evidently a different formula to the more established searchindustry conferences, and a world apart from research-driven events.
Third, the event did a great job of turning a group of different people (researchers, developers, and users) into
a cohort.
However, participation was not as high as we had expected, despite the massive advertising that had been
performed. Dissemination was done using media outlets as the primary venue, such as mailing lists and the
CeBIT LinkedIn group. Also most of the CeBIT exhibitors involved with Information Access were informed
via email about the event. With hindsight, also Facebook, Twitter and YouTube could have been exploited tofurther advertise the event.
To be noted that the target groups we addressed were quite different from the one we are used to, in fact there
was no impressive critical mass from academia, and most people attending CeBIT were more attracted by andinterested in the “product” itself rather than in a European project about evaluation. It might be that it was not
the right type of event to attract sufficient industrial attendees.
Moreover, some elements of the programme probably needed a little further thought: for example having aworkshop lasting an entire day, might have discouraged people from attending the event, in fact most people
from industry already had an agenda full of meetings and could not commit for the entire day. Considering
this, we should have possibly booked a time slot in their agendas beforehand.
Slide presentations are uploaded on the event website11, so those who did not manage to attend can have a gistof what was discussed.

6

References[1] Agosti, M., Berendsen, R., Bogers, T., Braschler, M., Buitelaar, P., Choukri, K., Di Nunzio, G. M., Ferro,N., Forner, P., Hanbury, A., Friberg Heppin, K., Hansen, P., Järvelin, A., Larsen, B., Lupu, M., Masiero, I.,
Müller, H., Peruzzo, S., Petras, V., Piroi, F., de Rijke, M., Santucci, G., Silvello, G., and Toms, E. (2012).
PROMISE Retreat Report - Prospects and Opportunities for Information Access. SIGIR Forum, 46(2):60-84.
[2] Agosti, M., Berendsen, R., Bogers, T., Braschler, M., Buitelaar, P., Choukri, K., Di Nunzio, G. M., Ferro,N., Forner, P., Hanbury, A., Friberg Heppin, K., Hansen, P., Järvelin, A., Larsen, B., Lupu, M., Masiero, I.,
Müller, H., Peruzzo, S., Petras, V., Piroi, F., de Rijke, M., Santucci, G., Silvello, G., and Toms, E. (2012).
PROMISE Retreat Report: Prospects and Opportunities for Information Access Evaluation. PROMISE
network of excellence, ISBN 978-88-6321-039-2, http://www.promise-noe.eu/promise-retreat-report-2012/11

http://www.promise-noe.eu/events/technology-transfer-day/programme

ACM SIGIR Forum

57

Vol. 47 No. 1 June 2013

[3] Braschler, M., Rietberger, S., Imhof, M., Järvelin, A., Hansen, P., Lupu, M., Gäde, M., Berendsen, R.,
Seco de Herrera, A.G. Best Practices Report. PROMISE network of excellence, http://www.promisenoe.eu/documents/10156/086010bb-0d3f-46ef-946f-f0bbeef305e8[4] Rietberger, S., Imhof, M., Braschler, M., Berendsen, R., Järvelin, A., Hansen, P., Seco de Herrera, A.G.,
Tsikrika, T., Lupu, M., Petras, V., Gäde, M., Kleineberg, M., Choukri, K. Tutorial on Evaluation in the Wild.
PROMISE network of excellence, http://www.promise-noe.eu/documents/10156/3f546a0b-be7c-48df-b228924cc5e185cb

ACM SIGIR Forum

58

Vol. 47 No. 1 June 2013