Creating gold standards and supervising outcomes – the future role of library staff in supporting information services based on Machine Learning
Creators: name: Borst, Timo
Publisher: name: Zenodo
Year: 2021
Subjects: Machine Learning
Training data
Library services

description: As in many other areas of industry and technology, in the information sector we can observe an uptake of experiments, services and products that are increasingly driven by software (development) and data commonly labelled as Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL), among them text classification and machine translation, query understanding and reformulation, information retrieval and (re-)ranking of search results, metadata generation by means of named entity recognition, image classification and object detection, as conducted e.g. in Optical Character Recognition, information assistants, such as chatbots or robots with a human interface. While particularly academic and research libraries already make use of these approaches at least on an experimental level in project settings, in my opinion it is sometimes hard to estimate internal efforts and skills that are not documented well enough, even though they might take a significant part. In particular, it is three challenges and corresponding skills from (data) librarians, that we conceive as a regular basis for the development of many, if not any AI and ML related information service: First, to produce or provide sufficient and reliable training data to optimize ML algorithms including their models and classifiers. Secondly, to test these algorithms by applying them to unlabelled data. Finally, to evaluate them in their interactive environment, as it is defined by surrounding information systems or general requirements, in particular information ethics.<br> To sketch the impact of these workflows more into detail, I refer to three exemplary current in-house projects at ZBW Leibniz Information Centre for Economics: <em>Automatic indexing of scientific full texts.</em> In this project, we first have to produce training resp. test data (the latter to be split into an extra corpus) as manually generated subject headings. The calculated results or “candidates” are evaluated by means of a supervisory tool, while information ethics forces the algorithm to distribute its suggestions or predictions across different sub-categories from a thesaurus to arrive at a more “balanced” indexing. <em>A chatbot for serving user requests.</em> In this project, we first have to classify training resp. test data, which is about conversational chatlogs. While chatbots are also about generating language in terms of answers, evaluation is done by means of Q&amp;A sessions to test a chatbot’s performance, but also to check automatic classification in particular of those conversations, which are ambiguous. <em>Automatic metadata generation by identifying the funders of a scientific work resp. publication.</em> Here, the task of manually identifying those entities comprises the detection and exclusion of ‚false positives‘, e.g. papers that mention other or no funders at all. While we benchmark the generated results against Boolean fulltext searches in terms of precision and recall, we conduct the final evaluation on the metadata within a faceted search application. As far as we can observe from our projects, the design and implementation of information services based on ML comprise three basic activities with respect to generating, curating and evaluating data that we might regard as constitutive for those kind of services. Hence, we can view it as a basic skillset that we still have to project into each individual information service.