Ali Shiri

University of Alberta, Edmonton, AB, Canada

Emily Villanueva

University of Alberta, Edmonton, AB, Canada

# Methodological Diversity in the Evaluation of Cultural Heritage Digital Libraries and Archives: Approaches, Frameworks, and Methods

**Abstract:**

Digital library evaluation has become increasingly important in information science, yet there has been minimal evaluative work focusing on digital cultural heritage. This article reports on a comprehensive review of methodologies, frameworks and techniques used in the evaluation of cultural heritage digital libraries and archives. Empirical studies are examined using Saracevic's digital library evaluation framework to identify models, frameworks, and methodologies in the literature and to categorize these past evaluative approaches. Through the classification of evaluative types and trends, we aim to develop a set of guidelines and recommendations for the future evaluation of cultural heritage digital libraries and archives.

**Keywords:** Cultural heritage, Digital libraries, Evaluation, Methodologies

## 1. Introduction

The heterogeneous materials and multicultural user groups represented in the cultural heritage field pose a unique challenge for both the design and evaluation of information systems (Petras, Stiller, & Gäde, 2013). Evaluation of cultural heritage digital libraries typically falls under the system-centric or user-centric perspective, and these varying approaches bring to light the features and challenges associated with different evaluative techniques (Petras, Stiller, & Gäde, 2013). While not specific to the cultural heritage field, Saracevic (2004) identified seven approaches to digital library evaluation, all addressing different components or goals: systems-centred, human-centred, usability-centred, anthropological, sociological, economic, and ethnographic. Saracevic (2000) is also credited for introducing the five main elements that frame digital library evaluations: construct, context, criteria, measures, and methodology. Other relevant evaluation frameworks have been Nicholson's (2004) holistic matrix model, which examines systems and use from an internal and external view, and Fuhr et al.'s (2001, 2007) DELOS evaluation framework, centred on three approaches: system evaluation, usefulness evaluation, and usability evaluation. Ultimately, past research has deemed interface usability, system performance, and collection value to be the most agreed upon evaluation criteria (Xie, 2008). While digital libraries have been an increasingly important area of study in library and information studies, the specificity and sensitivity required when dealing with cultural heritage in the digital space has often been overlooked.

In this paper, we discuss a recently completed comprehensive review on the evaluation of cultural heritage digital libraries and archives using Saracevic's (2000) evaluation framework,
which our analysis demonstrated to be one of the more widely used frameworks in evaluation research (Stiller, Gäde, & Petras, 2013). Saracevic's (2000) five-pronged evaluation framework was chosen because there were no widely accepted, identifiable frameworks created specifically for CHDLs. To complete this evaluation, we first located the existing literature on the evaluation of cultural heritage digital libraries from library and information studies databases and resources and compiled a bibliography of relevant literature. Second, we examined each article to identify the frameworks, approaches, methodologies and data gathering tools that had been used in previous evaluative studies. This information was tabulated and categorized in a large spreadsheet. Finally, we identified the specific elements, cultural or otherwise, that were being evaluated in these digital library studies following Saracevic's (2000) outline. In this paper, we also report a series of evaluative guidelines and recommendations for future research on cultural heritage digital libraries and archives.

## 2. Data Gathering and Analysis Methods

For this analysis, we gathered relevant studies through a systematic search of works related to the evaluation of cultural heritage digital libraries. Searches were conducted on the Directory of Open Access Journals (DOAJ), ACM Digital Library, Scopus, Library and Information Science Source, and the University of Alberta's general article search feature through EBSCO. Terms used in these searches included cultural heritage, digital libraries, digital archives, evaluation (user and system), methodologies, methods, approaches, and frameworks. A resulting list of 103 articles were collected before being reviewed for relevance. Relevance was determined through further examination of the content in the collected articles. Articles were removed if digital library evaluation was discussed, but no evaluation was conducted. Additionally, articles that described already existing evaluation frameworks but did not include a use case scenario of the framework were excluded. 59 articles remained in the study following this review process, indicating that these studies covered a specific CHDL evaluation study. Following this, each of the remaining articles was analyzed according to Saracevic's five evaluation elements (construct, context, criteria, measures, and methodology), with the relevant material extracted and placed in a spreadsheet. A sample dataset from the spreadsheet can be seen in Table 1. For this study, the Construct category resulted in a single group (cultural heritage institutions), as our focus was primarily on framework and criteria, not the specific institution involved in the evaluation. This conclusion was decided upon through our grounded theory approach, defined by Glaser and Strauss (1967) as 'the discovery of theory from data – systematically obtained and analysed in social research' (p.1). This qualitative method informed our analysis and interpretation of the collected CHDL data. While the Construct did not end up being central to this study, it is important to note that Europeana and CULTURA (CULTivating Understanding and Research through Adaptivity) were the most frequently evaluated projects, which can limit the large-scale applicability of the results of our work.

<table><caption>Table 1. Sample tabular presentation of cultural heritage digital library evaluative studies</caption><thead><tr><th>Title</th><th>Authors</th><th>Name</th><th>Framework / Approach</th><th>Evaluation</th><th>Method</th><th>Data type</th><th>Notes</th></tr></thead><tbody><tr><td>Methodological Considerations in Developing Cultural Heritage Digital Libraries: A Community-driven Framework</td><td>Shiri, A.</td><td>Digital Library North - Inuvialuit Settlement Region</td><td>Multi-disciplinary methodological framework, participatory</td><td>User-centric evaluation</td><td>Environmental scanning, open house, surveys, community leader engagement, information audit, community workshops, visual approach: photography</td><td>Qualitative, informal, surveys, photographs, participatory</td><td>N/A</td></tr><tr><td>An Evaluation Model for a Digital Library Services Tool</td><td>Dorward, J., Reinke, D., M.</td><td>SMETE Open Federation Digital Library</td><td>Instructional Architect (IA)</td><td>User-centred evaluation - Evaluation for IA involved both process and outcome components and was predicated on Program Theory Model.<br>The effectiveness of the DL in meeting the needs of its end-users must also be</td><td>The IA project was informed by a needs assessment (pilot survey of audience), expert review of software interface, prototype testing by members of the target audience, analysis of code changes by constituents, a case study of master teacher use of software, and a summative evaluation to assess overall product utility.</td><td>Qualitative pilot survey</td><td>N/A</td></tr><tr><td></td><td></td><td></td><td></td><td></td><td>This article looks at areas for further research and relevance to cross-cultural</td><td></td><td>This chapter has identified and discussed a set of critical</td></tr></tbody></table>
### 3. Results & Discussion

The frameworks and approaches to digital library evaluation that have been used in previous studies represent a combination of theoretical and technical frameworks found in the literature. Theoretical frameworks or approaches included multimodality, reflexivity, grounded theory, cross-cultural usability, and a multi-disciplinary methodological framework among others. Among the technical frameworks were the MUSETECH (Museum Technology) Model, MUSEF (Museum's Sites Evaluation Framework), the DML (Digital Music Lab) Framework, DELOS (Digital ERCIM Library Operational System) and NESTOR (NEsted SeTs for Object hieRarchies).

As noted by Petras, Stiller, and Gäde (2013), system-centric and user-centric evaluation are the primary evaluation models used in cultural heritage digital library and archive research, as well as the interface-based approach. Our findings echoed this sentiment, with a strong lean towards the user-centric approach. 53 of the identified articles related to cultural heritage digital library evaluation included at least one component of a user-centric approach. Among these were studies on user needs assessment (Marchionini, Plaisant, & Komlodi, 1998), usability evaluations (Jeng, 2008; Liew, 2005) and heuristic methods of evaluation (Skevakis, Makris, Kalokyri, Arapi, & Christodoulakis, 2014). A number of studies also combined the user-, interface-, and system-centric approach using various methods. The least common evaluation model was the system-centric approach, which was seen in 24 studies, either as the sole evaluation method or in tandem with another method. There was a strong focus on the efficiency of cultural heritage digital libraries as they pertained to the semantic web and linked data in the system-centric evaluations (de Boer et al., 2012; Freiere, Borbinha, & Calado, 2012; van Hooland et al., 2013; Wang et al., 2013).

The methods identified in our literature review were not as easily categorized as the evaluation models. There was a wide range of evaluation types and evaluation methods. While the majority of studies focused on qualitative data and techniques, there were quantitative components present in several studies. User evaluations included participants ranging from novice to expert, and data gathering techniques ranged from online surveys to in-person tasks. The questions asked and the tasks assigned differed based on the goals of the study and the target audience, which was usually students, researchers, stakeholders or the general public. System evaluations were more straightforward, and looked at metadata, linked data, and information retrieval. The combined evaluations tended to focus on the comparative aspect of evaluation, and the differences between manual and automatic evaluative practices.

Interestingly, what this study has demonstrated is that there is a lack of focus on cultural heritage material even in the evaluation of cultural heritage digital libraries and archives. The lack of diversity in the institutions being evaluated also demonstrated a bias towards established projects like CULTURA (http://www.cultura-strep.eu/home) and Europeana (https://www.europeana.eu/portal/en). While some studies examined system functionality and the potential of linked data for cultural heritage, most studies identified by our literature review focused on the user-experience, whether it was user-needs, usability of the interface, or search
functionality. Evaluation studies frequently relied on outside participation for their user-
evaluations, yet few studies identified in this project looked to a specific cultural community for
participatory engagement or feedback. Of the evaluative articles identified, few evaluated or
discussed any specific cultural elements present in the digital library. Although the evaluation of
cultural heritage digital libraries is becoming more common, there is still little attention given to
the unique needs and requirements of digital cultural heritage institutions. Some cultural aspects
that were discussed in the literature included the classification of Maori resources (Liew, 2005),
multilingual resources and consultation (Stiller, Gäde, & Petras, 2013; Sulé Duesa, Estivill Rius,
& Gascón García, 2011), the creation of culturally-appropriate metadata (Farnel et al., 2017), and
interoperability between cultural heritage institutions on the Canary Islands (Núñez & Ortiz
Repiso, 2019).

**4. Conclusion**

Our research has identified a number of patterns and trends in the evaluation of cultural
heritage digital libraries and archives, including the prioritization of qualitative methodologies.
While the review of methods, methodologies and frameworks of cultural heritage digital libraries
demonstrated a focus on user-centred evaluation and methods, it also highlighted a gap in
evaluative methodologies that overlooked the specific cultural elements, cultural context, and
community engagement that are so central to the development of cultural heritage digital
libraries. In this paper, we will present our guidelines and recommendations for future evaluation
of cultural heritage digital libraries, which ultimately aim for a greater focus on community
participation in the development and evaluation of heritage institutions, as well as greater
inclusion of cultural elements in digital library evaluation.

**Reference List:**

de Boer, Victor, Jan Wielemaker, Judith van Gent, Michiel Hildebrand, Antoine Isaac, Jacco van Ossenbruggen, and Guus Schreiber. 2012. "Supporting Linked Data Production for Cultural Heritage Institutes: The Amsterdam Museum Case Study." In *The Semantic Web: Research and Applications. ESWC 2012. Lecture Notes in Computer Science, Vol. 7295*, edited by E. Simperl, P. Cimiano, A. Polleres, O. Corcho, V. Presutti: 733-747. Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-30284-8_56

Farnel, Sharon, Ali Shiri, Sandra Campbell, Cathy Cockney, Dinesh Rathi, and Robyn Stobbs. 2017. "A Community-Driven Metadata Framework for Describing Cultural Resources: The Digital Library North Project." *Cataloging & Classification Quarterly* 55 (5): 289-306. doi:10.1080/01639374.2017.1312723

Freire, Nuno, Borbinha José, and Pável Calado. 2012. "An Approach for Named Entity Recognition in Poorly Structured Data." In *The Semantic Web: Research and Applications. ESWC 2012. Lecture Notes in Computer Science, Vol. 7295*, edited by E. Simperl, P. Cimiano, A. Polleres, O. Corcho, & V. Presutti: 718-732. Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-30284-8_55

Fuhr, Norbert, Preben Hansen, Michael Mabe, Andras Micsik, and Ingeborg Sølvberg. 2001. "Digital Libraries: A Generic Classification and Evaluation Scheme." In *Research and Advanced Technology for Digital Libraries. ECDL 2001. Lecture Notes in Computer Science*
*Science, Vol. 2163*, edited by P. Constantopoulos, I. T. Sølvberg: pp.187-199. Berlin, Heidelberg: Springer. doi:10.1007/3-540-44796-2_17

Fuhr, Norbert, Giannis Tsakonas, Trond Aalberg, Maristella Agosti, Preben Hansen, Sarantos Kapidakis, Claus-Peter Klas, László Kovács, Monica Landoni, András Micsik, Christos Papatheodorou, Carol Peters, and Ingeborg Sølvberg. 2007. "Evaluation of Digital Libraries." *International Journal of Digital Libraries*, 8: 21-38. doi:10.1007/s00799-007-0011-z

Glaser, Barney.G., and Anselm L. Strauss. 1967. *The Discovery of Grounded Theory: Strategies for Qualitative Research*. New York: Aldine de Gruyter.

Jeng, Judy. 2008. "Evaluation of the New Jersey Digital Highway." *Information Technology and Libraries* 27 (4): 17-24. doi:10.6017/ital.v27i4.3238

Liew, Chern L. 2005. "Cross-Cultural Design and Usability of a Digital Library Supporting Access to Maori Cultural Heritage Resources." In *Design and Usability of Digital Libraries: Case Studies in the Asia Pacific*, edited by Y. Theng, S. Foo: 285-297. Hershey, PA: IGI Global. doi:10.4018/978-1-59140-441-5.ch016

Marchionini, Gary, Catherine Plaisant, and Anita Komlodi. 1998. "Interfaces and Tools for the Library of Congress National Digital Library Program." *Information Processing & Management* 34 (5): 535-555. doi:10.1016/S0306-4573(98)00020-X

Nicholson, Scott. 2004. "A Conceptual Framework for the Holistic Measurement and Cumulative Evaluation of Library Services." *Journal of Documentation* 60 (2): 164-182. doi:10.1108/00220410410522043

Núñez, Maria E., and Virgina Ortiz Repiso. 2019. "Colecciones Digitales del Patrimonio Cultural de Canarias: Situación Actual y Propuesta de Actuación." *Revista General de Información y Documentación* 29 (1): 141-165. doi:10.5209/rgid.64552

Petras, Vivien, Juliane Stiller, and Maria Gäde. 2013. "Building for Success? Evaluating Digital Libraries in the Cultural Heritage Domain." In *Recent Developments in the Design, Construction, and Evaluation of Digital Libraries: Case Studies*, edited by C. Cool, K. B. Ng: 141-163. Hershey, PA: IGI Global. doi:10.4018/978-1-4666-2991-2.ch009

Saracevic, Tefko. 2000. "Digital Library Evaluation: Toward an Evolution of Concepts." *Library Trends* 49 (3): 350-369. Retrieved from
https://www.ideals.illinois.edu/bitstream/handle/2142/8343/librarytrendsv49i2i_opt.pdf

Saracevic, Tefko. 2004. "Evaluation of Digital Libraries: An Overview." Paper presented at the *DELOS Workshop on the Evaluation of Digital Libraries*. Retrieved from
http://dlib.ionio.gr/wp7/WS2004_Saracevic.pdf

Skevakis, Giannis, Konstantinos Makris, Varvara Kalokyri, Polyxeni Arapi, and Stavros Christodoulakis. 2014. "Metadata Management, Interoperability and Linked Data Publishing Support for Natural History Museums." *International Journal on Digital Libraries* 14 (3): 127-140. doi:10.1007/s00799-014-0114-2

Stiller, Juliane, Maria Gäde, and Vivien Petras. 2013. "Multilingual Access to Digital Libraries: The Europeana Use Case." *Information - Wissenschaft & Praxis* 64: 86-95. doi:10.1515/iwp-2013-0014.

Sulé Duesa, Andreu, Assumpció E. Rius, and Jesús Gascon García. 2011. "Evaluación de las Interfaces de Consulta de las Colecciones Digitales Patrimoniales Españolas." *Anales de Documentacion* 14 (2). Retrieved from
https://revistas.um.es/analesdoc/article/view/113931
van Hooland, Seth, Ruben Verborgh, Max De Wilde, Johannes Hercher, Erik Mannens, Rik Van de Walle. 2013. "Evaluating the Success of Vocabulary Reconciliation for Cultural Heritage Collections." *Journal of the Association for Information Science and Technology* 64: 464-479. doi:10.1002/asi.22763

Wang Shenghui, Antoine Isaac, Valentine Charles, Rob Koopman, Anthi Agoropoulou, and Titia van der Werf. 2013. "Hierarchical Structuring of Cultural Heritage Objects Within Large Aggregations." In *Research and Advanced Technology for Digital Libraries. TPDL 2013.* Lecture Notes in Computer Science, Vol. 8092, edited by T. Aalberg, C. Tapatheodorou, M. Dobreva, G. Tsakonas, C. J. Farrugia: 247-259. Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-40501-3_25

Xie, Hong I. 2008. "Users' Evaluation of Digital Libraries (DLs): Their Uses, Their Criteria, and Their Assessment." *Information Processing and Management* 44: 1346-1373. doi:10.1016/j.ipm.2007.10.003