How the Dutch prepared for certification
Barbara Sierman

Kees Waterman

KB National Library of the Netherlands
P.O. Box 90407
2509 LK, The Hague, The Netherlands
Barbara.Sierman@kb.nl

DANS
P.O. Box 93067
2509 AB, The Hague, The Netherlands
Kees.Waterman@dans.knaw.nl

ABSTRACTAs part of the national strategy for Cultural Heritage in the
Netherlands, under the umbrella of the Network Cultural Heritage(Dutch acronym: NDE), a working group consisting ofrepresentative digital preservationists from different large (mainly
cultural heritage) organizations worked to contribute to thecertification process of their organizations. This paper describes thevarious activities of the working group that resulted in a wellbalanced approach for certification based on (1) a phased approach,(2) a supporting tool for maturity level checking (3) translations of
the DSA and DIN/ nestor guidelines into Dutch, (4) lessons learnt
based on a survey amongst DSA certificate holders, (5) training
materials in Dutch.

KEYWORDS
Certification, DSA, nestor, trust, trusted digital repositories,
cultural heritage

1 INTRODUCTION
The growing maturity of digital preservation of the past 20 years isreflected in the need to be able to check whether organizations are
the trustworthy custodians of their digital collections. Several tools
were developed, from self-assessment tools and risk analysis tools
like Drambora [1] and the SPOT model [2], to official certification
tools like the Data Seal of Approval (DSA), the DIN 31644 / nestor
and the ISO standard 16363-2012. There is a clear need forbenchmarking and checklists, not only at libraries and archives as
the frontrunners in digital preservation, but also from fundingorganizations in relation to the growing amount of research datarepositories. These standards are used not only for self-assessmentand official audits but also for example in European projects like eARK to check the maturity of their participants in implementing
the e-ARK products [3]. Still, not everyone agrees that international
standards like the ISO 16363 are internationally applicable.[4] In
the Netherlands many organisations are taking steps in digitalpreservation. In order to raise the maturity level of theseorganizations a programme was started to popularize the concept
of audit and certification on a national scale.

2
2.1

ACTIVITIES OF THE NDECERTIFICATION WORKING GROUP
The certification working group

Funding by the Ministry of Education, Culture and Sciencefacilitated the founding of the Network Digital Heritage (Dutch
acronym: NDE), in which all the major Dutch heritageorganizations with digital collections work together in developinga system of common facilities and services for improving thevisibility, usability and sustainability of our digital heritage.[5] A
special NDE working group was started for Audit and Certificationof digital repositories (NDE-AUDIT wg), to continue the work
already started under the Dutch Coalition for Digital Preservation(Dutch acronym: NCDD). This working group ran a nationwide
campaign to raise awareness around auditing, created a roadmap
for audit and certification of repositories and above all propagatedthe benefits for organizations.
The participating organizations in the NDE-AUDIT working
group, consisting of six large cultural heritage organizations with avariety of digital collections had a different “maturity level” indigital preservation. Some already had acquired the DSA andstarted preparations for the nestor seal, others were still discussing
whether they were ready for certification. Apart from these content
holders one participant represented the Cultural Heritage
Inspectorate of the Ministry of Education, Culture and Science.
Originally it was planned to start at the same moment with the audit
and certification processes and to keep each other informed of thevarious steps taken. But because of the different levels and different
time schedules this did not work out. However, the variety inuptake also offered an excellent opportunity for developing abalanced view by combining the insights of the experienced and theless experienced.
To reach a wider audience, knowledge exchange about audit andcertification was not restricted to archives, (university) libraries andcultural heritage organizations. We deliberately also contactedsome suppliers of digital archiving software in use by these very
organizations. Although it was acknowledged that these suppliers
could not become certified themselves, as the certification is only
given to the collections holders in combination with the systems
they use to achieve this, several of them were very interested inacting together with their customers, either in consultancy or inbeing part of the certification effort.
The NDE-AUDIT working group planned a range of activities thatwill be described in more detail in the following paragraphs. These
will contain:






2.2

A phased approach of certification
Translations of two standards into Dutch
Further development of maturity checking software: theScoremodel
Survey to collect experiences with DSA v2
Training and communication

A phased approach of certification

Although the intention to become a certified organization is often
described as a goal in institutional policies [6], if the internal
challenges are serious and no external pressure for certification
exists, the intention might not be realized. A certification process
will require resources. The question “Am I ready for a certification
process” becomes important, as it is not wise to invest in a time andeffort consuming process if you are uncertain whether you canfulfill the basic standards.
At least in Europe the starting point for certification is the 3 level
European Framework [7] in which an organization starts with
applying for DSA, followed by a nestor seal and finally applying
for an official ISO audit based on ISO 16363.
We decided to use this framework but to add two important steps:
the Initial Self-Assessment and the Exploratory Phase.
In the past some organizations have started straight away with ISO
16363, amongst them LOCKSS. Apart from the benefits (moreabout this later) not seldom the feedback is that this certification
process was a hard one, and did cost a lot of time that could havebeen spent on other activities. In the NDE-AUDIT working group’s
opinion, adopting the European Framework model and to go for amore phased approach would be more suitable in our organizations,where not seldom digital preservation was only part of the activities
and often an activity in competition with the activities related tophysical collections. From bottom to top we distinguished thefollowing 5 phases:
Phase 1. Initial Self-Assessment
Phase 2. Exploratory Phase
Phase 3. DSA
Phase 4. DIN
Phase 5. ISO
In the following paragraphs these phases will be described with the
expected outcome of the phase.
Phase 1. Initial Self-Assessment
There are several basic tools for an organization to check its
maturity level and to get an initial idea of how mature theorganization really is with respect to digital preservation. The
NDE-AUDIT working group saw this step as an essential one. In

many organizations digital preservation is still a niche activity andby undertaking an Initial Self-Assessment we expect that theorganization will get a more reality based view on theirpreservation activities.
An extensive overview of tools can be found in one of thedeliverables of the European e-ARK project.[8] One of the tools
available for the Dutch audience is the Scoremodel, developed by
the Dutch organization DEN and the Flemish organization
PACKED, described more in detail in paragraph 2.4.
The outcome of this Initial Self-Assessment depends on the tools
used, but in case of using the Scoremodel a report is printed inwhich the strong and the weak areas are shown in a “spider web
score”. The next step for the organization can be to improve theidentified weak areas before the organization will be ready for eventhe basic level of certification (phase 3).
Phase 2. Exploratory phase
As the audit and certification process will take time and resources,
it is important to start a process in close harmony with the
management of an organization. We introduced a step in theprocess in which enough detailed information is collected that canbe used to support a management decision of “go/no go”. Although
it might not be possible to create exact figures about the timeneeded, at least what need to be identified is:Which departments will be involved. These are not only
the departments that are directly involved in thepreservation process, but also for example the legal
department, Human Resources department, Finance&Control department etc.
Which persons and in which role in these departments
need to be available during the audit and certification
process?

The availability of documentation of the preservation
activities is the main requirement for all levels ofcertification, as this documentation is necessary evidence
to prove that an organization is compliant. An estimatedguess need to be made how much documentation isalready available and in which areas one might expect
extra effort (and resources) is needed (gap analysis).
How well is the certification method understood in theorganization? Can one “translate” the DSA requirements
to the situation in their own organization? Are all
requirements clear? The NDE-AUDIT working group
did an investigation in which they took the DSA versionof 2016 and compared the terminology used in DSA withthe terminology used in their own organization (library,archive, AV organization). This offered an interesting
overview of different jargon used in various domains andstrengthen the case for discussing this before starting theaudit and certification process. It is very important thatan organization is aware of time needed to discuss theinterpretation of the requirements, as this is strongly
related to the expected effort needed.
Which part of the digital collection will be chosen as a candidate
for the audit and certification process? It is important that theboundaries of what will be part and what not are clear to everyone
involved.
The outcome of this exercise will be a management document witha summary of the findings on the basis of which the management
can make a decision whether they want to proceed or not. One ofthe participating organizations made use of this step whenorganizational changes and change in priorities became a threat for
the audit and certification exercise: based on this document it
became clear, even before the certification process started, that theresources needed would not be available.
Phase 3. Applying for the Data Seal of Approval (DSA)
When applying for the Data Seal of Approval, the following steps
can be distinguished [9]:Applying via the DSA form
Internal preparations
Self evaluation via the DSA Tool plus confirmation of
the received information from the DSA AdministrationPeer reviewer appointed by the DSA Board starts thereview, guided by the DSA tool. The peer review process
is supervised by the DSA Board

Feedback to the DSA applicant, either in asking for more
information or awarding the DSA Seal
Publication of the evidence given on the public DSA
website (except for confidential information) including
the feedback of the peer reviewer in order to have atransparent process.
The outcome of this process is a Data Seal of Approval that theorganization will post on the website related to the collection or
repository that was subject of the audit. The Data Seal of Approval
is valid for 3 years.
Phase 4. Applying for nestor Seal (DIN 31664) or selfassessment according to the ISO 16363
One of the participating organizations in the NDE-AUDIT working
group, DANS already took steps to apply for the nestor Seal, passed
the review and was the first organization to receive this Seal. When
applying for the nestor Seal, the following steps can be
distinguished:Contact nestor and acquire documentation and templatesInternal preparations and completing necessary
documents that will support prove of compliance with thenestor criteria
Add the self-assessments results to the 34 nestor criteria

Review of this self-assessment by 2 reviewers, appointed
by nestor.
Feedback of the results from the reviewers

Publication of the aforementioned documentation and theself-assessment on the website of the organization

Receiving the nestor seal including the year it wasawarded.
The nestor seal is valid indefinitely.
Phase 5. On site audit based on ISO 16363

As no one of the participating partners in the working group wasapplying for this audit method, we did not investigate the steps.
Currently (may 2017) there are no European certification bodies toperform a ISO 16363 audit, in the US ANAB (standards
organization) will take care of certifying auditors in the US.[10]2.3 Translations of two standards into Dutch
Coincidentally, another initiative was started by Regionaal
Historisch Centrum Limburg and funded by Archief2020 and
NCDD to translate the nestor DIN 31664 standard into Dutch.
Archief 2020 was a 4-year collaboration (from 2012-2016) between
archives with the goal to innovate the Dutch archives on all levels
with a focus on digital challenges. Motivation behind thetranslation of the nestor standard was the need of a set of
instruments to qualify an (archival) e-Depot as trustworthy and tohave an overview of the main ingredients for a trustworthy
designed e-Depot. Apart from that, although most Dutch people
understand the English language, a Dutch translation would lower
the barrier.
Members of the NDE-AUDIT working group helped tocontextualize the first draft translation and supported the translator
in domain specific jargon. Along this line it was soon decided totranslate the text of the Data Seal of Approval as well.
Since November 2016 a new version of the Data Seal of Approval
was published, in which several requirements were changed toreflect the merge of the original DSA standard with the ICSUWorld Data System requirements under the heading of “Unified
Requirements for Core [originally “basic”] Certification ofTrustworthy Data Repositories”.
Although the total set of requirements (16, formally called
Guidelines) stayed the same, some major changes were introduced.
There are new requirements on “Security”, “Confidentiality/ethics”
and “Expert guidance”. These headings already show the stillpresent “scientific data center” view of the DSA. Although thechanges are not many, some are significant. For example DSA
version 2 originally mentioned explicitly OAIS in guideline 13,
where it stated “The technical infrastructure explicitly supports thetasks and functions described in internationally accepted archival
standards like OAIS”. The new Requirements only mentioned
OAIS for repositories “with a preservation remit”.[11] These
differences between the versions of the DSA standards were one of
the arguments to introduce an investigation of the understandabilityof the standard in the Exploratory Phase. For example:“Confidentiality/ethics” might be less obvious for national libraries
as it might be for archives and social science data centers.
The translation exercise itself required some tough decisions. Firstit was agreed not to translate the well-known OAIS terminology
into Dutch, as we assumed the people that would use the standards,
should also be aware of the common digital preservation
terminology. In practice however, several requirements in DSA(and less in the DIN/nestor standard) asked for some contextual
information to make it understandable for different domains. For
example in the DSA standard the phrase “publication repository” isused but it is not clear whether here a “data publication repository”
or a repository with articles is meant. In several cases referencesare made to “citations”, which refers to a concept from the scientificdomain and is less important when certifying national libraries orarchives. So some requirements will need additional information to
explain the context. The more so because related to eachrequirement is a set of documentation that the organization need toprepare for the audit. It is not always clear from the text what kindof documentation is required and as yet no repository is certified
against this new standard, there is no evidence from otherorganizations that acquired the DSA according to these new
Requirements. We need to wait how this will work out in practice,
but it might have helped if an example list had accompanied thestandard (like is done in ISO 16363 with a Self-assessment
template).[12]Both translations of the standards into Dutch are published [13].
Especially the DSA one is now in use in two organizations in the
Netherlands that are preparing for this certificate, the National
Archive and the International Institute of Social History.

2.4 The maturity checking: the Scoremodel
The Scoremodel [14] is an online tool in the Dutch language, inwhich questions related to 7 areas of digital preservation need to beanswered, varying from mission and policy to ingest, knowledge
and organization, access, planning and quality control and storage
maintenance. Each question has an extensive explanation
describing how it is related to digital preservation. The outcome ofthis exercise is a report, including a spider diagram which gives aclear overview of the main strong and weak areas related to the 7
topics. In the report for each question the answer is (automatically)
analyzed, describing the risk related to the answer, the context andsuggested actions. advise.
In the new version of the Scoremodel (to be published in 2017) the
results are linked to DSA criteria. In this way users can estimate thereadiness for starting the DSA certification process. The
Scoremodel is a very helpful tool to get a first impression of thematurity of the organization with respect to digital preservation, as
well as giving the organization practical advise. It will take 2 to 4
hours to add the relevant information in the tool, of course
depending on available documentation.

2.5 Training and communication
The working group prepared two workshops in which a large groupof Dutch and Flemish organisations (50 in total) discussed their
preparations for audit and certification according to DSA. Several
smaller Dutch organisations were already preparing themselves for
DSA certification, some did this in collaboration with their
supplier. Apart from that the NDE program started with a virtual
learning environment for digital preservation in Dutch with a
special chapter on audit and certification. The main training topics
are related to having the right arguments to persuade higher
management, requirements for documentation, estimating how
much time will be involved in the certification process etc.

2.6

Survey to collect experience with DSA

One of the major activities of the NDE-AUDIT working group was
designing a survey in order to gather practical information fromcolleagues who already applied for the DSA Seal. The following
chapter will elaborate on this.

3

SURVEY FOR DSA CERTIFIEDORGANIZATIONSIn order to have more sound background information for the
Exploratory Phase in which management need to be convinced of
the usefulness of the audit and certification exercise and to get abetter estimation of the investments and benefits of certification,
we wanted to get information from organizations that already had
applied for the Data Seal of Approval (pre-2016 version). The
working group set up a survey to be held amongst theseorganizations, under the title “DSA-Experiences: help your peers!”
The outcome of the survey could offer us a comprehensive
overview of the experience with the certification process,especially as we specifically has some questions related to the first
application of the seal. This information we thought important to
inform our management about the certification process and to
implement the lessons learnt into training and advice for the Dutchorganizations’ planning to start the process.

3.1 Outline of the survey and the results
The DSA Guidelines were first published in 2008, and had an
updated version in 2010 and one in 2014-2015. As of March 2016
some 50 repositories had obtained the seal. Some organizations
have published their findings in meeting with DSA-requirements.
[14] Yet, no comprehensive overview is available on theexperiences regarding the process and results of certification of all
DSA-certified repositories. In an effort to collect such experiences
the NDE Audit working group requested permission from the DSA
Board to field a survey among all digital repositories that had
achieved DSA-certification by March 2016.
Despite the change from Guidelines to Requirements, the basic
tenets of core certification efforts will remain unchanged, however,and the NDE-AUDIT working group is convinced that theoutcomes of this survey can be of considerable significance toorganizations considering or actually preparing for DSAcertification – also after the introduction of the DSA’s revised
requirements in September 2016.
The survey was sent out to the email addresses of 50 DSA
repositories, or organizations that operated such repositories. TheDSA Secretariat was confident about the correct delivery to 47
recipients. Of these addressees 18 filled out and sent in the survey.
The total results and scores per questions can be found in theofficial report.[16] In this paper we summarize these findings,
without repeating the original answers. For convenience sake, wecategorized the answers and identified the following areas:







Repository characteristics [Q1-3]
Certifications achieved and planned [Q4-10]
Trigger for certification [Q11]
Certification efforts [Q12-13, 19-27]
DSA comprehensibility [Q14-18]
Benefits of DSA-certification [Q28-33]

Repository characteristics
Motivation to ask these questions: The DSA audit and certification
method was initially focused on research data repositories, but theDutch candidates for DSA certification are a more varied set oforganizations, consisting of libraries, archives, digital art
collections, data centers, an AV centre etc. To rightly interpret theanswers in the survey, an indication of the kind of digital archive
would be important, as would be the size of the organizations. All
18 repositories answered this question. The majority of the
respondents describe the type of their repository as domain or
subject-based, or as institutional, which is representative for thecurrent DSA community; they employ between 1 and 12 fte’s, ofwhich 0.2 to 10 fte’s work primarily on preservation tasks. It isnoteworthy that the average number of fte’s that work primarily on
preservation stands at 1; available human resources at therepositories are clearly modest in scope, although four repositories[out of 18] report that they employ more than 8 fte’s. Apparently,
also repositories with limited human resources have beensuccessful in applying for the seal.
Certifications achieved and planned
Motivation to ask these questions: The NDE-AUDIT working
group was very much interested in the “certification history” of the
DSA repositories. When did they do their first application for
DSA? Did they really renew the certification every 3 years? Were
they following the “European Framework model” and planning forthe next level? And above all, it was essential that all respondents
indicated they would report on their experiences during their first
application for the seal.
The respondents were asked which version of the DSA therepositories first obtained (2010 version or 2014-2015, between thetwo versions are small differences).
Out of 18 respondents, the majority of 14 obtained the more recent
version, while 4 of them obtained the 2010 version.
The responses are an indication of the larger uptake of thiscertification instrument in the DSA domains in recent years.
But how many had renewed their seal after having obtained theirfirst DSA-certification? All 18 repositories responded, of whichfour that had initially obtained the 2010 seal indicated they had
indeed renewed their seal; all others had not yet done so. Given thetime frame, there was no necessity to do this. Apart from that there
is no pressure from DSA to renew, it is expected that thecommunity will draw their own conclusions when an organization
has a seal on its website that is very outdated. A further question inwhich the repositories were asked for their intention to renew their
DSA-certificate showed that all (17 yes) but one stated theirintention to do so. We could say that, at least among the
respondents, the need for continuous maintenance and renewal of
the seal is an accepted practice.
But were they also willing to apply for the extended certification
by means of DIN/nestor certification? All 18 respondents answered
and most of them (16) had not applied at this level and were not inprocess of doing so, one of them was investigating DIN/nestor.

Asked whether they were contemplating to apply for the highest
level: the external audit on the basis of ISO 16363 no one of the 18
respondents indicated that they had applied or that they were in theprocess of doing so.
Trigger for certification
Motivation to ask this set of questions related to the trigger for
certification lied in the fact that the NDE-AUDIT working groupconsisted of a variety of organizations all with different motivations
to get certified. Building trust and showing that an independent
organization had recognized them as a trustworthy organization
was for some of them important for the suppliers of data to theirdata repository. In general it was felt that the “big hubs” as they
were called, meaning the main content holders in the Netherlands,
should get certified to establish trust in general and to be an
example for smaller organizations in the Netherlands. Apart fromthat there was an expectation that in the long run the fundingorganizations like the Ministry and research funding organizations
would require a certification.
From the wide array of the DSA survey respondents’ input, somemain considerations can be distilled:
One-third of the answers (7 out of 16) indicate that the repositories
were motivated by an existing, inherent recognition of theimportance of continuous professionalization and quality assurance
in their digital preservation remit. Five answers derive from therepositories’ recognition of the value of the DSA in showcasing
their value as a trusted digital repository to stakeholders. Anothergroup of five answers indicate that the repository was alreadyinvolved in the DSA’s development or acted on an invitation by the
DSA leadership. Four additional answers indicate that therepository’s interest was triggered by an internal directive(management) or an external obligation (e.g., condition forpartnering in a research infrastructure, funding).
To summarize: while the responses are varied, it is evident that therepositories were mostly triggered by a recognition that DSAcertification is a natural and appropriate instrument in (showcasing)their ongoing professionalization as trustworthy partners for longterm digital preservation.
Certification efforts
Motivation to ask these set of questions was the fact that one of thegoals of the survey was to deliver a rough overview of the effort
and time investment that would be useful to plan the Exploratory
Phase and the DSA Certification Process, based on real life
experience of first time applicants.
One remark should be made here: based on the answers to theexplicit question whether they had decided to keep track of the time
investment, all answered that they did not do so before applying forthe certification, and only a few (4 out of 16) had decided to do thisduring the certification process. One could assume that part of theanswers below are based on time sheets, some on other sources andsome on "memory".
The survey explicitly asked about the estimated time investment ingetting a first impression of the DSA as a certification instrument.

This is what we would call the Exploratory Phase. The majority(65%) reported an estimated time investment of 10-20 hours; 3chose the category 0-10 hours and 2 indicated 20-40 hours. A single
respondent indicated a larger amount of time, 60 hours or more.
The actual time investments in the first step of the DSA procedure:
the internal preparations, took more hours. The majority (60%)
reported a time investment of 50-100 or 100-200 hours; 4 chose thecategory 0-50, while 2 indicated larger investments (1 of 200-300
and 1 of 500 or more).
Followed by the steps of submission and the actual peer review(questions for clarification etc.) 17 repositories responded; 1skipped this question. The majority (78%) reported a time
investment of up to 50 or 50-100 hours; 2 chose the category 10020, while 2 estimated larger investments (1 of 200-300 and 1 of 500
or more). In a separate question the survey asked the respondents
for their biggest challenge in dealing with the peer reviewer’scomments. This question was primarily intended to identify(potentially) problematic aspects of the respondents’ interaction
with the peer reviewers. But most answers (8 out of 13) indicatedthat the respondents had in fact not encountered “challenges” or“problems.”
But did the time investments comply with or perhaps exceeded their
expectations? Responses (16 out of 18) were distributed quite
evenly through the full range of potential answers. If we leave out
the three respondents who reported that they had no preconceived
expectations on this issue, the largest subgroup (6) indicate they
had underestimated the required investments, a smaller subgroup(4) had correctly estimated this aspect and the smallest group (3)had overestimated the required investments.
To summarize: for developing a first impression of the DSA, most
respondents estimated a time investment of 1-20 hours; forpreparations for the DSA procedure, most respondents estimated atime investment of 50-100 or 100-200 hours; for the actual
certification process, most respondents estimated a time investmentof up to 50 or 50-100 hours. None of the respondents had decided
beforehand to keep a record of time investments, but four out
sixteen decided to do so at a later stage. Some repositories had no
preconceived idea of the required investments; the remaining ones
varied widely in their evaluation of expected time investments vs.
actual investments. The largest group of these (6 out of 13)
indicated they had underestimated the required investments. We
expected that a potential source for higher time investments than
expected might be the interactions with the peer reviewers, but
most respondents indicated that they had not experienced problems
in this regard.
DSA comprehensibility
Motivation to ask these questions was the fact that at the time of
the DSA Survey, the new version of the DSA-WDS was not yet
officially approved by the DSA Board. But the NDE working groupmade a comparison between the DSA Guidelines 2014-2015 andthe new version as published on the RDA website, as thedevelopment of the latest version was an activity of the RDA
Repository Audit and Certification DSA–WDS Partnership
Working Group.[17]The latest DSA version took a similar approach as ISO 16363 inchoosing the main topics, which are now Organizational
Infrastructure, Digital Object Management and Technology.
Although the total amount of the original guidelines (as it wascalled in version 2014-2015) are the same amount as the current
Requirements, 3 new Requirements appeared, covering“organizational infrastructure, “expert guidance” and “security”. It
is not possible to make a one to one comparison as there is nobackground information about the motivation behind the changes,
but the influence of the already existing WDS Criterions is visible.
Based on this comparison it became clear that some terminology
used was rather domain specific and not always compatible in otherdomains. This was the reason to explicitly ask whether DSA wascomprehensible to the variety of organizations: how clear andstraightforward were the DSA guidelines?
The majority of the respondents (65%; 17 out of 18 responded)
rated these aspects of the Guidelines as “adequate-excellent.” A
considerably smaller subgroup (4) rated these as “adequate,” andthe smallest group of respondents (2) opted for “poor-adequate.”
No respondent chose “poor” or “excellent.”
When asked to identify DSA Guidelines they found most difficult
to comply with, respondents listed every guideline in separate
comments – but most guidelines appeared in those comments only
once. The main reason was that they could “not yet” comply witha given guideline because no preservation plan, no written
workflow or no policy for acceptance of file formats was in placeat that moment. Apparently they improved the required
documentation during the certification process.
Benefits of DSA certification
The participants in the NDE-AUDIT working group made an
earlier attempt to convince Dutch organizations of the benefits ofgetting certified, and published a leaflet [18], inspired by thebenefits mentioned on the DSA website, although they are nolonger mentioned on the DSA website :- stakeholder confidence- improvements in communication- improvement in processes- transparency- differentiation from others- awareness raising about digital preservation.
But how did the DSA certified repositories experience this? 17
respondents out of 18 responded. Clearly the benefit of enhanced“transparency” was rated with the highest significance (8 times as“essential” and 8 times as “considerable”). An internal benefit
received the highest number of ratings as “considerable” (12):“awareness raising about digital preservation.” Two other benefits
that were highly rated are of a more external nature: “stakeholder
confidence” received 9 ratings as “considerable,” and sevenrespondents rated “differentiation from others” as a “considerable”
benefit.
It is noteworthy that one respondent rated the beneficial effects on“stakeholder confidence” as negligible. In a more detailed question
we asked for some more details about the impact of the DSA
certification on various aspects of their own organization andrepository. Four aspects scored high on “considerable” in 16 out of
18 respondents: “management’s recognition of the value of longterm preservation and sustained availability of digital assets”,“digital preservation policies” and “technical digital preservation
practices” and “the capacity to attract data producers”. Also, theimpact on the organization’s professional reputation was rated as“considerable” and “essential.” These ratings confirm the areas
where DSA-certification intends to have its strongest impact: by
imposing structured, professional and community-driven
expectations (guidelines or requirements) on the applicants’policies and work processes, DSA-certification guides andencourages organizations to describe, document, improve andmonitor their essential preservation tasks.
Reportedly, the impact was much smaller in the area of “financial
planning” and “allocation of financial resources.” Still, the impact
on “allocation of staff” was predominantly rated as “satisfactory.”
Given these benefits, would the repository also recommend thistype of certification to others within their domain?
The vast majority of the respondents (17 responded, 1 skipped thequestion: 88%) answered affirmatively; they were either 'certainly'(9) or 'very likely' (6) willing to do so.
When queried about their willingness to recommend DSAcertification to their peers, the vast majority of the respondents
answered affirmatively, and when asked why they would be willingto do so, most comments characterized the DSA as an instrument
to buttress quality assurance efforts. Another strand of motivations
related to the quality of the seal as a sign of professional maturity
to be showcased to stakeholders and colleagues. The majority of
the respondents rated the ratio between investments and benefits as“adequate-rewarding” to “rewarding-excellent”. The final question
was related to “lessons learned” or other relevant experiences topass on to future DSA-applicants. To this question the respondents’
input showed a high level of community sentiment. Most provided
a balanced review of their overall experience, placing the required
levels of investment on at least an equal footing with the perceived
and reported benefits. Key words in respondents’ final remarks
hovered around the need for thorough documentation andpreparation,
significant
contributions
toprofessional
accomplishment and a tangible boost to quality assurance in the
respondents’ work.

3.9

Lessons learnt for the NDE Working GroupThe Survey offered enough information to propagate this basic
certification to most of the organizations if they reserved some timeto get acquainted with DSA. In our opinion the Exploratory Phase
is a good description of this.
The outcomes of the DSA survey gave a significant insight in the
DSA certification process. On all aspects we were investigating, theanswers either confirmed our view – like the internal benefit ofgetting more management attention for digital preservation – aswell it gave us more insight in the required effort. There were also
some lessons to take with us, as for example to register the time
investment. We only need to wait now for some experience oforganizations applying for the DSA 2016 version as there is noexperience with that yet. One of the requirements that worried thepartners in the NDE-AUDIT working group most was the level ofrequired documentation, that seemed to be more than in the DSA
2014-2015 version.
Based on the information of the DSA survey and an educated guessof using the Scoremodel, the following time table was created:

Expected effort according to Dutch model1
Initial Self2 hrs adding
Assessment
information in model
based on(educated guess) and
Scoremodel
X hours internal
discussion (up toorganization)
2
Exploratory
1-20 hrs based on
Phaseoutcome survey
3
DSA
internal
50-200 hrs based on
preparations
outcome survey
DSA
submission
50-100 hrs based on
activity andoutcome survey
peer review
discussions

4

CONCLUSIONSOver the course of 3 years a group of Dutch preservationists
collaborating in the NCDD and the NDE worked on popularizing
audit and certification in an attempt to professionalize digitalpreservation in the Netherlands. Several large organizations
prepared themselves for certification, resulting in 2 partners
receiving the DSA pre-2016 Seal, one partner renewed this seal, 1
partner acquired the DIN/nestor seal as the first in the world en
currently 2 partners are preparing for the 2017 DSA version. Apartfrom these partners in the NDE-AUDIT group several smallerorganizations and their suppliers in the Netherlands are preparing
themselves, supported by the practical support of the outcomes of
the NDE-AUDIT group and meeting each other in regular training
sessions organized by the same working group, all traveling along
the originally designed Roadmap for Audit and Certification.

ACKNOWLEDGMENTS
A special thanks goes to the partners in the NDE-AUDIT Working
group: Margriet van Gorsel (National Archive), Annemieke deJong (Institute for Sound and Vision), Valentijn Gillissen (DANS),
Robert Gillisse (DEN), Ronald van der Steen (Cultural Heritage
Inspectorate), Madeleine de Smaele (4TU), Jacob Takema (KB)
and Marcel Ras (NCDD).
This work was partially supported by the Ministry of Education,
Culture and Science under the National Strategy for Cultural
Heritage and with investments of the participating organizations inthis working group: KB, DANS, NA, B&G, 4TU, DEN, IISG,NCDD.

REFERENCES
1

Digital Repository Audit Method Based on Risk Assessment,
http://www.repositoryaudit.eu/
2 Vermaaten, S, Lavoie B., Caplan, P. Identifying Threats to Successful Digital
Preservation: the SPOT Model for Risk Assessment. D-Lib Magazine, 2012,vol 18/19, http://www.dlib.org/dlib/september12/vermaaten/09vermaaten.html
3 Proença, D. Veira R. Borbinha, J.: A Maturity Model for Info Repository
characteristics for Information Governance – Final Version. Deliverable D 7.5 of thee-ARK project. http://www.eark-project.com/resources/project-deliverables/95-d75-1/file
4 Seles, A. 'The Transferability of Trusted Digital Repository Standards to an East
African Context'. Doctoral thesis, UCL (University College London).
http://discovery.ucl.ac.uk/id/eprint/1473881
5 Sierman, B. & Ras, M.: Best until… A national infrastructure for Digital Preservation in
the Netherlands, Proceedings of the 12th International Conference on Digital
Preservation, Chapel Hill, 2015,
https://phaidra.univie.ac.at/detail_object/o:429602
6 Sierman, B.: Maturity and preservation policies. Proceedings of the 11th International
Conference on Digital Preservation, Melbourne, 2014, p 259-266
https://phaidra.univie.ac.at/view/o:378066
7 Memorandum of understanding to create a European Framework for Audit andCertification of Digital Repositories,
http://www.trusteddigitalrepository.eu/Memorandum of Understanding.html
8 Proença, D. Veira R. Borbinha, J: Initial Assessment and Evaluation. Deliverable
D7.2 of the e-ARK project http://www.eark-project.com/resçources/project-deliverables/46d72initassess/file
9 Dillo, I. & De Leeuw, L., Het Data Seal of Approval: keurmerk voor duurzame en
betrouwbare databewaarplaatsen. Handboek Informatiewetenschap. De Jong, A.S.M.,
van Trier, G.M., Sieverts, E. & Koren M. (eds.). Alphen aan de Rijn, 2014.
10 http://www.iso16363.org/anab-opens-its-iso-16363-accreditation-program/
11 Guidance for requirement 9, see Core Trustworthy Data Repositories Requirements,
https://drive.google.com/file/d/0B4qnUFYMgSc-eDRSTE53bDUwd28/view
12 http://www.iso16363.org/preparing-for-an-audit/
13 For the Dutch translation of the DIN/nestor standard, see http://www.ncdd.nl/wpcontent/uploads/2016/02/Toelichting_nestor_keurmerk_NL.pdf (2016); for the Dutch
translation of the former DSA Requirements, now the Core Trustworthy Data Repository
Requirements, see http://www.ncdd.nl/wpcontent/uploads/2017/03/20170225_Basiseisen_voor_betrouwbare_archieven_DEF.pdf(2016).
14 https://www.scoremodel.org/nl/home
15 See the analysis of the Archaeology Data Service (UK) from 2011:
http://www.dcc.ac.uk/resources/case-studies/ads-dsa;
and the more recent report by the Finnish Social Science Data Archive from 2015:
http://www.datasealofapproval.org/en/assessment/fsd-dsa-case-study
16 Waterman, K. & Sierman, B.: Survey of DSA-certified digital repositories: Report on thefindings in a survey of all DSA-certified digital repositories on investments in andbenefits of acquiring the Data Seal of Approval (DSA). The Hague: Netherlands
Coalition for Digital Preservation, 2016. http://www.ncdd.nl/wpcontent/uploads/2016/10/201611_DE_Houdbaar_Report_DSA-survey_2016.pdf
17 https://www.rd-alliance.org/groups/repository-audit-and-certification-dsa%E2%80%93wdspartnership-wg.html
18 Doe ik het goed? Certificering van digitale repositories in Nederland. Den Haag, NCDD,
2014 http://ncdd.nl/site/wp-content/uploads/2014/06/NCDD-certific-folder-DIG-V3DEF.pdf