DATA QUALIFICATION METHODOLOGYIN THE LITERATURE SURVEY STAGE
Saori Yamada1, Takanori Kunimaru1, Kunio Ota1, Stratis Vomvoris2, Niels Giroud2
1

Nuclear Waste Management Organization of Japan (NUMO)
National Cooperative for the Disposal of Radioactive Waste (Nagra)2

1

4-1-23 Shiba, Minato-ku, 108-0014 Tokyo, Japan
Hardstrasse 73, CH-5430 Wettingen, Switzerland

2

Key words; Quality Management System, site investigation

Abstract
The site selection for a deep geological repository in Japan will be undertaken by means of a three-stage
process. The initial literature survey (LS) aims at selecting areas for the next preliminary investigation (PI) only
on the basis of data from the existing literature. The results of geological characterisation in the PI stage lead tothe final detailed investigation.
Key to the LS stage and the evaluation process is the development of a qualified assessment basis for eacharea. The challenge at this stage is that the vast amount of data in the assessment basis originates from a wide
spectrum of activities by third parties – field investigations, laboratory tests and underground constructions –none of which has been performed under NUMO’s Quality Management System.
NUMO has been executing a project to develop a strategy on how to evaluate, qualify and include data andinformation collected by the third parties in the assessment basis. The basic policy for data qualification was
defined and a qualification methodology was derived and applied to the data.

1. Introduction
The site selection for a deep geological repositoryof high-level radioactive waste in Japan will be
undertaken by means of a three-stage process. The
three-stage process consist of literature survey (LS),preliminary investigation (PI) and detailed
investigation (DI) (Figure 1). The initial LS aims atselecting areas for the next PI only on the basis ofdata from the existing literature. The results ofgeological characterisation in the PI stage lead to thefinal DI.
The Nuclear Waste Management Organization ofJapan (NUMO) initiated the siting process in 2002
with open solicitation of volunteer host municipalities
for the identification of a suitable repository site inthree-stage processes as specified in the Act on Final
Disposal of Specified Radioactive Waste. A logicaland progressive basis for the siting process has beendeveloped, which involves explicit exclusion
criteria1). With an amendment to the Basic Policy(22nd May, 2015), the Government committed tomaking a fresh start towards the initiation of the site
selection process and is taking the following steps:

Identifying geoscientific criteria and on this basis
areas where a geological disposal facility (GDF)
could potentially be located;

Undertaking information activities so that eachmunicipality can understand whether its area issuitable for inviting a survey of the geologicalenvironment for a GDF.

Through this process, it is expected thatmunicipalities in the potential areas will invite
NUMO to carry out feasibility studies to verify, forexample, that the geological stability of the potential
repository sites is not affected by volcanic activity,
active faults or other geological phenomena, as
defined in the exclusion criteria.
Preliminary
Investigation
Areas (PIAs)

Detailed
Investigation
Areas (DIAs)

Repository
Site (RS)Municipalities(Government’s Proposal)

Municipalities(Open Solicitation)

Literature
Survey (LS)Preliminary
Investigation (PI)

Detailed
Investigation (DI)- Geophysical survey- Borehole investigations etc- Excavation of test tunnel- Investigation in test tunnel

Selection criteria

Selectionof PIAs

Selection criteria

Selectionof DIAs

Selection criteria

Selectionof RS

Figure 1 Site selection process in Japan

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan2. Quality assurance in the LS stage
The first stage in the site selection process is the
LS stage, during which NUMO will collect available
information from the literature, analyse it andcompile it in such a way that the different areas canbe evaluated against the site selection criteria definedby the expert groups of the government. The result of
this evaluation will be the selection of a number ofareas for further investigations to be conducted under
the auspices of NUMO in the PI stage.
Key to the LS stage and the evaluation process isthe development of a qualified assessment basis foreach area. The assessment basis includes a wide
range of empirical data from field investigations in
the areas, laboratory tests on, for example, cores from
boreholes in these areas, or undergroundconstructions and studies. The data from theassessment basis will support the development of thesite descriptive model (SDM) (eg conceptual modelof hydrogeological structure, such as main types ofwater-conducting features, domains with different
average hydraulic properties etc) to be used in theevaluations and the derivation of the input parametervalues for any analytical or numerical models used.
For all these data used, it must thus be shown thatthey are qualified and are appropriate for theirintended use (fit for purpose). The challenge at thisstage is that the vast amount of data in the assessment
basis originates from a wide spectrum of activities by
third parties – field investigations, laboratory tests
and underground constructions – none of which has
been performed under NUMO’s Quality Management
System 2).
3. Strategy and General approach for data
qualification
As mentioned in chapter 2, the data qualification
methodology is intended for data used as input for thedevelopment of the SDM; this includes neither data
related to geological phenomena nor the geologicalmap. The basic policy for data collection andqualification consists of the following five principles:







All data found during the LS stage will be
documented;
Before using such data for the purpose ofevaluating sites for the next phase, each data set
will have to be qualified;
The evaluation will be graded (level of QA);
Data not considered further will be kept in the
database and the reason for not using them will be
described;
A stepwise qualification process will be followed.

As the first step, NUMO will collect all available
information without undertaking an evaluation of its
quality, in order to have as a complete a database as
possible. Each information or dataset entering

NUMO’s database will be characterised as ‘notqualified’ until the qualification process has been
undertaken. It is recognised that, whilst a certain
dataset may not be qualified in its entirety, it may stillcontain information that qualifies it for a specificapplication. A graded qualification to enable use of
the maximum information contained in a given
dataset is chosen.
Records that are not qualified will be designated assuch and the reasons for their disqualification will be
documented for future reference. In order to optimise
the resources allocated for the qualification, a
stepwise process was proposed3) (Figure 2).

Figure 2 Basic qualification process for literature data
The basic qualification process consists of threesteps, summarised below.
Step 1 Acceptability check
The basic acceptability check aims at assessing
whether there is sufficient information about the dataand it is in an acceptable form to allow initiation of
the technical qualification process. The checklist forsuch a test is to a large extent independent of the typeof data to be evaluated and consists of:Description of the data and creation of metadata
information: The description includes informationsuch as author of data, time and date of creation,
original purpose or project for which the data
were collected (later referred to as ‘provenance’);Form of data: For example, paper copy, analogue
data and digital data.

If Step 1 is successful, ie sufficient informationand in an acceptable form is available, then one canproceed with the next steps. If not, the data andinformation should be kept but identified as not
qualified and the reasons for the data rejection should
be documented. Note that the level of information
may vary among different records and it will be
useful during this step to also indicate the level of
information (in broad terms – low, acceptable, high).

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan

Step 2 Qualification of raw data
The qualification of raw data, if the raw data are
available and accessible, is discipline-specific but, ingeneral, includes information on:






Equipment and methods used;
Protocols and QA followed;
Conditions during the laboratory tests or field
investigations;
Graphic tests;
Consistency checks.

Raw data of this type include, for example,
laboratory tests for the determination of petrophysicalor hydrochemical properties. The qualification levelis related to the reliability, or degree of certainty, ofthe resulting values.
Step 3 Qualification of interpretation
In the majority of the cases a dataset from theliterature will not include raw data, but rather
information derived from these data. Examples of thistype of data are hydraulic conductivity ortransmissivity values derived from borehole
investigations, or geological sections interpreted fromgeophysical campaigns. The checks to assess thequalification are discipline-specific. These include,for example:






Assessment of the methods used for thegeneration and in case of no raw data, the method
used for the interpretation of the field data;
Plausibility tests;
Graphical tests (eg diagnostic plots using different
normalised parameter sets);
Internal consistency checks.

If access to raw data is possible, even if thequalification level of the existing interpretation is low,
the possibility exists to perform a re-analysis. Such a
decision will be affected by the potential role of thedata for a given application, for example,
development of an SDM in an area with scarce data,
the chances of a successful re-interpretation as wellas the amount of resources and time that would beneeded.
In terms of qualification grading, four qualificationlevels are considered as below:

Level A: Qualified - this is the highest level andwould imply that there is a high degree of
reliability to the dataset;
Level B: Qualified but can be used with some
restrictions – for example, a geologicalcross-section interpretation;
Level C: Qualified but can only be used in alimited way.

Not qualified or Level D: Assigned by default toany dataset being collected before it undergoes
evaluation. If the evaluation is negative, it would
remain at this level and be tagged with a short
description of the reasons.
General methods for evaluation of data reliability
Four alternative methods or a combination thereof
can be used to assess the quality of the literature data
and determine whether more specific technicalevaluation is required. These are shown in Figure 3
and can be thought together with thediscipline-specific processes as part of a tool-box forthe evaluation4). The result of their application can be
graded into different levels bounded by the two
end-members – fully qualified and not qualified.
With the exception of the qualification at the lowest
level, which would lead to discarding a particular
dataset, the qualification levels can contribute to thedecision as to whether additional analysis orinterpretation(discipline-specific)
shouldbeperformed.

Figure 3 Qualification tool-box
The provenance category groups information thatis directly related to the data, ie how the data havebeen collected, QA programme followed, and QC
applied for the generation of the data, the reliability –experience, track record, recognition – of thepersonnel or organisation responsible for the data, or,finally, if the data are from a refereed publication.
The other three categories correspond to: (a)further actions that have been performed to or withthe data, such as confirmatory testing, independent
analysis or a formal peer review, or (b) existence ofadditional information from other qualified sourcesthat would provide confidence in the data quality.
Note that confirmatory testing, independent analysis
or a formal peer review can also be performed as aresult of the qualification level determined for adataset, if it is expected that the qualification level
through one of these actions could increase.
4. Application to the in-situ hydraulic conductivity
datasetAn example of application of the general methods
shown above is described in this section. It focuses

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan

Figure 4 Main steps for the qualification of in-situ
hydraulic test data
Level of information
Each record in the NUMO’s database is describedwith a large number of attributes. Among these
attributes, the following are considered for theacceptability step:Formation and rock type (referred to in thefollowing as ‘geological information’);
Location and depth of the test interval for thehydraulic test (referred to in the following as‘geographical information’);Source of data (referred to in the following as‘reference’).

Geological information
In order to use hydraulic test data as input for the
SDM, NUMO needs to assess whether the data arerepresentative of the rock types used in the model.
Table 1 shows the matrix of the attributes considered.
The information given on the rock formation is rated
in three levels. The lowest quality rating is assigned
to records with no information on the rock type (egattribute field left blank or designated as unknown);
the highest quality rating is given to those records
where the information provided is unequivocal and asdetailed as possible. The level of information on therock type attribute is given four quality levels (Table1). The lowest level comprises all records for which
this attribute is described as unknown or left blank,
whereas the best level those with a complete
description of the rock type.

Formation

Geological information

Rock type

Work flow
The workflow presented in this section focuses on
the qualification of in-situ hydraulic test data and will
use some of the attributes of the corresponding
database. The main three steps are shown in Figure 4
and consist of: the assessment of the level of
information (acceptability level); the assessment of
the reliability of the value; and, the assignment of aqualification level.

Table 1 Rating of the level of information for thegeological information
Complete

Incomplete/Partial

Blank

Complete432Partial(eg argillaceous rock)

3

2

1

Rudimentary(eg sedimentary)

2

2

0

Blank

2

1

0

Geographical information
Four attributes are relevant for the location of aborehole in the dataset: latitude, longitude,
municipality and prefecture. The highest level of
information is the exact coordinates (latitude andlongitude) and it decreases as the number of attributes
left blank increases (Table 2).
Depth information is provided by four attributes:
Upper, lower, and middle depth of the testing interval,
and interval length. The dataset is split between
records with no depth information, records withindication of the middle of the interval, and records
with all four attributes provided. These three
categories of data will receive respectively the lowest,
intermediate and highest grade for the level of depth
information (Table 2).
Table 2 Rating of the level of information for thegeographical information
Depth
Geographical information

Locationon the acceptability check for one of thehydrogeological properties (hydraulic conductivity)
collected from existing literature.

Known depth andinterval length

Only middle ofinterval known

Blank

Lat./Long.
Address
Borehole #

4

3

1

Municipality

3

2

0

Prefecture

2

2

0

Blank

0

0

0

Reference
The third component for the qualification of thelevel of information is the source or reference for arecord and is used to discriminate data according to
the reliability of the original reference. The highest
information level is assigned in general topeer-reviewed publications, as well as to data
extracted from databases that are considered asreliable (Table 3). The lowest level is assigned to data
which have been found in a non-peer-reviewed
document and stem from a reference which is notavailable.
Table 3 Rating of the level of information for thereference
Reference
Peer-reviewed journal
Peer-reviewed report
Qualified database4

Quality certified data producer(eg ISO or JIS)

3

Conference abstract2

Non peer-reviewed and referring to an
unavailable source

1

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan

The information level ratings defined above for thedifferent categories of geological and location
information are combined in rating matrices which
provide on a scale from 0 (lowest) to 4 (highest) thelevel of information for these two groups ofinformation (Tables 1 and 2). The sum of the matrix
outputs for geological and geographical information
feeds into Table 4 which determines, depending on
the quality of the data source, the rating of the levelof information.

two attributes. All data that have not beendisqualified in the workflow for the level of
information start the workflow at the top. The firststep is to determine if the test method and/orinterpretation attribute are provided in the dataset, forhydraulic conductivity.
Data that were not disqualified
due to the level of information
Test method

Reliability level

Procedure canbe ended here

AR

Table 4 Rating of the level of information

Sourcetrustworthiness

7

6

5

4

3

2

1

AAL

BAL

BAL

BAL

CAL

CAL

DAL

DAL

3

AAL

BAL

BAL

CAL

CAL

CAL

DAL

DAL

2

BAL

BAL

BAL

CAL

CAL

DAL

DAL

DAL

1

CAL

CAL

CAL

CAL

DAL

DAL

DAL

DAL

0

Reliability of the value
The reliability of the value is evaluated mainly on
the test method and the interpretation method used.
Note that a record can include information on both
attributes, only on one and in some cases on neither.
The workflow discussed below considers all these
possibilities. A four-level qualification is also used
for the two attributes, as discussed in this section.
A two-step procedure was followed to estimate thereliability level of a documented value. The first step
aimed at grouping the methods registered in the
database to a much smaller number of types ofmethods keeping in mind the ultimate use of the data
values, namely the derivation of parameters for SDM.
Thus, the categories used were chosen to represent, tothe degree possible, the standard type of field tests
and interpretation methods used in sitecharacterisation in the context of SDM development.
The second step was to assign the rating of hydraulictesting method between 4 (highest) and 1 (lowest) toeach of the categories. A similar approach, iegrouping the methods, in categories, was also
followed for the interpretation methods.
The sum of the ratings of the test method and of
the interpretation is used to define the reliability level(Table 5) in four levels between AR (highest) and DR(lowest).
Table 5 Rating of the reliability of the value
Sum of rating of test and interpretation method
AR

Sourcetrustworthiness

7
AR

6
BR

5
BR

4
CR

3
CR

Reliability level

Procedure canbe ended here

AR

2
DR

Figure 5 shows the complete workflow that allows
determining the reliability level of the hydraulic test
result. The workflow has two distinct branches,
depending on whether there exists information on the

Reliability level

BR, CR, DR

BR, CR, DR

Procedure canbe ended here

Corroborating data

If raw data are available,
optional:
Perform a new interpretation

Reliability level
AR

BR, CR, DR

Figure 5 Workflow for the reliability of the value
If either or both are provided, a reliability level canbe assigned, as described in Table 5. The procedurecan be stopped at this stage with a preliminary
reliability level. This would be useful for a firstscreening of the data, selecting only the data points
with a reliable testing method. It is, however,possible to continue the procedure to potentially
increase the level of reliability for tests that wererated BR, CR or DR at the first step. The second test
considers the trustworthiness of the source or the
producer of the data. Finally, if raw data are available,
a new interpretation can potentially be performed.
For data where no method is specified, the only
resource is to consider the data provenance and thevalue itself (workflow to the right on Figure 5). The
quality of the data provenance is the same as thatused for a known method, but the maximum levelobtained with this method will be BR.
Qualification of a record
Both the level of information and the reliability ofthe value ratings are combined to provide the final
qualification level of the data. The qualificationlevels, on a scale from A (highest) to D (lowest), areshown in Table 6, and their relative definitions inTable 7.
Table 6 Qualification level based on the rating of thelevel of information and the reliability of thevalue
Qualification
level
Reliability ofthe value84

8BR, CR, DR

Sum of geological and geographical information

Disqualified

Reference

Level ofinformation

Methodunknown

Interpretation

Level of information
AAL

BAL

CAL

DAL

AR

A

A

B

B

BR

A

B

B

C

CR

B

B

C

C

DR

B

C

C

D

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan

Table 7 Description of the qualification levels
A

Qualified data
Qualified data, high-degree of reliability

B

Use with limitations
Qualified data, with some degree of uncertainty or
limitation in the level of information

C

Restricted use
High uncertainty in the data value or thematadata, for specific purposes onlyD

Disqualified
Data cannot be used in the current stage

This process is being tested on the actual database
and the results are promising in terms of correct
record screening and ease of application.

4.

descriptive models – Strategy and methodology
for the qualification of data collected from theliterature. Proc. ANS 16th International
High-Level Radioactive Waste Management
Conference, Charlotte, NC, USA, 9-13 Apr.
2017, pp.853-858, ANS.
Altman, W.D., Donnelly, J.P. and Kennedy, J.E.
(1988): Qualification of Existing Data for
High-Level Nuclear Waste Repositories.
Generic Technical Position, NUREG-1298.
USA, NRC.

5. Summary
In the initial LS stage, NUMO will collectavailable information from the literature and compile
it in such a way that the different areas can beevaluated against the site selection criteria. The
challenge at this stage is that the vast amount of datain the assessment basis originates from a wide
spectrum of activities by third parties none of which
has been performed under NUMO’s Quality
Management System. NUMO has been executing a
project to develop a strategy on how to evaluate andqualify such data in the assessment basis. The basic
policy for data qualification was defined and aqualification methodology was derived and applied tothe in-situ hydraulic conductivity dataset. In this case,the assessment of the level of information considers
the availability of information on the geology, thelocation and depth of the test and the reliability of thedata source. The assessment of the reliability of thevalue focuses on the value reported and its reliability.
The final step combines both level of information andreliability of the value to obtain the qualification of
the record in the assessment basis. A full applicationof these methods to existing and updated databases is
expected to take place in the near future with theinitiation of the LS stage.
References
1. NUMO (2004): Evaluating Site Stability for a
HLW Repository. NUMO-TR-04-04. Tokyo,
Japan.
2. Ota, K., Kunimaru, T., Nishio, H., Tanaka, T.,
Abumi, K., Horio, A., Arikawa, M., Yoshimura,
M., Suzuki, Y., Shiraishi, H. and Ishibashi, K.
(2014): Establishment of quality management
system for geological disposal project: (1)Development of quality management system forPreliminary Investigation stage. Abs. Atomic
Energy Society of Japan 2014 Fall Meeting,
Kyoto, Japan, 8-10 Sep. 2014, p.239, AESJ. (inJapanese)
3. Kunimaru, T., Ota, K., Giroud, N., Vomvoris, S.
and Yamada, S. (2017): Development of site

6th East Asia Forum on Radwaste Management Conference
November 27-29, 2017, Osaka, Japan