Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English Edition
February 4, 2022
(a revision of the Japanese version published on July 5, 2021)

Technical Report DigiARC-TR-2022-02
Digital Architecture Research Center
National Institute of Advanced Industrial Science and Technology (AIST)

Technical Report CPSEC-TR-2022003
Cyber Physical Security Research Center
National Institute of Advanced Industrial Science and Technology (AIST)

Technical Report
Artificial Intelligence Research Center
National Institute of Advanced Industrial Science and Technology (AIST)© 2022 National Institute of Advanced Industrial Science and TechnologyiTechnical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Foreword
In the project "Research and Development on the Quality Assessment Reference and Testbedof Machine-Learning /artificial intelligence systems" (JPNP20006) commissioned by the New
Energy and Industrial Technology Development Organization (NEDO), we are developing
Machine Learning Quality Management Guidelines [1] to explain the quality of machine learning.
While developing the guidelines, we have also been researching and developing techniques forevaluating and improving the quality of machine learning. Although this research anddevelopment is still ongoing, since we have obtained technical knowledge on the quality
evaluation described in the Machine Learning Quality Management Guidelines, we report on the
progress of this research and development for the recent two years (FY 2019~2020).

ii

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Table of Contents
1

2

3

4

5

6

7

Introduction ............................................................................................................................................................. 1
1.1

Overview of this research and development .................................................................................... 1

1.2

Author list........................................................................................................................................................ 3

1.3

Acknowledgements ..................................................................................................................................... 3

Visualization of Machine Learning Models ................................................................................................. 4
2.1

Survey on methods to support using machine learning .............................................................. 4

2.2

Developing prototype of model difference visualization tool ................................................... 6

2.3

Future work .................................................................................................................................................... 7

Improving Application of Data Augmentation in Deep Learning ...................................................... 8
3.1

Purpose ............................................................................................................................................................ 8

3.2

Feature combination mixup（FC-mixup） ....................................................................................... 8

3.3

Applying data augmentations to feature maps (Latent DA) ................................................... 10

Debug-Testing of DNN Software ................................................................................................................... 12
4.1

Direct cause of failure.............................................................................................................................. 12

4.2Internal indices .......................................................................................................................................... 13

4.3

Experiments: method and results ...................................................................................................... 14

4.4

Related work ............................................................................................................................................... 17

4.5

Conclusion .................................................................................................................................................... 18

Evaluation and Improvement of Robustness .......................................................................................... 19
5.1

Robustness measure (maximum safe radius) ............................................................................... 19

5.2

A survey on methods for evaluation and improvement of robustness .............................. 20

5.3

Conclusion .................................................................................................................................................... 26

Estimation of Generalization Error Upper Bounds .............................................................................. 27
6.1

Generalization error upper bounds .................................................................................................. 27

6.2

Generalization error upper bound estimation methods .......................................................... 29

6.3

Conclusion .................................................................................................................................................... 35

Adversarial Example Detection..................................................................................................................... 37
7.1

Research summary ................................................................................................................................... 37

7.2

Overview of adversarial example detection approaches ......................................................... 37

7.3

NIC system design overview ................................................................................................................ 39

7.4

NIC system implementation ................................................................................................................. 40

7.5

Computer experiment ............................................................................................................................. 41

8

AI Quality Management in Operation......................................................................................................... 44

9References.............................................................................................................................................................. 46iii

Machine Learning Quality Management Guideline,

National Institute of

1st English edition

1

Advanced Industrial Science and Technology

Introduction
Machine Learning Quality Management Guideline has been developed to clearly explain thequality of various industrial products using statistical machine learning (Version 2 [1]). The
second edition of the guideline focuses on the nine internal quality characteristics for machinelearning systems (Stability of the trained model, Reliability of underlying software system, etc.),
but techniques for evaluating and improving these internal quality characteristics have not been
sufficiently established yet. This document reports the current results on survey, research, anddevelopment of techniques for evaluating and improving the internal quality characteristics,
which are being conducted for supporting the development of the guideline.

1.1 Overview of this research and development
Figure 1.1 shows the relationship between the machine learning quality evaluation andimprovement techniques (the center yellow boxes in Figure 1.1, the number in each box shows
the chapter number explained in this report) that were researched and developed for the recent
two years (FY 2019~2020). The relations to the phases of the machine learning model lifecycle
and the nine internal quality characteristics are also shown. The techniques are briefly
introduced here, and the details are explained in Chapters 2 ~ 8.
Dataset
ML modelAnalysis

Input

Training

for training

ML quality evaluation and improvement techniques
researched and developed in FY 2019~2020
outputTechniques in training

Training algo.

2: Visualization of ML models

Training prog.

3: Improving Application of DA

TrainedML modelEvaluation

Input

outputfor testing
Infer prog.
Evaluation prog.

Operation

Evaluated
ML modelDomainoutputInput

Update Data in operation

A-1: Sufficiency ofproblem domain analysis
A-2: Coverage for
distinguished problem cases

4: Debug-Testing of DNN Software

B-1: Coverage ofdatasets

Techniques in evaluation

B-2: Uniformity ofdatasets

5: Evaluation of robustness

B-3: Adequacy ofdata

6: Evaluation of Generalization

C-1: Correctness of
trained models

Techniques in operation

C-2: Stability of
trained models

7: Detection of Adversarial data
Infer prog.
Monitor prog.

Internal quality characteristics
in the MLQM Guideline

8: AI quality management

D-1: Reliability ofunderlying software systems
E-1: Maintainability ofquality during operation

Figure 1.1 Machine learning quality evaluation and improvement techniques in this report–

Visualization of Machine Learning Models (in Chapter 2) :To visualize machine learning models from the two points of views, (1) comparison
results between multiple models and (2) the sensitivity of workers (e.g., annotators of
training data, designers of model structures) reflected in the models, we report theresults of a survey on such recent visualizations and our prototype visualization tool
for model comparison.

1Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition–

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Improving Application of Data Augmentation in Deep Learning (in Chapter 3):
To improve the data-diversity obtained by data augmentation and increase accuracy
and stability in deep learning, we devise new two data augmentation methods withsimple algorithms (FC-mixup and Latent DA) and report the results of their impact on
generalization performance in experiments [2].

–Debug-Testing of DNN Software (in Chapter 4):
To test the presence or absence of defects in training/learning programs, we derive alight-weight index from the internal index (neuron coverage) in trained DNN models,
and then report the results of several experiments and discuss the usefulness of thederived index [3].

–Evaluation and Improvement of Robustness (in Chapter 5):
To evaluate and improve robustness of machine learning models, we report on the
results of a survey on methods to measure the maximum safe radius (the maximum
value of noise that can be guaranteed not to cause misclassifications) as a measure ofrobustness for input noise including adversarial examples, and methods to increase thesafe radius.

–

Estimation of Generalization Error Upper Bounds (in Chapter 6):
To evaluate the generalization performance of machine learning models, we report on
the results of a survey on methods for estimating the upper bound on the expectedvalue of the error rate (i.e., generalization error) for all inputs, including unseen
samples.

–Adversarial Example Detection (in Chapter 7):
To establish a practical method for detecting adversarial examples, we report on the
results of a survey on the state-of-the-art adversarial example detection methods andclassifies them into four main categories, and then present the results of follow-up
experiments on representative methods.

–

AI Quality Management in Operation (in Chapter 8):
To maintain quality of machine learning models during operation, we report on the
results of a survey on detection and adaptation methods for changes in input-data
distribution over time (e.g., concept drift). The results include not only supervised
methods, but also unsupervised/semi-supervised methods.

2Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

1.2 Author list
The authors of each chapter are as follows:–

Chapter 1: Yoshinao Isobe (AIST CPSEC)–

Chapter 2: Yuri Miyagi (AIST AIRC)–

Chapter 3: Tomoumi Takase (AIST AIRC)–

Chapter 4: Shin Nakajima (NII)–

Chapter 5: Yoshinao Isobe (AIST CPSEC)–

Chapter 6: Yoshinao Isobe (AIST CPSEC)–

Chapter 7: Yusei Nakashima (Techmatrix)–

Chapter 8: Yoshihiro Okawa and Kenichi Kobayashi (Fujitsu)

1.3 Acknowledgements
This report is based on results obtained from the project "Research and Development on the
Quality Assessment Reference and Testbed of Machine-Learning /artificial intelligence systems"(JPNP20006), commissioned by the New Energy and Industrial Technology Development
Organization (NEDO).

3

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

2

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Visualization of Machine Learning Models
Information visualization is becoming a popular method to support the analysis of thestructure and behavior of machine learning models, which are known as black boxes. We havestarted research on a new method for visualizing machine learning models with the following
two objectives:–

Visualization of differences and comparison results between multiple models➢

Implementation of visualization based on expressions that are easy for humans tointerpret and understand–

Visualization of the sensitivity of workers (annotators of training data, designers ofmodel structures) reflected in the model➢

Proposal of new factors that can be used for quality assessment

This chapter first explains the results of a survey of recent machine learning modelvisualization methods, and analysis about workers using models. Then we report on an example
of a prototype visualization tool for model comparison developed in 2020, and future
implementation plans.

2.1 Survey on methods to support using machine learning
The basic purpose of visualization methods for machine learning is to improve theinterpretability of models, and this is closely related to XAI (Explainable AI), which has attracted
attention in recent years. There are no definitive definitions or evaluation methods for XAI,
however, many papers about the classification of XAI are published, and we can devise
visualization objectives and methods along these lines. In [4], the approaches to increaseinterpretability are classified into four categories:(1) Total explanation (Approximation of a complex model structure by a simple model)(2) Partial explanation (Explaining the rationale for decisions about model output results)(3) Design of explainable models (Creation of readable models at the design stage)(4) Explanation of the deep learning model (e.g., Highlighting the parts of the image datathat the model recognizes)
Especially (2) and (4) have much room for contribution by visualization. These machine learning
visualization methods are continuously being studied, and the number of survey papers isincreasing due to the diversity of applications and target cases. For example, Hohman et al. [5]described and classified deep learning visualization methods according to the 5W1H elements.
It also presents several overall directions and issues in the field of deep learning visualization.
Especially "improving interactions for model evaluation" and "improving interpretability
through active human involvement in models” are closely related to our research, which aims todevelop visualization methods for quality evaluation.

4

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

As research on machine learning visualization progresses and becomes popular in the realworld, there is a growing tendency for complex analysis to be performed in a single visualization
view. In the past, visualization methods basically focused on detailed analyses of single models
specialized in either data ([6]) or model structure ([7]). However, in recent years, research has
been conducted on combined visualization methods for data and model structure, as well as
methods that aim to compare multiple models. The number of elements that make up a machinelearning model is enormous, and it takes a lot of time and effort to create visualization results
for the number of models and compare them side by side. Besides, the differences in structure
and accuracy between the models to be compared are often small and features of the models
may be overlooked. Therefore, there is a high need for a visualization method that uses
expressions that emphasize the differences so that the differences can be found efficiently within
a limited screen. (For example, in [8], the pipeline from data input to output, hyperparametervalues, etc. for more than 10 models can be compared on a single screen.)
So far, we have introduced trends and examples of visualization methods regarding theproperties and accuracy of the models themselves. In parallel with this, we have also investigated
how the workers (annotators of training data, designers of model structures) involved in model
creation interact with the models. In fields such as image recognition, models with accuracy
beyond human recognition capabilities have been developed, but there is a persistent suggestion,
regardless of the field, that active human intervention is desirable to improve the accuracy ofmodels. There are many papers that discuss the following items regarding the relationshipbetween AI and humans and effective intervention methods in the modeling process:–Introduction of operations (adjustment and evaluation) to improve the accuracy of themodel in the learning process–

Designing an interface that is easy to use and can maintain the motivation of theoperator–

Collaboration with related fields such as cognitive science and psychology

As an example, Amershi et al. examined the psychological state of workers who wereassigned feedback to evaluate and improve several models [9]. The authors found that theworkers preferred to be able to directly tell the correct processing steps to models. They also
said that workers get more motivated to give more active feedback when they find their actions
are improving the accuracy of the model. Although there seem to be few examples ofvisualization of such information about the workers themselves and the impact of each worker
on the model, it can be adopted as a ground for quality assurance as follows:–

Show that their knowledge is sufficiently reflected in the model's behavior whendomain or machine learning experts participated in the creation of the model.

–

Indicate which workers' behavior is strongly reflected in the model and use this as aclue to identify elements (training data, parameters, etc.) that should be adjusted.

Based on the above survey results, we emphasized "comparative visualization of multiple
models" and "visualization of worker sensitivity" among model visualization methods and5Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

proceeded to design a visualization tool that has both properties.

2.2 Developing prototype of model difference visualization tool
Based on the results of the survey described above, we proceeded with the study and trial
production of a visualization tool specialized for the comparison of multiple models. Assumingthat the main users of this tool would be model designers and considering the possibility thatusers who were not familiar with visualization would be included, we combined basic
visualization methods (line graphs, bar graphs, etc.) and implemented them with the policy ofactively linking them (e.g., highlighting related parts). Figure 2.1 shows the overview ofprototype visualization views, which visualize the results of MNIST training and output for two
simple models.

Figure 2.1 The whole image of the prototype visualization views.
We created the tool on JupyterLab, mainly using the machine learning library PyTorch andthe visualization library Bokeh, so that we could compare the features of the two models:(1) Network of each model structure(2) Bar graph of output results for each class① Visualization for each model② Visualization of the difference between two models(3) Scatter plot of output result correlation between two selected classes for each model

6

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003(4) Line graph of accuracy(5) Thumbnail list of data classified with particularly high (low) confidence
Figure 2.2 shows an example of the results of classifying the output to MNIST for two models.
The horizontal axis represents the class from 0 to 9, and the vertical axis represents the amountof data. The color-coding of each bar represents the combination of correct (T) and incorrect (F)answers for the two models, for example, where TF (FT) means that only model 1 (2) correctlyclassified. Immediately after the start of learning (Figure 2.2, left), model 1 had a high percentage
of correct answers in classes 0, 1, and 7, and model 2 had a high percentage of correct answers
in classes 2, 6, and 8, indicating that each model had different strengths. At the advanced stageof learning (Figure 2.2, right), both models had high percentages of correct answers in many
classes. Besides, model 1 has a high percentage of correct answers, including classes 3, 4, 5, and7, which model 2 is not good at, indicating that model 1 is more advanced in learning than model
2 at this stage.

Figure 2.2 Example of comparing the output results of two models

2.3 Future work
In addition to the view shown in Figure 2.1, we will add a view for visualizing the sensibilityof workers. Specifically, we are working on the implementation of (1) time-series visualizationof the work content and accuracy changes in the training process, and (2) visualization of thedegree of effect to the model by the annotator and model designer. For the time being, we will
try to collect and visualize the information of workers based on logs of training and adjustment
tasks for models, so that the input in interactive operations, which would increase the burden
on users, will not be excessive. In addition, for each visualization view, including the two that weplan to add, we would like to enhance the coordination function and implement expressions thatdraw attention to important differences.

7

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

3

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Improving Application of Data Augmentation in Deep Learning
In this study, we developed a new data augmentation method, which is widely used in bothbasic and applied situations where deep learning is used, and verified its impact on the qualityof deep learning through experiments using benchmark datasets for multi-class classification.

3.1 Purpose
Data augmentation is a technique to increase the number of samples by adding
transformations to those, and it is highly effective in deep learning, which has a tendency to
reduce performance when the number of training samples is small. On the other hand, theeffectiveness of data augmentation strongly depends on data to be used, so data augmentation
methods and the parameters of each method must be selected appropriately. However, thetheoretical analysis of data augmentation is difficult, and the appropriate way to use it has not
yet been established, so it is often used empirically (or conventionally). This leads toinappropriate use, which impairs the quality of learning.
Therefore, in order to facilitate a move away from the empirical use of data augmentation,
this study focuses on data diversity. Increasing diversity is an intrinsic goal of data augmentation,and investigating its impact is expected to contribute to a better understanding of data
augmentation and its development. It has been demonstrated in the work of [10] that theincrease in data diversity due to data augmentation has a significant effect on the improvementof generalization performance. In this study, we propose a new augmentation method to increase
data diversity and verify its effectiveness. Recently, a method called RandAugment [11], which
dynamically applies randomly selected operations from multiple data augmentation operations
during training, has been attracting attention. While this method greatly improves diversity, it isdifficult to use effectively because many hyperparameters need to be adjusted. In this study, we
devised two methods with simple algorithms (FC-mixup in Section 3.2 and Latent DA in Section
3.3). Although the work in [10] also proposes a metric for evaluating diversity, verification using
this metric is our future work, and we only investigate its impact on generalization performancein the experiments in this study.
In addition, data augmentation is sometimes applied to the input data prior to training, but
in deep learning, data augmentation can be applied each time a sample is input to the model
during training, so it is possible to devise various ways of applying it. The methods in Sections
3.2 and 3.3 make good use of this property and apply data augmentation dynamically to the dataobtained in hidden layers of neural networks.

3.2 Feature combination mixup（FC-mixup）
Mixup [12] is a versatile data augmentation method that can be applied not only to image
data but also to tabular data. It is a method that generates a new sample by linearly interpolating

8

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

two samples using the same ratio of linear interpolation for each of the input values and labels,
as shown in the following equation.
{𝑥̃ = 𝜆𝑥𝑖 + (1 − 𝜆)𝑥𝑗𝑦̃ = 𝜆𝑦𝑖 + (1 − 𝜆)𝑦𝑗

Here, (𝑥𝑖 , 𝑦𝑖 ) and (𝑥𝑗 , 𝑦𝑗 ) represent the input values for the 𝑖-th and 𝑗-th samples, and 𝜆 isthe mixing rate sampled from the beta distribution. A modification of this method that can beused in hidden layers of neural networks is called Manifold mixup [13], but both methods havethe disadvantage that they generate samples only in a local area of the data distribution, on aline segment between two points, and are inappropriate for datasets with distributions in which
the properties of the points on the line segment vary nonlinearly.
Feature combination mixup (FC-mixup) proposed in this study is a method of mixing samples
in a different way from the conventional mixup, and its overview is shown in Figure 3.1. Here,suppose that two samples 𝐴 and 𝐵 in the same batch output features 𝑍A and 𝑍𝐵 in arandomly selected layer. 𝑑 is the total number of features in the layer, and FC-mixup randomly
extracts and combines 𝑑𝜆 features from 𝑍A and 𝑑(1 − 𝜆) features from 𝑍𝐵 to generate a
new sample 𝑍X . Since a large number of combinations are possible for a single value of 𝜆,different data can be generated depending on the random number, and thus a wide range ofsamples can be generated on the data distribution. FC-mixup is expressed as follows, so we mix
𝑍A and 𝑍𝐵 so that this equation is satisfied.
|𝑍A ∩ 𝑍X | = d𝜆This technique of generating new data by combining the parts of two samples is also seen in
Puzzle Mix [14], but it is limited to the input image. A similar technique is used in Adversarial
mixup resynthesis [15], but it is limited to autoencoders, while FC-mixup is designed for moregeneral use. In this study, we proposed Hybrid1 and Hybrid2, which are a fusion of FC-mixup
and Manifold mixup [13], to further increase the diversity of the generated data. Hybrid1 is amethod that uses either mixup with a probability of 0.5 for each batch, and Hybrid2 is a methodthat realizes both methods simultaneously.

Figure 3.1 Overview of FC-mixup

9

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

In order to investigate the performance of FC-mixup, we used several multi-classclassification datasets and compared the test accuracy between the conventional method (nodata augmentation, mixup at the input layer [12], Manifold mixup [13]) and the proposed
method (FC-mixup, Hybrid1, Hybrid2). The model includes WideResNet28-10, ResNet50,
multilayer perceptron (MLP), and small size convolutional neural network (small CNN) wereused as models. In addition to full datasets, experiments were also conducted on reduced data,
which was randomly extracted from 1,000 samples. From the results in Table 3.1, we can see thatthe proposed method shows the highest accuracy in most cases. Although some datasets show
lower accuracy than Manifold mixup (ImageNet, reduced CIFAR-10), using FC-mixup as one ofavailable methods may lead to the improvement of the accuracy.
Table 3.1 Test accuracy for multi-class classification data (underline indicates the proposed method)

3.3 Applying data augmentations to feature maps (Latent DA)
Data augmentation is usually applied to the input data, but in neural networks, it is possible
to apply it to outputs in hidden layers. Manifold mixup [13] actually realizes this, but it isspecialized for mixup [12] and cannot handle other operations. Since features are extracted
hierarchically in CNNs so that they become more complex as they approach the output layer, theeffect of data augmentation in various layers chosen randomly for each batch is different andproduces a variety of samples. It is also easy to implement, because data augmentation can beapplied to feature maps in the same way it is applied to the input image.
Examples of actual application of masking and translation to input images and feature maps
are shown in Figure 3.2. Here, samples are input to the model under training, and the images
immediately after applying data augmentation set to the same hyperparameters (mask positionand displacement) in different layers are displayed in the top row with their sizes aligned. The
feature maps of the sample at the last layer before fully connected layers are shown in the bottomrow, and their images are different depending on the layer where data augmentation was applied.
This result shows that applying data augmentation in various layers leads to an increase in the10Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

diversity of the generated data, and that the effect of such a training is different from that whendata augmentation is applied only to the input data.

Figure 3.2 Examples of applying data augmentation to input images and feature maps
obtained at hidden layers.
In order to compare the performance of data augmentation in the input layer with that infeature maps, we used various data augmentation operations and computed the test accuracy ofthe supervised trained models. Here, we trained WideResNet28-10 on CIFAR-10, Fashion-MNIST,
and SVHN (without extra data) datasets for 200 epochs. The results are shown in Figure 3.3. Ineach figure, the horizontal axis represents the accuracy [%] of the conventional method (Input
DA) and the vertical axis represents the accuracy [%] of the proposed method (Latent DA). As
can be seen from these results, the proposed method tends to show higher accuracy than theconventional method, and the proposed method gives higher accuracy even in the case where
the conventional method gives lower accuracy, such as the result using crop. These results showthat diverse samples generated by applying data augmentation to random layers are effective in
improving the performance.

Figure 3.3 Comparison of test accuracy between Input DA and Latent DA

11

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

4

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Debug-Testing of DNN Software
In the initial development stage of Deep Neural Network software (DNN software), we ensurethat the required functions and prediction performance are achieved through iterative trial-anderror processes, in which three viewpoints (elaborating and refining requirements, preparing
datasets for training, and selecting appropriate machine learning models) are considered. This
trial-and-error process corresponds to debugging in conventional program development. In thecase of DNN software, the debugging activities involve generating datasets for debug-testing,monitoring the training and learning status, and identifying and removing root causes thathinder the fulfilment of requirements. In the following, we will report on a debug-testing method
investigated in FY2020, discuss the experimental results obtained, and summarize our futureplans.

4.1 Direct cause of failure
A standard method of supervised DNN learning involves two types of programs: training (orlearning), and prediction (or inference). When training data is given and a learning task toachieve is made clear, a learning model for the target DNN software is selected, and some design
decisions on the method used in the training and learning process is fixed. If we use available
open-source machine learning frameworks, we may set up several parameters of the framework.
The next step is to construct training dataset. Then, we run the training/learning program(possibly provided by the machine learning framework) with the training model and training
dataset as input, and derive a trained DNN model as a computation result. More precisely, thetraining/learning program searches for a set of weight parameter values that define the trained
DNN model uniquely. This trained DNN model defines behavior of the prediction/inference
program.
From a user's point of view, a prediction/inference program is the entity to use. In the caseof a classification learning task, for example, the program calculates certainty levels ofprobabilities of classification results for an input data. By examining the output results, we candetermine whether the DNN software works as intended. When the program does not produce
results as expected, we localize possible fault locations and remove them. In other words, we
conduct debugging.
A failure may be occurred due to a flaw somewhere in the information used in the execution
process of the training/learning program, either in the training dataset, the training model, thelearning mechanism, or their combinations. However, direct causes of failure in prediction/
inference results are attributed to the trained DNN model or set of obtained weight parametervalues. While a root cause of failure is somewhere and often not known, the failure is attributed
to a defect in the weight parameter values or the trained DNN model. Thus, from users' point ofview, a certain distortion of the trained DNN model seems a direct cause of the failure [16]. A
method to measure such distortion degrees is needed regardless of the root causes.

12

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

In this chapter, we investigate whether we can detect faults in DNN software with an internal
metric to measure such distortion degrees of trained DNN models. The weight parameter values
in the DNN models are the output of the training/learning program, but there is no direct way tocheck its validity, because those expected weight parameter values cannot be known in advance.
If such expected parameter values were known, training/learning could be skipped. We can justuse those known values, as embedded in a trained DNN model, to implement a prediction/
inference program.

4.2 Internal indicesThis section first introduces the notion of neuron coverage (NC). We consider a learningmodel as a network of neurons. Given a threshold, neurons whose output values exceed thethreshold are said to be activated. When the number of neurons constituting the learning modelis N and the number of activated neurons is A, the neuron coverage is defined as the ratio ofactive neurons is (NC = A/N). In [17], NC is assumed to be criteria for test coverages of trained
DNN models; the research work investigates how the choice of input data for evaluation affects
NC values.

Cla

ithm
lgor
A
n
catio
ssifi

Figure 4.1 Trained DNN model.
In this chapter, NC is assumed to be used as an internal index [18] to represent distortion
degrees by appropriately choosing the target neurons to be considered. Figure 4.1 shows aschematic diagram of the trained DNN model. NCs are defined for the neurons in the final stageof the middle layer (or the penultimate layer as shaded gray), but not for all the neurons in the
trained DNN model as in [17].
In general, in machine learning techniques, this penultimate layer is often considered to holdmeaningful information. For example, in the case of an image classification task, the early stagesof the model is responsible for the correlation analysis (analysis of patterns of pixel values),
which plays a specific role in algorithms such as image recognition, and their calculation results
are summarized in the penultimate layer. In this chapter, we assume that direct causes of defects
are manifested in this internal layer. Furthermore, various statistical indices can be derived
based on NC values of this layer. We will investigate, through experiments, what derived index isappropriate depending on test objectives to be investigated.

13

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

4.3 Experiments: method and results
We present the results of several experiments and discuss the usefulness of the internal orderived indices mentioned in the previous section. First, we show the results of comparative
experiments when a training/learning program (or a learning framework) has faults in it. In thefollowing, BI is the training/learning program which is a bug-injected version of a probably
correct program PC.
Figure 4.2 depicts the accuracy (the percentage of reconstructed correct answers) for a test
dataset. In the experiments, a classical fully-connected network is chosen as the learning model,
and different number of neurons are placed in the middle layer, which implies that each model
is of different structural capacity. When we have a sufficient number of neurons (50 on thehorizontal axis), there is no significant difference in the accuracy between PC and BI. Thus, it isdifficult to distinguish between the PC and BI solely by examining their accuracy values, and thus
the presence or absence of a defect cannot be identified. In addition to this finding (Figure 4.2),
the results of an experiment to systematically investigate the situation further (Figure 4.3) arepresented below.
1

Accuracy

0.98

0.96
PC
BI

0.94

0.92

0.9
0

10

20

30

40

50

60

Structural Capacity

Figure 4.2 Learning models of different capacities.

Activated Neurons

1
0.9
0.8
0.7
0.6
0.5

PC

0.4

BI

0.3
0.2
0.1
0
0

10

20

30

40

50

60

Structural Capacity

Figure 4.3 Relationship with internal indicesFigure 4.3 plots values of the internal index (activated neurons or neuron coverage) on thevertical axis. Their absolute values, for example, of 10 for BI and 30 for PC are both around 0.7,
making it impossible to distinguish between BI and PC if we do not take into account thestructural capacity. The indices are not usable to examine the activated states of neurons.

14

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Therefore, we will study if there is an appropriate indicator to be derived from the internal indexof NC. As a set of data (a sample), in the test dataset, leads to a collection of neuron coverages,
we can obtain some statistics from the sample such as the mean μ and variance σ2, andcalculate σ/μ. Figure 4.4 shows the case where this derived index σ/μ is used on thehorizontal axis. From the values on the vertical axis, we can find out which leaning model has
which value by referring to Figure 4.3.
1

Accuracy

0.98

0.96
PC
BI

0.94

0.92

0.9
0

0.1

0.2

0.3

/µ

0.4

p

Figure 4.4 Derived index
Figure 4.4 shows that we can distinguish between the PC and BI. Although the internal index
cannot distinguish between the PC and BI with different capacities (Figure 4.3), a derived indexof σ/μ can discriminate between the PC and BI. We can see that the neuron coverage basicallycontains a piece of useful information.
Next, Figure 4.5 is a scatter plot of classification probability using corrupted data for theevaluation; the horizontal axis refers to the classification output by the BI and the vertical one
by the PC.
1.2

Probability of PC

1

0.8

0.6

0.4

0.2

0
0

0.2

0.4

0.6

0.8

1

1.2

Probability of BI

Figure 4.5 Classification certainty for corrupted data.
In Figure 4.5, a △ represents an output value for corrupted data, which is supposed to be
distributed on the dotted line passing through the origin, if we assume that the PC and BI output
the same value for the same data. In fact, it can be seen that □ selected from the test dataset

15

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003(without any corruption) mostly arranged on the dotted line. On the other hand, corrupted data(△) are distributed along the solid line, indicating that the PC is a better classification certainty
than the BI. It implies that the BI, containing bugs in it, is less robust, although the accuracy
remains the same as that of PC (Figure 4.2).
The following experiment confirms that differences in robustness can be detected by using
an internal index (Figure 4.6).

Figure 4.6 Differences in internal indices
The corrupted data described above were input, and the internal index for each input wasplotted on the horizontal axis. The □ distributed in a group on the right side shows the resultsof PC, and the ◇ distributed in a group on the left side shows the results of BI. The scatter plotshows that (1) the value of the internal index of PC is large, and (2) the correlation between theinternal index and prediction probability (certainty of classification) is negligible (0.033). Next,
we calculate σ/μ, which is 0.0876 for PC and 0.2183 for BI. Figure 4.6 shows results that
corrupted data affect the robustness, and that the value of σ / μ is considered to havecorrelations with the robustness.
Next, we conducted experiments to investigate how distorted training data affect the trained
DNN model. We plotted the accuracy for a test dataset common to all the cases. Thus, differencesin the vertical axis indicate a certain difference (distortion degree) in the training dataset used
for obtaining the trained DNN model (Figure 4.7).

Figure 4.7 Differences in training datasets.

16

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Figure 4.7 shows the two independent series for the PC (□) and BI (◇). From top to bottom
in a series of each measured points (from better to worse accuracy), a training dataset with alarger distortion is used. Because the test dataset is common, the data shift of the test data isrelatively larger as the distortion degrees in the training data is larger. Furthermore, the accuracy
decreases as the shift becomes large. Figure 4.7 also shows that the value of the horizontal axis(σ/μ) is clearly different between the PC (□) and BI (◇). It can be confirmed that the accuracy
and the robustness suggested by the σ/μ values are two independent perspectives.
From the above (Figure 4.7), the distortion in training dataset can be examined by themethod based on the accuracy. As is done in practice, the method based on the accuracy is usefulwhen checking the training dataset quality. On the other hand, if there is a possibility that otherfactors such as faults in a training/learning program are involved (multiple defects are assumed),
it is desirable to examine the values of the internal and derived indices (σ/μ) at the same time.

4.4 Related work
Neuron coverage (NC) is a simple quantitative measure introduced in DeepXplore [17] as atest coverage metric. In conventional software testing, test coverage is defined in terms of thebasic block of program codes, which is the statements executed by a given test input data. A
program is represented as a Control Flow Graph (CFG) whose nodes refer to executablestatements. In the simplest case, the criterion is whether or not a node in the CFG is containedin an execution path induced by an input test, i.e., whether or not the statements are executed.
As a DNN model is represented as a network, a kind of graphs, metrics similar to those for CFG
can be introduced. The neuron coverage concerns whether neurons located at nodes areactivated (output values of these neurons exceed a specified threshold), which is comparable to
the C0 criterion defined on the CFG. DeepXplore assumes that high NC values refer to thesituations where high percentage of neurons are exercised by input data, and discusses how to
generate new test input data to increase the NC values.
Neuron coverage would be a straightforward idea analogous to the conventional test
coverage criteria. Later, satisfying the criteria, to achieve 100% in terms of NC, is foundempirically not difficult. New metrics are proposed to take into account correlations among
multiple neurons or those in different layers [19], which may be comparable to more elaborated
coverage metrics, such as C1 or the others, in conventional software testing.
The original NC is simple and easy to use as a metric to guide or control automated test
generation processes. Usually, a classical data augmentation method picks up a seed data, fromwhich a series of new data is to be generated by pre-defined data transformation algorithms.
New test data are successively generated until the accumulated NC values is saturated. Whenreached the situation where no increase in the NC is seen, the generation method switches a seeddata to new one and continue the process [20]. The classical data augmentation method can be

17Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

replaced by other approaches such as test input generation based on GAN [21]. Test generation
method using GAN with a help of NC is reported in [22]. Although it is a simple metric, NC is now
considered as a practical criterion to control the automated test generation process (coverageguided test generations).
Some of early works on testing pre-trained DNN models adapt application-specificproperties as software test oracles; the DNN models for regression tasks in the auto-pilot car
application [20][21] use the calculated steering angle as the oracle. There is also a research work
[23] to investigate whether test inputs to increase the NC values are useful for detecting faults.
The usefulness of NC is dependent on what are considered failures. The work [23] also indicatesthat the correlation between NC and external indices such as the accuracy is weak. In this chapter,based on this observation that the correlation between the two is weak, an internal index basedon the NC is used for the test, which is not contradictory to the discussion in [23], but rather inthe same direction. Note that the test coverage is a criterion for terminating testing, while
detecting faults depends on whether the test input data executes corner cases. These two notions,the test coverage and corner cases, refer to different aspects. In fact, it has been reported thatthe enhancement of coverage does not necessarily leads to the improvement of the efficiency offault detection in conventional software testing. The same findings would be applicable to casesof DNN testing.
In this chapter, we use the NC value as a simple test index, from which a sort of distortion
degrees in trained DNN model is derived [16][18]. Our approach is based on a view that faults
in DNN models appear as inappropriate NC values, whereas existing works use NC as a criterion
for the test coverage. In our experiments, we were able to examine the reliability of the training
and learning programs and the robustness of the trained DNN models. These are two primaryconcerns in debug-testing.

4.5 Conclusion
In this chapter, we used an internal index based on the neuron coverage (NC) defined on the
penultimate layer for representing a sort of distortion degrees in trained DNN model. The NC isa scalar and easy to measure, and thus can be used as a light-weight test index. It, however,discards the information about the individual activated neurons, and thus lacks useful
information. In fact, Kim et al. [24] proposes a method to estimate the distribution of activatedneuron and to discuss the usefulness of input data for testing. Distribution on such neuron values
may be considered to have rich information. In future, we will study how to debug training
dataset by making use of such distribution information.

18

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

5

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Evaluation and Improvement of Robustness
In this chapter, robustness means the ability that a machine-learned model keeps correct

output even when noise is added to input (including adversarial examples). For example, itevaluates how much noise can be added to the model without changing the correct results. Oneof the measures of its robustness is the maximum safe radius (MSR). In this chapter, we explain
adversarial example and the maximum safe radius in a classifier based on a feedforward neuralnetwork, and then report the results of a survey on techniques for estimating and increasing the
maximum safe radius.

5.1 Robustness measure (maximum safe radius)
It is well known that machine-learned models on inference programs mis-classify input data
even when very small perturbations are added. Such perturbated data are called adversarial
examples [25], and adversarial examples have been actively researched in recent years. The set
𝐴𝑑𝑣𝛿 (𝑥) of all adversarial examples contained in the 𝛿-neighborhood (inside the sphere ofradius 𝛿 ∈ ℝ, where ℝ is the set of real numbers) of the input data sample 𝑥 ∈ ℝ𝑛 is defined
as follows:𝐴𝑑𝑣𝛿 (𝑥) = {𝑥′ | ‖ 𝑥 − 𝑥′ ‖ ≤ 𝛿 ⋀ 𝑓(𝑥) ≠ 𝑓(𝑥′)}where 𝑓(𝑥) is a function representing the machine-learned model that takes the input sample
𝑥 and return the classification, and ‖𝑥 − 𝑥′‖ is the distance between two data samples 𝑥 and
𝑥′. The 𝑝-norm is often used to define the distance.

Figure 5.1 An adversarial example from an image of a panda, which is mis-classified into a gibbon
Adversarial examples are explained by Figure 5.1. The left side in Figure 5.1 shows the input
space to the neural network and the right side shows the output space from the neural network.
The center of the red sphere in the input space represents an original input image of a panda,
and the inside of the sphere, whose radius is 𝛿, (i.e., 𝛿-neighborhood of the original image)19

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

represents the set of perturbated images obtained from the original image by adding noises
whose sizes are less than 𝛿. The set of outputs from the neural network for all the input images
in the 𝛿-neighborhood corresponds to the red region in the output space on the right. Here, apart (lower-right) of the red region in the output side is beyond the decision boundary and ismapped into the region of gibbons. It means misclassification, and the input images mapped to
the lower-right part are adversarial examples.
If there is no adversarial example in the 𝛿-neighborhood of the input data 𝑥 (i.e., inside thesphere whose radius is 𝛿 and center is 𝑥), then 𝛿 is said to be the safe radius of 𝑥. Then, themaximum safe radius of 𝑥, denoted by 𝑀𝑆𝑅(𝑥), is defined as follows:
𝑀𝑆𝑅(𝑥) = max {𝛿 | 𝐴𝑑𝑣𝛿 (𝑥) = ∅}When the maximum safe radius of 𝑥 is large, it is difficult to generate adversarial examples.
Therefore, the maximum safe radius can be used as a measure of the robustness to input
perturbations, including adversarial examples, of machine-learned models.
The radius 𝛿 in Figure 5.1 is not a safe radius because some perturbated input images insidethe 𝛿 -neighborhood are misclassified into gibbons. On the other hand, 𝛿 in the following
Figure 5.2 is the maximum safe radius because all the input images inside the 𝛿-neighborhoodin Figure 5.2 are correctly classified.

Figure 5.2 The maximum safe radius 𝛿

5.2 A survey on methods for evaluation and improvement of robustness
Table 5.1 shows recent research papers on methods for evaluation and improvement ofrobustness, where each small box in the table represents a research paper with reference andthe information on neural networks used in the experiments for evaluating the methods
proposed in the paper. The information is useful for comparing applicable scales of the methods.
Table 5.1 is categorized by the following perspectives:

20

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Table 5.1 Methods for evaluation and improvement of robustness (MSR: Maximum Safe Radius)
Evaluation of robustness

Improvement of robustness

Rigorous estimation of MSR
Katz et al. 2017 (Reluplex) [26]ACAS-XU-DNN, 300 ReLU nodes
6 hidden layers,
Rigorous(Limitation: hundreds of nodes)
Tjeng et al. 2019 [27]CIFAR-10, ResNet, 9-CNN, 2-layer,
107,496 ReLU units,
100~1,000 times faster than Reluplex

Deterministic

CertifiedEstimation of a lower bound (LB) of MSR

Training by detecting all the adversarial exes

Weng et al. 2018 (Fast-Lin) [28]Wong and Kolter 2018 [33]

CIFAR, 6-layer, 12,288 ReLU units

SVHN (32x32x3), 2-conv, 32-ch,

About 10,000 times faster than Reluplex

100, 10 hidden units, ReLU,(Non-applicable to ImageNet)

Boopathy et al. 2019 (CNN-Cert)[29]
CIFAR-10 (32x32x3), 5-layer,
10 filters, 29,360 hidden nodes,

Approximative

Faster than Fast-Lin

Probabilistic

Estimation of a probabilistic LB of MSR

Randomized smoothing after training

Weng et al. 2019 (PROVEN) [30]Lecuyer at el. 2019 [34]

CIFAR, 5-layer, CNN, ReLU

ImageNet (299x299x3),

almost same as CNN-Cert

Inception-v3 + auto-encoder
Cohen at el. 2019 [35]ImageNet (299x299x3),
ResNet-50 (50-layer)
Tighter certification than Lecuyer [34]Uncertified

Estimation of an upper bound (UB) of MSR

Training by detecting near adversarial exes

Carlini and Wagner 2017 [31]Madry et al. 2018 [36]ImageNet (299x299x3),

CIFAR (32x32x3),

Inception-v3

28-10 wide ResNet

Estimation of an approximation of MSR
Weng et al. 2018 (CLEVER) [32]ImageNet (299x299x3),
ResNet-50 (50-layer)

21Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition––

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Columns in Table 5.1 (application):➢

Evaluation of robustness by estimating MSR➢

Improvement of robustness by increasing data samples with a specified MSR

Row in Table 5.1 (certification and strictness):➢➢

Certification of no existence of adversarial examples in 𝛿-neighborhood


Rigorous estimation of MSRApproximative estimation of MSR⚫

Deterministic (no adversarial example exist)⚫

Probabilistic (the probability of no adversarial example is 𝜌%)

No certification of no existence of adversarial examples in 𝛿-neighborhood

The methods in Table 5.1 are explained in the following Subsections 5.2.1~5.2.7.
Certified and rigorous evaluation of robustness
Katz et al. [26] proposed a method, Reluplex, to verify that a machine-learned model satisfies
given properties. A demonstration tool that implements the method Reluplex has also beenreleased. Properties are constraints on input-output relations of machine-learned models, andReluplex can exhaustively and rigorously (soundly and completely) verify that there is no
adversarial example in the 𝛿-neighborhood of the input data sample. Therefore, the maximum
safe radius (MSR) can be estimated by checking the existence of adversarial examples bychanging the radius 𝛿 with binary search. Reluplex is an extended Simplex method (one ofsolvers for linear programming problems) with rules for the ReLU function and it is implemented
by a satisfiability-checking tool (SMT-Solver) with a module for the theory of real numbers.
Reluplex is a powerful tool to verify properties in addition to robustness, but the computational
cost is expensive and the number of neurons it can verify is a few hundred ReLUs at most.
Tjeng et al. [27] proposed an efficient method for estimating maximum safe radii. Then, they
implemented the method on a mixed integer linear programming (MILP) solver anddemonstrated that the tool can exactly estimate the maximum safe radii of a neural network with
100,000 ReLU-type neurons. Although it is still difficult to apply the rigorous solver-based tools
to practical large-scale machine-learned models, the scalability is being improved.
Certified, approximative, and deterministic evaluation of robustness
Weng et al. [28] proposed a method, Fast-Lin, to approximate the maximum safe radii of
ReLU-type neural network. Fast-Lin linearly approximates the output region with a polytope andestimates an approximation δ that is slightly smaller than the maximum safe radius, as shown in
Figure 5.3. It is guaranteed that there is no adversarial example inside the 𝛿-neighborhood
because the approximation δ does not exceed the maximum safe radius (i.e. sound). It means δ
is a safe radius and is a lower bound of the maximum safe radius (𝛿 ≤ 𝑀𝑆𝑅(𝑥)). It was reported
that Fast-Lin is 10,000 times faster than the rigorous method Reluplex by approximative convex

22

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

outer polytopes.

Figure 5.3 An approximation 𝛿 that is slightly smaller than the maximum safe radius (MSR)
Boopathy et al. proposed CNN-Cert, which is an improved version of Fast-Lin [29]. CNN-Certalso supports convolutional networks including not only the activation function ReLU but also
sigmoid, tanh, and arctan, and it improves approximation accuracy and is faster than Fast-Lin.
Certified, approximative, and probabilistic evaluation of robustness
Weng et al. [30] proposed a method, PROVEN, to approximate probabilistic maximum safety
radii. As shown in Figure 5.4, the probabilistic maximum safe radius 𝛿 with a probability 𝜌means that there is no adversarial example inside the 𝛿-neighborhood with a probability 𝜌. In
other words, it permits the existence of adversarial examples with the probability (1 − 𝜌).
PROVEN has been developed based on CNN-Cert, and the computational complexity has not
significantly increased from CNN-Cert.

Figure 5.4 An approximation 𝛿 that is slightly smaller than the probabilistic MSR with 𝜌

23Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Uncertified evaluation of robustness
Carlini and Wagner [31] proposed a method to detect the (almost) closest adversarial
example to the input data sample 𝑥 and estimate the distance 𝛿 as an approximative
maximum safety radius by using an existing optimization tool (Adam). However, it is notguaranteed that the distance 𝛿 estimated by the method is the shortest distance to theadversarial example, and there is a possibility that there are adversarial examples closer thanthe distance. In other words, it is an upper bound of the maximum safe radius (𝑀𝑆𝑅(𝑥) ≤ 𝛿).
Although it is not guaranteed that the distance δ estimated by the method is a safe radius, it isoften used for evaluation in recent papers on robustness as a measure of the maximum safe
radius.
Weng et al. [32] proposed the method CLEVER to estimate an approximate maximum safe
radius as an evaluation measure of robustness independent of attack methods. It was reported
that the method could be applied to relatively large neural networks and the image recognition
model Inception-v3 was evaluated in about 10 seconds. The method estimates an approximative
maximum safe radius based on the maximum effect in output caused by small changes in input,
where the maximum effect is approximated by the extreme value theory. As shown in Figure 5.5,the estimated value 𝛿 can be larger than the maximum safe radius, and thus there is apossibility that adversarial examples exist inside the 𝛿-neighborhood (i.e., it is not guaranteedthat 𝛿 is the safe radius).

Figure 5.5 An approximation of the maximum safe radius (uncertified)Certified, approximative, and deterministic improvement of robustness
Wong et al. [33] proposed a method (robust training) to train such that the maximum safe
radius of each data in the training dataset to be a specified value 𝛿. Although this method does
not guarantee that the maximum safe radius δ is obtained for every training data sample after
training, it also gives a method to estimate an approximative value (a safe radius) of the
maximum safe radius for each input data sample. In the robust training, neural networks try tolearn such that they correctly make inferences for not only training data samples but also the 𝛿neighborhood of every sample.

24

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

A sketch of the robust training is shown in Figure 5.6, where the black dotted line in the
output space represents the decision boundary learned by a normal training, and the red solidline represents the decision boundary learned by the robust training. The six training data
samples in the input space are correctly classified by both the boundaries, but some data in the𝛿-neighborhood of each sample are misclassified by the dotted boundary (normal training). On
the other hand, data in the 𝛿 -neighborhood of each sample are also learned in the robust
training as shown in the red boundary. The robust training can guarantee some safe radii, but it
is difficult to apply the training to practical large scale neural networks due to the low scalability.
Wong et al. [33] reports that the robust training was successfully applied to the datasets ofimages, MNIST (28 × 28) and SVHN (32 × 32) but was not applicable to ImageNet (256 × 256).

Figure 5.6 Robust-trining by input data with 𝛿-neighberhood
Certified, approximative, and probabilistic improvement of robustness
Lecuyer et al. [34] proposed a method to estimate maximum safe radii that can beprobabilistically guaranteed by randomized smoothing. In the randomized smoothing, theinference for the same input is repeated in a neural network where a noise layer is added after
training, and the final output is the average of the outputs obtained by the repeated inferences.
A sketch of the randomized smoothing is shown in Figure 5.7, where the black dotted line in
the output space represents the decision boundary without randomized smoothing, and the redsolid line represents the decision boundary with randomized smoothing. The randomizedsmoothing of Lecuyer et al. [34] improves robustness by smoothing decision boundaries withcertification of safe radii and has been successfully applied to guarantee the robustness ofmachine learned models for large-scale input data such as ImageNet (299 × 299 × 3). When thevariance of the added noise is increased, the guaranteed safe radius also increases, but on theother hand, the correctness (e.g., accuracy) decreases. Lecuyer et al. [34] applied the techniqueof differential privacy, where the output for two similar inputs is made statisticallyindistinguishable, to clarify the relations between certifiable approximative probabilistic
maximum safe radii, the standard deviation of noise, the number of inferences, and so on.
Cohen et al. [35] proposed a randomized smoothing based method that can estimate tighter
certifiable approximative probabilistic maximum safe radii than one of Lecuyer et al. [34].

25Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Although randomized smoothing needs repeated inferences (tens or hundreds of times
experimentally) for an input, it can probabilistically guarantee robustness even for large-scale
networks.

Neural network
Output

input
...

...

...

...

...

noise for smoothing

Figure 5.7 Improvement of robustness by randamized smoothing
Uncertified improvement of robustness
Madry et al. [36] proposed a method (adversarial training) to train such that maximum safe
radius of each data in the training dataset to be a specified value 𝛿. In the adversarial training,
samples to be potentially adversarial examples in 𝛿-neighborhood are detected during training
and are also used as training data. Compared to the robust training of Wong et al. [33], theadversarial training cannot guarantee robustness, but it is more applicable to larger networks.
In addition, compared to randomized smoothing, the adversarial training does not require
repeated inferences.

5.3 Conclusion
In general, improvement of robustness tends to decrease accuracy, and currently accuracy isoften more important. However, if robustness is not considered, accuracy may rapidly decrease
even by small input perturbations. Therefore, robustness is important in critical systems. The
methods related to the maximum safe radius, which is a measure of robustness, explained in thischapter have been proposed recently, and environments for applying such methods have notbeen established well yet. Since such methods have been experimentally applied also to practicalmachine learned models, we think that the maximum safe radius can be one of measures ofrobustness in a few years.

26

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

6

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Estimation of Generalization Error Upper Bounds
In this chapter, we report the results of a survey on techniques for estimating thegeneralization error upper bound in feedforward neural networks, especially classifiers, for the
purpose of guaranteeing the behavior of machine learned models for unseen input data.

6.1 Generalization error upper bounds
In this chapter, we focus on classifiers based on feedforward neural networks, as shown in
Figure 6.1, where it is assumed that there exists a correct classification class 𝑦 ∈ 𝒴 for each
input data sample 𝑥 ∈ 𝒳, and sample data (𝑥, 𝑦) are according to the distribution 𝒟. A trainedneural network is a function ℎ from the input space 𝒳 to the output space 𝒴. In this chapter,the function ℎ is called a hypothesis. A neural network can represent various hypotheses byadjusting parameters (weight matrix, etc.), where the set of representable hypotheses is denotedby ℋ. Therefore, training is to select a hypothesis ℎ from a set of hypotheses ℋ that fits thetraining dataset by a training algorithm.

Figure 6.1 Generalization error and empirical error
Then, the generalization error 𝐿(ℎ) of the hypothesis ℎ (a classifier) is the expected valueof the error rate for all input data according to distribution 𝒟, and is defined as follows.
𝐿(ℎ) = 𝔼(𝑥,𝑦)~𝒟 [𝑙(ℎ, (𝑥, 𝑦))]where 𝑙(ℎ, (𝑥, 𝑦)) is the loss function, which returns 0 if the inference ℎ(𝑥) matches thecorrect output 𝑦 for the input 𝑥 and return 1 otherwise (i.e., 0-1 loss function).
𝑙(ℎ, (𝑥, 𝑦)) = {01

if ℎ(𝑥) = 𝑦otherwise

The empirical error 𝐿̂𝑆 (ℎ) of the hypothesis ℎ is the error rate of the inferences ℎ(𝑥) for𝑚 input data samples 𝑥 in the dataset 𝑆~ 𝒟 𝑚 , which is according to distribution 𝒟.

27

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

𝐿̂𝑆 (ℎ) =1∑ 𝑙(ℎ, (𝑥, 𝑦))𝑚(𝑥,𝑦)∈𝑆

Especially, if 𝑆 is the training dataset, then 𝐿̂𝑆 (ℎ) is also called a training error.
In general, it is difficult to exactly calculate the generalization error because there are
numerous input data in the input space, but various estimation methods of generalization error
upper bound, which guarantee that the generalization error is probabilistically less than theupper bound, have been already proposed. Valle-Pérez and Louis [37] presented a table (Table 1
in the paper [37]) of classified generalization error upper bound estimation methods according
to their application conditions.
Table 6.1 A classification of generalization error upper bound estimation methods (Table 1 in [37])
Data independent

Data dependent

Uniform convergence

Rademacher complexity-based bounds

By VC dimension that is a complexity

By Rademacher complexity of hypothesis

measure of hypothesis set.

set and dataset.

Structural Risk Minimization-based bounds
By dividing a hypothesis set with non-uniform convergence into countable hypothesis
subsets with uniform convergence and by applying a complexity measure to each subset.
Margin-based bounds
By the output margin and norm of weightmatrix in a trained hypothesis
Sensitivity-based bounds
By the insensitivity to noise and norm of
weight matrix in a trained hypothesis

Other

Algorithm independentNon-uniform convergence

Algorithm independent

VC dimension-based bounds

Compression-based bounds

Stability-based bounds

By the dependency of a trained hypothesis

By the stability of the randomized training

to training dataset

algorithm to training dataset
Marginal likelihood-based bounds
By the marginal likelihood of the priordistribution of hypotheses

Table 6.1 shows the estimation methods, which are classified in the same perspectives inTable 1 of Valle-Pérez and Louis [37]. The estimation methods in the upper part of Table 6.1 arebasic and can be applied without any conditions on training algorithm, but the accuracy of theestimation is low (i.e., the difference between the generalization error and the upper bound is28

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

very large) because it estimates the generalization error for the worst hypothesis in the
hypothesis set. On the other hand, the estimation methods in the lower-right part of Table 6.1
impose some conditions on their application (e.g., the stochastic gradient descent method must
be applied, and it must achieve zero training-error), but the accuracy of the estimated upperbound is higher.

6.2 Generalization error upper bound estimation methods
In this section, we briefly explain each generalization error upper bound estimation method
in Table 6.1. For more details, refer to the paper [37] by Valle-Pérez and Louis.
Vapnik-Chervonenkis-based bounds
One of basic generalization error upper bound estimation methods is based on the VC(Vapnik-Chervonenkis) dimension [38]. The VC dimension 𝑉𝐶(ℋ) is a complexity measure of
the hypothesis set ℋ, which is the maximum number of data that can be divided by ℋ. For
example, the VC dimension of the hypothesis set of a linear classifier with 𝑛-dimensional inputs
is 𝑛 + 1. Although it is difficult to calculate the exact VC dimension of neural networks, theirapproximations and upper bounds can be estimated efficiently [39].
The generalization error upper bound can be estimated by the following inequality [37]using the VC dimension 𝑉𝐶(ℋ) of the hypothesis set ℋ prepared as candidate hypotheses.
This inequality holds with probability (1 − 𝛿) for any training data set 𝑆~𝒟 𝑚 (size: 𝑚) andany hypothesis ℎ ∈ ℋ.
1
𝑉𝐶(ℋ) √ln 𝛿̂
√𝐿(ℎ) ≤ 𝐿𝑆 (ℎ) + 144+𝑚𝑚
When the parameter 𝛿 is close to 0, the probability of this inequality is close to 100%, but
the right side becomes infinitely large. Therefore 𝛿 must be set to a appropriate value. For
example, if 𝛿 is set to 0.01, it can be guaranteed with probability 99% that the generalization
error 𝐿(ℎ) is less than the value on the right side of the above inequation.
Most of the generalization error upper bound estimated based on VC-dimension exceed
100% because it considers the case that the worst hypothesis is selected from the hypothesis
set ℋ (determined by the structure of the neural network, etc.). Therefore, it is difficult to use
it to evaluate the generalization performance of hypotheses (machine learned models).
It is noted that 𝑉𝐶(ℋ) becomes infinite and cannot be given an upper bound if the
hypothesis set ℋ does not have the uniform convergence property, where the uniform
convergence means that if sufficiently many training data samples are prepared then the
difference between the generalization error and the training error can be sufficiently small forany hypothesis ℎ ∈ ℋ. Since the hypothesis set does not always have the uniform convergenceproperty in practical machine learning, the generalized error upper bound estimation method

29Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

for non-uniform convergence is explained in Subsection 6.2.3.
Rademacher complexity-based bounds
The Rademacher complexity [38] is the complexity 𝑅(𝑆, ℋ) of the hypothesis set ℋ for
the dataset 𝑆. The complexity 𝑅(𝑆, ℋ) is the expected value of the supremum of the difference
between the two empirical errors |𝐿𝑆1 (ℎ) − 𝐿𝑆2 (ℎ)| for any hypothesis ℎ ∈ ℋ , when thedataset 𝑆 is randomly divided into two subsets 𝑆1 and 𝑆2.
The generalization error upper bound can be estimated by the following inequality [37]using the Rademacher complexity 𝑅(𝑆, ℋ). This inequality holds with probability (1 − 𝛿).
4
2 ln
√
𝛿
𝐿(ℎ) ≤ 𝐿̂𝑆 (ℎ) + 2𝑅(𝑆, ℋ) + 4𝑐𝑚where 𝑐 is a constant. Since the training dataset S is considered, the Rademacher complexity
bounds are lower than the VC dimension bounds. But since the hypothesis (the machine learned
model after training) is not considered, most of the generalization error upper bound exceed
100%. It is difficult to use the Rademacher complexity bounds to evaluate the generalization
performance of hypotheses by the same reason as the case of the VC dimension bounds.
Structural Risk Minimization-based bounds
The structural risk minimization (SRM) can be used for estimating the generalization error
upper bound if a hypothesis set ℋ can be countably divided into hypothesis subsets ℋ𝑖 withuniform convergence (i.e., ℋ = ⋃𝑖∈ℕ ℋ𝑖 ), namely ℋ has the non-uniform convergenceproperty which is sufficiently practical.
In the structural risk minimization for a hypothesis set ℋ with non-uniform convergence,
the generalization error upper bound can be estimated by the following inequality [37] using
each hypothesis subsets ℋ𝑖 . This inequality holds with probability (1 − 𝛿) if the training errorof the hypothesis ℎ is zero (i.e., 𝐿̂𝑆 (ℎ) = 0 ).
𝐿(ℎ) ≤

1
1
1(ln+ 𝐶𝑖 (𝑆) + ln )𝑚
𝑃𝑖 (ℎ)
𝛿

where 𝑃𝑖 (ℎ) is the distribution function of the hypotheses ℎ in the hypothesis subset ℋ𝑖 , and𝐶𝑖 (𝑆) is the arbitrary complexity measure of ℋ𝑖 and the training dataset 𝑆. For example, sinceℋ𝑖 has the uniform convergence property, 𝐶𝑖 (𝑆) can be instantiated by the VC-dimension
𝑉𝐶(ℋ𝑖 ), the Rademacher complexity 𝑅(𝑆, ℋ𝑖 ), and so on.
Generalization error upper bounds for practical hypothesis set ℋ with non-uniform
convergence can be estimated in the structural risk minimization. However, most of thegeneralization error upper bounds exceed 100% by the VC dimension or the Rademacher
complexity. It is still difficult to use it to evaluate the generalization performance of hypotheses.

30Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Margin based bounds
The output margin is the difference between the largest output of the neuron and the second
largest output of the neuron in the output layer. Even if two hypotheses have the same empiricalerror, the hypothesis with larger margins can be more robust for noise than the other one. The
empirical error 𝐿̂𝑆,𝛾 (ℎ) considering such output margins is defined as follows.
𝐿̂𝑆,𝛾 (ℎ) =1∑ 𝑙𝛾 (ℎ, (𝑥, 𝑦)),𝑚(𝑥,𝑦)∈𝑆

where 𝛾 is a threshold of output margins and 𝑙𝛾 (ℎ, (𝑥, 𝑦)) is the loss function with margin
defined as follows.
𝑙𝛾 (ℎ, (𝑥, 𝑦)) = {0 if ℎ(𝑥)[𝑦] ≤ 𝛾 + maxℎ(𝑥)[𝑦 ′ ]′𝑦 ≠𝑦1 otherwise

where, ℎ(𝑥)[𝑦] is the output value of the neuron assigned to the class 𝑦 ∈ 𝒴 in the output
layer of the neural network (i.e., ℎ(𝑥) = argmax𝑦∈𝒴 (ℎ(𝑥)[𝑦]) ). In the empirical error 𝐿̂𝑆,𝛾 (ℎ)
with margin, even if the output value of a neuron assigned to the correct class is larger than the
maximum output value of the other neurons, if the difference (output margin) is less than thethreshold 𝛾, it is counted as a misclassification.
In the margin-based bounds, the generalization error upper bound can be estimated by thefollowing inequality [40][41] using the margin-threshold 𝛾 and the weight matrix 𝑤, whereℎ𝑤 is the hypothesis derived from 𝑤. This inequality holds with probability (1 − 𝛿).
2𝐿(ℎ𝑤 ) ≤ 𝐿̂𝑆,𝛾 (ℎ𝑤 ) +√(42 ∑𝑑𝑖=1 (2√𝜔𝑖 + √2 ln(2𝑑))) ∏𝑑𝑖=1‖𝑤𝑖 ‖22 × ∑𝑑𝑖=1‖𝑤𝑖 ‖2𝐹𝑚+ ln ( )𝛿‖𝑤𝑖 ‖22𝛾 2𝑚where 𝑑 is the number of layers, 𝑤𝑖 is the weight matrix to the 𝑖-th layer, and 𝜔𝑖 is thenumber of elements in 𝑤𝑖 . Note that ‖𝑤‖2 is the spectral norm of matrix 𝑤, and ‖𝑤‖𝐹 is the
Frobenius norm of 𝑤.
The second term on the right-hand side of the inequality shows that the generalization gap(the difference between the generalization error and the empirical error) decreases as themargin threshold 𝛾 is increased. On the other hand, the first term 𝐿̂𝑆,𝛾 (ℎ𝑤 ) of the right-handside increases with 𝛾. They mean that the generalization errors of hypotheses that have larger
output margins become smaller, which is consistent with the empirical results. In general, since
the margin-based bounds also often exceed 100%, it is difficult to apply the bounds to absoluteevaluation of generalization performance, but the threshold 𝛾 is useful as a generalization
measure [40] for relative evaluation of generalization performance.
Sensitivity-based bounds
The sensitivity-based bound is a special case (for the Gaussian noise on weights) of the PAC-

31

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Bayesian bound. In the PAC-Bayesian bounds, the amount of information that a hypothesis has
obtained from the training dataset is estimated from the difference (called the KL-divergence)
between the two (prior and posterior) probability distributions of parameters before and after
training. Therefore, since the KL-divergence implies the dependency of the trained hypothesis
to the training dataset.
In the sensitivity based bounds, when the Gaussian noise 𝒩(0, 𝜎 2 ), whose mean is zero andstandard deviation is 𝜎, is added to each weight-parameter in 𝑤 after training (𝜔: the numberof parameters), where each parameter is randomly initialized by the Gaussian noise 𝒩(0, 𝜎 2 )before training, the upper bound of the expected value of the generalization error 𝐿(ℎ𝑤+𝑢 ) withnoise 𝑢 can be estimated by the following inequality [40][42] using the expected value of
empirical (training) error 𝐿̂𝑆 (ℎ𝑤+𝑢 ) with noise 𝑢 and the standard deviation 𝜎 of noise. Thisinequality holds with probability (1 − 𝛿).
1 ‖𝑤‖222𝑚
𝔼𝑢~𝒩(0,𝜎2)𝜔 [𝐿(ℎ𝑤+𝑢 )] ≤ 𝔼𝑢~𝒩(0,𝜎2 )𝜔 [𝐿̂𝑆 (ℎ𝑤+𝑢 )] + 4√ (+ ln
) ⋯ ⋯ (∗1 )2
𝑚 2𝜎
𝛿where 𝜔 is the number of weigh-parameters.
The first term on the RHS (right-hand side) of this inequality can be approximated as theaverage of several measurements of the empirical error 𝐿̂𝑆 (ℎ𝑤+𝑢 ) under the Gaussian noise 𝑢,
as shown in Figure 6.2.

Figure 6.2 The measurement of the empirical error with the Gaussian noise
The second term on the right-hand side of the inequality shows that the generalization gap
decreases as the standard deviation 𝜎 of the noise is increased. On the other hand, the first termon the RHS increases with the noise σ. They mean that the generalization errors of hypotheses
that are more insensitive to noise become smaller, which is consistent with the empirical results.
The sensitivity-based bounds also often exceed 100%. Figure 6.3 shows an example ofexpected generalization error upper bounds (i.e., the RHS of the inequality (∗1 )) estimated by
the sensitivity-based method for a simple feedforward neural network, where 𝛿 = 0.1. The
neural network consists of fully connected three layers, where the input layer has 28 × 28
neurons, the hidden layer has 64 neurons with ReLU, and the output layer has 10 neurons with
Softmax, and it was trained on the MNIST (brightness: [0, 1] for each pixel) dataset, whose size
is 54,000. As shown in Figure 6.3, the estimated upper bounds exceed 100%, for example, the32

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

expected generalization error upper bound (i.e., the RHS of (∗1 )) is 323% when 𝜎 = 0.05.
Therefore, it is difficult to directly apply the bounds to absolute evaluation of generalization
performance. However, similarly to the margin threshold 𝛾 explained in Subsection 6.2.4, thestandard deviation 𝜎 which increases the expected training error with noise to a constant value𝑒𝑇 (e.g., if 𝑒𝑇 = 10% then 𝜎 = 0.05 in Figure 6.3), is also useful as a generalization measure[40] for relative evaluation of generalization performance.

Figure 6.3 An example of expected generalization errors estimated by the sensitivity-based method
Compression-based bounds
In the compression-based bounds, for a hypothesis ℎ trained by using a training dataset 𝑆,
it is necessary to divide 𝑆 into a dependent subset 𝑇ℎ and an independent subset 𝑉ℎ (𝑆 =𝑇ℎ ∪ 𝑉ℎ ) such that the following two conditions are satisfied:–

The hypothesis ℎ depends only on the dependent subset 𝑇ℎ–

The empirical error of ℎ for the independent subset 𝑉ℎ (i.e., 𝐿̂𝑉ℎ (ℎ) = 0)

For the hypothesis ℎ, if the training data set 𝑆 (size: 𝑚) can be divided into a dependent subset
𝑇ℎ (size: 𝑘) and an independent subset 𝑉ℎ (size: 𝑚 − 𝑘), then the generalization error upperbound can be estimated by the following inequality [40] using the size 𝑘 of the subset 𝑇ℎ . Thisinequality holds with probability (1 − 𝛿).
𝐿(ℎ) ≤

8𝑘 𝑚
ln
𝑚 𝛿

The right-hand side shows that the generalization error upper bound becomes small by
reducing the dependence (𝑘/𝑚) of the hypothesis ℎ on the dataset 𝑆. The methods (called
compression schemes) for dividing a dataset into a dependent subset and an independent subset
are necessary for the application of this inequality and have been being developed. For example,
Brutzkus et al. [43] developed a compression scheme for two-layer neural networks trained withstochastic gradient descent method. More generic compression schemes are still in the researchphase.

33

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Stability-based bounds
In the stability-based bounds, the generalization error upper bounds are estimated based on
the stability of stochastic algorithms. The stability of a stochastic algorithm 𝒜 is defined by thesupremum 𝜖 on the expected value of the incremental loss 𝑙(𝒜(𝑆−{𝑧})∪{𝑧 ′ } , 𝑧) − 𝑙(𝒜𝑆 , 𝑧) when
any one data sample 𝑧 in the training dataset 𝑆 is replaced by another sample 𝑧 ′, where 𝒜𝑆
is the hypothesis obtained by training dataset 𝑆 with algorithm 𝒜. The small supremum 𝜖means that the stochastic algorithm is less dependent on the training dataset S, in other words,
it is stable.
Based on the stability of the algorithm 𝒜 for the dataset 𝑆 (size: 𝑚), for the hypothesis
𝒜𝑆𝑘 where 𝑆𝑘 is the subset of 𝑘 samples (𝑘 ≤ 𝑚) randomly selected from the dataset 𝑆, theupper bound of the difference between the generalization error and the empirical error can be
estimated by the following inequality [44].
4𝔼(𝑆~𝒟 𝑚 , 𝒜) [𝐿(𝒜𝑆𝑘 ) − 𝐿̂𝑆 (𝒜𝑆𝑘 )] ≤ 𝒪 (√𝑐(𝐿(ℎ𝑤1 ) − 𝐿∗ ) ∙

√𝑘
√𝑘
+ 𝑐𝜎
)
𝑚
𝑚

where 𝑐 is a constant related to the step size in the stochastic algorithm, ℎ𝑤1 is the initial
hypothesis derived from the initial weight matrix 𝑤1 , 𝐿∗ is the minimum error achievable in
the hypothesis set ℋ, and 𝜎 is the standard deviation of the stochastic gradient during training.
Although the generalization error 𝐿(ℎ𝑤1 ) of the initial hypothesis ℎ𝑤1 cannot be exactly
calculated, it can be approximated by its empirical error 𝐿̂𝑆 (ℎ𝑤1 ) because the initial weightmatrix is generally random.
In the paper [44], upper bounds close to the generalization errors are estimated based onstability, but the stability-based bounds require some conditions such that the loss function mustbe smooth, and each data sample must be used once at most (one-pass). The research forweakening the conditions (e.g., non-smooth loss function and multi-pass) has been ongoing.
Marginal likelihood-based bounds
Similarly to the sensitivity-based bounds explained in Subsection 6.2.5, the marginal
likelihood-based bound is also a special case of the PAC-Bayesian bound, but it uses the KLdivergence between prior- and posterior-distributions of hypotheses instead of weight matrix.
The hypothesis ℎ𝑤 ∈ ℋ is a function derived from a weight matrix 𝑤 ∈ 𝒲, and in general,
several weight matrixes 𝑤, 𝑤 ′ are mapped into the same hypothesis ℎ𝑤 = ℎ𝑤′ . Therefore, theKL-divergence between the prior- and posterior-distributions is less on the hypothesis set ℋ
than on the weight matrix set 𝒲. It means that the marginal likelihood-based bounds are tighterto the generalization errors than the sensitivity-based bounds. However, the marginal likelihoodbased bounds require the following two conditions.
–

The training error of ℎ for the training dataset 𝑆 must be zero (i.e., 𝐿̂𝑆 (ℎ) = 0).

–

The stochastic training algorithm must almost uniformly sample the zero-error region.

34

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

If the two conditions are satisfied, then the following equation holds for any zero-error
hypothesis ℎ ∈ ℋ0 (𝑆) = {ℎ ∈ ℋ | 𝐿𝑆 (ℎ) = 0},
𝑄(ℎ) =𝑃(ℎ),
𝑀0 (𝑆)

𝑤ℎ𝑒𝑟𝑒 𝑀0 (𝑆) =∑ 𝑃(ℎ)ℎ∈ℋ0 (𝑆)where 𝑃(ℎ) and 𝑄(ℎ) are the prior distributions (before training) and the posterior
distributions (after training) of hypotheses, respectively, and 𝑀0 (𝑆) is the probability of
selecting a hypothesis with zero training error for 𝑆 in the prior distribution 𝑃(ℎ), whichcorresponds to the marginal likelihood of 𝑆. It has been shown that the stochastic gradientdescent method (SGD) tends to satisfy the conditions [45].
When the above conditions are satisfied, the generalization error upper bound can be
estimated by the following inequality [37] for the hypothesis ℎ such that 𝑄(ℎ) ≥ 1 − 𝛾.
1

𝐿(ℎ) < 1 − (

−
𝑚
𝑚−1
)
𝑀0(𝑆) 𝛾 𝛿It has been shown [45] that simplicity bias is imposed when weight matrixes are mapped
into hypotheses in neural networks. Consequently, even if weight matrixes are initialized withuniform random numbers, the distribution 𝑃(ℎ) of the hypotheses derived from the matrixes
is not uniform because simpler hypotheses (with lower Kolmogorov complexity) tend to have ahigher probability. In general, it is expected that real-world hypotheses (rather than, for example,
artificial random functions) have a kind of regularity and/or structure. Therefore, the probability
𝑃(ℎ) of a hypothesis ℎ for the real world will be high, which means that 𝑀0 (𝑆) and 𝛾 can belarge, and then implies that the upper bounds can become small.
Valle-Pérez and Louis [37] proved some optimality of the marginal likelihood-based bounds
and performed experiments on the estimation of the upper bound by the two-class classification
problems but have not shown sufficiently comparisons with the other methods yet. The researchon the marginal likelihood-based bounds (e.g., multi-class classification and computation of
marginal likelihood) has been ongoing.

6.3 Conclusion
It is still difficult for absolute evaluation of the generalization performance of hypotheses toapply the methods for estimating the generalization error upper bounds introduced in thischapter because most of the estimation result exceed 100% (i.e., vacuous). Most research papers
do not concern about the absolute values of bounds because they aim at theoretically explaining
the empirical phenomenon (e.g., generalization performance can be obtained even in overparameterized network) of deep neural network. Even though the estimation results of thegeneralization error upper bounds exceed 100%, it is still useful as a relative generalization
measure for comparing the generalization performance of hypotheses, and such generalization
measures have been theoretically and empirically evaluated [40][46]. On the other hand, somerecent papers have been presented the methods for estimating the non-vacuous upper bound35

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003(i.e., less than 100%) of generalization errors and the upper bounds have been becoming close
to the generalization errors. For example, Zhou et al. [47] formalized a generalization bound forcompressed trained neural network and provided a non-vacuous generalization guarantee forthe realistic dataset ImageNet. And Garg et al. [48] proposed a method, named RATT, thatleverages unlabeled data to estimate generalization bounds and their several experiments
showed that the bounds are non-vacuous. We think that the generalization error upper boundcan be one of measures of generalization performance in a few years.

36Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

7

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Adversarial Example Detection

7.1 Research summary
With the goal of practically establishing a method for determining whether a given input
image is an adversarial example, we focus on the following points regarding attacks anddetection methods that generate adversarial examples. We are conducting a survey of typicaltechnologies.
–

Supporting adversarial example detection program code and confirmation bycomputational experiment–

Reproduction of experimental results of adversarial example detection method papers

Adversarial example detection stands for detecting adversarial examples from given inputs,
and existing state-of-the-art adversarial example detection methods can be divided into fourmain categories.
① Metric based approaches (example [49])② Denoisers approaches (example [50])③ Prediction inconsistency based approaches (example [51])④ Neural Network Invariant Checking approaches (example [52])In this chapter, we report the results of additional test experiments to compare and evaluate
adversarial example detection methods based on each of these approaches ① to ④ .

Asreported in the paper [52], it was confirmed that the approach of ④ (NIC: Neural Network
Invariant Checking) shows the highest detection rate among ① to ④ .

In this follow-up

experiment, the published implementation code was used for ① to ③, but the implementation
code was not published for ④, so a computer experiment was conducted by implementing a
NIC according to the paper [52]. Therefore, this chapter mainly describes NIC ④.
After explaining the outline of the four approaches, the method of detecting adversarial
examples by the NIC is explained, and the implementation method is described. Finally, theresults of the follow-up experiments of each approach and the experiments by NIC are described.

7.2 Overview of adversarial example detection approaches
In this section, the four state-of-the-art approaches to adversarial example detection are
overviewed.
Metric based approaches
A method of performing statistical measurements of inputs (and outputs of each neuron) to37Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

detect adversarial examples, Ma et al. recently proposed the use of a measurement called Local
Intrinsic Dimensionality (LID) [49]. This method estimates the LID value that evaluates thespace-filling capacity of the area surrounding the sample by calculating the distance distributionof the sample and the number of neighbors in each layer, and the adversarial example tends tohave a large LID value. It uses certain properties to detect adversarial examples. LID is superior
to traditional kernel density estimation (KD) and Bayesian uncertainty (BU) for detecting
adversarial examples and is currently the state-of-the-art technology for this type of detector.
Denoisers approaches
It is a method of detecting adversarial examples by removing noise in a preprocessing stepfor each input. In this method, the training model or noise remover (encoder and decoder) is
trained to filter the image so that the key components in the training model can be highlighted.
This filter can be used to remove noise added by an attacker to generate adversarial examples
and correct misclassification. MagNet [50] is a method of detecting adversarial examples using
detectors and reformers (trained automatic encoders and automatic decoders).
Prediction inconsistency based approach
A method of detecting adversarial examples by measuring the discrepancy between theoriginal neural network and the neural network enhanced by human perceptible attributes.
Feature Squeezing [51], the state-of-the-art detection technique of this method, can achieve very
high detection rates against a variety of attacks. Feature squeezing focuses on detecting gradientbased attacks, focusing on the ability of attackers to generate adversarial examples through theunnecessarily large input feature space of deep neural networks DNN. The procedure fordetecting adversarial examples by feature squeezing is shown below.
1.

Apply squeezing technology (a technology that reduces the color depth of an image andsmooths the image) to the original input image to generate multiple squeezed images.

2.

Input the original input image and multiple squeeze images into the deep neuralnetwork, and measure the distance between the inference result (prediction vector) of
the input image and the inference result of each squeeze image.

3.

When one of the differences (distances) between the original input image and thesqueeze image exceeds the threshold value, the original input image is detected as anadversarial example.
Neural Network Invariant Checking approaches

A method of detecting adversarial examples by measuring the discrepancy between theoriginal neural network and the neural network enhanced by human perceptible attributes.
Feature Squeezing [51], the state-of-the-art detection technique of this method, can achieve very

38

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

high detection rates against a variety of attacks. Feature squeezing focuses on value invariants(VIs) and provenance invariants (PIs) inside neural networks in deep neural network NICs [52].
The value invariant VI is the distribution of possible neuron values in each layer, and the history
invariant PI is the possible neuron value pattern of two consecutive layers (summary ofcorrelation between features across two layers). If an input violates these invariants, the input
is detected as an adversarial example. Train these invariant VIs and PIs with benign input data
and model them as a one-class classification (OCC) problem that detects adversarial examples.
A higher detection rate has been reported than the methods based on (1) to (3) explained above[52]. The outline and the implementation of the NIC system design are explained in detail in
Sections 7.3 and 7.4, respectively.

7.3 NIC system design overview
The procedure for building and detecting the NIC detector (steps A to C: during training, D to
E: during execution) is explained using Figure 7.1 [52]. This invariant VI, PI training uses only
non-adversarial benign data.

Figure 7.1 Outline of system design (Fig. 8 of thesis [52])–Step A: Collect the output value of each neuron at each layer of each training data input.

–

Step B: For each layer 𝑘 (e.g., L1, L2), extract the submodels from the input layer tothe 𝑘 layer and add a new softmax layer with the same output label as the originalmodel. Then create a derived model (DerivedModel in Figure 7.1)

39

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition–

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Step C: Enter each benign training data for all derived models and collect the final
output of these models (i.e., the output probability values of the individual classes). For
each set of consecutive layers, we train using the distribution of the classification
results of this derivative model. This trained distribution is the PI for these two layers.

–

Step D: Input each test data 𝑡 (for example, the image of “4” in Figure 7.1) to allderivative models in addition to the original model, and observe the activation value ofeach layer of the original model. Collect the value OV (for example, OV(L1, 𝑡) in Figure7.1) and the classification result (set) of the derivative model of consecutive layers.
From this classification result, the observed source OP (for example, OP(L1, L2, 𝑡), etc.)
is obtained.

–

Step E: Calculate the probability D that the OV and OP fit the corresponding VI and PI
distributions. The possibility that the input 𝑡 is adversarial is predicted at the sametime by aggregating all these D values.

7.4 NIC system implementation
In order to detect adversarial examples based on NIC, a direct sum space (vector) isconstructed from PI and VI, and an OSVM (One Class Support Vector Machine) for classifying thisvector is constructed. When the input to the layer 𝑙 of the trained DNN (Deep Neural Network)
model (hereinafter referred to as M) is 𝑥𝑙 , the output 𝑓𝑙 of the layer 𝑙 is given by the following
equation.
𝑓𝑙 = 𝜎(𝑥𝑙 ∙ 𝑤𝑙𝑇 + 𝑏𝑙 )Where 𝜎 is the activation function of the layer 𝑙, 𝑤𝑙𝑇 is the weight matrix, and 𝑏𝑙 is thebias. At this time, the direct sum spaces classified by VI, PI, and OSVM are obtained as follows.
–

VI calculation: The VI of each layer 𝑙 of model M is determined by solving the following
optimization problem.
𝑉𝐼𝑙 = min [ ∑ 𝐽(𝑓𝑙 ∘ 𝑓𝑙−1 ∘ ⋯ ∘ 𝑓1 (𝑥) ⋯ 𝑤 𝑇 − 1)]𝑥∈𝑋𝑏

Here, 𝐽 is the error evaluation function, and 𝑋𝑏 is the batch used to create M. Also, ∘is a monoid, in this case a vectorized version of 𝑓𝑘 .
–PI calculation: 𝑃𝐼𝑙,𝑙+1 (𝑥) is based on the classification output of the derived models of
the layers 𝑙 and 𝑙 + 1 . The probability that 𝑥 is benign (non-adversarial) is
estimated by solving the following optimization problem.
𝑃𝐼𝑙,𝑙+1 (𝑥) = min [ ∑ 𝐽(𝑐𝑜𝑛𝑐𝑎𝑡(𝐷𝑙 (𝑥), 𝐷𝑙+1 (𝑥)) ⋯ 𝑤 𝑇 − 1)]
𝑥∈𝑋𝑏

40

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Here, a derivative model 𝐷𝑙 of the layer 𝑙 is defined as follows, with the softmax layer
added after the layer 𝑙.
𝐷𝑙 = softmax ∘ 𝑓𝑙 ∘ 𝑓𝑙−1 ∘ ⋯ ∘ 𝑓1–

Direct sum space of PI and VI: From the VI and PI obtained by the above optimization,
the following direct sum space (vector) is created for each batch of training data ofmodel M.
𝑉𝐼1 ⨁ 𝑃𝐼1,2 ⨁ 𝑉𝐼2 ⨁ 𝑃𝐼2,3 ⋯ 𝑉𝐼𝐵 ⨁ 𝑃𝐼𝐵−1,𝐵 ⨁ 𝑉𝐼𝐵
This vector is 𝐿 × 3 dimensions (𝐿 is the number of layers of M), which is the vector
space (direct sum space) of the number 𝐵. The NIC performs OSVM on this space.

7.5 Computer experiment
In order to confirm the effect of adversarial example detection technology (NIC), theexperiment of the paper [52] was retested in the following experimental environment.
–

Hardware environment: AIST ABCI [53]–Datasets: Two common image datasets, MNIST [54] and CIFAR-10 [55], were used forimage classification experiments. MNIST is a grayscale image dataset used for
handwritten digit recognition, and CIFAR-10 is a color image dataset used for object
recognition. For NIC, we also conducted an experiment on LFW (face image) [56].

–Attacks: Non-targeted attacks (FGSM 𝐿2 ,𝐿∞ ), targeted attacks JSMA, and gradientbased attacks (CW 𝐿2) were used to generate adversarial examples. The Cleverhans
library [57] was used to implement FGSM and JSMAFirst, in order to evaluate the adversarial example detection method based on each of theapproaches ① to ③, the published implementation code of LID [49], MagNet [50], and featuresqueezing [51] was used to evaluate each paper. A follow-up experiment was conducted. As aresult, the detection rates reported in each paper could be confirmed, and among these three,feature squeezing showed the highest detection rate.
Next, in order to evaluate the adversarial example detection method based on the approach④, an experiment was conducted using the NIC code implemented in Section 7.4. Table 7.1 toTable 7.3 show the results of adversarial example detection and computational experiments on
the MNIST, CIFAR-10, and LFW datasets, respectively. Here, the correct answer rate is the rate at
which adversarial examples are input to the classifier (OSVM) described in Section 7.4 and aredetermined to be adversarial examples. The CNN model used in the experiment is LeNet5, andthe OSVM Kernel is RBF (MNIST: γ = 0.1 to 0.27, CIFAR-10: γ = 0.11 to 0.2, LFW: γ = 0.005 to0.90). In the results of this experiment, high detection performance was confirmed not only forthe dataset and attack method reported in the paper [52], but also for the unreported dataset
LFW and attack method (FGSM 𝐿∞ ).

41

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Table 7.1 Adversarial example detection computational experiment results for MNIST dataset
Data Set
MNIST

AttackInvariant

FGSM 𝐿2

FGSM 𝐿∞

JSMA

CW2

TrojanPerformance

NumberPerformance reportedof datain the paper [52]

VI

97%

2800

PI

98%

84%

NIC

97%

100%

VI

98%

PI

98%

―

NIC

98%

―

VI

100%

PI

100%

100%

NIC

100%

100%

VI

100%

PI

100%

96%

NIC

100%

100%

VI

100%

PI

100%

100%

NIC

100%

100%

2800

280

280

3200

100%

―

83%

95%

100%

Table 7.2 Adversarial example detection computational experimental results for CIFAR-10 dataset
Data Set

AttackInvariant

CIFAR-10

FGSM 𝐿

2

FGSM 𝐿∞

JSMA

CW2

TrojanPerformance

NumberPerformance reportedof datain the paper [52]

VI

99%

PI

99%

52%

NIC

99%

100%

VI

100%

PI

100%

―

NIC

100%

―

VI

97%

PI

95%

100%

NIC

96%

100%

VI

98%

PI

95%

89%

NIC

96%

100%

VI

100%

PI

100%

100%

NIC

100%

100%

42

6400

6400

320

320

3200

100%

―

62%

88%

100%

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Table 7.3 Adversarial example detection computational experiment results for LFW dataset
Data Set
LFW

Attack
FGSM 𝐿2

FGSM 𝐿∞

JSMA

CW2

TrojanInvariant

Performance

NumberPerformance reportedof datain the paper [52]

VI

98%

28222

PI

98%

―

NIC

98%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

2822

280

840

3200

43

―

―

―

―

―

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

8

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

AI Quality Management in Operation
In this chapter, we report on the results of a survey on the latest technologies for detectingchanges in data distribution over time, called concept drift, and adapting machine learningmodels to the changed distribution as a technique for AI quality management during operation.
Concept drift is one of the main causes of performance degradation of machine learningmodels running in AI systems during operation. In order to maintain quality that is satisfied at
the beginning of the operation of the system throughout the operation period, it is necessary tocontinuously monitor whether drift occurs or not. In addition, if necessary, we retrain themachine learning models in the system with the latest data to adapt them to the distribution ofdata changed after the drift occurs. As the use of machine learning technologies has been
expanded in recent years, AI systems operating with such technologies will require processing alarge amount of data without their true labels (ground truths) in a short period of time, including
types of data that have not been handled in the past.
In the fiscal year 2019-2020, we conducted a survey on the latest technologies for detecting
and adapting to the concept drift to maintain the performance of machine learning models
during operation. As a result of this survey, we found that most of the methods developed so far
are supervised methods that use true labels of data additionally acquired during operation forthe detection and adaptation. However, such true labels are not always available or are often
costly even if they are available. In order to expand the applicability of the detection andadaptation methods and reduce their operational costs, we found that an "unsupervised
method" that does not use the true labels or a "semi-supervised method" that uses only a limited
number of the true labels is promising. We summarized the results of the surveys organized anddiscussed from this perspective.
For details on the survey on detection methods, see Section 7.8 of the Machine Learning
Quality Management Guidelines [1]. In addition, adaptation methods are summarized in our
survey result [58]. Table 8.1 shows the comparison of our survey with the other existing surveys
on concept drift detection and adaptation methods. Gama et al. summarized their survey resultin [59] and Lu et al. added recently published drift detection and adaptation methods in [60].
Those survey papers mainly focus on introducing "supervised" methods that use true labels of
operational data for drift detection and adaptation. On the other hand, Ishida et al. introduced"unsupervised" concept drift detection methods that do not use true labels of data for drift
detection in [61]. In comparison with those existing survey results, we introduced"unsupervised" and "semi-supervised" concept drift adaptation methods that do not use or use
only a limited number of true labels as mentioned above. Furthermore, we introduced those drift
adaptation methods based on the characteristic of each method. In detail, we listed ten
remarkable unsupervised/semi-supervised drift adaptation methods and classified themaccording to: i) types of drift that can be dealt with effectively, ii) processes where true labels ofdata are required during operation and the percentage of the labeled data used in verifications
shown in the papers, and iii) machine learning models or clustering methods used in each44

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

method. Finally, we closed our survey by discussing further development of unsupervised andsemi-supervised concept drift adaptation methods using knowledge obtained from relevant
unsupervised domain adaptation techniques.
Table 8.1 Comparison of survey papers on concept drift detection and adaptation
Detection
Supervised
Unsupervised /Semi-supervised

Adaptation

Gama et al.[59], Lu et al.[60]Ishida et al.[61]45

Okawa and Kobayashi [58](Ours)

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

9

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

References

Chapter 1:[1] National Institute of Advanced Industrial Science and Technology (AIST), Machine Learning
Quality Management Guideline, Digital Architecture Research Center, Cyber Physical
Security Research Center, Artificial Intelligence Research Center, Technical Report DigiARCTR-2022-01/ CPSEC-TR-2022002 https://www.digiarc.aist.go.jp/publication/aiqm/[2] Tomoumi Takase, Dynamic batch size tuning based on stopping criterion for neuralnetwork training, Neurocomputing, Volume 429, pp.1-11, 2021.
[3] Shin Nakajima, Software Testing with Statistical Partial Oracles, 10th SOFL+MSVL, 2021.
Chapter 2:[4] Satoshi Hara, My Bookmark : Interpretability in Machine Learning, Journal of JapaneseSociety for Artificial Intelligence, vol. 33, no. 3, pp. 366-369, 2018 (in Japanese).
[5] Fred Hohman, Minsuk Kahng, Robert Pienta, Duen Horng Chau, Visual Analytics in Deep
Learning: An Interrogative Survey for the Next Frontiers, IEEE Transactions on
Visualization and Computer Graphics, vol. 25, no. 8, pp. 2674-2693, 2018.
[6] Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, Liu Ren, Do Convolutional Neural
Networks Learn Class Hierarchy?, IEEE Transactions on Visualization and Computer
Graphics, vol. 24, no. 1, pp. 152-162, 2018.
[7] Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, Shixia Liu, Analyzing the Training Processes of
Deep Generative Models, IEEE Transactions on Visualization and Computer Graphics, vol.24,
no.1, pp.77-87, 2018.
[8] Jorge Piazentin Ono, Sonia Castelo, Roque Lopez, Enrico Bertini, Juliana Freire, Claudio Silva,
PipelineProfiler: A Visual Analytics Tool for the Exploration of AutoML Pipelines, IEEE
Transactions on Visualization and Computer Graphics, vol.27, no.2, pp.390-400, 2021.
[9] Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza, Power to the People: The
Role of Humans in Interactive Machine Learning. AI Magazine, vol.35, no.4, pp.105-120,
2014.
Chapter 3:[10] Gontijo-Lopes, R., Smullin, S. J., Cubuk, E. D.,

and Dyer, E., Affinity and Diversity:

Quantifying Mechanisms of Data Augmentation. arXiv preprint arXiv:2002.08973, 2020.
[11] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q., RandAugment: Practical Automated Data
Augmentation with a Reduced Search Space. In Neural Information Processing Systems, 33,
2020.
[12] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D., Mixup: Beyond Empirical Risk

46

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

Minimization. In International Conference on Learning Representations, 2018.
[13] Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y.,
Manifold Mixup: Better Representations by Interpolating Hidden States. In International
Conference on Machine Learning, pp. 6438–6447, PMLR, 2019.
[14] Kim, J-H., Choo, W., and Song, H. O., Puzzle mix: Exploiting saliency and local statistics for
optimal mixup. In International Conference on Machine Learning, 2020.
[15] Beckham, C., Honari, S., Verma, V., Lamb, A., Ghadiri, F., Hjelm, R. D., Bengio, Y., and Pal, C. On
adversarial mixup resynthesis. In Neural Information Processing Systems, 2019.
Chapter 4:[16] Nakajima, S., Quality Issues in Machine Learning from Software Engineering Viewpoints,
Maruzen Publisher, 2020. (in Japanese)[17] Pei, K., et al., DeepXplore: Automated Whitebox Testing of Deep Learning Systems, In Proc.
26th SOSP, 2017, pp.1-18.
[18] Nakajima, S., Distortion and Faults in Machine Learning Software, In Post-Proc. 9th
SOFL+MSVL, 2020, pp.29-41.
[19] Ma, L., et al., DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems, In
Proc. ASE, 2018, pp.120-131.
[20] Tian, Y., et al., DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous
Cars, In Proc. 40th ICSE, 2018, pp.303-314.
[21] Zhang, M., et al., DeepRoad: GAN-Based Metamorphic Testing and Input Validation
Framework for Autonomous Driving Systems, In Proc. ASE, 2018, pp.132-142.
[22] Zhang, P, et al., CAGFuzz: Coverage-Guided Adversarial Generative Fussing Testing of Deep
Learning Systems, arXiv:1911.07931, 2019.
[23] Harel-Canada, F., et al., Is Neuron Coverage a Meaningful Measure for Testing Deep Neural
Networks? In ESEC/FSE, 2020, pp.851-862.
[24] Kim, J. et al., Guiding Deep Learning System Testing Using Surprise Adequacy, In Proc. 41st
ICSE, 2019, pp.1039-1049.
Chapter 5:[25] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian
Goodfellow, and Rob Fergus, Intriguing properties of neural networks, The International
Conference on Learning Representations (ICLR 2014), pp.1-10, 2014.
https://arxiv.org/abs/1312.6199[26] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer, Reluplex: An
Efficient SMT Solver for Verifying Deep Neural Networks, International Conference on
Computer-Aided Verification (CAV), 2017. https://arxiv.org/abs/1702.01135[27] Vincent Tjeng, Kai Xiao, and Russ Tedrake, Evaluating robustness of neural networks with
mixed integer programming, International Conference on Learning Representations (ICLR),

47

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003

2019. https://arxiv.org/abs/1711.07356[28] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit
S. Dhillon, and Luca Daniel, Towards Fast Computation of Certified Robustness for ReLU
Networks, International Conference on Machine Learning, PMLR 80, pp.5276-5285, 2018.
https://arxiv.org/abs/1804.09699[29] Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel, CNN-Cert: An
Efficient Framework for Certifying Robustness of Convolutional Neural Networks, The
Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019), pp.3240-3247, 2019.
https://arxiv.org/abs/1811.12395[30] Tsui-Wei Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan
Oseledets, and Luca Daniel, PROVEN: Verifying Robustness of Neural Networks with a
Probabilistic Approach, International Conference on Machine Learning (ICML 2019), PMLR
vol. 97, pp.6727-6736, 2019. http://proceedings.mlr.press/v97/weng19a.html[31] Nicholas Carlini and David Wagner, Towards Evaluating the Robustness of Neural Networks,
IEEE Symposium on Security and Privacy (SP), pp.39-57, 2017.
https://arxiv.org/abs/1608.04644[32] Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh,and Luca Daniel, Evaluating the Robustness of Neural Networks: An Extreme Value Theory
Approach, International Conference on Learning Representations (ICLR 2018), 2018.
https://arxiv.org/abs/1801.10578[33] Eric Wong and J. Zico Kolter, Provable defenses against adversarial examples via the convex
outer adversarial polytope, International Conference on Machine Learning (ICML 2018),
PMLR vol. 80, pp.5283-5292, 2018. https://arxiv.org/abs/1711.00851[34] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana,
Certified Robustness to Adversarial Examples with Differential Privacy, The IEEE
Symposium on Security and Privacy (SP), 2019. https://arxiv.org/abs/1802.03471[35] Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter, Certified Adversarial Robustness via
Randomized Smoothing, The 36th International Conference on Machine Learning (ICML2019), 2019. https://arxiv.org/abs/1902.02918[36] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian
Vladu, Towards Deep Learning Models Resistant to Adversarial Attacks, The Sixth
International Conference on Learning Representations (ICLR 2018), 2018.
https://arxiv.org/abs/1706.06083
Chapter 6:[37] Guillermo Valle-Pérez and Ard A. Louis, Generalization bounds for deep learning,
arXiv:2012.04115v2, 2020. https://arxiv.org/abs/2012.04115[38] Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory
to Algorithms, Cambridge University Press, 2014.
https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/index.html

48Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003[39] Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian, Nearly-tight VCdimension and Pseudodimension Bounds for Piecewise Linear Neural Networks, Journal of
Machine Learning Research, Vol.20, No.63, pp.1−17, 2019.
https://jmlr.org/papers/v20/17-612.html[40] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio,
Fantastic Generalization Measures and Where to Find Them, International Conference on
Learning Representations (ICLR 2020). https://arxiv.org/abs/1912.02178[41] Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst, PAC-Bayesian Margin Bounds
for Convolutional Neural Networks, arXiv:1801.00171, 2018.
https://arxiv.org/abs/1801.00171[42] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro, Exploring
Generalization in Deep Learning, NeurIPS 2017. arXiv:1706.08947.
https://arxiv.org/abs/1706.08947[43] Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz, SGD Learns Overparameterized Networks that Provably Generalize on Linearly Separable Data, ICLR 2018.
https://openreview.net/forum?id=rJ33wwxRb[44] Ilja Kuzborskij and Christoph H. Lampert, Data-Dependent Stability of Stochastic Gradient
Descent, 2017. arXiv:1703.01678. https://arxiv.org/abs/1703.01678[45] Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis, Deep learning generalizes
because the parameter-function map is biased towards simple functions, ICLR 2019.
arXiv:1805.08522. https://arxiv.org/abs/1805.08522[46] Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan
Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M. Roy, In search of robust measuresof generalization, NeurIPS 2020. arXiv:2010.11924. https://arxiv.org/abs/2010.11924[47] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz,
Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression
Approach, ICLR 2019. https://arxiv.org/abs/1804.05862[48] Saurabh Garg, Sivaraman Balakrishnan, J. Zico Kolter, and Zachary C. Lipton, RATT:
Leveraging Unlabeled Data to Guarantee Generalization, ICML 2021.
arXiv:2105.00303. https://arxiv.org/abs/2105.00303
Chapter 7:[49] X. Ma, Characterizing adversarial subspaces using Local Intrinsic Dimensionality, 2018.
[50] D. Meng , Magnet: a two-pronged defense against adversarial examples, in Proceedings ofthe 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017.
[51] W. Xu, Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks, inProceedings of the 2018 Network and Distributed Systems Security Symposium (NDSS),
2018.
[52] Shiqing Ma, NIC: Detecting Adversarial Samples with Neural Network Invariant Checking,
Network and Distributed Systems Security Symposium (NDSS), NDSS 2019.

49

Technical Report on Machine Learning
Quality Evaluation and Improvement
1st English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2022-02 / CPSEC-TR-2022003[53] National Institute of Advanced Industrial Science and Technology (AIST), AI Bridging Cloud
Infrastructure, https://abci.ai/ja/[54] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document
recognition, Proceedings of the IEEE, vol. 86, no. 11, pp.2278–2324, 1998. [Online].
Available: http://yann.lecun.com/exdb/mnist/[55] A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, 2009.
[56] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, Labeled faces in the wild: A databasefor studying face recognition in unconstrained environments, University of Massachusetts,
Amherst, Tech. Rep. 07-49, October 2007.
[57] Nicolas Papernot, Ian Goodfellow, Ryan Sheatsley, Reuben Feinman, and Patrick McDaniel.
cleverhans v1.0.0: an adversarial machine learning library. arXiv preprint arXiv:1610.00768,
2016.
Chapter 8:[58] Yoshihiro Okawa and Kenichi Kobayashi, A Survey on Concept Drift Adaptation
Technologies for Unlabeled Data in Operation, Proceedings of the 35th Annual Conference of
the Japanese Society for Artificial Intelligence, pp.1-4, 2021 (in Japanese),https://doi.org/10.11517/pjsai.JSAI2021.0_2G4GS2f03.
[59] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia,
A survey on concept drift adaptation, ACM Computer Surveys, vol. 46, no. 4, pp.1-37, 2014.
[60] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan ZhangJ, Learning under
Concept Drift: A Review, in IEEE Transactions on Knowledge and Data Engineering, vol. 31,
no. 12, pp. 2346-2363, 2019.
[61] Tsutomu Ishida, Hiroaki Kingetsu, Yasuto Yokota, Yoshihiro Okawa, Kenichi Kobayashi, andKatsuhito Nakazawa, Evaluation of Concept Drift Detection Methods for Unlabeled Data in
Operation, Proceedings of the 34th Annual Conference of the Japanese Society for Artificial
Intelligence, pp.1-4, 2020 (in Japanese),https://doi.org/10.11517/pjsai.JSAI2020.0_4Rin105.

50