Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English Edition
January 20, 2023
(a revision of the Japanese version published on August 2, 2022)

Technical Report DigiARC-TR-2023-02
Digital Architecture Research Center
National Institute of Advanced Industrial Science and Technology (AIST)

Technical Report CPSEC-TR-2023002
Cyber Physical Security Research Center
National Institute of Advanced Industrial Science and Technology (AIST)

Technical Report
Artificial Intelligence Research Center
National Institute of Advanced Industrial Science and Technology (AIST)© 2023 National Institute of Advanced Industrial Science and TechnologyiTechnical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Foreword
In the project "Research and Development on the Quality Assessment Reference and Testbedof Machine-Learning /artificial intelligence systems" (JPNP20006) commissioned by the New
Energy and Industrial Technology Development Organization (NEDO), we are developing
Machine Learning Quality Management Guidelines [1] to explain the quality of machine learning.
While developing the guidelines, we have also been researching and developing techniques forevaluating and improving the quality of machine learning. Although this research anddevelopment is still ongoing, since we have obtained technical knowledge on the quality
evaluation described in the Machine Learning Quality Management Guidelines, we report on the
progress of this research and development for the recent three years (FY 2019~2021).

ii

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Table of Contents
1

2

3

4

5

6

7

8

Introduction ............................................................................................................................................................. 1
1.1

Overview of this research and development .................................................................................... 1

1.2

Author list........................................................................................................................................................ 4

1.3

Acknowledgements ..................................................................................................................................... 4

Visualization of Machine Learning Models ................................................................................................. 5
2.1

Survey on methods to support using machine learning .............................................................. 5

2.2

Visualization of model structure and worker information ........................................................ 7

2.3

Future work ................................................................................................................................................. 11

Improved Quality through Better Application of Data Augmentation ......................................... 12
3.1

Research purpose...................................................................................................................................... 12

3.2

Improved application layer for data augmentation.................................................................... 12

3.3

Proposal for a new mixing method by improving Mixup ......................................................... 15

Debug-Testing of DNN Software ................................................................................................................... 18
4.1

Direct cause of failure.............................................................................................................................. 18

4.2Internal indices .......................................................................................................................................... 19

4.3

Experiments: method and results ...................................................................................................... 20

4.4

Related work ............................................................................................................................................... 23

4.5

Conclusion .................................................................................................................................................... 24

Debugging and Testing of Training Data ................................................................................................... 25
5.1

Three Problem Settings .......................................................................................................................... 25

5.2

Debugging Problems of Training Data ............................................................................................. 25

5.3

Outliers and Neuron Coverage ............................................................................................................ 29

5.4

Experiments and Discussions .............................................................................................................. 31

5.5

Conclusion .................................................................................................................................................... 34

Evaluation and Improvement of Robustness .......................................................................................... 35
6.1

Robustness measure (maximum safe radius) ............................................................................... 35

6.2

A survey on methods for evaluation and improvement of robustness .............................. 36

6.3

Conclusion .................................................................................................................................................... 42

Generalization Bounds of Machine-Learned Models........................................................................... 43
7.1

Generalization bounds ............................................................................................................................ 43

7.2

The theory of generalization bounds................................................................................................ 44

7.3

Computational examples of generalization bounds ................................................................... 48

7.4

Towards the evaluation of “the stability of trained models” .................................................. 52

Adversarial Example Detection..................................................................................................................... 54
8.1

Research summary ................................................................................................................................... 54

8.2

Overview of adversarial example detection approaches ......................................................... 54

8.3

NIC system design overview ................................................................................................................ 56

8.4

NIC system implementation ................................................................................................................. 57

8.5

Computer experiment ............................................................................................................................. 58

iii

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

8.6

Implementation of the NIC framework............................................................................................ 60

8.7

Evaluation of the effectiveness of NIC with the Kullback-Leibler divergence ................. 65

9

AI Quality Management in Operation......................................................................................................... 68

10

References ......................................................................................................................................................... 70iv

Machine Learning Quality Management Guideline,

National Institute of

2nd English edition

1

Advanced Industrial Science and Technology

Introduction
Machine Learning Quality Management Guideline has been developed to clearly explain thequality of various industrial products including statistical machine learning (3rd Edition [1]). The
third edition of the guideline focuses on the nine internal quality characteristics (e.g., Stability ofthe trained model, Reliability of underlying software system, etc.) for machine learning systems,
but techniques for evaluating and improving these internal quality characteristics have not been
sufficiently established yet. This document reports the current results on survey, research, anddevelopment of techniques for evaluating and improving the internal quality characteristics,
which are being conducted for supporting the development of the guideline.

1.1 Overview of this research and development
Figure 1.1 shows the relationship between the machine learning quality evaluation andimprovement techniques (the center yellow boxes in Figure 1.1, where the number in each box
shows the chapter number explained in this report) that were researched and developed for the
recent three years (FY 2019~2021). The relations to the phases of the machine learning model
lifecycle and the nine internal quality characteristics are also shown. The techniques are briefly
introduced here, and the details are explained in Chapters 2 ~ 9.

ML quality evaluation and improvement techniques
researched and developed in FY 2019~2021

DatasetAnalysis

ML modelTechniques in development
2: Visualization of ML models

Development

Training data
Input
Testing data

A-2: Coverage for
distinguished problem cases
B-1: Coverage ofdatasets

Training prog.

4: Debug-testing of DNN Software

B-2: Uniformity ofdatasets

5: Debug-testing of training data

B-3: Adequacy ofdata

6: Evaluation of robustness

Operation

Input

Data in operation

3: Improving application of DA

A-1: Sufficiency ofproblem domain analysis

Inference prog.
Evaluation prog.

ML modelDomainUpdateTraining algo.

Internal quality characteristics
in the MLQM Guideline

Inference prog.
Monitor prog.

7: Evaluation of Generalization

C-1: Correctness of
trained models

Techniques in operation

C-2: Stability of
trained models

8: Detection of Adversarial data
9: AI quality management

D-1: Reliability ofunderlying software systems
E-1: Maintainability ofquality during operation

Figure 1.1 Machine learning quality evaluation and improvement techniques in this report–

Visualization of Machine Learning Models (in Chapter 2) :To support the quality evaluation work of machine learning models, we attempted tovisualize the difference and comparison results between multiple models and thesensitivity of the workers (annotators and model designers) reflected in each model.
We proceeded with the implementation of a tool to visualize the work procedures of
the workers involved in creating the models and their influence on the models with1

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

multiple views[2][3].
–Improved Quality through Better Application of Data Augmentation (in Chapter 3):
To improve the data-diversity obtained by data augmentation and increase accuracy
and stability in deep learning, we devised new two data augmentation methods withsimple algorithms (FC-mixup and Latent DA) and report the results of their impact on
generalization performance in experiments [4]. In addition, for the Latent DA method,
we have been developing AdaLASE, for dynamically selecting appropriate layers for thedata augmentation.

–

Debug-Testing of DNN Software (in Chapter 4):
The failures of DNN (Deep Neural Network) models can be considered from two
viewpoints of causes. One of them is the direct cause during inference (by prediction
and inference programs) and the other one is the root cause during training (bytraining and learning programs, training models, and training data). We proposed anindicator and an analysis method for evaluating the presence of bugs in training
programs by the internal information (e.g., neuron coverage) of DNN models, and thenconfirmed that the indicator is useful by experiments [5].

–Debugging and Testing of Training Data (in Chapter 5):
For the case that failures in DNN (Deep Neural Network) models are caused by training
data bias, we researched methods for detecting such bias from two quality viewpoints:
model accuracy and model robustness. We proposed a method to evaluate the bias by
the internal states (e.g., neuron coverage) in the DNN models and confirmed that themethod is useful for debugging the training data by experiments.

–

Evaluation and Improvement of Robustness (in Chapter 6):
To evaluate and improve robustness of machine learned models, we report on the
results of a survey on methods to measure the maximum safe radius (the maximum
value of noise that can be guaranteed not to cause misclassifications) as a measure ofrobustness for input noise including adversarial examples, and methods to increase thesafe radius.

–

Generalization Bounds of Machine-Learned Models (in Chapter 7):
To evaluate the generalization performance of machine-learned models, we report on
the results of a survey on theorems for generalization bounds, that are the upperbounds on the expected values of the error rates (i.e., generalization errors) for all
inputs, including unseen data-samples. Then, we confirmed that (perturbated)
generalization bounds based on testing errors (i.e., test-set bounds) can be used forevaluating the generalization performance, by experiments.

2

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition–

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Adversarial Example Detection (in Chapter 8):
To establish a practical method for detecting adversarial examples, we report on the
results of a survey on the state-of-the-art adversarial example detection methods andclassifies them into four main categories, and then present the results of follow-up
experiments on representative methods. Consequently, we confirmed that NIC method
shows the highest detection rate. Then, we constructed the NIC framework fordetecting adversarial examples based on the NIC method and evaluated it by the
Kullback-Leibler divergence for explaining the reason why the NIC method is effective.

–

AI Quality Management in Operation (in Chapter 9):
To maintain quality of machine learning models even for unseen data and/or changing
trends during operation, we report on the results of a survey on detection andadaptation methods for changes in input-data distribution over time (e.g., concept
drift), and also a survey on the latest unsupervised domain adaptation methods (e.g.,label-shift). The surveys include not only supervised methods but also unsupervised/semi-supervised methods that are promising approaches from the viewpoints ofoperational costs and practical adaptability.

3Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

1.2 Author list
The authors of each chapter are as follows:–

Chapter 1: Yoshinao Isobe (AIST CPSEC)–

Chapter 2: Yuri Miyagi (AIST AIRC)–

Chapter 3: Tomoumi Takase (AIST AIRC)–

Chapter 4: Shin Nakajima (NII)–

Chapter 5: Shin Nakajima (NII)–

Chapter 6: Yoshinao Isobe (AIST CPSEC)–

Chapter 7: Yoshinao Isobe (AIST CPSEC)–

Chapter 8: Yusei Nakashima and Keiichi Nishida (Techmatrix)–

Chapter 9: Yoshihiro Okawa and Kenichi Kobayashi (Fujitsu)

1.3 Acknowledgements
This report is based on results obtained from the project "Research and Development on the
Quality Assessment Reference and Testbed of Machine-Learning /artificial intelligence systems"(JPNP20006), commissioned by the New Energy and Industrial Technology Development
Organization (NEDO).

4

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

2

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Visualization of Machine Learning Models
Information visualization is becoming a popular method to support the analysis of thestructure and behavior of machine learning models, which are known as black boxes. We havestarted research on a new method for visualizing machine learning models with the following
two objectives:–

Visualization of differences and comparison results between multiple models➢

Implementation of visualization based on expressions that are easy for humans tointerpret and understand–

Visualization of the sensitivity of workers (annotators of training data, designers ofmodel structures) reflected in the model➢

Proposal of new factors that can be used for quality assessment

In this chapter, we first describe the results of a survey of recent machine learning modelvisualization techniques. Then, we introduce the results of a prototype visualization tool forobserving model and worker information, developed in 2020-2021, and our future
implementation policy.

2.1 Survey on methods to support using machine learning
The basic purpose of visualization methods for machine learning is to improve theinterpretability of models, and this is closely related to XAI (Explainable AI), which has attracted
attention in recent years. There are no definitive definitions or evaluation methods for XAI,
however, many papers about the classification of XAI are published, and we can devise
visualization objectives and methods along these lines. In [6], the approaches to increaseinterpretability are classified into four categories:(1) Total explanation (Approximation of a complex model structure by a simple model)(2) Partial explanation (Explaining the rationale for decisions about model output results)(3) Design of explainable models (Creation of readable models at the design stage)(4) Explanation of the deep learning model (e.g., Highlighting the parts of the image datathat the model recognizes)
Especially (2) and (4) have much room for contribution by visualization. These machine learning
visualization methods are continuously being studied, and the number of survey papers isincreasing due to the diversity of applications and target cases. For example, Hohman et al. [7]described and classified deep learning visualization methods according to the 5W1H elements.
It also presents several overall directions and issues in the field of deep learning visualization.
Especially "improving interactions for model evaluation" and "improving interpretability
through active human involvement in models” are closely related to our research, which aims todevelop visualization methods for quality evaluation.

5

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

As research on machine learning visualization progresses and becomes popular in the realworld, there is a growing tendency for complex analysis to be performed in a single visualization
view. In the past, visualization methods basically focused on detailed analyses of single models
specialized in either data ([8]) or model structure ([9]). However, in recent years, research has
been conducted on combined visualization methods for data and model structure, as well as
methods that aim to compare multiple models. The number of elements that make up a machinelearning model is enormous, and it takes a lot of time and effort to create visualization results
for the number of models and compare them side by side. Besides, the differences in structure
and accuracy between the models to be compared are often small and features of the models
may be overlooked. Therefore, there is a high need for a visualization method that uses
expressions that emphasize the differences so that the differences can be found efficiently within
a limited screen. (For example, in [10], the pipeline from data input to output, hyperparametervalues, etc. for more than 10 models can be compared on a single screen.)
So far, we have introduced trends and examples of visualization methods regarding theproperties and accuracy of the models themselves. In parallel with this, we have also investigated
how the workers (annotators of training data, designers of model structures) involved in model
creation interact with the models. In fields such as image recognition, models with accuracy
beyond human recognition capabilities have been developed, but there is a persistent suggestion,
regardless of the field, that active human intervention is desirable to improve the accuracy ofmodels. There are many papers that discuss the following items regarding the relationshipbetween AI and humans and effective intervention methods in the modeling process:–Introduction of operations (adjustment and evaluation) to improve the accuracy of themodel in the learning process–

Designing an interface that is easy to use and can maintain the motivation of theoperator–

Collaboration with related fields such as cognitive science and psychology

As an example, Amershi et al. examined the psychological state of workers who wereassigned feedback to evaluate and improve several models [11]. The authors found that theworkers preferred to be able to directly tell the correct processing steps to models. They also
said that workers get more motivated to give more active feedback when they find their actions
are improving the accuracy of the model. Although there seem to be few examples ofvisualization of such information about the workers themselves and the impact of each worker
on the model, it can be adopted as a ground for quality assurance as follows:–

Show that their knowledge is sufficiently reflected in the model's behavior whendomain or machine learning experts participated in the creation of the model.

–

Indicate which workers' behavior is strongly reflected in the model and use this as aclue to identify elements (training data, parameters, etc.) that should be adjusted.

6

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

2.2 Visualization of model structure and worker information
Based on the above research results, we place particular importance on "comparativevisualization of multiple models" and "visualization of worker's sensitivity" among machinelearning model visualization methods. We proceed to design a visualization tool with bothproperties. Figure 2.1 shows an overview of the proposed method.

Figure 2.1 Overview of machine learning models and worker information visualization methods
Logging of differences between models
First, we collect logs of the structure of the model to be visualized (the adjustment process
and test results). In the current implementation, we suppose image classification as a case study
and obtain the model designer's parameter adjustment process and test results as text files using
Cometl.ml, a machine learning experiment management tool. For the annotators, we do notdirectly collect work logs, but indirectly evaluate their work based on how the model designers
selected data and applied preprocessing.
From these logs, we calculate differences between models (the amount of change from themodel used immediately before). Differences between models are classified into three
categories: training data, model structure, and optimization algorithm, and are calculated foreach. The difference in training data is calculated by adding up the data used, the number ofclasses, and the difference in parameters used for preprocessing. The difference in model
structure is obtained by creating pairs of layers that comprise the two models and summing thedissimilarities (differences in layer types and parameters) of each pair. For the difference in
optimization algorithms, a constant is assigned if the algorithm types are different. If they are
the same, the difference is calculated from the difference in parameters. After obtaining the three
types of differences, we obtain the overall change in the model by summing these values.

7Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Developing prototype of model difference visualization tool
In 2020, we implemented views on basic information such as model structure and output
results, and in 2021, added views to visualize the progress of model adjustments and testing byworkers.
Figure 2.2 shows the overview of prototype visualization views, which visualize the resultsof MNIST for two simple models developed in 2020. Assuming that the main users of this tool
would be model designers and considering the possibility that users who were not familiar withvisualization would be included, we combined basic visualization methods (line graphs, bargraphs, etc.) and implemented them with the policy of actively linking them (e.g., highlighting
related parts).

Figure 2.2 Visualization views on model structure and output results
We created the tool on JupyterLab, mainly using the machine learning library PyTorch andthe visualization library Bokeh, so that we could compare the features of the two models:(1) Network of each model structure(2) Bar graph of output results for each class① Visualization for each model② Visualization of the difference between two models(3) Scatter plot of output result correlation between two selected classes for each model(4) Line graph of accuracy(5) Thumbnail list of data classified with particularly high (low) confidence

8Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 2.3 shows an example of the results of classifying the output to MNIST for two models.
The horizontal axis represents the class from 0 to 9, and the vertical axis represents the amountof data. The color-coding of each bar represents the combination of correct (T) and incorrect (F)answers for the two models, for example, where TF (FT) means that only model 1 (2) correctlyclassified. Immediately after the start of learning (Figure 2.3, left), model 1 had a high percentage
of correct answers in classes 0, 1, and 7, and model 2 had a high percentage of correct answers
in classes 2, 6, and 8, indicating that each model had different strengths. At the advanced stageof learning (Figure 2.3, right), both models had high percentages of correct answers in many
classes. Besides, model 1 has a high percentage of correct answers, including classes 3, 4, 5, and7, which model 2 is not good at, indicating that model 1 is more advanced in learning than model
2 at this stage.

Figure 2.3 Examples of comparing the output results of two models
Next, we introduce the visualization results generated by the time-series visualization
function for model test results, created in 2021. Figure 2.4 visualizes the amount of change inaccuracy and model structure when multiple image classification models are tested in sequence.
The horizontal axis represents the order of the tests, and the vertical axis means top-1 for eachmodel. The boxes colored gray to yellow correspond to one model. The tool visualizes three small
icons inside each box, except for the box for the first model used. The colors of these icons andboxes represent the amount of change from the model used immediately before, with higher
saturation meaning greater change. Specifically, the top icon (red) represents the training data,
the middle icon (blue) represents the model structure, and the bottom icon (green) represents
the amount of change related to the optimization algorithm. The boxes (yellow) reflect the total
amount of these changes. The center coordinates of each box are connected by edges to clearly
indicate the change in accuracy.

9

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 2.4 An example of visualization of a worker's model adjustment history
Table 2.1 Parameters of the model used for work history visualization

The area circled in red in Figure 2.4 shows the visualization results of the process of adjusting
the ResNet model trained on CIFAR-100. Referring to the hyperparameter tuning scenario
conducted in [12], the model was trained and tested 6 times and logged while changing theparameters as shown in Table 2.1. 𝑙 is the learning rate, 𝑚 is the momentum value. 𝑝 and 𝑎
are the erasing probability and max erasing area when random erasing was applied to thetraining data. 𝑑 is the depth of the ResNet model used.
The numbers in the red boxes in Figure 2.4 correspond to the Indexes in Table 2.1; in 2 and
5, of the three icons, the top icon representing changes in the training data is highlighted inorange. This reflects the fact that the values of 𝑝 and 𝑎, parameters related to the training data,
were changed significantly when going from model 1 to 2 and from 4 to 5. Similarly, in 3 and 4,
the bottom icons are highlighted in yellow-green, indicating changes related to the optimization
algorithm. Specifically, it reflects the change in 𝑙 and 𝑚. The color of the box also indicates thatthe third model had the largest amount of change from the model used immediately prior.
Compared to model 2, four parameters (𝑙, 𝑚, 𝑝, and 𝑎) have been changed. On the other hand,the overall change in model accuracy was small, indicating that the impact of this parameter
adjustment procedure was limited.
The box to the right of the red area plots the progress of the test after changing the training
data to MNIST and ImageNet. The two boxes near the center of Figure 2.4 are plotted in brightyellow, which coincides with the timing of the change in the data set and model used. Thus, we
can observe the long-term working history of multiple cases, in addition to showing the detailed
adjustment process of a particular model.

10Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

2.3 Future work
In future work, we would like to work on extending the visualization function with thefollowing policy. First, we will aim to visualize the results of long-term evaluation of models andwork contents and recommendations for improvement for a single or a small number of workers.
Then, we will compare the work patterns of a large number of workers and visualize thesimilarity and classification results among models or workers from an overhead perspective. By
observing these visualization results, we would like to be able to estimate the skill level ofworkers and classify their work characteristics (work patterns).

11

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

3

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Improved Quality through Better Application of Data Augmentation
This chapter describes the results of developing a new method for applying data

augmentation in neural network learning and evaluating its effect to learning quality throughexperiments.

3.1 Research purpose
Data augmentation is a technique to increase the number of samples by adding deformations
to the data, and it is highly effective in deep learning, which has a tendency of performancedegradation when the number of training samples is small. On the other hand, the effectivenessof data augmentation strongly depends on the data used, so the selection of data augmentation
methods and the parameters of each method must be set appropriately. However, theoreticalanalysis of data augmentation is difficult, and general ways to use it have not yet been
established. This leads to unintentional and inappropriate use, which in turn compromises thequality of learning. In fact, there are many cases that training performance is degraded by setting
inappropriate values for the amount of deformation of each data augmentation method, such as
mask size or rotation angle, or where the user is puzzled as to what data augmentation method
to select for the actual data to be used.
Therefore, to move away from the empirical use of data augmentation, this study focused ondata diversity. Increasing diversity is the essential goal of data augmentation, and it has been
demonstrated in the work of [13] that increasing diversity has a significant impact on improving
generalization performance. Recently, a technique called RandAugment [14], which dynamicallyapplies randomly selected operations from multiple data augmentation operations during
training, has attracted much attention, but while it greatly improves diversity, effectively using it
is not easy because many parameters need to be adjusted. In this study, we proposed thefollowing two new methods for applying data augmentation related to data diversity, andimproved the algorithms and evaluated their performance.
–

We apply data augmentation at various layers of the neural network, including hidden
layers, and perform automatic optimization of the applied layer (Section 3.2).

–

We improve the Mixup method [15], a promising data augmentation method, andpropose a new way to mix samples (Section 3.3).

3.2 Improved application layer for data augmentation
Data augmentation at hidden layers
Generally, data augmentation is considered to be applied to input data, but in neuralnetworks, it is also possible to apply data augmentation to hidden layers. There are several
previous studies on this subject, but most of them are not versatile methods, such as Manifold

12Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

mixup [16], which limits the method to mixup [15], or other methods that require specificnetworks and datasets. In this study, we considered applying various data augmentation
methods used for image data, such as affine transformation and mask processing, in the hidden
layers. Since features are extracted hierarchically in CNNs, data augmentation can be applied invarious layers randomly selected for each minibatch to generate a wide variety of samples. As
with application to input images, data augmentation can be applied to the feature maps obtainedat the intermediate layers, making implementation easy.
An example of actual application of mask processing and translation to an input image andfeature map is shown in Figure 3.1. Here, a sample is input to the model in training, and theimages are shown in the upper row, aligned in size, immediately after data augmentation wasapplied at different layers with the same parameters (mask position and translation amount).
The feature maps in the final layer of the sample are shown in the lower row. They are different
images depending on the layer where the data augmentation was applied. This result shows thatdata augmentation at various layers leads to an increase in the diversity of the generated data
and results in learning different from when data augmentation is applied only to the input data.

Figure 3.1 Example of applying data augmentation to input images and feature maps obtained at
hidden layers
To compare the performance of data augmentations in the input layer and that in featuremaps, we used various data augmentations and obtained test accuracies for models trained withsupervision. Here, WideResNet28-10 was trained for 200 epochs using the CIFAR-10, FashionMNIST, and SVHN (without extra data) datasets. The results are shown in Figure 3.2. In eachfigure, the horizontal axis represents the accuracy [%] of the conventional method (Input DA)
and the vertical axis represents the accuracy of the proposed method (Latent DA). As can be seenfrom these results, the proposed method tends to show higher accuracy than the conventional
method, and the proposed method presented higher accuracy even in cases where theconventional method presented lower accuracy, such as the results using Crop. These results
indicate that the diverse samples generated by the application of data augmentation to random
layers are effective in improving performance.

13

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 3.2 Comparison of test accuracy between input DA and latent DA
Selecting appropriate layers for data augmentation
Although previous studies have shown that data augmentation at hidden layers is effective,
the question arises as to which layer is optimal for data augmentation. Although it is possible tofind the optimal layer by repeatedly training with different layers of data augmentation andcomparing the values of validation accuracy, it is an inefficient and impractical method because
it increases the overall training time. Therefore, in this study, we worked on developing a method
to dynamically discover the optimal layer for data augmentation in a single training session.
The approach is to prepare a parameter called the acceptance rate for each layer, update theacceptance rate during training, and apply data augmentation in the layer selected
probabilistically according to the acceptance rate. The updating of the acceptance rate is doneusing the gradient descent method as shown below.
𝑞𝑙 ← 𝑞𝑙 − 𝜂

𝜕𝐿𝑣𝑎𝑙
,
𝜕𝑞𝑙

where 𝑞𝑙 is the acceptance rate of layer 𝑙, 𝐿𝑣𝑎𝑙 is the value of the error when the validation
data is input, and 𝜂 is the step width of the update. In practice, the values of the validation data
should not be included in the algorithm, so the update is performed by creating pseudovalidation data with the training data with data augmentation. In the initial state of training, allacceptance rates are set to equal values so that the sum is 1, and the acceptance rate is updated
for each minibatch. This optimization is expected to improve the generalization performance byincreasing the acceptance rate of layers suitable for data augmentation and decreasing theacceptance rate of layers unsuitable for data augmentation.
We named this method Adaptive Layer Selection (AdaLASE) and compared it to conventional
methods. Using CIFAR-10 as the data and ResNet18 as the model, we compared test accuracies
for no data augmentation, data augmentation on input, data augmentation at random layers, andAdaLASE. Figure 3.3 (a) and (b) show the results using Cutout and Mixup, respectively. The meanand standard deviation of the accuracy for five different initial values are shown for each method.
These results show that AdaLASE can perform as well as or better than conventional methods.

14

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Future plans include a detailed analysis of how layers are selected and whether AdaLASE is
working properly by looking at the change in acceptance rate during training.

(a) Cutout

(b)

Mixup

Figure 3.3 Comparison of test accuracy between AdaLASE and conventional methods

3.3 Proposal for a new mixing method by improving Mixup
In actual training, data augmentation often involves the simultaneous use of multiple
methods, such as cropping, rotating, and flipping. Therefore, we focus on the compatibility
between methods when multiple methods are used in this way, and in particular, we consider
discussing the compatibility from the viewpoint of data diversity. As a first step in this approach,we propose a new method that is a variant of an existing method and use it simultaneously withthe original method to increase the diversity of the data generated and improve performance.
The method for formulating the diversity is described in the work of [13]. In this study, we firstcompare only the accuracy and verify whether the proposed method improves the performance.
Here, we have improved Mixup [15], one of the data augmentation methods. This method
generates a new sample by linear interpolation of two samples, and takes the same ratio of linear
interpolation for each of the input values and labels, as shown in the following equation.
{𝑥̃ = 𝜆𝑥𝑖 + (1 − 𝜆)𝑥𝑗,𝑦̃ = 𝜆𝑦𝑖 + (1 − 𝜆)𝑦𝑗

where (𝑥𝑖 , 𝑦𝑖 ) and (𝑥𝑗 , 𝑦𝑗 ) represent the input values for the 𝑖-th and 𝑗-th samples, and 𝜆 isthe mixing ratio sampled from the beta distribution. Mixup was chosen as the subject of thisstudy because of its versatility and because it can be used for many numerical data, including
not only images but also time series data, and therefore, the impact of improving the Mixupmethod would be significant.
An improved version of mixup so that it can also be performed in a hidden layer of a neuralnetwork is called manifold mixup [16], but both mixups generate samples only in a localized
region of the data distribution, on a line segment between two points, and are inappropriate fordata sets with distributions in which the properties of the points on that line segment varynonlinearly.
The Feature Combination Mixup (FC-mixup) proposed in this study is a method of mixing
samples in a different way from conventional mixups, and is outlined in Figure 3.4. Suppose that
two samples A and B in the same minibatch output the features 𝑍A and 𝑍B in a randomly

15

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

selected layer. 𝑑 is the total number of features in that layer, FC-mixup randomly extracts andcombines 𝑑𝜆 features from 𝑍A and 𝑑(1 − 𝜆) from 𝑍B and generates a new sample 𝑍X . Since
the number of possible combinations is large for a single value of 𝜆 , different data can begenerated depending on the random number, and thus samples can be generated over a wide
range of the data distribution. FC-mixup is expressed as follows, so 𝑍A and 𝑍B are mixed sothat this equation is satisfied.
|𝑍A ∩ 𝑍X | = d𝜆Figure 3.4 Overview of FC-mixup
This technique of generating new data by combining the parts of two data sets is also foundin Puzzle Mix [17], but the target is limited to the input image. A similar technique is used inAdversarial mixup resynthesis [18], but it is limited to use in autoencoders, while FC-mixup isdesigned for more general use. To increase the diversity of the generated data, a method thatsimultaneously uses FC-mixup and Manifold mixup [16] is referred to here as the Hybrid method.
In our experiments, we used several multi-class classification datasets to compare theclassification accuracy of the test data between the conventional method (no data augmentation,
mixup at the input layer [15], Manifold Mixup [16]) and the proposed method (FC-mixup, Hybrid
method). MNIST, CIFAR-10, CIFAR-100, SVHN, and TinyImageNet were used for the data. Models
used were a multilayer perceptron (MLP) with one intermediate layer, a small convolutional
neural network (CNN), ResNet18, and ResNet50. In addition to the full-size data, experiments
were conducted on reduced data with 1,000 randomly selected samples. Means and standard
deviations in five trials with different initial values were obtained and compared.
The results in Table 3.1 show that in most cases the proposed method gives the highest
accuracy. Overall, FC-mixup tended to give better performance than the Hybrid method. It canbe said that the results of the present study are promising results, indicating that using the FC-

16Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

mixup and Hybrid methods is likely to improve the quality. Detailed analysis of the compatibility
between data augmentation methods focusing on diversity will be the subject of future work.
Table 3.1 Comparison of test accuracy on multi-class classification data17

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

4

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Debug-Testing of DNN Software
In the initial development stage of Deep Neural Network software (DNN software), we ensurethat the required functions and prediction performance are achieved through iterative trial-anderror processes, in which three viewpoints (elaborating and refining requirements, preparing
datasets for training, and selecting appropriate machine learning models) are considered. This
trial-and-error process corresponds to debugging in conventional program development. In thecase of DNN software, the debugging activities involve generating datasets for debug-testing,monitoring the training and learning status, and identifying and removing root causes thathinder the fulfilment of requirements. In the following, we will report on a debug-testing method
investigated in FY2020, discuss the experimental results obtained, and summarize our futureplans.

4.1 Direct cause of failure
A standard method of supervised DNN learning involves two types of programs: training (orlearning), and prediction (or inference). When training data is given and a learning task toachieve is made clear, a learning model for the target DNN software is selected, and some design
decisions on the method used in the training and learning process is fixed. If we use available
open-source machine learning frameworks, we may set up several parameters of the framework.
The next step is to construct training dataset. Then, we run the training/learning program(possibly provided by the machine learning framework) with the training model and training
dataset as input, and derive a trained DNN model as a computation result. More precisely, thetraining/learning program searches for a set of weight parameter values that define the trained
DNN model uniquely. This trained DNN model defines behavior of the prediction/inference
program.
From a user's point of view, a prediction/inference program is the entity to use. In the caseof a classification learning task, for example, the program calculates certainty levels ofprobabilities of classification results for an input data. By examining the output results, we candetermine whether the DNN software works as intended. When the program does not produce
results as expected, we localize possible fault locations and remove them. In other words, we
conduct debugging.
A failure may be occurred due to a flaw somewhere in the information used in the execution
process of the training/learning program, either in the training dataset, the training model, thelearning mechanism, or their combinations. However, direct causes of failure in prediction/
inference results are attributed to the trained DNN model or set of obtained weight parametervalues. While a root cause of failure is somewhere and often not known, the failure is attributed
to a defect in the weight parameter values or the trained DNN model. Thus, from users' point ofview, a certain distortion of the trained DNN model seems a direct cause of the failure [19]. A
method to measure such distortion degrees is needed regardless of the root causes.

18

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

In this chapter, we investigate whether we can detect faults in DNN software with an internal
metric to measure such distortion degrees of trained DNN models. The weight parameter values
in the DNN models are the output of the training/learning program, but there is no direct way tocheck its validity, because those expected weight parameter values cannot be known in advance.
If such expected parameter values were known, training/learning could be skipped. We can justuse those known values, as embedded in a trained DNN model, to implement a prediction/
inference program.

4.2 Internal indicesThis section first introduces the notion of neuron coverage (NC). We consider a learningmodel as a network of neurons. Given a threshold, neurons whose output values exceed thethreshold are said to be activated. When the number of neurons constituting the learning modelis N and the number of activated neurons is A, the neuron coverage is defined as the ratio ofactive neurons is (NC = A/N). In [20], NC is assumed to be criteria for test coverages of trained
DNN models; the research work investigates how the choice of input data for evaluation affects
NC values.

Cla

ithm
lgor
A
n
catio
ssifi

Figure 4.1 Trained DNN model.
In this chapter, NC is assumed to be used as an internal index [21] to represent distortion
degrees by appropriately choosing the target neurons to be considered. Figure 4.1 shows aschematic diagram of the trained DNN model. NCs are defined for the neurons in the final stageof the middle layer (or the penultimate layer as shaded gray), but not for all the neurons in the
trained DNN model as in [20].
In general, in machine learning techniques, this penultimate layer is often considered to holdmeaningful information. For example, in the case of an image classification task, the early stagesof the model is responsible for the correlation analysis (analysis of patterns of pixel values),
which plays a specific role in algorithms such as image recognition, and their calculation results
are summarized in the penultimate layer. In this chapter, we assume that direct causes of defects
are manifested in this internal layer. Furthermore, various statistical indices can be derived
based on NC values of this layer. We will investigate, through experiments, what derived index isappropriate depending on test objectives to be investigated.

19

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

4.3 Experiments: method and results
We present the results of several experiments and discuss the usefulness of the internal orderived indices mentioned in the previous section. First, we show the results of comparative
experiments when a training/learning program (or a learning framework) has faults in it. In thefollowing, BI is the training/learning program which is a bug-injected version of a probably
correct program PC.
Figure 4.2 depicts the accuracy (the percentage of reconstructed correct answers) for a test
dataset. In the experiments, a classical fully-connected network is chosen as the learning model,
and different number of neurons are placed in the middle layer, which implies that each model
is of different structural capacity. When we have a sufficient number of neurons (50 on thehorizontal axis), there is no significant difference in the accuracy between PC and BI. Thus, it isdifficult to distinguish between the PC and BI solely by examining their accuracy values, and thus
the presence or absence of a defect cannot be identified. In addition to this finding (Figure 4.2),
the results of an experiment to systematically investigate the situation further (Figure 4.3) arepresented below.
1

Accuracy

0.98

0.96
PC
BI

0.94

0.92

0.9
0

10

20

30

40

50

60

Structural Capacity

Figure 4.2 Learning models of different capacities.

Activated Neurons

1
0.9
0.8
0.7
0.6
0.5

PC

0.4

BI

0.3
0.2
0.1
0
0

10

20

30

40

50

60

Structural Capacity

Figure 4.3 Relationship with internal indicesFigure 4.3 plots values of the internal index (activated neurons or neuron coverage) on thevertical axis. Their absolute values, for example, of 10 for BI and 30 for PC are both around 0.7,
making it impossible to distinguish between BI and PC if we do not take into account thestructural capacity. The indices are not usable to examine the activated states of neurons.

20Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Therefore, we will study if there is an appropriate indicator to be derived from the internal indexof NC. As a set of data (a sample), in the test dataset, leads to a collection of neuron coverages,
we can obtain some statistics from the sample such as the mean μ and variance σ2, andcalculate σ/μ. Figure 4.4 shows the case where this derived index σ/μ is used on thehorizontal axis. From the values on the vertical axis, we can find out which leaning model has
which value by referring to Figure 4.3.
1

Accuracy

0.98

0.96
PC
BI

0.94

0.92

0.9
0

0.1

0.2

0.3

/µ

0.4

p

Figure 4.4 Derived index
Figure 4.4 shows that we can distinguish between the PC and BI. Although the internal index
cannot distinguish between the PC and BI with different capacities (Figure 4.3), a derived indexof σ/μ can discriminate between the PC and BI. We can see that the neuron coverage basicallycontains a piece of useful information.
Next, Figure 4.5 is a scatter plot of classification probability using corrupted data for theevaluation; the horizontal axis refers to the classification output by the BI and the vertical one
by the PC.
1.2

Probability of PC

1

0.8

0.6

0.4

0.2

0
0

0.2

0.4

0.6

0.8

1

1.2

Probability of BI

Figure 4.5 Classification certainty for corrupted data.
In Figure 4.5, a △ represents an output value for corrupted data, which is supposed to be
distributed on the dotted line passing through the origin, if we assume that the PC and BI output
the same value for the same data. In fact, it can be seen that □ selected from the test dataset

21

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002(without any corruption) mostly arranged on the dotted line. On the other hand, corrupted data(△) are distributed along the solid line, indicating that the PC is a better classification certainty
than the BI. It implies that the BI, containing bugs in it, is less robust, although the accuracy
remains the same as that of PC (Figure 4.2).
The following experiment confirms that differences in robustness can be detected by using
an internal index (Figure 4.6).

Figure 4.6 Differences in internal indices
The corrupted data described above were input, and the internal index for each input wasplotted on the horizontal axis. The □ distributed in a group on the right side shows the resultsof PC, and the ◇ distributed in a group on the left side shows the results of BI. The scatter plotshows that (1) the value of the internal index of PC is large, and (2) the correlation between theinternal index and prediction probability (certainty of classification) is negligible (0.033). Next,
we calculate σ/μ, which is 0.0876 for PC and 0.2183 for BI. Figure 4.6 shows results that
corrupted data affect the robustness, and that the value of σ / μ is considered to havecorrelations with the robustness.
Next, we conducted experiments to investigate how distorted training data affect the trained
DNN model. We plotted the accuracy for a test dataset common to all the cases. Thus, differencesin the vertical axis indicate a certain difference (distortion degree) in the training dataset used
for obtaining the trained DNN model (Figure 4.7).

Figure 4.7 Differences in training datasets.

22

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 4.7 shows the two independent series for the PC (□) and BI (◇). From top to bottom
in a series of each measured points (from better to worse accuracy), a training dataset with alarger distortion is used. Because the test dataset is common, the data shift of the test data isrelatively larger as the distortion degrees in the training data is larger. Furthermore, the accuracy
decreases as the shift becomes large. Figure 4.7 also shows that the value of the horizontal axis(σ/μ) is clearly different between the PC (□) and BI (◇). It can be confirmed that the accuracy
and the robustness suggested by the σ/μ values are two independent perspectives.
From the above (Figure 4.7), the distortion in training dataset can be examined by themethod based on the accuracy. As is done in practice, the method based on the accuracy is usefulwhen checking the training dataset quality. On the other hand, if there is a possibility that otherfactors such as faults in a training/learning program are involved (multiple defects are assumed),
it is desirable to examine the values of the internal and derived indices (σ/μ) at the same time.

4.4 Related work
Neuron coverage (NC) is a simple quantitative measure introduced in DeepXplore [20] as atest coverage metric. In conventional software testing, test coverage is defined in terms of thebasic block of program codes, which is the statements executed by a given test input data. A
program is represented as a Control Flow Graph (CFG) whose nodes refer to executablestatements. In the simplest case, the criterion is whether or not a node in the CFG is containedin an execution path induced by an input test, i.e., whether or not the statements are executed.
As a DNN model is represented as a network, a kind of graphs, metrics similar to those for CFG
can be introduced. The neuron coverage concerns whether neurons located at nodes areactivated (output values of these neurons exceed a specified threshold), which is comparable to
the C0 criterion defined on the CFG. DeepXplore assumes that high NC values refer to thesituations where high percentage of neurons are exercised by input data, and discusses how to
generate new test input data to increase the NC values.
Neuron coverage would be a straightforward idea analogous to the conventional test
coverage criteria. Later, satisfying the criteria, to achieve 100% in terms of NC, is foundempirically not difficult. New metrics are proposed to take into account correlations among
multiple neurons or those in different layers [22], which may be comparable to more elaborated
coverage metrics, such as C1 or the others, in conventional software testing.
The original NC is simple and easy to use as a metric to guide or control automated test
generation processes. Usually, a classical data augmentation method picks up a seed data, fromwhich a series of new data is to be generated by pre-defined data transformation algorithms.
New test data are successively generated until the accumulated NC values is saturated. Whenreached the situation where no increase in the NC is seen, the generation method switches a seeddata to new one and continue the process [23]. The classical data augmentation method can be

23Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

replaced by other approaches such as test input generation based on GAN [24]. Test generation
method using GAN with a help of NC is reported in [25]. Although it is a simple metric, NC is now
considered as a practical criterion to control the automated test generation process (coverageguided test generations).
Some of early works on testing pre-trained DNN models adapt application-specificproperties as software test oracles; the DNN models for regression tasks in the auto-pilot car
application [23][24] use the calculated steering angle as the oracle. There is also a research work
[26] to investigate whether test inputs to increase the NC values are useful for detecting faults.
The usefulness of NC is dependent on what are considered failures. The work [26] also indicatesthat the correlation between NC and external indices such as the accuracy is weak. In this chapter,based on this observation that the correlation between the two is weak, an internal index basedon the NC is used for the test, which is not contradictory to the discussion in [26], but rather inthe same direction. Note that the test coverage is a criterion for terminating testing, while
detecting faults depends on whether the test input data executes corner cases. These two notions,the test coverage and corner cases, refer to different aspects. In fact, it has been reported thatthe enhancement of coverage does not necessarily leads to the improvement of the efficiency offault detection in conventional software testing. The same findings would be applicable to casesof DNN testing.
In this chapter, we use the NC value as a simple test index, from which a sort of distortion
degrees in trained DNN model is derived [19][21]. Our approach is based on a view that faults
in DNN models appear as inappropriate NC values, whereas existing works use NC as a criterion
for the test coverage. In our experiments, we were able to examine the reliability of the training
and learning programs and the robustness of the trained DNN models. These are two primaryconcerns in debug-testing.

4.5 Conclusion
In this chapter, we used an internal index based on the neuron coverage (NC) defined on the
penultimate layer for representing a sort of distortion degrees in trained DNN model. The NC isa scalar and easy to measure, and thus can be used as a light-weight test index. It, however,discards the information about the individual activated neurons, and thus lacks useful
information. In fact, Kim et al. [27] proposes a method to estimate the distribution of activatedneuron and to discuss the usefulness of input data for testing. Distribution on such neuron values
may be considered to have rich information. In future, we will study how to debug training
dataset by making use of such distribution information.

24Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

5

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Debugging and Testing of Training Data

5.1 Three Problem Settings
In early stages of software development, in which programs are constructed to employ Deep
Neural Networks (DNNs) [28], debugging and testing is performed to ensure that the core DNN
components behave as expected. This is the process of feeding appropriate data to the DNN
components and checking whether the predicted output is exactly what is expected. If the output
is faulty in some ways, the DNN component under test contains a defect. The purpose of
debugging is to identify and remove such unknown defects.
Defects in DNN components are the direct cause, but not the root cause, of failures. In thestandard method for building DNN components [29], three distinctive components are basicallyinvolved: (a) the machine learning infrastructure, (b) the training model (a template of the DNN
model), and (c) the training data. The root cause is one of them or their certain combinations
leading to the failure that the DNN component exhibits. The problem setting of the inspection
differs depending on where the root cause is assumed [30].
The basis of DNN component construction is to make use of a training dataset consisting ofa huge number of training data and derive the information inherent in those data by means ofstatistical methods so as to obtain a DNN model (a nonlinear function) inductively. In a naive
way, we may examine the DNN model to identify root causes. However, since the DNN model is anonlinear function to exhibit some functional behavior, the software testing method using
indirect test oracles is often employed; we feed evaluation data to the DNN model and check
whether output results are valid or not [31].
In the case (a) above, the core of the machine learning infrastructure is a numerical programthat solves an optimization problem, and the metamorphic testing method is known to be useful[32]. In the case (b), the learning model is not obviously flawed. It is to find an optimal or suboptimal learning model for the target machine learning task, which has been, in a sense, one of
the main challenges of the DNN technology [28]. In this chapter, we discuss the case (c), i.e.,
debugging and testing methods of training data.

5.2 Debugging Problems of Training Data
Debugging and testing of training data is to revise (add or delete) the training data so as toobtain a DNN model that exhibits the intended functional behavior. This view is based on theobservation that the bias of the training data affects much the trained DNN model. In thefollowing, we specifically consider the debugging problem of training data for supervisedmachine learning classification tasks.

25

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Model Accuracy and Model Robustness
In the supervised task of classifying input data into 𝐶 categories, a datapoint 𝑧 is a tuple(𝑧 = 〈𝑥, 𝑦〉) consisting of two types information, a multidimensional vector 𝑥 and its correct
answer tag (or simply a label) 𝑦 (see Figure 5.1). The DNN model, derived from a given
training dataset 𝑆 (𝑆 = {𝑧 (𝑘) | 𝑘 = 1, … , 𝑁 }), is inspected against input evaluation data 𝑥. Its
output is a 𝐶 dimensional classification probability vector 𝑃𝑥 corresponding to the data 𝑥. If
𝑃𝑥 [𝑗] (the 𝑗-th component of 𝑃𝑥 ), the component with the largest value 𝑗, is equal to y (𝑦 =argmax(𝑗∈[1,𝐶]) 𝑃𝑥 [𝑗]), then the DNN model is considered to return a correct answer. In this case,the multidimensional vector 𝑃𝑥 , in particular, the probability of the 𝑗-th component 𝑃𝑥 [𝑗], is oneof the good indicators of the model accuracy for the data 𝑥. For a collection of evaluation data
𝐸 (𝐸 = {〈𝑥 (ℓ) , 𝑦 (ℓ) 〉 | ℓ = 1, … , 𝑀 }), Accuracy is the number of correct answers (percentage ofcorrect answers) for the collection. In addition, the variability of the probabilities of theclassification categories (sometimes referred to as Gini Impurity) is an indicator of the model
accuracy as well.
The accuracy for the training dataset 𝑆 and the one for the other dataset 𝐸 , different
dataset from 𝑆 , are compared. While the accuracy for 𝑆 is good, the accuracy is sometimes
worse for 𝐸. This phenomenon is known as overfitting to the training dataset. Usually, both 𝑆
and 𝐸 are constructed from one large data pool 𝐷 , and are considered as different samples
following the same data distribution; 𝐸 in this case is sometimes called a testing dataset ascompared with the training dataset of 𝑆. When there is no overfitting where the accuracies are
not much different each other, the DNN model is considered to exhibit good generalization
performance.
In the training data debugging problem, the evaluation data 𝐸 may be selected from adataset other than 𝐷 . For example, in positive testing, where the goal is to confirm that thesystem exhibits the expected behavior, as in the evaluation of generalization performance, we
can choose 𝐸 from 𝐷 , in which 𝐸 is different from 𝑆 . However, to test the behavior in
exceptional situations, we may choose a dataset 𝐹 for the evaluation that is not included in 𝐷.
Model accuracy, measured with the percentage of correct answers, is not a good indicator for 𝐹.
The evaluation criterion is model robustness, which expresses how the prediction probability isdecreased depending on how much a data in 𝐹 is deviate from data in 𝐷 or 𝑆.
In the development in practice, if the expected prediction performance is not achieved for agiven 𝐷, new data is collected and the training data itself is revised. Then, the DNN components
are derived using the new training dataset, namely in an iterative manner. Moreover, during
testing, we evaluate the model accuracy and model robustness in view of both positive andexceptional testing.
Memorization of Training Data
Overfitting or overlearning significantly affects the prediction performance (the model

26

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

accuracy and model robustness) of DNN models. Therefore, basic machine learning methods
have been studied extensively to mitigate those problems; the study includes regularization or
dropout [33]. In spite that such methods are adopted, the expected prediction performancecannot be obtained if the training dataset is inadequately biased. The debugging problem oftraining data is to improve the prediction performance of DNN models by revising the training
dataset. Simply, it is to eliminate the inappropriate bias. However, it is difficult to evaluate thedegree of bias as well as the appropriateness or inappropriateness of the bias.
One traditional approach to evaluate the bias of the training data (sample) is to examine
statistical characteristics of the sample. For example, given that 𝑆 = {〈𝑥 (𝑘) , 𝑦 (𝑘) 〉 | 𝑘 = 1, … , 𝑁 } ,let 𝑆 𝐶 = {〈𝑥, 𝑐〉 | 〈𝑥, 𝑐〉 ∈ 𝑆 and 𝑐 = 1, … , 𝐶 } where 𝑐 is a correct answer tag. If the sizes of 𝑆 𝐶
are equal in size, then we may say that there is no bias among 𝑆 𝐶 from the viewpoint of thecorrect answer tag. However, each 𝑆 𝐶 follows some data distribution 𝜌𝐶 and we don't knowwhether 𝑆 𝐶 is sampled faithfully in regard to 𝜌𝐶 . To check this, we need to know 𝜌𝐶 , however,the data 𝑥 is multidimensional, and such a multidimensional data distribution is not easy toestimate.
Alternatively, the prediction performance of DNN models is investigated by testing resultswith input evaluation data. DNN models derived from the same training data may exhibit
different prediction performance, depending on the method of the machine learning. In otherwords, it is not enough to examine the statistical characteristics of the training data for the
purpose of debugging the training data, but it is also necessary to consider how the bias of the
training data is reflected in the trained DNN model.
The relationship between DNN models and training data bias can be discussed in terms of
the DNN models remembering the labels of the training data. Now, when the training data 𝑆
contains a datapoint 〈𝑎, 𝑡〉 (〈𝑎, 𝑡〉 ∈ 𝑆), we can construct 𝑆′ so that the 〈𝑎, 𝑡〉 is removed fromthe training data 𝑆 (𝑆 ′ = 𝑆 ∖ {〈𝑎, 𝑡〉}). Let each DNN model obtained by training with either 𝑆
or 𝑆’ be 𝑀 or 𝑀’ respectively. Then, the result, 𝑃𝑎 for 𝑀 or 𝑃𝑎′ for 𝑀’, is obtained for thecommon input data 𝑎 . If the classification result 𝑡 for 𝑃𝑎 [𝑡] is very likely and 𝑃𝑎′ [𝑡] is lesslikely, then 𝑀 is said to memorize the datapoint 〈𝑎, 𝑡〉 used as one the training data. From thisdefinition, we can see that the DNN model memorizes the training data in the overfitting
situation, where 𝑃𝑎 [𝑡] is apparently more likely than 𝑃′𝑎 [𝑡].
For DNN models, it is known that the Membership Inference is possible. The problem is tofind out if a datapoint 〈𝑥, 𝑦〉 (〈𝑥, 𝑦〉 ∈ 𝐷) was included in the training dataset (〈𝑥, 𝑦〉 ∈ 𝑆) justfrom the information obtained by feeding data to the trained model, 𝑀 𝑆 (𝑥). Black box methods
make use of the classification probability vector 𝑃𝑥 [34], or white-box methods use theinformation of the loss function ℓ(𝑌(𝑊; 𝑥), 𝑦) calculated in the process of executing 𝑀 𝑆 (𝑥)[35], where 𝑊 is the training parameter or weight and 𝑌(𝑊; 𝑥) is the internal representationof the prediction for the input 𝑥.
Intuitively, Membership Inference method is based on the observation that the distributionof 𝑃𝑥 or ℓ(𝑌(𝑊; 𝑥), 𝑦) is different depending on whether the datapoint 〈𝑥, 𝑦〉 is included in
the training dataset 𝑆 or not. Furthermore, these differences in the distributions are somehow
attributed to the memorization of training data including overfitting cases [35]. Thus, the27Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

approach to mitigate the threats of Membership Inference is to remove those data, that arememorized easily, from the training dataset, in addition to employing a machine learning
method that avoids overfitting [36].
We now examine the situation involved with the memorization of training data. Consider aclassification problem as in Figure 5.1; we assume 𝑎 ≠ 𝑏 whereas 𝑡 = 𝑢 . Figure 5.1 (a)illustrates a situation where the prediction probability of 〈𝑏, 𝑢〉, a training data moved awayfrom 〈𝑎, 𝑡〉, decreases as the distance between them becomes large. Figure 5.1 (b) shows thatremoving that datapoint 〈𝑎, 𝑡〉 from 𝑆 does not significantly affect the prediction probabilityof the data 〈𝑏, 𝑢〉 when the training data are dense in 𝑆. In other words, the removed training
data is not memorized in that it does not significantly affect the prediction results. Figure 5.1 (c)represents a situation where the training data are sparse. Contrary to Figure 5.1 (b), it represents
that the influence becomes large and is firmly remembered. Such outlier data significantly affects
the predictive classification performance of the DNN model.
Finally, we consider the Membership Inference viewed from the training data debugging
problem. In the situation where training data are memorized, the distribution of either 𝑃𝑥 orℓ(𝑌(𝑊; 𝑥), 𝑦) is very different depending on whether the datapoint is included in the training
dataset 𝑆 or not. The Membership Inference method makes use of the fact that the predictive
performance for 𝑧 ′, far from training 𝑧 datapoints, is poor. In other words, we can think of theMembership Inference as a test of model robustness; the phenomenon of training data
memorization is related to model robustness.

(a) Predicted probability in the neighborhood (b) Dense region (c) Sparse region
Figure 5.1 Training data placement and prediction certainty.
Here, we refer to the schematic situation in Figure 5.1. Removing the dense data shown in
Figure 5.1 (b) would have little impact on the model accuracy. On the other hand, removing thedata in a sparse region as shown in Figure 5.1 (c) would improve model robustness, but would
reduce model accuracy in the neighborhood because there would no longer be data to support
their predictive classification results. Alternatively, adding new data in the neighborhood
without removing this datapoint will make the region dense and improve the local model
accuracy. Therefore, detecting outliers in the training data set 𝑆 is important for debugging
dataset.
Figure 5.1 schematically illustrates that the predictive classification performance of the input

28

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

data is affected by the location relationship with the training data. However, it does not say how
the location relationship is defined, i.e., from what aspects of the data, the location relationshipis defined. Conversely, now the question is how the location relation should be defined when
discussing the difference in prediction classification performance; the outlier detection problem
will become clear when such criteria are precisely defined.

5.3 Outliers and Neuron Coverage
We consider outlier detection methods for the purpose of training data debugging.
Outliers in Training Data
The debugging problem of training data is to find out outliers in the training dataset and to
decide how to deal with the outliers according to the purpose of the DNN model underdevelopment. How we handle the outliers is related to the requirements specification of the DNN
model. Thus, the general discussion of training data debugging may be limited withinestablishing a technique for outlier detection.
In general, outliers are data that have different characteristics from the data that make up
the majority, and whether or not they are outliers is defined based on the data distribution(statistical data model) that the collection of target data exhibits [37]. For example, if theprobability density function of the data distribution is known, then we can check whether thedata are outlier or not based on the likelihood of the data.
In a naive way, we consider whether it is an outlier or not based on the empirical distributionof the training data. However, the training data is a multi-dimensional vector, and it is difficult toknow the empirical distribution in a compact form. For example, it is difficult to apply methods
such as Kernel Density Estimation, and as a result, the outlier detection method based on
likelihood is not practical. Alternatively, analysis methods similar to Combination Testing, which
is known in the field of software testing, may be applied. By selecting components (features) thatare considered having a large impact on the empirical distribution and focusing on such
representative dimensions, we may conduct analysis as an approximate of the case on the whole
empirical distribution. While practically applicable, outliers are rare by definition, and theeffectiveness of this approximate method is questionable.
For a slight change of perspective, the robustness radius of the standard method of analyzing
model robustness [38] is considered. For two datapoints 〈𝑥, 𝑦〉 and 〈𝑥 ′ , 𝑦 ′ 〉 and the predictiveclassification results for each of the outputs 𝑃𝑥 [𝑦] and 𝑃𝑥 ′ [𝑦 ′ ], let the robust radius 𝛿 be thetolerance level 𝜀 of the difference between the outputs; for a given 𝜀, the robust radius is the
maximum difference of input data that satisfies 𝛿 ( | 𝑃𝑥 [𝑦] − 𝑃𝑥 ′ [𝑦 ′ ] | ≤ 𝜀 when | 𝑥 − 𝑥 ′ |𝑝 ≤𝛿 ). Here, we define the difference of input data in terms of 𝐿𝑝 -norm. In a naive way, for a given
𝜀 for given input data, we consider that the model robustness is good if the robustness radius 𝛿is large. However, the calculated radius 𝛿𝑝 is dependent on the choice of the norm 𝐿𝑝 . While29

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

the definition of model robustness by the robust radius is strict, the analysis in the space of input
data requires further discussion or interpretation of whether the norm used is appropriate ornot, which complicates the problem.
We consider now how the training data 〈𝑎, 𝑡〉 affect the prediction results of the other data〈𝑏2, 𝑢2〉 that are classified to different classification categories (𝑡 ≠ 𝑢2) (see Figure 5.1 (a)). The
two datapoints have different classification categories and can be assumed to be far apart in the
input data space. We assume that the training dataset 𝑆 contains 〈𝑎, 𝑡〉 and let 𝑆’ be the one
to be removed 〈𝑎, 𝑡〉 from 𝑆. Further, let the DNN models obtained from 𝑆 and 𝑆’ be 𝑀 and
𝑀’ respectively, and let the predictive classification results for the data 〈𝑏2, 𝑢2〉 be 𝑃𝑏2 and′𝑃𝑏2. With the method of Influence Functions, which analyzes how 𝑆 and 𝑆’ affect the error

function, we are able to know that there exists 〈𝑏2, 𝑢2〉 such that the values of 𝑃𝑏2[𝑢2] and′ [𝑢2]𝑃𝑏2
are different [39]. It shows that the presence or absence of the training datapoint 〈𝑎, 𝑡〉
affects the classification probability of 〈𝑏2, 𝑢2〉. Therefore, it is difficult to obtain the desired
information by analyzing the differences in the input data space (𝑎 ≠ 𝑏2) .
From the above, we can see that it is difficult to systematically detect the desired outliers by
analyzing a collection of training data in the input data space. The reason for this is that model
accuracy and model robustness are affected not only by the training data but also by various
factors involved in the training process, such as the machine learning method. However, we do
not claim that the analysis in the input data space is completely ineffective. Such an analysis
would give us a vague idea of the empirical distribution of the training data.
In this chapter, we think that even if the features of the input data space are related to model
accuracy and model robustness, they are not appropriate as a systematic training data
debugging method. We will study systematic methods for detecting outliers in training data.
Active Neurons
Neuron coverage is defined as the ratio of active neurons to the number of target neurons
considered [40]. 𝑀 𝑆 (𝑥) denotes the situation where the input signal (of x) propagates throughthe DNN model and activates each neuron. When the output of a particular neuron exceeds agiven threshold, we call it active, an active neuron.
Neuron coverage was initially proposed as a coverage criterion for coverage-driven test data
generation [40]. The active neurons for the input data 𝑥 provide a useful information in thatthey influence the output results. On the other hand, the neurons not involved in the predictive
inference process, are considered to be inactive. The input data that produce inactive neurons
do not effectively test all the neurons, and then new input test data are needed so that theyfurther activate the inactive neurons. When a set of input data makes all the neurons active, theset of test data are considered to reach 100% of the coverage.
After the original proposal in [40], there have been several research works to study thepractical usefulness of the neuron coverage as a test coverage criterion [41][42][43]. In
particular, it has been recognized that 100% of the neuron coverage is not difficult to achieve
and thus is weak as a test coverage criterion, which is similar to the case of the C0 criterion in30

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

conventional software testing methods.
On the other hand, we may consider that the training was appropriate in the first place,producing inactive neurons not involved in the predictive inference process. In this case, we can
add new input data to the training data and conduct re-training [40]. This suggests the idea ofusing the neuron coverage as a criterion for evaluating the quality of the model 𝑀 𝑆 . The
following is a discussion from the viewpoint of the neuron coverage as a model quality
evaluation criterion [44].
In DNN models 𝑀 𝑆 for classification tasks, the upstream layers near the input perform
encoding 𝐸 ′ (Encoding), and is followed by classifying 𝐶 ′ . Classifying is done after theencoding (𝑀 𝑆 = 𝐶 ′ ∘ 𝐸 ′ ); 𝑀 𝑆 (𝑥) = (𝐶 ′ ∘ 𝐸 ′ )(𝑥) = 𝐶 ′ (𝐸 ′ (𝑥)). When the output is a classification
probability vector, we place softmax functions in the final layer (logits) of the output; 𝑀 𝑆 =SOFTMAX ∘ 𝐶 ∘ 𝐸 ′ . Next, we may place a layer of Fully Connected Network (FCN) between 𝐸’
and 𝐶; 𝑀 𝑆 = SOFTMAX ∘ 𝐶 ∘ FCN ∘ 𝐸.
In FCN, a neuron in a layer considered is connected to all the neurons in the next layer, thus
the output is swap-invariant, which means that the output is preserved when the neurons areexchanged within the same layer. Therefore, the neuron coverage may be useful to summarize
the neuron activity in FCN layers. On the other hand, when the constituent neurons play a
specific functional role, such as in SOFTMAX or CNN, it is questionable whether the neuron
coverage, which considers all neurons equally, provides useful information. In fact, two different
definitions are studied for CNNs, and depending on which one is adopted, the value of neuron
coverage is different [45]. In this chapter, we consider neuron coverage for the FCN layer.
A series of experiments are conducted [45] in which training data are systematicallygenerated by means of a classical data augmentation method and the effects on neuron coverage
are investigated. The results showed that the difference in training data had influenced theneuron coverages at the 𝐸′ layer, while only a small effect was made on the 𝐶 layers. In addition,although the testing data are changed, very small differences are observed on the last layer in 𝐶
(Penultimate Layer of the whole model). It implies that the differences in the training data arereflected in the FCN layer where 𝐸 ′ = FCN ∘ 𝐸 as introduced early.
In addition, in previous experiments [32][44][46] in which we have measured the neuron
coverage on the FCN located as the last layer of 𝐶, we observed little correlation between theclassification prediction probability and neuronal coverage. Therefore, the neuron coverage may
be considered to represent an aspect independent of the information contributing to the model
accuracy. If it is found to be correlated with the model robustness, we can expect that theneuronal coverage on a particular layer is useful as a method to detect outliers for our purposes.

5.4 Experiments and Discussions
In our experiments, we used the machine learning model such that 𝑀 𝑆 = SOFTMAX ∘ 𝐶 ∘FCN ∘ 𝐸 , and the MNIST dataset. In particular, the training dataset 𝑆 was entered as theevaluation input, and the neuron coverage at the FCN layer was measured, whose results are31Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

shown in Figure 5.2.
Figure 5.2 (a) shows the frequency distribution of the neuron coverage for the input data(the result of KDE). Figure 5.2 (b) is a scatter plot of the neuron coverage of the input data on thehorizontal axis and the prediction probability of the same input data plotted on the vertical axis.
The red dots represent the data with correct predictions and the blue dots represent theincorrect one. The correct prediction rate of the training data and the test data were both 99%.

(a) Neuron coverage vs data frequency(b) Neuron coverage vs prediction probability
Figure 5.2 Analysis results of training data
Figure 5.2 (a) shows that the neuron coverage in the FCN layer is distributed between 0.38
and 0.84, and the median and mean values are 0.60. In Figure 5.2 (b), we confirmed again that
there is almost no correlation between the neuronal coverage and the predicted classification
probability. While there is a large difference in neuron coverage (between 0.38 and 0.84) , it canbe considered that the magnitude of the contribution to prediction classification differs
regardless of whether the answer is correct or incorrect. In other words, the neuron coverage
takes a large value even when the contribution to being incorrect may be large.
In Figure 5.2, the area circled by the ellipse, for example, represents the training data withlarger-than-average neuron coverage, resulting in a significant impact on the output, leading tothe observed classification probabilities. Therefore, it can be considered to be a faithfulrepresentation of the characteristics (classification probabilities) of the target data. Supposethat we chose training data with smaller-than-average neuron coverage. According to Figure 5.2(b), the predicted classification probabilities of the outputs are scattered, which is similar to thecase for the elliptical regions in Figure 5.2 (b), while it is not certain that the small neuron
coverage had an appropriate impact on the output. In other words, those training data may beconsidered not to play significant roles to obtain 𝑀 𝑆 . In this chapter, we consider the training
data that has small neuron coverage, regardless of the predicted classification probability, to be
an outlier.
We now sort the training data based on the neuron coverage and create training datasets
𝑆 (𝐾) of the same size. Then, we use 𝑆 (𝐾) to derive a trained training model 𝑀(𝐾) , andevaluate 𝑀(𝐾) by means of a common dataset for the evaluation. In this experiment, the entire

32Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

sorted training data was studied, and no adjustments were made at the level of individual data.

(a) Median output value vs. test data frequency (b) Neuron vs. magnitude of displacement
Figure 5.3 Neuron activity vector
Figure 5.3 shows the results of the five trained models 𝑀(𝐾) (K = 1, …, 5), in the form of theneuron activity vectors of the FCN layer, where the test dataset was used for the evaluation. The
neuron activity vector represents an internal feature, which is a multidimensional vector thatconsists of the output values of the constituent neurons.
The training datasets 𝑆 (𝐾) (K = 1, …, 5) are those obtained by random selection, right-side
editing (replacing the training data enclosed in the ellipse in the Figure 5.2), left-side editing,both-sides editing, random selection, and data augmentation after random selection. These
correspond to the bar graph from left to right in Figure 5.3 (a). The accuracies are 97.14%,
96.89%, 96.90%, 96.94%, and 95.81%; there is no significant difference except for the 𝑀(5) . The
fact that there is no difference from 𝑀(1) to 𝑀(4) is consistent with the way that the training
data sorting method (see Figure 5.2 (b)). In addition, because data augmentation is applied toobtain 𝑆 (5) and thus its distribution characteristics are somewhat different from those of thetest dataset used.
Figure 5.3 (a) shows bar graphs, and each bar represents the number of test data for which
the median of the vector components is within the range of values on the horizontal axis. It
represents that most of the test data have a small median value and thus a small contribution to
the predictive classification result. In addition, the data distribution in the 𝑆 (𝐾) differs, where𝑆 (1) can be considered to follow the distribution of the original training dataset 𝐷 because it israndomly selected. In Figure 5.3 (b), we consider this 𝑆 (1) as a reference, and for eachcomponent of the neuron activity vector, we show the difference between the case of 𝑆 (1) andthe others. As the difference is large, the effect by the deviation from the 𝑆 (1) distribution (andthus the original 𝐷) is large. In other words, it suggests that the effect of training data editing islarge.
From the above, it can be said that the training data could be re-arranged and crafted, basedon the neuron coverage, in such a way that the effect on the model accuracy is minute while thediscrepancy of the neuron activity vector is apparent (Figure 5.3 (b)). For example, 𝑆 (3) is theresult of left-side editing, in which the training data with small neuron coverage were removed.
Qualitatively, Figure 5.3 implies that 𝑆 (3) is considered to have contributed to the removal of33

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

outliers. In order to discuss further quantitatively, it is necessary to establish an empirical test
method standardized, a kind of test-time augmentation method, for the model robustness.

5.5 Conclusion
In FY2021, we studied debugging and testing methods of training data for the case wherebias in training data is the root cause of defects. Here, defects are judged from two quality
characteristics, the model accuracy and model robustness. In general, it is necessary to debug
training data considering that these two characteristics have a trade-off relationship. In the study,
the main points of the training data debugging method are studied in view of the notions
obtained for the membership inference, which has been discussed in the context of privacy
quality characteristics, and attributed to the problem of outlier detection. However, how todefine outliers is non-trivial. We proposed a method to estimate outliers in training data bycomputing the neuron coverage from the activated states inside the model and extracting theoutliers from the bias in the distribution of the neuron coverage values. Experiments showedthat the proposed method might provide a piece of information to aid debugging of training data.
In the future, we will establish a method to detect outliers for the purpose of training data
debugging through refining the experiment methodology presented here and conducting
experiments systematically. Furthermore, we will study methods to conduct debugging from theviewpoints of both model accuracy and model robustness.

34

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

6

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Evaluation and Improvement of Robustness
In this chapter, robustness means the ability that a machine-learned model keeps correct

output even when noise is added to input (including adversarial examples). For example, itevaluates how much noise can be added to the model without changing the correct results. Oneof the measures of its robustness is the maximum safe radius (MSR). In this chapter, we explain
adversarial example and the maximum safe radius in a classifier based on a feedforward neuralnetwork, and then report the results of a survey on techniques for estimating and increasing the
maximum safe radius.

6.1 Robustness measure (maximum safe radius)
It is well known that machine-learned models on inference programs mis-classify input data
even when very small perturbations are added. Such perturbated data are called adversarial
examples [47], and adversarial examples have been actively researched in recent years. The set
𝐴𝑑𝑣𝛿 (𝑥) of all adversarial examples contained in the 𝛿 -neighborhood (inside the sphere ofradius 𝛿 ∈ ℝ, where ℝ is the set of real numbers) of the input data sample 𝑥 ∈ ℝ𝑛 is defined
as follows:𝐴𝑑𝑣𝛿 (𝑥) = {𝑥′ | ‖ 𝑥 − 𝑥 ′ ‖ ≤ 𝛿 ⋀ 𝑓(𝑥) ≠ 𝑓(𝑥′)},where 𝑓(𝑥) is a function representing the machine-learned model that takes the input sample
𝑥 and return the classification, and ‖𝑥 − 𝑥′‖ is the distance between two data samples 𝑥 and
𝑥′. The 𝑝-norm is often used to define the distance.

Figure 6.1 An adversarial example from an image of a panda, which is mis-classified into a gibbon
Adversarial examples are explained by Figure 6.1. The left side in Figure 6.1 shows the input
space to the neural network and the right side shows the output space from the neural network.
The center of the red sphere in the input space represents an original input image of a panda,
and the inside of the sphere, whose radius is 𝛿 , (i.e., 𝛿 -neighborhood of the original image)35

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

represents the set of perturbated images obtained from the original image by adding noises
whose sizes are less than 𝛿. The set of outputs from the neural network for all the input images
in the 𝛿-neighborhood corresponds to the red region in the output space on the right. Here, apart (lower-right) of the red region in the output side is beyond the decision boundary and ismapped into the region of gibbons. It means misclassification, and the input images mapped to
the lower-right part are adversarial examples.
If there is no adversarial example in the 𝛿-neighborhood of the input data 𝑥 (i.e., inside thesphere whose radius is 𝛿 and center is 𝑥), then 𝛿 is said to be the safe radius of 𝑥. Then, themaximum safe radius of 𝑥, denoted by 𝑀𝑆𝑅(𝑥), is defined as follows:
𝑀𝑆𝑅(𝑥) = max {𝛿 | 𝐴𝑑𝑣𝛿 (𝑥) = ∅}.
When the maximum safe radius of 𝑥 is large, it is difficult to generate adversarial examples.
Therefore, the maximum safe radius can be used as a measure of the robustness to input
perturbations, including adversarial examples, of machine-learned models.
The radius 𝛿 in Figure 6.1 is not a safe radius because some perturbated input images insidethe 𝛿 -neighborhood are misclassified into gibbons. On the other hand, 𝛿 in the following
Figure 6.2 is the maximum safe radius because all the input images inside the 𝛿-neighborhoodin Figure 6.2 are correctly classified.

Figure 6.2 The maximum safe radius 𝛿

6.2 A survey on methods for evaluation and improvement of robustness
Table 6.1 shows recent research papers on methods for evaluation and improvement ofrobustness, where each small box in the table represents a research paper with reference andthe information on neural networks used in the experiments for evaluating the methods
proposed in the paper. The information is useful for comparing applicable scales of the methods.
Table 6.1 is categorized by the following perspectives:

36

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Table 6.1 Methods for evaluation and improvement of robustness (MSR: Maximum Safe Radius)
Evaluation of robustness

Improvement of robustness

Rigorous estimation of MSR
Katz et al. 2017 (Reluplex) [48]ACAS-XU-DNN, 300 ReLU nodes
6 hidden layers,

Rigorous(Limitation: hundreds of nodes)
Tjeng et al. 2019 [49]
CIFAR-10, ResNet, 9-CNN, 2-layer,
107,496 ReLU units,
100~1,000 times faster than Reluplex

Deterministic

CertifiedEstimation of a lower bound (LB) of MSR

Training by detecting all the adversarial exes

Weng et al. 2018 (Fast-Lin) [50]

Wong and Kolter 2018 [55]

CIFAR, 6-layer, 12,288 ReLU units

SVHN (32x32x3), 2-conv, 32-ch,

About 10,000 times faster than Reluplex

100, 10 hidden units, ReLU,(Non-applicable to ImageNet)

Boopathy et al. 2019 (CNN-Cert)[51]
CIFAR-10 (32x32x3), 5-layer,
10 filters, 29,360 hidden nodes,

Approximative

Faster than Fast-Lin

Probabilistic

Estimation of a probabilistic LB of MSR

Randomized smoothing after training

Weng et al. 2019 (PROVEN) [52]Lecuyer at el. 2019 [56]CIFAR, 5-layer, CNN, ReLU

ImageNet (299x299x3),

almost same as CNN-Cert

Inception-v3 + auto-encoder
Cohen at el. 2019 [57]ImageNet (299x299x3),
ResNet-50 (50-layer)
Tighter certification than Lecuyer [56]Uncertified

Estimation of an upper bound (UB) of MSR

Training by detecting near adversarial exes

Carlini and Wagner 2017 [53]Madry et al. 2018 [58]ImageNet (299x299x3),

CIFAR (32x32x3),

Inception-v3

28-10 wide ResNet

Estimation of an approximation of MSR
Weng et al. 2018 (CLEVER) [54]ImageNet (299x299x3),
ResNet-50 (50-layer)

37

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition––

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Columns in Table 6.1 (application):➢

Evaluation of robustness by estimating MSR➢

Improvement of robustness by increasing data samples with a specified MSR

Row in Table 6.1 (certification and strictness):➢➢

Certification of no existence of adversarial examples in 𝛿-neighborhood


Rigorous estimation of MSRApproximative estimation of MSR⚫

Deterministic (no adversarial example exist)⚫

Probabilistic (the probability of no adversarial example is 𝜌%)

No certification of no existence of adversarial examples in 𝛿-neighborhood

The methods in Table 6.1 are explained in the following Subsections 6.2.1~6.2.7.
Certified and rigorous evaluation of robustness
Katz et al. [48] proposed a method, Reluplex, to verify that a machine-learned model satisfies
given properties. A demonstration tool that implements the method Reluplex has also beenreleased. Properties are constraints on input-output relations of machine-learned models, andReluplex can exhaustively and rigorously (soundly and completely) verify that there is no
adversarial example in the 𝛿-neighborhood of the input data sample. Therefore, the maximum
safe radius (MSR) can be estimated by checking the existence of adversarial examples bychanging the radius 𝛿 with binary search. Reluplex is an extended Simplex method (one ofsolvers for linear programming problems) with rules for the ReLU function and it is implemented
by a satisfiability-checking tool (SMT-Solver) with a module for the theory of real numbers.
Reluplex is a powerful tool to verify properties in addition to robustness, but the computational
cost is expensive and the number of neurons it can verify is a few hundred ReLUs at most.
Tjeng et al. [49] proposed an efficient method for estimating maximum safe radii. Then, they
implemented the method on a mixed integer linear programming (MILP) solver anddemonstrated that the tool can exactly estimate the maximum safe radii of a neural network with
100,000 ReLU-type neurons. Although it is still difficult to apply the rigorous solver-based tools
to practical large-scale machine-learned models, the scalability is being improved.
Certified, approximative, and deterministic evaluation of robustness
Weng et al. [50] proposed a method, Fast-Lin, to approximate the maximum safe radii of
ReLU-type neural network. Fast-Lin linearly approximates the output region with a polytope andestimates an approximation δ that is slightly smaller than the maximum safe radius, as shown in
Figure 6.3. It is guaranteed that there is no adversarial example inside the 𝛿 -neighborhood
because the approximation δ does not exceed the maximum safe radius (i.e. sound). It means δ
is a safe radius and is a lower bound of the maximum safe radius (𝛿 ≤ 𝑀𝑆𝑅(𝑥)). It was reported
that Fast-Lin is 10,000 times faster than the rigorous method Reluplex by approximative convex

38

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

outer polytopes.

Figure 6.3 An approximation 𝛿 that is slightly smaller than the maximum safe radius (MSR)
Boopathy et al. proposed CNN-Cert, which is an improved version of Fast-Lin [51]. CNN-Certalso supports convolutional networks including not only the activation function ReLU but also
sigmoid, tanh, and arctan, and it improves approximation accuracy and is faster than Fast-Lin.
Certified, approximative, and probabilistic evaluation of robustness
Weng et al. [52] proposed a method, PROVEN, to approximate probabilistic maximum safety
radii. As shown in Figure 6.4, the probabilistic maximum safe radius 𝛿 with a probability 𝜌means that there is no adversarial example inside the 𝛿-neighborhood with a probability 𝜌. In
other words, it permits the existence of adversarial examples with the probability (1 − 𝜌) .
PROVEN has been developed based on CNN-Cert, and the computational complexity has not
significantly increased from CNN-Cert.

Figure 6.4 An approximation 𝛿 that is slightly smaller than the probabilistic MSR with 𝜌

39Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Uncertified evaluation of robustness
Carlini and Wagner [53] proposed a method to detect the (almost) closest adversarial
example to the input data sample 𝑥 and estimate the distance 𝛿 as an approximative
maximum safety radius by using an existing optimization tool (Adam). However, it is notguaranteed that the distance 𝛿 estimated by the method is the shortest distance to theadversarial example, and there is a possibility that there are adversarial examples closer thanthe distance. In other words, it is an upper bound of the maximum safe radius (𝑀𝑆𝑅(𝑥) ≤ 𝛿).
Although it is not guaranteed that the distance δ estimated by the method is a safe radius, it isoften used for evaluation in recent papers on robustness as a measure of the maximum safe
radius.
Weng et al. [54] proposed the method CLEVER to estimate an approximate maximum safe
radius as an evaluation measure of robustness independent of attack methods. It was reported
that the method could be applied to relatively large neural networks and the image recognition
model Inception-v3 was evaluated in about 10 seconds. The method estimates an approximative
maximum safe radius based on the maximum effect in output caused by small changes in input,
where the maximum effect is approximated by the extreme value theory. As shown in Figure 6.5,
the estimated value 𝛿 can be larger than the maximum safe radius, and thus there is apossibility that adversarial examples exist inside the 𝛿-neighborhood (i.e., it is not guaranteedthat 𝛿 is the safe radius).

Figure 6.5 An approximation of the maximum safe radius (uncertified)Certified, approximative, and deterministic improvement of robustness
Wong et al. [55] proposed a method (robust training) to train such that the maximum safe
radius of each data in the training dataset to be a specified value 𝛿. Although this method does
not guarantee that the maximum safe radius δ is obtained for every training data sample after
training, it also gives a method to estimate an approximative value (a safe radius) of the
maximum safe radius for each input data sample. In the robust training, neural networks try tolearn such that they correctly make inferences for not only training data samples but also the 𝛿neighborhood of every sample.

40

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

A sketch of the robust training is shown in Figure 6.6, where the black dotted line in the
output space represents the decision boundary learned by a normal training, and the red solidline represents the decision boundary learned by the robust training. The six training data
samples in the input space are correctly classified by both the boundaries, but some data in the𝛿-neighborhood of each sample are misclassified by the dotted boundary (normal training). On
the other hand, data in the 𝛿 -neighborhood of each sample are also learned in the robust
training as shown in the red boundary. The robust training can guarantee some safe radii, but it
is difficult to apply the training to practical large scale neural networks due to the low scalability.
Wong et al. [55] reports that the robust training was successfully applied to the datasets ofimages, MNIST (28 × 28) and SVHN (32 × 32) but was not applicable to ImageNet (256 × 256).

Figure 6.6 Robust-trining by input data with 𝛿-neighberhood
Certified, approximative, and probabilistic improvement of robustness
Lecuyer et al. [56] proposed a method to estimate maximum safe radii that can beprobabilistically guaranteed by randomized smoothing. In the randomized smoothing, theinference for the same input is repeated in a neural network where a noise layer is added after
training, and the final output is the average of the outputs obtained by the repeated inferences.
A sketch of the randomized smoothing is shown in Figure 6.7, where the black dotted line in
the output space represents the decision boundary without randomized smoothing, and the redsolid line represents the decision boundary with randomized smoothing. The randomizedsmoothing of Lecuyer et al. [56] improves robustness by smoothing decision boundaries withcertification of safe radii and has been successfully applied to guarantee the robustness ofmachine learned models for large-scale input data such as ImageNet (299 × 299 × 3). When thevariance of the added noise is increased, the guaranteed safe radius also increases, but on theother hand, the correctness (e.g., accuracy) decreases. Lecuyer et al. [56] applied the techniqueof differential privacy, where the output for two similar inputs is made statisticallyindistinguishable, to clarify the relations between certifiable approximative probabilistic
maximum safe radii, the standard deviation of noise, the number of inferences, and so on.
Cohen et al. [57] proposed a randomized smoothing based method that can estimate tighter
certifiable approximative probabilistic maximum safe radii than one of Lecuyer et al. [56].

41

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Although randomized smoothing needs repeated inferences (tens or hundreds of times
experimentally) for an input, it can probabilistically guarantee robustness even for large-scale
networks.

Neural network
Output

input
...

...

...

...

...

noise for smoothing

Figure 6.7 Improvement of robustness by randamized smoothing
Uncertified improvement of robustness
Madry et al. [58] proposed a method (adversarial training) to train such that maximum safe
radius of each data in the training dataset to be a specified value 𝛿. In the adversarial training,
samples to be potentially adversarial examples in 𝛿-neighborhood are detected during training
and are also used as training data. Compared to the robust training of Wong et al. [55], theadversarial training cannot guarantee robustness, but it is more applicable to larger networks.
In addition, compared to randomized smoothing, the adversarial training does not require
repeated inferences.

6.3 Conclusion
In general, improvement of robustness tends to decrease accuracy, and currently accuracy isoften more important. However, if robustness is not considered, accuracy may rapidly decrease
even by small input perturbations. Therefore, robustness is important in critical systems. The
methods related to the maximum safe radius, which is a measure of robustness, explained in thischapter have been proposed recently, and environments for applying such methods have notbeen established well yet. Since such methods have been experimentally applied also to practicalmachine learned models, we think that the maximum safe radius can be one of measures ofrobustness in a few years.

42

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

7

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Generalization Bounds of Machine-Learned Models
In this chapter, we briefly introduce well-known theorems on generalization bounds, that are

the expected values of the error rates of machine-learned models for all input data, and we show
some computational results obtained by applying the theorems. We are aiming at guaranteeing
the behavior of machine-learned models even for unseen input data.

7.1 Generalization bounds
In this chapter, we model machine-learned neural networks as shown in Figure 7.1, with the
input-output relations. In particular, we focus on feed-forward neural networks trained asclassifiers by supervised deep learning and denote the input-output relation as a function 𝑦 =𝑓𝑤 (𝑥), where 𝑥 and 𝑦 are an input and the correct output, respectively, and 𝑤 is the weights(training parameters) on connections between neurons in the neural network. In the field ofstatistical learning theory, the function 𝑓𝑤 is often called a hypothesis, but it is called a machinelearned model in this chapter, as in the other chapters. Since a neural network can express
multiple machine-learned models 𝑓𝑤 by adjusting weights 𝑤 ∈ 𝒲 , the set of expressible
machine-learned models 𝑓𝑤 in the network is denoted by ℱ. Hence, Machine learning means to
select a machine-learned model 𝑓𝑤 from ℱ by a training algorithm such that 𝑓𝑤 fits thetraining dataset. The model 𝑓𝑤 can be denoted by 𝑓 when the parameter 𝑤 is not important.

Figure 7.1 Generalization error and empirical error
The generalization error 𝐿(𝑓𝑤 ) of the machine-learned model 𝑓𝑤 (a classifier) is the
expected value of the error rate of 𝑓𝑤 for every pair (𝑥, 𝑦), of an input 𝑥 and the correct output
𝑦, randomly selected according to the distribution 𝒟, and is defined as follows:43

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

𝐿(𝑓𝑤 ) = 𝔼(𝑥,𝑦)~𝒟 [𝕀(𝑦 ≠ 𝑓𝑤 (𝑥))],where 𝕀(𝑏) is the following indicator function, that returns 0 if 𝑏 is true and returns 1
otherwise, and therefore 𝕀(𝑦 ≠ 𝑓𝑤 (𝑥)) is the 0-1 loss function:𝕀(𝑏) = if (𝑏 = true) then 1 else 0.
For example, at the top left in Figure 7.1, the input space, that is drawn in 2 dimensions for
simplicity but exactly has 784 dimensions, means “the distribution 𝒟 of images (28 × 28 =784 pixels with 256 grayscale) looking like numbers.” In this case, the generalization error isthe expected value of the error rate of 𝑓𝑤 for every image looking like numbers. Here, note that
it is not the error rate for all the (256784) images in the input space.
The empirical error 𝐿̂𝑆 (𝑓𝑤 ) of the machine-learned model 𝑓𝑤 is the error rate of 𝑓𝑤 for 𝑚
input data samples in the dataset 𝑆~ 𝒟 𝑚 selected according to distribution 𝒟:
𝐿̂𝑆 (𝑓𝑤 ) =1∑ [𝕀(𝑦 ≠ 𝑓𝑤 (𝑥))].
𝑚(𝑥,𝑦)∈𝑆

Especially, if 𝑆 is the training dataset, then 𝐿̂𝑆 (𝑓𝑤 ) is also called the training error, and if 𝑇 is
the testing dataset, then 𝐿̂ 𝑇 (𝑓𝑤 ) is also called the testing error.

7.2 The theory of generalization bounds
Even though it is almost impossible to exactly compute generalization errors because there
are innumerable data in input-spaces, various theorems on generalization bounds, that are(upper) bounds of generalization errors, have been proposed for guaranteeing that “thegeneralization error of a machine-learned model is less than a generalization bound withprobability at least 𝑝% ,” where p is the confidence of the bound. In this section, we briefly
classify the theorems on the generalization bounds in Subsection 7.2.1, and then we explain thetheorems in Subsections 7.2.2~7.2.5.
A classification of generalization bounds
In this subsection, according to the classification of the generalization bounds as shown in
Figure 7.2, we briefly explain the feasibility for applying generalization bounds to the evaluationof generalization performance of machine-learned models. In Figure 7.2, it seems difficult toapply the VC bounds or the Rademacher bounds for evaluating the generalization performanceof (trained) machine-learned models because they give upper bounds of the worst model in theset ℱ of expressible machine-learned models in a given neural network. The PAC-Bayes bounds
are effective for comparing the generalization performance of two or more machine-learned
models [59][60] even though the absolute values of the bounds often are vacuous (close to or
more than 100%). The Chernoff bounds can give upper bounds very close to the generalization
errors by providing a large amount of testing data separately from the training dataset.

44

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Independent oftraining datasets
Independent oftraining algorithms
Based ontraining errors
Dependent on
training algorithms

Generalization
boundsBased ontesting errors

Dependent on
training datasets

e.g., VC boundse.g., Rademacher boundse.g., PAC-Bayes boundse.g., Chernoff bounds

Figure 7.2 A Classification of generalization bounds and well-known examples of bounds
In most cases, generalization bounds based on training errors (i.e., training datasets) is larger
than generalization bounds based on testing errors (i.e., testing datasets). This tendency is also
seen not only for the PAC-Bayes bounds but also for the others such as bounds based on output
margins [61], bounds based on the stability of the training algorithm [62], and so on.
The advantage of generalization bounds based on training errors is that they can be applied
to the study of training (algorithms) for reducing generalization errors. The other advantage isthat they can be computed only by training datasets without additional datasets such as testingdatasets. For example, when the number of data samples is very small (e.g., a few dozen), it wasreported that the PAC-Bayes bounds, where all samples were used for training, were able to belower than the Chernoff bounds, where the samples were separated for training and testing [63].
Recently, several methods have been proposed for computing non-vacuous generalization
bounds (less than 100%) even based on training errors. For example, such methods use
distributions of machine-learned models (i.e., input-output functions instead of weights) in the
PAC-Bayes bounds [64], or random labelled data in training [65], or model compression [66].
Furthermore, methods have also been proposed to optimize the distribution (the mean and thestandard deviation) of each weight to reduce generalization errors using the PAC-Bayes bounds
as objective functions [67][68]. In the near future, it is expected that generalization bounds
based on training errors will also be effective as a measure of the generalization performance ofmachine-learned models, but currently, we consider that it is more realistic to adopt
generalization bounds based on testing errors.
VC bounds
The VC dimension 𝑉𝐶(ℱ) is a complexity measure of the set ℱ of expressible machinelearned models and it means the maximum number of data that can be divided by ℱ [69]. Then,by using the VC dimension 𝑉𝐶(ℱ), the theorem, VC bounds [59], guarantees that the following
inequality holds with probability (1 − 𝛿) at least, for any training data set 𝑆~𝒟 𝑚 (size: 𝑚)and for any machine-learned model 𝑓 ∈ ℱ:

45

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

1
𝑉𝐶(ℱ) √ln 𝛿̂
𝐿(𝑓) ≤ 𝐿𝑆 (𝑓) + 144√+,𝑚𝑚
where 𝛿 ∈ (0,1) represents the uncertainty of the inequality.
Most of computation results of VC-bounds exceed 100% because the bounds consider the
worst case such that any model is selected from the set ℱ for any training dataset when the setℱ of expressible machine-learned models (i.e., architecture of a neural network) is given, inother words, the bounds are independent of training dataset and training algorithm. Therefore,
it is not appropriate to use it to evaluate the generalization performance of trained machinelearned models.
Rademacher bounds
The Rademacher complexity is the complexity 𝑅(𝑆, ℋ) of the set ℱ of expressible machinelearned models for the dataset 𝑆 [69]. Then, by using the complexity 𝑅(𝑆, ℋ), the theorem,
Rademacher bounds [59], guarantees that the following inequality holds with probability (1 −𝛿) at least, for any training data set 𝑆~𝒟 𝑚 (size: 𝑚) and for any machine-learned model 𝑓 ∈
ℱ:
4
2 ln
√
𝛿
𝐿(𝑓) ≤ 𝐿̂𝑆 (𝑓) + 2𝑅(𝑆, ℱ) + 4𝑐,𝑚where 𝑐 is a constant.
The Rademacher bounds are lower than the VC bounds because the training dataset 𝑆 isconsidered, but the most of bounds also exceed 100% because the bounds are still independentof training algorithm. Therefore, it is not appropriate to use the Rademacher bounds to evaluate
the generalization performance of trained machine-learned models by the same reason as the
VC bounds.
PAC-Bayes bounds
PAC (Probably Approximately Correct) represents that a machine-learned model 𝑓𝑤 trainedby a training dataset is an approximation of the correct model and the generalization error gap
is less than a threshold with a probability. PAC-Bayes considers the expected value 𝔼𝑤~𝑄 [𝐿(𝑓𝑤 )]of the generalization error of the probabilistic machine-learned model 𝑓𝑤 whose weights 𝑤
are randomly selected according to the probability-distribution 𝑄 instead of fixed values.
Although several theorems on the PAC-Bayes bounds have been proved, two well-known
theorems (Catoni bounds and Maurer bounds) are introduced in this subsection. The theorem,
Catoni bounds [70], guarantees that the following inequality holds with probability (1 − 𝛿) atleast, for any posterior distribution 𝑄 and for any 𝛽, 𝛿 > 0 , when a training dataset 𝑆~𝒟 𝑚(size: 𝑚) and a prior distribution 𝑃 whose weights independent of 𝑆 are given:

46

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

𝔼𝑤~𝑄 [𝐿(𝑓𝑤 )] ≤National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

11
1(1 − exp (−𝛽𝔼𝑤~𝑄 [𝐿̂𝑆 (𝑓𝑤 )] − (KL(𝑄 ∥ 𝑃) + ln ))).
1 − exp(−𝛽)𝑚𝛿The theorem, Maurer bound [71], guarantees that the following inequality holds with probability(1 − 𝛿) at least, if 𝑚 ≥ 8:
𝔼𝑤~𝑄 [𝐿(𝑓𝑤 )] ≤ 𝑘𝑙 −1 (𝔼𝑤~𝑄 [𝐿̂𝑆 (𝑓𝑤 )],1
2√𝑚(𝐾𝐿(𝑄 ∥ 𝑃) + ln (
))),
𝑚
𝛿

where 𝐾𝐿(𝑄 ∥ 𝑃) is the KL-divergence (Kullback-Leibler divergence) that shows the difference
between the two distributions 𝑄 and 𝑃, and it is defined as follows:
𝐾𝐿(𝑄 ∥ 𝑃) = ∫ 𝑄(𝑤) ln (𝒲

and 𝑘𝑙−1 (𝑞,

𝑄(𝑤)
) 𝑑𝑤,
𝑃(𝑤)

𝑏) is the binary KL-inversion defined by𝑘𝑙−1 (𝑞, 𝑏) = sup{ 𝑝 ∈ [𝑥, 1] ∶ 𝑘𝑙(𝑞 ∥ 𝑝) ≤ 𝑏 },where 𝑘𝑙(𝑞 ∥ 𝑝) is the binary KL-divergence (the KL-divergence of the Bernoulli distributionsof 𝑞 and 𝑝) defined as follows:
𝑞𝑘 (1 − 𝑞)1−𝑘𝑞
1−𝑞𝑘𝑙(𝑞 ∥ 𝑝) = ∑ 𝑞𝑘 (1 − 𝑞)1−𝑘 ln ( 𝑘) = 𝑞 ln ( ) + (1 − 𝑞) ln ().
𝑝 (1 − 𝑝)1−𝑘𝑝1−𝑝𝑘∈{0,1}As shown in the Catoni bounds and the Maurer bounds, the PAC-Bayes bounds contain theposterior distribution 𝑄 of the trained machine-learned model (i.e., they depend on the
training algorithms). Compared with the VC bounds and the Rademacher bounds, the PAC-Bayes
bounds can give generalization bounds closer to the generalization errors, but the computation
results of the PAC-Bayes bounds are often close to 100% (i.e., vacuous).
Chernoff bounds
There are generalization bounds based on testing errors for the dataset 𝑇~𝒟 𝑚 (size: 𝑚)that is not used in training (i.e., held-out dataset prepared for evaluation). For example, thetheorem, Chernoff bounds [72], guarantees that the following inequality holds with probability(1 − 𝛿) at least:𝐿(𝑓𝑤 ) ≤ 𝑘𝑙 −1 (𝐿̂ 𝑇 (𝑓𝑤 ),11ln ( ) ),𝑚𝛿

where 𝑘𝑙 −1 (𝑞, 𝑏) is the binary KL-inversion explained in Subsection 7.2.4. Here, note that thetesting error 𝐿̂ 𝑇 (𝑓𝑤 ) is used for expressing the upper bounds. The Chernoff bounds can give tightgeneralization bounds (i.e., close to the generalization errors) when sufficient testing dataset are
prepared.

47

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

7.3 Computational examples of generalization bounds
In this section, we report some computational examples of generalization bounds based on
the theorems introduced in Section 7.2. At first, in Subsection 7.3.1, it is explained how tocompute generalization bounds. Next, in Subsection 7.3.2, the computation results are shown.
Computation of generalization bounds
As introduced in Subsection 7.2.5, the Chernoff bounds are useful for evaluating thegeneralization performance of trained machine-learned models because they can give tightgeneralization bounds, thus close to the generalization errors, and therefore the bounds aremeaningful as absolute values. On the other hand, it has also been reported [59][60] that thegeneralization bounds, called perturbation bounds, of machine-learned models whose weights
are perturbated, for example, by the Gaussian noise as shown in Figure 7.3, are useful as
generalization measures for relatively comparing the generalization performance of machinelearned models. In this section, the standard deviation 𝜎𝑖 of the Gaussian noise 𝒩(0, 𝜎𝑖2 )added to each weight 𝑤𝑖 is decided to be proportional to the magnitude of 𝑤𝑖 such that 𝜎𝑖 =𝑟|𝑤𝑖 |, where 𝑟 is a positive constant, called SD-rate. Such addition of noise is called magnitudeaware perturbation [59].

Figure 7.3 A machine-learned model whose weights are perturbated by the Gaussian noise
The theorems on the PAC-Bayes bounds can be applied for computing such generalization
bounds of machine-learned models with probability distribution of perturbated weights. In thissection, we use the posterior distribution 𝑄 equal to the prior distribution 𝑃 trained by atraining dataset 𝑆 in the PAC-Bayes bounds (i.e., 𝑄 = 𝑃). As explained in Subsection 7.2.4, theposterior distribution 𝑄 can depend on the dataset used for computing generalization bounds,
while the prior distribution 𝑃 cannot depend on it. Hence, we compute the generalization
bound by using a testing dataset 𝑇, separated from 𝑆, in this case of 𝑄 = 𝑃. The advantage of
𝑄 = 𝑃 is that the KL-divergence 𝐾𝐿(𝑄 ∥ 𝑃) is zero. In general, one of the reasons why the PACBayes bounds are close to 100% is that the KL-divergences are large. In this section, we apply
the PAC-Bayes bounds in the case of 𝑄 = 𝑃 to the computation of generalization bounds based48

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

on testing errors (i.e., by using testing datasets).
In order to apply the PAC-Bayes bounds, it is necessary to compute the expected value
𝔼𝑢~𝒩(0,𝑟 2 )𝜔 [𝐿̂ 𝑇 (𝑓𝑤⨁𝑢 )] of the testing error of the machine learned model 𝑓𝑤 with noise 𝑢 ,where 𝜔 is the total number of weights (training parameters), 𝑟 is the rate of noise to weight
magnitude (i.e., SD-rate), and 𝑤⨁𝑢 represents that the noise 𝑢 is added to weights 𝑤
proportionally to the magnitude of each weight element (i.e., for each element 𝑖 ∈ {1, … , 𝜔} ,(𝑤⨁𝑢)𝑖 = 𝑤𝑖 + 𝑢𝑖 |𝑤𝑖 | ). However, since it is difficult to exactly compute the expected value, we
compute the upper bound of the expected value from the average of testing errors computedwith noise in 𝑛 times. The upper bound 𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ) can be defined as follows, for the set 𝑈 ={𝑢𝑖 | 𝑖 ∈ {1, … , 𝑛}} of randomly sampled 𝑛 Gaussian noises 𝑢𝑖 ~ 𝒩(0, 𝑟 2 )𝜔 :𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ) = 𝑘𝑙−1 (112∑ 𝐿̂ 𝑇 (𝑓𝑤⨁𝑢 ) ,
ln ),
|𝑈|
|𝑈| 𝛿 ′
𝑢∈𝑈

where 𝛿 ′ is the uncertainty for using the average of testing errors instead of the expected value.
Indeed, the following inequality holds with probability (1 − 𝛿 ′ ) at least (e.g. see Section 6 in[68]):
𝔼𝑢~𝒩(0,𝑟 2 )𝜔 [𝐿̂ 𝑇 (𝑓𝑤⨁𝑢 )] ≤ 𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ).
Consequently, in this section, we use the following three expressions for computing
generalization bounds using a testing dataset 𝑇 (size: 𝑚),(1) Chernoff bounds: 𝑘𝑙 −1 (𝐿̂ 𝑇 (𝑓𝑤 ),(2) Catoni bounds:11ln ( ) ) ,
𝑚
𝛿

1
1
1
(1 − exp (−𝛽 𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ) −
ln (
))) ,
1 − exp(−𝛽)𝑚𝛿 − 𝛿′(3) Maurer bounds: 𝑘𝑙 −1 (𝐿̅𝑇,𝑈,𝛿′ (𝑓𝑤 ),

1
2√𝑚
ln (
)),
𝑚
𝛿 − 𝛿′

where 𝐾𝐿(𝑄 ∥ 𝑃) in the Catoni bounds and the Maurer bounds has disappeared because it iszero if 𝑄 = 𝑃. Note that the denominators in the logarithms in the expressions (2) and (3)are replaced by (𝛿 − 𝛿 ′ ) from 𝛿 because a part of the uncertainty 𝛿 of generalization bounds
must be used as the uncertainty 𝛿 ′ of expected values. In this section, the uncertainty 𝛿 ′ isobtained as the solution of the following equation (e.g., by the Newton method) for
approximately minimize the expression (3):𝑚𝑚

(21− 𝑛 √𝑚 ) 𝛿 ′ 𝑛 + 𝛿 ′ − 𝛿 = 0,
where 𝑚 is the size of the testing dataset and 𝑛 is the number of testing errors repeatedly
computed with noise added. Similarly, the parameter 𝛽 is given for approximately minimizing
the expression (2) as follows:
1
)
𝛿
𝛽=√
.
𝑚 𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ) (1 − 𝐿̅ 𝑇,𝑈,𝛿′ (𝑓𝑤 ))2 ln (

49

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

The binary KL-inversion 𝑘𝑙 −1 (𝑞, 𝑏) in the expressions (1) and (3) can be approximately
computed, for example, by the Newton method (e.g., see Appendix C in [67]).
Computational results of generalization bounds
In this subsection, we report the computational results of the generalization bounds by theexpressions (1), (2), and (3) described in Subsection 8.3.1, for the following two types ofneural networks MLP and CNN trained on the dataset MNIST (pixels: 28 × 28, grayscale: [0,1],
training size: 27,000, and testing size: 10,000) of handwritten digit images.
––

MLP (Multi-Layer Perceptron):➢

The total number of the training parameters: 118,282➢

The layers: 3 fully connected layers

CNN (Convolutional Neural Network):➢

The total number of the training parameters: 121,930➢

The layers: 2 convolutional layers, 2 pooling layers, and 2 fully connected layers

For comparing generalization performance, 4 machine-learned models for each neuralnetwork MLP/CNN were trained in the parameters shown in Table 7.1
Table 7.1 The training parameters in the 4 machine-learned models
Model ID

Dropout rate

𝐿2-Regularization

#1

0

0

#2

0

0.001

#3

0.2

0

#4

0.2

0.001

Figure 7.4 shows the computational results of the generalization bounds for the machinelearned models of MLP and CNN (4 models for each), for the SD-rates 𝑟 = 0, … , 0.7, where theChernoff bounds are used for the case 𝑟 = 0 (no noise) and the Catoni bounds are used for thecases 𝑟 = 0.1, … , 0.7. The computational results by the Maurer bounds are omitted because they
were always almost 1% higher than the Catoni bounds. The confidence (1 − 𝛿) , i.e., theprobability that the generalization errors are less than the generalization bounds, is 90% in
Figure 7.4. The upper bounds of expected values of testing errors with noise added are computedfrom the averages of 5,000 testing errors with noise added for 10,000 test data samples (i.e.,𝑚 = 10,000 and 𝑛 = 5,000). For the cases of 𝑟 = 0 and 0.3 in Figure 7.4, the generalization
gaps (the difference between the generalization bounds and the testing errors) and the
expectation gaps (the difference between the expected values of testing errors and the averages)
are shown in Figure 7.5 and Figure 7.6, respectively.

50

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002(a) The generalization bounds of MLP#1~4(b) The generalization bounds of CNN#1~4
Figure 7.4 The computational results by the Chernoff (𝑟 = 0) and the Catoni bounds (𝑟 > 0)(a) The generalization bounds of MLP#1~4(b) The generalization bounds of CNN#1~4
Figure 7.5 The generalization gaps in the Chernoff (𝑟 = 0)(a) The generalization bounds of MLP#1~4(b) The generalization bounds of CNN#1~4
Figure 7.6 The generalization gaps and the expectation gaps in the Catoni bounds (𝑟 = 0.3)

51Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

As shown in Figure 7.5, the generalization bounds by the Chernoff bounds are very close to
the testing errors and the values of the bounds are meaningful on their own (as absolute values).
On the other hand, as a relative evaluation between machine-learned models, the generalization
bounds in Figure 7.5 do not show any difference from the (normal) testing errors by data
samples. For example, the generalization bound of CNN#2 with 𝐿2-regularization is larger thanthe bound of CNN#1 as shown in Figure 7.5 (b). It seems that the generalization bounds are notenough to evaluate generalization performance. Then, in order to evaluate the generalization
performance at different viewpoints from data samples, it is useful to add noise to weights inneural networks. Figure 7.4 and Figure 7.6 clearly show that the 𝐿2 -regularization and thedropout can suppress increase of the generalization bounds when noise increases. Althoughsuch suppression-effect is empirically well-known, it is an advantage of the generalization
bounds that they can quantitatively evaluate such effect and can probabilistically guarantee theupper bounds of the generalization errors according to the statistical learning theory.
In the rest of this subsection, as an example of generalization bounds based on training errors,
we show a computational result of MLP#4 (i.e., with the dropout and the regularization). In the
training of MLP#4, each initial weight 𝑤0𝑖 was randomly selected according to the normal
2
distribution 𝑃𝑖 = 𝒩(0, 𝜎0𝑖) , i.e., the mean is 0 and the standard deviation is 𝜎0𝑖 . For eachweight 𝑤𝑖 , the KL-divergence 𝐾𝐿(𝑄𝑖 ∥ 𝑃𝑖 ) between the prior distribution 𝑃𝑖 and the posterior
distribution 𝑄𝑖 = 𝒩(𝑤𝑖 , (𝑟|𝑤𝑖 |)2 ) is equal to or larger than (1/2𝑟 2 ). Here, the equality holds if
𝜎0𝑖 = 𝑟|𝑤𝑖 |, but we cannot assume the equality because 𝜎0𝑖 cannot depend on the training
dataset (i.e., on 𝑤𝑖 ). Then, the minimum of the total KL-divergence of a neural network is(𝜔/2𝑟 2 ) , where 𝜔 is the total number of weights. For example, the minimum of the KLdivergence of MLP#4 (𝜔 = 118,282) in the case of 𝑟 = 0.5 is 236,564. Then, the generalization
bound of the MLP#4 (𝑟 = 0.5 ) computed by the Catoni bounds is 99.99% with probability
90% at least, although the average of 5,000 training errors was 2.17%. The several techniques
for reducing the generalization bounds based on training errors have been proposed, but it isthought to be currently practical to use testing datasets (i.e., the generalization bounds based on
testing errors).

7.4 Towards the evaluation of “the stability of trained models”
The stability of trained models is one of the nine internal quality characteristics described in
Machine Learning Quality Management Guideline [1] and it represents that machine-learned
components reasonably behave even for unseen input data. The theorems and the computations
on the generalization bounds explained in this chapter make it possible to theoreticallyguarantee the probabilistic upper bounds of the generalization errors (i.e., expected values oferror rates for all data including unseen data). Therefore, the generalization bounds will be oneof useful methods for evaluating “the stability of trained models.”
In Subsection 7.3.2, it has been shown that the Chernoff bounds based on testing errors cangive meaningful values close to generalization errors, In addition, it has been also shown that52

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

the PAC-Bayes bounds (e.g., the Catoni bounds) can gives meaningful perturbated generalization
bounds based on testing errors (not on training errors) for probabilistic machine learned models
by adding noise to the weights. Such perturbation bounds make it possible to quantitatively
evaluate the performance at different viewpoints from normal testing by data samples.
As an additional investigation, we note the relation between perturbated generalization
bounds and the randomized smoothing introduced in Subsection 6.2.6. In the randomizedsmoothing, certifiable approximative probabilistic maximum safe radii can be estimated byadding noise to input data. We are still investigating the possibility that the perturbatedgeneralization bounds can theoretically guarantee robustness in a similar way.

53

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

8

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Adversarial Example Detection

8.1 Research summary
With the goal of practically establishing a method for determining whether a given input
image is an adversarial example, we focus on the following points regarding attacks anddetection methods that generate adversarial examples. We are conducting a survey of typicaltechnologies.
–

Supporting adversarial example detection program code and confirmation bycomputational experiment–

Reproduction of experimental results of adversarial example detection method papers–

Implementation of the framework for detecting adversarial examples

Adversarial example detection stands for detecting adversarial examples from given inputs,
and existing state-of-the-art adversarial example detection methods can be divided into fourmain categories.
① Metric based approaches (example [73])② Denoisers approaches (example [74])③ Prediction inconsistency based approaches (example [75])④ Neural Network Invariant Checking (NIC) approaches (example [76])In this chapter, we report the results of additional test experiments to compare and evaluate
adversarial example detection methods based on each of these approaches ① to ④. As reported
in the paper [76], it was confirmed that the approach of ④ (NIC: Neural Network Invariant
Checking) shows the highest detection rate among ① to ④. In this follow-up experiment, thepublished implementation code was used for ① to ③, but the implementation code was not
published for ④, so a computer experiment was conducted by implementing the NIC according
to the paper [76]. Therefore, this chapter mainly describes the NIC ④.
After explaining the outline of the four approaches, the method of detecting adversarial
examples by the NIC is explained, and the implementation method is described. Then, the resultsof the follow-up experiments of each approach and the experiments by the NIC are described.
Finally, we report the implementation of the NIC framework and the effectiveness evaluation.

8.2 Overview of adversarial example detection approaches
In this section, the four state-of-the-art approaches to adversarial example detection are
overviewed.

54Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Metric based approaches
A method of performing statistical measurements of inputs (and outputs of each neuron) to
detect adversarial examples, Ma et al. recently proposed the use of a measurement called Local
Intrinsic Dimensionality (LID) [73]. This method estimates the LID value that evaluates thespace-filling capacity of the area surrounding the sample by calculating the distance distributionof the sample and the number of neighbors in each layer, and the adversarial example tends tohave a large LID value. It uses certain properties to detect adversarial examples. LID is superior
to traditional kernel density estimation (KD) and Bayesian uncertainty (BU) for detecting
adversarial examples and is currently the state-of-the-art technology for this type of detector.
Denoisers approaches
It is a method of detecting adversarial examples by removing noise in a preprocessing stepfor each input. In this method, the training model or noise remover (encoder and decoder) is
trained to filter the image so that the key components in the training model can be highlighted.
This filter can be used to remove noise added by an attacker to generate adversarial examples
and correct misclassification. MagNet [74] is a method of detecting adversarial examples using
detectors and reformers (trained automatic encoders and automatic decoders).
Prediction inconsistency based approach
A method of detecting adversarial examples by measuring the discrepancy between theoriginal neural network and the neural network enhanced by human perceptible attributes.
Feature Squeezing [75], the state-of-the-art detection technique of this method, can achieve very
high detection rates against a variety of attacks. Feature squeezing focuses on detecting gradientbased attacks, focusing on the ability of attackers to generate adversarial examples through theunnecessarily large input feature space of deep neural networks DNN. The procedure fordetecting adversarial examples by feature squeezing is shown below.
1.

Apply squeezing technology (a technology that reduces the color depth of an image andsmooths the image) to the original input image to generate multiple squeezed images.

2.

Input the original input image and multiple squeeze images into the deep neuralnetwork, and measure the distance between the inference result (prediction vector) of
the input image and the inference result of each squeeze image.

3.

When one of the differences (distances) between the original input image and thesqueeze image exceeds the threshold value, the original input image is detected as anadversarial example.

55

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Neural Network Invariant Checking (NIC) approaches
The NIC (Neural Network Invariant Checking) method focuses on value invariants (VIs) andprovenance invariants (PIs) inside deep neural networks [76]. The value invariant VI is the
distribution of possible neuron values in each layer, and the provenance invariant PI is thepossible neuron value pattern of two consecutive layers (summary of correlation between
features across two layers). If an input violates these invariants, the input is detected as anadversarial example. The NIC [76] method trains these invariant VIs and PIs with benign input
data and model them as a one-class classification (OCC) problem that detects adversarial
examples. A higher detection rate has been reported than the methods based on (1) to (3)explained above. The outline and the implementation of the NIC system design are explained indetail in Sections 8.3 and 8.4, respectively.

8.3 NIC system design overview
The procedure for building the NIC detector (steps A to C: during training, D to E: duringexecution) is explained by using Figure 8.1 [76]. This invariant VI, PI training uses only nonadversarial benign data.

Figure 8.1 Outline of system design (Fig. 8 of thesis [76])–Step A: Collect the output value of each neuron at each layer of each training data input.

–

Step B: For each layer 𝑘 (e.g., L1, L2), extract the sub-models from the input layer tothe 𝑘 layer and add a new softmax layer with the same output label as the originalmodel. Then create a derived model (DerivedModel in Figure 8.1)

56

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition–

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Step C: Enter each benign training data for all derived models and collect the final
output of these models (i.e., the output probability values of the individual classes). For
each set of consecutive layers, we train using the distribution of the classification
results of this derivative model. This trained distribution is the PI for these two layers.

–

Step D: Input each test data 𝑡 (for example, the image of “4” in Figure 8.1) to allderivative models in addition to the original model, and observe the activation value ofeach layer of the original model. Collect the value OV (for example, OV(L1, 𝑡) in Figure8.1) and the classification result (set) of the derivative model of consecutive layers.
From this classification result, the observed source OP (for example, OP(L1, L2, 𝑡), etc.)
is obtained.

–

Step E: Calculate the probability D that the OV and OP fit the corresponding VI and PI
distributions. The possibility that the input 𝑡 is adversarial is predicted at the sametime by aggregating all these D values.

8.4 NIC system implementation
In order to detect adversarial examples based on NIC, a direct sum space (vector) isconstructed from PI and VI, and for classifying this vector, an OSVM (One Class Support Vector
Machine) is constructed. When the input to the layer 𝑙 of the trained DNN (Deep Neural
Network) model (hereinafter referred to as M) is 𝑥𝑙 , the output 𝑓𝑙 of the layer 𝑙 is given by
the following equation:
𝑓𝑙 = 𝜎(𝑥𝑙 ∙ 𝑤𝑙𝑇 + 𝑏𝑙 ),where 𝜎 is the activation function of the layer 𝑙, 𝑤𝑙𝑇 is the weight matrix, and 𝑏𝑙 is the bias.
At this time, the direct sum spaces classified by VI, PI, and OSVM are obtained as follows.
–

VI calculation: The VI of each layer 𝑙 of model M is determined by solving the following
optimization problem.
𝑉𝐼𝑙 = min [ ∑ 𝐽(𝑓𝑙 ∘ 𝑓𝑙−1 ∘ ⋯ ∘ 𝑓1 (𝑥) ⋯ 𝑤 𝑇 − 1)]𝑥∈𝑋𝑏

Here, 𝐽 is the error evaluation function, and 𝑋𝑏 is the batch used to create M. Also, ∘is a monoid, in this case a vectorized version of 𝑓𝑘 .
–PI calculation: 𝑃𝐼𝑙,𝑙+1 (𝑥) is based on the classification output of the derived models of
the layers 𝑙 and 𝑙 + 1 . The probability that 𝑥 is benign (non-adversarial) is
estimated by solving the following optimization problem.
𝑃𝐼𝑙,𝑙+1 (𝑥) = min [ ∑ 𝐽(𝑐𝑜𝑛𝑐𝑎𝑡(𝐷𝑙 (𝑥), 𝐷𝑙+1 (𝑥)) ⋯ 𝑤 𝑇 − 1)]
𝑥∈𝑋𝑏

57

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Here, a derivative model 𝐷𝑙 of the layer 𝑙 is defined as follows, with the softmax layer
added after the layer 𝑙.
𝐷𝑙 = softmax ∘ 𝑓𝑙 ∘ 𝑓𝑙−1 ∘ ⋯ ∘ 𝑓1–

Direct sum space of PI and VI: From the VI and PI obtained by the above optimization,
the following direct sum space (vector) is created for each batch of training data ofmodel M.
𝑉𝐼1 ⨁ 𝑃𝐼1,2 ⨁ 𝑉𝐼2 ⨁ 𝑃𝐼2,3 ⋯ 𝑉𝐼𝐵 ⨁ 𝑃𝐼𝐵−1,𝐵 ⨁ 𝑉𝐼𝐵
This vector is 𝐿 × 3 dimensions (𝐿 is the number of layers of M), which is the vector
space (direct sum space) of the number 𝐵. The NIC performs OSVM on this space.

8.5 Computer experiment
In order to confirm the effect of adversarial example detection technology (NIC), theexperiment of the paper [76] was retested in the following experimental environment.
–

Hardware environment: AIST ABCI [77]–

Datasets: Two common image datasets, MNIST [78] and CIFAR-10 [79], were used forimage classification experiments. MNIST is a grayscale image dataset used for
handwritten digit recognition, and CIFAR-10 is a color image dataset used for object
recognition. For NIC, we also conducted an experiment on LFW (face image) [80].

–Attacks: Non-targeted attacks (FGSM 𝐿2 ,𝐿∞ ), targeted attacks JSMA, and gradientbased attacks (CW 𝐿2) were used to generate adversarial examples. The Cleverhans
library [81] was used to implement FGSM and JSMAFirst, in order to evaluate the adversarial example detection method based on each of theapproaches ① to ③, the published implementation code of LID [73], MagNet [74], and featuresqueezing [75] was used to evaluate each paper. Then, follow-up experiments were conducted.
As the result, the detection rates reported in each paper were able to be confirmed, and among
these three, feature squeezing showed the highest detection rate.
Next, in order to evaluate the adversarial example detection method based on the approach④, an experiment was conducted using the NIC code implemented in Section 8.4. Table 8.1 toTable 8.3 show the results of adversarial example detection and computational experiments on
the MNIST, CIFAR-10, and LFW datasets, respectively. Here, the correct answer rate is the rate at
which adversarial examples are input to the classifier (OSVM) described in Section 7.4 and aredetermined to be adversarial examples. The CNN model used in the experiment is LeNet5, andthe OSVM Kernel is RBF (MNIST: γ = 0.1 to 0.27, CIFAR-10: γ = 0.11 to 0.2, LFW: γ = 0.005 to0.90). In the results of this experiment, high detection performance was confirmed not only forthe dataset and attack method reported in the paper [76], but also for the unreported dataset
LFW and attack method (FGSM 𝐿∞ ).

58

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Table 8.1 Adversarial example detection computational experiment results for MNIST dataset
Data Set

AttackInvariant

Performance

MNISTFGSM 𝐿2

VI
PI
NIC
VI

97%
98%
97%
98%

PI

98%

―

NIC

98%

―

VI
PI
NIC
VI
PI
NIC
VI
PI
NIC

100%
100%
100%
100%
100%
100%
100%
100%
100%

FGSM 𝐿∞

JSMA

CW2

TrojanNumberof data
2800

2800

280

280

3200

Performance reported
in the paper [76]
100%
84%
100%
―

83%
100%
100%
95%
96%
100%
100%
100%
100%

Table 8.2 Adversarial example detection computational experimental results for CIFAR-10 dataset
Data Set

AttackInvariant

Performance

CIFAR-10

FGSM 𝐿2

VI
PI
NIC
VI

99%
99%
99%
100%

PI

100%

―

NIC

100%

―

VI
PI
NIC
VI
PI
NIC
VI
PI
NIC

97%
95%
96%
98%
95%
96%
100%
100%
100%

FGSM 𝐿∞

JSMA

CW2

Trojan59

Numberof data
6400

6400

320

320

3200

Performance reported
in the paper [76]
100%
52%
100%
―

62%
100%
100%
88%
89%
100%
100%
100%
100%

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Table 8.3 Adversarial example detection computational experiment results for LFW dataset
Data Set

AttackInvariant

Performance

LFW

FGSM 𝐿2

VI

98%

PI

98%

―

NIC

98%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

VI

100%

PI

100%

―

NIC

100%

―

FGSM 𝐿∞

JSMA

CW2

TrojanNumberof data
28222

2822

280

840

3200

Performance reported
in the paper [76]
―

―

―

―

―

8.6 Implementation of the NIC framework
We have implemented a simplified NIC method based on Sections 8.3 and 8.4 in order to
conduct the computer experiments for confirming the effectiveness of NIC in Section 8.5. In thesimplified implementation, we have found some implementation issues in the original paper[76]. In this section, while clarifying the issues, we reconsider the algorithm in order to construct
the NIC framework for high detection rates of adversarial examples on the testbed, that is used
for creating an environment (attack, defense and detection) to benchmark vulnerability toadversarial examples.
Overview of the NIC framework
The NIC framework consists of five parts: taking output from each layer; calculating VI and PI
for normal data; calculating VI, PI and NIC for adversarial examples; evaluating OSVM anddisplaying results. The use case of the NIC framework is shown in Figure 8.2. In addition, theprocess steps for detecting adversarial examples are shown in Figure 8.3.

60

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 8.2 NIC framework use cases

Figure 8.3 Processing procedures for adversarial example detection by the NIC framework
As shown in Figure 8.3, the overall processing procedure for adversarial example detection
by the NIC framework consists of five parts. The function of each part (input, processing andoutput) is shown in Table 8.4.
Output of OSVM evaluation results
The NIC framework has been implemented using scikit-learn, that is a Python machinelearning library. For example, the scikit-learn's OneClassSVM class is used for implementing thefinal part of the OSVM as shown in Figure 8.3 as follows.
class sklearn.svm.OneClassSVM(array, kernel='rbf', gamma='auto', nu=0.3)
Here, the meaning of each argument is as follows.
・ array: parameters trained by normal data and used for detecting adversarial examples in
NIC.
・ kernel: the RBF kernel is used as the algorithm for One Class SVM.
・ gamma: the gamma parameter of the RBF kernel is set to 'auto'.
・ nu: the upper limit for the percentage of training error and the lower limit for thepercentage of support vector are set to 0.3 in this case.

61

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Table 8.4 Functions of the parts comprising the adversarial example detection process procedure
Output extraction from each layer
input

Normal data (images)
Adversarial examples (image).
Trained models, trained on normal data (models trained on normal data)processing

Obtain the output of each layer of the trained model for normal and adversarial
examples and save it in 'numpy in numpy' format.

output (e.g.

Output data from each layerof dynamo)
VI and PI calculations for normal data
input

Output data from each layer of normal dataprocessing

Calculate VI, PI from the output data of each layer of normal data.

output (e.g.

VI, PIof dynamo)
VI and PI calculations for adversarial examples
input

Output data from each layer of adversarial examples
Created at the time of calculation to PI with normal data Derived model of PI

processing

Compute VI, PI from the output of each layer of adversarial examples.

output (e.g.

VI, PIof dynamo)
Calculation of NIC
input

VI of normal data, PI
VI of adversarial examples, PI

processing

NIC of normal data is created from VI and PI of normal data and NIC ofadversarial examples is calculated from VI and PI of adversarial examples,
respectively.

output (e.g.

NIC for normal data, NIC for adversarial examplesof dynamo)
Evaluation and display of results in OSVM.
input

NIC of normal data
NIC for adversarial examples.

processing

Train OSVM on normal data to create a model, and use this trained model tojudge adversarial examples; OSVM uses sk-learn's one class svm API. The
judgement results are then displayed.

output (e.g.

Assessment Resultsof dynamo)

62

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 8.4 shows output values from each layer when one normal data and its adversarial
examples are input to the NIC framework, where the horizontal axis is the ID of the model
derived to calculate the NIC at each layer (Note. There are multiple outputs from each layer for
one image, for example, a convolution layer in CNN), and the vertical axis represents the signed
distance of each NIC to the One Class SVM classification hyperplane of the NIC of the normal data,that is the closeness to the normal data in this case. The black dots in Figure 8.4 (a) represent
the output relative to the normal data, the red dots in Figure 8.4 (b) are the outputs foradversarial examples. In this calculation, the adversarial examples in Figure 8.4 (b) weregenerated by using the FGSM 𝐿∞ attack method.

(a) Normal input data(b) Adversarial input data

Figure 8.4 Comparison of NIC framework outputs
After training One Class SVM by normal data, One Class SVM function 𝑓(𝑥) can be used for
detecting adversarial examples such that if 𝑓(𝑥) ≥ 0 then the input 𝑥 is normal otherwise itis adversarial. Most of the output for normal data are close to zero as shown in Figure 8.4 (a),while approximately 94% of the outputs for adversarial examples are explicitly less than zero as
shown in Figure 8.4 (b). This difference of the output between Figure 8.4 (a) and (b) explainsthat NIC can effectively detect adversarial examples.
Generation of adversarial examples
As shown in Figure 8.3, NIC framework does not include the program for generating
adversarial examples. We recommend for using CleverHans [82] if adversarial examples arenecessary. Figure 8.5 shows some examples in the normal (original) images of handwritten
numbers (MNIST) and the adversarial examples generated from the normal images by attack

63

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

method FGSM 𝐿2 with the misclassified labels inferred for the adversarial images. As shown in
the inference results (label 8) in Figure 8.5 (b), all generated adversarial examples are
misclassified as 8.

(a) Original MNIST data(b) Generated adversarial examples

Figure 8.5 Example of adversarial example generation from MNIST (handwritten numbers) images
and its decision results
Reducing calculation costs for VI, PI and VIC
The calculation method for VI, PI and NIC in the original paper [76] has been explained in
Section 8.4, but if the calculation method is used, then the dimension of each data (vector)
becomes very large, due to the problem so-called 'dimension demon'. Therefore, we have tried
to reduce the dimension as much as possible. In the following section, we explain how eachcalculation is simplified.
–

Calculation of VI: in the NIC framework, let 𝑋𝐵 = 1 for clarifying the correspondence
between the input data (both normal and adversarial data) and the VI, PI and NIC (i.e.,
for the accuracy of the verification). In addition, as all input data are normalized andcalculated, the following simplified formula is used:
𝑉𝐼𝑙 = 𝑓𝑙 ∘ 𝑓𝑙−1 ∘ ⋯ ∘ 𝑓2 ∘ 𝑓1 .

–Calculation of PI: as in the case VI above, let 𝑋𝐵 = 1. Then, the following simplified

64

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

formula is used:
𝑃𝐿𝑙,𝑙−1 = 𝑐𝑜𝑛𝑐𝑎𝑡(𝐷𝑙 , 𝐷𝑙−1 ) ∘ ⋯ ∘ 𝑐𝑜𝑛𝑐𝑎𝑡(𝐷2 , 𝐷1 ).
–NIC calculations: for dimensionality suppression, 𝑋𝐵 is set as follows:
𝑋𝐵 = (The number of layers from which output are obtained)

8.7 Evaluation of the effectiveness of NIC with the Kullback-Leibler divergenceThis section reports the results of the evaluation of the effectiveness of the NIC by calculating
the degree of divergence between the images of normal and adversarial examples and the NIC
by using the Kullback-Leibler divergence.
Kullback-Leibler divergence
The Kullback-Leibler divergence, denoted by 𝐾𝐿(𝑃 ∥ 𝑄) , is a measure of the degree ofdivergence between two probability distributions 𝑃 (the probability density functions 𝑝) andQ (the probability density function 𝑞 ). The Kullback-Leibler divergence is defined by thefollowing equation.
𝐾𝐿(𝑃 ∥ 𝑄) = ∫ 𝑝(𝑥) log

𝑝(𝑥)
𝑞(𝑥)

The Kullback-Leibler divergence is 0 when the two distributions are the same, and it increases
as the divergence increases (the convergence is not guaranteed due to the presence of log).
Figure 8.6 shows a simple calculation example of the Kullback-Leibler divergence. In Figure 8.6(a), both of the distributions 𝑃 and 𝑄 are the same normal distribution whose mean andvariance are 0.5 and 0.5, respectively, and then the 𝐾𝐿(𝑃 ∥ 𝑄) is 0. In Figure 8.6 (b), the meansof 𝑃 and 𝑄 are 0.5 and 0.55, and the variance of them are 0.5 and 0.55, respectively, and thenthe 𝐾𝐿(𝑃 ∥ 𝑄) is 0.053.

(a) In the case of the same distributions(b) In the case of the different distributions

Figure 8.6 Example of Kullback-Leibler divergence calculation

65

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Kullback-Leibler divergence estimation
The Kullback-Leibler divergence assumes that the probability distributions to be compared
are fixed, but in practice, both normal and adversarial data are simply sets of images and the
distributions are unknown. Fortunately, a method for approximating the Kullback-Leibler
divergence between sets with unknown probability distributions [83] is known. The outline of
the approximation method calculates the Kullback-Leibler divergence as a solution of an
optimization problem on the following linear polynomial of 𝑟𝜃 (𝑥) as the constraint forminimizing the density ratio 𝑟(𝑥) = 𝑝(𝑥)/𝑞(𝑥):𝑏𝑟𝜃 (𝑥) = ∑ 𝜃𝑗 𝜓𝑗 (𝑥) = 𝜽𝑇 𝝍(𝑥),𝑗=1

where 𝜓𝑗 (𝑥) is the RBF kernel and is defined by
𝜓𝑗 (𝑥) = exp (−‖𝑥 − 𝑥 ′ ‖2),2ℎ2

where ℎ is a determinable constant and is the bandwidth.
Then, the Kullback-Leibler divergence can be approximately calculated by the linear
polynomial 𝑟𝜃 (𝑥) obtained as the solution of the optimization problem as follows [83]:𝑛1𝐾𝐿(𝑃 ∥ 𝑄) ~ ∑ log 𝑟(𝒙𝑖 )𝑛𝑖=1

Effectiveness evaluation of NIC
In Section 8.5, we have shown that the NIC method can effectively detect adversarial
examples as anomaly data by experiments. In this section, we show the degree of divergence
between normal data and adversarial examples by comparing the Kullback-Leibler divergenceof them for explaining the reason why NIC is effective.
At first, Figure 8.7 shows the computational results of the Kullback-Leibler of normal data
and adversarial examples (generated by the attack method FGSM 𝐿2) for 50 image data samples,as shown in Figure 8.5. The approximate value of the Kullback-Leibler divergence for the FGSM
in Figure 8.7 is 0.46. Here, note that the average value of the multiple Kullback-Leibler divergenceis shown in Figure 8.7 because there are multiple values of NIC for one image as explained in
Figure 8.4.
Next, Figure 8.8 shows the computational results of the Kullback-Leibler of NIC of the normal
data and the adversarial examples used in Figure 8.7. The approximate value of the KullbackLeibler divergence in Figure 8.8 is 4.47. Therefore, the Kullback-Leibler divergence in Figure 8.8
is about 10 times larger than one in Figure 8.7. We conjecture that the results mean that the
perturbations added to normal data can be extracted as more explicit difference by the NIC
method.

66

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Figure 8.7 The Kullback-Leibler divergence for normal and adversarial examples

Figure 8.8 The Kullback-Leibler divergence of NIC for normal and adversarial examples

67

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

9

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

AI Quality Management in Operation
In this chapter, we report on the results of a survey on the latest technologies for detectingchanges in data distribution over time, called concept drift, and adapting machine learningmodels to the changed distribution for AI quality management during operation. In addition, we
also introduce the results of a survey on the latest unsupervised domain adaptation technologies
published at recent international major conferences on machine learning and computer vision
for further development of the AI quality management technologies.
Concept drift is one of the main causes of performance degradation of machine learningmodels running in AI systems during operation. In order to maintain quality that is satisfied at
the beginning of the operation of the system throughout the operation period, it is necessary tocontinuously monitor whether drift occurs or not. In addition, if necessary, we retrain themachine learning models in the system with the latest data to adapt them to the distribution ofdata changed after the drift occurs. As the use of machine learning technologies has been
expanded in recent years, AI systems operating with such technologies will require processing alarge amount of data without their true labels (ground truths) in a short period of time, including
types of data that have not been handled in the past.
In the fiscal year 2019-2020, we conducted a survey on the latest technologies for detecting
and adapting to the concept drift to maintain the performance of machine learning models
during operation. As a result of this survey, we found that most of the methods developed so far
are supervised methods that use true labels of data additionally acquired during operation forthe detection and adaptation. However, such true labels are not always available or are often
costly even if they are available. In order to expand the applicability of the detection andadaptation methods and reduce their operational costs, we found that an "unsupervised
method" that does not use the true labels or a "semi-supervised method" that uses only a limited
number of the true labels is promising. We summarized the results of the surveys organized anddiscussed from this perspective.
For details on the survey on detection methods, see Section 7.8 of the Machine Learning
Quality Management Guidelines [1]. In addition, adaptation methods are summarized in our
survey result [84]. Table 9.1 shows the comparison of our survey with the other existing surveys
on concept drift detection and adaptation methods. Gama et al. summarized their survey resultin [85] and Lu et al. added recently published drift detection and adaptation methods in [86].
Those survey papers mainly focus on introducing "supervised" methods that use true labels of
operational data for drift detection and adaptation. On the other hand, Ishida et al. introduced"unsupervised" concept drift detection methods that do not use true labels of data for drift
detection in [87]. In comparison with those existing survey results, we introduced"unsupervised" and "semi-supervised" concept drift adaptation methods that do not use or use
only a limited number of true labels as mentioned above. Furthermore, we introduced those drift
adaptation methods based on the characteristic of each method. In detail, we listed ten
remarkable unsupervised/semi-supervised drift adaptation methods and classified them68

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

according to: i) types of drift that can be dealt with effectively, ii) processes where true labels ofdata are required during operation and the percentage of the labeled data used in verifications
shown in the papers, and iii) machine learning models or clustering methods used in eachmethod. Finally, we closed our survey by discussing further development of unsupervised andsemi-supervised concept drift adaptation methods using knowledge obtained from relevant
unsupervised domain adaptation techniques.
Table 9.1 Comparison of survey papers on concept drift detection and adaptation
Detection
Supervised
Unsupervised /Semi-supervised

Adaptation

Gama et al.[85], Lu et al.[86]Ishida et al.[87]Okawa and Kobayashi [84], [88](Ours)In the future operation of AI systems, there is a growing need for new adaptation techniquesthat do not use the original training data (i.e., source data) to adapt machine learning models
from the viewpoint of data privacy and portability in addition to that can deal with changes otherthan those in the distribution of input data. In particular, adaptation techniques that do notdepend on such training data (source data) are called "source-free domain adaptation
techniques" or "test-time adaptation techniques (if they adapt online)”. These source-free andtest-time adaptation technologies have been attracting more attention because they can reduce
costs not only on management and transmission of source data for adaptation but also on
security for data storage.
In FY2021, following the above-mentioned surveys, we conducted a survey on the latest
research trends in unsupervised adaptation techniques to data changes presented at majorinternational conferences in the fields of machine learning and computer vision held in 20192021, focusing on unsupervised concept drift adaptation techniques and unsupervised domain
adaptation techniques. The result of this survey is summarized in [88]. In detail, we listed andintroduced 15 remarkable concept drift detection and unsupervised domain adaptation
methods and classified them according to: i) kinds of adaptation problems, ii) kinds of data andlabels used in detection and adaptation, iii) availability for adaptation to label shift, and iv) kindsof validation tasks. According to the results of this survey, it is shown that there has beendevelopment of the source-free adaptation and test-time adaptation techniques mentioned
above and adaptation techniques that are able to adapt to changes other than the distribution of
input data, such as label shifts. Furthermore, some techniques have been validated not only forimage classification problems, but also for semantic segmentation and object detection
problems. These research trends in unsupervised adaptation techniques are expected to solve
new problems in AI operations, such as maintaining data privacy, and to be used in various
situations in future AI operations.

69

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

10 References
Chapter 1:[1] National Institute of Advanced Industrial Science and Technology (AIST), Machine Learning
Quality Management Guideline (3rd English Edition), Digital Architecture Research Center,
Cyber Physical Security Research Center, Artificial Intelligence Research Center, Technical
Report DigiARC-TR-2023-01/ CPSEC-TR-2023001 , 2023
https://www.digiarc.aist.go.jp/publication/aiqm/[2] Yuri Miyagi, Masaki Onishi, Machine Learning Model Comparison Visualization Focusing on
Worker Information, The 24th Meeting on Image Recognition and Understanding 2021, I3122, 2021.
[3] Yuri Miyagi, Masaki Onishi, Comparative Visualization Method Focusing on Workers for
Evaluation of Machine Learning Models, The 49th Symposium on Visualization, OS12, 2021.
[4] Tomoumi Takase, Dynamic batch size tuning based on stopping criterion for neuralnetwork training, Neurocomputing, Volume 429, pp.1-11, 2021.
[5] Shin Nakajima, Software Testing with Statistical Partial Oracles, 10th SOFL+MSVL, 2021.
Chapter 2:[6] Satoshi Hara, My Bookmark : Interpretability in Machine Learning, Journal of JapaneseSociety for Artificial Intelligence, vol. 33, no. 3, pp. 366-369, 2018 (in Japanese).
[7] Fred Hohman, Minsuk Kahng, Robert Pienta, Duen Horng Chau, Visual Analytics in Deep
Learning: An Interrogative Survey for the Next Frontiers, IEEE Transactions on
Visualization and Computer Graphics, vol. 25, no. 8, pp. 2674-2693, 2018.
[8] Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, Liu Ren, Do Convolutional Neural
Networks Learn Class Hierarchy?, IEEE Transactions on Visualization and Computer
Graphics, vol. 24, no. 1, pp. 152-162, 2018.
[9] Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, Shixia Liu, Analyzing the Training Processes of
Deep Generative Models, IEEE Transactions on Visualization and Computer Graphics, vol.24,
no.1, pp.77-87, 2018.
[10] Jorge Piazentin Ono, Sonia Castelo, Roque Lopez, Enrico Bertini, Juliana Freire, Claudio Silva,
PipelineProfiler: A Visual Analytics Tool for the Exploration of AutoML Pipelines, IEEE
Transactions on Visualization and Computer Graphics, vol.27, no.2, pp.390-400, 2021.
[11] Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza, Power to the People: The
Role of Humans in Interactive Machine Learning. AI Magazine, vol.35, no.4, pp.105-120,
2014.
[12] Heungseok Park, Jinwoong Kim, Minkyu Kim, Ji-Hoon Kim, Jaegul Choo, Jung-Woo Ha and
Nako

Sung,

VISUALHYPERTUNER:

VISUALANALYTICS

HYPERPARAMTER TUNING OF DEEP NEURAL NETWORKS, 2019.

70

FOR

USER-DRIVEN

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Chapter 3:[13] Gontijo-Lopes, R., Smullin, S. J., Cubuk, E. D.,

and Dyer, E., Affinity and Diversity:

Quantifying Mechanisms of Data Augmentation. arXiv preprint arXiv:2002.08973, 2020.
[14] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q., RandAugment: Practical Automated Data
Augmentation with a Reduced Search Space. In Neural Information Processing Systems, 33,
2020.
[15] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D., Mixup: Beyond Empirical Risk
Minimization. In International Conference on Learning Representations, 2018.
[16] Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y.,
Manifold Mixup: Better Representations by Interpolating Hidden States. In International
Conference on Machine Learning, pp. 6438–6447, PMLR, 2019.
[17] Kim, J-H., Choo, W., and Song, H. O., Puzzle mix: Exploiting saliency and local statistics for
optimal mixup. In International Conference on Machine Learning, 2020.
[18] Beckham, C., Honari, S., Verma, V., Lamb, A., Ghadiri, F., Hjelm, R. D., Bengio, Y., and Pal, C. On
adversarial mixup resynthesis. In Neural Information Processing Systems, 2019.
Chapter 4:[19] Nakajima, S., Quality Issues in Machine Learning from Software Engineering Viewpoints,
Maruzen Publisher, 2020. (in Japanese)[20] Pei, K., et al., DeepXplore: Automated Whitebox Testing of Deep Learning Systems, In Proc.
26th SOSP, 2017, pp.1-18.
[21] Nakajima, S., Distortion and Faults in Machine Learning Software, In Post-Proc. 9th
SOFL+MSVL, 2020, pp.29-41.
[22] Ma, L., et al., DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems, In
Proc. ASE, 2018, pp.120-131.
[23] Tian, Y., et al., DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous
Cars, In Proc. 40th ICSE, 2018, pp.303-314.
[24] Zhang, M., et al., DeepRoad: GAN-Based Metamorphic Testing and Input Validation
Framework for Autonomous Driving Systems, In Proc. ASE, 2018, pp.132-142.
[25] Zhang, P, et al., CAGFuzz: Coverage-Guided Adversarial Generative Fussing Testing of Deep
Learning Systems, arXiv:1911.07931, 2019.
[26] Harel-Canada, F., et al., Is Neuron Coverage a Meaningful Measure for Testing Deep Neural
Networks? In ESEC/FSE, 2020, pp.851-862.
[27] Kim, J. et al., Guiding Deep Learning System Testing Using Surprise Adequacy, In Proc. 41st
ICSE, 2019, pp.1039-1049.
Chapter 5:71Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002[28] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, The MIT Press 2016.
[29] Simon Haykin, Neural Networks and Learning Machines (3ed.), Pearson India 2016.
[30] Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Anath Grama, MODE:
Automated Neural Network Model Debugging via State Differential Analysis and Input
Selection, In Proc. 26th ESE/FSE, pp.175-186, 2018.
[31] Shin Nakajima, Software Testing with Statistical Partial Oracles – Applications to Neural
Network Software, In Proc. 10th SOFL+MSVL, pp.275-192, 2021.
[32] Shin Nakajima and Tsong Yueh Chen, Generating Biased Dataset for Metamorphic Testing of
Machine Learning Programs, In Proc. 31st ICTSS, pp.56-64, 2019.
[33] Gregor Montavon, Genevieve B. Orr, and Klaus-Robert Muller (eds.), Neural Networks: Tricksof the Trade (2ed.), Springer 2012.
[34] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov, Membership Inference
Attacks Against Machine Learning Models, arXiv:1610.05820v2, 2017.
[35] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha, Privacy Risk in Machine
Learning: Analyzing the Connection to Overfitting, arXiv:1709.01604v5, 2018.
[36] Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl
A. Gunter, and Kai Chen, Understanding Membership Inferences on Well-Generalized
Learning Models, arXiv:1802.04489, 2018.
[37] Charu C. Aggarwal, Outlier Analysis (2ed.), Springer 2017.
[38] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer, Replux: An
Efficeint SMT Solver for Verifying Deep Neural Networks, In Proc. 29th CAV, pp.97-117, 2017.
[39] Pang Wei Koh and Percy Liang, Understanding Black-box Predictions via Influence
Functions, arXiv:1703.04730v3, 2020.
[40] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana, DeepXplore: Automated Whitebox
Testing of Deep Learning Systems, In Proc. 26th SOSP, pp.1-18, 2017.
[41] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting
Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. DeepGauge: Multi-Granularity TestingCriteria for Deep Learning Systems, In Proc. 33rd ASE, pp.120-131, 2018.
[42] Yizhen Dong, Peixin Zhang, Jingyi Wang, Shuang Liu, Jun Sun, Jianye Hao, Xinyu Wang, Li
Wang, Jin Song Dong, and Dai Ting. There is Limited Correlation between Coverage andRobustness for Deep Neural Networks. arXiv:1911.05904, 2019.
[43] Fabrice Harel-Canada, Lingxiao Wang, Muhammad Ali Gulzar, and Miryung Kim, In Proc. 28th
ESEC/FSE, pp.851-862, 2020.
[44] Shin Nakajima, Distortion and Faults in Machine Learning Software, In Proc. 9th SOFL+MSVL,
pp.29-41, 2019.
[45] Stephanie Abrecht, Maram Akila, Sujan Sai Gannamaneni, Konrad Groh, ChristianHeinzemann, Sebastian Houben, and Matthjas Woehrle, Revisiting Neuron Coverage and Its
Application to Test Generation, In Proc. SAFECOMP 2020 Workshop, pp.289-301, 2020.
[46] National Institute of Advanced Industrial Science and Technology (AIST), Machine Learning
Quality Management Guideline (1st English Edition), Chapter 4, Digital Architecture
Research Center, Cyber Physical Security Research Center, Artificial Intelligence Research

72Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Center, Technical Report DigiARC-TR-2022-01/ CPSEC-TR-2022002.
https://www.digiarc.aist.go.jp/publication/aiqm/
Chapter 6:[47] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian
Goodfellow, and Rob Fergus, Intriguing properties of neural networks, The International
Conference on Learning Representations (ICLR 2014), pp.1-10, 2014.
https://arxiv.org/abs/1312.6199[48] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer, Reluplex: An
Efficient SMT Solver for Verifying Deep Neural Networks, International Conference on
Computer-Aided Verification (CAV), 2017. https://arxiv.org/abs/1702.01135[49] Vincent Tjeng, Kai Xiao, and Russ Tedrake, Evaluating robustness of neural networks with
mixed integer programming, International Conference on Learning Representations (ICLR),
2019. https://arxiv.org/abs/1711.07356[50] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit
S. Dhillon, and Luca Daniel, Towards Fast Computation of Certified Robustness for ReLU
Networks, International Conference on Machine Learning, PMLR 80, pp.5276-5285, 2018.
https://arxiv.org/abs/1804.09699[51] Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel, CNN-Cert: An
Efficient Framework for Certifying Robustness of Convolutional Neural Networks, The
Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019), pp.3240-3247, 2019.
https://arxiv.org/abs/1811.12395[52] Tsui-Wei Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan
Oseledets, and Luca Daniel, PROVEN: Verifying Robustness of Neural Networks with a
Probabilistic Approach, International Conference on Machine Learning (ICML 2019), PMLR
vol. 97, pp.6727-6736, 2019. http://proceedings.mlr.press/v97/weng19a.html[53] Nicholas Carlini and David Wagner, Towards Evaluating the Robustness of Neural Networks,
IEEE Symposium on Security and Privacy (SP), pp.39-57, 2017.
https://arxiv.org/abs/1608.04644[54] Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh,and Luca Daniel, Evaluating the Robustness of Neural Networks: An Extreme Value Theory
Approach, International Conference on Learning Representations (ICLR 2018), 2018.
https://arxiv.org/abs/1801.10578[55] Eric Wong and J. Zico Kolter, Provable defenses against adversarial examples via the convex
outer adversarial polytope, International Conference on Machine Learning (ICML 2018),
PMLR vol. 80, pp.5283-5292, 2018. https://arxiv.org/abs/1711.00851[56] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana,
Certified Robustness to Adversarial Examples with Differential Privacy, The IEEE
Symposium on Security and Privacy (SP), 2019. https://arxiv.org/abs/1802.03471[57] Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter, Certified Adversarial Robustness via

73

Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

Randomized Smoothing, The 36th International Conference on Machine Learning (ICML2019), 2019. https://arxiv.org/abs/1902.02918[58] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian
Vladu, Towards Deep Learning Models Resistant to Adversarial Attacks, The Sixth
International Conference on Learning Representations (ICLR 2018), 2018.
https://arxiv.org/abs/1706.06083
Chapter 7:[59] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio,
Fantastic Generalization Measures and Where to Find Them, International Conference on
Learning Representations (ICLR 2020). https://arxiv.org/abs/1912.02178[60] Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan
Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M. Roy, In search of robust measuresof generalization, NeurIPS 2020. arXiv:2010.11924. https://arxiv.org/abs/2010.11924[61] Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst, PAC-Bayesian Margin Bounds
for Convolutional Neural Networks, arXiv:1801.00171, 2018.
https://arxiv.org/abs/1801.00171[62] Ilja Kuzborskij and Christoph H. Lampert, Data-Dependent Stability of Stochastic Gradient
Descent, 2017. arXiv:1703.01678. https://arxiv.org/abs/1703.01678[63] Andrew Y. K. Foong, Wessel P. Bruinsma, David R. Burt, and Richard E. Turner, How Tight
Can PAC-Bayes be in the Small Data Regime? Neural Information Processing Systems(NeurIPS), 2021. https://arxiv.org/abs/2106.03542[64] Guillermo Valle-Perez and Ard A. Louis, Generalization bounds for deep learning,
arXiv:2012.04115v2, 2020. https://arxiv.org/abs/2012.04115[65] Saurabh Garg, Sivaraman Balakrishnan, J. Zico Kolter, and Zachary C. Lipton, RATT:
Leveraging Unlabeled Data to Guarantee Generalization, ICML 2021.
arXiv:2105.00303. https://arxiv.org/abs/2105.00303[66] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz,
Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression
Approach, ICLR 2019. https://arxiv.org/abs/1804.05862[67] Gintare Karolina Dziugaite and Daniel M. Roy, Computing Nonvacuous Generalization
Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training
Data, Thirty-Third Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
arXiv:1703.11008. https://arxiv.org/abs/1703.11008[68] María Perez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvari, Tighter riskcertificates for neural networks, Journal of Machine Learning Research, 2021.
arXiv:2007.12911. https://arxiv.org/abs/2007.12911[69] Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory
to Algorithms, Cambridge University Press, 2014.
https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/index.html

74Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002[70] Oliver Catoni, PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical
Learning, Institute of Mathematical Statistics, Lecture Notes-Monograph Series, vol. 56,
2007. https://www.jstor.org/stable/i20461497[71] Andreas Maurer, A Note on the PAC Bayesian Theorem, arXiv:cs/0411099, 2004.
https://arxiv.org/abs/cs/0411099[72] John Langford, Tutorial on Practical Prediction Theory for Classification, JMLR, vol.6, No.10,
pp.273–306, 2005. https://jmlr.org/papers/v6/langford05a.html
Chapter 8:[73] X. Ma, Characterizing adversarial subspaces using Local Intrinsic Dimensionality, 2018.
[74] D. Meng , Magnet: a two-pronged defense against adversarial examples, in Proceedings ofthe 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017.
[75] W. Xu, Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks, inProceedings of the 2018 Network and Distributed Systems Security Symposium (NDSS),
2018.
[76] Shiqing Ma, NIC: Detecting Adversarial Samples with Neural Network Invariant Checking,
Network and Distributed Systems Security Symposium (NDSS), NDSS 2019.
[77] National Institute of Advanced Industrial Science and Technology (AIST), AI Bridging Cloud
Infrastructure, https://abci.ai/ja/[78] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document
recognition, Proceedings of the IEEE, vol. 86, no. 11, pp.2278–2324, 1998. [Online].
Available: http://yann.lecun.com/exdb/mnist/[79] A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, 2009.
[80] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, Labeled faces in the wild: A databasefor studying face recognition in unconstrained environments, University of Massachusetts,
Amherst, Tech. Rep. 07-49, October 2007.
[81] Nicolas Papernot, Ian Goodfellow, Ryan Sheatsley, Reuben Feinman, and Patrick McDaniel.
cleverhans v1.0.0: an adversarial machine learning library. arXiv preprint arXiv:1610.00768,
2016.
[82] CleverHans, https://github.com/cleverhans-lab/cleverhans[83] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori, Density Ratio Estimation in
Machine Learning, Cambridge University Press, 2012.
Chapter 9:[84] Yoshihiro Okawa and Kenichi Kobayashi, A Survey on Concept Drift Adaptation
Technologies for Unlabeled Data in Operation, Proceedings of the 35th Annual Conference of
the Japanese Society for Artificial Intelligence, pp.1-4, 2021 (in Japanese),https://doi.org/10.11517/pjsai.JSAI2021.0_2G4GS2f03.
[85] Joao Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia,

75Technical Report on Machine Learning
Quality Evaluation and Improvement
2nd English edition

National Institute ofAdvanced Industrial Science and Technology
DigiARC-TR-2023-02 / CPSEC-TR-2023002

A survey on concept drift adaptation, ACM Computer Surveys, vol. 46, no. 4, pp.1-37, 2014.
[86] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan ZhangJ, Learning under
Concept Drift: A Review, in IEEE Transactions on Knowledge and Data Engineering, vol. 31,
no. 12, pp. 2346-2363, 2019.
[87] Tsutomu Ishida, Hiroaki Kingetsu, Yasuto Yokota, Yoshihiro Okawa, Kenichi Kobayashi, andKatsuhito Nakazawa, Evaluation of Concept Drift Detection Methods for Unlabeled Data in
Operation, Proceedings of the 34th Annual Conference of the Japanese Society for Artificial
Intelligence, pp.1-4, 2020 (in Japanese),https://doi.org/10.11517/pjsai.JSAI2020.0_4Rin105.
[88] Yoshihiro Okawa and Kenichi Kobayashi, Recent Research Trends in Unsupervised
Adaptation Techniques for Data Changes, Proceedings of the 36th Annual Conference of theJapanese Society for Artificial Intelligence, pp.1-4, 2022 (in Japanese),https://doi.org/10.11517/pjsai.JSAI2022.0_3Yin240.

76