Studies on Computational
Learning via Discretization

Mahito Sugiyama
Doctoral dissertation, 2012

Department of Intelligence Science and Technology
Graduate School of Informatics
Kyoto University

A doctoral dissertation
submitted in partial ful illment of the requirements
for the degree of Doctor of Informatics.
Department of Intelligence Science and Technology,
Graduate School of Informatics,
Kyoto University.
Typeset withXETEX, Version 3.1415926-2.3-0.9997.5 (TeX Live 2011),
XY-pic, Version 3.8.5.
Copyright © 2012 Mahito Sugiyama
All rights reserved.

Abstract
This thesis presents cutting-edge studies on computational learning. The key issue
throughout the thesis is amalgamation of two processes; discretization of continuous objects and learning from such objects provided by data.
Machine learning, or data mining and knowledge discovery, has been rapidly
developed in recent years and is now becoming a huge topic in not only researchcommunities but also businesses and industries. Discretization is essential forlearning from continuous objects such as real-valued data, since every datum obtained by observation in the real world must be discretized and converted fromanalog (continuous) to digital (discrete) form to store in databases and manipulate on computers. However, most machine learning methods do not pay attention
to the process: they use digital data in actual applications whereas assume analog
data (usually real vectors) in theories. To bridge the gap, we cut into computational
aspects of learning from theory to practice through three parts in this thesis.
Part I addresses theoretical analysis, which forms a disciplined foundation of
the thesis. In particular, we analyze learning of igures, nonempty compact sets in
Euclidean space, based on the Gold-style learning model aiming at a computational
basis for binary classi ication of continuous data. We use fractals as a representation system, and reveal a learnability hierarchy under various learning criteria in
the track of traditional analysis of learnability in the Gold-style learning model. We
show a mathematical connection between machine learning and fractal geometryby measuring the complexity of learning using the Hausdorff dimension and the
VC dimension. Moreover, we analyze computability aspects of learning of iguresusing the framework of Type-2 Theory of Effectivity (TTE).
Part II is a way from theory to practice. We start from designing a new measure in a computational manner, called coding divergence, which measures the difference between two sets of data, and go further by solving the typical machinelearning tasks: classi ication and clustering. Speci ically, we give two novel clustering algorithms, COOL (COding Oriented cLustering) and BOOL (Binary cOding
Oriented cLustering). Experiments show that BOOL is faster than the K-means
algorithm, and about two to three orders of magnitude faster than two state-ofthe-art algorithms that can detect non-convex clusters of arbitrary shapes.
Part III treats more complex problems: semi-supervised and preference learning, by bene iting from Formal Concept Analysis (FCA). First we construct a SELF
(SEmi-supervised Learning via FCA) algorithm, which performs classi ication andlabel ranking of mixed-type data containing both discrete and continuous variables. Finally, we investigate a biological application; we challenge to ind ligand
candidates of receptors from databases by formalizing the problem as multi-label
classi ication, and develop an algorithm LIFT (Ligand FInding via Formal ConcepT
Analysis) for the task. We experimentally show their competitive performance.

Acknowledgments
I am deeply grateful to all the people who have supported me along the way. Firstof all, I would like to sincerely thank to my supervisor Prof. Akihiro Yamamoto,who is my thesis committee chair. His comments and suggestions had inestimablevalue for my study. I would also like to thank to the other committee members,
Prof. Tatsuya Akutsu and Prof. Toshiyuki Tanaka, for reviewing this thesis and fortheir meticulous comments.
Special thanks to my co-authors Prof. Hideki Tsuiki and Prof. Eiju Hirowatari,
who have been greatly tolerant and supportive and gave insightful comments andsuggestions. I am also indebted to Mr. Kentaro Imajo, Mr. Keisuke Otaki, and Mr.
Tadashi Yoshioka, who are also my co-authors and colleagues in our laboratory.
My deepest appreciation goes to Prof. Shigeo Kobayashi who was my supervisor during my Master’s course. It has been a unique chance for me to learn fromhis biological experience and his never-ending passion of scienti ic discovery.
I have had the support and encouragement of Prof. Takashi Washio, Prof. Shinichi Minato, Dr. Yoshinobu Kawahara, and Dr. Matthew de Brecht. I would like toexpress my gratitude to Dr. Marco Cuturi for his constant support in the English
language throughout this thesis.
I would like to warmly thank all of the people who helped or encouraged me invarious ways during my doctoral course: Dr. Koichiro Doi, Dr. Ryo Yoshinaka, andmy colleagues in our laboratory.
Apart from individuals, I gratefully appreciate the inancial support of the Japan
Society for the Promotion of Science (JSPS) and Japan Student Services Organization that made it possible to complete my thesis.
Finally, I would like to thank to my family: my mother Hiroko, my father-inlaw Masayuki Fujidai, and my wife Yumiko. In particular, Yumiko’s support wasindispensable to complete my doctoral course.
My father and my mother-in-law passed away during my doctoral course. I
would like to devote this thesis to them.

Contents
Abstracti

Acknowledgmentsii

1 Introduction
1.1 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1
4

I

7

Theory2 Learning Figures as Computable Classi ication
2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Formalization of Learning . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Exact Learning of Figures . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Explanatory Learning . . . . . . . . . . . . . . . . . . . . . .
2.3.2 Consistent Learning . . . . . . . . . . . . . . . . . . . . . . .
2.3.3 Reliable and Refutable Learning . . . . . . . . . . . . . . . .
2.4 Effective Learning of Figures . . . . . . . . . . . . . . . . . . . . . . .
2.5 Evaluation of Learning Using Dimensions . . . . . . . . . . . . . . .
2.5.1 Preliminaries for Dimensions . . . . . . . . . . . . . . . . . .
2.5.2 Measuring the Complexity of Learning with Dimensions . .
2.5.3 Learning the Box-Counting Dimension Effectively . . . . . .
2.6 Computational Interpretation of Learning . . . . . . . . . . . . . . .
2.6.1 Preliminaries for Type-2 Theory of Effectivity . . . . . . . .
2.6.2 Computability and Learnability of Figures . . . . . . . . . .
2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8
12
13
21
21
24
25
28
31
31
33
35
36
36
38
41

II

43

From Theory to Practice

3 Coding Divergence
3.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Mathematical Background . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 The Cantor Space . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Embedding the Euclidean Space into the Cantor Space . . .
3.3 Coding Divergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 De inition and Properties . . . . . . . . . . . . . . . . . . . .
3.3.2 Classi ication Using Coding Divergence . . . . . . . . . . . .
3.3.3 Learning of Coding Divergence . . . . . . . . . . . . . . . . .
3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44
46
47
47
47
49
49
51
51
53

iv

CONTENTS
3.4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Results and Discussions . . . . . . . . . . . . . . . . . . . . .
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Outlook: Data Stream Classi ication on Trees . . . . . . . . . . . . .
3.6.1 CODE approach . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . .

53
55
55
57
57
58

4 Minimum Code Length and Gray Code for Clustering
4.1 Minimum Code Length . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Minimizing MCL and Clustering . . . . . . . . . . . . . . . . . . . . .
4.2.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . .
4.2.2 COOL Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 G-COOL: COOL with Gray Code . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Gray Code Embedding . . . . . . . . . . . . . . . . . . . . . .
4.3.2 Theoretical Analysis of G-COOL . . . . . . . . . . . . . . . . .
4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . .
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62
65
66
66
66
68
68
69
71
71
73
75

5 Clustering Using Binary Discretization
5.1 Clustering Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.1 Formulation of Databases and Clustering . . . . . . . . . . .
5.1.2 Naïve BOOL . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.3 Relationship between BOOL and DBSCAN . . . . . . . . . . .
5.2 Speeding Up of Clustering through Sorting . . . . . . . . . . . . . . .
5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . .
5.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

76
77
78
79
81
82
86
86
88
90
94

III With Formal Concept Analysis

95

6 Semi-supervised Classi ication and Ranking
6.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 The SELF Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.1 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . .
6.2.2 Clustering and Making Lattices by FCA . . . . . . . . . . . .
6.2.3 Learning Classi ication Rules . . . . . . . . . . . . . . . . . .
6.2.4 Classi ication . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

96
97
99
99
102
103
107
108
108
110
111
118

7 Ligand Finding by Multi-label Classi ication
119
7.1 The LIFT Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

CONTENTS
7.1.1 Multi-label Classi ication and Ranking . . . . . . . . . . . . .
7.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . .
7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8 Conclusion

123
129
129
130
132
133

A Mathematical Background
136
A.1 Sets and Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
A.2 Topology and Metric Space . . . . . . . . . . . . . . . . . . . . . . . . 137
Symbols

140

Bibliography

144

Publications by the Author

157

Index

159

v

List of Figures
1.1 Measurement of cell by microscope . . . . . . . . . . . . . . . . . . .
1.2 Binary encoding of real numbers in [0, 1] . . . . . . . . . . . . . . .

2
3

2.1
2.2
2.3
2.4
2.5

Framework of learning igures . . . . . . . . . . . . . . . . . . . . . . 11
Generation of the Sierpiński triangle . . . . . . . . . . . . . . . . . . 16
Learnability hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Positive and negative examples for the Sierpiński triangle . . . . . . 34
The commutative diagram representing F E -I - and F E E -I learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.1
3.2
3.3
3.4
3.5
3.6
3.7

Two examples of computing the binary-coding divergence . . . . .
Tree representation of the Cantor space over Σ = {0, 1} . . . . . . .
The (one-dimensional) binary embedding 𝜑 . . . . . . . . . . . . .
Experimental results of accuracy for real data . . . . . . . . . . . . .
Examples of calculating similarities . . . . . . . . . . . . . . . . . . .
Experimental results for synthetic data . . . . . . . . . . . . . . . . .
Experimental results for real data . . . . . . . . . . . . . . . . . . . .

46
48
49
56
58
61
61

4.1 Examples of computing MCL with binary and Gray code embedding
4.2 Gray code embedding 𝜑 . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Examples of level-1 and 2 partitions with binary and Gray code embedding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Representative clustering results . . . . . . . . . . . . . . . . . . . .
4.5 Experimental results for synthetic data . . . . . . . . . . . . . . . . .
4.6 Experimental results for speed and quality for synthetic data . . . .
4.7 Experimental results for real data . . . . . . . . . . . . . . . . . . . .

63
69

5.1
5.2
5.3
5.4
5.5

Example of clustering using BOOL . . . . . . . . . . . . . . . . . . . .
Illustration of 𝑙-neighborhood . . . . . . . . . . . . . . . . . . . . . .
Illustrative example of clustering process by speeded-up BOOL . .
Clustering speed and quality for randomly generated synthetic data
Clustering speed and quality with respect to distance parameter 𝑙and noise parameter 𝑁 . . . . . . . . . . . . . . . . . . . . . . . . . .
5.6 Experimental results for synthetic databases DS1 - DS4 . . . . . . .
5.7 Experimental results (contour maps) for four natural images . . . .
5.8 Experimental results for geospatial satellite images . . . . . . . . .

70
71
72
73
74
77
81
85
87
88
91
92
93

6.1 Flowchart of SELF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.2 The bipartite graph corresponding to the context in Example 6.5 . 103
6.3 Closed set lattice (concept lattice) . . . . . . . . . . . . . . . . . . . . 104

LIST OF FIGURES
6.4
6.5
6.6
6.7
6.8

Closed set lattices (concept lattices) at discretization levels 1 and 2
Experimental results of accuracy with varying the labeled data size
Experimental results of accuracy with varying the feature size . . .
Experimental results of accuracy with varying the feature size . . .
Experimental results of correctness and completeness with varying
the labeled data size . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.9 Experimental results of correctness and completeness with varying
the feature size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.10 Experimental results of correctness and completeness with varying
the feature size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.1 Ligand-gated ion channel . . . . . . . . . . . . . . . . . . . . . . . . .
7.2 Concept lattice constructed from the context in Example 7.1 . . . .
7.3 Concept lattice from the context in Example 7.2 with its geometricinterpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.4 Concept lattices constructed from contexts in Example 7.8 . . . . .
7.5 Experimental results of accuracy for each receptor family . . . . . .

106
112
113
114
115
116
117
120
122
123
128
131

vii

List of Tables
1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

2.1 Relationship between the conditions for each inite sequence andthe standard notation of binary classi ication . . . . . . . . . . . . .

20

4.1 Experimental results of running time and MCL for real data . . . .

75

5.1
5.2
5.3
5.4

79
84
89
90

Database 𝜏, and discretized databases Δ (𝜏) and Δ (𝜏) . . . . . . .
Database 𝜏 and sorted database 𝑆 (𝜏) . . . . . . . . . . . . . . . . .
Experimental results of running time . . . . . . . . . . . . . . . . . .
Experimental results for UCI data . . . . . . . . . . . . . . . . . . . .

6.1 Statistics for UCI data . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.1 Statistics for families of receptors . . . . . . . . . . . . . . . . . . . . 130

List of Algorithms & Procedures
2.1 Classi ier ℎ of hypothesis 𝐻 . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Learning procedure that F E -I -learns 𝜅(ℋ) . . . . . . . . . . .
2.3 Learning procedure that F E E -I -learns 𝜅(ℋ) . . . . . . . . . .

18
22
31

3.1
3.2
3.3
3.4

Learner 𝜓 that learns 𝐶 (𝑋, 𝑌) . . . . . . . . . . . . . . . . . . . . . .
Learning algorithm M that learns 𝐶 (𝑋, 𝑌) . . . . . . . . . . . . . . .
Construction of tree and calculation of the similarity . . . . . . . . .
CODE procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

52
54
59
60

4.1 COOL algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67

5.1 Naïve BOOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Speeded-up BOOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

78
83

6.1
6.2
6.3
6.4

100
101
105
107

Data preprocessing for discrete variables . . . . . . . . . . . . . . .
Data preprocessing for continuous variables . . . . . . . . . . . . .
SELF algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Classi ication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.1 LIFT algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

1
INTRODUCTION

L

measuring the size of a cell. One of the most straightforward
ways is to use a microscope equipped with a pair of micrometers; an ocular micrometer
micrometer and a stage micrometer. The ocular micrometer is a glass disk with aruled scale (like a ruler) located at a microscope eyepiece, which is used to measure the size of magni ied objects. The stage micrometer is used for calibration,
because the actual length of the marks on the scale of the ocular micrometer isdetermined by the degree of magni ication. Here we consider only four objectives
whose magni ication is 1×, 2×, 4×, and 8×, for simplicity, and do not consider
magni ication of the eyepiece.
Figure 1.1 shows an example of measurement of a cell. Let the length of the cell
be 𝑥 and marks represent 1 𝜇m in length without any magni ication. We obtain
2 𝜇m ≤ 𝑥 ≤ 3 𝜇m
if we use the objective with 1× magni ication. We call the width 3 − 2 = 1 𝜇m theerror of measurement. This is a very rough value, but the result can be re ined, that error
is, the error can be reduced if we use a high-power objective. Then, we have2 𝜇m ≤ 𝑥 ≤ 2.5 𝜇m
2.25 𝜇m ≤ 𝑥 ≤ 2.5 𝜇m(2×),(4×), and2.25 𝜇m ≤ 𝑥 ≤ 2.375 𝜇m(8×),

and errors are 0.5 𝜇m, 0.25 𝜇m, and 0.125 𝜇m, respectively. Thus we can see thatevery datum in the real world obtained by a microscope has a numerical error datum
which depends on the degree of magni ication, and if we magnify 𝑘×, the error
becomes 𝑘 𝜇m. This is not only the case for a microscope and is fundamental forevery measurement: Any datum obtained by an experimental instrument, which
is used for scienti ic activity such as proposing and testing a working hypothesis,must have some numerical error (cf. Baird, 1994).
In the above discussion, we (implicitly) used a real number to represent the real number
true length 𝑥 of the cell, which is the standard way to treat objects in the real world
mathematically. However, we cannot directly treat such real numbers on a computer — an in inite sequence is needed for exact encoding of a real number without

2

INTRODUCTION

Scale of ocular micrometer

Magnification
1×
2×
4×

Magnified cells

Figure 1.1 | Measurement of a cell bya microscope.

continuum

discretization
binary encoding

8×

any numerical error. This is why both of the cardinalities of the set of real numbersℝ and the set of in inite sequences Σ are the continuum, whereas that of the set of∗inite sequences Σ is the same as ℵ , the cardinality of the set of natural numbersℕ. Therefore, we cannot escape from discretization of real numbers to treat themon a computer in inite time.
In a typical computer, numbers are represented through the binary encoding
scheme. For example, numbers 1, 2, 3, 4, 5, 6, 7, and 8 are represented as𝟶𝟶𝟶, 𝟶𝟶𝟷, 𝟶𝟷𝟶, 𝟶𝟷𝟷, 𝟷𝟶𝟶, 𝟷𝟶𝟷, 𝟷𝟷𝟶, 𝟷𝟷𝟷,

binary representation

respectively and, in the following, we focus on real numbers in [0, 1] (the closed
interval from 0 to 1) to go into the “real” world more deeply.
Mathematically, the binary encoding, or binary representation, of real numbers
in [0, 1] is realized as a surjective function 𝜌 from Σ to ℝ with Σ = {𝟶, 𝟷} such that
𝜌(𝑝) =𝑝 ⋅2()

for an in inite binary sequence 𝑝 = 𝑝 𝑝 𝑝 … (Figure 1.2). For instance,𝜌(𝟷𝟶𝟶𝟶 … ) = 0.5, 𝜌(𝟶𝟷𝟶𝟶 … ) = 0.25, and 𝜌(𝟷𝟷𝟷𝟷 … ) = 1.
Thus, for an unknown real number 𝑥 such that 𝑥 = 𝜌(𝑝), if we observe the irst bit
𝑝 , we can determine to𝑝 ⋅2≤𝑥≤𝑝 ⋅2+2.

This means that this datum has an error 2 = 0.5, which is the width of the interval. In the same way, if we observe the second, the third, and the fourth bits 𝑝 ,𝑝 , and 𝑝 , we have

𝑝 ⋅2

(

)

≤𝑥≤

𝑝 ⋅2

(

)

+2

(for 𝑝 𝑝 ) ,

𝑝 ⋅2

(

)

≤𝑥≤

𝑝 ⋅2

(

)

+2

(for 𝑝 𝑝 𝑝 ) , and

𝑝 ⋅2

(

)

≤𝑥≤

𝑝 ⋅2

(

)

+2

(for 𝑝 𝑝 𝑝 𝑝 ) .

INTRODUCTION

3

Position

4
3
2
1
0
0

ρ(01001...) = 0.3

0.5

1

Figure 1.2 | Binary encoding of real
numbers in [0, 1]. The position i is 1 if
it is on the line, and 0 otherwise.

Thus a pre ix, a truncated inite binary sequence, has a partial information about thetrue value 𝑥, which corresponds to a measured datum by a microscope. The error()
becomes 2
when we obtain the pre ix whose length is 𝑘. Thus observing
the successive bit corresponds to magnifying the object to double. In this way, we
can reduce the error but, the important point is, we cannot know the exact true
value of the object. In essentials, only such an observable information, speci ically,a pre ix of an in inite binary sequence encoding a real number, can be used on acomputer, and all computational processings must be based on discrete structure
manipulation on such approximate values.
Recently, computation for real numbers has been theoretically analyzed in thearea of computable analysis (Weihrauch, 2000), where the framework of Type-2 Type-2 Theory of Eﬀectivity, TTE
Theory of Effectivity (TTE) has been introduced based on a Type-2 machine, which Type-2 machineis an extended mathematical model of a Turing machine. This framework treats Turing machine
computation between in inite sequences; i.e., treats manipulation for real numbers through their representations (in inite sequences). The key to realization ofreal number computation is to guarantee the computation between streams as fol- stream
lows: when a computer reads longer and longer pre ixes of the input sequence, it
produces longer and longer pre ixes of the resulting sequence. Such procedure iscalled effective computing.
eﬀective computing
Here we go to the central topic of the thesis: machine learning, which “is a scienti ic discipline concerned with the design and development of algorithms thatallow computers to evolve behaviors based on empirical data”¹. Machine learning,
including data mining and knowledge discovery, has been rapidly developed in recent years and is now becoming a huge topic in not only research communities but
also businesses and industries.
Since the goal is to learn from empirical data obtained in the real world, basically, discretization lies in any process in machine learning for continuous objects.
However, most machine learning methods do not pay attention to discretization
as a principle for computation of real numbers. Although there are several discretization techniques (Elomaa and Rousu, 2003; Fayyad and Irani, 1993; Friedman et al., 1998; Gama and Pinto, 2006; Kontkanen et al., 1997; Lin et al., 2003; Liuet al., 2002; Skubacz and Hollmén, 2000), they treat discretization as just the data
preprocessing for improving accuracy or ef iciency, and the process discretization
itself is not considered from computational point of view. Now, the mainstream
in machine learning is an approach based on statistical data analysis techniques,so-called statistical machine learning, and they also (implicitly) use digital data in
actual applications on computers whereas assume analog data (usually vectors of¹Reprinted from Wikipedia (http://en.wikipedia.org/wiki/Machine_learning)

4

INTRODUCTION
real numbers) in theory. For example, methods originated from the perceptron
are based on the idea of regulating analog wiring (Rosenblatt, 1958), hence theytake no notice of discretization.
This gap is the motivation throughout this thesis. We cut into computational
aspects of learning from theory to practice to bridge the gap. Roughly speaking,we build an “analog-to-digital (A/D) converter” into machine learning processes.

1.1 Main Contributions
This thesis consists of three parts. We list the main contributions for each part in
the following with referring the publications by the author. We also summarize ourcontributions in Table 1.1 by categorizing them into learning types. See pp. 157 –158 for the list of publications.

Part I: Theory
All results presented in this part have been published in [P1,P2].
Chapter 2: Learning Figures as Computable Classi ication• We formalize learning of igures using fractals based on the Gold-style learning model towards fully computable binary classi ication (Section 2.2). We
construct a representation system for learning using self-similar sets basedon the binary representation of real numbers, and show desirable propertiesof it (Lemma 2.2, Lemma 2.3, and Lemma 2.4).
• We construct the learnability hierarchy under various learning criteria, summarized in Figure 2.3 (Section 2.3 and 2.4). We introduce four criteria forlearning: explanatory learning (Subsection 2.3.1), consistent learning (Subsection 2.3.2), reliable and refutable learning (Subsection 2.3.3), and effective
learning (Section 2.4).
• We show a mathematical connection between learning and fractal geometryby measuring the complexity of learning using the Hausdorff dimension andthe VC dimension (Section 2.5). Speci ically, we give the lower bound to thenumber of positive examples using the dimensions.
• We also show a connection between computability of igures and learnabilityof igures discussed in this chapter using TTE (Section 2.6). Learning can beviewed as computable realization of the identity from the set of igures to thesame set equipped with the iner topology.

Part II: From Theory to PracticeAll results presented in this part have been published in [B1,P3,P4,P6,P7,P8]. Chapter 3 is based on [B1,P3,P7], Chapter 4 on [P4,P6], and Chapter 5 on [P8].
Chapter 3: Coding Divergence1.1 MAIN CONTRIBUTIONS• We propose a measure of the difference between two sets of real-valued data,
called coding divergence, to computationally unify two processes of discretization and learning (De inition 3.5).
• We construct a classi ier using the divergence (Subsection 3.3.2), and experimentally illustrate its robust performance (Section 3.4).
Chapter 4: Minimum Code Length and Gray Code for Clustering• We design a measure, called the Minimum Code Length (MCL), that can scorethe quality of a given clustering result under a ixed encoding scheme (De inition 4.1).
• We propose a general strategy to translate any encoding method into a cluster
algorithm, called COOL (COding-Oriented cLustering) (Section 4.2). COOL has
a low computational cost since it scales linearly with the data set size.
• We consider the Gray Code as the encoding scheme to present G-COOL (Section 4.3). G-COOL can ind clusters of arbitrary shapes and remove noise.
• G-COOL is theoretically shown to achieve internal cohesion and external isolation and is experimentally shown to work well for both synthetic and real
datasets (Section 4.4).
Chapter 5: Clustering Using Binary Discretization• We present a new clustering algorithm, called BOOL (Binary cOding Oriented
cLustering), for multivariate data using binary discretization (Section 5.1, 5.2).
It can detect arbitrarily shaped clusters and is noise tolerant.
• Experiments show that BOOL is faster than the K-means algorithm, and about
two to three orders of magnitude faster than two state-of-the-art algorithms
that can detect non-convex clusters of arbitrary shapes (Section 5.3).
• We also show the robustness of BOOL to changes in parameters, whereas mostalgorithms for arbitrarily shaped clusters are known to be overly sensitive tosuch changes (Section 5.3).

Part III: With Formal Concept Analysis
All results presented in this part have been published in [J1,J2,P5,C1]. Chapter 6 isbased on [J1,P5] and Chapter 7 on [J2,C1].
Chapter 6: Semi-supervised Classi ication and Ranking• We present a new semi-supervised learning algorithm, called SELF (SEmisupervised Learning via FCA), which performs multiclass classi ication andlabel ranking of mixed-type data containing both discrete and continuous variables (Section 6.2). SELF uses closed set lattices, which have been recently
used for frequent pattern mining within the framework of the data analysis
technique of Formal Concept Analysis (FCA).
• SELF can weight each classi ication rule using the lattice, which gives a partial
order of preference over class labels (Section 6.2).
• We experimentally demonstrate competitive performance of SELF in classi ication and ranking compared to other learning algorithms (Section 6.3).

5

6

INTRODUCTION
Supervised Learning
Chapter 2
Chapter 3

Theoretical Analysis of Learning Figures
LLLL 2009 [P1], ALT 2010 [P2]
Coding Divergence: Measuring the Similarity between Two Sets
Book [B1], ACML 2010 [P3], ALSIP 2011 [P7]

Unsupervised Learning
Chapter 4
Chapter 5(G-)COOL: Clustering with the MCL and the Gray CodeLLLL 2011 [P4], ECML PKDD 2011 [P6]
BOOL: Clustering Using Binary Discretization
ICDM 2011 [P8]Semi-supervised Learning
Chapter 6
Chapter 7
Table 1.1 | Contributions.

SELF: Semi-supervised Learning via FCA
ICCS 2011 [P5], IDA [J1]
LIFT: Ligand Finding via FCAILP 2011 [C1], IPSJ TOM [J2]Chapter 7: Ligand Finding by Multi-label Classi ication• We mathematically model the problem of ligand inding, which is a crucial
problem in biology and biochemistry, as multi-label classi ication.
• We develop a new algorithm LIFT (Ligand FInding via Formal ConcepT Analysis) for multi-label classi ication, which can treat ligand data in databases inthe semi-supervised manner.
• We experimentally show that LIFT effectively solves our task compared toother machine learning algorithms using real data of ligands and receptors
in the IUPHAR database.

Part I
Theory“The symbol is de ined as a set of points in this square, viz.
the set occupied by printer’s ink.”— Alan Mathison Turing, On Computable Numbers, with the Application to the
Entscheidungsproblem

2
LEARNING FIGURES ASCOMPUTABLE CLASSIFICATION
discretization

Valiant-style learning modelD

is a fundamental process in machine learning from analog data.
For example, Fourier analysis is one of the most essential signal processing
methods and its discrete version, discrete Fourier analysis, is used for learning or
recognition on a computer from continuous signals. However, in the method, only
the direction of the time axis is discretized, so each data point is not purely discretized. That is to say, continuous (electrical) waves are essentially treated asinite/in inite sequences of real numbers, hence each value is still continuous (analog). The gap between analog and digital data therefore remains.
This problem appears all over machine learning from observed multivariate
data as mentioned in Introduction. The reason is that an in inite sequence is needed
to encode a real vector exactly without any numerical error, since the cardinalityof the set of real numbers, which is the same as that of in inite sequences, is muchlarger than that of the set of inite sequences. Thus to treat each data point on
a computer, it has to be discretized and considered as an approximate value withsome numerical error. However, to date, most machine learning algorithms ignore the gap between the original value and its discretized representation. Thisgap could result in some unexpected numerical errors¹. Since now machine learning algorithms can be applied to massive datasets, it is urgent to give a theoretical foundation for learning, such as classi ication, regression, and clustering, frommultivariate data, in a fully computational manner to guarantee the soundness of
the results of learning.
In the ield of computational learning theory, the Valiant-style learning model(also called PAC, Probably Approximately Correct, learning model), proposed by
Valiant (1984), is used for theoretical analysis of machine learning algorithms. In
this model, we can analyze the robustness of a learning algorithm in the face ofnoise or inaccurate data and the complexity of learning with respect to the rateof convergence or the size of the input using the concept of probability. Blumer
et al. (1989) and Ehrenfeucht et al. (1989) provided the crucial conditions for¹Müller (2001) and Schröder (2002b) give some interesting examples in the study of computation
for real numbers.

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
learnability, that is, the lower and upper bounds for the sample size, using the VC(Vapnik-Chervonenkis) dimension (Vapnik and Chervonenkis, 1971). These results VC dimension
can be applied to targets for continuous values, e.g., the learning of neural networks (Baum and Haussler, 1989). However, this learning model does not it todiscrete and computational analysis of machine learning. We cannot know whichclass of continuous objects is exactly learnable and what kind of data are needed tolearn from a inite expression of discretized multivariate data. Although Valiantstyle learning from axis-parallel rectangles have already been investigated by Longand Tan (1998), which can be viewed as a variant of learning from multivariate
data with numerical error, they are not applicable in the study since our goal is toinvestigate computational learning focusing on a common ground between “learning” and “computation” of real numbers based on the behavior of Turing machine
without any probability distribution, and we need to distinguish abstract mathematical objects such as real numbers and their concrete representations, or codes,
on a computer.
Instead, in this chapter we use the Gold-style learning model (also called identi- Gold-style learning model
ication in the limit), which is originally designed for learning of recursive functions(Gold, 1965) and languages (Gold, 1967). In the model, a learning machine is assumed to be a procedure, i.e., a Turing machine (Turing, 1937) which never halts,that receives training data from time to time, and outputs representations (hypotheses) of the target from time to time. All data are usually assumed to be given
in time. Starting from this learning model, learnability of classes of discrete objects, such as languages and recursive functions, has been analyzed in detail undervarious learning criteria (Jain et al., 1999b). However, analysis of learning for continuous objects, such as classi ication, regression, and clustering for multivariate
data, with the Gold-style learning model is still under development, despite suchsettings being typical in modern machine learning. To the best of our knowledge,
the only line of studies by Hirowatari and Arikawa (1997); Apsītis et al. (1999);
Hirowatari and Arikawa (2001); Hirowatari et al. (2003, 2005, 2006) devoted tolearning of real-valued functions, where they addressed the analysis of learnableclasses of real-valued functions using computable representations of real numbers. We therefore need a new theoretical and computational framework for modern machine learning based on the Gold-style learning model with discretizationof numerical data.
In this chapter we consider the problem of binary classi ication for multivari- binary classiﬁcation
ate data, which is one of the most fundamental problems in machine learning andpattern recognition. In this task, a training dataset consists of a set of pairs(𝑥 , 𝑦 ), (𝑥 , 𝑦 ), … , (𝑥 , 𝑦 ) ,where 𝑥 ∈ ℝ is a feature vector, 𝑦 ∈ {0, 1} is a label, and the 𝑑-dimensional feature vector
Euclidean space ℝ is a feature space. The goal is to learn a classi ier from the classiﬁer
given training dataset, that is, to ind a mapping ℎ ∶ ℝ → {0, 1} such that, for all𝑥 ∈ ℝ , ℎ(𝑥) is expected to be the same as the true label of 𝑥. In other words, sucha classi ier ℎ is the characteristic function of a subset
characteristic function
𝐿= 𝑥∈ℝℎ(𝑥) = 1

of ℝ , which has to be similar to the true set
𝐾= 𝑥∈ℝ

the true label of 𝑥 is 1

9

10

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

ﬁgure

representation

binary representation

as far as possible. Throughout the chapter, we assume that each feature is normalized by some data preprocessing such as min-max normalization for simplicity, that is, the feature space is the unit interval (cube) ℐ = [0, 1] × … × [0, 1] inthe 𝑑-dimensional Euclidean space ℝ . In many realistic scenarios, each target 𝐾
is a closed and bounded subset of ℐ , i.e., a nonempty compact subset of ℐ , calleda igure. Thus here we address the problem of binary classi ication by treating it
as “learning of igures”.
In this machine learning process, we implicitly treat any feature vector throughits representation, or code on a computer, that is, each feature vector 𝑥 ∈ ℐ isrepresented by a sequence 𝑝 over some alphabet Σ using an encoding scheme 𝜌.
Here such a surjective mapping 𝜌 is called a representation and should map theset of “in inite” sequences Σ to ℐ since there is no one-to-one correspondence
between inite sequences and real numbers (or real vectors). In this chapter, we
use the binary representation 𝜌 ∶ Σ → [0, 1] with Σ = {𝟶, 𝟷}, which is de ined by𝜌(𝑝) ≔(𝑝 ⋅2)

for an in inite sequence 𝑝 = 𝑝 𝑝 𝑝 … . For example,
𝜌(𝟶𝟷𝟶𝟶 … ) = 0.25, 𝜌(𝟷𝟶𝟶𝟶 … ) = 0.5, and 𝜌(𝟶𝟷𝟷𝟷 … ) = 0.5.
However, we cannot treat in inite sequences on a computer in inite time and, instead, we have to use discretized values, i.e., truncated inite sequences in any actualmachine learning process. Thus in learning of a classi ier ℎ for the target igure 𝐾,
we cannot use an exact data point 𝑥 ∈ 𝐾 but have to use a discretized inite se∗quence 𝑤 ∈ Σ which tells us that 𝑥 takes one of the values in the set {𝜌(𝑝) ∣ 𝑤 ⊏ 𝑝}(𝑤 ⊏ 𝑝 means that 𝑤 is a pre ix of 𝑝). For instance, if 𝑤 = 𝟶𝟷, then 𝑥 should be in∗
the interval [0.25, 0.5]. For a inite sequence 𝑤 ∈ Σ , we de ine
𝜌(𝑤) ≔ 𝜌(𝑝) 𝑤 ⊏ 𝑝 with 𝑝 ∈ Σusing the same symbol 𝜌. From a geometric point of view, 𝜌(𝑤) means a hyper
rectangle whose sides are parallel to the axes in the space ℐ . For example, for thebinary representation 𝜌, we have𝜌(𝟶) = [0, 0.5], 𝜌(𝟷) = [0.5, 1], 𝜌(𝟶𝟷) = [0.25, 0.5],
and so on. Therefore in the actual learning process, while a target set 𝐾 and eachpoint 𝑥 ∈ 𝐾 exist mathematically, a learning machine can only treat inite sequences as training data.
Here the problem of binary classi ication is stated in a computational manner
as follows: given a training dataset(𝑤 , 𝑦 ), (𝑤 , 𝑦 ), … , (𝑤 , 𝑦 )∗(𝑤 ∈ Σ for each 𝑖 ∈ {1, 2, … , 𝑛}), where𝑦 =1 if 𝜌(𝑤 ) ∩ 𝐾 ≠ ∅ for a target igure 𝐾 ⊆ ℐ
0 otherwise,∗

positive example

learn a classi ier ℎ ∶ Σ → {0, 1} for which ℎ(𝑤) should be the same as the truelabel of 𝑤. Each training datum (𝑤 , 𝑦 ) is called a positive example if 𝑦 = 1 and a

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

11

Positive examples
Negative examples
Learner

Self-similar set represented Figure 2.1 | Framework of learning
ﬁgures.
by hypothesis

Target figure

negative example if 𝑦 = 0.
negative example
Assume that a igure 𝐾 is represented by a set 𝑃 of in inite sequences, i.e.,{𝜌(𝑝) ∣ 𝑝 ∈ 𝑃} = 𝐾, using the binary representation 𝜌. Then learning the igure isdifferent from learning the well-known pre ix closed set Pref(𝑃), de ined as
preﬁx closed set∗

Pref(𝑃) ≔ 𝑤 ∈ Σ

𝑤 ⊏ 𝑝 for some 𝑝 ∈ 𝑃 ,∗since generally Pref(𝑃) ≠ { 𝑤 ∈ Σ ∣ 𝜌(𝑤) ∩ 𝐾 ≠ ∅ } holds. For example, if 𝑃 ={ 𝑝 ∈ Σ ∣ 𝟷 ⊏ 𝑝 }, the corresponding igure 𝐾 is the interval [0.5, 1]. The in inite
sequence 𝟶𝟷𝟷𝟷 … is a positive example since 𝜌(𝟶𝟷𝟷𝟷 … ) = 0.5 and 𝜌(𝟶𝟷𝟷𝟷 … ) ∩𝐾 ≠ ∅, but it is not contained in Pref(𝑃). Solving this mismatch between objects oflearning and their representations is one of the challenging problems of learning
continuous objects based on their representation in a computational manner.
For inite expression of classi iers, we use self-similar sets known as a family self-similar set
of fractals (Mandelbrot, 1982) to exploit their simplicity and the power of expres- fractals
sion theoretically provided by the ield of fractal geometry. Speci ically, we canapproximate any igure by some self-similar set arbitrarily closely (derived fromthe Collage Theorem given by Falconer (2003)) and can compute it by a simple
recursive algorithm, called an IFS (Iterated Function System) (Barnsley, 1993; Fal- IFS (Iterated Function System)
coner, 2003). This approach can be viewed as the analog of the discrete Fourier
analysis, where FFT (Fast Fourier Transformation) is used as the fundamental recursive algorithm. Moreover, in the process of sampling from analog data in discrete Fourier analysis, scalability is a desirable property. It requires that when thesample resolution increases, the accuracy of the result is monotonically re ined.
We formalize this property as effective learning of igures, which is inspired by effective computing in the framework of Type-2 Theory of Effectivity (TTE) studiedin computable analysis (Schröder, 2002a; Weihrauch, 2000). This model guarantees that as a computer reads more and more precise information of the input, it
produces more and more accurate approximations of the result. Here we interpret
this model from computation to learning, where if a learner (learning machine) receives more and more accurate training data, it learns better and better classi iers
(self-similar sets) approximating the target igure.
To summarize, our framework of learning igures (shown in Figure 2.1) is asfollows: Positive examples are axis-parallel rectangles intersecting the target igure, and negative examples are those disjoint with the target. A learner reads apresentation (in inite sequence of examples), and generates hypotheses. A hypothesis is a inite set of inite sequences (codes), which is a discrete expressionof a self-similar set. To evaluate “goodness” of each classi ier, we use the conceptof generalization error and measure it by the Hausdorff metric since it induces the Hausdorﬀ metricstandard topology on the set of igures (Beer, 1993).

12

LEARNING FIGURES AS COMPUTABLE CLASSIFICATIONThe rest of the chapter is organized as follows: We review related work in comparison to the present work in Section 2.1. We formalize computable binary classiication as learning of igures in Section 2.2 and analyze the learnability hierarchy
induced by variants of our model in Section 2.3 and Section 2.4. The mathematicalconnection between fractal geometry and the Gold-style learning model with theHausdorff and the VC dimensions is presented in Section 2.5 and between computability and learnability of igures in Section 2.6. Section 2.7 gives the summaryof this chapter.

2.1 Related Work
Statistical approaches to machine learning are achieving great success (Bishop,
2007) since they are originally designed for analyzing observed multivariate data
and, to date, many statistical methods have been proposed to treat continuous objects such as real-valued functions. However, most methods pay no attention todiscretization and the inite representation of analog data on a computer. For
example, multi-layer perceptrons are used to learn real-valued functions, since
they can approximate every continuous function arbitrarily and accurately. However, a perceptron is based on the idea of regulating analog wiring (Rosenblatt,1958), hence such learning is not purely computable, i.e., it ignores the gap between analog raw data and digital discretized data. Furthermore, although several
discretization techniques have been proposed (Elomaa and Rousu, 2003; Fayyad
and Irani, 1993; Gama and Pinto, 2006; Kontkanen et al., 1997; Li et al., 2003; Linet al., 2003; Liu et al., 2002; Skubacz and Hollmén, 2000), they treat discretization
as data preprocessing for improving the accuracy or ef iciency of machine learning algorithms. The process of discretization is not therefore considered from acomputational point of view, and “computability” of machine learning algorithms
is not discussed at suf icient depth.
There are several related work considering learning under various restrictionsin the Gold-style learning model (Goldman et al., 2003), the Valiant-style learningmodel (Ben-David and Dichterman, 1998; Decatur and Gennaro, 1995), and otherlearning context (Khardon and Roth, 1999). Moreover, recently learning from partial examples, or examples with missing information, has attracted much attention
in the Valiant-style learning model (Michael, 2010, 2011). In this chapter we also
consider learning from examples with missing information, which are truncatedinite sequences. However, our model is different from them, since the “missing information” in this chapter corresponds to measurement error of real-valued data.
As mentioned in Introduction (Chapter 1), our motivation comes from actual measurement/observation of a physical object, where every datum obtained by an experimental instrument must have some numerical error in principle (Baird, 1994).
For example, if we measure the size of a cell by a microscope equipped with micrometers, we cannot know the true value of the size but an approximate value
with numerical error, which depends on the degree of magni ication by the micrometers. In this chapter we try to treat this process as learning from multivariate data, where an approximate value corresponds to a truncated inite sequenceand error becomes small as the length of the sequence increases, intuitively. The
asymmetry property of positive and negative examples is naturally derived fromthe motivation. The model of computation for real numbers within the frameworkof TTE its to our motivation, which is unique in computational learning theory.

2.2 FORMALIZATION OF LEARNING
Self-similar sets can be viewed as a geometric interpretation of languages recognized by 𝜔-automata (Perrin and Pin, 2004), irst introduced by Büchi (1960), 𝜔-automaton
and learning of such languages has been investigated by De La Higuera and Janodet (2001); Jain et al. (2011). Both works focus on learning 𝜔-languages fromtheir pre ixes, i.e. texts (positive data), and show several learnable classes. Thisapproach is different from ours since our motivation is to address computability
issues in the ield of machine learning from numerical data, and hence there is a
gap between pre ixes of 𝜔-languages and positive data for learning in our setting.
Moreover, we consider learning from both positive and negative data, which is a
new approach in the context of learning of in inite words.
To treat values with numerical errors on computers, various effective methods have been proposed in the research area of numerical computation with result veri ication (Oishi, 2008). Originally, they also used an interval as a representation of an approximate value and, recently, some ef icient techniques withloating-point numbers have been presented (Ogita et al., 2005). While they focus
on computation with numerical errors, we try to embed the concept of errors into
learning based on the computation schema of TTE using interval representationof real numbers. Considering relationship between our model and such methods
discussed in numerical computation with result veri ication and constructing ef icient algorithms using the methods is an interesting future work.

2.2

Formalization of Learning

To analyze binary classi ication in a computable approach, we irst formalize learning of igures based on the Gold-style learning model. Speci ically, we de ine targets of learning, representations of classi iers produced by a learning machine,
and a protocol for learning. In the following, let ℕ be the set of natural numbers
including 0, ℚ the set of rational numbers, and ℝ the set of real numbers. The
set ℕ (resp. ℝ ) is the set of positive natural (resp. real) numbers. The 𝑑-foldproduct of ℝ is denoted by ℝ and the set of nonempty compact subsets of ℝ is∗denoted by 𝒦 .
Throughout this chapter, we use the binary representation 𝜌 ∶ (Σ ) → ℐ as binary representation
the canonical representation for real numbers. If 𝑑 = 1, this is de ined as follows:Σ = {𝟶, 𝟷} and𝜌 (𝑝) ≔

𝑝 ⋅2

(

)

(2.1)

for an in inite sequence 𝑝 = 𝑝 𝑝 𝑝 … . Note that Σ denotes the set {𝑎 𝑎 … 𝑎 ∣𝑎 ∈ Σ} and Σ = Σ. For example, 𝜌 (𝟶𝟷𝟶𝟶 … ) = 0.25, 𝜌 (𝟷𝟶𝟶𝟶 … ) = 0.5, andso on. Moreover, by using the same symbol 𝜌, we introduce a representation 𝜌 ∶∗∗Σ → 𝒦 for inite sequences de ined as follows:𝜌 (𝑤) ≔ 𝜌 (↑𝑤) = [ 𝜌(𝑤𝟶𝟶𝟶 … ), 𝜌(𝑤𝟷𝟷𝟷 … ) ]
=

𝑤 ⋅2

(

)

,

𝑤 ⋅2

(

)

+2

| |

,

(2.2)

where ↑𝑤 = { 𝑝 ∈ Σ ∣ 𝑤 ⊏ 𝑝 }. For instance, 𝜌 (𝟶𝟷) = [0.25, 0.5] and 𝜌 (𝟷𝟶) =[0.5, 0.75].

13

14

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
In a 𝑑-dimensional space with 𝑑 > 1, we use the 𝑑-dimensional binary representation 𝜌 ∶ (Σ ) → ℐ de ined in the following manner.
𝜌 (⟨ 𝑝 , 𝑝 , … , 𝑝 ⟩) ≔ 𝜌 (𝑝 ), 𝜌 (𝑝 ), … , 𝜌 (𝑝 ) ,(2.3)

where 𝑑 in inite sequences 𝑝 , 𝑝 , … , and 𝑝 are concatenated using the tupling
function ⟨⋅⟩ such that
⟨𝑝 ,𝑝 ,…,𝑝 ⟩ ≔ 𝑝 𝑝 …𝑝 𝑝 𝑝 …𝑝 𝑝 𝑝 …𝑝 ….
∗

(2.4)

∗

Similarly, we de ine a representation 𝜌 ∶ (Σ ) → 𝒦 by
𝜌 (⟨ 𝑤 , 𝑤 , … , 𝑤 ⟩) ≔ 𝜌 (↑⟨ 𝑤 , 𝑤 , … , 𝑤 ⟩),where⟨𝑤 ,𝑤 ,…,𝑤 ⟩ ≔ 𝑤 𝑤 …𝑤 𝑤 𝑤 …𝑤 …𝑤 𝑤 …𝑤 .
∗languagewith |𝑤 | = |𝑤 | = ⋯ = |𝑤 | = 𝑛. Note that, for any 𝑤 = ⟨ 𝑤 , … , 𝑤 ⟩ ∈ (Σ ) ,|𝑤 | = |𝑤 | = ⋯ = |𝑤 | always holds, and we denote the length by |𝑤| in this∗chapter. For a set of inite sequences, i.e., a language 𝐿 ⊂ Σ , we de ine𝜌 (𝐿) ≔ 𝜌 (𝑤) 𝑤 ∈ 𝐿 .

contraction

We omit the superscript 𝑑 of 𝜌 if it is understood from the context.
∗
A target set of learning is a set of igures ℱ ⊆ 𝒦 ixed a priori, and one of themis chosen as a target in each learning term. A learning machine uses self-similarsets, known as fractals and de ined by inite sets of contractions. This approach isone of the key ideas in this chapter. Here, a contraction is a mapping CT ∶ ℝ → ℝ
such that, for all 𝑥, 𝑦 ∈ 𝑋, 𝑑(CT(𝑥), CT(𝑦)) ≤ 𝑐𝑑(𝑥, 𝑦) for some real number 𝑐 with
0 < 𝑐 < 1. For a inite set of contractions 𝐶, a nonempty compact set 𝐹 satisfying
CT(𝐹)

𝐹=
∈

self-similar set

is determined uniquely (see the book (Falconer, 2003) for its formal proof). The∗
set 𝐹 is called the self-similar set of 𝐶. Moreover, if we de ine a mapping 𝐂𝐓 ∶ 𝒦 →∗𝒦 by𝐂𝐓(𝐾) ≔

(2.5)

CT(𝐾)
∈

and de ine
𝐂𝐓 (𝐾) ≔ 𝐾 and 𝐂𝐓(𝐾) ≔ 𝐂𝐓(𝐂𝐓 (𝐾))for each 𝑘 ∈ ℕ recursively, then𝐹=

𝐂𝐓 (𝐾)(2.6)

s

2.2 FORMALIZATION OF LEARNING∗

for every 𝐾 ∈ 𝒦 such that CT(𝐾) ⊂ 𝐾 for every CT ∈ 𝐶. This means that we havea level-wise construction algorithm with 𝐂𝐓 to obtain the self-similar set 𝐹.
Actually, a learning machine produces hypotheses, each of which is a inite language and becomes a inite expression of a self-similar set that works as a classi ier.
∗
Formally, for a inite language 𝐻 ⊂ (Σ ) , we consider 𝐻 , 𝐻 , 𝐻 , … such that 𝐻
is recursively de ined as follows:𝐻 ≔ {𝜆},𝐻 ≔ ⟨𝑤 𝑢 ,𝑤 𝑢 ,…,𝑤 𝑢 ⟩⟨𝑤 ,𝑤 ,…,𝑤 ⟩ ∈ 𝐻⟨𝑢 ,𝑢 ,…,𝑢 ⟩ ∈ 𝐻

and.

We can easily construct a ixed program which generates 𝐻 , 𝐻 , 𝐻 , … when receiving a hypothesis 𝐻. We give the semantics of a hypothesis 𝐻 by the following
equation:
𝜅(𝐻) ≔

𝜌(𝐻 ).

(2.7)

Since ⋃ 𝜌(𝐻 ) ⊃ ⋃ 𝜌(𝐻 ) holds for all 𝑘 ∈ ℕ, 𝜅(𝐻) = lim → ⋃ 𝜌(𝐻 ). We∗
denote the set of hypotheses {𝐻 ⊂ (Σ ) ∣ 𝐻 is inite} by ℋ and call it the hypothesis
space. We use this hypothesis space throughout the chapter. Note that, for a pair hypothesis spaceof hypotheses 𝐻 and 𝐿, 𝐻 = 𝐿 implies 𝜅(𝐻) = 𝜅(𝐿), but the converse may not hold.
Example 2.1
Assume 𝑑 = 2 and let a hypothesis 𝐻 be the set {⟨ 𝟶, 𝟶 ⟩, ⟨ 𝟶, 𝟷 ⟩, ⟨ 𝟷, 𝟷 ⟩} = {𝟶𝟶, 𝟶𝟷,𝟷𝟷}. We have
𝐻 = ∅,

𝐻 = {⟨ 𝟶, 𝟶 ⟩, ⟨ 𝟶, 𝟷 ⟩, ⟨ 𝟷, 𝟷 ⟩} = {𝟶𝟶, 𝟶𝟷, 𝟷𝟷},𝐻 =⟨ 𝟶𝟶, 𝟶𝟶 ⟩, ⟨ 𝟶𝟶, 𝟶𝟷 ⟩, ⟨ 𝟶𝟷, 𝟶𝟷 ⟩, ⟨ 𝟶𝟶, 𝟷𝟶 ⟩, ⟨ 𝟶𝟶, 𝟷𝟷 ⟩,⟨ 𝟶𝟷, 𝟷𝟷 ⟩, ⟨ 𝟷𝟶, 𝟷𝟶 ⟩, ⟨ 𝟷𝟶, 𝟷𝟷 ⟩, ⟨ 𝟷𝟷, 𝟷𝟷 ⟩= {𝟶𝟶𝟶𝟶, 𝟶𝟶𝟶𝟷, 𝟶𝟶𝟷𝟷, 𝟶𝟷𝟶𝟶, 𝟶𝟷𝟶𝟷, 𝟶𝟷𝟷𝟷, 𝟷𝟷𝟶𝟶, 𝟷𝟷𝟶𝟷, 𝟷𝟷𝟷𝟷}, …
and the igure 𝜅(𝐻) de ined in the equation (2.7) is the Sierpiński triangle (Figure Sierpiński triangle
2.2). If we consider the following three mappings:
CT

𝑥
𝑥

CT

𝑥
𝑥

CT

𝑥
𝑥

1 𝑥
2 𝑥
1 𝑥
=
2 𝑥
=

=

1 𝑥
2 𝑥

+

0
,
0

+

0
,
1/2

+

1/2
,
1/2

the three squares CT (ℐ ), CT (ℐ ), and CT (ℐ ) are exactly the same as 𝜌(𝟶𝟶),𝜌(𝟶𝟷), and 𝜌(𝟷𝟷), respectively. Thus each sequence in a hypothesis can be viewed
as a representation of one of these squares, which are called generators for a self- generator
similar set since if we have the initial set ℐ and generators CT (ℐ ), CT (ℐ ), andCT (ℐ ), we can reproduce the three mappings CT , CT , and CT and construct
the self-similar set from them. Note that there exist in initely many hypotheses 𝐿
such that 𝜅(𝐻) = 𝜅(𝐿) and 𝐻 ≠ 𝐿. For example, 𝐿 = {⟨ 𝟶, 𝟶 ⟩, ⟨ 𝟷, 𝟷 ⟩, ⟨ 𝟶𝟶, 𝟷𝟶 ⟩,⟨ 𝟶𝟶, 𝟷𝟷 ⟩, ⟨ 𝟶𝟷, 𝟷𝟷 ⟩}.

15

16

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

1

0
1

ρ(H0)

1

0
ρ(H4)

1

0
1

Figure 2.2 | Generation of the Sierpiński triangle from the hypothesis H= {⟨0,1⟩, ⟨0,1⟩, ⟨1,1⟩} (Example 2.1). 0 0

1

0

ρ(H1)

1

0
ρ(H5)

1

0

1

0
1

0

ρ(H2)

1

0
ρ(H6)

1

0

1

0
1

0

ρ(H3)

0
ρ(H7)

1

0

1

Lemma 2.2: Soundness of hypotheses
For every hypothesis 𝐻 ∈ ℋ, the set 𝜅(𝐻) de ined by the equation (2.7) is a selfsimilar set.
Proof. Let 𝐻 = {𝑤 , 𝑤 , … , 𝑤 }. We can easily check that the set of rectangles
𝜌(𝑤 ), … , 𝜌(𝑤 ) is a generator de ined by the mappings CT , … , CT , where each
CT maps the unit interval ℐ to the igure 𝜌(𝑤 ). De ine 𝐂𝐓 and 𝐂𝐓 in the same
way as the equations (2.5) and (2.6). For each 𝑘 ∈ ℕ,
𝜌(𝐻 ) = 𝐂𝐓 (ℐ )holds. It follows that the set 𝜅(𝐻) is exactly the same as the self-similar set de inedby the mappings CT , CT , … , CT , that is, 𝜅(𝐻) = ⋃ CT (𝜅(𝐻)) holds.
generalization error
Hausdorﬀ metric

To evaluate the “goodness” of each hypothesis, we use the concept of generalization error, which is usually used to score the quality of hypotheses in a machinelearning context. The generalization error of a hypothesis 𝐻 for a target igure 𝐾,
written by GE(𝐾, 𝐻), is de ined by the Hausdorff metric 𝑑H on the space of igures,
GE(𝐾, 𝐻) ≔ 𝑑H (𝐾, 𝜅(𝐻)) = inf 𝛿 𝐾 ⊆ 𝜅(𝐻) and 𝜅(𝐻) ⊆ 𝐾

𝛿-neighborhood,

where 𝐾 is the 𝛿-neighborhood of 𝐾 de ined by𝐾 ≔ 𝑥∈ℝ𝑑E (𝑥, 𝑎) ≤ 𝛿 for some 𝑎 ∈ 𝐾 .

The metric 𝑑E is the Euclidean metric such that𝑑E (𝑥, 𝑎) =(𝑥 − 𝑎 )for 𝑥 = (𝑥 , … , 𝑥 ), 𝑎 = (𝑎 , … , 𝑎 ) ∈ ℝ . The Hausdorff metric is one of the stan∗
dard metrics on the space since the metric space (𝒦 , 𝑑H ) is complete (in the sense2.2 FORMALIZATION OF LEARNING
of topology) and GE(𝐾, 𝐻) = 0 if and only if 𝐾 = 𝜅(𝐻) Beer (1993); Kechris (1995).
∗
The topology on 𝒦 induced by the Hausdorff metric is called the Vietoris topology. Since the cardinality of the set of hypotheses ℋ is smaller than that of the set Vietoris topology∗
of igures 𝒦 , we often cannot ind the exact hypothesis 𝐻 for a igure 𝐾 such that
GE(𝐾, 𝐻) = 0. However, following the Collage Theorem given by Falconer (2003),
we show that the power of representation of hypotheses is still suf icient, that is,
we always can approximate a given igure arbitrarily closely by some hypothesis.
Lemma 2.3: Representational power of hypotheses∗
For any 𝛿 ∈ ℝ and for every igure 𝐾 ∈ 𝒦 , there exists a hypothesis 𝐻 such that
GE(𝐾, 𝐻) < 𝛿.
Proof. Fix a igure 𝐾 and the parameter 𝛿. Here we denote the diameter of the set
𝜌(𝑤) with |𝑤| = 𝑘 by diam(𝑘). Then we havediam(𝑘) = √𝑑 ⋅ 2.

For example, diam(1) = 1/2 and diam(2) = 1/4 if 𝑑 = 1, and diam(1) = 1/√2and diam(2) = 1/√8 if 𝑑 = 2. For 𝑘 with diam(𝑘) < 𝛿, let∗

𝐻 = 𝑤 ∈ (Σ )|𝑤| = 𝑘 and 𝜌(𝑤) ∩ 𝐾 ≠ ∅ .

We can easily check that the diam(𝑘)-neighborhood of the igure 𝐾 contains 𝜅(𝐻)and diam(𝑘)-neighborhood of 𝜅(𝐻) contains 𝐾. Thus we have GE(𝐾, 𝐻) < 𝛿.
There are many other representation systems that meet the following condition. One of remarkable features of our system with self-similar sets will be shownin Lemma 2.37. Moreover, to work as a classi ier, every hypothesis 𝐻 has to be∗∗computable, that is, the function ℎ ∶ (Σ ) → {0, 1} such that, for all 𝑤 ∈ (Σ ) ,computableℎ(𝑤) =

1 if 𝜌(𝑤) ∩ 𝜅(𝐻) ≠ ∅,0 otherwise(2.8)

should be computable. We say that such ℎ is the classi ier of 𝐻. The computability classiﬁerof ℎ is not trivial, since for a inite sequence 𝑤, the two conditions ℎ(𝑤) = 1 and𝑤 ∈ 𝐻 are not equivalent. Intuitively, this is because each interval represented
by a inite sequence is closed. For example, in the case of Example 2.1, ℎ(𝟷𝟶) = 1 closed
because 𝜌(𝟷𝟶) = [0.5, 1] × [0, 0.5] and 𝜌(𝟷𝟶) ∩ 𝜅(𝐻) = {(0.5, 0.5)} ≠ ∅ whereas
𝟷𝟶 ∉ 𝐻 for any 𝑘 ∈ ℕ. Here we guarantee this property of computability.
Lemma 2.4: Computability of classi iers
For every hypothesis 𝐻 ∈ ℋ, the classi ier ℎ of 𝐻 de ined by the equation (2.8) is
computable.
Proof. First we consider whether or not the boundary of an interval is containedin 𝜅(𝐻). Suppose 𝑑 = 1 and let 𝐶 be a inite set of contractions and 𝐹 be theself-similar set of 𝐶. We have the following property: For every interval [𝑥, 𝑦] =CT ∘ CT ∘ … ∘ CT (ℐ ) such that CT ∈ 𝐶 for all 𝑖 ∈ {1, … , 𝑛} (𝑛 ∈ ℕ), we have𝑥 ∈ 𝐹 (resp. 𝑦 ∈ 𝐹) if and only if 0 ∈ CT(ℐ ) (resp. 1 ∈ CT(ℐ )) for some CT ∈ 𝐶.
This means that if [𝑥, 𝑦] = 𝜌(𝑣) with a sequence 𝑣 ∈ 𝐻 (𝑘 ∈ ℕ) for a hypothesis

17

18

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

Algorithm 2.1: Classi ier ℎ of hypothesis 𝐻
Input: Finite sequence 𝑤 and hypothesis 𝐻
Output: Class label 1 or 0 of 𝑤
1: 𝑘 ← 0
2: repeat3:𝑘←𝑘+1
4: until min ∈ |𝑣| > |𝑤|
5: for each 𝑣 ∈ 𝐻
6:
if 𝑤 ⊑ 𝑣 then7:
output 1 and halt
8:
else if C
B(𝑤, 𝑣, 𝐻) = 1 then9:
output 1 and halt
10:
end if
11: end for
12: output 0
function C
B(𝑤, 𝑣, 𝐻)
1: 𝑎 ← 𝑎 𝑎 … 𝑎// 𝑎 is a inite sequence whose length is 𝑑
2: for each 𝑎 in {𝑎 , 𝑎 , … , 𝑎 }3:
if 𝑤 ⊑ 𝑣 then 𝑎 ← ⊥
4:
else
5:
if 𝑤 ⊑ 𝑣 then 𝑎 ← 𝟶
6:else if 𝑤 ⊑ 𝑣 then 𝑎 ← 𝟷
7:else return 0
8:
end if9:
end if
10: end for
11: for each 𝑢 ∈ 𝐻
12:
if 𝑢 = 𝑎𝑎𝑎 … 𝑎 then return 1
13: end for
14: return 0

𝐻, we have 𝑥 ∈ 𝜅(𝐻) (resp. 𝑦 ∈ 𝜅(𝐻)) if and only if 𝑢 ∈ {𝟶} (resp. 𝑢 ∈ {𝟷} ) forsome 𝑢 ∈ 𝐻.
We show a pseudo-code of the classi ier ℎ in Algorithm 2.1 and prove that theoutput of the algorithm is 1 if and only if ℎ(𝑤) = 1, i.e., 𝜌(𝑤) ∩ 𝜅(𝐻) ≠ ∅. In
the algorithm, 𝑣 and 𝑣 denote the previous and subsequent binary sequences of𝑣 with |𝑣 | = |𝑣 | = |𝑣 | in the lexicographic order, respectively. For example,
if 𝑣 = 𝟶𝟶𝟷, 𝑣 = 𝟶𝟶𝟶 and 𝑣 = 𝟶𝟷𝟶. Moreover, we use the special symbol ⊥meaning unde inedness, that is, 𝑣 = 𝑤 if and only if 𝑣 = 𝑤 for all 𝑖 ∈ {0, 1, … , |𝑣|−1} with 𝑣 ≠ ⊥ and 𝑤 ≠ ⊥.
The “if” part: For an input of a inite sequence 𝑤 and a hypothesis 𝐻, if ℎ(𝑤) =1, there are two possibilities as follows:
1. For some 𝑘 ∈ ℕ, there exists 𝑣 ∈ 𝐻 such that 𝑤 ⊑ 𝑣. This is because 𝜌(𝑤) ⊇𝜌(𝑣) and 𝜌(𝑤) ∩ 𝜅(𝐻) ≠ ∅.

2.2 FORMALIZATION OF LEARNING

19

2. The above condition does not hold, but 𝜌(𝑤) ∩ 𝜅(𝐻) ≠ ∅.
In the irst case, the algorithm goes to line 7 and stops with outputting 1. The
second case means that the algorithm uses the function C
B. Sinceℎ(𝑤) = 1, there should exist a sequence 𝑣 ∈ 𝐻 such that 𝑢 = 𝑎 for some 𝑢 ∈ 𝐻,
where 𝑎 is obtained in lines 1–10. C
B
therefore returns 1.
The “only if” part: In Algorithm 2.1, if 𝑣 ∈ 𝐻 satis ies conditions in line 6 orline 8, ℎ(𝑤) ∩ 𝜅(𝐻) ≠ ∅. Thus ℎ(𝑤) = 1 holds.
∗

The set { 𝜅(𝐻) ∣ 𝐻 ⊂ (Σ ) and the classi ier ℎ of 𝐻 is computable } exactly corresponds to an indexed family of recursive concepts / languages discussed in com- indexed family of recursive concepts
putational learning theory (Angluin, 1980), which is a common assumption for∗learning of languages. On the other hand, there exists some class of igures ℱ ⊆ 𝒦that is not an indexed family of recursive concepts. This means that, for some igure 𝐾, there is no computable classi ier which classi ies all data correctly. Therefore we address the problems of both exact and approximate learning of igures toobtain a computable classi ier for any target igure.
We consider two types of input data streams, one includes both positive andnegative data and the other includes only positive data, to analyze learning basedon the Gold-style learning model. Formally, each training datum is called an ex∗
ample and is de ined as a pair (𝑤, 𝑙) of a inite sequence 𝑤 ∈ (Σ ) and a label example𝑙 ∈ {0, 1}. For a target igure 𝐾, we de ine𝑙≔

1 if 𝜌(𝑤) ∩ 𝐾 ≠ ∅ (positive example),0 otherwise (negative example).

positive example
negative example

In the following, for a target igure 𝐾, we denote the set of inite sequences of posi∗tive examples {𝑤 ∈ (Σ ) ∣ 𝜌(𝑤)∩𝐾 ≠ ∅} by Pos(𝐾) and that of negative examples
by Neg(𝐾). From the geometric nature of igures, we obtain the following monotonicity of examples:monotonicity
Lemma 2.5: Monotonicity of examples
If (𝑣, 1) is an example of 𝐾, then (𝑤, 1) is an example of 𝐾 for all pre ixes 𝑤 ⊑ 𝑣,
and (𝑣𝑎, 1) is an example of 𝐾 for some 𝑎 ∈ Σ . If (𝑤, 0) is an example of 𝐾, then∗(𝑤𝑣, 0) is an example of 𝐾 for all 𝑣 ∈ (Σ ) .
Proof. From the de inition of the representation 𝜌 in the equations (2.1) and (2.3),
if 𝑤 ⊑ 𝑣, we have 𝜌(𝑤) ⊇ 𝜌(𝑣), hence (𝑤, 1) is an example of 𝐾. Moreover,𝜌(𝑣𝑎) = 𝜌(𝑢)∈holds. Thus there should exist an example (𝑣𝑎, 1) for some 𝑎 ∈ Σ . Furthermore,∗
for all 𝑣 ∈ Σ , 𝜌(𝑤𝑣) ⊂ 𝜌(𝑤). Therefore if 𝐾 ∩ 𝜌(𝑤) = ∅, then 𝐾 ∩ 𝜌(𝑤𝑣) = ∅ for∗all 𝑣 ∈ (Σ ) , and (𝑤𝑣, 0) is an example of 𝐾.
We say that an in inite sequence 𝜎 of examples of a igure 𝐾 is a presentation of presentation
𝐾. The 𝑖th example is denoted by 𝜎(𝑖 − 1), and the set of all examples occurring in
𝜎 is denoted by range(𝜎)². The initial segment of 𝜎 of length 𝑛, i.e., the sequence²The reason for this notation is thatof examples.

can be viewed as a mapping from ℕ (including 0) to the set

20

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
Target igure 𝐾

Table 2.1 | Relationship between theconditions for each ﬁnite sequencew ∈ ∑* and the standard notation ofbinary classiﬁcation.

Hypothesis 𝐻𝑤 ∈ Pos(𝐾)(𝜌(𝑤) ∩ 𝐾 ≠ ∅)𝑤 ∈ Neg(𝐾)(𝜌(𝑤) ∩ 𝐾 = ∅)ℎ(𝑤) = 1(𝜌(𝑤) ∩ 𝜅(𝐻) ≠ ∅)True positive

False positive(Type I error)ℎ(𝑤) = 0(𝜌(𝑤) ∩ 𝜅(𝐻) = ∅)False negative(Type II error)

True negative𝜎(0), 𝜎(1), … , 𝜎(𝑛 − 1), is denoted by 𝜎[𝑛 − 1]. A text of a igure 𝐾 is a presentation
𝜎 such that
{ 𝑤 (𝑤, 1) ∈ range(𝜎) } = Pos(𝐾)informant( = { 𝑤 𝜌(𝑤) ∩ 𝐾 ≠ ∅ } ),and an informant is a presentation 𝜎 such that
{ 𝑤 (𝑤, 1) ∈ range(𝜎) } = Pos(𝐾) and{ 𝑤 (𝑤, 0) ∈ range(𝜎) } = Neg(𝐾).
Table 2.1 shows the relationship between the standard terminology in classi ication and our de initions. For a target igure 𝐾 and the classi ier ℎ of a hypothesis𝐻, the set{ 𝑤 ∈ Pos(𝐾) ℎ(𝑤) = 1 }true positive

corresponds to true positive,{ 𝑤 ∈ Neg(𝐾) ℎ(𝑤) = 1 }false positive (type I error)

false positive (type I error),{ 𝑤 ∈ Pos(𝐾) ℎ(𝑤) = 0 }false negative (type II error)

false negative (type II error), and{ 𝑤 ∈ Neg(𝐾) ℎ(𝑤) = 0 }true negative
consistentlearnertrue negative.
Let ℎ be the classi ier of a hypothesis 𝐻. We say that the hypothesis 𝐻 is consistent with an example (𝑤, 𝑙) if 𝑙 = 1 implies ℎ(𝑤) = 1 and 𝑙 = 0 implies ℎ(𝑤) = 0,
and consistent with a set of examples 𝐸 if 𝐻 is consistent with all examples in 𝐸.
A learning machine, called a learner, is a procedure, (i.e. a Turing machine thatnever halts) that reads a presentation of a target igure from time to time, andoutputs hypotheses from time to time. In the following, we denote a learner by M
and an in inite sequence of hypotheses produced by M on the input 𝜎 by M , andM (𝑖 − 1) denotes the 𝑖th hypothesis produced by M. Assume that M receives 𝑗
examples 𝜎(0), 𝜎(1), … , 𝜎(𝑗 − 1) so far when it outputs the 𝑖th hypothesis M (𝑖 −1). We do not require the condition 𝑖 = 𝑗, that is, the inequation 𝑖 ≤ 𝑗 usuallyholds since M can “wait” until it receives enough examples. We say that an in inite
sequence of hypotheses M converges to a hypothesis 𝐻 if there exists 𝑛 ∈ ℕ suchthat M (𝑖) = 𝐻 for all 𝑖 ≥ 𝑛.

2.3 EXACT LEARNING OF FIGURES21

FIGEX-INF = FIGCONS-INF = FIGRELEX-INF = FIGEFEX-INF
FIGREFEX-INF

FIGEX-TXT = FIGCONS-TXT

FIGRELEX-TXT

FIGREFEX-TXT

FIGEFEX-TXT = ∅2.3

Figure 2.3 | Learnability hierarchy. In
each line, the lower set is a proper
subset of the upper set.

Exact Learning of Figures

We analyze “exact” learning of igures. This means that, for any target igure 𝐾,
there should be a hypothesis 𝐻 such that the generalization error is zero (i.e., 𝐾 =𝜅(𝐻)), hence the classi ier ℎ of 𝐻 can classify all data correctly with no error, that is,ℎ satis ies the equation (2.8). The goal is to ind such a hypothesis 𝐻 from examples(training data) of 𝐾.
In the following two sections (Sections 2.3 and 2.4), we follow the standard
path of studies in computational learning theory (Jain et al., 1999b; Jain, 2011;
Zeugmann and Zilles, 2008), that is, we de ine learning criteria to understand various learning situations and construct a learnability hierarchy under the criteria.
We summarize our results in Figure 2.3.

2.3.1 Explanatory Learning
The most basic learning criterion in the Gold-style learning model is E -learning(EX means EXplain), or learning in the limit, proposed by Gold (1967). We call learning in the limit
these criteria F E -I - (INF means an informant) and F E -T -learning (TXT
means a text) for E -learning from informants and texts, respectively. We introduce these criteria into the learning of igures, and analyze the learnability.
De inition 2.6: Explanatory learning∗
A learner M F E -I -learns (resp. F E -T -learns) a set of igures ℱ ⊆ 𝒦 FIGEX-INF-learning
if for all igures 𝐾 ∈ ℱ and all informants (resp. texts) 𝜎 of 𝐾, the outputs M FIGEX-TXT-learning
converge to a hypothesis 𝐻 such that GE(𝐾, 𝐻) = 0.
For every learning criterion CR introduced in the following, we say that a set ofigures ℱ is CR-learnable if there exists a learner that CR-learns ℱ, and denote by
CR the collection of CR-learnable sets of igures following the standard notation ofthis ield (Jain et al., 1999b).
First, we consider F E -I -learning. Informally, a learner can F E -I learn a set of igures if it has an ability to enumerate all hypotheses and to judge
whether or not each hypothesis is consistent with the received examples (Gold,22

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

Procedure 2.2: Learning procedure that F E -I

-learns 𝜅(ℋ)

Input: Informant 𝜎 = (𝑤 , 𝑙 ), (𝑤 , 𝑙 ), … of igure 𝐾 ∈ 𝜅(ℋ)
Output: In inite sequence of hypotheses M (0), M (1), …
1: 𝑖 ← 0
2: 𝐸 ← ∅ // 𝐸 is a set of received examples
3: repeat4:
read 𝜎(𝑖) and add to 𝐸 // 𝜎(𝑖) = (𝑤 , 𝑙 )5:search the irst hypothesis 𝐻 consistent with 𝐸
through a normal enumeration
6:
output 𝐻 // M (𝑖) = 𝐻
7:𝑖←𝑖+1
8: until forever

normal enumeration

1967). Here we introduce a convenient enumeration of hypotheses. An in inite sequence of hypotheses 𝐻 , 𝐻 , … is called a normal enumeration if { 𝐻 ∣ 𝑖 ∈ ℕ } = ℋ
and, for all 𝑖, 𝑗 ∈ ℕ, 𝑖 < 𝑗 implies
max |𝑣| ≤ max |𝑤|.
∈∈

We can easily implement a procedure that enumerates ℋ through a normal enumeration.
Theorem 2.7
The set of igures 𝜅(ℋ) = {𝜅(𝐻) ∣ 𝐻 ∈ ℋ} is F E -I

-learnable.

Proof. This learning can be done by the well-known strategy of identi ication by
enumeration. We show a pseudo-code of a learner M that F E -I -learns 𝜅(ℋ)
in Procedure 2.2. The learner M generates hypotheses through normal enumeration. If M outputs a wrong hypothesis 𝐻, there must exist a positive or negative
example that is not consistent with the hypothesis since, for a target igure 𝐾∗ ,Pos(𝐾∗ ) ⊖ Pos(𝜅(𝐻)) ≠ ∅for every hypothesis 𝐻 with 𝜅(𝐻) ≠ 𝐾∗ , where 𝑋 ⊖ 𝑌 denotes the symmetric difference, i.e., 𝑋 ⊖ 𝑌 = (𝑋 ∪ 𝑌) ⧵ (𝑋 ∩ 𝑌). Thus the learner M changes the wrong
hypothesis and reaches to a correct hypothesis 𝐻∗ such that 𝜅(𝐻∗ ) = 𝐾∗ in inite
time. If M produces a correct hypothesis, it never changes it, since every example
is consistent with it. Therefore M F E -I -learns 𝜅(ℋ).
Next, we consider F E -T -learning. In learning of languages from texts, thenecessary and suf icient conditions for learning have been studied in detail (Angluin, 1980, 1982; Kobayashi, 1996; Lange et al., 2008; Motoki et al., 1991; Wright,
1989), and characterization of learnability using inite tell-tale sets is one of the
crucial results. We interpret these results into the learning of igures and show
the F E -T -learnability.

2.3 EXACT LEARNING OF FIGURES
De inition 2.8: Finite tell-tale set (cf. Angluin, 1980)
Let ℱ be a set of igures. For a igure 𝐾 ∈ ℱ, a inite subset 𝒯 of the set of
positive examples Pos(𝐾) is a inite tell-tale set of 𝐾 with respect to ℱ if for all ﬁnite tell-tale set
igures 𝐿 ∈ ℱ, 𝒯 ⊂ Pos(𝐿) implies Pos(𝐿) ⊄ Pos(𝐾) (i.e., 𝐿 ⊄ 𝐾). If every 𝐾 ∈ ℱ
has inite tell-tale sets with respect to ℱ, we say that ℱ has a inite tell-tale set.
Theorem 2.9
Let ℱ be a subset of 𝜅(ℋ). Then ℱ is F E -T -learnable if and only if there isa procedure that, for every igure 𝐾 ∈ ℱ, enumerates a inite tell-tale set 𝑊 of 𝐾
with respect to ℱ.
This theorem can be proved in exactly the same way as that for learning of languages (Angluin, 1980). Note that such procedure does not need to stop. Using
this theorem, we show that the set 𝜅(ℋ) is not F E -T -learnable.
Theorem 2.10
The set 𝜅(ℋ) does not have a inite tell-tale set.
Proof. Fix a igure 𝐾 = 𝜅(𝐻) ∈ 𝜅(ℋ) such that #𝐻 ≥ 2 and ix a inite set 𝑇 ={ 𝑤 , 𝑤 , … , 𝑤 } contained in Pos(𝐾). For each inite sequence 𝑤 , there exists𝑢 ∈ Pos(𝐾) such that 𝑤 ⊏ 𝑢 with 𝑤 ≠ 𝑢 . For the igure 𝐿 = 𝜅(𝑈) with
𝑈 = {𝑢 , … , 𝑢 }, 𝑇 ⊂ Pos(𝐿) and Pos(𝐿) ⊂ Pos(𝐾) hold. Therefore 𝐾 has no inite
tell-tale set with respect to 𝜅(ℋ).
Corollary 2.11
The set of igures 𝜅(ℋ) is not F E -T

-learnable.

In any realistic situation of machine learning, however, this set 𝜅(ℋ) is too large
to search for the best hypothesis since we usually want to obtain a “compact” representation of a target igure. Thus we (implicitly) have an upper bound on thenumber of elements in a hypothesis. Here we give a fruitful result for the above
situation, that is, if we ix the number of elements #𝐻 in each hypothesis 𝐻 a priori, the resulting set of igures becomes F E -T -learnable. Intuitively, this isbecause if we take 𝑘 large enough, the set {𝑤 ∈ Pos(𝐾) ∣ |𝑤| ≤ 𝑘} becomes a inite
tell-tale set of 𝐾. For a inite subset of natural numbers 𝑁 ⊂ ℕ, we denote the setof hypotheses {𝐻 ∈ ℋ ∣ #𝐻 ∈ 𝑁} by ℋ .
Theorem 2.12
There exists a procedure that, for all inite subsets 𝑁 ⊂ ℕ and all igures 𝐾 ∈𝜅(ℋ ), enumerates a inite tell-tale set of 𝐾 with respect to 𝜅(ℋ ).
Proof. First, we assume that 𝑁 = {1}. It is trivial that there exists a procedure
that, for an arbitrary igure 𝐾 ∈ 𝜅(ℋ ), enumerates a inite tell-tale set of 𝐾 withrespect to 𝜅(ℋ ), since we always have 𝐿 ⊄ 𝐾 for all pairs of igures 𝐾, 𝐿 ∈ 𝜅(ℋ ).
Next, ix a inite set 𝑁 ⊂ ℕ with 𝑁 ≠ {1}. Let us consider the procedure thatenumerates elements of the sets
Pos (𝐾), Pos (𝐾), Pos (𝐾), … .
We show that this procedure enumerates a inite tell-tale set of 𝐾 with respect to𝜅(ℋ ). Notice that the number of elements #Pos (𝐾) monotonically increases
when 𝑘 increases whenever 𝐾 ∉ 𝜅(ℋ{ } ).

23

24

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
For each level 𝑘 and for a igure 𝐿 ∈ 𝜅(ℋ),
𝐿 ⊂ 𝐾 and Pos(𝐿) ⊇Pos (𝐾)∈{ , ,…, }impliesPos (𝐾).

Pos(𝐿) =(2.9)∈{ , ,…, }Here we de ine the setℒ = { 𝐿 ∈ 𝜅(ℋ ) 𝐿 ⊂ 𝐾 and 𝐿 satis ies the condition (2.9) }for each level 𝑘 ∈ ℕ. Then we can easily check that the minimum size of hypothesis
min ( )∈ℒ #𝐻 monotonically increases as 𝑘 increases. This means that there exists a natural number 𝑛 such that ℒ = ∅ for every 𝑘 ≥ 𝑛, since for each hypothesis
𝐻 ∈ ℋ we must have #𝐻 ∈ 𝑁. Therefore the set
Pos (𝐾)𝒯=∈{ , ,…, }is a inite tell-tale set of 𝐾 with respect to 𝜅(ℋ ).
Corollary 2.13
For all inite subsets of natural numbers 𝑁 ⊂ ℕ, the set of igures 𝜅(ℋ ) is
F E -T -learnable.

2.3.2 Consistent Learning
In a learning process, it is natural that every hypothesis generated by a learner isconsistent with the examples received by it so far. Here we introduce F C-I and F C-T -learning (CONS means CONSistent). These criteria correspondto C-learning that was irst introduced by Blum and Blum (1975)³. This model
was also used (but implicitly) in the Model Inference System (MIS) proposed by
Shapiro (1981, 1983), and studied in the computational learning of formal languages and recursive functions (Jain et al., 1999b).

FIGCONS-INF-learning
FIGCONS-TXT-learning

De inition 2.14: Consistent learning-T -learns) a set of igures ℱ ⊆A learner M F C-I -learns (resp. F C∗
𝒦 if M F E -I -learns (resp. F E -T -learns) ℱ and for all igures 𝐾 ∈ ℱ
and all informants (resp. texts) 𝜎 of 𝐾, each hypothesis M (𝑖) is consistent with
𝐸 that is the set of examples received by M until just before it generates the
hypothesis M (𝑖).
Assume that a learner M achieves F E -I -learning of 𝜅(ℋ) using Procedure
2.2. We can easily check that M always generates a hypothesis that is consistent
with the received examples.
³Consistency was also studied in the same form by Barzdin (1974) in Russian.

2.3 EXACT LEARNING OF FIGURESCorollary 2.15
F E -I = F C-I.

Suppose that ℱ ⊂ 𝜅(ℋ) is F E -T -learnable. We can construct a learner M inthe same way as in the case of EX-learning of languages from texts (Angluin, 1980),
where M always outputs a hypothesis that is consistent with received examples.
Corollary 2.16
F E -T = F C-T.

2.3.3 Reliable and Refutable Learning
In this subsection, we consider target igures that might not be represented exactly
by any hypothesis since there are in initely many such igures, and if we have nobackground knowledge, there is no guarantee of the existence of an exact hypothesis. Thus in practice this approach is more convenient than the explanatory or
consistent learning considered in the previous two subsections.
To realize the above case, we use two concepts, reliability and refutability. Reliable learning was introduced by Blum and Blum (1975) and Minicozzi (1976)
and refutable learning by Mukouchi and Arikawa (1995) and Sakurai (1991) incomputational learning of languages and recursive functions to introduce targets
which cannot be exactly represented by any hypotheses, and developed in literatures (Jain et al., 2001; Merkle and Stephan, 2003; Mukouchi and Sato, 2003). Here
we introduce these concepts into the learning of igures and analyze learnability.
First, we treat reliable learning of igures. Intuitively, reliability requires thatan in inite sequence of hypotheses only converges to a correct hypothesis.
De inition 2.17: Reliable learning
A learner M F R E -I -learns (resp. F R E -T -learns) a set of igures FIGRELEX-INF-learning∗ℱ ⊆ 𝒦 if M satis ies the following conditions:
FIGRELEX-TXT-learning
1. The learner M F E -I -learns (resp. F E -T -learns) ℱ.
∗
2. For any target igure 𝐾 ∈ 𝒦 and its informants (resp. texts) 𝜎, the in inite
sequence of hypotheses M does not converge to a wrong hypothesis 𝐻
such that GE(𝐾, 𝜅(𝐻)) ≠ 0.
We analyze reliable learning of igures from informants. Intuitively, if a learnercan judge whether or not the current hypothesis 𝐻 is consistent with the target
igure, i.e., 𝜅(𝐻) = 𝐾 or not in inite time, then the target igure is reliably learnable.
Theorem 2.18
F E -I = F R

E -I

.

Proof. Since the statement F R E -I
⊆ F E -I is trivial, we prove theopposite F E -I ⊆ F R E -I . Fix a set of igures ℱ ⊆ 𝜅(ℋ) with ℱ ∈F E -I , and suppose that a learner M F E -I -learns ℱ using Procedure 2.2.
The goal is to show that ℱ ∈ F R E -I . Assume that a target igure 𝐾 belongs∗
to 𝒦 ⧵ ℱ. Here we have the following property: for all igures 𝐿 ∈ ℱ, there must∗
exist a inite sequences 𝑤 ∈ (Σ ) such that𝑤 ∈ Pos(𝐾) ⊖ Pos(𝐿),

25

26

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
hence for any M’s current hypothesis 𝐻, M changes 𝐻 if it receives a positive or
negative example (𝑤, 𝑙) such that 𝑤 ∈ Pos(𝐾) ⊖ Pos(𝜅(𝐻)). This means that anin inite sequence of hypotheses does not converge to any hypothesis. Thus wehave ℱ ∈ F R E -I .
In contrast, we have an interesting result in the reliable learning from texts.
We show in the following that F E -T
≠ F R E -T holds and that a setof igures ℱ is reliably learnable from positive data only if any igure 𝐾 ∈ ℱ is asingleton. Remember that ℋ denotes the set of hypotheses {𝐻 ∈ ℋ ∣ #𝐻 ∈ 𝑁}
for a subset 𝑁 ⊂ ℕ and, for simplicity, we denote ℋ{ } by ℋ for 𝑛 ∈ ℕ.
Theorem 2.19
The set of igures 𝜅(ℋ ) is F R

E -T-learnable if and only if 𝑁 = {1}.

Proof. First we show that the set of igures 𝜅(ℋ ) is F R E -T -learnable.
From the self-similar sets property of hypotheses, we have the following: a ig∗
ure 𝐾 ∈ 𝜅(ℋ) is a singleton if and only if 𝐾 ∈ 𝜅(ℋ ). Let 𝐾 ∈ 𝒦 ⧵ 𝜅(ℋ ), andassume that a learner M F E -T -learns 𝜅(ℋ ). We can naturally suppose that
M changes the current hypothesis 𝐻 whenever it receives a positive example (𝑤, 1)
such that 𝑤 ∉ Pos(𝜅(𝐻)) without loss of generality. For any hypothesis 𝐻 ∈ ℋ ,∗there exists 𝑤 ∈ (Σ ) such that𝑤 ∈ Pos(𝐾) ⧵ Pos(𝜅(𝐻))since 𝐾 is not a singleton. Thus if the learner M receives such a positive example(𝑤, 1), it changes the hypothesis 𝐻. This means that an in inite sequence of hypotheses does not converge to any hypothesis. Therefore 𝜅(ℋ ) is F R E -T learnable.
Next, we prove that 𝜅(ℋ ) is not F R E -T -learnable for any 𝑛 > 1. Fix
such 𝑛 ∈ ℕ with 𝑛 > 1. We can easily check that, for a igure 𝐾 ∈ 𝜅(ℋ ) and any of∗
its inite tell-tale sets 𝒯 with respect to 𝜅(ℋ ), there exists a igure 𝐿 ∈ 𝒦 ⧵𝜅(ℋ )such that 𝐿 ⊂ 𝐾 and 𝒯 ⊂ Pos(𝐿). This means that
Pos(𝐿) ⊆ Pos(𝐾) and 𝒯 ⊆ Pos(𝐿)hold. Thus if a learner M F E -T -learns 𝜅(ℋ ), M for some presentation 𝜎of some such L must converge to some hypothesis in ℋ . Consequently, we have𝜅(ℋ ) ∉ F R E -T .
Corollary 2.20
F R E -T ⊂ F E -T.

Sakurai (1991) proved that a set of concepts 𝒞 is reliably EX-learnable from texts
if and only if 𝒞 contains no in inite concept (p. 182, Theorem 3.1)⁴. However, wehave shown that the set 𝜅(ℋ ) is F R E -T -learnable, though all igures 𝐾 ∈𝜅(ℋ ) correspond to in inite concepts since Pos(𝐾) is in inite for all 𝐾 ∈ 𝜅(ℋ ).
The monotonicity of the set Pos(𝐾) (Lemma 2.5), which is a constraint naturally
derived from the geometric property of examples, causes this difference.
Next, we extend F E -I - and F E -T -learning by paying our attention to
refutability. In refutable learning, a learner tries to learn igures in the limit, but it
⁴The literature (Sakurai, 1991) was written in Japanese. The same theorem was mentioned by Mukouchi and Arikawa (1995, p. 60, Theorem 3).

2.3 EXACT LEARNING OF FIGURES
understands that it cannot ind a correct hypothesis in inite time, that is, outputs
the refutation symbol △ and stops if the target igure is not in the considered space.
De inition 2.21: Refutable learning
A learner M F R E -I -learns (resp. F R E -T -learns) a set of igures FIGREFEX-INF-learning∗ℱ ⊆ 𝒦 if M satis ies the followings. Here, △ denotes the refutation symbol.
FIGREFEX-TXT-learning
1. The learner M F E -I -learns (resp. F E -T -learns) ℱ.
2. If 𝐾 ∈ ℱ, then for all informants (resp. texts) 𝜎 of 𝐾, M (𝑖) ≠ △ for all 𝑖 ∈ ℕ.
∗
3. If 𝐾 ∈ 𝒦 ⧵ℱ, then for all informants (resp. texts) 𝜎 of 𝐾, there exists 𝑚 ∈ ℕsuch that M (𝑖) ≠ △ for all 𝑖 < 𝑚, and M (𝑖) = △ for all 𝑖 ≥ 𝑚.
Conditions 2 and 3 in the above de inition mean that a learner M refutes the set ℱ∗
in inite time if and only if a target igure 𝐾 ∈ 𝒦 ⧵ ℱ. To characterize refutablelearning, we prepare the following lemma, which is a translation of Mukouchi andArikawa (1995, Lemma 4).
Lemma 2.22
Suppose a learner M F R E -I -learns (resp. F R E -T -learns) a set of∗igures ℱ, and let 𝐾 ∈ 𝒦 ⧵ℱ. For every informant (resp. text) 𝜎 of 𝐾, if M outputs△ after receiving 𝜎[𝑛], then for any 𝐿 ∈ ℱ, the set of examples range(𝜎[𝑛]) is notconsistent with 𝐿.
Here we compare F R

E -I

Theorem 2.23
F R E -I ⊈ F E -T

-learnability with other learning criteria.

and F E -T

⊈F R

E -I

.

Proof. First we consider F R E -I
⊈ F E -T . We show an example of aset of igures ℱ with ℱ ∈ F R E -I and ℱ ∉ F E -T in the case of 𝑑 = 2.
Let 𝐾 = 𝜅({⟨ 𝟶, 𝟶 ⟩, ⟨ 𝟷, 𝟷 ⟩}), 𝐾 = 𝜅({⟨ 𝑤, 𝑤 ⟩ ∣ 𝑤 ∈ Σ ⧵ {𝟷 }}) for every 𝑖 ≥ 1, andℱ = {𝐾 ∣ 𝑖 ∈ ℕ}. Note that 𝐾 is the line 𝑦 = 𝑥 and 𝐾 ⊂ 𝐾 for all 𝑖 ≥ 1.
We prove that ℱ ∈ F R E -I . If a target igure 𝐾 ⊃ 𝐾 , it is trivial that,for any 𝐾’s informant 𝜎, the set of examples range(𝜎[𝑛]) for some 𝑛 ∈ ℕ is notconsistent with any 𝐾 ∈ ℱ (consider a positive example for a point 𝑥 ∈ 𝐾 ⧵ 𝐾 ).
Otherwise if 𝐾 ⊂ 𝐾 , there should exist a negative example ⟨ 𝑣, 𝑣 ⟩ ∈ Neg(𝐾). Then
we have 𝐾 ≠ 𝐾 for all 𝑖 > |𝑣|. Thus a leaner can refute candidates {𝐾 , 𝐾 , … , 𝐾| | }in inite time. Therefore from Lemma 2.22, ℱ ∈ F R E -I holds.
Next we show that ℱ ∉ F E -T . Let 𝐾 be the target igure. For any inite setof positive examples 𝒯 ⊂ Pos(𝐾 ), there exists a igure 𝐾 ∈ ℱ such that 𝐾 ⊂ 𝐾
and 𝒯 is consistent with 𝐾 . Therefore it has no inite tell-tale set with respect toℱ and hence ℱ ∉ F E -T from Theorem 2.9.
Second we check F E -T ⊈ F R E -I . Assume that ℱ = 𝜅(ℋ{ } ) anda target igure 𝐾 is a singleton {𝑥} with 𝐾 ∉ ℱ. It is clear that, for any 𝐾’s informant 𝜎 and 𝑛 ∈ ℕ, range(𝜎[𝑛]) is consistent with some igure 𝐿 ∈ ℱ. Thusℱ ∉ F R E -I whereas ℱ ∈ F E -T .
Theorem 2.24
F R E -T ⊈ F R

E -I

and F R

E -I

⊈F R

E -T

.

Proof. It is trivial that F R E -T
⊈ F R E -I since we have 𝜅(ℋ{ } ) ∉F R E -I in the above proof and 𝜅(ℋ{ } ) ∈ F R E -T from Theorem 2.19.

27

28

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
Next we consider F R E -I ⊈ F R E -T . We show a counterexample in the case of 𝑑 = 2. Let ℱ = {𝐾} with 𝐾 = 𝜅({⟨ 𝟶, 𝟶 ⟩, ⟨ 𝟷, 𝟷 ⟩}) and M be a
F R E -T -learner of ℱ. Notice that the igure 𝐾 is the line 𝑦 = 𝑥. For simplicity, in the following we assume that every text 𝜎 of a igure 𝐾 is an in inite sequenceof inite sequences 𝑤 , 𝑤 , 𝑤 , … such that {𝑤 } ∈ℕ = Pos(𝐾).
For a dense subset 𝑥 , 𝑥 , … of the igure 𝐾, de ine 𝑋 = {𝑥 , 𝑥 , … , 𝑥 }. Then,for each 𝑛 ∈ ℕ, there exists a pre ix 𝜎 ( ) of a text for 𝑋 ( ) with some monotonically increasing function 𝑟 ∶ ℕ → ℕ satisfying the following three conditions:
1. The learner M makes a mind change (changes the current hypothesis) when
it receives 𝜎 ( ) ,2. range(𝜎 ( ) ) ⊇ {𝑤 ∈ Pos(𝐾) ∣ |𝑤| ≤ 𝑛}, and3. 𝜎 ( ) ⊑ 𝜎 ( ) (with 𝑛 ≥ 1).
If 𝑛 goes to in inite, 𝜎 ( ) becomes a text for 𝐾, but M’s output does not converge
to any hypothesis. Thus ℱ ∉ F R E -T .
On the other hand, we can easily check that ℱ ∈ F R E -I .
These results mean that both F R E -I - and F R E -T -learning are
dif icult, but they are incomparable in terms of learnability. Furthermore, we havethe following hierarchy.
Theorem 2.25
F R E -T ≠ ∅ and F R

E -T

⊂F R

E -I

.

Proof. Let a set of igures ℱ be a singleton {𝐾} such that 𝐾 = 𝜅(𝑤) for some 𝑤 ∈∗(Σ ) . Then there exists a learner M that F R E -T -learns ℱ, since all M has
to do is to check whether or not, for a given positive example (𝑣, 1), 𝑣 ⊑ 𝑢 for some𝑢 ∈ Pos(𝐾) = {𝑥 ∣ 𝑥 ⊑ 𝑤𝑤𝑤 … }.
Next, let ℱ = {𝐾} such that 𝐾 = 𝜅(𝐻) with #𝐻 ≥ 2. We can easily check thatℱ ∉ F R E -T because if a target igure 𝐿 is a proper subset of 𝐾, no learner
can refute ℱ in inite time. Conversely, ℱ ∈ F R E -I since for all 𝐿 with 𝐿 ≠ 𝐾,
there exists an example with which the hypothesis 𝐻 is not consistent.
Corollary 2.26
F R E -T ⊂ F R

E -T

.

2.4 Eﬀective Learning of Figures

generalization error
eﬀectivity

In learning under the proposed criteria, i.e., explanatory, consistent, reliable, andrefutable learning, each hypothesis is just considered as exactly “correct” or not,that is, for a target igure 𝐾 and for a hypothesis 𝐻, 𝐻 is correct if GE(𝐾, 𝐻) = 0
and is not correct if GE(𝐾, 𝐻) ≠ 0. Thus we cannot know the rate of convergenceto the target igure and how far it is from the recent hypothesis to the target. It
is therefore more useful if we consider approximate hypotheses by taking various
generalization errors into account in the learning process.
We de ine novel learning criteria, F E E -I - and F E E -T -learning (EFmeans EFfective), to introduce into learning the concept of effectivity, which has
been analyzed in computation of real numbers in the area of computable analysis(Weihrauch, 2000). Intuitively, these criteria guarantee that for any target igure,a generalization error becomes smaller and smaller monotonically and converges

2.4 EFFECTIVE LEARNING OF FIGURES
to zero. Thus we can know when the learner learns the target igure “well enough”.
Furthermore, if a target igure is learnable in the limit, then the generalization error goes to zero in inite time.
De inition 2.27: Effective learning
A learner M F E E -I -learns (resp. F E E -T∗
𝒦 if M satis ies the following conditions:-learns) a set of igures ℱ ⊆ FIGEFEX-INF-learning
FIGEFEX-TXT-learning

1. The learner M F E -I -learns (resp. F E -T -learns) ℱ.
∗
2. For an arbitrary target igure 𝐾 ∈ 𝒦 and all informants (resp. texts) 𝜎 of𝐾, for all 𝑖 ∈ ℕ,
GE(𝐾, M (𝑖)) ≤ 2 .
This de inition is inspired by the Cauchy representation (Weihrauch, 2000, De inition 4.1.5) of real numbers.
Effective learning is related to monotonic learning (Lange and Zeugmann, 1993,
1994; Kinber, 1994; Zeugmann et al., 1995) originally introduced by Jantke (1991)
and Wiehagen (1991), since both learning models consider monotonic convergence of hypotheses. In contrast to their approach, where various monotonicity
over languages was considered, we geometrically measure the generalization error of a hypothesis by the Hausdorff metric. On the other hand, the effective learning is different from B -learning developed in the learning of languages and recursive functions (Jain et al., 1999b) since B -learning only guarantees that generalization errors go to zero in inite time. This means that B -learning is not effective.
First we show that we can bound the generalization error of the hypothesis 𝐻
using the diameter diam(𝑘) of the set 𝜌(𝑤) with |𝑤| = 𝑘. Recall that we havediam(𝑘) = √𝑑 ⋅ 2(see proof of Lemma 2.3). In the following, we denote by 𝐸 the set of examples{(𝑤, 𝑙) ∣ |𝑤| = 𝑘} in 𝜎 and call each example in it a level-𝑘 example.
level-k example
Lemma 2.28
Let 𝜎 be an informant of a igure 𝐾 and 𝐻 be a hypothesis that is consistent withthe set of examples 𝐸 = {(𝑤, 𝑙) ∣ |𝑤| = 𝑘}. We have the inequality
GE(𝐾, 𝐻) ≤ diam(𝑘).
Proof. Since 𝐻 is consistent with 𝐸 ,𝜅(𝐻) ∩ 𝜌(𝑤)≠ ∅ if (𝑤, 1) ∈ 𝐸 ,= ∅ if (𝑤, 0) ∈ 𝐸 .

For 𝛿 = diam(𝑘), the 𝛿-neighborhood of 𝜅(𝐻) contains 𝐾 and the 𝛿-neighborhoodof 𝐾 contains 𝜅(𝐻). Thus it follows that GE(𝐾, 𝐻) = 𝑑H (𝐾, 𝜅(𝐻)) ≤ diam(𝑘).
Theorem 2.29
The set of igures 𝜅(ℋ) is F E E -I

-learnable.

29

30

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
Proof. We show the pseudo-code of M that F E E -I
2.3. We use the function-learns 𝜅(ℋ) in Procedure𝑔(𝑘) = ⌈ 𝑘 + log √𝑑 ⌉.
Then for all 𝑘 ∈ ℕ, we havediam(𝑔(𝑘)) = √𝑑 ⋅ 2

( )

≤2

.

The learner M stores examples, and when it receives all examples at the level 𝑔(𝑘),it outputs a hypothesis. Every 𝑘th hypothesis M (𝑘) is consistent with the set of( )examples 𝐸. Thus we haveGE(𝐾, M (𝑘)) ≤ diam(𝑔(𝑘)) ≤ 2
for all 𝑘 ∈ ℕ from Lemma 2.28.
Assume that 𝐾 ∈ 𝜅(ℋ). If M outputs a wrong hypothesis, there must be a positive or negative example that is not consistent with the hypothesis, and it changes
the wrong hypothesis. If it produces a correct hypothesis, then it never changes thecorrect hypothesis, since every example is consistent with the hypothesis. Thus
there exists 𝑛 ∈ ℕ with GE(𝐾, M (𝑖)) = 0 for all 𝑖 ≥ 𝑛. Therefore, M F E E -I learns 𝜅(ℋ).
Corollary 2.30
F E E -I = F R

E -I= F E -I.
∗

The learner with Procedure 2.3 can treat the set of all igures 𝒦 as learning tar∗gets, since for any igure 𝐾 ∈ 𝒦 , it can approximate the igure arbitrarily closelyusing only the igures represented by hypotheses in the hypothesis space ℋ.
In contrast to F E -T -learning, there is no set of igures that is F E E -T learnable.
Theorem 2.31
F E E -T = ∅.
Proof. We show a counterexample of a target igure which no learner M can approximate effectively. Assume that 𝑑 = 2 and a learner M F E E -T -learns∗
a set of igures ℱ ⊆ 𝒦 . Let us consider two target igures 𝐾 = {(0, 0), (1, 1)}∗
and 𝐿 = {(0, 0)}. For 𝐿’s text 𝜎, for all examples (𝑤, 1) ∈ range(𝜎), 𝑤 ∈ {𝟶𝟶} .
Since M F E E -T -learns ℱ, it should output the hypothesis 𝐻 as M (2) suchthat GE(𝐿, 𝐻) < 1/4. Suppose that M receives 𝑛 examples before outputting the
hypothesis 𝐻. Then there exists a presentation 𝜏 of the igure 𝐾 such that 𝜏[𝑛 −1] = 𝜎[𝑛 − 1], and M outputs the hypothesis 𝐻 with receiving 𝜏[𝑛 − 1]. However,GE(𝐾, 𝐻) ≥ √2−1/4 holds from the triangle inequality, contradicting our assump∗tion that M F E E -T -learns ℱ. This proof can be applied for any ℱ ⊆ 𝒦 ,thereby we have F E E -T = ∅.
Since F R

E -T

F E E -T≠ ∅, we have the relation

⊂F R

E -T

.

This result means that we cannot learn any igures “effectively” by using only positive examples.

2.5 EVALUATION OF LEARNING USING DIMENSIONS

Procedure 2.3: Learning procedure that F E E -I

-learns 𝜅(ℋ)

Input: Informant 𝜎 = (𝑤 , 𝑙 ), (𝑤 , 𝑙 ), … of igure 𝐾 ∈ 𝜅(ℋ)
Output: In inite sequence of hypotheses M (0), M (1), …
1: 𝑖 ← 0
2: 𝑘 ← 0
3: 𝐸 ← ∅ // 𝐸 is a set of received examples
4: repeat5:
read 𝜎(𝑖) and add to 𝐸 // 𝜎(𝑖) = (𝑤 , 𝑙 )
( )
6:
if 𝐸
⊆ 𝐸 then( )// 𝐸= {(𝑤, 𝑙) ∈ range(𝜎) ∣ |𝑤| = 𝑔(𝑘)}// 𝑔(𝑘) = ⌈ 𝑘 + log √𝑑 ⌉7:
search the irst 𝐻 that is consistent with 𝐸
through a normal enumeration
8:
output 𝐻 // M (𝑖) = 𝐻
9:
𝑘←𝑘+1
10:
end if11:𝑖←𝑖+1
12: until forever

2.5

Evaluation of Learning Using Dimensions

Here we show a novel mathematical connection between fractal geometry andGold-style learning under the proposed learning model described in Section 2.2.
More precisely, we measure the number of positive examples, one of the complexities of learning, using the Hausdorff dimension and the VC dimension. The
Hausdorff dimension is known as the central concept of fractal geometry, whichmeasures the density of igures, and VC dimension is the central concept of the
Valiant-style learning model (PAC learning model) (Kearns and Vazirani, 1994),
which measures the complexity of classes of hypotheses.

2.5.1 Preliminaries for Dimensions
First we introduce the Hausdorff and related dimensions: the box-counting dimension, the similarity dimension, and the VC dimension.
For 𝑋 ⊆ ℝ and 𝑠 ∈ ℝ with 𝑠 > 0, de ine|𝑈|ℌ (𝑋) ≔ inf

𝒰 is a 𝛿-cover of 𝑋.

∈𝒰

The 𝑠-dimensional Hausdorff measure of 𝑋 is lim → ℌ (𝑋), denoted by ℌ (𝑋). We s-dimensional Hausdorﬀ measure
say that 𝒰 is a 𝛿-cover of 𝑋 if 𝒰 is countable, 𝑋 ⊆ ⋃ ∈ 𝒰 𝑈, and |𝑈| ≤ 𝛿 for all
𝑈 ∈ 𝒰. When we ix a set 𝑋 and view ℌ (𝑋) as a function with respect to 𝑠, it has at
most one value where the value ℌ (𝑋) changes from ∞ to 0 (Federer, 1996). Thisvalue is called the Hausdorff dimension of 𝑋. Formally, the Hausdorff dimension of Hausdorﬀ dimension

31

32

LEARNING FIGURES AS COMPUTABLE CLASSIFICATIONa set 𝑋, written as dimH 𝑋, is de ined by
dimH 𝑋 ≔ sup 𝑠 ℌ (𝑋) = ∞ = inf 𝑠 ≥ 0 ℌ (𝑋) = 0 .

box-counting dimension

The box-counting dimension, also known as the Minkowski-Bouligand dimension, is one of widely used dimensions since its mathematical calculation and empirical estimation are relatively easy compared to the Hausdorff dimension. Moreover, if we try to calculate the box-counting dimension, which is given as the limit of
the following equation (2.10) by decreasing 𝛿, the obtained values often converge
to the Hausdorff dimension at the same time. Thus we can obtain an approximate
value of the Hausdorff dimension by an empirical method. Let 𝑋 be a nonempty
bounded subset of ℝ and 𝑁 (𝑋) be the smallest cardinality of a 𝛿-cover of 𝑋. The
box-counting dimension dimB 𝑋 of 𝑋 is de ined bydimB 𝑋 ≔ lim→

log 𝑁 (𝑋)− log 𝛿(2.10)

if this limit exists. We have
dimH 𝑋 ≤ dimB 𝑋

similarity dimension

for all 𝑋 ⊆ ℝ .
It is usually dif icult to ind the Hausdorff dimension of a given set. However,
we can obtain the dimension of a certain class of self-similar sets in the following
manner. Let 𝐶 be a inite set of contractions, and 𝐹 be the self-similar set of 𝐶. The
similarity dimension of 𝐹, denoted by dimS 𝐹, is de ined by the equation
dimS

𝐿(CT)

= 1,

∈

contractivity factor

where 𝐿(CT) is the contractivity factor of CT, which is de ined by the in imum of
all real numbers 𝑐 with 0 < 𝑐 < 1 such that 𝑑(CT(𝑥), CT(𝑦)) ≤ 𝑐𝑑(𝑥, 𝑦) for all
𝑥, 𝑦 ∈ 𝑋. We have
dimH 𝐹 ≤ dimS 𝐹
and if 𝐶 satis ies the open set condition,
dimH 𝐹 = dimB 𝐹 = dimS 𝐹

open set condition(Falconer, 2003). Here, a inite set of contractions 𝐶 satis ies the open set condition
if there exists a nonempty bounded open set 𝑂 ⊂ ℝ such that CT(𝑂) ⊂ 𝑂 for all
CT ∈ 𝐶 and CT(𝑂) ∩ CT (𝑂) = ∅ for all CT, CT ∈ 𝐶 with CT ≠ CT .
Intuitively, the VC dimension (Vapnik and Chervonenkis, 1971; Valiant, 1984)
is a parameter of separability, which gives lower and upper bounds for the sample
size in the Valiant-style, or PAC, learning model (Kearns and Vazirani, 1994). For∗
all ℛ ⊆ ℋ and 𝑊 ⊆ Σ , de ineΠℛ (𝑊) ≔ { Pos(𝜅(𝐻)) ∩ 𝑊 𝐻 ∈ ℛ } .

VC dimension

If Πℛ (𝑊) = 2 , we say that 𝑊 is shattered by ℛ. Here the VC dimension of ℛ,
denoted by dimVC ℛ, is the cardinality of the largest set 𝑊 shattered by ℛ.

2.5 EVALUATION OF LEARNING USING DIMENSIONS

2.5.2 Measuring the Complexity of Learning with Dimensions
We show that the Hausdorff dimension of a target igure gives a lower bound to thenumber of positive examples. Remember that Pos (𝐾) = {𝑤 ∈ Pos(𝐾) ∣ |𝑤| = 𝑘}and the diameter diam(𝑘) of the set 𝜌(𝑤) with |𝑤| = 𝑘 is √𝑑2 . Moreover, for all∗𝑘 ∈ ℕ, #{𝑤 ∈ (Σ ) ∣ |𝑤| = 𝑘} = 2 .
Theorem 2.32∗
For every igure 𝐾 ∈ 𝒦 and for any 𝑠 < dimH 𝐾, if we take 𝑘 large enough,#Pos (𝐾) ≥ 2 .
Proof. Fix 𝑠 < dimH 𝐾. From the de inition of the Hausdorff measure,ℌdiam( ) (𝐾) ≤ #Pos (𝐾) ⋅ (√𝑑2
since diam(𝑘) = √𝑑2). If we take 𝑘 large enough,ℌdiam( ) (𝐾) ≥ (√𝑑)because ℌ (𝐾) is monotonically increasing with decreasing 𝛿, and goes to ∞. Thus)

#Pos (𝐾) ≥ ℌdiam( ) (𝐾)(√𝑑2≥ (√𝑑) (√𝑑2)

=2

holds.
Moreover, if a target igure 𝐾 can be represented by some hypothesis, that is,𝐾 ∈ 𝜅(ℋ), we can use the exact dimension dimH 𝐾 as a bound for the number#Pos (𝐾). We use the mass distribution principle (Falconer, 2003, Mass distribu- mass distribution principle
tion principle 4.2): Let 𝜇 be a mass distribution, which is a measure on a bounded
subset of ℝ satisfying 0 < 𝜇(ℝ ) < ∞, over 𝐾. Assume that for some 𝑠 there
exists 𝑐 > 0 and 𝜀 > 0 such that 𝜇(𝑈) ≤ 𝑐|𝑈| holds for all sets 𝑈 with |𝑈| < 𝜀.
Then we have ℌ (𝐾) ≥ 𝜇(𝐾)/𝑐.
Theorem 2.33
For every igure 𝐾 ∈ 𝜅(ℋ) with 𝐾 = 𝜅(𝐻), if we take 𝑘 large enough,#Pos (𝐾) ≥ 2

dimH

.

Proof. From the de inition of the Hausdorff measure,dim

Hℌdiam((𝐾) ≤ #Pos (𝐾) ⋅ (√𝑑2
)

)

dimH

.

Let 𝜇 be a mass distribution such that 𝜇(𝜌(𝑤)) = 2
that𝑐 = (√𝑑)dimH

for each 𝑤 ∈ (Σ ) . Assume.

We have𝜇(𝜌(𝑤)) = 2≤ 𝑐 ⋅ diam(𝑘)

dimH

= √𝑑
=2

dimH
dimH

⋅ √𝑑2
.

dimH

33

34

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
1

Figure 2.4 | Positive and negative examples for the Sierpiński triangle at
level 1 and 2. White (resp. gray)squares mean positive (resp. negative) examples.

0

1

0

0

1

0

1

Thus from the mass distribution principle,
ℌ

dimH

dimH

(𝐾) ≥ 1/𝑐 = (√𝑑).

Therefore if we take 𝑘 large enough,dim

H#Pos (𝐾) ≥ ℌdiam((𝐾)(√𝑑2))

dimH≥ (√𝑑)
=2

dimH

(√𝑑2

)

dimH

dimH

holds.
Example 2.34
Let us consider the igure 𝐾 in Example 2.1. It is known that dimH 𝐾 = log 3/ log 2 =1.584 … . From Theorem 2.33,
Pos (𝐾) ≥ 2

dimH

=3

holds at level 1 andPos (𝐾) ≥ 4

dimH

=9

holds at level 2. Actually, Pos (𝐾) = 4 and Pos (𝐾) = 13. Note that 𝐾 is alreadycovered by 3 and 9 intervals at level 1 and 2, respectively (Figure 2.4).
The VC dimension can also be used to characterize the number of positive examples. De ineℋ ≔ { 𝐻 ∈ ℋ | |𝑤| = 𝑘 for all 𝑤 ∈ 𝐻 } ,and call each hypothesis in the set a level-𝑘 hypothesis. We show that the VC dimen∗sion of the set of level 𝑘 hypotheses ℋ is equal to #{𝑤 ∈ (Σ ) ∣ |𝑤| = 𝑘} = 2 .
Lemma 2.35
At each level 𝑘, we have dimVC ℋ = 2 .
Proof. First of all,dimVC ℋ ≤ 2
is trivial since #ℋ = 2 . Let ℋ denote the set {𝐻 ∈ ℋ ∣ #𝐻 = 𝑛}. For all
𝐻 ∈ ℋ , there exists 𝑤 ∈ Pos(𝜅(𝐻)) such that 𝑤 ∉ Pos(𝜅(𝐺)) for all 𝐺 ∈ ℋ
with 𝐻 ≠ 𝐺. Thus if we assume ℋ = {𝐻 , … , 𝐻 }, there exists the set of inite2.5 EVALUATION OF LEARNING USING DIMENSIONS
sequences 𝑊 = {𝑤 , … , 𝑤 } such that for all 𝑖 ∈ {1, … , 2 }, 𝑤 ∈ Pos(𝜅(𝐻 ))∗and 𝑤 ∉ Pos(𝜅(𝐻 )) for all 𝑗 ∈ {1, … , 2 } with 𝑖 ≠ 𝑗. For every pair 𝑉, 𝑊 ⊂ (Σ ) ,𝑉 ⊂ 𝑊 implies 𝜅(𝑉) ⊂ 𝜅(𝑊). Therefore the set 𝑊 is shattered by ℋ , meaningthat we have dimVC ℋ = 2 .
Therefore we can rewrite Theorems 2.32 and 2.33 as follows.
Theorem 2.36∗
For every igure 𝐾 ∈ 𝒦 and for any 𝑠 < dimH 𝐾, if we take 𝑘 large enough,#Pos (𝐾) ≥ (dimVC ℋ )/.

Moreover, if 𝐾 ∈ 𝜅(ℋ) with 𝐾 = 𝜅(𝑊), if we take 𝑘 large enough,#Pos (𝐾) ≥ (dimVC ℋ )dimH /.

These results demonstrate a relationship among the complexities of learning igures (numbers of positive examples), classes of hypotheses (VC dimension), andtarget igures (Hausdorff dimension).

2.5.3 Learning the Box-Counting Dimension Effectively
One may think that F E E -I -learning can be achieved without the proposed
hypothesis space, since if a learner just outputs igures represented by a set of received positive examples, the generalization error becomes smaller and smaller.
Here we show that one “quality” of a target igure, the box-counting dimension,is also learned in F E E -I -learning, whereas if a learner outputs igures represented by a set of received positive examples, the box-counting dimension (andalso the Hausdorff dimension) of any igure is always 𝑑.
Recall that for all hypotheses 𝐻 ∈ ℋ, dimH 𝜅(𝐻) = dimB 𝜅(𝐻) = dimS 𝜅(𝐻),since the set of contractions encoded by 𝐻 meets the open set condition.
Theorem 2.37
Assume that a learner M F E E -I∗

-learns 𝜅(ℋ). For all target igures 𝐾 ∈ 𝒦 ,lim dimB 𝜅(M (𝑘)) = dimB 𝐾.
→

Proof. First, we assume that a target igure 𝐾 ∈ 𝜅(ℋ). For every informant 𝜎 of 𝐾,
M converges to a hypothesis 𝐻 with 𝜅(𝐻) = 𝐾. Thuslim dimB 𝜅(M (𝑘)) = dimB 𝐾 = dimH 𝐾.
→∗

Next, we assume 𝐾 ∈ 𝒦 ⧵ 𝜅(ℋ). Since GE(𝐾, M (𝑖)) ≤ 2 holds for every𝑖 ∈ ℕ, for each 𝑘 ∈ ℕ, we have some 𝑖 ≥ 𝑘 such that the hypothesis M (𝑖) is
consistent with the set of level-𝑘 examples 𝐸 = {(𝑤, 𝑙) ∈ range(𝜎) ∣ |𝑤| = 𝑘}.
ThusdimS 𝜅(M (𝑖)) = dimB 𝜅(M (𝑖)) =log #Pos (𝐾)− log 2

.

35

36

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION
Falconer (2003) shows that the box-counting dimension dimB 𝐾 is de ined equivalently bydimB 𝐾 = lim→

log #Pos (𝐾)− log 2,where 2 is the length of one side of an interval 𝜌(𝑤) with |𝑤| = 𝑘. Therefore
from the de inition of the box-counting dimension,
lim dimH 𝜅(M (𝑖)) = lim→→

log #Pos (𝐾)− log 2= dimB 𝐾

holds.

2.6 Computational Interpretation of Learning
Recently, the notion of “computability” for continuous objects has been introduced(Schröder, 2002a; Weihrauch, 2000, 2008; Weihrauch and Grubba, 2009; Tavanaand Weihrauch, 2011) in the framework of Type-2 Theory of Effectivity (TTE),
where we treat an uncountable set 𝑋 as objects for computing through in inite
sequences over a given alphabet Σ. Using the framework, we analyze our learning model from the computational point of view. Some studies by de Brecht andYamamoto (2009); de Brecht (2010) have already demonstrated a close connection between TTE and the Gold-style learning model, and our analysis becomes an
instance and extension of their analysis.

2.6.1 Preliminaries for Type-2 Theory of Effectivity

representation

Type-2 machine

computability

equivalentFirst, we prepare mathematical notations for TTE. In the following in this section,
we assume Σ = {𝟶, 𝟷, [, ], ‖}. A partial (resp. total) function 𝑔 from a set 𝐴 to a set
𝐵 is denoted by 𝑔 ∶⊆ 𝐴 → 𝐵 (resp. 𝑔 ∶ 𝐴 → 𝐵). A representation of a set 𝑋 is a∗surjection 𝜉 ∶⊆ 𝐶 → 𝑋, where 𝐶 is Σ or Σ . We see 𝑝 ∈ dom(𝜉) as a name of theencoded element 𝜉(𝑝).
∗Computability of string functions 𝑓 ∶⊆ 𝑋 → 𝑌, where 𝑋 and 𝑌 are Σ or Σ , isde ined via a Type-2 machine, which is a usual Turing machine with one-way input
tapes, some work tapes, and a one-way output tape (Weihrauch, 2000). The function 𝑓 ∶⊆ 𝑋 → 𝑌 computed by a Type-2 machine 𝑀 is de ined as follows: When∗𝑌 is Σ , 𝑓 (𝑝) ≔ 𝑞 if 𝑀 with input 𝑝 halts with 𝑞 on the output tape, and when
𝑌 is Σ , 𝑓 (𝑝) ≔ 𝑞 if 𝑀 with input 𝑝 writes step by step 𝑞 onto the output tape.
We say that a function 𝑓 ∶⊆ 𝐶 → 𝐷 is computable if there is a Type-2 machine thatcomputes 𝑓, and a inite or in inite sequence 𝑝 is computable if the constant function 𝑓 which outputs 𝑝 is computable. A Type-2 machine never changes symbols
that have already been written onto the output tape, thus each pre ix of the output
depends only on a pre ix of the input.
By treating a Type-2 machine as a translator between names of some objects,a hierarchy of representations is introduced. A representation 𝜉 is reducible to 𝜁,
denoted by 𝜉 ≤ 𝜁, if there exists a computable function 𝑓 such that 𝜉(𝑝) = 𝜁(𝑓(𝑝))for all 𝑝 ∈ dom(𝜉). Two representations 𝜉 and 𝜁 are equivalent, denoted by 𝜉 ≡ 𝜁,
if both 𝜉 ≤ 𝜁 and 𝜁 ≤ 𝜉 hold. As usual, 𝜉 < 𝜁 means 𝜉 ≤ 𝜁 and not 𝜁 ≤ 𝜉.
Computability for functions is de ined through representations and computability of string functions.

2.6 COMPUTATIONAL INTERPRETATION OF LEARNING
De inition 2.38: Computability and realization
Let 𝜉 and 𝜁 be representations of 𝑋 and 𝑌, respectively. An element 𝑥 ∈ 𝑋 is
𝜉-computable if there is some computable 𝑝 such that 𝜉(𝑝) = 𝑥. A function 𝜉-computable𝑓 ∶⊆ 𝑋 → 𝑌 is (𝜉, 𝜁)-computable if there is some computable 𝑔 such that(𝜉, 𝜁)-computable𝑓 ∘ 𝜉(𝑝) = 𝜁 ∘ 𝑔(𝑝)for all 𝑝 ∈ dom(𝜉). This 𝑔 is called a (𝜉, 𝜁)-realization of 𝑓.

(𝜉, 𝜁)-realization

Thus the abstract function 𝑓 is “realized” by the concrete function (Type-2 machine) 𝑔 through the two representations 𝜉 and 𝜁.
∗
Various representations of the set of nonempty compact sets 𝒦 are well studied by Brattka and Weihrauch (1999); Brattka and Presser (2003). Let
𝒬= 𝐴⊂ℚ

𝐴 is inite and nonempty∗

and de ine a representation 𝜈𝒬 ∶⊆ Σ → 𝒬 by𝜈𝒬 ([𝑤 ‖𝑤 ‖ … ‖𝑤 ]) ≔ {𝜈ℚ (𝑤 ), 𝜈ℚ (𝑤 ), … , 𝜈ℚ (𝑤 )},∗where 𝜈ℚ ∶⊆ (Σ ) → ℚ is the standard binary notation of rational numbers
de ined by𝜈ℚ (⟨ 𝑤 , 𝑤 , … , 𝑤 ⟩)
|

≔

|

|

𝑤 ⋅2

(

)

,

|

|

𝑤 ⋅2

(

)

,…,

|

𝑤 ⋅2

(

)

and “[”, “]”, and “‖” are special symbols used to separate two inite sequences. For a
inite set of inite sequences {𝑤 , … , 𝑤 }, for convenience we introduce the mapping 𝜄 which translates the set into a inite sequence de ined by 𝜄(𝑤 , … , 𝑤 ) ≔[𝑤 ‖ … ‖𝑤 ]. Note that 𝜈ℚ (⟨ 𝑤 , … , 𝑤 ⟩) = (min 𝜌(𝑤 ), … , min 𝜌(𝑤 )) for our
representation 𝜌 introduced in equation (2.2). The standard representation of the∗
topological space (𝒦 , 𝑑H ), given by Brattka and Weihrauch (1999, De inition 4.8),
is de ined in the following manner.
De inition 2.39: Standard representation of igures∗
De ine the representation 𝜅H ∶⊆ Σ → 𝒦 of igures by 𝜅H (𝑝) = 𝐾 if 𝑝 =𝑤 ‖𝑤 ‖𝑤 ‖ … ,𝑑H (𝐾, 𝜈𝒬 (𝑤 )) < 2
for each 𝑖 ∈ ℕ, and lim → 𝜈𝒬 (𝑤 ) = 𝐾.
∗

Note that the topology on 𝒦 induced by the Hausdorff metric 𝑑H coincides with∗
the Vietoris topology on 𝒦 (Beer, 1993). This representation 𝜅H is known to be Vietoris topology∗
an admissible representation of the space (𝒦 , 𝑑H ), which is the key concept in TTE admissible representation0(Schröder, 2002a; Weihrauch, 2000), and is also known as the Σ1 -admissible representation proposed by de Brecht and Yamamoto (2009).

37

38

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

2.6.2 Computability and Learnability of Figures
First, we show computability of igures in 𝜅(ℋ).
Theorem 2.40
For every igure 𝐾 ∈ 𝜅(ℋ), 𝐾 is 𝜅H -computable.
Proof. It is enough to prove that there exists a computable function 𝑓 such that
𝜅(𝐻) = 𝜅H (𝑓(𝐻)) for all 𝐻 ∈ ℋ. Fix a hypothesis 𝐻 ∈ ℋ such that 𝜅(𝐻) = 𝐾. For
all 𝑘 ∈ ℕ and for 𝐻 de ined by∗

𝐻 ≔ 𝑤 ∈ (Σ )𝑤 ⊑ 𝑣 with 𝑣 ∈ 𝐻 for some 𝑚, and |𝑤| = 𝑘 ,we can easily check that𝑑H (𝐾, 𝜈𝒬 (𝜄(𝐻 ))) < diam(𝑘) = √𝑑 ⋅ 2Moreover, for each 𝑘, √𝑑 ⋅ 2

( )

<2

.

, where𝑔(𝑘) = ⌈𝑘 + log √𝑑⌉.
Therefore there exists a computable function 𝑓 which translates 𝐻 into a representation of 𝐾 given as follows: 𝑓(𝐻) = 𝑝 such that 𝑝 = 𝑤 ‖𝑤 ‖ … such that
𝜄(𝐻 ( ) ) = 𝑤 for all 𝑘 ∈ ℕ.
Thus a hypothesis 𝐻 can be viewed as a “code” of a Type-2 machine that producesa 𝜅H -representation of the igure 𝜅(𝐻).
Both informants and texts are also representations (in the sense of TTE) of∗
compact sets. De ine the mapping 𝜂INF by 𝜂INF (𝜎) ≔ 𝐾 for every 𝐾 ∈ 𝒦 and∗informant 𝜎 of 𝐾, and the mapping 𝜂TXT by 𝜂TXT (𝜎) ≔ 𝐾 for every 𝐾 ∈ 𝒦 and text
𝜎 of 𝐾. Trivially 𝜂INF < 𝜂TXT holds, that is, some Type-2 machine can translate 𝜂INF
to 𝜂TXT , but no machine can translate 𝜂TXT to 𝜂INF . Moreover, we have the following
hierarchy of representations.
Lemma 2.41
𝜂INF < 𝜅H , 𝜂TXT ≰ 𝜅H , and 𝜅H ≰ 𝜂TXT .
Proof. First we prove 𝜂INF ≤ 𝜅H , that is, there is some computable function 𝑓 suchthat 𝜂INF (𝜎) = 𝜅H (𝑓(𝜎)). Fix a igure 𝐾 and its informant 𝜎 ∈ dom(𝜂INF ). For all𝑘 ∈ ℕ, we have𝑑H (𝐾, Pos (𝐾)) ≤ diam(𝑘) = √𝑑 ⋅ 2
and Pos (𝐾) can be obtained from 𝜎. Moreover, for each 𝑘, √𝑑 ⋅ 2
where

( )

< 2

,

𝑔(𝑘) = ⌈𝑘 + log √𝑑⌉.
Therefore there exists a computable function 𝑓 that translates 𝜎 into a representation of 𝐾 as follows: 𝑓(𝜎) = 𝑝, where 𝑝 = 𝑤 ‖𝑤 ‖ … such that 𝑤 = 𝜄(Pos ( ) (𝐾))for all 𝑘 ∈ ℕ.
Second, we prove 𝜂TXT ≰ 𝜅H . Assume that the opposite, 𝜂TXT ≤ 𝜅H holds. Then
there exists a computable function 𝑓 such that 𝜂TXT (𝜎) = 𝜅H (𝑓(𝜎)) for every igure2.6 COMPUTATIONAL INTERPRETATION OF LEARNING∗

𝐾 ∈ 𝒦 . Fix a igure 𝐾 and its text 𝜎 ∈ dom(𝜂TXT ). This means that for any small
𝜀 ∈ ℝ, 𝑓 can ind a pre ix 𝜎[𝑚] such that 𝑑H (𝐾, 𝐼) ≤ 𝜀, where 𝐼 = {𝜌(𝑤) ∣ (𝑤, 1) ∈range(𝜎[𝑚])}. However, since 𝜎 contains only positive examples, no computablefunction can ind such 𝑚. It follows that 𝜂TXT ≰ 𝜅H .
Third, we prove 𝜅H ≰ 𝜂INF and 𝜅H ≰ 𝜂TXT . There is a igure 𝐾 such that∗𝐾 ∩ 𝜌(𝑤) = {𝑥} for some 𝑤 ∈ Σ , i.e., 𝐾 and 𝜌(𝑤) intersect in only one point 𝑥.
Such a 𝑤 must be in 𝜎 as a positive example, that is, 𝑤 ∈ Pos(𝐾). However, a representation of 𝐾 can be constructed without 𝑤. There exists an in inite sequence𝑝 ∈ 𝜅H with 𝑝 = 𝑤 ‖𝑤 ‖ … such that 𝑥 ∉ 𝜈𝒬 (𝑤 ) for all 𝑘 ∈ ℕ. Therefore there
is no computable function that outputs an example (𝑤, 1) from 𝑝, meaning that
𝜅H ≰ 𝜂INF and 𝜅H ≰ 𝜂TXT .
Here we interpret learning of igures as computation based on TTE. If we see theoutput of a learner, i.e., an in inite sequence of hypotheses, as an in inite sequence
encoding a igure, the learner can be viewed as a translator of codes of igures. Naturally, we can assume that the hypothesis space ℋ is a discrete topological space,that is, every hypothesis 𝐻 ∈ ℋ is isolated and is an open set itself. De ine themapping limℋ ∶ ℋ → ℋ, where ℋ is the set of in inite sequences of hypotheses in ℋ, by limℋ (𝜏) ≔ 𝐻 if 𝜏 is an in inite sequence of hypotheses that converges
to 𝐻, i.e., there exists 𝑛 ∈ ℕ such that 𝜏(𝑖) = 𝜏(𝑛) for all 𝑖 ≥ 𝑛. This coincides with0
the naïve Cauchy representation given by Weihrauch (2000) and Σ2 -admissible rep- naïve Cauchy representation
resentation of hypotheses introduced by de Brecht and Yamamoto (2009). For any Σ0 -admissible representation∗
set ℱ ⊆ 𝒦 , let ℱ denote the space ℱ equipped with the discrete topology, that 2
is, every subset of ℱ is open, and the mapping idℱ ∶ ℱ → ℱ be the identity onℱ. The computability of this identity is not trivial, since the topology of ℱ is iner
than that of ℱ. Intuitively, this means that ℱ has more informative than ℱ. We
can interpret learnability of ℱ as computability of the identity idℱ . The results inthe following are summarized in Figure 2.5.
Theorem 2.42∗
A set ℱ ⊆ 𝒦 is F E -I -learnable (resp. F E -T -learnable) if and only if
the identity idℱ is (𝜂INF , 𝜅∘limℋ )-computable (resp. (𝜂TXT , 𝜅∘limℋ )-computable).
Proof. We only prove the case of F E -I -learning, since we can prove the caseof F E -T -learning in exactly the same way.
The “only if” part: There is a learner M that F E -I -learns ℱ, hence for all
𝐾 ∈ ℱ and all 𝜎 ∈ dom(𝜂INF ), M converges to a hypothesis 𝐻 ∈ ℋ such that
𝜅(𝐻) = 𝐾. Thus
idℱ ∘ 𝜂INF (𝜎) = 𝜅 ∘ limℋ (M ),(2.11)and this means that idℱ is (𝜂INF , 𝜅 ∘ limℋ )-computable.
The “if” part: For some M, the above equation (2.11) holds for all 𝜎 ∈ dom(𝜂INF ).
This means that M is a learner that F E -I -learns ℱ.
Here we consider two more learning criteria, F F -I -learning and F F T -learning, where the learner generates only one correct hypothesis and halts.
This learning corresponds to inite learning, or one shot learning, introduced by ﬁnite learning
Gold (1967) and Trakhtenbrot and Barzdin (1970) and it is a special case of learn- one shot learning
ing with a bound of mind change complexity, the number of changes of hypothesis,
introduced by Freivalds and Smith (1993) and used to measure the complexity oflearning classes (Jain et al., 1999b). We obtain the following theorem.

39

40

LEARNING FIGURES AS COMPUTABLE CLASSIFICATION

INF(ℱ)

M /

𝜂


ℱ

idℱ

∗

ℋ

/

INF(𝒦 )𝜅 ∘ limℋ

𝜂

/

ℋ


𝒦

ℱ

M


∗

id

/

𝛾 ≡ 𝜅H𝒦∗

Figure 2.5 | The commutative diagram representing FIGEX-INF-learning of ℱ (left), and FIGEFEX-INF-learning of ℱ
(both left and right). In this diagram, INF(ℱ) denotes the set of informants of 𝐾 ∈ ℱ.
Theorem 2.43∗
A set ℱ ⊆ 𝒦 is F F -I -learnable (resp. F F -T -learnable) if and only
if the identity idℱ is (𝜂INF , 𝜅)-computable (resp. (𝜂TXT , 𝜅)-computable).
Proof. We only prove the case of F F -I -learning, since we can prove the caseof F F -T -learning in exactly the same way.
The “only if” part: There is a learner M that F F -I -learns ℱ, hence for all
𝐾 ∈ ℱ and all 𝜎 ∈ dom(𝜂INF ) of 𝐾, we can assume that M = 𝐻 such that 𝜅(𝐻) = 𝐾.
Thus we have
idℱ ∘ 𝜂INF (𝜎) = 𝜅(M ).

(2.12)

This means that idℱ is (𝜂INF , 𝜅)-computable.
The “if” part: For some M, the above equation (2.12) holds for all 𝜎 ∈ dom(𝜂INF ).
This means that M is a learner that F F -I -learns ℱ.
Finally, we show a connection between effective learning of igures and thecomputability of igures. Since F E E -T
= ∅ (Theorem 2.31), we only treat∗
learning from informants. We de ine the representation 𝛾 ∶⊆ ℋ → 𝒦 by 𝛾(𝑝) ≔𝐾 if 𝑝 = 𝐻 , 𝐻 , … such that 𝐻 ∈ ℋ and 𝑑H (𝐾, 𝜅(𝐻 )) ≤ 2 for all 𝑖 ∈ ℕ.
Lemma 2.44𝛾 ≡ 𝜅H .
Proof. First we prove 𝛾 ≤ 𝜅H . For the function 𝑔 ∶ ℕ → ℝ such that𝑔(𝑖) = ⌈ 𝑖 + log √𝑑 ⌉,()≤ 2 for all 𝑖 ∈ ℕ. Thus there exists a comwe have diam(𝑔(𝑖)) = √𝑑 ⋅ 2
putable function 𝑓 such that, for all 𝑝 ∈ dom(𝛾), 𝑓(𝑝) is a representation of 𝜅H
since, for an in inite sequence of hypotheses 𝑝 = 𝐻 , 𝐻 , … , all 𝑓 has to do is to()generate an in inite sequence 𝑞 = 𝑤 ‖𝑤 ‖𝑤 ‖ … such that 𝑤 = 𝜄(𝐻 ( ) ) for all𝑖 ∈ ℕ, which results in𝑑H (𝐾, 𝜈𝒬 (𝑤 )) ≤ diam(𝑔(𝑖)) = √𝑑 ⋅ 2
for all 𝑖 ∈ ℕ.

()≤2

2.7 SUMMARY
Next, we prove 𝜅H ≤ 𝛾. Fix 𝑞 ∈ dom(𝜅H ) with 𝑞 = 𝑤 ‖𝑤 ‖ … . For each 𝑖 ∈ ℕ,
let 𝑤 = 𝜄(𝑤 , , 𝑤 , , … , 𝑤 , ). Then the set {𝑤 , , … , 𝑤 , }, which we denote 𝐻 ,becomes a hypothesis. From the de inition of 𝜅H ,𝑑H (𝐾, 𝜅(𝐻 )) ≤ 2
holds for all 𝑖 ∈ ℕ. This means that, for the sequence 𝑝 = 𝑤 , 𝑤 , … , 𝛾(𝑝) = 𝐾. We
therefore have 𝛾 ≡ 𝜅H .
By using this lemma, we interpret effective learning of igures as the computability of two identities (Figure 2.5).
Theorem 2.45∗
A set ℱ ⊆ 𝒦 is F E E -I -learnable if and only if there exists a computablefunction 𝑓 such that 𝑓 is a (𝜂INF , 𝜅 ∘ limℋ )-realization of the identity idℱ , and 𝑓 is∗∗also a (𝜂INF , 𝛾)-realization of the identity id ∶ 𝒦 → 𝒦 .
Proof. We prove the latter half of the theorem, since the former part can be proved
exactly as for Theorem 2.42.
The “only if” part: We assume that a learner M F E E -I -learns ℱ. For all∗
𝐾 ∈ 𝒦 and all 𝜎 ∈ dom(𝜂INF ),id ∘ 𝜂INF (𝜎) = 𝛾(M )holds since the identity id is (𝜂INF , 𝛾)-computable.
The “if” part: For some M, id∘𝜂INF (𝜎) = 𝛾(M ) for all 𝜎 ∈ dom(𝜂INF ). It follows
that M is a learner that F E E -I -learns ℱ.
Thus in F E E -I - and F E E -T -learning of a set of igures ℱ, a learner
M outputs a hypothesis 𝑃 with 𝜅(𝑃) = 𝐾 in inite time if 𝐾 ∈ ℱ, and M outputs∗
the “standard” representation of 𝐾 if 𝐾 ∈ 𝒦 ⧵ ℱ since we prove that 𝛾 ≡ 𝜅H in
Lemma 2.44. Informally, this means that there is not too much loss of informationof igures even if they are not explanatory learnable.

2.7

Summary

We have proposed the learning of igures using self-similar sets based on the Goldstyle learning model towards a new theoretical framework of binary classi ication
focusing on computability, and demonstrated a learnability hierarchy under various learning criteria (Figure 2.3). The key to the computable approach is amalgamation of discretization of data and the learning process. We showed a novel
mathematical connection between fractal geometry and the Gold-style learningmodel by measuring the lower bound of the size of training data with the Hausdorff dimension and the VC dimension. Furthermore, we analyzed our learningmodel using TTE (Type-2 Theory of Effectivity) and presented several mathematical connections between computability and learnability.
Although we theoretically showed that fractals (self-similar sets) can be effectively used as a representation system for learning of igures, there may be more
ef icient systems in terms of time and space complexities. To examine learning
under such systems, our framework can be used as a basis.

41

42

LEARNING FIGURES AS COMPUTABLE CLASSIFICATIONIn these days, most methods in machine learning are based on a statistical approach (Bishop, 2007). The reason is that many data in the real world are in analog (real-valued) form, and the statistical approach can treat such analog data directly in theory. However, all learning methods are performed on computers. Thismeans that all machine learning algorithms actually treat discretized digital data
and, now, most research pays no attention to the gap between analog and digital
data. In this chapter we have proposed a novel and completely computable learning method for analog data, and have analyzed the method precisely. This work
will be a theoretical foundation for computable learning from analog data, such asclassi ication, regression, and clustering.

Part II
From Theory to Practice“the whole Universe is made up of these two classes together”— George Boole, An Investigation of the Laws of Thought

3
CODING DIVERGENCE

T
coding divergenceis to give a practical learning method for continuous
data in a computational manner started from theoretical analysis. We integrate two fundamental processes: discretization of continuous data and learningof a model that explains them. In particular, we treat the problem of measuring thedifference between two sets of real-valued data, since such a problem is basic fora lot of tasks in machine learning, e.g., classi ication and clustering. We propose anovel measure, called coding divergence, to measure similarity between such sets,
and construct a classi ier using it. The key concept of the divergence is the simple
procedure separation: The coding divergence measures the complexity of separating a positive dataset from a negative dataset, which corresponds to the height ofa decision tree separating them, intuitively.
Motivation

negative control
positive control

statistical hypothesis test

In experimental science, a lot of experiments are designed including a negative
control and a positive control. Positive controls con irm that the procedure is competent for observing the effect, and negative controls con irm that the procedure
is not observing an unrelated effect. Testing the effect of a treatment group is theobjective of such an experiment. A typical situation is to test the effect of a new
drug, where the result of applying the new drug (treatment group) is comparedagainst placebo (negative control).
The standard method for the above task is a statistical hypothesis test, which
has developed from the works by Neyman and Pearson (1928, 1933) and Fisher(1925, 1956). However, it is well known that there exist many fundamental problems in statistical hypothesis testing such as non-veri iable assumptions for populations and arbitrary 𝑝 values (Johnson, 1999) when we apply such methods toactual datasets. As a result, even in the top journals such as Nature, Science, andCell, we can easily ind inappropriate usage of statistical hypothesis testing¹. Thusan alternative method to analyze experimental results is required.
All the scientists have to do is to judge to which controls a treatment groupbelongs and, obviously, this typical task in experimental science can be viewed as¹Nature says “please help us” to deal with various statistical methods appropriately (http://www.
nature.com/nature/authors/gta/).

CODING DIVERGENCE
classi ication in machine learning, that is, a pair of negative and positive controls is classiﬁcation
given as a training dataset, and a classi ier labels a treatment group (corresponds
to a test set) “negative” or “positive” using the training set. Note that labels of allelements in the test set are same. Therefore, comparing the similarity between thenegative control and the test set to the one between the positive control and thetest set is a direct way, and the coding divergence, which will be introduced in thischapter, can achieve such task.
Note that lots of existing classi ication methods based on statistics could notwork well, since most of them have been proposed for large datasets, while typical datasets are small in experimental science such as physiology. Moreover, mostclassi ication methods considered in the machine learning community usually classify each test datum, whereas we have to classify a set of test data in controlled
experiments, and the coding divergence is designed for the task.

Contributions
The coding divergence, proposed in this chapter, uses no statistics and no probabilistic models, and purely depends on the topological structure (in the sense ofmathematics) of the Cantor space Σ , which is known as the standard topological space of the set of in inite sequences (Weihrauch, 2000). Thus, for example,
it has no assumptions for the probability distributions of datasets. This property
enables us to develop a robust machine learning algorithm for various datasets.
We identify each real-valued datum with a real vector in the 𝑑-dimensional
Euclidean space ℝ , and realize discretization of a real-valued datum through an
topological mapping from ℝ into the Cantor space Σ . A topology uses open sets open set
to axiomatize the notion of approximation, and it enables us to treat each discretized datum as a base element of an open set of the Cantor space.
The simplest model, which is consistent with the given two sets, is learned in
the Cantor space with assuming that one set is a set of positive examples and theother set is a set of negative examples, i.e., the learned model explains all positive
examples and does not explain any negative examples. This model corresponds tothe minimum open set in the Cantor space Σ . The coding divergence is obtained minimum open set
from the length of the code that encodes the learned model in the Cantor space.
Figure 3.1 shows two examples of computing the binary-coding divergence (the binary-coding divergencecoding divergence with the binary encoding) where each datum is a vector in ℝ .
The unit cube ℐ = [0, 1] × [0, 1] is partitioned recursively, and each partitioned
cube is encoded by the binary encoding (see Section 3.2.2 and Figure 3.3). In theleft panel (Figure 3.1a), to separate one set from the other set, we need only one
partition in each dimension, and need 2/10 symbols par datum. In contrast in theright panel (Figure 3.1b), three recursive partitions are needed in each dimension,
and we need 26/10 symbols par datum. The binary-coding divergence is determined directly from these numbers of symbols, that is, 2/10+2/10 = 0.4 in Figure
3.1a and 26/10 + 26/10 = 5.2 in Figure 3.1b.
This chapter is organized as follows: we give mathematical background in Section 3.2. Section 3.3 is the main part in this chapter, where we introduce the coding
divergence and a classi ier using the divergence. We analyze performance of theclassi ier experimentally in Section 3.4. Section 3.5 summarizes this chapter.

45

46

CODING DIVERGENCE11

0.5 (0,1)

(1,1)

(0,0)
0

(1,0)
0.5

0

(a)

0.5

1

0

0

0.5

1

(b)

Figure 3.1 | Two examples of computing the binary-coding divergence. Each ○ and × denotes a real-valued
datum, and each cube is encoded by the binary encoding (same as the base-2 embedding in Deﬁnition 3.3). In
the left panel (a), we need a pair of codes (0,0) (encoding [0, 1/2] × [0, 1/2]) to separate ○ from ×, and (1,1)(encoding [1/2, 1] × [1/2, 1]) to separate × from ○, hence we can separate them with 2/10 symbols par datum,and the binary-coding divergence is 2/10 + 2/10 = 0.4 (For each pair of codes, we ignore symbols ''('', '','', and '')'',
and actually tupled into one code. See the equation (2.4)). In contrast in the right panel (b), we need codes (00,00),(00,01), (010,010), (011,011), and (110,100) to separate ○ from ×, and (10,10), (11,11), (111,101), (010,011),
and (011,010) to separate × from ○. Thus they are separated by (4 + 4 + 6 + 6 + 6)/10 symbols par datum, andthe divergence is 26/10 + 26/10 = 5.2.

3.1 Related Work
Liu et al. (2008) used the similar idea of our divergence to anomaly detection. They
constructed decision trees by partitioning intervals randomly and recursively, andused the height of them to measure the complexity of separating each datum. Compared to the method, our contribution is as follows: We treat the way of partition
as the process of discretization and formulate it as a mapping (encoding) of ℝ
into the Cantor space Σ , and show that the height of a decision tree corresponds
to the size of an open set in the topological space Σ . This gives the theoretical justi ication for the process “partition”. Furthermore, this is the irst time to perform
classi ication using such idea.
Discretization is also used to construct decision trees from continuous data.
For example, C4.5 (Quinlan, 1996) is one of famous algorithms to develop decision trees together with discretization. Our approach is different from them since
we realize discretization process by methods of encoding real numbers, give theoretical support by modern mathematical theories in Computable Analysis, andintegrate the process into computational learning.
Kernel methods including Support Vector Machines (SVMs) are known to be
one of the most famous and powerful classi ication methods (Bishop, 2007), where
any form of data, for example sequences, images, and graphs, are mapped into ahigh dimensional feature space, which is the 𝑑-dimensional Euclidean space ℝ ,or more generally the in inite-dimensional Hilbert space, to measure the similarity
between each pair of data. In contrast, our strategy is inverted: Every real-valued
datum in ℝ is mapped into the Cantor space Σ . The “discrete” space Σ mightseem to be strange as a feature space, but is natural for machine learning fromreal-valued data with discretization.

3.2 MATHEMATICAL BACKGROUND
Some studies use the Kolmogorov complexity for machine learning such as
clustering (Cilibrasi and Vitányi, 2005; Li et al., 2003) by measuring the similarity between a pair of data. Compared to the methods, our approach has mainly
two advantages as follows: First, the coding divergence is computable (Section
3.3.3) whereas the Kolmogorov complexity is not, hence they use actual compression algorithms such as gzip to obtain an approximate value of the Kolmogorov
complexity, which result in a gap between theory and practice; second, our approach is more general than theirs, because we can treat any continuous objects
through encoding of them, however they cannot since they do not consider encoding process of objects.

3.2

Mathematical BackgroundWe introduce our mathematical background using the framework of Type-2 Theory of Effectivity (TTE) studied in the area of computable analysis (Weihrauch,2000). Remember that ↑𝑤 denotes the set { 𝑝 ∈ Σ ∣ 𝑤 ⊑ 𝑝 } and, in addition, we
denote by ↑𝑊 the set { 𝑝 ∈ Σ ∣ 𝑤 ⊑ 𝑝 for some 𝑤 ∈ 𝑊 } for a set of inite sequences∗𝑊 ⊆ Σ . The size of 𝑊 is de ined by |𝑊| ≔ ∑ ∈ |𝑤|. For example, if 𝑊 ={𝟷𝟷, 𝟶, 𝟷𝟶𝟶}, then |𝑊| = 2 + 1 + 3 = 6.

3.2.1 The Cantor SpaceFirst, we introduce the Cantor topology into the set of in inite sequences Σ , which Cantor topology
is known as the standard topology on it (Weihrauch, 2000).
De inition 3.1: The Cantor topology∗
De ine 𝜏 ≔ { ↑𝑊 ∣ 𝑊 ⊆ Σ }. We say that 𝜏 is the Cantor topology over Σ,and the topological space (Σ , 𝜏 ) is the Cantor space.
Cantor space
We abbreviate (Σ , 𝜏 ) as Σ if 𝜏 is understood from the context. Then, the set∗{ ↑𝑤 ∣ 𝑤 ∈ Σ } becomes a base of the topology 𝜏 .
Example 3.2
Let an alphabet Σ = {𝟶, 𝟷}. For example, the set ↑𝟶𝟶 = { 𝑝 ∈ Σ ∣ 𝟶𝟶 ⊑ 𝑝 } is abase element of the Cantor space Σ , and ↑𝟷𝟶 is also a base element. Of course,
both sets ↑𝟶𝟶 and ↑𝟷𝟶 are open sets in the space Σ , i.e., ↑𝟶𝟶 ∈ 𝜏 and ↑𝟷𝟶 ∈ 𝜏 .
Assume 𝑊 = {𝟶𝟶, 𝟷𝟶}. Then the set ↑𝑊 = { 𝑝 ∈ Σ ∣ 𝟶𝟶 ⊑ 𝑝 or 𝟷𝟶 ⊑ 𝑝 } is an
open set. Note that ↑𝑊 = 𝟶𝟶Σ ∪ 𝟷𝟶Σ .
The Cantor space Σ can be visualized by a tree, where each in inite sequence 𝑝 ∈Σ corresponds to an in inite descending path from the root (Figure 3.2). Then abase element ↑𝑤 is a full subtree and an open set is a union of full subtrees, andthe length |𝑤| corresponds to the depth of the root of the subtree.

3.2.2 Embedding the Euclidean Space into the Cantor SpaceLet us approach to the 𝑑-dimensional Euclidean space ℝ through the Cantor space
Σ using topological mappings between ℝ and Σ .
We say that a surjective function 𝜌 ∶⊆ Σ → ℝ is a representation of ℝ , and representation
an injective function 𝜑 ∶⊆ ℝ → Σ is an embedding of ℝ , meaning that repre- embedding

47

48

CODING DIVERGENCE
↑010

↑1010

Figure 3.2 | Tree representation of
the Cantor space over ∑= {0, 1}. Subtrees ↑010 and ↑1010 are base elements, and the set ↑010 ∪ ↑1010 isopen.
sentations and embeddings are dual concepts. In the previous chapter, the binary
representation was used as the canonical representation of real numbers. In theield of computable analysis, computability of ℝ or other continuous objects (i.e.,
uncountable sets) are treated through representations with Σ (Schröder, 2002a;
Weihrauch, 2000), and here we apply this theoretical framework to machine learning. For simplicity, we turn an embedding 𝜑 into a bijective function by replacing the codomain Σ by its actual image 𝜑(ℝ ), and assume that a representation
𝜌=𝜑 .
When we identify an element in ℝ with an object with analog quantity, an embedding 𝜑 can be viewed as a mathematical realization of an encoding method oran actual A/D converter, where a continuous signal (analog quantity) is converted
to a sequence of bits (digital quantity). The true value 𝑥 of the analog quantity is
encoded as an in inite sequence 𝜑(𝑥) by the embedding 𝜑, and any pre ix 𝑤 of 𝜑(𝑥)is a discretized digital value. The pre ix 𝑤 tells us 𝑥 is in the interval 𝜑 (↑𝑤), andthe width of the interval𝜑(↑𝑤) = sup 𝑑(𝑥, 𝑦) 𝑥, 𝑦 ∈ 𝜑(↑𝑤)(𝑑 is the Euclidean metric) corresponds to an error of 𝑤. This modeling re lects thefundamental property of observation together with discretization, that is, every(discretized) real-valued datum must have some error intrinsically (Baird, 1994).
We now de ine the standard embedding, the base-𝛽 embedding.

base-𝛽 embedding

De inition 3.3: Base-𝜷 embedding
Assume 𝑑 = 1. The base-𝛽 embedding of the unit interval ℐ = [0, 1] is a mapping𝜑 ∶ ℐ → Σ that maps 𝑥 to an in inite sequence 𝑝 composed as follows: For all𝑖 ∈ ℕ, 𝑥 = 0 implies 𝑝 = 0, and 𝑥 ≠ 0 implies𝑝 =(if 𝑧 < 𝑥 ≤ 𝑧 + 𝛽,()(
if 𝑧 + 𝛽<𝑥≤𝑧+𝛽⎨ ...
⎪⎩𝛽 − 1

if 𝑧 + 𝛽

where
𝑧=

)

⎧0
⎪1

𝑝 𝛽

(

)

if 𝑖 ≠ 0, and 𝑧 = 0 otherwise.

(

) (

)

)

,

<𝑥≤𝑧+𝛽

(

) (

)

,

Position

3.3 CODING DIVERGENCE
4
3
2
1
0
0

φ2(0.3) = 01001...

φ2((0.625, 0.75]) = ↑101

1

49

Figure 3.3 | The (one-dimensional)
binary embedding φ2 . The position i
is 1 if it is on the line, and 0 otherwise.

Especially, we call the base-2 embedding the binary embedding. Figure 3.3 denotes binary embedding
the binary embedding 𝜑 , where each horizontal line means that the corresponding position is 1 on the line and 0 otherwise. For example, let 𝑝 = 𝜑 (0.3). Then𝑝 = 0 since the position 0 is not on the line, and 𝑝 = 1 since the position 1 is on
the line, and so on.
By using the tupling function (see Equation (2.4)), we de ine the 𝑑-dimensional
base-𝛽 embedding 𝜑 from ℐ ⊂ ℝ to Σ by𝑑-dimensional base-𝛽 embedding
𝜑 (𝑥 , 𝑥 , … , 𝑥 ) ≔ ⟨𝜑 (𝑥 ), 𝜑 (𝑥 ), … , 𝜑 (𝑥 )⟩.

(3.1)

Example 3.4
For the binary embedding 𝜑 , we have 𝜑 (0.3) = 𝟶𝟷𝟶 … and 𝜑 (0.5) = 𝟶𝟷𝟷 … .
Thus 𝜑 (0.3, 0.5) = ⟨𝟶𝟷𝟶 … , 𝟶𝟷𝟷 … ⟩ = 𝟶𝟶𝟷𝟷𝟶𝟷 … .
We abbreviate 𝑑 of 𝜑 if it is understood from the context.

3.3

Coding DivergenceWe give a novel measure of the difference between sets in the 𝑑-dimensional Euclidean space ℝ , called the coding divergence¹, with assuming that an element coding divergencein ℝ corresponds to a numerical (real-valued) data point. This divergence isthe main contribution in this chapter. We design a classi ier using it in Subsection 3.3.2, and construct a learner that learns the divergence in Subsection 3.3.3.

3.3.1 De inition and Properties
Let 𝑋 and 𝑌 be a pair of nonempty inite sets in the unit interval ℐ ⊂ ℝ . We encode
them by an embedding 𝜑 from ℐ to the Cantor space Σ .
We set a model of given data as an open set in the Cantor space, since this prop- model
erty “open” is desirable for machine learning as follows: For a set 𝑃, if 𝑃 is open, open set
then there exists a set of inite sequences 𝑊 such that ↑𝑊 = 𝑃 (remember that𝑊Σ = { 𝑝 ∈ Σ ∣ 𝑤 ⊑ 𝑝 for some 𝑤 ∈ 𝑊 }). This means that 𝑃 is initely observable (Smyth, 1992), that is, for any in inite sequence 𝑝, we can decide whether or
not 𝑝 ∈ 𝑃 by observing just some pre ix of 𝑝. Thus from sets of in inite sequences
𝜑(𝑋) and 𝜑(𝑌), we can obtain an open set as a model of them in inite time
We say that a set of in inite sequences 𝑃 ⊆ Σ is consistent with an ordered consistentpair (𝜑(𝑋), 𝜑(𝑌)) if 𝑃 ⊇ 𝜑(𝑋) and 𝑃 ∩ 𝜑(𝑌) = ∅, hence if we see 𝜑(𝑋) and 𝜑(𝑌) as¹The usage of the word “divergence” follows the original de inition of the Kullback-Leibler divergence (Kullback and Leibler, 1951).

50

CODING DIVERGENCE
positive and negative examples, respectively, then the set 𝑃 “explains” all elementsin 𝜑(𝑋) and does not explain any elements in 𝜑(𝑌).
De inition 3.5: Coding divergence
Given an embedding 𝜑 ∶ ℐ → Σ . For a pair of nonempty inite sets 𝑋, 𝑌 ⊂ ℐ,
de ine the coding divergence with respect to 𝜑 bycoding divergence
𝐶 (𝑋, 𝑌) ≔∞if 𝑋 ∩ 𝑌 ≠ ∅,𝐷 (𝑋; 𝑌) + 𝐷 (𝑌; 𝑋) otherwise,

where 𝐷 is the directed coding divergence with respect to 𝜑 de ined bydirected coding divergence
𝐷 (𝑋; 𝑌) ≔binary-coding divergence

consistent learning1
min { |𝑂| 𝑂 is open, and consistent with (𝜑(𝑋), 𝜑(𝑌)) } .
#𝑋

The standard embedding for the coding divergence is the base-𝛽 embedding 𝜑and, especially, the coding divergence with respect to 𝜑 is written by 𝐶 (𝑋, 𝑌). If
𝛽 = 2, we call 𝐶 (𝑋, 𝑌) the binary-coding divergence.
Intuitively, the procedure to obtain the directed coding divergence is as follows:
We encode given sets 𝑋, 𝑌 into the Cantor space Σ by an embedding 𝜑, ind thesimplest model (minimum open set) 𝑂 that is consistent with an ordered pair ofsets of in inite sequences (𝜑(𝑋), 𝜑(𝑌)), and measure the size |𝑂|. Thus the directed
coding divergence 𝐷 (𝑋; 𝑌) can be viewed as the result of consistent learning (cf.
Subsection 2.3.2) from positive examples 𝜑(𝑋) and negative examples 𝜑(𝑌) in thecomputational learning theory context (Jain et al., 1999b).
Example 3.6
Suppose that 𝑋 = {√0.2, 0.8} and 𝑌 = {𝜋 / 10}. Then in the binary embedding 𝜑(De inition 3.3), these sets are encoded into in inite sequences as follows: 𝜑 (𝑋) ={𝟶𝟷𝟷 … , 𝟷𝟷𝟷 … } and 𝜑 (𝑌) = {𝟶𝟷𝟶 … }. Thus for an open set ↑𝑉 with 𝑉 = {𝟶𝟷𝟷, 𝟷},↑𝑉 is minimum and consistent with (𝜑 (𝑋), 𝜑 (𝑌)). Therefore we have 𝐷 (𝑋; 𝑌) =(3 + 1)/2 = 2. Similarly, ↑𝑊 with 𝑊 = {𝟶𝟷𝟶} is minimum and consistent with(𝜑 (𝑌), 𝜑 (𝑋)), hence 𝐷 (𝑌; 𝑋) = 3/1 = 3. Consequently, 𝐶 (𝑋, 𝑌) = 𝐷 (𝑋; 𝑌) +𝐷 (𝑌; 𝑋) = 5.
It is trivial that the coding divergence is not a metric since 𝐶 (𝑋, 𝑌) ≠ 0 for all
nonempty inite sets 𝑋, 𝑌 ⊂ ℐ.
Lemma 3.7
The coding divergence satis ies the following conditions:
1. 𝐶 (𝑋, 𝑌) > 0.
2. 𝐶 (𝑋, 𝑌) = 𝐶 (𝑌, 𝑋).
Furthermore, it does not satisfy the triangle inequality.
Proof. The conditions 𝐶 (𝑋, 𝑌) > 0 and 𝐶 (𝑋, 𝑌) = 𝐶 (𝑌, 𝑋) are proved directlyfrom the de inition. We can easily ind an example, where the triangle inequality
does not hold. For example, let 𝑋 = {0.1}, 𝑌 = {0.8}, and 𝑍 = {0.2}, and assume that we use the binary embedding. We have 𝐶 (𝑋, 𝑌) = 2, 𝐶 (𝑌, 𝑍) = 2, and𝐶 (𝑍, 𝑋) = 6, thus 𝐶 (𝑋, 𝑌) + 𝐶 (𝑌, 𝑍) < 𝐶 (𝑍, 𝑋).

3.3 CODING DIVERGENCE

3.3.2 Classi ication Using Coding Divergence
We construct a lazy learner that classi ies a given dataset using the coding diver- lazy learner
gence. It classi ies a test set by measuring its similarity (coding divergence) to a
training dataset. It is quite simple and has no parameters, hence we can apply it
to any type of real-valued datasets without any consideration. We will show its
robustness in Section 3.4.
Assume that there are two classes 𝐴 and 𝐵, and a pair (𝑋, 𝑌) is given as a training dataset (𝑋 and 𝑌 are nonempty inite sets in ℝ ), where 𝑋 (resp. 𝑌) belongs
to 𝐴 (resp. 𝐵). Moreover, suppose that we have a dataset 𝑍 ⊂ ℝ in which every
datum is unlabeled, and we just know that all labels of elements in 𝑍 are same.
The classi ier performs as follows: First, it computes 𝐶 (𝑋, 𝑍) and 𝐶 (𝑌, 𝑍),
and next, judges
𝑍 belongs to the class𝐴𝐵

if 𝐶 (𝑋, 𝑍) > 𝐶 (𝑌, 𝑍),
otherwise.

Generally, the computability of 𝐶 (𝑋, 𝑍) and 𝐶 (𝑌, 𝑍) is not guaranteed. However,
if we use the base-𝛽 embedding 𝜑 such as the binary embedding (De inition 3.3),
then it is effectively computable. We give the learner 𝜓 that learns the coding divergence with respect to 𝜑 in the next subsection.

3.3.3 Learning of Coding Divergence
Here we integrate discretization of encoded real-valued data (in inite sequences)and learning of the coding divergence with respect to 𝜑 , and construct a learnerthat learns the divergence 𝐶 (𝑋, 𝑌) from 𝜑 (𝑋) and 𝜑 (𝑌).
Procedure 3.1 shows the learner 𝜓 that learns the coding divergence 𝐶 (𝑋, 𝑌)
from a pair of given sets of encoded in inite sequences 𝜑 (𝑋) and 𝜑 (𝑌). It continues to discretize each sequence, obtains longer and longer pre ixes, and generates
approximate values of the 𝐶 (𝑋, 𝑌). For inputs 𝜑 (𝑋) and 𝜑 (𝑌), the output of𝜓 ( inite or in inite sequence) is denoted by 𝜓(𝜑 (𝑋), 𝜑 (𝑌)), hence for example
𝜓(𝜑 (𝑋), 𝜑 (𝑌))(0) is the irst output, 𝜓(𝜑 (𝑋), 𝜑 (𝑌))(1) is the second output,
and so on.
To learn the coding divergence, a learner has to judge whether or not an open
set 𝑂 is consistent with given sets (𝜑 (𝑋), 𝜑 (𝑌)), and we show that it is decidable
in inite time.
Lemma 3.8
For every open set 𝑂 and every 𝑃, 𝑄 ⊆ Σ , it is decidable that whether or not 𝑂 isconsistent with (𝑃, 𝑄) in inite time.
Proof. Let 𝑂 = 𝑊Σ and 𝑘 = max ∈ |𝑤|, and de ine 𝑃[𝑘] ≔ { 𝑝[𝑘] ∣ 𝑝 ∈ 𝑃 } and𝑄[𝑘] ≔ { 𝑞[𝑘] ∣ 𝑞 ∈ 𝑄 }. We show that we can check the consistency only using 𝑊,
𝑃[𝑘], and 𝑄[𝑘] (thus it is decidable). The condition 𝑂 ⊇ 𝑃 holds if and only if for
all 𝑥 ∈ 𝑃[𝑘], there exists 𝑤 ∈ 𝑊 such that 𝑤 ⊑ 𝑥. Moreover, 𝑂 ∩ 𝑄 = ∅ if and only
if 𝑦 ⋢ 𝑤 and 𝑤 ⋢ 𝑦 for all 𝑤 ∈ 𝑂 and 𝑦 ∈ 𝑄[𝑘].
If 𝑋∩𝑌 = ∅, then 𝜓 halts in inite time, and the last output is exactly the same as
𝐶 (𝑋, 𝑌). Otherwise if 𝑋 ∩ 𝑌 ≠ ∅, then 𝜓 continues to output approximate values

51

52

CODING DIVERGENCE

Procedure 3.1: Learner 𝜓 that learns 𝐶 (𝑋, 𝑌)
Input: Pair (𝜑 (𝑋), 𝜑 (𝑌)) (both 𝑋 and 𝑌 are nonempty inite sets in ℐ)
Output: Finite or in inite sequence converging to 𝐶 (𝑋, 𝑌)
function M (𝜑 (𝑋), 𝜑 (𝑌))1: L
(𝜑 (𝑋), 𝜑 (𝑌), ∅, ∅, #𝑋, #𝑌, 0)function L(𝑃, 𝑄, 𝐻 , 𝐻 , 𝑚, 𝑛, 𝑘)
1: 𝑉 ← D(𝑃, 𝑘)
2: 𝑊 ← D(𝑄, 𝑘)
3: 𝐻 ← 𝐻 ∪ {𝑣 ∈ 𝑉 ∣ 𝑣 ∉ 𝑊}
4: 𝐻 ← 𝐻 ∪ {𝑤 ∈ 𝑊 ∣ 𝑤 ∉ 𝑉}
5: 𝑃 ← {𝑝 ∈ 𝑃 ∣ 𝑝 ∉ 𝐻 Σ }6: 𝑄 ← {𝑞 ∈ 𝑄 ∣ 𝑞 ∉ 𝐻 Σ }7: output 𝑚 ∑ ∈ |𝑣| + 𝑛 ∑ ∈ |𝑤|
8: if 𝑃 = ∅ and 𝑄 = ∅ then halt
9: else return L(𝑃, 𝑄, 𝐻 , 𝐻 , 𝑚, 𝑛, 𝑘 + 1)
function D(𝑃, 𝑘)
1: return {𝑝[𝑛] ∣ 𝑝 ∈ 𝑃}, where 𝑛 = (𝑘 + 1)𝑑 − 1of the 𝐶 (𝑋, 𝑌) = ∞ forever. We can easily prove that for all 𝑖, 𝑗 ∈ ℕ with 𝑖 < 𝑗,
𝜓(𝜑 (𝑋), 𝜑 (𝑌))(𝑖) ≤ 𝜓(𝜑 (𝑋), 𝜑 (𝑌))(𝑗), andlim 𝜓(𝜑 (𝑋), 𝜑 (𝑌))(𝑖) = ∞.
→eﬀective
Type-2 machineThis means that we can obtain more and more accurate values of the coding divergence, even though we cannot obtain the exact value in inite time. This property
corresponds to the effective computability realized by the Type-2 machine, where
while a computer reads more and more precise information (longer and longer
pre ixes) of the input, it produces more and more accurate approximations of theresult. This machine is a natural extension of the usual Turing machine, and used to
introduce computability of continuous objects (Weihrauch, 2000). Furthermore,this property is an effective version of the consistency in statistical context.
Example 3.9
Let us consider the case in Figure 3.1, and assume that 𝑋 and 𝑌 consist of all ○ and×, respectively. Let 𝑃 = 𝜑 (𝑋) and 𝑄 = 𝜑 (𝑌). Every pair of codes is tupled(𝑃, 0) returns
into one code by the tupling function. In Figure 3.1a, D{𝟶𝟶} (corresponds to (𝟶, 𝟶)) and D(𝑄, 0) returns {𝟷𝟷} (corresponds to(𝟷, 𝟷)). Thus 𝜓 outputs 2/10 + 2/10 = 0.4 and halts. In Figure 3.1b, both D (𝑃, 0) and D(𝑄, 0) return {𝟶𝟶, 𝟷𝟷}, and D(𝑃, 1) returns{𝟶𝟶𝟶𝟶, 𝟶𝟶𝟶𝟷, 𝟶𝟶𝟷𝟷, 𝟷𝟷𝟷𝟶}, and D(𝑄, 1) returns {𝟷𝟷𝟶𝟶, 𝟷𝟷𝟷𝟷, 𝟶𝟶𝟷𝟷,
𝟷𝟷𝟷𝟶}. Thus 𝐻 = {𝟶𝟶𝟶𝟶, 𝟶𝟶𝟶𝟷} and 𝐻 = {𝟷𝟷𝟶𝟶, 𝟷𝟷𝟷𝟷}. Furthermore, for the
rest of data, D
performs similarly, and inite sequences 001100, 001111,
111000 and 111011, 001110, 001101 are added to 𝐻 and 𝐻 , respectively. Thus 𝜓
outputs 26/10 + 26/10 = 5.2 and halts.

3.4 EXPERIMENTS

3.4

Experiments

We evaluate the performance of the classi ier given in Subsection 3.3.2 by experiments to analyze the coding divergence empirically. We compared accuracy ofclassi ication performed by the classi ier obtained from sensitivity and speci icity
to those of other classi ication methods.
Theoretically, our classi ier can be applied to actual analog data such as (raw)real-valued data and continuous signals. However, it is dif icult to collect such typeof datasets and apply our classi ier to them directly. Thereby in the following we
use just discretized real-valued data stored in databases.

3.4.1 Methods
First, we construct the learning algorithm M for empirical experiments that computes an approximate value of the coding divergence with respect to the base-𝛽
embedding and always halts in inite time (Algorithm 1). The algorithm M does
not receive in inite sequences, but receives just inite sequences.
De ine for each 𝑖 ∈ ℕ,
𝜑 (𝑋)[𝑖] ≔ 𝑤 |𝑤| = 𝑖 and 𝑝 ∈ ↑𝑤 for some 𝑝 ∈ 𝜑 (𝑋) .
The algorithm M is slightly different from the learner 𝜓, since it receives only inite sequences 𝜑 (𝑋)[𝑖] and 𝜑 (𝑌)[𝑗]. We can get the exact coding divergence(i.e., M(𝜑 (𝑋)[𝑖], 𝜑 (𝑌)[𝑗]) = 𝐶 (𝑋, 𝑌)) in the usual situation where 𝜑 (𝑋)[𝑖] ∩𝜑 (𝑌)[𝑗] = ∅ holds. Moreover, the output of M (denoted by M(𝜑 (𝑋)[𝑖], 𝜑 (𝑌)[𝑗]))has the monotonicity with respect to 𝑖 and 𝑗: For all pairs of 𝑖, 𝑗 and 𝑖′, 𝑗′ with 𝑖 ≤ 𝑖′ monotonicity
and 𝑗 ≤ 𝑗′, we have
𝐶 (𝑋, 𝑌) − M(𝜑 (𝑋)[𝑖], 𝜑 (𝑌)[𝑗])≥ 𝐶 (𝑋, 𝑌) − M(𝜑 (𝑋)[𝑖′], 𝜑 (𝑌)[𝑗′]) ,andlim 𝐶 (𝑋, 𝑌) − M(𝜑 (𝑋)[𝑖], 𝜑 (𝑌)[𝑗]) = 0.

, →Thus, intuitively, if we obtain more and more accurate data (longer and longer sequences), then approximation of the coding divergence becomes better and better,meaning that M is an effective algorithm.
The computational complexity of M is 𝑂(𝑚𝑛𝑑), where 𝑚 and 𝑛 are the cardinality of 𝑋 and 𝑌, respectively, since in each level 𝑘, updating 𝑉 and 𝑊 (lines 7 and
8 in Algorithm 1) takes 𝑂(𝑚𝑛𝑑).
To treat data that are not in the unit interval ℐ, we use min-max normalizationthat maps a value 𝑥 to 𝑥′, where𝑥′ =𝑥 − min{𝑋 ∪ 𝑌 ∪ 𝑍}.
max{𝑋 ∪ 𝑌 ∪ 𝑍} − min{𝑋 ∪ 𝑌 ∪ 𝑍}

The classi ier with the above algorithm M was implemented in R version 2.12.1(R Development Core Team, 2011). We tested the performance of it by evaluating
accuracy, the standard error measure of classi ication (Han and Kamber, 2006).

53

54

CODING DIVERGENCE

Algorithm 3.2: Learning algorithm M that learns 𝐶 (𝑋, 𝑌)
Input: Pair (𝜑 (𝑋)[𝑖], 𝜑 (𝑌)[𝑗]) (𝑋, 𝑌 ⊂ ℐ and 𝑖, 𝑗 ∈ ℕ)
Output: Approximate value of 𝐶 (𝑋, 𝑌)
function M (𝑉, 𝑊)
1: (𝐷 , 𝐷 ) ← L(𝑉, 𝑊, 0, 0, 0, min{𝑖, 𝑗})
2: return 𝐷 / #𝑉 + 𝐷 / #𝑊
function L(𝑉, 𝑊, 𝐷 , 𝐷 , 𝑘, 𝑘)
1: 𝑉dis ← D(𝑉, 𝑘)
2: 𝑊dis ← D(𝑊, 𝑘)
3: 𝑉sep ← {𝑣 ∈ 𝑉dis ∣ 𝑣 ∉ 𝑊dis }4: 𝑊sep ← {𝑤 ∈ 𝑊dis ∣ 𝑤 ∉ 𝑉dis }5: 𝐷 ← 𝐷 + ∑ ∈ sep |𝑣|
6: 𝐷 ← 𝐷 + ∑ ∈ sep |𝑤|
7: 𝑉 ← {𝑣 ∈ 𝑉 ∣ 𝑣 ∉ 𝑉sep Σ }8: 𝑊 ← {𝑤 ∈ 𝑊 ∣ 𝑤 ∉ 𝑊sep Σ }9: if 𝑉 = ∅ and 𝑊 = ∅ then return (𝐷 , 𝐷 )10: else if 𝑘 = 𝑘then return (𝐷 + 𝑛#𝑉, 𝐷 + 𝑛#𝑊)
11: else return L(𝑃, 𝑄, 𝐷 , 𝐷 , 𝑘 + 1, 𝑘)∗

function D(𝑉, 𝑘) // 𝑉 ⊂ Σ
1: return {𝑣[𝑛] ∣ 𝑣 ∈ 𝑉} (𝑛 = (𝑘 + 1)𝑑 − 1)

Ten datasets were collected from UCI Machine Learning Repository (Frank andAsuncion, 2010): abalon, transfusion, sonar, glass, segmentation, ionosphere, madelon, magic, waveform, and yeast. Every datum in each dataset is real-valued typeand belongs to one of two classes (if there are more than two classes, we picked up
just two classes). We used the size of each dataset 10 and 30, since one of applications is controlled experiments in life science, where such small sizes are typical.
We repeated the following procedure 10,000 times:
1. Sample 𝑑 attributes (𝑑 is 1, 2, or 3),
2. collect 𝑛 data for a training set 𝑋 and for a test set 𝑇 from one class by independent random sampling without replacement, and 𝑌 and 𝑇 from the otherclass, where 𝑛 = 10 or 30,
3. Using 𝑋 and 𝑌, classify test sets 𝑇 and 𝑇 by our method and other classi ication methods.
The binary-coding divergence was used throughout all experiments.
Let 𝑡pos be the number of true positives, that is, the number of the case in which
𝑇 is classi ied correctly, and 𝑡neg be the number of true negatives. We calculated
the accuracy by (𝑡pos + 𝑡neg )/20000, since the sensitivity is 𝑡pos /10000 and thespeci icity is 𝑡neg /10000. For reference, we used SVM with the RBF kernel, SVM
with the polynomial kernel, and 𝑘-nearest neighbor classi iers (𝑘 = 1 and 5). We
used the function ksvm in the “kernlab” package for SVM (Karatzoglou et al., 2004),
and the function knn in the “class” package for the 𝑘-nearest neighbor classi iers,3.5 SUMMARY
which have been implemented in R. Note that these methods classify each element
in a test set, thereby we classi ied 𝑇 (or 𝑇 ) to the class in which greater numberof elements in 𝑇 (or 𝑇 ) are classi ied.

3.4.2 Results and Discussions
The experimental result of accuracy is shown in Figure 3.4. Let us compare the result using the binary-coding divergence to those of other methods. In most cases,
the accuracy of our classi ier shows the highest value, and only in some cases
other methods are more accurate than our method (e.g., low dimensional datasets
in yeast). This result shows the robustness of our classi ier for various datasets.
The simple process “separation by encoding” of our theoretical background mightcause this robustness.
Moreover, results of other methods highly depend on kinds of datasets. For
example, 1- and 5-nearest neighbor methods are better than SVMs in ionosphere,
but are worse in abalon. This means that these methods require adjustment ofparameters depending on datasets. However, our method has no parameters anddoes not need any adjustment. Thus we can apply our method effectively without
special background knowledge about datasets.

3.5

Summary

In this chapter, we have proposed a novel measure of the difference between sets,called the coding divergence, and integrated discretization of analog data and learning of models that explains given data through computational learning of it, where
the extracted model corresponds to the minimum open set in the Cantor space
Σ . The key idea is separation of intervals, which corresponds to realization of an
encoding process of analog data by a mathematical embedding of the Euclidean
space into the Cantor space. Furthermore, we have constructed a classi ier, thelazy learner using the coding divergence, and shown the robust performance of itby empirical experiments.
Our mathematical framework is general and has possibility to develop further
in the ield of machine learning and data mining, since any type of datasets can behandled through an appropriate embedding from such objects to the Cantor space.
This approach is new and, intuitively, opposite to the one using kernel functions.
For any other measures or metrics used in learning, the property of invarianceis usually required to guarantee the soundness of them. Since the difference between two coding divergences is used for learning, providing the invariance of thedifference with respect to geometric transformation, such as parallel translation,
rotation, and other af ine transformations, is an important future work.
Furthermore, since our proposed measure is quite simple, it can be applied tovarious tasks in machine learning and data mining. For example, we can measure
the difference between clusters by the coding divergence. This means that we canperform hierarchical clustering using the coding divergence directly. Thus applying our measure to such tasks is a future work.
Another future work is an application for other actual datasets. For example,
image matching is an important topic in computer vision, where each image is assumed to be a set of real vectors. One of well-known distances is the Hausdorff
metric used in the previous chapter, and some metrics based on the Hausdorff

55

CODING DIVERGENCE

Accuracy

Accuracy

abalon
0.55
0.5
0.45
0.4
0.8
0.7
0.6
0.5

Accuracy

1.0transfusion

Binary-coing divergence
SVM (RBF kernel)
SVM (polynomial kernel)
1−nearest neighbor5−nearest neighbor

0.8
0.7
0.6
0.5
sonar

1.0

glass

0.9
0.8
0.7
segmentation

1.0

0.9

0.8

0.8

0.6

0.5

0.7

0.45

0.6

0.4

0.5

0.9
0.8
0.7
0.6

waveform

ionospheremagic

Accuracy

madelonAccuracy

56

0.6

yeast

0.5
0.4
(1, 10) (2, 10) (3, 10) (1, 30) (2, 30) (3, 30)(1, 10) (2, 10) (3, 10) (1, 30) (2, 30) (3, 30)(Number of attributes, Size of each data set)

(Number of attributes, Size of each data set)

Figure 3.4 | Experimental results of accuracy using real-valued datasets collected from UCI repository. We applied
our classiﬁer (using the binary-coding divergence) and other four classiﬁcation methods: SVM with the RBF kernel,
SVM with the polynomial kernel, k-nearest-neighbor classiﬁers (k = 1, 5), to ten real-valued datasets: abalon,transfusion, sonar, glass, segmentation, ionosphere, madelon, magic, waveform, and yeast. We examined six cases:
the number of used attributes are 1, 2, or 3, and the size of each sampled training/test datasets are 10 or 30.

3.6 OUTLOOK: DATA STREAM CLASSIFICATION ON TREES
metric have been proposed (Huttenlocher et al., 1993; Zhao et al., 2005). Triviallyour method can be applied to such topics. In preliminary experiments, we havechecked that the accuracy of classi ication using the coding divergence is better
than that using the Hausdorff metric. Thus our method might be a better classi ier
than that with the Hausdorff metric.

3.6

Outlook: Data Stream Classiﬁcation on Trees

In this section, we show our preliminary results as outlook of the coding divergence: classi ication of data streams. Machine learning from data streams (Bab- data streamcock et al., 2002), which rapidly grow without limit, is one of challenging problems and, to date, many techniques have been proposed (Aggarwal et al., 2004;
Aggarwal and Yu, 2010; Gaber et al., 2005; Gama and Kosina, 2011). For instance,
Domingos and Hulten (2000) proposed VFDT for data stream classi ication, whichlearns a decision tree whose size is kept small by using Hoeffding bounds. However, since the amount of data increasingly grows like the data deluge², a morescalable and effective algorithm is required.
We construct an algorithm, called CODE (Classi ier with the binary encODingon trEes), for classi ication of numerical data streams. It incrementally constructs
a trie-like tree structure from received labeled data by discretizing them based on trie
the binary encoding scheme. The coding divergence is used to measure the similarity between stored labeled data on a tree and an unlabeled datum. The succinct
tree structure leads fast computation of the coding divergence, which can be applied to massive data streams.

3.6.1 CODE approach
We revise the coding divergence to evaluate the similarity between a set of storedlabeled data points (training data) 𝑋 and an unlabeled data point (test datum) 𝑥.
De inition 3.10: Similarity based on the binary encoding
Given a dataset 𝑋 and a datum 𝑥, de ine𝑠(𝑥; 𝑋) ≔1
min 𝜔 (↑𝑊) ↑𝑊 ⊇ 𝜑 (𝑋) and ↑𝑊 ∩ {𝜑 (𝑥)} = ∅ ,
#𝑋

where
𝜔 (↑𝑤) ≔ |𝑤| ⋅ #(↑𝑤 ∩ 𝜑 (𝑋)) and𝜔 (↑𝑊) ≔𝜔 (↑𝑤).
∈

Note that if we de ine 𝜔 (↑𝑤) ≔ |𝑤|, 𝑠(𝑥; 𝑋) is exactly the same as the directed
coding divergence 𝐷(𝑋; {𝑥}), thus the similarity 𝑠(𝑥; 𝑋) can be viewed as weighted
coding divergence between 𝑥 and 𝑋.
Next, we show that the similarity is effectively calculated using a trie-like data
structure, which naturally extends the domain of data from ℐ to ℝ . Procedure 3.3²This phrase is from the special issue of Science (http://www.sciencemag.org/site/special/
data/).

57

58

CODING DIVERGENCE

Figure 3.5 | Examples of calculating
similarities. White and black points
are labeled and unlabeled data, respectively. In the left-hand side, thesimilarity between labeled and unlabeled data is ((3 + 5 + 3) ⋅ 2)/11 =2. In the right-hand side, the similarity is ((3 + 3) ⋅ 2 + (1 + 2 + 2) ⋅ 4)/11= 32/11 = 2.91.
constructs a tree 𝒯 from labeled data, which corresponds to training phase formaking a model, and computes the similarity between 𝑥 and 𝒯 if the input is unlabeled, which corresponds to test phase using the model. In any tree 𝒯, we assume
that every node 𝑣 has three properties: key(𝑣), org(𝑣), and num(𝑣), where key(𝑣)is the key of 𝑣, org(𝑣) is a value of an original datum pointed from 𝑣, and num(𝑣)is the number of leaf nodes of 𝑣. Then, for any unlabeled 𝑥, the similarity between
𝑥 and 𝒯 computed by Procedure 3.3 is exactly the same as the similarity 𝑠(𝑥; 𝑋).
The binary encoding process of 𝑥 is performed by the simple arithmetic operation ⌊𝑥 ⋅ 2 ⌋ at each discretization level 𝑘, and this operation is applicable to any
real number in ℝ. We thus can apply our algorithm to any dataset, where each datum is a point in the 𝑑-dimensional Euclidean space ℝ . Figure 3.5 illustrates two
examples of computing the similarity on trees.
Procedure 3.3 incrementally updates a tree 𝒯 and its time complexity is 𝑂(𝑁 +𝐷), where 𝑁 = max ∈𝒯 #{𝑣′ ∈ 𝒯 ∣ 𝑣′ is a child of 𝑣} and 𝐷 is the maximum depth of𝒯. The number 𝑁 grows exponentially when the dimension 𝑑 increases. To escapefrom this “curse of dimensionality”, we introduce the additional input parameter𝑎 and we do not add a node if ∑|𝑐 − 𝑐′ | ≤ 𝑎 holds for some 𝑣’s child 𝑣′ suchthat key(𝑣′) = (𝑐′ , … , 𝑐′ ) at the line 4 in Procedure 3.3.
Here we present the entire CODE algorithm in Procedure 3.4, which achieves
multi-class classi ication from numerical data streams using the similarity de ined
above. Let ℒ = {𝑙 , 𝑙 , … , 𝑙 } be the domain of class labels. CODE makes 𝑚 trees
𝒯 , 𝒯 , … , 𝒯 for respective labels using Procedure 3.3 from labeled data and, if it
receives an unlabeled datum, it computes the similarity for each tree and classi ies
the datum to the class whose similarity is maximum.

3.6.2 Experiments
We experimentally evaluate CODE to determine its scalability and effectiveness.
Methods
We used Mac OS X version 10.6.5 with two 2.26-GHz Quad-Core Intel Xeon CPUs
and 12 GB of memory. CODE was implemented in C and compiled with gcc 4.2.1,
which can be used in R, version 2.12.2 (R Development Core Team, 2011).

3.6 OUTLOOK: DATA STREAM CLASSIFICATION ON TREES

Procedure 3.3: Construction of tree and calculation of the similarity
Input: Data point 𝑥 and tree 𝒯
Output: Updated tree 𝒯 or the similarity between 𝑥 and 𝒯
function T(𝑥, 𝒯)
1: 𝑘 ← 1, 𝑣 ← the root of 𝒯
2: repeat3:(𝑐 , … , 𝑐 ) ← (⌊𝑥 ⋅ 2 ⌋, … , ⌊𝑥 ⋅ 2 ⌋)4:
if a node 𝑣 has a child 𝑣′ with key(𝑣′) = (𝑐 , … , 𝑐 ) then5:
if 𝑥 is unlabeled then6:𝑠 ← 𝑠 + ∑{num(𝑢) ⋅ 𝑘 ⋅ 𝑑 ∣ 𝑢 is a brother of 𝑣}
7:end if8:
if a node 𝑣′ is a leaf then9:𝑦 ← org(𝑣′)10:
add a node 𝑣′′ as a child of 𝑣′,
where key(𝑣′′) = (⌊𝑦 ⋅ 2 ⌋, … , ⌊𝑦 ⋅ 2 ⌋) and org(𝑣′′) = 𝑦
11:
end if12:𝑣 ← 𝑣′
13:else
14:
if 𝑥 is labeled then15:
add a 𝑣’s child 𝑣′, where key(𝑣′) = (𝑐 , … , 𝑐 ) and org(𝑣′) = 𝑥
16:
return 𝒯
17:else if 𝑥 is unlabeled then18:
return (𝑠 + num(𝑣) ⋅ 𝑘 ⋅ 𝑑) / the number of leaf nodes of 𝒯
19:
end if20:
end if21:𝑘←𝑘+1
22: until foreverTo evaluate scalability with respect to the data size, synthetic data were generated randomly using the R clusterGeneration package (Qiu and Joe, 2006),
where the parameter sepVal was 0.35, 𝑑 = 5, and the number of data points 𝑛 =1, 000, 000. The number of classes was 5. We used 990, 000 data as labeled data
to evaluate scalability in constructing trees and the remaining 10, 000 data as unlabeled test data to check accuracy of classi ication. Moreover, to check effectivity
in classi ication, ive real datasets were collected from the UCI machine learning
repository (Frank and Asuncion, 2010): iris, waveform, vertebral, wine, and yeast,
and we obtained accuracy by ten-fold cross validation.
We adopted two tree-based methods as control: one is the tree method implemented in the R tree package as a typical tree-based batch classi ication algorithm,
and the other is VFDT (Domingos and Hulten, 2000) which is a representative fasttree-based classi ication algorithm for data streams. We used VFDT implemented
as a module of the VFML toolkit (Hulten and Domingos, 2003). Notice that, since
the tree method in R is a batch algorithm, we irst stored all data and applied themethod to them, whereas CODE and VFDT incrementally make (train) trees whenever labeled data arrive.

59

60

CODING DIVERGENCE

Procedure 3.4: CODE procedure
Input: Data stream 𝑥 , 𝑥 , 𝑥 , …
Output: Sequence of labels for each unlabeled datum
1: repeat2:
read 𝑥
3:
if 𝑥 is labeled then4:𝒯 ←T(𝑥 , 𝒯 ), where 𝑥 ’s label is 𝑙5:else if 𝑥 is unlabeled then6:
output argmax ∈ℒ T
(𝑥 , 𝒯 )7:
end if8:𝑖←𝑖+1
9: until forever

Results and Discussion
Figure 3.6 shows results; running time and accuracy of classi ication, for synthetic
data. Since the tree construction algorithm of CODE is simple, it is faster than thetree algorithm in R and VFDT, especially for large datasets. Moreover, accuracy of
CODE is higher than VFDT when the data size is small. These results indicate that
CODE is a scalable high-throughput algorithm and can be applied for classi icationof massive data stream.
In real UCI datasets, CODE is competitive compared to the tree method and
VFDT (Figure 3.7) and shows the best performance in two datasets (waveform andvertebral). Thus CODE should work ef iciently and effectively for various size ofdatasets including small and large datasets.
To summarize, CODE has been shown to be competitive compared to other algorithms in terms of scalability and effectiveness. We will continue to investigate
the presented approach. To treat more complex data streams, such as a stream
containing concept drift, is one of future works.

3.6 OUTLOOK: DATA STREAM CLASSIFICATION ON TREES

61

1.0
0.8
Accuracy

Running time (s)

15

10

5

0.6
0.4
CODE
Tree
VFDT

0.2
0.0

0
0

2×105 4×105 6×105 8×105 10×105
Number of data

0

2×105 4×105 6×105 8×105 10×105
Number of data

CODE
Tree
VFDT

1.0
0.8
Accuracy

Figure 3.6 | Experimental results forsynthetic data.

0.6
0.4
0.2
0.0
iris

waveform

vertebral

wine

yeast

Figure 3.7 | Experimental results forreal data. Data show mean ± s.e.m.

4
MINIMUM CODE LENGTH AND
GRAY CODE FOR CLUSTERING

C

compression
minimum description length (MDL)

Kolmogorov complexity

minimum code length (MCL)

embeddingis a fundamental task in data analysis, and many clustering algorithms have been developed in the ields of machine learning and knowledge
discovery (Berkhin, 2006; Halkidi et al., 2001; Jain et al., 1999a). Several clustering
algorithms have been recently proposed that focus on the compression of data.
Kontkanen et al. (2005) proposed the minimum description length (MDL) approach to clustering by taking advantage of an information theoretic framework.
However, data encoding has to be optimized to ind the best clusters; that is, allencoding schemes are considered within the clustering process under the MDL
criterion. As a result, their approach takes quadratic time with respect to the data
set size and can only be handled in practice using a stochastic algorithm (Kontkanen and Myllymäki, 2008). Cilibrasi and Vitányi (2005) proposed a clustering
algorithm based on the Kolmogorov complexity. Since their method measures the
distance between two data points on the basis of compression of inite sequences(i.e., discrete variables), it is dif icult to apply it to multivariate data of continuous
variables. Moreover, although there are other approaches (Keogh et al., 2007; Liet al., 2001) that focus on compression of data, they perform simple agglomerative
hierarchical clustering, so it takes quadratic time with respect to the data set size.
These approaches are therefore not suitable for clustering massive data sets.
Here we propose a new measure, called the minimum code length (MCL), toscore the quality of a given clustering result under a ixed encoding scheme. Thisuse of ixed encoding enables the performance of fast (i.e., linear complexity withrespect to the data set size) and exact clustering since we do not need to optimize
data encoding. We present a clustering algorithm, called COOL (COding-Oriented
cLustering), that always inds the best clusters; i.e., the globally optimal clusters
which minimizes MCL, and requires only the lower bounds for the number and sizeof the clusters. The discretization of continuous variables with the ixed encoding
scheme coincides with the clustering process itself — a hierarchy of clusters is
introduced automatically by increasing the accuracy of discretization.
Mathematically, an encoding, or embedding, is a mapping from real numbers toin inite sequences over some alphabet (Tsuiki, 2002), and discretization is realizedMINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING
Binary embedding

Gray-code embedding
Level-3000 001 011 010

00

01
0

0

0.25

00
1

0.5

0.75

01

11

0
1 0

0.25

Level-2

10

Level-1

1
0.5

0.75

1

Figure 4.1 | Examples of computing MCL with binary (left) or Gray code (right) embedding. These onedimensional data sets are in [0,1] and partitioned into three clusters, ○, ◊, and ∆. Level means the length ofeach preﬁx. With binary embedding, cluster ∆ is separated at level 1, and ○ and ◊ are separated at level 2. They
are encoded by 1, 00, and 01, respectively, so the MCL is 1 + 2 + 2 = 5. With Gray code embedding, the intervals
overlap, and adjacent clusters are merged at each level. As a result, ∆ is separated at level 2, and ○ and ◊ areseparated at level 3. Their representatives are {11,10}, {000,001}, and {011,010}, respectively, so the MCL is 4 + 6+ 6 = 16.
by truncation of in inite sequences. For example, in the binary embedding 𝜑 , every real number in [0, 1] is translated into an in inite sequence composed of 𝟶 and
𝟷; e.g., 𝜑 (0) = 𝟶𝟶𝟶 … , 𝜑 (0.2) = 𝟶𝟶𝟷 … , and 𝜑 (0.4) = 𝟶𝟷𝟷 … , where the irst
bit is 𝟶 if the value is in the interval [0, 0.5] and 𝟷 if in (0.5, 1]. If these sequences aretruncated at the irst bit, all of them become 𝟶, and hence they are considered as in
the same cluster since they cannot be distinguished. If they are then truncated at
the second bit, both 0 and 0.2 become 𝟶𝟶, and 0.4 becomes 𝟶𝟷. Thus, two clusters
are generated: 𝐶 = {0, 0.2} and 𝐶 = {0.4}. This means that representatives of
𝐶 and 𝐶 are 𝟶𝟶 and 𝟶𝟷, respectively. Finally, if they are truncated at the third bit,
0 and 0.2 are separated. The hierarchy is therefore constructed as {{0, 0.2, 0.4}},{{0, 0.2}, {0.4}}, and {{0}, {0.2}, {0.4}}.
The complexity of making clusters can be measured by the length of the cluster
representatives. In the above example, 2 + 2 = 4 for clusters {0, 0.2} and {0.4}(𝟶𝟶 and 𝟶𝟷), and 3 + 3 + 2 = 8 for {0}, {0.2}, and {0.4} (𝟶𝟶𝟶, 𝟶𝟶𝟷, and 𝟶𝟷). We
call these values the MCL since we cannot distinguish a pair of data points fromdifferent clusters if their truncated codes are shorter than the MCL.
Since COOL does not optimize an embedding scheme within the clustering process, the clustering result strongly depends on the embedding used. This meansthat we have to carefully choose an appropriate embedding for effective clustering.
In this chapter, we consider the Gray code as an embedding scheme for COOL — Gray code
resulting in an algorithm we call G-COOL. Gray code was originally developed forbinary encoding of natural numbers and has become especially important in conversion between analog and digital information (Knuth, 2005). From the geometrical point of view, Gray code embedding is the partitioning of each interval into
overlapping smaller intervals. This enables clusters with arbitrary shapes to befound, which cannot be done with binary embedding. There is theoretical support
for clustering by G-COOL as shown in Lemma 4.9 and Theorem 4.11. Figure 4.1
illustrates examples of computing the MCL with binary and Gray code embedding.
The motivation for using Gray code scheme comes from Computable Analy-

63

64

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING
sis (Weihrauch, 2000), a well-established mathematical framework for addressing analytical and computational aspects of real numbers through representationof real numbers as in inite sequences. Computability for real numbers depends
on the embedding method used, and computation makes sense only if the method
meets a key mathematical property: “admissibility” (see the book (Weihrauch,2000) for its mathematical de inition and properties). It is thus natural that theclustering results depends on the embedding method. Gray code has been shown
to be admissible (Tsuiki, 2002) whereas binary embedding is not, and this property is a key for embedding that can detect arbitrarily shaped clusters.
This chapter is organized as follows: Section 4.1 introduces the MCL, and Section 4.2 gives a formal de inition of clustering based on the MCL and explains theintegration of COOL with the computation of the MCL. In Section 4.3, we introduce
Gray code embedding and analyze G-COOL theoretically. Section 4.4 describes theexperiments, presents the results, and discusses them. Section 4.5 summarized
the key points with reviewing related work.
Related Work
Many types of shape-based clustering, or spatial clustering, methods have been
proposed for inding arbitrarily shaped clusters, including partitional algorithms(Chaoji et al., 2009, 2011), the mass-based clustering algorithm (Ting and Wells,2010), density-based clustering algorithms (e.g., DBSCAN (Ester et al., 1996) and
DENCLUE (Hinneburg and Keim, 1998)), agglomerative hierarchical clustering algorithms (e.g., CURE (Guha et al., 1998), CHAMELEON (Karypis et al., 1999)), andgrid-based algorithms (e.g., STING (Wang et al., 1997) and Wave Cluster (Sheikholeslami et al., 1998)). However, most of them are not practical. Their clusteringresults are sensitive to the input parameters, which have to be tuned manually, sothey work well only if all parameters are tuned appropriately by the user. As aresult, these methods are not well suited for users who are not specialized in machine learning. Furthermore, most of them are not scalable: their time complexity is quadratic or cubic with respect to data size. Compared to these methods,
G-COOL is robust to the input parameters and always inds the globally optimalclusters under the MCL criterion. Moreover, G-COOL is usually faster than most ofthese methods since the time complexity is linear with respect to data size.
Many cluster validity methods have been proposed for quantitative evaluationof clustering results (Handl et al., 2005). These measures are usually divided into
two categories: internal (e.g., connectivity and Silhouette width) and external (e.g.,
𝐹-measure and Rand index). The internal measures are intrinsic to actual clustering while the external measures need information that may not be available in an
actual situation. Our proposed measure, MCL, can be categorized as an internalmeasure. Its effectiveness has been shown experimentally (see Section 4.4).
NotationIn the following, ℝ denotes the 𝑑-dimensional Euclidean space. A data point 𝑥 isa vector in ℝ , and a dataset 𝑋 is a inite set of data points. For a pair of sets 𝑋 and
𝑌, 𝑋 ⧵ 𝑌 means the relative complement of 𝑌 in 𝑋.
Clustering is the partition of a dataset 𝑋 into 𝐾 subsets 𝐶 , … , 𝐶 , called clusters,
where 𝐶 ≠ ∅, 𝐶 ∩ 𝐶 = ∅ with 𝑖 ≠ 𝑗, and ⋃ ∈{ ,…, } 𝐶 = 𝑋. Here we say that a
set 𝒞 = {𝐶 , … , 𝐶 } holding the above properties is a partition of 𝑋 and denote the4.1 MINIMUM CODE LENGTH
set of all possible partitions by 𝒞(𝑋); i.e., 𝒞(𝑋) = {𝒞 ∣ 𝒞 is a partition of 𝑋}. For acluster 𝐶, #𝐶 denotes the number of data points in 𝐶.
∗
The set of inite and in inite sequences over an alphabet Σ is denoted by Σand Σ , respectively. The length |𝑤| of a inite or an in inite sequence 𝑤 is thenumber of positions for symbols other than ⊥ (the unde inedness character) in 𝑤.
For example, if 𝑤 = 𝟷𝟷⊥𝟷𝟶𝟶⊥⊥⊥ … , |𝑤| = 5. For a set of sequences 𝑊, the size of
𝑊 is de ined by |𝑊| ≔ ∑ ∈ |𝑤|.
An embedding of ℝ is an injection 𝜑 from ℝ to Σ (cf. Subsection 3.2.2). Fora pair of in inite sequences 𝑝, 𝑞, we write 𝑝 ≤ 𝑞 if 𝑝(𝑖) = 𝑞(𝑖) for all 𝑖 with 𝑝(𝑖) ≠ ⊥,where 𝑝(𝑖) denotes the 𝑖th position (including 0) of 𝑝. This means that 𝑞 is more∗
speci ic than 𝑝 since ⊥ denotes unde inedness. Moreover, if 𝑤⊥ ≤ 𝑝 for 𝑤 ∈ Σ ,we write 𝑤 ⊏ 𝑝 (𝑤 is a pre ix of 𝑝). Remember that ↑𝑤 = {𝑝 ∈ range(𝜑) ∣ 𝑤 ⊏ 𝑝}∗∗
for 𝑤 ∈ Σ and ↑𝑊 = {𝑝 ∈ range(𝜑) ∣ 𝑤 ⊏ 𝑝 for some 𝑤 ∈ 𝑊} for 𝑊 ⊆ Σ .

4.1

Minimum Code Length

The minimum code length, or MCL, is used to measure partitions under a ixed
embedding 𝜑. We de ine, for 𝑝 ∈ range(𝜑) and 𝑃 ⊂ range(𝜑),Φ(𝑝 ∣ 𝑃) ≔ 𝑤 ∈ Σ

𝑝 ∈ ↑𝑤, and 𝑃 ∩ ↑𝑣 = ∅for all 𝑣 with |𝑣| = |𝑤| and 𝑝 ∈ ↑𝑣∗.

Every element in Φ(𝑝 ∣ 𝑃) is a pre ix of 𝑝 that discriminates 𝑝 from 𝑃. Trivially,Φ(𝑝 ∣ 𝑃) = ∅ if 𝑝 ∈ 𝑃.
The MCL is introduced here in accordance with the above preparations.
De inition 4.1: MCL
Given an embedding 𝜑, for a partition 𝒞 = {𝐶 , … , 𝐶 } of a dataset 𝑋, we de ine
MCL(𝒞) ≔𝐿 (𝒞),∈{ ,…, }where
𝜑(𝐶 ) ⊆ ↑𝑊 and
𝐿 (𝒞) ≔ min|𝑊|Φ (𝜑(𝑥) ∣ 𝜑(𝑋 ⧵ 𝐶 ))

𝑊⊆

.

∈

Intuitively, this gives the code length of the maximally compressed representativesof a given partition through discretization using ixed embedding 𝜑 since the following property holds: For a partition 𝒞 of 𝑋, if we discretize each data point 𝑥 ∈ 𝑋
into a inite sequence 𝑐(𝑥) with 𝜑 (i.e., 𝑐(𝑥) ⊏ 𝜑(𝑥)) such that ⋃ ∈ 𝑐(𝑥) <MCL(𝒞), then there must exist a pair of data points 𝑥, 𝑦 ∈ 𝑋 satisfying ↑𝑐(𝑥) ∩↑𝑐(𝑦) ≠ ∅ and 𝑥 ∈ 𝐶 , 𝑦 ∈ 𝐶 with 𝑖 ≠ 𝑗. Therefore, we cannot discriminate 𝑥 from𝑦 and thus cannot ind the partition 𝒞 from compressed codes 𝑐(𝑋).
Example 4.2
Suppose we use binary embedding 𝜑 . Assume a one-dimensional dataset 𝑋 ={0.1, 0.2, 0.8, 0.9} and partitions 𝒞 = {{0.1, 0.2}, {0, 8, 0.9}} and 𝒞 = {{0.1},

65

66

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING{0.2, 0.8}, {0.9}}. Then, MCL(𝒞 ) = 𝐿 (𝒞 ) + 𝐿 (𝒞 ) = 1 + 1 = 2 since 𝜑 ([0, 0.5])= ↑𝟶 and 𝜑 ((0.5, 1]) = ↑𝟷, and MCL(𝒞 ) = 𝐿 (𝒞 ) + 𝐿 (𝒞 ) + 𝐿 (𝒞 ) = 3 + (3 +3) + 3 = 12 because we have 𝜑 ([0, 0.125]) = ↑𝟶𝟶𝟶, 𝜑 ((0.125, 0.25]) = ↑𝟶𝟶𝟷,
𝜑 ((0.75, 0.875]) = ↑𝟷𝟷𝟶, and 𝜑 ((0.875, 1]) = ↑𝟷𝟷𝟷. Note that 𝜑 ([0, 0.25]) =↑𝟶𝟶, hence 0.1 and 0.2 cannot be discriminated using code 𝟶𝟶, and that 𝜑 ((0.75,1]) = ↑𝟷𝟷, hence 0.8 and 0.9 cannot be discriminated using 𝟷𝟷.
The MCL is calculated for 𝑂(𝑛𝑑) time complexity by using a radix sort, where 𝑛 is
the size of 𝑋 (i.e., 𝑛 = #𝑋), and 𝑑 is the dimension of 𝑋. This is why if the discretizeddataset {𝑝(0)𝑝(1) … 𝑝(𝑘 − 1) ∣ 𝑝 ∈ 𝜑(𝑋)} at level 𝑘 is sorted in advance, each data
point simply needs to be compared with the subsequent one for each dimension,and the MCL is obtained by checking from 𝑘 = 1, 2, 3, … .

4.2 Minimizing MCL and Clustering
We formulate clustering using the MCL as a criterion and describe clustering algorithm COOL, which inds the globally optimal partition that minimizes the MCL.

4.2.1 Problem Formulation
The clustering problem with the MCL is de ined as follows.
De inition 4.3: Clustering under the MCL criterion
Clustering of a dataset 𝑋 under the MCL criterion means inding the globally optimal partition that minimizes the MCL with more than 𝐾 clusters; that is, inding
𝒞 such that𝒞

∈ argmin MCL(𝒞),𝒞∈𝒞( )

where 𝒞(𝑋)

= {𝒞 ∈ 𝒞(𝑋) ∣ #𝒞 ≥ 𝐾}.

In this framework, we assume that a lower bound on the number of clusters 𝐾 isgiven to avoid overgeneralization since, if we search for the optimal partition 𝒞 in
𝒞(𝑋) (i.e., all possible partitions) instead of 𝒞(𝑋) , we always have the nonsense
result 𝒞 = {𝑋}.

4.2.2 COOL Algorithm
Our COOL algorithm ef iciently solves the optimization problem (De inition 4.3) byintegrating the computation of MCL within the clustering step. By contrast, naïve
approach that would compare the MCLs of all possible partitions would result in analgorithm with exponential time complexity. The pseudo-code of COOL is shownin Algorithm 4.1.
COOL is a level-wise clustering algorithm that inds the optimal partition 𝒞
by enumerating level-𝑘 partitions (𝑘 = 1, 2, 3, … ).

level-𝑘 partition

De inition 4.4: Level-k partition
For a dataset 𝑋 and an embedding 𝜑, the level-𝑘 partition 𝒞 is de ined as follows: Every pair of data points 𝑥, 𝑦 ∈ 𝑋 are contained in the same cluster if andonly if 𝑣 = 𝑤 for some 𝑣 ⊏ 𝜑(𝑥) and 𝑤 ⊏ 𝜑(𝑦) with |𝑣| = |𝑤| = 𝑘.

4.2 MINIMIZING MCL AND CLUSTERING

Algorithm 4.1: COOL algorithm
Input: Dataset 𝑋, lower bound on the cluster size 𝐾, and noise parameter 𝑁
Output: Optimal partition 𝒞 and noise data
Function C(𝑋, 𝐾, 𝑁)
1: Find partitions 𝒞 , … , 𝒞
2: (𝒞 , MCL) ← F C
3: return (𝒞 , 𝑋 ⧵ ⋃ 𝒞 )such that #𝒞< 𝐾 ≤ #𝒞
(𝑋, 𝐾, {𝒞 , … , 𝒞 })Function F C(𝑋, 𝐾, {𝒞 , … , 𝒞 })1: if 𝐾 = 1 then2:
return (𝒞 , |𝑊|), where 𝜑(⋃ 𝒞 ) ⊆ ↑𝑊 and |𝑤| = 𝑙 for all 𝑤 ∈ 𝑊
3: end if
4: Find 𝑘 such that #𝒞< 𝐾 ≤ #𝒞
5: 𝒞 ← 𝒞
6: MCL ← MCL(𝒞 )7: for each 𝐶 in 𝒞 ∪ … ∪ 𝒞
8:𝐿 ← min { |𝑊| ∣ 𝜑(𝐶) ⊆ ↑𝑊 and |𝑤| = 𝑗 for all 𝑤 ∈ 𝑊 }9:(𝒞, 𝐿′) ← F C(𝑋 ⧵ 𝐶, 𝐾 − 1, {𝒞 , … , 𝒞 })10:
if 𝐿 + 𝐿′ < MCL then11:
𝒞 ←𝐶∪𝒞12:
MCL ← 𝐿 + 𝐿′
13:
end if
14: end for
15: return (𝒞 , MCL)This means that if 𝑥, 𝑦 ∈ 𝑋 are in the same cluster, there exists a chain of data
points 𝑧 , 𝑧 , … , 𝑧 (𝑚 ≥ 2) such that, for all 𝑖 ∈ {1, 2, … , 𝑚 − 1}, 𝑧 = 𝑥, 𝑧 = 𝑦,and 𝑤 = 𝑤
for some 𝑤 ⊏ 𝜑(𝑧 ) and 𝑤⊏ 𝜑(𝑧 ) with |𝑤 | = |𝑤 | = 𝑘.
Obviously, the level-𝑘 partition is determined uniquely. The time complexity ofinding the level-𝑘 partition is 𝑂(𝑛𝑑), where 𝑛 and 𝑑 are the size and dimensionof the dataset, respectively, since, if the discretized dataset {𝑝 𝑝 … 𝑝∣ 𝑝 ∈𝜑(𝑋)} at level 𝑘 is initially sorted using a radix sort, clusters are constructed bycomparing each data point to the next data point for each dimension.
The most important feature of the level-𝑘 partition is that the optimal partition
𝒞 in De inition 4.3 is obtained by searching for only clusters contained in thelevel-𝑘 partition.
Lemma 4.5
For every cluster 𝐶 ∈ 𝒞 , 𝐶 is contained in some level-𝑘 partition, that is, 𝐶 ∈ 𝒞
for some 𝑘 ∈ ℕ.
Proof. Let 𝒞 be a partition such that, for every 𝐶 ∈ 𝒞, 𝐶 ∈ 𝒞 for some 𝑘, and a pairof clusters 𝐶, 𝐶′ ∈ 𝒞 is ixed. Then, from the de inition of the level-𝑘 partition, thefollowing condition holds: For all pairs of clusters 𝐷, 𝐷′ such that 𝐷 ∪ 𝐷′ = 𝐶 ∪ 𝐶′
and 𝐷 ∩ 𝐷′ = ∅, we have MCL(𝒞) ≤ MCL(𝒞′), where 𝒞′ = (𝒞 ⧵ {𝐶, 𝐶′}) ∪ {𝐷, 𝐷′}.
Therefore, for the optimal partition 𝒞 , 𝐶 ∈ 𝒞 with 𝑘 ∈ ℕ for all 𝐶 ∈ 𝒞 .

67

68

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERINGThe level-𝑘 partition has a hierarchical structure: For each cluster 𝐶 ∈ 𝒞 ,there must exist a set of clusters 𝒟 ⊆ 𝒞such that ⋃ 𝒟 = 𝐶. Thus, COOL works
through divisive hierarchical clustering. The MCL of the level-𝑘 partition used inline 6 of the function FINDCLUSTERS in Algorithm 4.1 can thus be easily calculated:
Let 𝒞 be a set of clusters {𝐶 , … , 𝐶 }. For each 𝐶 and for the minimum level 𝑙such that 𝐶 ∈ 𝒞 ,
𝐿 (𝒞 ) = min{|𝑊| ∣ 𝜑(𝐶 ) ⊆ ↑𝑊 and |𝑤| = 𝑙 for all 𝑤 ∈ 𝑊}
holds. This means that we can obtain the MCL of the level-𝑘 partition by checking
only sequences with length 𝑙.
Next we show that COOL can solve the optimization problem in De inition 4.3.
Proposition 4.6
The COOL algorithm (Algorithm 4.1) always outputs the globally optimal partition 𝒞 .
Proof. Let #𝒞 = 𝐾. Then there must exist 𝑘 ∈ ℕ such that 𝐾 ≤ #𝒞 and #𝒞 <𝐾 for all 𝑘′ < 𝑘 since the number of clusters in the level-𝑘 partition #𝒞 increases
monotonically with respect to increase of 𝑘. Fix a cluster 𝐶 ∈ 𝒞 , and let 𝒞′ be
the optimal partition for the dataset 𝑋 ⧵ 𝐶. Then we can easily check that 𝒞′ ∪ {𝐶}coincides with 𝒞 . Moreover, from Lemma 4.5 and the de inition of the level-𝑘
partition, for all 𝐶 ∈ 𝒞 , 𝐶 ∈ 𝒞 for some 𝑙 ∈ {1, … , 𝑘}. Thus, COOL inds the
optimal partition 𝒞 by recursive computing in Algorithm 4.1 (lines 4 - 7) withixing each cluster in 𝒞 ∪ … ∪ 𝒞 .

noise parameter

COOL can ind the globally optimal partition 𝒞 ef iciently, and its time complexity is 𝑂(𝑛𝑑) and 𝑂(𝑛𝑑 + 𝐾! ) in the best and worst cases, respectively, sinceinding 𝑚 partitions in the irst line of the function COOL takes 𝑂(𝑛𝑑), and the function FINDCLUSTERS takes 𝑂(𝐾! ) in the worst case. Usually, 𝐾 ≪ 𝑛 holds, so complexity becomes 𝑂(𝑛𝑑).
Furthermore, noise is directly removed by COOL using a lower bound on thesize of each cluster 𝑁, which we call the noise parameter. For a partition 𝒞, we denote the set {𝐶 ∈ 𝒞 ∣ #𝐶 ≥ 𝑁} by 𝒞 . For example, let a dataset 𝑋 = {0.1, 0.4, 0.5,
0.6, 0.9} and 𝒞 = {{0.1}, {0.4, 0.5, 0.6}, {0.9}}. Then, 𝒞 = {{0.4, 0.5, 0.6}}, and two
data points, 0.1 and 0.9, are detected as noise.

4.3 G-COOL: COOL with Gray CodeWe use Gray code embedding for COOL, and show its powerful clustering abilityby theoretical analysis. We call COOL with Gray code embedding G-COOL.

4.3.1 Gray Code Embedding
Gray code embedding is illustrated in Figure 4.2. Its rich mathematical propertiesare described elsewhere (Tsuiki, 2002). Gray code was originally simply binary
encoding of natural numbers, as mentioned in introduction. For example, natural
numbers 1, 2, … , 8 are represented in Gray code as 000, 001, 011, 010, 110, 111,
101, 100, whereas, in binary code, they are represented as 000, 001, 010, 011, 100,

4.3 G-COOL: COOL WITH GRAY CODE

Position

5
3
2
1

0
0 γG(0.25) = 0⊥1000... 0.5

1

Figure 4.2 | Gray code embedding
φG . Position i is 1 if it is on the line,⊥ if on the end point, and 0 otherwise. Diagonal lines are auxiliary
lines. For example, if p = φG (0.25),
p = 0⊥1000… because position 0 is
not on the line, 1 is on the end point,
2 is on the line, and every i ≥ 3 is noton the line.

101, 110, 111. The importance of Gray code is that only one bit differs between
one code and its successor, that is, the Hamming distance between them is always
one. Here ℐ denotes the unit interval [0, 1] × … × [0, 1] ⊂ ℝ , and Σ , denotes
the set of in inite sequences for which, in each sequence, at most 𝑑 positions are⊥. For example, if Σ = {𝟶, 𝟷} and 𝑑 = 2, then 𝟶⊥𝟷𝟶𝟶 … ∈ Σ , , ⊥⊥𝟷𝟷𝟶 … ∈ Σ , ,and 𝟶⊥𝟷⊥⊥𝟶 … ∉ Σ , . In the following, we consider only real vectors in ℐ.
De inition 4.7: Gray code embedding(One-dimensional) Gray code embedding is an injective function, 𝜑 ∶ ℐ → Σ , Gray code embedding(𝑑 = 1), that maps 𝑥 ∈ ℐ to an in inite sequence 𝑝 𝑝 𝑝 … : For each 𝑖, 𝑝 ≔ 𝟷 if
2 𝑚−2

(

)

<𝑥<2 𝑚+2()

holds for an odd number 𝑚, 𝑝 ≔ 𝟶 if the same holds for an even number 𝑚,()
and 𝑝 ≔ ⊥ if 𝑥 = 2 𝑚 − 2
for some integer 𝑚.
Moreover, by using the tupling function (see Equation (2.4)), we can de ine 𝑑dimensional Gray code embedding 𝜑 ∶ ℐ → Σ , as
𝜑 (𝑥 , … , 𝑥 ) ≔ ⟨𝜑 (𝑥 ), … , 𝜑 (𝑥 )⟩.
We abbreviate 𝑑 of 𝜑 if it is understood from the context.
Example 4.8
For one-dimensional data points 𝑥 = 0.2, 𝑦 = 0.5, and 𝑧 = 0.7, we have 𝜑 (𝑥) =𝟶𝟶𝟷𝟶 … , 𝜑 (𝑦) = ⊥𝟷𝟶𝟶 … , and 𝜑 (𝑧) = 𝟷𝟷𝟷𝟶 … with the Gray code embedding, while 𝜑 (𝑥) = 𝟶𝟶𝟶𝟷 … , 𝜑 (𝑦) = 𝟶𝟷𝟷𝟷 … , and 𝜑 (𝑧) = 𝟷𝟶𝟷𝟷 … with thebinary embedding. For a two-dimensional data point (𝑥, 𝑦), we have 𝜑 (𝑥, 𝑦) =𝟶⊥𝟶𝟷𝟷𝟶𝟶𝟶 … , and for a three-dimensional data point (𝑥, 𝑦, 𝑧), 𝜑 (𝑥, 𝑦, 𝑧) = 𝟶⊥𝟷𝟶𝟷𝟷𝟷𝟶𝟷𝟶𝟶𝟶 … with Gray code embedding.

4.3.2 Theoretical Analysis of G-COOL
Here we show that G-COOL achieves internal cohesion and external isolation without any distance calculation or data distribution. In the following, we measure thedistance between 𝑥, 𝑦 ∈ ℝ by the 𝐿 metric, where the distance is de ined by𝑑 (𝑥, 𝑦) ≔ max∈{ ,…, }𝑥 −𝑦 .

69

70

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING
id

Value

a
b
c
d
e

Level 1

0.14
0.48
0.51
0.73
0.77

Binary embedding

Level 2BinaryGray

Binary

Gray

𝟶
𝟶
𝟷
𝟷
𝟷

𝟶
𝟶, ⊥𝟷
𝟷, ⊥𝟷
𝟷, ⊥𝟷
𝟷

𝟶𝟶
𝟶𝟷
𝟷𝟶
𝟷𝟶
𝟷𝟷

𝟶𝟶
𝟶𝟷, ⊥𝟷𝟶
𝟷𝟷, ⊥𝟷𝟶
𝟷𝟷, 𝟷⊥𝟷
𝟷𝟶, 𝟷⊥𝟷

Gray-code embedding
Level-2 partition

0⊥1
00

01

10

11

00

⊥10

01

1⊥1

11

10
Level-1 partition

Figure 4.3 | Examples of level-1 and
2 partitions with binary and Gray code
embedding.

⊥1
0
0

0.25

1
0.5

0.75

0
1 0

0.25

1
0.5

0.75

1

Lemma 4.9
For the level-𝑘 partition 𝒞 of a dataset 𝑋 with Gray code embedding 𝜑 , two()
data points 𝑥, 𝑦 ∈ 𝑋 are in the same cluster if 𝑑 (𝑥, 𝑦) < 2
and are not in()
the same cluster only if 𝑑 (𝑥, 𝑦) ≥ 2.
()

Proof. From the de inition of Gray code embedding, if 𝑑 (𝑥, 𝑦) < 2
for 𝑥, 𝑦 ∈𝑋, there must exist a inite sequence 𝑤 with |𝑤| = 𝑘 such that 𝑤 ⊏ 𝜑 (𝑥) and𝑤 ⊏ 𝜑 (𝑦). Thus, 𝑥 and 𝑦 are in the same cluster in the level-𝑘 partition 𝒞 . This()
means that, if 𝑥 and 𝑦 are in the different clusters in 𝒞 , 𝑑 (𝑥, 𝑦) ≥ 2.
Informally, the redundancy of Gray code embedding enables the powerful property
described in the above lemma, that is, for an in inite sequence 𝑝 = 𝜑 (𝑥), there
may be two pre ixes, 𝑣 ⊏ 𝑝 and 𝑣 ⊏ 𝑝 with |𝑣| = |𝑤|.
Example 4.10
Let us consider the situation illustrated in Figure 4.3, where we have ive data
points: 𝑥a = 0.14, 𝑥b = 0.48, 𝑥c = 0.51, 𝑥d = 0.73, and 𝑥e = 0.77. In binary embedding, the unit interval ℐ = [0, 1] is divided into two intervals [0, 0.5] and [0.5, 1]
at level-1, while it is divided into three intervals [0, 0.5], [0.25, 0.75], and [0.5, 1] inGray code embedding. Thus, there are three overlapping clusters {𝑥a , 𝑥b } (encoded
as 𝟶), {𝑥b , 𝑥c , 𝑥d } (encoded as ⊥𝟷), and {𝑥c , 𝑥d , 𝑥e } (encoded as 𝟷). Actually, there is
only one cluster {𝑥a , 𝑥b , 𝑥c , 𝑥d , 𝑥e } since they are merged. At level-2, we have fourclusters with binary embedding although some data points such as 𝑥b and 𝑥c areclose. On the other hand, we have two natural clusters {𝑥a } and {𝑥b , 𝑥c , 𝑥d , 𝑥e } with
Gray code embedding.
Intuitively, this lemma theoretically supports the claim that G-COOL inds natural clusters. For a data point 𝑥 ∈ 𝑋, we say that a data point 𝑦 ∈ 𝑋 is the nearest4.4 EXPERIMENTS
G-COOL
1.0

COOL with binary embedding
1.0

0.5

0.5

0

K-means
1.0

0.5

0
0

0.5

1.0

0
0

0.5

1.0

0

0.5

1.0

neighbor of 𝑥 if 𝑦 ∈ argmin ∈ 𝑑 (𝑥, 𝑥′).
Theorem 4.11
The optimal partition 𝒞 of a dataset 𝑋 generated by G-COOL has the following
property: For every data point 𝑥 ∈ 𝐶 with 𝐶 ∈ 𝒞 and #𝐶 ≥ 2, its nearest
neighbor 𝑦 ∈ 𝐶.
Proof. From Lemma 4.5, every cluster 𝐶 ∈ 𝒞 is contained in 𝒞 for some 𝑘. Thus,
from Lemma 4.9, any 𝑥 ∈ 𝐶 with 𝐶 ∈ 𝒞 , #𝐶 = 1 or its nearest neighbor 𝑦 ∈ 𝐶.
This property of Gray code (Lemma 4.9) enables clusters with the condition in
Theorem 4.11 to be quickly found, whereas the naïve solution results in more than𝑂(𝑛 ). Figure 4.4 illustrates the results of G-COOL for a two-dimensional dataset
for which 𝐾-means could not ind natural clusters. We can see that COOL withbinary embedding also failed to ind such clusters.

4.4

71

Experiments

We empirically evaluate the effectiveness of G-COOL and the proposed measure,
MCL. We use low-dimensional synthetic and real datasets, which are common in
spatial clustering setting.

4.4.1 Methods
Environment
G-COOL was implemented in R version 2.12.2 (R Development Core Team, 2011),
and all experiments were performed in R. We used Mac OS X version 10.6.5 withtwo 2.26-GHz Quad-Core Intel Xeon CPUs and 12 GB of memory.
Datasets
The synthetic datasets were used to evaluate robustness against the number ofclusters, the size of the datasets, and noise existence. They were randomly generated using the R clusterGeneration package (Qiu and Joe, 2006), and the parameters were set as follows: sepVal = 0.34, numNonNoisy = 2, numNoisy = 1,
numOutlier = 500, and rangeN = 𝑐(1000, 2000). We generated 20 datasets toobtain the mean and s.e.m. (standard error of the mean). The size of each cluster

Figure 4.4 | Clustering results for GCOOL and COOL with binary embedding (K = 2, N = 50) and K-means (K =2). Dataset size is 10,500 (where 500
points are noise). G-COOL detects twonatural clusters. The other two methods cannot ﬁnd such clusters.

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING

Connectivity

MCL

5000
500

Running time (s)Adjusted Rand index

1.0
0.9
0.6
0.3

2
4
6
Number of clusters

100
10
1
0.1

2
4
6
Number of clusters

Figure 4.5 | Experimental results forsynthetic datasets. MCL and connectivity should be minimized, and Silhouette width and adjusted Rand index should be maximized. Data show
mean ± s.e.m.

0.6

1000

50000

Silhouette width

72

2
4
6
Number of clusters

20

0.5
0.4
0.3
2
4
6
Number of clusters
G-COOLDBSCANK-means

15
10
5
0

2
4
6
Number of clusters

was around 1,500, so the size of the datasets varied from ∼3,000 to ∼10,500. Eachdataset was three-dimensional, where one dimension was composed of noise andabout 500 data point were added to each dimension as noise.
Five real datasets were collected from the Earth-as-Art website¹, which contains geospatial satellite images (see Table 4.1 and Figure 4.7). Similar datasets
were used in experiments with the state-of-the-art study (Chaoji et al., 2011). Each
image was pre-processed using ImageJ software (Rasband, 1997–2011); they werereduced to 200 × 200 pixels and translated into binary images.
With G-COOL, each dataset was translated using min-max normalization (Hanand Kamber, 2006) so that the dataset was in the unit interval ℐ, where each value
𝑥 of 𝑖th dimension 𝑋 of a dataset 𝑋 was mapped to 𝑥′ = (𝑥 − min 𝑋 )/(max 𝑋 −min 𝑋 ) and the runtime for the translation was included in the G-COOL running
time.
Control Methods
As control methods, we used 𝐾-means and DBSCAN because 𝐾-means is the standard clustering algorithm and DBSCAN is a typical method for inding arbitrarily
shaped clusters, and their source codes are publicly available. DBSCAN was executed using the R fpc package. We tuned the parameters of DBSCAN to obtain thebest results.
Evaluation
With the synthetic datasets, performance was evaluated using internal and external measures. As internal measures, we used the MCL (with Gray code), the connectivity (takes values in [0, ∞], to be minimized) (Handl et al., 2005), and the Silhouette width (takes values in [−1, 1], to be maximized) (Rousseeuw, 1987). As
an external measure, we used the adjusted Rand index (takes values in [−1, 1], to¹http://eros.usgs.gov/imagegallery/

0.6
0.4
0.2
0

0 2 4 6 8 10
The noise parameter N

40
20
0

0 2 4 6 8 10
The noise parameter N
1.0
0.8

60

0 2 4 6 8 10
The noise parameter N
2.0

Silhouette width

80
Connectivity

5000
4000
3000
2000
1000
0

Running time (s)Adjusted Rand index

MCL

4.4 EXPERIMENTS

73

0.6
0.4
0.2
0

0 2 4 6 8 10
The noise parameter N

1.5
1.0
0.5
0

0 2 4 6 8 10
The noise parameter N

be maximized) (Hubert and Arabie, 1985), which takes into account the groundtruth and is popular in the clustering literature. The measures were calculated using the R clValid (Brock et al., 2008), cluster (Maechler et al., 2005), and clues(Chang et al., 2010) packages, respectively. For the real datasets, we used the MCL
and simply show scatter plots of the results since we had no information on the
ground truth.

4.4.2 Results and Discussion
The results obtained with the synthetic datasets (Figure 4.5) show that the qualityof clusters obtained with G-COOL is signi icantly higher for three of the four quality measures (determined by paired 𝑡-test) and is competitive for the other one(adjusted Rand index). Moreover, it is faster than DBSCAN. These results showthat the MCL works reasonably well as a measure of cluster quality compared toexisting ones.
Note that we need to input only the lower bounds for the number and size ofthe clusters in G-COOL, whereas we have to tune the parameters carefully in DBSCAN and other shape-based (spatial) clustering algorithms. Therefore, G-COOL
is more ef icient and effective than existing clustering algorithms. Moreover, as
shown in Figure 4.6, cluster quality is stable with respect to the noise parameter 𝑁 (i.e., lower bound on cluster size) even if the dataset contains noise, when
𝑁 is large enough. If 𝑁 is too small, then each noise is detected as a cluster. Thus,
when clustering using G-COOL, all we have to do is set the parameter large enough,meaning that G-COOL is equally useful as 𝐾-means.
For all the real datasets, G-COOL inds natural clusters (𝑁 was set as 50 for all
the datasets), as shown in Figure 4.7, whereas 𝐾-means results in inferior clustering quality (we did not perform DBSCAN since it takes too much time and needs
manual tuning of the input parameters). Moreover, MCL of clustering results for
G-COOL is much smaller than those for 𝐾-means (Table 4.1). These results showthat G-COOL is robust and that it can ind arbitrarily shaped clusters without careful tuning of the input parameters.

Figure 4.6 | Clustering speed andquality for G-COOL and synthetic datasets. MCL and connectivity should be
minimized, and Silhouette width andadjusted Rand index should be maximized. G-COOL shows robust performance if the noise parameter N islarge enough. Data show mean ±s.e.m.

74

MINIMUM CODE LENGTH AND GRAY CODE FOR CLUSTERING(a) Delta(b) Binary image(c) G-COOL(d) -means(e) Europe(f) Binary image(g) G-COOL(h) -means(i) Norway(j) Binary image(k) G-COOL(l) -means(m) Ganges(n) Binary image(o) G-COOL(p) -means(q) Dragon(r) Binary image(s) G-COOL(t) -means

Figure 4.7 | Results for real datasets obtained from satellite images by G-COOL and K-means. G-COOL ﬁnds all
natural clusters whereas K-means does not.

4.5 SUMMARY
Name
Delta
DragonEuropeNorwayGanges

4.5

𝑛
20748
29826
17380
22771
18019

𝐾
4
2
6
5
6

Running time (s)

75

MCL

G-COOL𝐾-means

G-COOL𝐾-means

1.158
0.595
2.404
0.746
0.595

0.012
0.026
0.041
0.026
0.026

4010
3906
2320
1820
2320

4922
7166
12210
6114
12526

Summary

We have proposed an internal measure, the minimum code length (MCL), to evaluate the results of clustering and presented a clustering algorithm COOL that always
inds the globally optimal partition; i.e., clusters that have the minimum MCL. Intuitively, COOL produces the maximally compressed clusters using a ixed encoding
scheme and does not take optimization of encoding into account. Moreover, Graycode is used for the encoding, resulting in an algorithm called G-COOL. Theoretically and empirically, G-COOL has been shown to be noise tolerant and to have theability to ind arbitrarily shaped clusters ef iciently. The result is an effective solution to two essential problems, how to measure the goodness of clustering results
and how to ind good clusters.
G-COOL’s results are robust to changes in the input parameters, and does notassume a data distribution and does not need a distance calculation. Thus, it canbe effectively applied to other machine learning tasks, such as anomaly detection.
Theoretical analysis of relationship between admissibility of encoding schemes incomputing real numbers and the ability of clustering to detect arbitrarily shaped
clusters is necessary future work.

Table 4.1 | Running time (in seconds)
and MCL for real datasets. In table,n and K denote the number of data
points and clusters, respectively. Clustering results are shown in Figure 4.7.

5
CLUSTERING USINGBINARY DISCRETIZATION

T
spatial clustering
shape-based clustering

K(MacQueen, 1967) is widely used due to its simplicity
and ef iciency while it is one of the earliest clustering algorithms. The maindrawback is its limited clustering ability; that is, non-convex clusters cannot befound and noise is not iltered since the separation of data is based on a Voronoi
diagram. To ind clusters with arbitrary shapes, many spatial clustering, or shapebased clustering, algorithms have been proposed (Han et al., 2001) (see Section 5.4
for related work). One of major applications of such algorithms is the identi ication of similar regions from geographical databases. However, most of them are
not scalable. Their time complexity is quadratic or even cubic with respect to thenumber of data points. Moreover, the clustering results are sensitive to the input
parameters, which have to be tuned manually, and they detect reasonable clusters
only if all parameters are tuned appropriately.
Ideally, a clustering algorithm should be• fast, applicable to massive data sets with a complexity that scales linearly with
its size;• robust in the sense that its does not require too much parameter tuning inorder to produce meaningful results;• adaptive, be able to detect non-convex clusters.

binary discretization

The contribution of this chapter is to present a spatial clustering algorithm,
called BOOL (Binary cOding Oriented cLustering), that meets all the above threerequirements. The ef iciency and effectiveness of our BOOL algorithm are shown
experimentally (see Section 5.3). The key idea is to translate input data onto binary representations using binary discretization and use such binary codes to sortand merge data points for high-speed large scale clustering. A hierarchy of clusters is naturally produced by increasing the precision of the discretization. Eachhierarchy level is constructed in only two steps: discretization of each numerical
data point and agglomeration of adjacent smaller clusters.
The clustering procedure of BOOL is illustrated in Figure 5.1. We assume thatevery data point is in a two-dimensional space [0, 1] × [0, 1]. First, data points

5.1 CLUSTERING STRATEGY
Original data
𝑥
𝑥
𝑥
𝑥
𝑥

2

1

2

1

2

0.15
0.66
0.72
0.48
0.74

0.32
0.71
0.86
0.89
0.48

0
1
1
0
1

0
1
1
1
0

0
2
2
1
2

1
2
3
3
1

Discretization at level 1
x3
x2

x1

Level 2

1

Original datax4

Level 1

x5

x4

1
0

x1

Discretization at level 2

x3

3

x2

2

x5

1

x4

x3
x2
x5

x1

0
0

1

0

1

2

3

are discretized at level 1; i.e., each value 𝑥(𝑖) (𝑖 = 1 or 2) is discretized to 𝟶 if
0 ≤ 𝑥(𝑖) ≤ 0.5 and 𝟷 if 0.5 < 𝑥(𝑖) ≤ 1. For instance, two data points (0.15, 0.32)
and (0.48, 0.89) are discretized to (𝟶, 𝟶) and (𝟶, 𝟷), respectively. Next, clusters
using the following de inition are constructed; two data points are considered to
be in the same cluster if and only if the number of positions where two values differ
is at most one and the difference in the values is less than 𝑙 (the precise conditionis given in De inition 5.2 ). Intuitively, if 𝑙 = 1, the two values lie in the same squareor the adjacent squares from the geometrical point of view. Thus, at this level, alldata are in the same cluster for all 𝑙 ≥ 1. Next, all data points are discretized at
level 2; each value 𝑥(𝑖) is discretized to 𝟶 if 0 ≤ 𝑥(𝑖) ≤ 0.25, 𝟷 if 0.25 < 𝑥(𝑖) ≤ 0.5,
𝟸 if 0.5 < 𝑥(𝑖) ≤ 0.75, and 𝟹 if 0.75 < 𝑥(𝑖) ≤ 1. If 𝑙 = 1, two clusters are found: adata point 𝑥 is in one cluster, and 𝑥 , 𝑥 , 𝑥 , and 𝑥 are in the other cluster.
Due to the naïve approach of this clustering process, the time complexity isquadratic with respect to the number of data points (see Section 5.1) since eachpoint has to be compared to all the other ones. However, if the database is sorted
in advance, each data point simply needs to be compared with the subsequent one,so clustering can be performed with linear complexity (see Section 5.2).
This chapter is organized as follows: Section 5.1 introduces the naïve versionof BOOL to clearly explain the clustering process and analyze the relationship between BOOL and DBSCAN (Ester et al., 1996), and Section 5.2 discusses the speeding up by using sorting. The experimental results are presented and discussed
in Section 5.3. Related works are reviewed in Section 5.4, and the key points are
summarized in Section 5.5.

5.1

77

Clustering Strategy

In this section, we formulate the clustering problem and describe the naïve versionof our BOOL algorithm to enable the reader to get a clear understanding of the

Figure 5.1 | Example of clustering using BOOL.

78

CLUSTERING USING BINARY DISCRETIZATION

Algorithm 5.1: Naïve BOOL
Input: Database 𝜏 = (𝐻, 𝑋), lower bound on number of clusters 𝐾,
noise parameter 𝑁, and distance parameter 𝑙
Output: Partition 𝒞
function N
B(𝜏, 𝐾, 𝑁)
1: 𝑘 ← 1 // 𝑘 is level of discretization
2: repeat3:
𝒞 ← { {𝑥} ∣ 𝑥 ∈ set(𝑋) }4:for each object 𝑥 ∈ set(𝑋)5:for each object 𝑦 ∈ set(𝑋)6:
if 𝑑 (Δ (𝑥), Δ (𝑦)) ≤ 1 and 𝑑 (Δ (𝑥), Δ (𝑦)) ≤ 𝑙 then7:
delete clusters 𝐶 ∋ 𝑥 and 𝐷 ∋ 𝑦 from 𝒞, and add 𝐶 ∪ 𝐷
8:end if
9:end for10:
end for11:𝒞 ← { 𝐶 ∈ 𝒞 ∣ #𝐶 ≥ 𝑁 } // noise iltering
12:
𝑘←𝑘+1
13: until #𝒞 ≥ 𝐾
14: output 𝒞

clustering process. Note that results obtained with the naïve version and with the
speeded-up version are exactly the same — time complexity is the only difference
between them. The pseudo-code of naïve BOOL is shown in Algorithm 5.1.

5.1.1 Formulation of Databases and Clustering
We treat the target data set as a table, or a relation (Date, 2003; Garcia-Molina
et al., 2008; Simovici and Djeraba, 2008); 𝜏 which is a pair (𝐻, 𝑋) of a header 𝐻 anda body 𝑋. A header 𝐻 is a inite set of attributes, where each ℎ ∈ 𝐻 is referred toas the domain of ℎ, denoted by Dom(ℎ). A body 𝑋 is a sequence of tuples, or data
points, 𝑥 , 𝑥 , … , 𝑥 , where each tuple 𝑥 is de ined as a total function from 𝐻 to
Dom(𝐻) = {Dom(ℎ) ∣ ℎ ∈ 𝐻} such that 𝑥 (ℎ) ∈ Dom(ℎ) for all ℎ ∈ 𝐻. We denotethe number of tuples, the table size, 𝑛 by |𝜏|. When we treat the body 𝑋 as a set in
the set theoretic manner, we denote it by set(𝑋), that is, set(𝑋) = {𝑥 , 𝑥 , … , 𝑥 }.
This means that we do not take the order and multiplicity into account in set(𝑋).
Throughout this chapter, the domain Dom(ℎ) of each attribute ℎ ∈ 𝐻 is considered to be the closed interval [0, 1] ⊂ ℝ. For our experiments using synthetic
and real data sets (Section 5.3), we irst perform pre-processing (min-max normalization) (Han and Kamber, 2006) so that each value for each attribute is in thedomain [0, 1]. We assume that the header 𝐻 is always the set of natural numbers{1, 2, … , 𝑑}, so each tuple corresponds to a real vector in the 𝑑-dimensional Euclidean space ℝ . We call a tuple an object and a relation a database.
Let 𝐽 be a subset of the header 𝐻. For each object 𝑥, the projection of 𝑥 on 𝐽,
denoted by 𝑥| , is exactly the same as the restriction of 𝑥 to 𝐽, which is the function
from 𝐽 to Dom(𝐻) such that 𝑥| (ℎ) = 𝑥(ℎ) for all ℎ ∈ 𝐽. For a table 𝜏 = (𝐻, 𝑋),

5.1 CLUSTERING STRATEGY
𝜏

Δ (𝜏)

Δ (𝜏)

𝐻

1

2

3

𝑥
𝑥
𝑥

0.66 0.71 0.27
0.72 0.86 0.46
0.14 0.53 0.04

𝐻

1

2

3

𝐻

1

2

3

Δ (𝑥 )Δ (𝑥 )Δ (𝑥 )

1
1
0

1
1
1

0
0
0

Δ (𝑥 )Δ (𝑥 )Δ (𝑥 )

2
2
0

2
3
2

1
1
0

Table 5.1 | Database τ, and discretized databases ∆1 (τ) and ∆2 (τ).

the projection of 𝜏 is the database 𝜏| = (𝐽, 𝑋| ), where 𝑋| is de ined by 𝑋| ∶=𝑥 | ,𝑥 | ,…,𝑥 | .
Clustering is the partitioning of the set of objects set(𝑋) of a database 𝜏 into
𝐾 disjoint subsets 𝐶 , 𝐶 , … , 𝐶 , called clusters, such that 𝐶 ≠ ∅, 𝐶 ∩ 𝐶 = ∅with 𝑖 ≠ 𝑗, and ⋃ ∈{ ,…, } 𝐶 = set(𝑋). Here, a database is treated as a sequence ofobjects (tuples) to take ordering and multiplicity of objects into account just likethe relational database theory, while a cluster is a inite set of objects in the set
theoretic manner and ordering and multiplicity is not considered. We say that a
set of clusters 𝒞 = {𝐶 , … , 𝐶 } with the above properties is a partition of set(𝑋).
The number of objects in 𝐶 is denoted by #𝐶.

5.1.2 Naïve BOOL
BOOL performs divisive hierarchical clustering, so the number of clusters increases
monotonically with the hierarchy level, and the union of some clusters becomes acluster at the previous lower level. At each level, BOOL determines the numberof clusters automatically. This ability is the major difference between BOOL andother divisive hierarchical methods.
First, we introduce discretization of objects in a database based on binary encoding scheme. Recall that the domain of each attribute is assumed to be in theinterval [0, 1].
De inition 5.1: Discretization
For an object 𝑥 in a database 𝜏, discretization at level 𝑘 is an operator Δ for discretization𝑥, where each value 𝑥(𝑖) is mapped to a natural number 𝑚 such that 𝑥(𝑖) = 0
implies 𝑚 = 𝟶, and 𝑥(𝑖) ≠ 0 implies

𝑚=

⎧𝟶
⎪𝟷

if 0 < 𝑥(𝑖) ≤ 2 ,if 2< 𝑥(𝑖) ≤ 2⎨ ...
⎪⎩2 − 1

if 2

(

)

79

,

< 𝑥(𝑖) ≤ 1.

We use the same operator Δ for discretization of a database 𝜏; i.e., each object 𝑥of 𝜏 is discretized to Δ (𝑥) in the database Δ (𝜏) = (𝐻, Δ (𝑋)). Table 5.1 shows
an example of discretization at levels 1 and 2. In the following, we ix the level of
discretization as 𝑘 and explain the construction of clusters at level 𝑘.
Next, we make clusters through discretization Δ by measuring the distances
between objects using the 𝐿 and 𝐿 metrics, where the distance between a pair

80

CLUSTERING USING BINARY DISCRETIZATION
of objects 𝑥 and 𝑦 is de ined by𝑑 (𝑥, 𝑦) =𝛿(𝑥(𝑖), 𝑦(𝑖)), where 𝛿 =0 if 𝑥(𝑖) = 𝑦(𝑖),1 if 𝑥(𝑖) ≠ 𝑦(𝑖),𝑑 (𝑥, 𝑦) = max |𝑥(𝑖) − 𝑦(𝑖)|∈{ ,…, }𝑙-neighborhood at level 𝑘in the 𝐿 and 𝐿 metrics, respectively. Here, for an object 𝑥 in a database 𝜏 = (𝐻, 𝑋)
and a natural number 𝑙 ∈ ℕ, we de ine 𝑥’s 𝑙-neighborhood at level 𝑘 by
𝑁 (𝑥) ≔ 𝑦 ∈ set(𝑋) 𝑑 (Δ (𝑥), Δ (𝑦)) ≤ 1 and 𝑑 (Δ (𝑥), Δ (𝑦)) ≤ 𝑙 .
(5.1)

distance parameter

We call 𝑙 the distance parameter. This condition means that the number of attributes for which two values differ is at most one and that the difference in thevalues is less than 𝑙 (see Figure 5.2). Note that, if 𝑙 = 1,
𝑁 (𝑥) = {𝑦 ∈ set(𝑋) ∣ 𝑑 (Δ (𝑥), Δ (𝑦)) ≤ 1}
holds, where 𝑑 is the Manhattan distance (𝐿 metric).

directly reachable at level 𝑘reachable at level 𝑘

De inition 5.2: Reachability
Given a distance parameter 𝑙 ∈ ℕ and a database 𝜏 = (𝐻, 𝑋). An object 𝑥 ∈set(𝑋) is directly reachable at level 𝑘 from an object 𝑦 ∈ set(𝑋) if 𝑥 ∈ 𝑁 (𝑦).
Moreover, 𝑥 is reachable at level 𝑘 from an object 𝑧 ∈ set(𝑋) if there exists a
chain of objects 𝑧 , 𝑧 , … , 𝑧 (𝑝 ≥ 2) in 𝜏 such that 𝑧 = 𝑧, 𝑧 = 𝑥, and 𝑧
isdirectly reachable at level 𝑘 from 𝑧 for every 𝑖 ∈ {1, 2, … , 𝑝 − 1}.
Trivially, this notion of reachability is symmetric; that is, if an object 𝑥 is reachable
at level 𝑘 from 𝑦, then 𝑦 is reachable from 𝑥. Consequently, a partition of a databaseat discretization level 𝑘 is naturally produced.

level-𝑘 partition

De inition 5.3: Level-k partition
A partition of a database 𝜏 = (𝐻, 𝑋) is a level-𝑘 partition, denoted by 𝒞 , if itsatis ies the following condition: For all pairs of objects 𝑥, 𝑦 ∈ set(𝑋), the pairis in the same cluster if and only if 𝑦 is reachable at level 𝑘 from 𝑥.
The level-𝑘 partition is determined uniquely. Intuitively, if we see each discretizedobject as a cluster, then adjacent clusters are agglomerated when 𝑙 = 1. It takes𝑂(𝑛 𝑑) time to construct the level-𝑘 partition with the naïve version shown in Algorithm 5.1, where 𝑛 is the number of objects and 𝑑 is the number of attributes,
since we have to calculate distances 𝑑 and 𝑑 for all pairs of objects. In the next
section, we discuss speeding up of this process by sorting the database.
Example 5.4
Consider the database 𝜏 given in Table 5.1. Fix 𝑙 = 1. Then, level-1 partition 𝒞 ={{𝑥 , 𝑥 , 𝑥 }} since𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 0 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 0,𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1,𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1,

5.1 CLUSTERING STRATEGY
𝑙=1

81

𝑙=3
𝑥
l=2

𝑥
𝑥

Figure 5.2 | Illustration of lneighborhood. For each distanceparameter l and an object x, if an
object y is in a dotted square, y ∈Nkl (x) deﬁned in the equation (5.1).

hence 𝑁 (𝑥 ) = {𝑥 , 𝑥 }, 𝑁 (𝑥 ) = {𝑥 , 𝑥 }, and 𝑁 (𝑥 ) = {𝑥 , 𝑥 }; 𝑥 and 𝑥 arereachable at level 1 from 𝑥 . Level-2 partition 𝒞 = {{𝑥 , 𝑥 }, {𝑥 }} since𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 1,𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 2 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 2,𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 3 and 𝑑 (Δ (𝑥 ), Δ (𝑥 )) = 2,
hence 𝑁 (𝑥 ) = {𝑥 }, 𝑁 (𝑥 ) = {𝑥 }, and 𝑁 (𝑥 ) = ∅; 𝑥 and 𝑥 are not reachable
from 𝑥 , and 𝑥 is reachable from 𝑥 .
Level-𝑘 partitions have a hierarchical structure, that is, for the level-𝑘 and 𝑘 +1 partitions 𝒞 and 𝒞 , the following condition holds: For every cluster 𝐶 ∈𝒞 , there exists a set of clusters 𝒟 ⊆ 𝒞such that ⋃ 𝒟 = 𝐶. This is why, for
two objects 𝑥 and 𝑦, if 𝑦 is directly reachable at level 𝑘 from 𝑥, then 𝑦 is directly
reachable at level 𝑘 from 𝑥 for all 𝑘 ≤ 𝑘. BOOL uses this property; it increases
discretization level 𝑘 to make more than 𝐾 clusters, where 𝐾 is the input parameter
denoting the lower bound on the number of clusters.
Furthermore, by using the lower bound on cluster size 𝑁, noise iltering can bedone easily and directly.
De inition 5.5: NoiseGiven a natural number 𝑁 ∈ ℕ. Let 𝒞 be the level-𝑘 partition of a database𝜏 = (𝐻, 𝑋). For each cluster 𝐶 ∈ 𝒞 , every object in 𝐶 is noise if #𝐶 < 𝑁.
noise
We call 𝑁 the noise parameter. For example, for the level-2 partition 𝒞 in Example noise parameter
5.4, the cluster {𝑥 } is detected as noise if 𝑁 = 2.

5.1.3 Relationship between BOOL and DBSCAN
The BOOL approach can be viewed as a mix of hierarchical clustering with densitybased clustering. BOOL uses only 𝐿 and 𝐿 metrics to de ine reachability and thisrestriction allows for sorting to be leveraged for clustering shown in the next section. Here we theoretically discuss the relationship between BOOL and DBSCAN(Ester et al., 1996), which is a typical density-based clustering algorithm, with respect to the reachability.
First, we prepare the notion of density-reachability in DBSCAN referring theEps-neighborhood
literature (Ester et al., 1996). For an object 𝑥, the Eps-neighborhood of 𝑥 is

82

CLUSTERING USING BINARY DISCRETIZATION𝑁

directly density-reachable(𝑥) = {𝑦 ∈ set(𝑋) ∣ 𝔡(𝑥, 𝑦) ≤ Eps},

where 𝔡 is an arbitrary distance function such as Euclidean distance. Then, an
object 𝑥 is directly density-reachable from 𝑦 with respect to Eps and MinPts if𝑞∈𝑁

density-reachable

density-connected(5.2)(𝑥) and #𝑁( ) ≥ MinPts,

and an object 𝑥 is density-reachable from 𝑧 with respect to Eps and MinPts if there isa chain of objects 𝑧 , 𝑧 , … , 𝑧 such that 𝑧 = 𝑧, 𝑧 = 𝑥, and 𝑧 is directly densityreachable from 𝑧 . Here, in DBSCAN, each cluster is de ined to be a set of densityconnected points which is maximal with respect to density-reachability, where an
object 𝑥 is density-connected to an object 𝑦 with respect to Eps and MinPts if there
is an object 𝑜 such that both 𝑥 and 𝑦 are density-reachable from 𝑜 with respect to
Eps and MinPts.
Compared to our approach, we can easily show the following relationship between BOOL and DBSCAN since, if MinPts = 1, the property of density-connectivity
is exactly the same as density-reachability.
Theorem 5.6
Given a database 𝜏 = (𝐻, 𝑋). Assume that the distance parameter 𝑙 = 1. The
level-𝑘 partition 𝒞 of 𝜏 is exactly the same as the output of DBSCAN for the discretized input database Δ (𝜏) if the distance 𝔡 in the equation (5.2) is the Manhattan distance 𝑑 and parameters are set as Eps = 𝑙 and MinPts = 1.
For 𝑙 > 1, if we modify the de inition of the Eps-neighborhood by𝑁(𝑥) = {𝑦 ∈ set(𝑋) ∣ 𝑑 (𝑥, 𝑦) ≤ 1 and 𝑑 (𝑥, 𝑦) ≤ Eps},

we have the same result. In contrast, the de inition of noise is different. Noise in
DBSCAN is de ined using MinPts as low density area, whereas BOOL does not use
any density; no object is noise in BOOL if MinPts = 1.
The time complexity of DBSCAN is 𝑂(𝑛 𝑑), thus it is dif icult to apply it for massive data in data mining and knowledge discovery. It is known that the complexity
can be reduced to 𝑂(𝑛𝑑 log 𝑛) using 𝑘-𝑑 tree for data indexing, but practically DBSCAN is still computationally expensive (Han and Kamber, 2006). In the next section, we show that linear complexity with respect to the number of objects can berealized in BOOL by sorting binary representations of original data, which resultsin speedups when compared with DBSCAN and other algorithms.
Moreover, DBSCAN is quite sensitive to the parameter settings as describedby Han and Kamber (2006). The hierarchical clustering of BOOL using the input
parameter 𝑙 of a natural number realizes robust clustering, and manual parametersetting in BOOL is quite easy.

5.2 Speeding Up of Clustering through Sorting
In this section, we discuss speeding up of the naïve version of BOOL by sorting theobjects, resulting in linear time complexity. The pseudo-code of speeded-up BOOL
is shown in Algorithm 5.2.
First, we introduce a sorting operator. We sort a given database 𝜏 by using
values in the discretized database Δ (𝜏).

5.2 SPEEDING UP OF CLUSTERING THROUGH SORTING

Algorithm 5.2: Speeded-up BOOL
Input: Database 𝜏 = (𝐻, 𝑋), lower bound on number of clusters 𝐾,
noise parameter 𝑁, and distance parameter 𝑙
Output: Partition 𝒞
function B(𝜏, 𝐾, 𝑁)
1: 𝑘 ← 1 // 𝑘 is discretization level2: repeat3:𝒞←MH(𝜏, 𝑘, 𝑁)
4:𝑘←𝑘+1
5: until #𝒞 ≥ 𝐾
6: output 𝒞
function MH(𝜏, 𝑘, 𝑁)
1: 𝒞 ← { {𝑥} ∣ 𝑥 ∈ set(𝑋) }2: 𝑋 ← Δ (𝑋) // discretize 𝑋 at level 𝑘
3: 𝑋 ← 𝑆 ∘ 𝑆 ∘ … ∘ 𝑆 (𝑋), where 𝜎 = 𝑋 |4: ℎ ← 𝑑
5: repeat
6:
𝑋 ← 𝑆 (𝑋) such that 𝜎 = 𝑋 |7:𝒞←A(𝜏, 𝒞, ℎ)8:ℎ←ℎ−1
9: until ℎ = 0
10: 𝒞 ← {𝐶 ∈ 𝒞 ∣ #𝐶 ≥ 𝑁}
11: output 𝒞
function A(𝜏, 𝒞, ℎ)
1: for each object 𝑥 ∈ set(𝑋)2:𝑦 ← successive object of 𝑥
3:
if 𝑦 is directly reachable from 𝑥 (i.e., 𝑦 ∈ 𝑁 (𝑥)) then4:
delete clusters 𝐶 ∋ 𝑥 and 𝐷 ∋ 𝑦 from 𝒞, and add 𝐶 ∪ 𝐷
5:
end if
6: end for
7: output 𝒞

De inition 5.7: SortingLet 𝜏 = (𝐻, 𝑋) be a database and 𝜎 be a sequence of natural numbers such thatthe size of 𝜎 is same as that of 𝑋. The expression 𝑆 (𝜏) is the database 𝜏 for which
objects are sorted in the order indicated by 𝜎, where ties keep the original order.
Obviously, this operation 𝑆 can be realized by using a standard sorting algorithm
such as quick sort. Here we only consider sorting of database 𝜏 by using a column
in the discretized database Δ (𝜏), so we can use bucket sort since the domain of bucket sorteach attribute of Δ (𝜏) is from 𝟶 to 2 − 1, and this size can reasonably it in thebucket of the main memory for most real databases (𝑘 is usually less than 10 inpractice). The operation of 𝑆 should therefore take only 𝑂(𝑛) time, where 𝑛 is thenumber of objects.

83

84

CLUSTERING USING BINARY DISCRETIZATION𝜏Table 5.2 | Database τ = ({1,2,3}, X)
and sorted database Sσ (τ), where σ=∆2 (X)|2 .

𝜎

𝐻

1

2

3

𝑥
𝑥
𝑥

0.66 0.71 0.27
0.72 0.86 0.46
0.14 0.53 0.04

2
3
2

𝑆 (𝜏)
𝐻

1

2

3

𝑥
𝑥
𝑥

0.66 0.71 0.27
0.14 0.53 0.04
0.72 0.86 0.46

Example 5.8
Consider database 𝜏 given in Table 5.2 (same as in Table 5.1). and let 𝜎 be thesequence Δ (𝑋)| ; i.e., the column of the discretized database Δ (𝜏) = ({1, 2, 3},
Δ (𝑋)) for attribute 2. Then, 𝑆 (𝜏) becomes the database in Table 5.2.
Next we describe the main theorem used for speeding up BOOL. It states clustering can be completed by comparing each object to the next object 𝑑 times withsorting.
Theorem 5.9
For any database 𝜏 and any natural number 𝑘, the output of the function MH(𝜏, 𝑘, 0) in Algorithm 5.2 is the level-𝑘 partition 𝒞 .

-Proof. Suppose that 𝒞 is the output of the function M
H. From thefunction A, it is trivial that, for all pairs of objects 𝑥 and 𝑦 of 𝜏 such that they
are in the same cluster in 𝒞, the same holds in 𝒞 . Thus, it is enough to prove that,for all pairs of objects 𝑥 and 𝑦, if 𝑦 is directly reachable from 𝑥, they are in the samecluster in 𝒞.
Fix two objects 𝑥 and 𝑦 such that 𝑦 is directly reachable from 𝑥. We can assume
that there is no object 𝑧 such that 𝑑 (Δ (𝑧), Δ (𝑥)) = 0 or 𝑑 (Δ (𝑧), Δ (𝑦)) = 0(i.e., 𝑧 = 𝑥 or 𝑧 = 𝑦) without loss of generality. Then there must exist ℎ ∈ 𝐻 ={1, 2, … , 𝑑} such that |𝑥(ℎ) − 𝑦(ℎ)| ≤ 𝑙 and |𝑥(𝑖) − 𝑦(𝑖)| = 0 for all 𝑖 ∈ 𝐻 with𝑖 ≠ ℎ. By sorting 𝑆 (𝑋) using 𝜎 = Δ (𝑋)| (line 6 of the function), it is easy to seethat 𝑦 becomes the successive object of 𝑥 since other attributes have already beensorted, so they are in the same cluster. Thus 𝒞 is exactly the same as 𝒞 .
We can easily check that the time complexity of the function M
Hof Algorithm 5.2 is 𝑂(𝑛𝑑) in the best case and 𝑂(𝑛𝑑 ) in the worst case, sincechecking the condition with the distances 𝑑 and 𝑑 (line 3 of function A) takes𝑂(1) time in the best case and 𝑂(𝑑) time in the worst case. Thus, the time complexity of BOOL is 𝑂(𝑛𝑑𝑘) or 𝑂(𝑛𝑑 𝑘), where #𝒞< 𝐾 and #𝒞 ≥ 𝐾, and it isusually 𝑂(𝑛𝑑) or 𝑂(𝑛𝑑 ) since 𝑘 ≪ 𝑛 holds. Most real databases used in spatial
clustering have only two or three attributes, so BOOL is fast although the complexity is quadratic with respect to the number of attributes in the worst case (actual
running time is shown in next section).
Example 5.10
We explain the clustering process of the function M
H
in BOOL using
an example shown in Figure 5.3. We consider the level-2 partition of a database 𝜏(Figure 5.3, (a)) with the number of attributes 𝑑 = 2, and assume that the distanceparameter 𝑙 = 1.
First, the database 𝜏 (Figure 5.3, (a)) is discretized to Δ (𝜏) in line 2 in the function (Figure 5.3, (b)). Next, it is sorted two times by bucket sort for each attribute5.2 SPEEDING UP OF CLUSTERING THROUGH SORTING(a) Original database 𝜏 (b) Discretized (c) Sort by
database Δ (𝜏) attribute 2(d) Sort byattribute 1

𝐻

1

2

𝐻

1

2

𝐻

1

2

𝐻

1

2

𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥

0.36
0.48
0.42
0.19
0.72
0.92
0.40
0.97
0.36
0.77

0.11
𝑥
0.29
𝑥
0.61
𝑥
0.21 ⟶ 𝑥
0.88
𝑥
0.63
𝑥
0.51
𝑥
0.03
𝑥
0.87
𝑥
0.81
𝑥

1
1
1
0
2
3
1
3
1
3

𝑥
0
𝑥
1
𝑥
2
0⟶ 𝑥
𝑥
3
𝑥
2
𝑥
2
𝑥
0
𝑥
3
𝑥
3

1
0
3
1
1
3
1
2
1
3

0
𝑥
𝑥
0
0
𝑥
1⟶ 𝑥
2
𝑥
2
𝑥
2
𝑥
𝑥
3
3
𝑥
3
𝑥

0
1
1
1
1
1
2
3
3
3

0
0
1
2⟶
2
3
3
0
2
3

(e) Sort by attribute 2 and compare eachobject to the subsequent object
𝐻

1

2

Class

𝑥
𝑥
𝑥
⟶ 𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥

0
1
3
1
1
1
3
1
2
3

0
0
0
1
2
2
2
3
3
3

A
A
B
C
D
D
E
F
F
F

3
2

⟶

1
0
0

1

2

3

(f) Sort by attribute 1 and compare eachobject to the subsequent object
𝐻

1

2

Class

𝑥
𝑥
𝑥
⟶ 𝑥
𝑥
𝑥
𝑥
𝑥
𝑥
𝑥

0
1
1
1
1
1
2
3
3
3

0
0
1
2
2
3
3
0
2
3

A
A
A
A
A
A
A
B
A
A

3
2
1
0
0

1

2

3

Figure 5.3 | Illustrative example of clustering process by speeded-up BOOL.

85

86

radix sort

CLUSTERING USING BINARY DISCRETIZATION
in line 3 (Figure 5.3, (c) and (d)). This sorting is exactly the same as radix sort if
we see each discretized object Δ (𝑥) = 𝑚 as the integer 10𝑚(1) + 𝑚(2). Then, thediscretized database is sorted again together with merging clusters in lines 5–9(Figure 5.3, (e) and (f)). For each attribute ℎ, BOOL sorts the database by bucket
sort in advance, and compare each object to the subsequent object. If one object isdirectly reachable from the other object, BOOL categorizes these objects into thesame cluster (i.e., gives the same class label). By repeating this clustering process
for each attribute, every pair of objects are in the same cluster if and only if one isreachable from the other. In this example, by sorting with attribute 2, six clusters
A, B, C, D, E, and F are found (Figure 5.3, (e)) and, by sorting with attribute 1, iveclusters A, C, D, E, and F are merged into one cluster (Figure 5.3, (f)).

5.3 Experiments
We evaluate BOOL experimentally to determine its scalability and effectiveness forvarious types of databases including: synthetic databases with two attributes, real
databases with three attributes generated from natural images, large real databases with two attributes generated from geospatial images, and real databaseswith various sizes of attributes from the UCI repository.

5.3.1 Methods
Environment
We used Mac OS X version 10.6.5 with two 2.26-GHz Quad-Core Intel Xeon CPUs
and 12 GB of memory. BOOL was implemented in C and compiled with gcc 4.2.1.
All experiments were performed in the R environment, version 2.12.2 (R Development Core Team, 2011).
Databases
Synthetic and real databases were used. To evaluate scalability with respect tothe size of the databases and the number of clusters, synthetic databases weregenerated randomly using the R clusterGeneration package (Qiu and Joe, 2006),
with parameter sepVal set to 0.34. Each database had two attributes (𝑑 = 2), andthe number of objects (𝑛) and clusters (𝐾) varied from 100 to 1,000,000 and from
5 to 40, respectively. We used databases with 𝑛 = 10,000 and 𝐾 = 5 to evaluate therobustness of BOOL. Each database contained 𝑛/100 outliers (noise points). The
databases were generated 20 times, and the mean and s.d. (standard deviation)
were calculated. Four synthetic databases (DS1, DS2, DS3, and DS4) were takenfrom the CLUTO website¹; they had been used as benchmarks in the evaluationsof other spatial clustering algorithms such as CHAMELEON (Karypis et al., 1999),
SPARCL (Chaoji et al., 2009), and ABACUS (Chaoji et al., 2011).
The real databases consisted of four natural images, four large satellite images,
and six databases from the UCI machine learning repository (Frank and Asuncion, 2010). The natural images (shown in Figure 5.7, (a), (d), (g), and (j)) were¹http://glaros.dtc.umn.edu/gkhome/views/cluto/

5.3 EXPERIMENTS
(a)

(b)
10

1
10-2
10-4
102 103
104
105
106
Number of objects n

1.00

Adjusted Rand index

0.98
0.96
0.94
0.92
0.9
102 103
104
105
106
Number of objects n

Running time (s)

102

1

BOOL
K-means
DBSCAN

10-1

10-2
10-3

(d)
Adjusted Rand index

Running time (s)

104

(c)

87

5 10 15 20 25 30 35 40
Number of clusters K

1.0
0.9
0.8
0.7
0.6
0.5
0.4

5 10 15 20 25 30 35 40
Number of clusters K

taken from the website of the Berkeley segmentation database and benchmark²(Martin et al., 2001); they were the same ones used as benchmarks in a previous study by Chaoji et al. (2011). Each image was 481 × 321 in size, and 154,401
pixels in total, and the RGB (red-green-blue) values for each pixel were obtainedby pre-precessing. Finally, each image was translated into a database with 3 attributes and 154,401 objects. Four another real databases were collected from the
Earth-as-Art website³, which contains geospatial satellite images. Similar databases were used in the literature (Chaoji et al., 2011). Each image was originallycomposed of 7, 296×7, 296 pixels (Figures 5.8 (a), (e), (i), and (m)) and was translated into the binary image using ImageJ software (Rasband, 1997–2011) (Figures 5.8 (b), (f), (j), and (n)). The six UCI databases (ecoli, sonar, shuttle, wdbc,
wine, and wine quality) were used to test the effectiveness of BOOL for databaseswith various attribute sizes.
Every database was translated using min-max normalization (Han and Kamber, 2006) in advance so that each value was in the unit interval [0, 1], The runtime
for this translation was included in the BOOL running time.
Control Algorithms
We used 𝐾-means and DBSCAN as control algorithms. 𝐾-means is a commonlyused and ef icient clustering algorithm, and DBSCAN is a well-known and typical spatial clustering algorithm. DBSCAN was executed using the R fpc package.
For databases DS1 - DS4 and the natural image databases, we also compared therunning time of BOOL with those of state-of-the-art clustering algorithms ABACUS(Chaoji et al., 2011) and SPARCL (Chaoji et al., 2009). Their source codes are notpublicly available, so we simply used the reported results (Chaoji et al., 2011) toavoid implementation biases. The four synthetic databases and four natural im²http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/³http://eros.usgs.gov/imagegallery/

Figure 5.4 | Clustering speed andquality for randomly generated synthetic databases, where K = 5 for (a)and (c) and n = 10,000 for (b) and (d).
Adjusted Rand index should be maximized. Data show mean ± s.d.

CLUSTERING USING BINARY DISCRETIZATION

0.015
0.010
0.005
0

Adjusted Rand index(c)Figure 5.5 | Clustering speed andquality for various values of distanceparameter l and noise parameter N.
Adjusted Rand index should be maximized. Data show mean ± s.d.

(b)

0.020

Running time (s)Running time (s)

(a)

(d)

1.00
0.98
0.96
0.94
0.92
0.90

2
4
6
8
10
Distance parameter l

0.020
0.015
0.010
0.005
0

2
4
6
8
10
Distance parameter lAdjusted Rand index

88

1.2
1.0
0.8
0.6
0.4
0.2
0
-0.2

0

2
4
6
8
Noise parameter N

10

0

2
4
6
8
Noise parameter N

10ages are identical to those used in a previous evaluation (Chaoji et al., 2011), so
the comparison here is reasonable.
Evaluation
Clustering quality was evaluated using the adjusted Rand index (takes values in[−1, 1], to be maximized) (Hubert and Arabie, 1985), which takes into account the
ground truth and is popular in the clustering literature. For the synthetic databases DS1 - DS4, natural images, and satellite images, we simply show scatterand contour plots of the results since we had no information on the ground truth.
Moreover, the number of clusters was not clear for the natural images, so we used
the MAKEHIERARCHY function in Algorithm 5.2 to obtain clusters at some hierarchy
level. For 𝐾-means, we used the reported number of clusters (Chaoji et al., 2011).
All parameters for BOOL and DBSCAN were tuned to obtain the best results.

5.3.2 Results and Discussion
As discussed below, our proposed algorithm, BOOL, shows the best performancein terms of both clustering speed and quality. The results indicate that BOOL is
ef icient and effective for spatial clustering.
The clustering speed and quality results for randomly generated synthetic databases are plotted in Figure 5.4. For BOOL, the distance and noise parameters, 𝑙 and𝑁, were set to 1 and 𝑛/100, respectively. In most cases, BOOL shows the best performance. BOOL is faster than 𝐾-means and from two to six orders of magnitude
faster than DBSCAN. Moreover, the quality of the obtained clusters is higher thanthose for 𝐾-means and DBSCAN in terms of the adjusted Rand index. The clustering speed and quality of BOOL for various values of 𝑁 and 𝑙 are plotted in Figure 5.5,
where 𝑁 was ixed to 100 for Figures 5.5, (a) and (c), and 𝑙 to 1 for Figures 5.5, (b)and (d). Every database contains noise points, and BOOL treats each one as a cluster if 𝑁 is small (𝑁 = 0, 1, 2, and 3). This is why the adjusted Rand index is low for5.3 EXPERIMENTS
Name

𝑛

𝑑

𝐾

BL

KM

DB

AB

SP

DS1
DS2
DS3
DS4

8000
8000
10000
8000

2
2
2
2

6
6
9
8

0.004
0.004
0.010
0.005

0.008
0.008
0.036
0.018

9.959
10.041
15.832
9.947

1.7
1.3
1.9
1.7

1.8
1.5
2.5
1.8

Horse
Mushroom
PyramidRoad

154401154401154401154401

3
3
3
3

–
–
–
–

0.253
0.761
0.187
0.180

0.674
1.449
0.254
0.209

–
–
–
–

31.2
29.3
11.3
14.9

41.8
–
–
–

Delta
EuropeNorwayGanges27543260
23071720
33879834
24546151

2
2
2
2

6
6
4
8

34.39827.60723.86632.977273.23144.98546.30258.273

–
–
–
–

–
–
–
–

–
–
–
–

small 𝑁. These results show that BOOL is robust with respect to 𝑁 and 𝑙 and, forlots of databases, all we have to do is set 𝑙 as 1 and 𝑁 large enough (𝑁 ≥ 4).
The results for the synthetic databases DS1–DS4 are summarized in Table 5.3
and illustrated in Figure 5.6. For all databases, BOOL is the fastest and about threeorders of magnitude faster than the two state-of-the-art clustering algorithms, ABACUS and SPARCL, when (𝑙, 𝑁) = (1, 100), (1, 100), (7, 100), and (11, 40) for DS1,
DS2, DS3, and DS4, respectively. BOOL inds all reasonable clusters in DS1 andDS2. It also effectively ilters out the noise, as shown in Figure 5.6, whereas 𝐾means does not. Note that ABACUS and SPARCL cannot detect noise effectively,
as shown in previous studies (Chaoji et al., 2009, 2011). BOOL does not ind someclusters in DS3 and DS4, which is counterintuitive. The reason is that such clusters
are connected by dense noise, so BOOL cannot divide them into smaller clusters.
However, the results are still much better than those with 𝐾-means. These results
indicate that BOOL is the fastest spatial clustering algorithm that can ind arbitrarily shaped clusters for use in typical spatial clustering tasks.
For the real databases obtained from the natural images, BOOL again shows thebest performance; i.e., it is the fastest of the compared clustering algorithms (Table 5.3) for (𝑘, 𝑙, 𝑁) = (7, 1, 50), (7, 15, 20), (8, 5, 100), and (10, 15, 400) for Horse,
Mushroom, Pyramid, and Road, respectively (𝑘 is the hierarchy level). BOOL isfaster than 𝐾-means, and about two orders of magnitude faster than ABACUS andSPARCL (we did not perform DBSCAN since it takes too much time and needs manual tuning of the input parameters). Moreover, the quality of the clustering is muchbetter than that with 𝐾-means, as shown in Figure 5.7. These results show that
BOOL is ef icient and effective for spatial clustering for natural images. Furthermore, they show that BOOL works well for hierarchical clustering, meaning that
we can input the hierarchy level directly rather than entering the lower bound on
the number of clusters.
For all the real databases from geospatial satellite images, BOOL is faster than𝐾-means and inds natural clusters (parameters 𝑙 and 𝑁 were set as 1 and 100000,
respectively, for all the databases), as shown in Figures 5.8, (c), (g), (k), and (o),whereas 𝐾-means results in inferior clustering quality (Figures 5.8, (d), (h), (l),and (p)). These results show that BOOL has high scalability and that it can indarbitrarily shaped clusters without careful tuning of the input parameters.

89

Table 5.3 | Running time (in seconds) for synthetic databases, real
databases from natural images, andreal databases from geospatial satellite images with BOOL (BL), K-means(KM), DBSCAN (DB), ABACUS (AB), andSPARCL (SP). In table, n, d, and K denote number of objects, attributes,
and clusters, respectively. Results for
ABACUS and SPARCL are taken fromthe literature (Chaoji et al., 2011).
Clustering results are shown in Figures 5.6, 5.7, and 5.8.

90

CLUSTERING USING BINARY DISCRETIZATION
Name

Table 5.4 | Results for UCI databaseswith BOOL (BL), K-means (KM) and
DBSCAN (DB). In table, n, d, and K
denote the number of objects, attributes, and clusters, respectively.
For all databases, BOOL showed theequal or better performance compared to K-means and DBSCAN.

ecoli
sonar
shuttlewdbc
wine
wine q.

𝑛
336
208
14500
569
178
4898

𝑑
7
60
9
30
13
11

𝐾
8
2
7
2
3
7

Running time

Adjusted Rand index

BL

KM

DB

BL

KM

DB

0.001
0.004
0.025
0.004
0.002
0.019

0.002
0.005
0.065
0.004
0.002
0.026

0.111
0.149
–
0.222
0.047
7.601

0.5745 0.4399 0.1223
0.0133 0.0064 0.0006
0.7651 0.1432–
0.6806 0.4914 0.5530
0.4638 0.3347 0.2971
0.0151 0.0099 0.0134

Finally, for the UCI databases, BOOL again shows the best performance compared to 𝐾-means and DBSCAN (Table 5.4), showing that BOOL works well not only
for low-dimensional databases but also for high-dimensional databases. These results show that BOOL should work ef iciently and effectively for various types of
databases including small databases, large databases, and databases containing
many clusters.

5.4 Related Workdiscretization

From the point of view of discretization, several techniques (Fayyad and Irani, 1993;
Liu et al., 2002; Skubacz and Hollmén, 2000) have been proposed to treat multivariate data in the discrete manner. In particular, recently Lin et al. (2007, 2003)
have been proposed SAX, symbolic representation for time series data, which is
becoming a more and more popular discretization technique. They showed thatthe quality of clusters becomes better with SAX in both agglomerative hierarchical clustering and partitional clustering using 𝐾-means. However, since the above
discretization methods including SAX are just data preprocessing, it is dif icult to
directly achieve ef icient spatial clustering for inding arbitrarily shaped clusters.
On the other hand, in our BOOL approach, the discretization process is integrated
into the clustering process and clusters are ef iciently detected with effective useof sorting (see Example 5.10 and Figure 5.3). Informally, the geometric information of “topology” (i.e., metric) of the Euclidean space is appropriately embedded
into the discrete space through the binary encoding, and this embedding enables
effective and ef icient spatial clustering with binary words of original data.
Many clustering algorithms have been proposed for arbitrary shape clustering(Berkhin, 2006; Halkidi et al., 2001; Jain et al., 1999a), and some notable clustering
algorithms, SPARCL (Chaoji et al., 2009) and ABACUS (Chaoji et al., 2011), have recently been proposed for massive databases. Although we do not compare BOOL
with these algorithms in exactly the same way, our experiments clearly showsthat BOOL is the fastest among these state-of-the-art clustering algorithms andthat it is robust to the input parameters. Other new approaches include a massbased clustering MassTER algorithm (Ting and Wells, 2010). It also has linear timecomplexity with respect to the data size, but their results indicate that it is muchslower than BOOL. For instance, MassTER takes 59 seconds for a database with𝑛 = 70,000 and 𝑑 = 3.
In the following, we brie ly review spatial (shape-based) clustering algorithms,
including density-based clustering algorithms, agglomerative hierarchical clustering algorithms, and grid-based algorithms, and compare them to BOOL.

5.4 RELATED WORK(a) DS1(b) BOOL(c) -means(d) DS2(e) BOOL(f) -means(g) DS3(h) BOOL(i) -means(j) DS4(k) BOOL(l) -means

Figure 5.6 | Synthetic databases DS1 - DS4 and clustering results obtained using BOOL and K-means.

91

92

CLUSTERING USING BINARY DISCRETIZATION(a) Horse(b) BOOL(c) -means(d) Mushroom(e) BOOL(f) -means(g) Pyramid(h) BOOL(i) -means(j) Road(k) BOOL(l) -means

Figure 5.7 | Four natural images and corresponding contour maps obtained using BOOL and K-means from preprocessed databases with three attributes (RGB). Pixels of the same color are in the same cluster.

5.4 RELATED WORK(a) Delta(b) Binary image(c) BOOL(d) -means(e) Europe(f) Binary image(g) BOOL(h) -means(i) Norway(j) Binary image(k) BOOL(l) -means(m) Ganges(n) Binary image(o) BOOL(p) -means

Figure 5.8 | Results for geospatial satellite images obtained using BOOL and K-means. BOOL ﬁnds most of naturalclusters whereas 𝐾-means does not.

93

94

CLUSTERING USING BINARY DISCRETIZATION
Density-based clustering, such as that used in DBSCAN (Ester et al., 1996) and
DENCLUE (Hinneburg and Keim, 1998), is one of the most commonly used approaches to detect arbitrarily shaped clusters. However, it is quite sensitive tothe parameter settings, as described by (Han and Kamber, 2006) and it is computationally expensive. Despite BOOL and DBSCAN have close relationship as shownin Section 5.1.3, our experiments show that BOOL is robust and much faster than
DBSCAN (see Section 5.3).
Guha et al. (1998) introduced the hierarchical agglomerative clustering algorithm CURE for inding arbitrarily shaped clusters. The key to CURE is data partitioning to obtain ine cluster representatives. This approach is similar to ours since
we obtain binary words as representatives by discretization and agglomerate clusters by using the obtained representatives. However, the time complexity of CURE
is large and quadratic with respect to data size, so CURE needs data sampling when
it is applied to massive data sets. In contrast, the time complexity of BOOL is linear,and it is much faster than CURE. Moreover, it can perform exact clustering on massive data sets. Another hierarchical approach, CHAMELEON (Karypis et al., 1999),
achieves spatial clustering by using a graph partitioning algorithm. However, several parameters must be tuned for effective clustering.
Unlike CURE, which is based on data partitioning, grid-based algorithms suchas BANG (Schikuta and Erhart, 1997), STING (Wang et al., 1997), and WaveCluster(Sheikholeslami et al., 1998) focus on space partitioning. They summarize data information into a grid structure and achieve clustering by using this structure. For
example, STING assembles summary statistics into a tree of grid-cells. This ideaof data summarization using space partitioning is the same as in BOOL. However,compared to other such algorithms, BOOL is very simple — it simply discretizes
each data point by using binary encoding, without using statistics. This simplicity
enables it to achieve fast and robust clustering by effectively using data sorting. In
contrast, BOOL shares the major issue about high-dimensional data with the above
space partitioning algorithms. High-dimensional data are usually too sparse toclustering since the number of partitioned regions exponentially increases withrespect to the number of dimensions. In the extreme case in BOOL, every object
is not reachable from any other objects even at discretization level 1, hence BOOL
does not work well for such high-dimensional data. Thus combining other techniques such as dimension reduction methods is needed.

5.5 Summary
We have developed a spatial clustering algorithm called BOOL that is the fastest
such algorithm with respect to inding arbitrarily shaped clusters. It is robust,
highly scalable, and noise tolerant, and especially effective for low-dimensional
data such as geographical images. The key to performance is discretization basedon the binary encoding and sorting of data points. BOOL can be viewed as a restricted version of DBSCAN, which is a typical density-based clustering algorithm,
but our hierarchical approach with discretization realizes not only highly speedingup but also robust clustering which can be easily handled by the user.
BOOL is simple, hence it can be extended to other data mining tasks such as
anomaly detection, semi-supervised learning, and structured data clustering. Furthermore, BOOL can be used for preliminary exploration, especially for various
tasks in computer vision and pattern recognition from massive images.

Part III
With Formal Concept Analysis“je n’y emploie point d’autres caractères que 0 et 1,
et puis allant à deux, je recommence”— Gottfried Wilhelm Leibniz, Explication De L’arithmétique Binaire

6SEMI-SUPERVISEDCLASSIFICATION AND RANKING
mixed-type datasemi-supervised learning

Formal Concept Analysis (FCA)

closed set latticeN

including both discrete (binary or nominal) andcontinuous (real-valued) variables are collected by researchers in various research domains from biology to economics. However, despite recent rapid development of many data analysis techniques in the ields of machine learning, data
mining, and knowledge discovery, only few algorithms such as the decision treebased classi ier (Murthy, 1998) can directly handle such mixed-type data. In particular, to the best of our knowledge, no learning algorithm treats mixed-type data
in a semi-supervised manner based on discrete approaches.
Semi-supervised learning is a special form of classi ication (Zhu and Goldberg,
2009; Chapelle et al., 2006); a learning algorithm uses both labeled and unlabeleddata to learn classi ication rules. In real tasks, it is often dif icult to obtain enoughlabeled data since the task of labeling has a high cost in terms of time and money,
whereas lots of unlabeled data can be collected easily. The goal of semi-supervised
learning is to construct a better classi ier using such large amount of unlabeleddata together with labeled data in short supply.
To effectively use unlabeled mixed-type data for learning (training), we in thischapter propose a novel semi-supervised learning algorithm, called SELF (SEmisupervised Learning via Formal Concept Analysis), which can directly treat mixedtype data. SELF adopts a popular semi-supervised learning strategy, called clusterand-label (Dara et al., 2002; Demiriz et al., 1999), where a clustering algorithm
is irst applied, followed by labeling each cluster using labeled data. One of theremarkable features of SELF is that it performs the clustering process using Formal Concept Analysis (FCA) (Davey and Priestley, 2002; Ganter and Wille, 1998),
which is a mathematical theory for data analysis and knowledge representation introduced by Wille (1982). Recently, Pasquier et al. (1999) proposed to use closed
patterns (itemsets) obtained by FCA as condensed “lossless” representations ofpatterns. This new approach has been the subject of further research and extensions (Beslon et al., 2010; Plantevit et al., 2010; Saquer and Jitender, 2010; Zaki,
2000). In SELF, the labeling process is performed on a closed set lattice, which is
the result of FCA. Informally, this structure describes the maximally general classiication rules that explain the training data, thus preventing over itting. Moreover,6.1 RELATED WORK
each classi ication rule can be weighted using the closed set lattice by counting thenumber of clusters classi ied by the rule, resulting in the preference of class labels preference
as a partial order of them for each unlabeled datum. Furthermore, FCA and closed
set lattices enable us to naturally treat incomplete data including missing values.
To summarize, this chapter provides a contribution to both the ields of semisupervised learning and FCA:
1. To semi-supervised learning: we present a novel approach based on an algebraic framework without assuming any data distribution.
2. To FCA: we study a novel application, semi-supervised learning, using FCA andclosed set lattices.
The behavior of SELF is outlined as a lowchart in Figure 6.1, and this chapter
is organized along it after discussing about related work in Section 6.1. The data
preprocessing phase to construct a context from a given dataset to apply FCA is
explained in Subsection 6.2.1. Missing values are handled in this phase. The learning phase is described in Subsections 6.2.2 and 6.2.3; Subsection 6.2.2 shows dataclustering and making closed set lattices by FCA and Subsection 6.2.3 explains thetraining algorithm of SELF to learn classi ication rules. Classi ication by learned
rules is considered in Subsection 6.2.4. Section 6.3 gives empirical evaluation of
SELF and, inally, the key points and future work are summarized in Section 6.4.

6.1

Related Work

Many studies used FCA for machine learning (Kuznetsov, 2004), such as classi ication (Ganter and Kuznetsov, 2000, 2003), clustering (Zhang et al., 2008), frequent pattern and association rule mining (Jaschke et al., 2006; Pasquier et al.,
1999; Valtchev et al., 2004), and bioinformatics (Blinova et al., 2003; Kaytoue et al.,2011b; Kuznetsov and Samokhin, 2005). Ganter and Kuznetsov (2000) investigated the problem of binary classi ication of real-valued data and proposed algorithms based on the JSM-method that produces hypotheses (classi iers) using JSM-method
positive and negative examples. Their idea of using the lattice structure derived
by FCA for classi ication is similar to our approach, but the way of treating continuous variables is different. Their method discretizes continuous variables byinequations, called conceptual scaling (Ganter and Wille, 1998), that are given a conceptual scalingpriori, while SELF automatically discretizes them along with the learning process
and no background knowledge and assumption about data are needed.
On the other hand, in machine learning context, decision tree-based algorithms
such as C4.5 (Quinlan, 1993, 1996) can directly treat mixed-type data by discretizing continuous variables, and there are several discretization techniques (Fayyad
and Irani, 1993; Liu et al., 2002; Skubacz and Hollmén, 2000) to treat continuous variables in a discrete manner. Our approach is different from them since we
integrate discretization process into learning process and avoid over itting using
closed set lattices effectively. SELF uses cluster-and-label, or called label propagation, which is a popular approach in semi-supervised learning as mentioned in
the beginning of this chapter (Cheng et al., 2009; Dara et al., 2002; Demiriz et al.,
1999; Hwang and Kuang, 2010; Wang and Zhang, 2006). First SELF makes clusters
without label information by FCA, followed by giving preferences of class labels foreach cluster. However, to date, most of such approaches are designed for only continuous variables and, to the best of our knowledge, no semi-supervised learning

97

98

SEMI-SUPERVISED CLASSIFICATION AND RANKING

Input training data
Preprocess fordiscrete variables

Data preprocessing phasek←0

Learning phase

Discretize continuous
variables at level kMake a contextk←k+1
Make a closed set lattice(concept lattice) by FCA
Extract classi cation rules
using labeled data

All labeled data
are contained inconsistent concepts

Figure 6.1 | A ﬂowchart of the proposed SELF algorithm. It learns classiﬁcation rules from training data andapplies them to classify test data.
Here we say that a concept is lconsistent if all labels contained in theconcept are same.

Remove objects containedin consistent concepts

NO

YES
Classify test data

Classi cation phase

Output a result

algorithm based on cluster-and-label can treat mixed-type data including discretevariables appropriately. Since SELF uses FCA for clustering, it needs no distancecalculation and no data distribution, which is one of features of SELF.
There exists only one study by Kok and Domingos (2009) which is related tothe idea of putting original data on lattices. They proposed a learning algorithm via
hypergraph lifting, which constructs clusters by hypergraphs and learns on them.
Their idea is thus similar to ours since we also “lift” raw data to the space of aclosed set lattice via FCA. However, it is dif icult to treat continuous variables intheir approach, thereby our approach can be more useful for machine learningand knowledge discovery from mixed-type data.
SELF achieves not only semi-supervised learning but also label ranking using
the preference for each class label. Recently, the concept of preference has attracted more and more attention in arti icial intelligence including machine learning and knowledge discovery, resulting in formalization of the research topic of“preference learning” (Fürnkranz and Hüllermeier, 2010). In particular, label rank-

6.2 THE SELF ALGORITHM
ing (Cheng et al., 2010; Hülermeier et al., 2008; Vembu and Gärtner, 2010) has
been treated in preference learning as an extension of traditional supervised classi ication, where the objective is to obtain a ranker which gives a (partial) orderof labels for each datum. SELF is the irst algorithm that treats label ranking of
mixed-type data by weighting each classi ication rule through closed set lattices.

6.2

The SELF Algorithm

We present the SELF algorithm in this section, which is the main part of this chapter. The behavior of SELF is illustrated in Figure 6.1; irst it performs data preprocessing to make a context from a given dataset, second it constructs concept
lattices by FCA, and third it learns the preference for each class label. Notations
about databases are same as those in Chapter 5 (see Subsection 5.1.1).

6.2.1 Data Preprocessing
The aim of data preprocessing is to construct a (formal) context, a binary matrix
specifying a set of objects and their attributes, to apply FCA to training data. Since
we are interested in mixed-type data, we consider two types of variables, discreteand continuous. Formally, if a feature ℎ ∈ 𝐻 is discrete, Dom(ℎ) = 𝑆 ∪ {⊥} for somecountable set 𝑆. For instance, 𝑆 = {T, F} if the feature ℎ is binary and 𝑆 is a ( inite)
set of symbols if 𝑗 is nominal (categorical). If ℎ is continuous, Dom(ℎ) = ℝ ∪ {⊥},
where ℝ is the set of real numbers.
De inition 6.1: Context
In FCA, we call a triplet (𝐺, 𝑀, 𝐼) context. Here 𝐺 and 𝑀 are sets and 𝐼 ⊆ 𝐺 × 𝑀 context
is a binary relation between 𝐺 and 𝑀. The elements in 𝐺 are called objects, andthose in 𝑀 are called attributes.
For a given table 𝜏 = (𝐻, 𝑋), we always identify the set of objects 𝐺 with set(𝑋) ={𝑥 , 𝑥 , … , 𝑥 }.
In the data preprocessing, for each feature ℎ ∈ 𝐻 of a table 𝜏, we independently
construct a context (𝐺, 𝑀 , 𝐼 ) and combine them into a context (𝐺, 𝑀, 𝐼). For thisprocess, we always qualify attributes to be disjoint by denoting each element 𝑚 qualifyof the attribute 𝑀 by ℎ.𝑚 following the notations used in the database systems
literature (Garcia-Molina et al., 2008).
First, we focus on preprocessing for discrete variables. Since a context is alsoa discrete representation of a dataset, this process is directly achieved as follows:
For each feature ℎ, the set of attributes
𝑀 = { ℎ.𝑚 𝑚 ∈ Dom(ℎ) ⧵ {⊥} }and, for each value 𝑥 (ℎ),(𝑥 , ℎ.𝑚) ∈ 𝐼 if and only if 𝑥 (ℎ) = 𝑚.
In this way, discrete values are translated into a context and missing values arenaturally treated. Algorithm 6.1 performs this translation.

99

100

SEMI-SUPERVISED CLASSIFICATION AND RANKING

Algorithm 6.1: Data preprocessing for discrete variables
Input: Table 𝜏 = (𝐻, 𝑋) whose variables are discrete
Output: Context (𝐺, 𝑀 , 𝐼 )function C
D(𝜏)1: 𝐺 ← set(𝑋)2: for each ℎ ∈ 𝐻
3:𝑀 ← { ℎ.𝑚 ∣ 𝑚 ∈ Dom(ℎ) ⧵ {⊥} }4:
𝐼 ← { (𝑥, ℎ.𝑥(ℎ)) ∣ 𝑥 ∈ 𝐺 and 𝑥(ℎ) ≠ ⊥ }5: end for
6: combine all (𝐺, 𝑀 , 𝐼 ) with ℎ ∈ 𝐻 into (𝐺, 𝑀 , 𝐼 )7: return (𝐺, 𝑀 , 𝐼 )Example 6.2
Given a table 𝜏 = (𝐻, 𝑋) with 𝐻 = {1, 2, 3} and 𝑋 = 𝑥 , 𝑥 such that(𝑥 (1), 𝑥 (2), 𝑥 (3)) = (T, ⊥, C),(𝑥 (1), 𝑥 (2), 𝑥 (3)) = (F, F, ⊥).
This table can be represented in the following manner.
𝐻
𝑋

𝑥
𝑥

1

2

3

T
F

⊥
F

C
⊥

The domains are given as Dom(1) = Dom(2) = {T, F} and Dom(3) = {A, B, C}.
Here we have
𝐺 = {𝑥 , 𝑥 },(𝑀 , 𝐼 ) = ({1.T, 1.F}, {(1, 1.T), (𝑥 , 1.F)}),(𝑀 , 𝐼 ) = ({2.T, 2.F}, {(𝑥 , 2.F)}),(𝑀 , 𝐼 ) = ({3.A, 3.B, 3.C}, {(𝑥 , 3.C)}).
Thus we have the context (𝐺, 𝑀, 𝐼) such that
𝑀 = 𝑀 ∪ 𝑀 ∪ 𝑀 = {1.T, 1.F, 2.T, 2.F, 3.A, 3.B, 3.C},
𝐼 = 𝐼 ∪ 𝐼 ∪ 𝐼 = {(𝑥 , 1.T), (𝑥 , 3.C), (𝑥 , 1.F), (𝑥 , 2.F)}.
It is visualized as a cross-table as follows:
1.T
𝑥
𝑥

1.F

2.T

2.F

×

3.A

3.B

3.C
×

×

×

Second, we make a context from continuous variables using discretization. This
process is embedded in the learning process (see Figure 6.1) and discretizing resolution increases along with the process. The degree of resolution is denoted by a

6.2 THE SELF ALGORITHM

Algorithm 6.2: Data preprocessing for continuous variables
Input: Table 𝜏 = (𝐻, 𝑋) whose variables are continuous,
discretization level 𝑘
Output: Context (𝐺, 𝑀 , 𝐼 )function C
C(𝜏, 𝑘)1: 𝐺 ← set(𝑋)2: for each ℎ ∈ 𝐻
3:𝑀 ← { ℎ.1, ℎ.2, … , ℎ.2 }4:Normalize values in the feature ℎ by min-max normalization
5:
𝐼 ←∅
6:
for each 𝑥 ∈ 𝐺
7:
if 𝑥(ℎ) = 0 then 𝐼 ← 𝐼 ∪ {(𝑥, 1)}8:else if 𝑥(ℎ) ≠ 0 and 𝑥(ℎ) ≠ ⊥ then9:
𝐼 ← 𝐼 ∪ {(𝑥, ℎ.𝑎)}, where (𝑎 − 1)/2 < 𝑥(ℎ) ≤ 𝑎/2
10:
end if
11:
end for
12: end for
13: combine all (𝐺, 𝑀 , 𝐼 ) with ℎ ∈ 𝐻 into (𝐺, 𝑀 , 𝐼 )14: return (𝐺, 𝑀 , 𝐼 )natural number 𝑘, called discretization level and, in the following, we explain how discretization levelto discretize continuous variables at ixed level 𝑘. First we use min-max normalization (Han and Kamber, 2006) so that every datum is in the closed interval [0, 1].
For every feature ℎ, each value 𝑥(ℎ) is mapped to a value 𝑦(ℎ) such that𝑦(ℎ) =

𝑥(ℎ) − min ∈ ( ) 𝑥(ℎ).
max ∈ ( ) 𝑥(ℎ) − min ∈ ( ) 𝑥(ℎ)

Next, we discretize values in [0, 1] and make a context using the binary encodingof real numbers, following the approach we have used in the previous chapters. At
discretization level 𝑘, 𝑀 for a feature ℎ ∈ 𝐻 is always the set {ℎ.1, ℎ.2, … , ℎ.2 }.
For each value 𝑥 (ℎ), if 𝑥 (ℎ) = 0, then (𝑥 , ℎ.1) ∈ 𝐼 . Otherwise if 𝑥 (ℎ) ≠ 0, then(𝑥 , ℎ.𝑎) ∈ 𝐼 if and only if
𝑎−1< 𝑥 (ℎ) ≤𝑎.
2
2
If 𝑥 (ℎ) = ⊥, then (𝑥 , 𝑚) ∉ 𝐼 for all 𝑚 ∈ 𝑀 . This means that if we encode thevalue 𝑥 (ℎ) as an in inite sequence 𝑝 = 𝑝 𝑝 𝑝 … , a context at level 𝑘 is decided
by the irst 𝑘 bits 𝑝 𝑝 … 𝑝 . Each value is converted to exactly one relation ofa context if it is not missing. Algorithm 2 shows the above process for making acontext from continuous variables.
Example 6.3
Given a table 𝜏 = (𝐻, 𝑋) with 𝐻 = {1, 2, 3, 4} and 𝑋 = 𝑥 , 𝑥 such that(𝑥 (1), 𝑥 (2), 𝑥 (3)) = (T, C, 0.35, 0.78),(𝑥 (1), 𝑥 (2), 𝑥 (3)) = (⊥, ⊥, 0.813, ⊥).

101

102

SEMI-SUPERVISED CLASSIFICATION AND RANKING
It can be represented as follows:
𝐻
𝑋

𝑥
𝑥

1

2

3

4

T
⊥

C
⊥

0.35
0.813

0.78
⊥

where the irst and second features are discrete with Dom(1) = {T, F} and Dom(2)= {A, B, C}, and the third and forth are continuous. Assume that discretization level𝑘 = 1. We have
𝐺 = {𝑥 , 𝑥 },(𝑀 , 𝐼 ) = ({1.T, 1.F}, {(𝑥 , 1.T)}),(𝑀 , 𝐼 ) = ({2.A, 2.B, 2.C}, {(𝑥 , 2.C)}),(𝑀 , 𝐼 ) = ({3.1, 3.2}, {(𝑥 , 3.1), (𝑥 , 3.2)}),(𝑀 , 𝐼 ) = ({4.1, 4.2}, {(𝑥 , 4.2)}).
Thus we have the context (𝐺, 𝑀, 𝐼) such that 𝑀 = 𝑀 ∪ 𝑀 ∪ 𝑀 ∪ 𝑀 and 𝐼 =𝐼 ∪ 𝐼 ∪ 𝐼 ∪ 𝐼 , which is visualized as a cross-table as follows:
1.T
𝑥
𝑥

×

1.F

2.A

2.B

2.C

3.1

×

×

3.2

4.1

4.2
×

×

6.2.2 Clustering and Making Lattices by FCA
From a context obtained by the data preprocessing, we generate closed sets asclusters of data points and construct closed set lattices by FCA. First we summarize FCA (see literatures (Davey and Priestley, 2002; Ganter and Wille, 1998) for
detail). We always assume that a given table 𝜏 is converted into a context (𝐺, 𝑀, 𝐼)
by Algorithms 6.1 and 6.2.
De inition 6.4: Concept
For subsets 𝐴 ⊆ 𝐺 and 𝐵 ⊆ 𝑀, we de ine
𝐴′ ≔ {𝑚 ∈ 𝑀 ∣ (𝑔, 𝑚) ∈ 𝐼 for all 𝑔 ∈ 𝐴},
𝐵′ ≔ {𝑔 ∈ 𝐺 ∣ (𝑔, 𝑚) ∈ 𝐼 for all 𝑚 ∈ 𝐵}.
concept, extent, intent
Galois connection
closedconcept latticeThen, a pair (𝐴, 𝐵) with 𝐴 ⊆ 𝐺 and 𝐵 ⊆ 𝑀 is called a concept of a context (𝐺, 𝑀, 𝐼)
if 𝐴′ = 𝐵 and 𝐴 = 𝐵′. The set 𝐴 is called an extent and 𝐵 an intent.
Each operator ′ is a Galois connection between the power set lattices on 𝐺 and𝑀, respectively, hence the mapping ′′ becomes a closure operator on the context(𝐺, 𝑀, 𝐼). This means that, for each concept (𝐴, 𝐵), 𝐴 and 𝐵 are (algebraic) closed
sets. Note that a subset 𝐴 ⊆ 𝐺 (resp. 𝐵 ⊆ 𝑀) is the extent (resp. intent) of someconcept if and only if 𝐴′′ = 𝐴 (resp. 𝐵′′ = 𝐵). Thus a set of objects 𝐴 ⊆ 𝐺 forms acluster if and only if 𝐴′′ = 𝐴. Each object usually belongs to more than one cluster,hence this method is not “crisp” clustering.
The set of concepts over (𝐺, 𝑀, 𝐼) is written by 𝔅(𝐺, 𝑀, 𝐼) and called the concept lattice. If we focus on either one of the set of objects or attributes, this lat-6.2 THE SELF ALGORITHM

103

tice is called the closed set lattice. In particular, in the context of frequent pattern closed set latticemining, a set of attributes corresponds to an itemset and the lattice is called theclosed itemset lattice. For a pair of concepts (𝐴 , 𝐵 ) ∈ 𝔅(𝐺, 𝑀, 𝐼) and (𝐴 , 𝐵 ) ∈𝔅(𝐺, 𝑀, 𝐼), we write (𝐴 , 𝐵 ) ≤ (𝐴 , 𝐵 ) if 𝐴 ⊆ 𝐴 . Then (𝐴 , 𝐵 ) ≤ (𝐴 , 𝐵 )holds if and only if 𝐴 ⊆ 𝐴 (and if and only if 𝐵 ⊇ 𝐵 ). This relation ≤ becomesan order on 𝔅(𝐺, 𝑀, 𝐼) in the mathematical sense and ⟨𝔅(𝐺, 𝑀, 𝐼), ≤⟩ becomes acomplete lattice.
Let 𝒞 ⊆ 𝔅(𝐺, 𝑀, 𝐼). A concept (𝐴, 𝐵) ∈ 𝒞 is a maximal element of 𝒞 if (𝐴, 𝐵) ≤(𝑋, 𝑌) and (𝑋, 𝑌) ∈ 𝒞 imply (𝐴, 𝐵) = (𝑋, 𝑌) for all (𝑋, 𝑌) ∈ 𝒞. We write the set of
maximal elements of 𝒞 by Max𝒞.
Many algorithms are available for constructing closed set lattices, or concept
lattices, and the algorithm proposed by Makino and Uno (2004) is known to be
one of the fastest algorithms. Their algorithm enumerates all maximal bipartite
cliques in a bipartite graph with 𝑂(Δ ) delay, where Δ is the maximum degree ofthe given bipartite graph, that is,
Δ = max #𝐽 𝐽 ⊆ 𝐼, where𝑔 = ℎ for all (𝑔, 𝑚), (ℎ, 𝑙) ∈ 𝐽, or𝑚 = 𝑙 for all (𝑔, 𝑚), (ℎ, 𝑙) ∈ 𝐽(#𝐽 is the number of elements in 𝐽) in the FCA context. Since we can easily checkthat each context and concept exactly coincide with a bipartite graph and a maximal bipartite clique, respectively (Figure 6.2), we can use their algorithm directly.
For empirical experiments, we use the program LCM by Uno et al. (2005) provided
by the authors to enumerate all concepts and construct the closed set lattice.
Example 6.5
Given the following context:

𝑥
𝑥
𝑥
𝑥

1

2

×

×
×

3

4

5

×
×

×

×
×

×

x1

1

x2

2

x3

3

x4

4

There exist eight concepts in total;

5(∅, {1, 2, 3, 4, 5}), ({𝑥 }, {1, 2, 4}), ({𝑥 }, {2, 4, 5}), ({𝑥 }, {3}),({𝑥 , 𝑥 }, {2, 4}), ({𝑥 , 𝑥 }, {2, 5}), ({𝑥 , 𝑥 , 𝑥 }, {2}), and ({𝑥 , 𝑥 , 𝑥 , 𝑥 }, ∅),Figure 6.2 | The bipartite graph corresponding to the context in Example 6.5.

and Δ = 3. We show the closed set lattice in Figure 6.3. Let
𝒞=(∅, {1, 2, 3, 4, 5}), ({𝑥 }, {1, 2, 4}), ({𝑥 }, {2, 4, 5}),({𝑥 , 𝑥 }, {2, 4}), ({𝑥 , 𝑥 }, {2, 5})}.

Then max 𝒞 = {({𝑥 }, {1, 2, 4}), ({𝑥 , 𝑥 }, {2, 4}), ({𝑥 , 𝑥 }, {2, 5})}.

6.2.3 Learning Classi ication Rules
Here we present the main learning algorithm of SELF in Algorithm 3, which obtains
a set of classi ication rules from a table 𝜏 for training. In this chapter, a classi ication rule is a pair of a set of attributes and a label since, intuitively, every unlabeled classiﬁcation rule104

SEMI-SUPERVISED CLASSIFICATION AND RANKING∅
x1, x2, x3, x4
2x1, x2, x4
2, 5

2, 4
x1, x2x2, x43

Figure 6.3 | The closed set lattice(concept lattice) constructed from thecontext given in Example 6.5. In thisdiagram, each dot denotes a concept,
which are treated as a cluster in SELF.

label

2, 4, 5x2

x3

1, 2, 4x1

1, 2, 3, 4, 5∅

tuple (datum) is classi ied to the associated label if it has the same attributes. SELF
generates a set of classi ication rules at each discretization level. We give the precise algorithm of classi ication in the next subsection.
We introduce some notations. For each object 𝑔 ∈ 𝐺, we denote a label, an
identi ier of a class, of 𝑔 by Λ(𝑔), and if 𝑔 is unlabeled; i.e., the label information is
missing, we write Λ(𝑔) = ⊥. Moreover, we de ine
Γ(𝐺) ≔ { 𝑔 ∈ 𝐺 Λ(𝑔) ≠ ⊥ }hence objects in Γ(𝐺) are labeled and those in 𝐺 ⧵ Γ(𝐺) are unlabeled.

l-consistent

De inition 6.6: l-consistency
For a concept (𝐴, 𝐵) ∈ 𝔅(𝐺, 𝑀, 𝐼), we say that it is 𝑙-consistent if Γ(𝐴) ≠ ∅ andΛ(𝑔) = Λ(ℎ) for all 𝑔, ℎ ∈ Γ(𝐴).
Note that a concept with Γ(𝐴) = ∅ (all labels are missing) is not 𝑙-consistent.
First SELF performs data preprocessing and makes the context (𝐺, 𝑀, 𝐼) from agiven table at each discretization level 𝑘 using the algorithms given in Section 6.2.1.
Second it constructs the concept lattice 𝔅(𝐺, 𝑀, 𝐼) using both labeled and unlabeled tuples and inds 𝑙-consistent concepts using labeled tuples (objects). Third
it outputs the sets of classi ication rules such thatℛ = { (𝐵, 𝜆) (𝐴, 𝐵) ∈ Max𝒞 and 𝜆 = Λ(𝑔) with 𝑔 ∈ Γ(𝐴) } ,where

𝒞 = { (𝐴, 𝐵) ∈ 𝔅(𝐺, 𝑀, 𝐼) (𝐴, 𝐵) is 𝑙-consistent }at discretization level 𝑘. The lattice enables us to avoid over itting since, informally,
attributes of maximal concepts correspond to the most general classi ication rules.
If some objects that are not contained in 𝑙-consistent concepts remains, it re ines
discretization; i.e., increases discretization level, and repeats the above procedure
for the remaining objects.
Moreover, SELF weights each classi ication rule. For a classi ication rule 𝑅 =(𝐵, 𝜆), the weight 𝜔(𝑅) is de ined as follows:
𝜔(𝑅) ≔ # { (𝐶, 𝐷) ∈ 𝔅(𝐺, 𝑀, 𝐼) 𝐷 ⊇ 𝐵 } .

6.2 THE SELF ALGORITHM

Algorithm 6.3: SELF algorithm
Input: Table 𝜏 = (𝐻, 𝑋)
Output: Classi ication rules ℛ , ℛ , … , ℛ
function M (𝜏)1: Divide 𝜏 vertically into two tables 𝜏 and 𝜏 , where 𝜏 contains all
discrete variables in 𝜏 and 𝜏 contains all continuous variables in 𝜏
2: (𝐺, 𝑀 , 𝐼 ) ← C
D(𝜏 )// make a context from discrete variables of 𝜏
3: 𝑘 ← 1 // 𝑘 is discretization level
4: L(𝜏 , 𝐺, 𝑀 , 𝐼 , 𝑘) // use this function recursivelyfunction L(𝜏 , 𝐺, 𝑀 , 𝐼 , 𝑘)
1: (𝐺, 𝑀 , 𝐼 ) ← CONTEXTC(𝜏 , 𝑘)// make a context from continuous variables of 𝜏 at level 𝑘
2: make (𝐺, 𝑀, 𝐼) from (𝐺, 𝑀 , 𝐼 ) and (𝐺, 𝑀 , 𝐼 )3: construct the concept lattice 𝔅(𝐺, 𝑀, 𝐼) from (𝐺, 𝑀, 𝐼)
4: 𝒞 ← { (𝐴, 𝐵) ∈ 𝔅(𝐺, 𝑀, 𝐼) ∣ (𝐴, 𝐵) is 𝑙-consistent }5: ℛ ← { (𝐵, Λ(𝑔)) ∣ (𝐴, 𝐵) ∈ Max𝒞 and 𝑔 ∈ Γ(𝐴) }6: output ℛ
7: 𝐺 ← 𝐺 ⧵ { 𝑔 ∣ 𝑔 ∈ 𝐴 for some (𝐴, 𝐵) ∈ 𝒞 }8: remove corresponding attributes and relations from 𝑀 and 𝐼
9: remove corresponding tuples from 𝜏
10: if Γ(𝐺) = ∅ then halt
11: else L(𝜏 , 𝐺, 𝑀 , 𝐼 , 𝑘 + 1)
12: end if

Intuitively, the weight of a rule 𝑅 means its importance since it is the number ofclusters classi ied by the rule. Using the weight of rules, label ranking is achieved(see the next subsection).
Example 6.7
Given a dataset 𝜏 = (𝐻, 𝑋) and its labels as follows:

𝐻

𝑋

𝑥
𝑥
𝑥
𝑥
𝑥

1

2

3

Label

T
F
T
F
T

C
A
B
A
C

0.28
0.54
⊥
0.79
0.81

1
1
⊥
2
⊥

where Dom(1) = {T, F} ∪ {⊥}, Dom(2) = {A, B, C} ∪ {⊥}, and Dom(3) = ℝ ∪ {⊥}.
At discretization level 1, we have the following context:

105

106

SEMI-SUPERVISED CLASSIFICATION AND RANKING∅
x1, x2, x3, x4, x5

Level 11.T
x1, x3, x5
3.2

Figure 6.4 | The closed set lattices
x2, x4, x5(concept lattices) at discretization levels 1 and 2 constructed during the
1.F,2.A,3.2
learning phase in Example 6.7. In
x2, x4these diagrams, each black dot denotes the maximal l-consistent concept in the set of concepts covered by
the dotted line.

1.T
𝑥
𝑥
𝑥
𝑥
𝑥

1.F,2.Ax2, x4

Level 2

1.F,2.A,3.3
x4

1.T,2.C
x1, x5
1.T,2.C,3.2
x5

1.F,2.1,3.4
x4

1.T,1.F,2.A,2.B,2.C,
3.1,3.2,3.3,3.4∅

1.T,2.C,3.1 1.T,2.B
x1
x3

1.T,1.F,2.A,2.B,2.C,3.1,3.2
∅

1.F

2.A

×

×

2.B

×
×

2.C

3.1

×

×

3.2
×

×
×

×

×

×
×

×

We show the closed set lattice in the left-hand side in Figure 6.4. By SELF, we obtainℛ = {({1.T}, 1)} since the concept ({𝑥 , 𝑥 , 𝑥 }, {1.T}) is the maximal 𝑙-consistentconcept, and there is no 𝑙-consistent concept that contains 𝑥 or 𝑥 . This classi ication rule means “For a tuple 𝑥, if 𝑥(1) = T, then 𝑥 is classi ied to the class 1”. The
weight is calculated as 𝜔({1.T}, 1) = 6. SELF removes objects 𝑥 , 𝑥 , and 𝑥 contained in the 𝑙-consistent concepts and proceeds to the next level. At discretizationlevel 2, we have the following context:
1.T
𝑥
𝑥

1.F

2.A

×
×

×
×

2.B

2.C

3.1

3.2

3.3

3.4

×
×

The right-hand side in Figure 6.4 shows the closed set lattice of the above context,
and we obtain ℛ = {({1.F, 2.A, 3.3}, 1), ({1.F, 2.A, 3.4}, 2)}. For instance, the irst
rule means “For a tuple 𝑥, if 𝑥(1) = F, 𝑥(2) = A, and 0.5 < 𝑥(3) ≤ 0.75, its classlabel is 1”. The weight are 2 for both rules.
We show that SELF always stops in inite time if there are no con licting objects.
Namely, for a table 𝜏 = (𝐻, 𝑋), if there is no pair 𝑥, 𝑦 ∈ set(𝑋) such that Λ(𝑥) ≠ Λ(𝑦)and 𝑥(ℎ) = 𝑦(ℎ) for all ℎ ∈ 𝐻, Algorithm 3 stops in inite time. This statement isproved in the following way: if discretization level 𝑘 is large enough, we have theconcept lattice 𝔅(𝐺, 𝑀, 𝐼), where for every object 𝑥 ∈ 𝐺, there exists a concept(𝐴, 𝐵) such that 𝐴 = {𝑥} since there is no pair 𝑥, 𝑦 ∈ 𝐺 satisfying 𝑥(ℎ) = 𝑦(ℎ)
for all ℎ ∈ 𝐻. Thus each object 𝑥 with Λ(𝑥) ≠ ⊥ must be contained in some 𝑙consistent concept, and the algorithm stops. Note that the algorithm works evenif Γ(𝐺) = 𝐺; i.e., all objects have labels, hence it also can be viewed as a supervisedclassi ication method.

6.2 THE SELF ALGORITHM

Algorithm 6.4: Classi ication
Input: Classi ication rules ℛ , ℛ , … , ℛ , table 𝜐 = (𝐻, 𝑦), andthe set of labels ℒ
Output: Preference of each label
function C(ℛ , ℛ , … , ℛ , 𝜐)
1: Divide 𝜐 vertically into two tables 𝜐 and 𝜐 , where 𝜐 contains all
discrete variables in 𝜐 and 𝜐 contains all continuous variables in 𝜐
2: (𝐺, 𝑀 , 𝐼 ) ← C
D(𝜐 )// make a context from discrete values of 𝜐
3: for each 𝜆 ∈ ℒ
4:lPref(𝜆) ← 0
5:for each 𝑘 ∈ {1, 2, … , 𝐾}
6:
(𝐺, 𝑀 , 𝐼 ) ← C
C(𝜐 , 𝑘)// make a context from continuous values of 𝜐 at level 𝑘
7:
make a context (𝐺, 𝑀, 𝐼) from (𝐺, 𝑀 , 𝐼 ) and (𝐺, 𝑀 , 𝐼 )8:lPref(𝜆) ← lPref(𝜆) + ∑ ∈𝒬 𝜔(𝑅), where𝒬 = { (𝐵, 𝜆) ∈ ℛ ∣ (𝑦, 𝑏) ∈ 𝐼 for all 𝑏 ∈ 𝐵 }9:
end for10:
output lPref(𝜆)11: end for

The time complexity of learning by SELF is 𝑂(𝑛𝑑) + 𝑂(Δ 𝑁) such that𝑁=

max∈{ , ,…, }#𝔅(𝐺 , 𝑀 , 𝐼 ),where (𝐺 , 𝑀 , 𝐼 ) is the context at discretization level 𝑘 and 𝐾 is the level where
SELF stops since data preprocessing takes 𝑂(𝑛𝑑), making a concept lattice takes
less than 𝑂(Δ 𝑁), and obtaining classi ication rules takes less than 𝑂(𝑁).

6.2.4 Classi ication
Now we have sets of classi ication rules ℛ , ℛ , … , ℛ for each discretizationlevel from training mixed-type data including labeled and unlabeled data using
Algorithms 6.1, 6.2, and 6.3. In this section, we show how to classify a new unlabeled datum using the rules. We assume that such a new datum is given as a table𝜐 = (𝐻, 𝑦), where the body 𝑙-consists of only one tuple 𝑦.
Algorithm 6.4 performs classi ication using the obtained rules ℛ , ℛ , … , ℛ .
The algorithm is levelwise; i.e., at each level 𝑘, it makes a context (𝐺, 𝑀, 𝐼) from thetable 𝜐 = (𝐻, 𝑦) and apply the set of rules ℛ to it. Let ℒ be the domain of classlabels. SELF checks all rules in ℛ at each discretization level 𝑘 and, for each label
𝜆 ∈ ℒ, it outputs the preference of the label 𝜆.

107

108

SEMI-SUPERVISED CLASSIFICATION AND RANKING
De inition 6.8: l-preference
For a label 𝜆 ∈ ℒ, its 𝑙-preference is de ined asl-preferencelPref(𝜆) ≔

𝜔(𝑅),
∈𝒬

where
𝒬 = { (𝐵, 𝜆) ∈ ℛ(𝑦, 𝑏) ∈ 𝐼 for all 𝑏 ∈ 𝐵 }by summing up weights of rules.
Note that the set 𝐺 is always a singleton {𝑦} in the classi ication phase. The resultmeans that if lPref(𝜆) > lPref(𝜆′) for labels 𝜆 and 𝜆′, 𝜆 is preferable than 𝜆′, and viceversa, and if lPref(𝜆) = lPref(𝜆), the preference of 𝜆 and 𝜆′ are same, resulting in
the partial order over the set of labels ℒ. Thus the task of label ranking is performedby the preference lPref. Moreover, if we pick up the label
𝜆 ∈ argmax ∈ℒ lPref(𝜆),(6.1)

multiclass classi ication is also achieved directly.
Example 6.9
Let us consider the case discussed in Example 6.7. A tuple 𝑦 such that(𝑦(1), 𝑦(2), 𝑦(3)) = (T, B, 0.45)
satis ies only the rule ({1.T}, 1) ∈ ℛ . Thus we have lPref(1) = 6 and lPref(2) = 0
for labels 1 and 2, respectively. A tuple 𝑧 with(𝑧(1), 𝑧(2), 𝑧(3)) = (F, A, 0.64)
satis ies only the rule ({1.F, 2.A, 3.3}), hence lPref(1) = 0 and lPref(2) = 2.

6.3 Experiments
Here we empirically evaluate SELF. Our experiments consist of two parts: one isabout multiclass classi ication, and the other is about label ranking.

6.3.1 Methods
Environment
SELF was implemented in R version 2.12.1 (R Development Core Team, 2011) andall experiments were performed in the R environment. For enumeration of all concepts and construction of a closed set lattice from a context, we used LCM¹ distributed by Uno et al. (2005), which was implemented in C.
¹http://research.nii.ac.jp/~uno/codes.htm

6.3 EXPERIMENTS
Name# Data# Classes

abaloneallbp
annealarrhythmia
australiancrx
echocardiogram
heart
hepatitis
horse colic

4177
2800
798
452
690
690
131
270
155
368

28
3
5
13
2
2
2
2
2
2

109

# Features
Discrete Continuous
1
2
28
5
7
9
1
7
13
8

7
3
10
5
4
6
7
6
6
2

Table 6.1 | Statistics for datasets collected from UCI repository used for experiments.

Datasets
We collected ten mixed-type datasets from UCI repository (Frank and Asuncion,2010): abalone, allbp, anneal, arrhythmia, australian, crx, echocardiogram, heart,
hepatitis, and horse colic. Their basic statistics are summarized in Table 6.1. Datasets allbp, anneal, arrhythmia, australian, crx, echocardiogram, hepatitis, and horse
colic included missing values, which were directly treated in SELF. In other learning algorithms, we ignored all tuples which have missing values since they cannot
treat such datasets appropriately.
In label ranking, we used four datasets: abalone, allbp, anneal, and arrhythmia,
which have more than three classes. The other datasets had only two classes andcould not be used for label ranking evaluation.
Control Algorithms
In multiclass classi ication, three learning algorithms were adopted: the decisiontree-based classi ier implemented in R supplied in the tree package (Ripley, 1996),
SVM with the RBF kernel (𝐶 = 5 and 𝛾 = 0.05) in the kernlab package (Karatzoglou et al., 2004), and the 𝑘 nearest neighbor algorithm (𝑘 = 1 and 5) in theclass package. Notice that only the decision tree-based algorithm can treat mixedtype data directly, which is one of typical such learning algorithms. All discretevalues were treated as continuous in SVM and 𝑘NN.
Evaluation
In classi ication, for each dataset, the following procedure was repeated 20 times
and the mean and s.e.m. (standard error of the mean) of accuracy was obtained:
1) the number of labeled data or features was ixed, where the range was from 10
to 100 and 2 to 10, respectively, 2) labeled training data were sampled randomly,3) labels of the remaining data were predicted by respective learning algorithms,
and 4) the accuracy was obtained. The equation (6.1) was used to determine themost preferable label for each unlabeled datum. If there exists more than two suchlabels, we chose the smallest one.
We adopted two criteria: correctness and completeness, which was used in the correctness
literature (Cheng et al., 2010), to evaluate partial orders of labels in label ranking. completenessCorrectness coincides with the gamma rank correlation (Goodman and Kruskal,

110

SEMI-SUPERVISED CLASSIFICATION AND RANKING
1979), which is the normalized difference between the number of correctly ranked
pairs and that of incorrectly ranked pairs. Let ℒ be the set of class labels and wedenote by ≺∗ the ground truth of the partial order over the set of labels ℒ. Assume
that ≺ is a predicated partial order. Here we de ine
𝐶 ≔ # { (𝜆, 𝜆′) ∈ ℒ × ℒ 𝜆 ≺ 𝜆′ and 𝜆 ≺∗ 𝜆′ } ,𝐷 ≔ # { (𝜆, 𝜆′) ∈ ℒ × ℒ 𝜆 ≺ 𝜆′ and 𝜆′ ≺∗ 𝜆 } .
Then, the correctness is de ined byCR(≺, ≺∗ ) ≔

𝐶−𝐷
.
𝐶+𝐷

Trivially, the correctness takes a value in [−1, 1], and CR(≺, ≺∗ ) = 1 if ≺=≺∗ andCR(≺, ≺∗ ) = −1 if ≺ is the inversion of≺∗ . Thus the correctness should be maximized. Moreover, to evaluate the degree of completeness of a predicted partial
order, we use the completeness de ined as follows:CP(≺) ≔

𝐶+𝐷.
#{(𝜆, 𝜆′) ∈ ℒ × ℒ ∣ 𝜆 ≺∗ 𝜆′ or 𝜆′ ≺∗ 𝜆}The completeness takes a value in [0, 1] and should be maximized.

6.3.2 Results
Multiclass Classi ication
We evaluated SELF in multiclass classi ication. Speci ically, we examined SELF’s
behavior with respect to the number of labeled data and the number of features;
the number of labeled data was varied from 10 to 100, and the number of features from 2 to 10. When we ixed the number of labeled data, we used all features
for abalone, anneal, australian, crx, echocardiogram, heart, and hepatitis, and onlyused features 1, 2, 3, 18, 20 in allbp, 1, 2, … , 6, 22, 22, … , 25 in arrhythmia, and 1, 2,
4, 5, … 11 in horse colic, since we could not inish experiments in reasonable time
for such dense datasets. The above features seem to be representative for eachdataset. Otherwise if we ixed the number of features, we examined two cases inwhich the number of labeled data for training were 10 or 100. Such small amountof labeled data is typical in evaluation in semi-supervised learning; for example,
the numbers 10 and 100 were adopted in benchmarks in the literature¹ (Chapelle
et al., 2006, §21).
To analyze effectivity of unlabeled data in the semi-supervised manner, wetrained SELF in two ways; one is using both labeled and the remaining all unlabeled data for training, and the other is using only labeled data for training without any unlabeled data. In the following, we denote “SELF” in the former case and“SELF (w/o)” in the latter case. All experiments were carried out in the transductive setting (Vapnik and Sterin, 1977), that is, test data coincide with the unlabeledtraining data. This setting is common in empirical evaluation of semi-supervised
learning methods (Chapelle et al., 2006, §21).
¹This content is available at http://olivier.chapelle.cc/ssl-book/benchmarks.pdf.

6.3 EXPERIMENTS
For control, three learning algorithms were adopted: the decision tree-basedclassi ier, SVM with the RBF kernel, and the 𝑘 nearest neighbor algorithm (𝑘 = 1
and 5). All the above algorithms are typical for supervised learning and hence did
not use unlabeled data in training.
Figure 6.5 and Figures 6.6, 6.7 show the accuracy with respect to changes in
the number of labeled data and the number of features, respectively. In every case,the accuracy of SELF was much better than that of SELF (w/o), and the accuracy
was getting better according as the number of labeled data increases. Moreover,
SELF’s performance is getting better with increase in the number of features. SELF
therefore can effectively use unlabeled data and features for learning.
In comparison with the tree algorithm which can treat mixed-type data directly, SELF showed better performance in all datasets in Figure 6.5. Moreover,compared to other learning algorithms of SVM and 𝑘NN, SELF also achieved thebest performance in abalone, anneal, and horse colic. When the number of labeleddata is small (about 10 – 40), SELF outperformed other learning algorithms in all
datasets except allbp, as shown in Figures 6.5 and 6.6.

Label Ranking
We examined effectivity of SELF for label ranking. In consideration of the lack ofbenchmark data for label ranking, we adopted the following procedure for label
ranking: we trained SELF using all labeled data on the respective dataset and obtained the ranking for each datum, and used them as the ground truth. Cheng et al.
(2010); Hülermeier et al. (2008) who studied label ranking used the naïve Bayesclassi ier to make the ground truth of rankings from datasets. However, the mathematical theory is totally different from those of SELF, hence their approach is not
appropriate to our case.
Figure 6.8 shows the results of label ranking by SELF with varying the number of labeled data, and Figures 6.9 and 6.10 show those with respect to the number of features, where the number of labeled data is 10 for Figure 6.9 and 100 for
Figure 6.10. The correctness of SELF is better than SELF (w/o) in abalone, and issimilar between them in the other datasets for all conditions. In contrast, the completeness of SELF is much higher than that of SELF (w/o) in most cases. The main
reason might be that lots of data are not classi ied to any class in SELF (w/o).

6.3.3 Discussion
Our experiments about classi ication (Figures 6.5, 6.6, 6.7) show that SELF has
competitive performance compared to other machine learning algorithms, where
unlabeled data can be used effectively in training. This result means that dataclustering using the closed set lattices works well for semi-supervised learningof mixed-type data. Moreover, SELF can explicitly produce classi ication rules likethe decision tree-based algorithm, hence SELF’s results can be easily interpreted.
Furthermore, in label ranking (Figures 6.8 – 6.10), SELF outperformed SELF (w/o)
in most cases in terms of completeness, and the performance got higher with increase of the number of labeled data. Our results therefore show that unlabeleddata are also effectively used in SELF in the task of label ranking.

111

SEMI-SUPERVISED CLASSIFICATION AND RANKING

abaloneallbp

anneal
0.9

Accuracy

0.9
0.15

0.8

0.8

0.10
0.05

0.7

0.7

0.6

0.6

0.5

0.5

0.4
0.3

0.4
20

40

60

80

100

20

arrhythmia

Accuracy

0.4
0.3

20

0.5
80

100

20

40

60

80

100

20

0.8
0.7

0.6

0.6

0.5

0.5

0.5

0.4

80

100

40

60

80

100

80

100

0.6

0.4
60

100

hepatitis

0.7

0.7

40

80

0.5

heart

0.4

60

0.6

0.8

20

40

crx

0.7

echocardiogram

Accuracy

100

0.7

0.1
60

80

0.8

0.6

40

60

0.8

0.2

20

40

australian

0.5

0.3
20

40

60

80

Number of labeled data

100

20

40

60

Number of labeled data

horse_colic

Accuracy

112

0.75
0.70
0.65
0.60
0.55
0.50
0.45

SELF
SELF (w/o)
Tree
SVM
1−NN
5−NN
20

40

60

80

100

Number of labeled data

Figure 6.5 | Experimental results of accuracy for ten mixed-type datasets from UCI repository with varying thenumber of labeled data. We performed SELF using both labeled and unlabeled data (SELF) and using only labeleddata (SELF (w/o)), and compared them to the decision tree-based classiﬁer (Tree), SVM with the RBF kernel (SVM),
and the k-nearest neighbor algorithm (1-NN, 5-NN). Data show mean ± s.e.m.

6.3 EXPERIMENTS

abaloneallbp

Accuracy

0.10
0.08
0.06

0.9

0.8

0.8

0.6

0.7

0.4

0.6

0.04

0.2

0.5
2

4

6

8

10

0.0
2

4

arrhythmia

Accuracyanneal

6

8

10

4

australian

10

8

10

8

10

0.6

0.6

0.10

8

0.7

0.7

0.15

6

crx

0.5

0.5

0.4

0.05

0.3
2

Accuracy

2

4

6

8

10

2

4

echocardiogram

0.70
0.65
0.60
0.55
0.50
0.45
0.40
0.35

6

8

10

6

8

10

6

hepatitis
0.6
0.5

0.6

4

4

heart
0.7

2

2

0.4

0.5

0.3

0.4

0.2
2

4

6

8

Number of features

10

2

4

6

Number of features

horse colic

Accuracy

0.7
0.6

SELF
SELF (w/o)
Tree
SVM
1−NN
5−NN

0.5
0.4
0.3
2

4

6

8

10

Number of features

Figure 6.6 | Experimental results of accuracy for ten mixed-type datasets from UCI repository with varying thenumber of features. The number of labeled data was ﬁxed at 10 in each experiment. We performed SELF using bothlabeled and unlabeled data (SELF) and using only labeled data (SELF (w/o)), and compared them to the decisiontree-based classiﬁer (Tree), SVM with the RBF kernel (SVM), and the k-nearest neighbor algorithm (1-NN, 5-NN).
Data show mean ± s.e.m.

113

SEMI-SUPERVISED CLASSIFICATION AND RANKING

abaloneallbp

Accuracy

0.16
0.14
0.12
0.10
0.08
2

4

6

8

10

0.8
0.6
0.4
0.2
0.0
2

4

arrhythmia

6

8

10

10

4

Accuracy

echocardiogram

6

8

10

2

0.65

0.7

0.6

0.4

0.5

6

8

10

10

0.5

0.6

0.55

4

6

hepatitis
0.7

2

4

heart
0.8

0.50

8

0.3
2

0.70

0.60

10

0.4

0.3
8

8

0.5

0.4
6

10

0.6

0.5

0.2

8

0.7

0.6

0.3

6

crx

0.7

0.4

4

4

0.8

0.8

2

2

australian

0.5

Accuracyanneal

0.9
0.8
0.7
0.6
0.5
0.4
0.3

0.18

0.3
2

4

6

8

Number of features

10

2

4

6

Number of features

horse colic

0.8
0.7

Accuracy

114

0.6

SELF
SELF (w/o)
Tree
SVM
1−NN
5−NN

0.5
0.4
0.3
0.2
2

4

6

8

10

Number of features

Figure 6.7 | Experimental results of accuracy for ten mixed-type datasets from UCI repository with varying thenumber of features. The number of labeled data was ﬁxed at 100 in each experiment. We performed SELF using
both labeled and unlabeled data (SELF) and using only labeled data (SELF (w/o)), and compared them to the decision tree-based classiﬁer (Tree), SVM with the RBF kernel (SVM), and the k-nearest neighbor algorithm (1-NN,
5-NN). Data show mean ± s.e.m.

6.3 EXPERIMENTS

abalonallbp

Correctness

−0.4

0.95

−0.5

0.90

−0.6

0.85

−0.7

0.80

−0.8

0.75

SELF
SELF (w/o)

0.70

−0.9
20

40

60

80

100

20

anneal

40

60

80

100

80

100

80

100

arrhythmia
0.4

Correctness

0.85

0.2

0.80

0.0

0.75

−0.2
−0.4

0.70

−0.6
20

40

60

80

100

20

40

Completenessabalon

allbp

0.35
0.30
0.25
0.20
0.15
0.10
0.05

0.9
0.8
0.7
0.6
20

40

60

80

100

20

annealCompleteness

60

40

60

arrhythmia

0.9
0.8
0.7
0.6
0.5
0.4
0.3

0.7
0.6
0.5
0.4
0.3
0.2
0.1
20

40

60

80

Number of labeled data

100

20

40

60

80
100
SELF
SELF
(w/o)
Number of labeled data

Figure 6.8 | Experimental results of correctness and completeness (should be maximized) for four mixed-type
datasets from UCI repository with varying the number of labeled data. We performed SELF using both labeled andunlabeled data (SELF) and using only labeled data (SELF (w/o)), Data show mean ± s.e.m.

115

SEMI-SUPERVISED CLASSIFICATION AND RANKING

abalone

allbp

−0.65

0.8
0.6
0.4

Correctness

−0.70
−0.75
−0.80

0.2
0.0
−0.2

−0.85
−0.90
2

4

6

8

10

SELF
SELF (w/o)

2annealCorrectness

6

8

10

8

10

8

10

8
SELF

10

−0.4

0.4

−0.5

0.2

−0.6

0.0

−0.7

−0.2
2

4

6

8

10

abaloneCompleteness4

arrhythmia

0.6

0.30
0.25
0.20
0.15
0.10
2

4

6

2

4

8

10

6

allbp

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2

0.05

2

anneal4

6arrhythmia

0.8

Completeness

116

0.4

0.7
0.6

0.3

0.5

0.2

0.4

0.1

0.3
2

4

6

8

Number of features

10

2

4

6

Number of features
SELF (w/o)

Figure 6.9 | Experimental results of correctness and completeness (should be maximized) for mixed-type datasetsfrom UCI repository with varying the number of features. The number of labeled data was ﬁxed at 10 in eachexperiment. We performed SELF using both labeled and unlabeled data (SELF) and using only labeled data (SELF(w/o)), Data show mean ± s.e.m.

6.3 EXPERIMENTS

Correctness

abalone
−0.4

1.0

−0.5

0.5

−0.6

0.0

−0.7

−0.5

−0.8

allbp

SELF
SELF (w/o)

−1.0
2

4

6

8

10

2

annealCorrectness

8

10

8

10

8

10

8

10

0.4

0.8

0.3

0.7
0.6

0.2

0.5
0.1

0.4
2

4

6

8

10

abalone

2

4

6

allbp

1.0

0.35

Completeness6

arrhythmia

0.9

0.8

0.30
0.25

0.6

0.20

0.4

0.15

0.2

0.10

0.0

0.05
2

4

6

8

10

2

annealCompleteness

4

4

6

arrhythmia
0.70
0.65
0.60
0.55
0.50
0.45
0.40

0.90
0.85
0.80
0.75
0.70
2

4

6

8

Number of features

10

2

4

6

Number of features

Figure 6.10 | Experimental results of correctness and completeness (should be maximized) for mixed-type datasets from UCI repository with varying the number of features. The number of labeled data was ﬁxed at 100 in eachexperiment. We performed SELF using both labeled and unlabeled data (SELF) and using only labeled data (SELF(w/o)), Data show mean ± s.e.m.

117

118

SEMI-SUPERVISED CLASSIFICATION AND RANKING

6.4 Summary
We have proposed a novel semi-supervised learning algorithm, called SELF, for
mixed-type data including both discrete and continuous variables, and experimentally showed its competitive performance. The key strategy is data clustering withclosed set lattices using FCA, and the present study shows the effectivity of thelattices in semi-supervised learning. To our best knowledge, this approach is theirst direct semi-supervised method for mixed-type data, and also the irst one to
exploit closed set lattices in semi-supervised learning. Moreover, we can directly
treat missing values on SELF, meaning that SELF can be used for various practical
datasets. To date, many semi-supervised learning methods use data distribution
and probabilities, whereas SELF uses only the algebraic structure of data without
any background knowledge. Our results with lattice-based data analysis provide
new insight to machine learning and knowledge discovery.
There are two future works; one is analysis of SELF from the FCA point of view.
Re inement of discretization of continuous variables must have some connection
with reduction of a context (Ganter and Wille, 1998) since if we extend a context
by re ining real-valued variables, the original attributes are removed by reduction.
Moreover, ambiguity of data, such as intervals including truth values, might betreated using such techniques in FCA. Thereby analysis of mathematical connection between them is a future work. The other is theoretical analysis in the computational learning theory context. de Brecht and Yamamoto (2010) have proposed
Alexandrov concept space for learning from positive data. Our proposed method
might be an instance of the study, since the concept lattice is similar to the Alexandrov space. Thus theoretical analysis of our framework is also a future work.

7
LIGAND FINDING BY
MULTI-LABEL CLASSIFICATION

R

located at the surface of cells, which receive receptor
chemical signals from outside of the cells. Since receptors have crucial roles
for signal processing in organisms, to date, enormous studies have been devoted
to investigate their biochemical functions. The key approach in an experiment is to
use receptor speci icity with respect to a ligand, which triggers a cellular response ligand
by binding to a receptor, for controlling the receptor actions (Figure 7.1). However,inding new convenient ligands is dif icult; choosing ligand candidates relies on
expert knowledge of biologists and conducting experiments to test whether or notcandidates work in vivo or in vitro costs high in terms of time and money. Thus anin silico approach is required for helping biologists.
In this chapter, we adopt a machine learning, or knowledge discovery and data
mining, approach to ind candidates of ligands. Speci ically, we formulate the problem of ligand inding as multi-label classi ication recently discussed in the ield of multi-label classiﬁcationpreference learning (Fürnkranz and Hüllermeier, 2010), where each training da- preference learning
tum used for learning is associated with not a single class label but a set of possiblelabels. Here, for each ligand, receptors to which it binds correspond to class labelsof the ligand, and our goal is to predict labels (i.e., receptors) of ligands from databases of receptors and ligands. A ligand can often bind to more than two receptors; this is why our problem is not traditional single-label but multi-label classiication. Moreover, we try to predict labels in a semi-supervised manner (Chapelle semi-supervised learning
et al., 2006; Zhu and Goldberg, 2009). Semi-supervised learning is a special formof classi ication, where a learning algorithm uses both labeled and unlabeled datain training. Commonly, only few labeled data are assumed to be available since
the labeling task costs high in a real situation. Semi-supervised learning therefore
its to our goal since, in our problem, only few ligands for each receptor have been
discovered yet lots of ligands for other receptors are available.
Formally, the problem of semi-supervised multi-label classi ication is stated asfollows: Given a sequence 𝑋 = 𝑥 , 𝑥 , … , 𝑥 , where each 𝑥 is a tuple, or feature tuple
vector, and a domain of labels ℒ. Each tuple 𝑥 is associated with a set of labels feature vector
𝐿 ⊆ ℒ. Since we consider semi-supervised learning, 𝐿 = ∅ is allowed. The goal
is, for any tuple (test datum) 𝑦, to predict the preference of labels with respect to preference

120

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION

Ligandbinds

Receptor(ion channel)

Ions
Cell

Channel opens
Cell

Figure 7.1 | A ligand-gated ion channel, which is a typical receptor.

mixed-type data

Formal Concept Analysis (FCA)

concept latticeclosed set lattice𝑦, and we can decide whether or not 𝑦 is associated with a label 𝜆 for each 𝜆 ∈ ℒ.
Information about receptors and ligands is donated to various databases, suchas KEGG¹, and in this chapter we use the IUPHAR database (Sharman et al., 2011)².
In the database, every ligand is characterized by seven features as follows: hydrogen bond acceptors, hydrogen bond donors, rotatable bonds, topological polar
surface area, molecular weight, XLogP, and number of Lipinski’s rules broken. We
abbreviate them in this chapter HBA, HBD, RB, TPS, MW, XLogP, and NLR, respectively. Here, TPS, MW, and XLogP take continuous (real-valued) values while theothers, HBA, HBD, RB, and NLR, take discrete values. Thus to design an effectiveclassi ier for the IUPHAR database, we have to appropriately treat mixed-type data
including both discrete and continuous variables.
Recently, semi-supervised learning is one of active research ields in machinelearning and knowledge discovery, and now various semi-supervised learning algorithms have already been developed (Chapelle et al., 2006; Zhu and Goldberg,2009). However, most of them are designed for real-valued variables (not discretevariables) and cannot be applied to mixed-type data directly and, moreover, they
do not treat multi-label classi ication considered in this chapter. Therefore, in thischapter, we construct a new learning algorithm, called LIFT (Ligand FInding via
Formal ConcepT Analysis), which is designed for semi-supervised multi-label classi ication of mixed-type data and hence it solves the ligand inding problem fromthe IUPHAR database. The basic strategy of LIFT is similar to the learning method
SELF presented in the previous chapter, which directly handles mixed-type datain the semi-supervised manner. Since SELF cannot treat multi-label classi ication,
we redesign the essential algorithm of SELF to it to multi-label classi ication.
LIFT uses “label propagation”, or cluster-and-label, which is a typical approachin semi-supervised learning (Dara et al., 2002; Demiriz et al., 1999). This meansthat it irst makes clusters without label information, followed by giving preferences of class labels for each cluster. In LIFT, the clustering process is achieved
by Formal Concept Analysis (FCA) (Davey and Priestley, 2002; Ganter and Wille,
1998), which is a mathematical data analysis technique originally proposed byWille (1982). One of successful applications of FCA in data mining is for frequent
pattern and association rule mining proposed by Pasquier et al. (1999), whereclosed patterns (itemsets) obtained by FCA is used as condensed “lossless” representations of original patterns. Using FCA, informally, we can introduce a lattice
structure, called a concept lattice or a closed set lattice, which is a partially ordered
set of data clusters with respect to subset inclusion, into original data. Many studies used FCA for machine learning and knowledge discovery, such as classi ication¹http://www.genome.jp/kegg/²http://www.iuphar-db.org/index.jsp

7.1 THE LIFT ALGORITHM(Ganter and Kuznetsov, 2003), clustering (Zhang et al., 2008), and bioinformatics(Blinova et al., 2003; Kaytoue et al., 2011b; Kuznetsov and Samokhin, 2005), but
ligand inding presented in this chapter is a novel application of FCA.
To date, no study treats machine learning for ligand inding in the (multi-class)classi ication point of view. Recently, to the best of our knowledge, there exists
only one related study by Ballester and Mitchell (2010), which investigated a machine learning approach to predict the af inity of ligands, the strength of docking. aﬃnity
Another approach was performed by King et al. (1996) for modeling structureactivity relationships (SAR), which can be applied to ligand inding. However, theirgoal is to understand the chemical model by describing relations using inductive
logic programming (ILP), thus their approach is different from ours. On the other
hand, most in silico studies about receptors and ligands tried to construct a predictive model using domain-speci ic knowledge, such as the potential energy of acomplex, the two-dimensional co-ordinates, and the free energy of binding (Huang
et al., 2006; Moitessier et al., 2008), and lots of scoring methods were proposed;e.g., AMBER (Cornell et al., 1995), AutoDock (Huey et al., 2007), and DrugScore(Gohlke et al., 2000). However, to use such a method, some domain-speci ic background knowledge is required and results depend on them. In contrast, our approach relies on only databases, thereby the user does not need any backgroundknowledge and can easily use and understand results.
This chapter is organized as follows: Section 7.1 presents the LIFT algorithm.
Section 7.2 gives experimental results with methodologies and discussion. Finally
we summarize our results and discuss our future works in Section 7.3.

7.1

The LIFT Algorithm

We present the algorithm, LIFT (LIgand Finding via Formal ConcepT Analysis), in
this section. LIFT uses exactly the same algorithms for lattice construction from agiven database. Namely, Algorithms 6.1 and 6.2 are used for data preprocessing,and FCA is used for clustering (see Section 6.2.1 and 6.2.2 for detail). In the following, we write the concept lattice (the set of concepts) 𝔅(𝐺, 𝑀, 𝐼) generated from agiven table 𝜏 at discretization level 𝑘 by 𝔅 (𝜏).
Example 7.1
Given a table 𝜏 = (𝐻, 𝑋), where 𝐻 = {HBD, TPS, MW} and 𝑋 = 𝑥 , 𝑥 , 𝑥 such that(𝑥 (HBD), 𝑥 (TPS), 𝑥 (MW)) = (0, 0.61, 0.98),(𝑥 (HBD), 𝑥 (TPS), 𝑥 (MW)) = (0, 0.44, 0.74),(𝑥 (HBD), 𝑥 (TPS), 𝑥 (MW)) = (1, 0.72, 0.34),
which is visualized as follows:
𝐻
𝑋

𝑥
𝑥
𝑥

HBD

TPS

MW

0
0
1

0.61
0.44
0.72

0.98
0.74
0.34

121

122

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION∅
x1, x2, x3

H.0, M.2
x1, x2

T.2
x1, x3

H.0, T.1, M.2
x2

Figure 7.2 | The concept lattice constructed from the context in Example 7.1. Feature names HBD, TPS, andMW are abbreviated as H, T, and M, respectively.

H.0, T.2, M.2
x1

H.1, T.2, M.1
x3

H.0, H.1, T.1, T.2, M.1, M.2∅

Let discretization level 𝑘 = 1. For each feature, we have the context as follows:𝑀= {HBD.0, HBD.1},

𝐼𝑀𝐼= {(𝑥 , HBD.0), (𝑥 , HBD.0), (𝑥 , HBD.1)},= {TPS.1, TPS.2},= {(𝑥 , TPS.2), (𝑥 , TPS.1), (𝑥 , TPS.2)},

𝑀
𝐼= {MW.1, MW.2},= {(𝑥 , MW.2), (𝑥 , MW.2), (𝑥 , MW.1)}.

Thus we have the context (𝐺, 𝑀, 𝐼) such that𝐺 = {𝑥 , 𝑥 , 𝑥 },
𝑀=𝑀
∪𝑀
𝐼=𝐼

∪𝐼

∪𝑀
∪𝐼

,
,

which is visualized as a cross-table in the following.
HBD.0
𝑥
𝑥
𝑥

HBD.1

×
×

TPS.1

TPS.2

MW.1

×

×
×

×
×

×

MW.2

×

The set 𝔅 (𝜏) consists of seven concepts in total:(∅, {HBD.0, HBD.1, TPS.1, TPS.2, MW.1, MW.2}),({𝑥 }, {HBD.0, TPS.2, MW.2}), ({𝑥 }, {HBD.0, TPS.1, MW.2}),({𝑥 }, {HBD.1, TPS.2, MW.1}), ({𝑥 , 𝑥 }, {HBD.0, MW.2}),({𝑥 , 𝑥 }, {TPS.2}), 𝑎𝑛𝑑({𝑥 , 𝑥 , 𝑥 }, ∅).
We show the corresponding concept lattice in Figure 7.2.
Example 7.2
Let us consider a table 𝜏 = (𝐻, 𝑋) such that 𝐻 = {TPS, MW, XLogP} and 𝑋 =7.1 THE LIFT ALGORITHM

123

T.1
x1, x2, x3
T.1, X.2
x2, x3

T.1, M.1
x1, x2

T.1, M.2, X.2 T.1, M.1, X.2
x3
x2

T.1, M.1, X.1
x1

T.1, T.2, M.1, M.2, X.1, X.2∅

Figure 7.3 | The concept lattice fromthe context in Example 7.2 with its geometric interpretation.

𝑥 , 𝑥 , 𝑥 such that(𝑥 (TPS), 𝑥 (MW), 𝑥 (XLogP)) = (0.23, 0.12, 0.18),(𝑥 (TPS), 𝑥 (MW), 𝑥 (XLogP)) = (0.35, 0.03, 0.74),(𝑥 (TPS), 𝑥 (MW), 𝑥 (XLogP)) = (0.41, 0.79, 0.91),
which is represented as follows:
𝐻
𝑋

𝑥
𝑥
𝑥

TPS

MW

XLogP

0.23
0.35
0.41

0.12
0.03
0.79

0.18
0.74
0.91

The concept lattice (Figure 7.3) constructed from the context:
T.1
𝑥
𝑥
𝑥

×
×
×

T.2

M.1

M.2

×
×

X.1

X.2

×
×

×
×

corresponds to a hierarchy of cubes in three-dimensional Euclidean space.

7.1.1 Multi-label Classi ication and Ranking
Here we discuss classi ication on concept lattices using label information. Our
strategy is to design preference, a kind of weight, for each label of a given test datum preference(unlabeled tuple) 𝑦 based on concepts produced by FCA, and achieve multi-label
classi ication based on the preference. Moreover, we show that label ranking canbe performed using the preference.

124

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION

m-consistency

First LIFT translates a tuple 𝑦 of a table 𝜐 = (𝐻, 𝑦) into a context with just one
object using Algorithms 6.1 and 6.2; i.e., 𝐺 is the singleton {𝑦}. We always assume
that the header 𝐻 is exactly the same as that of a table 𝜏 = (𝐻, 𝑋) of training data.
The key idea is, for each concept (𝐴, 𝐵) ∈ 𝔅 (𝜏) obtained from a table 𝜏 oftraining data, to treat the set of attributes 𝐵 as a classi ication rule. For an unlabeled
tuple 𝑦, we check whether or not the object 𝑦 has the all attributes of the concept(𝐴, 𝐵), since this condition means that the object 𝑦 has the same properties of theobjects 𝐴, meaning that 𝑦 is classi ied to the same class of objects in 𝐴. We call thisproperty 𝑚-consistency which is formally de ined as follows:

m-consistent

De inition 7.3
For a context ({𝑦}, 𝑀, 𝐼) and a concept (𝐴, 𝐵), the object 𝑦 is 𝑚-consistent with
(𝐴, 𝐵) if both conditionsclassiﬁcation rule𝐵 ⊆ { 𝑚 ∈ 𝑀 (𝑦, 𝑚) ∈ 𝐼 } and 𝐵 ≠ ∅hold.
The notion of 𝑚-consistency has the monotonicity with respect to the order ≤on a concept lattice. If an object 𝑦 is 𝑚-consistent with a concept (𝐴, 𝐵), it is 𝑚consistent with any concept (𝐶, 𝐷) such that (𝐴, 𝐵) ≤ (𝐶, 𝐷), and if an object 𝑦
is not 𝑚-consistent with a concept (𝐴, 𝐵), it is not 𝑚-consistent with any concept(𝐶, 𝐷) such that (𝐶, 𝐷) ≤ (𝐴, 𝐵). Thus, for the set of concepts 𝔅 (𝜏), if we de ine
𝒞(𝑦) ≔ (𝐴, 𝐵) ∈ 𝔅 (𝜏) 𝑦 is 𝑚-consistent with (𝐴, 𝐵) ,there always exist inite concepts (𝐴 , 𝐵 ), (𝐴 , 𝐵 ), … , (𝐴 , 𝐵 ) such that↑(𝐴 , 𝐵 ) = 𝒞(𝑦),∈{ , ,…, }where↑(𝐴, 𝐵) = (𝐶, 𝐷) ∈ 𝔅 (𝜏) (𝐴, 𝐵) ≤ (𝐶, 𝐷) .
Here we give the formal de inition of the 𝑚-preference of a label. We denotethe set of labels associated with a tuple (object) 𝑥 by Λ(𝑥). Notice that Λ(𝑥) was asingle label in the previous chapter since we considered single-class classi ication,
but here it is a set of labels to take multi-class into account. Thereby, for a set oftuples (objects) 𝐴, Λ(𝐴) denotes the set ⋃ ∈ Λ(𝑥). LIFT allows unlabeled data fortraining, hence Λ(𝑥) could be empty, meaning that the object 𝑥 is unlabeled. This
is why LIFT is a semi-supervised learning algorithm.
De inition 7.4: m-preference at discretization level kGiven tables 𝜏 = (𝐻, 𝑋) and 𝜐 = (𝐻, 𝑦) with |𝜐| = 1. For each discretizationlevel 𝑘 and each label 𝜆 ∈ ℒ, we de ine the 𝑚-preference of 𝜆 at discretizationlevel 𝑘 with respect to the tuple 𝑦 by
m-preference at discretization level k
mPref (𝜆 ∣ 𝜏) ≔#Λ(𝐴)𝑦 is 𝑚-consistent with (𝐴, 𝐵) ∈ 𝔅 (𝜏)such that 𝜆 ∈ Λ(𝐴)where #Λ(𝐴) denotes the number of elements in Λ(𝐴), and we assume #Λ(𝐴)= 0 if #Λ(𝐴) = 0 for simplicity.

7.1 THE LIFT ALGORITHM
We do not take the size #𝐴 of the extent 𝐴 into account, since the distribution oftraining data with respect to labels is often biased, especially in biological data.
Example 7.5
Let 𝜏 = (𝐻, 𝑋) be a table in Example 7.2 (see Figure 7.3) and tables 𝜐 = (𝐻, 𝑦) and𝜎 = (𝐻, 𝑧) be(𝑦(TPS), 𝑦(MW), 𝑦(XLogP)) = (0.12, 0.41, 0.31),(𝑧(TPS), 𝑧(MW), 𝑧(XLogP)) = (0.31, 0.22, 0.89).
Assume that Λ(𝑥 ) = {A}, Λ(𝑥 ) = {B}, and Λ(𝑥 ) = {C} in 𝑋. Binary relations 𝐼
and 𝐼 at discretization level 1 for objects 𝑦 and 𝑧 are
𝐼 = {(𝑦, TPS.1), (𝑦, MW.1), (𝑦, XLogP.1)},
𝐼 = {(𝑧, TPS.1), (𝑧, MW.1), (𝑧, XLogP.2)}.
The object 𝑦 is 𝑚-consistent with three concepts ({𝑥 , 𝑥 , 𝑥 }, {TPS.1}), ({𝑥 , 𝑥 },{TPS.1, MW.1}), and ({𝑥 }, {TPS.1, MW.1, XLogP.1}), and 𝑧 is 𝑚-consistent withfour concepts ({𝑥 , 𝑥 , 𝑥 }, {TPS.1}), ({𝑥 , 𝑥 }, {TPS.1, MW.1}), ({𝑥 , 𝑥 }, {TPS.1,
XLogP.2}), and ({𝑥 }, {TPS.1, MW.1, XLogP.2}). Thus we have the 𝑚-preference
1 1
1 1+ + 1 = 1.83, mPref (B ∣ 𝜏) = + = 0.83,
3 2
3 2
1
mPref (C ∣ 𝜏) = = 0.33,
3
1 1
1 1 1
mPref (A ∣ 𝜏) = + = 0.83, mPref (B ∣ 𝜏) = + + + 1 = 2.33,
3 2
3 2 2
1 1
mPref (C ∣ 𝜏) = + = 0.83.
3 2
These results of 𝑚-preferences re lect the similarity between data, since 𝑦 and 𝑧
are most similar to the irst and second tuples of 𝑋, respectively.
mPref (A ∣ 𝜏) =It is easy to achieve multi-class classi ication from the 𝑚-preference at someixed discretization level. However, this 𝑚-preference would not be enough to exploit information from obtained data. We show a simple representative case in thefollowing, which shows the anti-monotonicity of the notion of 𝑚-consistency with anti-monotonicity
respect to discretization level.
Example 7.6
Let 𝜏 = (𝐻, 𝑋) be a table such that 𝐻 = {HBD, TPS} and 𝑋 = 𝑥 , 𝑥 , where(𝑥 (HBD), 𝑥 (TPS)) = (0, 0.56), Λ(𝑥 ) = {A}(𝑥 (HBD), 𝑥 (TPS)) = (0, 0.91), Λ(𝑥 ) = {B}and 𝜐 = (𝐻, 𝑦) be a table such that(𝑦(HBD), 𝑦(TPS)) = (0, 0.11),
which is shown as follows:
𝐻
𝑋
𝑦

𝑥
𝑥

HBD

TPS

Label

0
0

0.56
0.91

A
B

0

0.11

125

126

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION
At discretization level 1, we have the context
H.0
𝑥
𝑥

×
×

𝑦

×

T.1

T.2
×
×

×

and there are two concepts({𝑥 , 𝑥 }, {HBD.0, TPS.2}), (∅, {HBD.0, TSP.1, TPS.2}).
The object 𝑦 is not 𝑚-consistent with any concept, hencemPref (A ∣ 𝜏) = 0, mPref (B ∣ 𝜏) = 0.
However, at discretization level 2 with the following context,
H.0
𝑥
𝑥

×
×

𝑦

×

T.1

T.2

T.3

T.4

×
×
×

there are four concepts({𝑥 , 𝑥 }, {HBD.0}), ({𝑥 }, {HBD.0, TPS.3}),({𝑥 }, {HBD.0, TPS.4}), (∅, {HBD.0, TSP.1, TPS.2}),and 𝑦 is 𝑚-consistent with the concept ({𝑥 , 𝑥 }, {HBD.0}), thus
mPref (A ∣ 𝜏) = 0.5, mPref (B ∣ 𝜏) = 0.5.
Therefore 𝑦 can be classi ied to both classes A and B.
Ideally, every discretization levels should be taken into account to obtain the
𝑚-preference of labels. One of straightforward ways is to obtain the 𝑚-preferenceof a label by summing up 𝑚-preferences for each discretization level. However, if
we de ine the 𝑚-preference by
mPref (𝜆 ∣ 𝜏) ≔mPref (𝜆 ∣ 𝜏),

this 𝑚-preference goes to in inity in many cases. We therefore introduce the maximum level 𝑘of discretization as a parameter.
De inition 7.7: m-preference
Given tables 𝜏 and 𝜐, where |𝜐| = 1, and a natural number 𝑘
𝜆 ∈ ℒ, we de ine the 𝑚-preference of 𝜆 by
m-preferencemPref (𝜆 ∣ 𝜏) ≔for a tuple 𝑦.

mPref (𝜆 ∣ 𝜏)

. For each label

7.1 THE LIFT ALGORITHM

Algorithm 7.1: LIFT algorithm
Input: Tables 𝜏 = (𝐻, 𝑋) and 𝜐 = (𝐻, 𝑦), and maximum level 𝑘
Output: 𝑚-preference mPref for each label 𝜆 ∈ ℒfunction LIFT(𝜏, 𝜐, 𝑘)
1: 𝑘 ← 1 // 𝑘 is discretization level2: for each label 𝜆 ∈ ℒ
3:mPref (𝜆 ∣ 𝜏) ← 0 // initialization
4: end for5: return LEARNING(𝜏, 𝜐, 𝑘, 𝑘)function LEARNING(𝜏, 𝜐, 𝑘, 𝑘)
1: make a concept lattice 𝔅 (𝜏) from 𝜏 using Algorithms 6.1 and 6.2
2: for each label 𝜆 ∈ ℒ
3:compute the 𝑚-preference mPref (𝜆 ∣ 𝑋) at discretization level 𝑘
4:
mPref (𝜆 ∣ 𝑋) ← mPref (𝜆 ∣ 𝑋) + mPref (𝜆 ∣ 𝑋)
5: end for
6: if 𝑘 = 𝑘then
7:
return (mPref (𝜆 ∣ 𝜏)) ∈ℒ
8: else9:
return LEARNING(𝜏, 𝜐, 𝑘 + 1, 𝑘)
10: end if

We abbreviate “∣ 𝜏” of the expression mPref (𝜆 ∣ 𝜏) if it is understood from context.
We give the LIFT algorithm in Algorithm 7.1, which calculates the 𝑚-preference
for each label.
Example 7.8
Let us consider a table 𝜏 = (𝐻, 𝑋) with 𝐻 = {HBD, RB, TPS, MW}, where labels areassociated with each tuple, and a table 𝜐 = (𝐻, 𝑦) with an unlabeled tuple 𝑦, as
shown in the following.
𝐻
𝑋

𝑥
𝑥
𝑥

𝑦
Assume that 𝑘
H.0
𝑥
𝑥
𝑥

×

𝑦

×

HBD

TPS

MW

Labels

0
1
2

0.98
0.41
0.12

0.88
0.48
0.71

A
B
A

0

0.77

0.79

C
C

= 2. At discretization level 1 with the context
H.1

H.2

T.1

×

×
×

T.2

M.1

×
×

M.2
×

×
×
×

×

127

128

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION∅
x1, x2, x3

M.2
x1, x3
H.0, T.2, M.2

Figure 7.4 | Concept lattices con1
structed from contexts 𝔅 (τ) (left)2
and 𝔅 (τ) (right) in Example 7.8. The
tuple y is m-consistent with concepts
denoted by black dots.

∅
x1, x2, x3
H.0, T.4, M.4

T.1
x2, x3
H.2, T.1, M.2

x1

x1

x2

x3

H.0,H.1,H.2,T.1,T.2,T.3,
T.4,M.1,M.2,M.3,M.4

H.1, T.1, M.1

x3

H.1, T.2, M.2 H.2, T.1, M.3

x2∅

H.0, H.1, H.2, T.1, T.2, M.1, M.2∅

we have
mPref (A) = 1.5, mPref (B) = 0, and mPref (C) = 0.5,
since 𝑦 is 𝑚-consistent with two concepts(𝐴 , 𝐵 ) = ({𝑥 , 𝑥 }, {MW.2}) and(𝐴 , 𝐵 ) = ({𝑥 }, {HBD.0, TPS.2, MW.2}),
where Λ(𝐴 ) = {A, C} and Λ(𝐴 ) = {A} (see Figure 7.4). Remember that we always ignore the concept whose attribute is empty. At discretization level 2 withthe context
H.0
𝑥
𝑥
𝑥

×

𝑦

×

H.1

H.2

T.1

T.2

T.3

T.4

M.1

M.2

M.3

×
×

×
×

M.4
×

×

×

×
×

×

we have
mPref (A) = 1, mPref (𝐵) = 0, and mPref (C) = 0,
since 𝑦 is 𝑚-consistent with only one concept ({𝑥 }, {HBD.0, TPS.4, MW.4}). Finally
we have
mPref (A) = 2.5, mPref (B) = 0, and mPref (C) = 0.5
for each label.
From the 𝑚-preference obtained by LIFT, multi-label classi ication can be performed, that is, an unlabeled tuple 𝑦 is associated with a set of labels 𝐿 ⊆ ℒ suchthat 𝐿 = {𝜆 ∈ ℒ ∣ mPref (𝜆) ≠ 0}. Furthermore, a partial order ⪯ of labels
can be derived from 𝑚-preferences, where 𝜆 ⪯ 𝜆 (𝜆 is preferable than 𝜆 ) if
mPref (𝜆 ) ≤ mPref (𝜆 ). Thus we can also achieve the label ranking problem
using the 𝑚-preference.
The time complexity of LIFT is 𝑂(𝑛𝑑) + 𝑂(Δ 𝑁), which is same as that of SELF.

7.2 EXPERIMENTS
Example 7.9
For training and test data given in Example 7.8, labels A and C are associated with𝑦 since both mPref (A) and mPref (C) are larger then 0. Moreover, we have theorder B ≤ C ≤ A of label ranking for the tuple 𝑦.

7.2

Experiments

We evaluate the LIFT algorithm using real data of ligands and receptors comparedto SVM and the decision tree-based algorithm. We also experimentally measure
the effectiveness of unlabeled data for training in semi-supervised learning by LIFT.

7.2.1 Methods
Environment
LIFT was implemented in R and all experiments were performed in R version 2.12.2(R Development Core Team, 2011). LIFT uses LCM distributed by Uno et al. (2005)
to construct a concept lattice 𝔅 (𝜏), which was implemented in C. In all experiments, we used Mac OS X version 10.6.5 with two 2.26-GHz Quad-Core Intel Xeon
CPUs and 12 GB of memory.
Databases
We collected the entire 1,782 ligand data in the IUPHAR database (Sharman et al.,2011)³. In the database, each ligand is characterized by seven features: HBA, HBD,
RB, TPS, MW, XLogP, and NLR as described in introduction of this chapter. Receptors, which corresponds to class labels, are classi ied into families, such as 5Hydroxytryptamine receptors and Acetylcholine receptors, hence we picked up
the eleven largest families from the database and used respective families as datasets for each training. Statistics of receptor families is shown in Table 7.1. In
semi-supervised learning of LIFT, entire ligands except the focusing receptor family were used as unlabeled training data.
Learning Algorithms
To measure the effectiveness of unlabeled ligand data, we used LIFT in two cases:
only labeled data were used in training in the irst case (denoted by LIFT (w/o) in
Figure 7.5), and all ligands except test data were used as unlabeled training data inthe second case. The maximum level 𝑘
was set at 5 throughout all experiments.
As a control method for evaluation of LIFT, we adopted SVM with the RBF kernel
and the decision tree-based method implemented in R (Ripley, 1996), since thetree method is a typical learning algorithm which can be applied to mixed-type
data containing both discrete and continuous variables. We used the function ksvm
in the kernlab package for SVM (Karatzoglou et al., 2004), where all discrete values
are treated as continuous. Note that these control methods are typical supervised
learning methods and cannot use unlabeled data in the learning phase. Moreover,³http://www.iuphar-db.org/index.jsp

129

130

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION
Family name

Table 7.1 | Families of receptors. The
number of ligands and receptors correspond to the data size and the number of class labels, respectively.

# Ligands# Receptors

286
100
162
111
69
120
76
73
78
61
81

53
68
40
35
40
37
34
9
58
71
14

5-Hydroxytryptamine receptors
Acetylcholine receptors
Adenosine receptors
Adrenoceptors
Dopamine receptors
Histamine receptors
Neuropeptide Y receptors
Metabotropic glutamate receptors
Transient receptor potential channels
Voltage-gated potassium channels
Ionotropic glutamate receptors

since they are algorithms designed for single-label classi ication, we just used theirst label for each training datum.
Evaluation
Let 𝜐 = (𝐻, 𝑌) be a test table with 𝑌 = 𝑦 , 𝑦 , … , 𝑦 and the domain of labels ℒ be
{𝜆 , 𝜆 , … , 𝜆 }. Assume that we have the 𝑚-preference {mPref (𝜆 ∣ 𝜏) ∣ 1 ≤ 𝑖 ≤ 𝑙}for each label 𝜆 ∈ ℒ by LIFT, where
mPref (𝜆∣ 𝜏) ≥ mPref (𝜆∣ 𝜏) ≥ mPref (𝜆∣ 𝜏) ≥ … ≥ mPref (𝜆 ∣ 𝜏).

for each tuple 𝑦 ∈ set(𝑌). In LIFT, we de ine the accuracy acc(𝜐) by∑acc(𝜐) ≔# 𝜆 ∈ Λ(𝑦 ) 𝑗 ∈ {𝑝 , 𝑝 , … , 𝑝# (∑#Λ(𝑦 ))},

which takes values in [0, 1] and to be maximized. This means that when 𝑦 is associated with 𝑞 labels, we check whether or not each label is in top-𝑞 labels determined by the 𝑚-preference. Notice that we do not take labels 𝜆 # ( ) , … , 𝜆 into
account to obtain the accuracy since the database has only positive informationand 𝜆 ∉ Λ(𝑦) does not means that the ligand 𝑦 does not bind to the receptor 𝜆.
For the decision-tree based method and SVM, the accuracy is obtained byacc(𝜐) ≔#{𝑦 ∣ 1 ≤ 𝑖 ≤ 𝑛, 𝑓(𝑦 ) ∈ Λ(𝑦 )},𝑛where 𝑓(𝑦 ) is the output for the tuple 𝑦 by respective learning algorithms.
Mean and s.e.m. (standard error of the mean) of accuracy was obtained foreach dataset (receptor family) by 10-fold crossvalidation.

7.2.2 Results and Discussion
Results are shown in Figure 7.5. These results clearly show that LIFT is more effective than the typical classi ication algorithms of SVM and the tree algorithm forligand inding. Accuracy obtained by LIFT is signi icantly higher than that by SVM
and the tree algorithm in eight cases out of eleven cases (checked by paired 𝑡-test,

7.2 EXPERIMENTS

0.8

Accuracy

0.6
0.4
0.2
0

5-Hydroxytryptamine receptors

Acetylcholine
receptors

Adenosine
receptors

Adrenoceptors

Dopamine
receptors

Histamine
receptors

Neuropeptide Y
receptors

Metabotropic
glutamate receptors

0.8

Accuracy

0.6
0.4
0.2
0

0.8
LIFT (w/o)
LIFT
Tree
SVM

Accuracy

0.6
0.4* : p < 0.05
0.2
0Transient Receptor
Voltage-Gated Ionotropic glutamate
Potential Channels Potassium Channels
receptors

Figure 7.5 | Accuracy for each receptor family obtained by LIFT without unlabeled training data (LIFT (w/o)), LIFT,
the tree algorithm, and SVM with the RBF kernel. Data show mean ± s.e.m.

131

132

LIGAND FINDING BY MULTI-LABEL CLASSIFICATION𝛼 = 0.05). One of reasons of the difference might be that LIFT can treat multilabels effectively whereas SVM and the tree algorithm cannot. Moreover, SVM
treated discrete values as continuous, presumably resulting in lower accuracy.
Since each family has many classes from 9 to 71, accuracy of LIFT, which is morethan 50 % in most cases, is high enough. Furthermore, unlabeled training data canbe used effectively in LIFT in the semi-supervised manner. Our results therefore
indicate that LIFT should be valuable for inding new ligands and contribute tobiology and biochemistry.
By using LIFT, we can ind new ligand candidates from any training data, hence
LIFT can be used as a tool for actual biological experiments to narrow down newligand candidates. Checking such candidates obtained by LIFT in biological experiments is a future work.

7.3 Summary
In this chapter, we have proposed the semi-supervised learning algorithm, called
LIFT, for ligand inding from databases. LIFT performs preference learning, thatis, multi-label classi ication and ranking, in the semi-supervised manner. First,
every dataset is translated into a (formal) context, followed by clustering of it by
FCA by putting on a concept lattice, where each continuous (real-valued) value isdiscretized based on the binary encoding scheme. Then, on the lattice, the preferences (formally, 𝑚-preferences) of class labels for unlabeled test data are obtainedby taking labels of training data into account.
Since LIFT is a lexible learning algorithm, it can be applied to any databases invarious domains. Thus considering contributions to other domains is one of the
future works. Another future work is to treat incremental databases in LIFT, because lots of databases are frequently updated whereas LIFT cannot directly treatsuch incremental databases. LIFT can display weighted classi ication rules, which
are easily-interpreted, thus analysis of learned rules from biological point of viewis also a future work. Furthermore, using biological background knowledge suchas the structure of a receptor for learning is an interesting future work.

8
CONCLUSION

T

several approaches to computational learning. Namely,
Part I analyzed theories of computational learning, Part II developed machinelearning algorithms using discretization, and Part III presented preference learning algorithms using Formal Concept Analysis.
In Chapter 2, we presented learning of igures, nonempty compact sets in Euclidean space, based on the Gold-style learning model for a computable foundation
for binary classi ication of multivariate data. Encoding real vectors with no numerical error requires in inite sequences, resulting in the gap between each real vector
and its discretized representation used for the actual machine learning process.
Our motivation was to bridge the gap to overcome poor computational analysis
in binary classi ication as well as in other machine learning tasks such as regression and clustering. In the chapter, we amalgamated two processes; discretization
and binary classi ication. Each learning target, the set of real vectors classi ied as
positive, is treated as a igure. A learner receives discretized vectors as input dataand outputs a sequence of discrete representations of the target igure in the formof self-similar sets, known as fractals. The generalization error of each output ismeasured by the Hausdorff metric. Using this learning framework, we revealed
the hierarchy of learnabilities under various learning criteria in the track of traditional analysis of learnabilities in the Gold-style learning model. Moreover, we
showed a mathematical connection between machine learning and fractal geometry by measuring the complexity of learning using the Hausdorff dimension andthe VC dimension. Furthermore, we analyzed computability aspects of learning ofigures using the framework of Type-2 Theory of Effectivity (TTE).
In Chapter 3, we proposed a novel measure of the difference between two setsof data, called the coding divergence, and unify two processes discretization andlearning computationally. Discretization of continuous data was realized by a topological mapping (in the sense of mathematics) from the 𝑑-dimensional Euclidean
space ℝ into the Cantor space Σ , and the simplest model was learned in the Cantor space, which corresponds to the minimum open set separating the given two
sets of data. Furthermore, we constructed a classi ier using the divergence, andexperimentally illustrated robust performance of it.
We proposed in Chapter 4 approaches to exploit compression algorithms forclustering numerical data. Our irst contribution was to design a measure, called

134

CONCLUSION
the Minimum Code Length (MCL), that can score the quality of a given clusteringresult under the light of a ixed encoding scheme. Our second contribution was topropose a general strategy to translate any encoding method into a cluster algorithm, called COOL (COding-Oriented cLustering). COOL has a low computational
cost since it scales linearly with the data set size. The clustering results of COOL
were also shown to minimize MCL. To illustrate further this approach, we considered the Gray Code as the encoding scheme to present G-COOL. G-COOL can ind
clusters of arbitrary shapes and remove noise. Moreover, it is robust to change in
the input parameters; it requires only two lower bounds for the number of clusters and the size of each cluster, whereas most algorithms for inding arbitrarily
shaped clusters work well only if all parameters are tuned appropriately. G-COOL
was theoretically shown to achieve internal cohesion and external isolation andwas experimentally shown to work well for both synthetic and real data sets.
In Chapter 5, we presented a fast spatial clustering algorithm for multivariate
data, called BOOL (Binary cOding Oriented cLustering), which can detect arbitrarily shaped clusters and is noise tolerant. BOOL handles data using a two-step procedure: data points are irst discretized and represented as binary words; clusters
are then iteratively constructed by agglomerating smaller clusters using this representation. This latter step is carried out with linear complexity with respect tothe number of data points by sorting such binary representations, which resultsin dramatic speedups when compared with other clustering techniques. Experiments showed that BOOL is faster than K-means, and about two to three orders ofmagnitude faster than two state-of-the-art algorithms that can detect non-convex
clusters of arbitrary shapes. We also showed that results of BOOL are robust tochanges in parameters, whereas most algorithms for arbitrarily shaped clusters
are known to be overly sensitive to such changes. The key to the robustness of
BOOL is the hierarchical structure of clusters introduced automatically by increasing the precision of the discretization.
We proposed in Chapter 6 a new approach for semi-supervised learning using closed set lattices, which have been recently used for frequent pattern mining
within the framework of the data analysis technique of Formal Concept Analysis(FCA). We presented a learning algorithm, called SELF (SEmi-supervised Learning
via FCA), which performs as a multiclass classi ier and a label ranker for mixedtype data containing both discrete and continuous variables, whereas only few
learning algorithms such as the decision tree-based classi ier can directly handle
mixed-type data. From both labeled and unlabeled data, SELF constructs a closed
set lattice, which is a partially ordered set of data clusters with respect to subset inclusion, via FCA together with discretizing continuous variables, followed bylearning classi ication rules through inding maximal clusters on the lattice. Moreover, it can weight each classi ication rule using the lattice, which gives a partial
order of preference over class labels. We experimentally illustrated the competitive performance of SELF in classi ication and ranking compared to other learningalgorithms using UCI datasets.
To date, enormous studies have been devoted to investigate biochemical functions of receptors, which have crucial roles for signal processing in organisms. Ligands are key tools in experiments since receptor speci icity with respect to themenables us to control activity of receptors. However, inding ligands is dif icult;choosing ligand candidates relies on expert knowledge of biologists and conducting test experiments in vivo or in vitro has a high cost. In Chapter 7, we investigated the ligand inding problem with a machine learning approach by formalizing

CONCLUSION
the problem as multi-label classi ication mainly discussed in the area of preference
learning. We developed an algorithm, called LIFT (Ligand FInding via Formal ConcepT Analysis), for multi-label classi ication, which can treat ligand data in databases in a semi-supervised manner. The key to LIFT is to achieve clustering by
putting an original dataset on lattices using the data analysis technique of Formal
Concept Analysis (FCA), followed by obtaining the preference for each label using
the lattice structure. Experiments using real data of ligands and receptors in the
IUPHAR database showed that LIFT effectively solves the task compared to othermachine learning algorithms.
To conclude, we believe that our studies on computational learning will become
one of key approaches to machine learning for next generation. Discrete structure
manipulation is now becoming more and more important to effectively and ef iciently treat big data and structured data (Minato, 2011). This thesis is a bridge
between data analysis in the real world, mainly for multivariate data, and discreteand computational manipulations of such data on computers.

135

APPENDIX A
MATHEMATICAL BACKGROUNDM

of this thesis is summarized. Namely, sets, functions, topology, and metric space. See literatures (Dieudonné, 1960; Hopcroft
and Ullman, 1979) for complete explanation about the basic concepts.

A.1 Sets and Functions
Let 𝐴 and 𝐵 be sets. If all elements in 𝐴 are contained in 𝐵, we write 𝐴 ⊆ 𝐵. If both𝐴 ⊆ 𝐵 and 𝐴 ≠ 𝐵 hold, we write 𝐴 ⊂ 𝐵. The notation 𝐴 ⧵ 𝐵 means the relativecomplement of 𝐵 in 𝐴. The cardinality of a set 𝐴 is denoted by #𝐴, and the power
set of 𝐴 is denoted by 2 . The Cartesian product of two sets 𝐴 and 𝐵, denoted by
𝐴 × 𝐵, is the set of ordered pairs such that𝐴 × 𝐵 = { (𝑎, 𝑏) 𝑎 ∈ 𝐴 and 𝑏 ∈ 𝐵 } .
correspondence
sourcetargetgraph
image

A correspondence from a set 𝐴 to a set 𝐵 is a triplet 𝑓 = (𝐴, 𝐵, 𝐺(𝑓)), where𝐺(𝑓) ⊆ 𝐴 × 𝐵. The set 𝐴 is called the source, 𝐵 the target, and 𝐺(𝑓) the graph of 𝑓.
For a subset 𝑋 ⊆ 𝐴, the image of 𝑋 under 𝑓 is de ined byinverseand the inverse of 𝑓 is de ined by𝑓(𝑋) ≔ { 𝑏 ∈ 𝐵 (𝑎, 𝑏) ∈ 𝐺(𝑓) for some 𝑎 ∈ 𝑋 } ,𝑓domainrange≔ 𝐵, 𝐴, 𝐺(𝑓) , where 𝐺(𝑓) ≔ { (𝑏, 𝑎) (𝑎, 𝑏) ∈ 𝐺(𝑓) } .

The domain and the range of 𝑓 are written by dom(𝑓) and range(𝑓), respectively,
which are de ined asdom(𝑓) ≔ 𝑓 (𝐵),range(𝑓) ≔ 𝑓(𝐴).

partial function

A correspondence 𝑓 = (𝐴, 𝐵, 𝐺(𝑓)) is written by 𝑓 ∶⊆ 𝐴 ⇉ 𝐵 in general. In
particular, for a correspondence 𝑓 ∶⊆ 𝐴 ⇉ 𝐵, if an image 𝑓({𝑎}) has only one
elements for every 𝑎 ∈ dom(𝑓), 𝑓 is called a partial function from 𝐴 to 𝐵, written

A.2 TOPOLOGY AND METRIC SPACE
by 𝑓 ∶⊆ 𝐴 → 𝐵. Moreover, if dom(𝑓) = 𝐴 holds for a partial function 𝑓 ∶⊆ 𝐴 → 𝐵,
𝑓 is called a total function, written by 𝑓 ∶ 𝐴 → 𝐵. If 𝑓 is a total or partial function, total function
𝑓 is called a function or a mapping.
function, mapping
For a partial function 𝑓 ∶⊆ 𝐴 → 𝐵, we de ine
𝑓(𝑎) ≔

𝑓({𝑎}) if 𝑎 ∈ dom(𝑓),div
if 𝑎 ∉ dom(𝑓).

A.2 Topology and Metric Space
Let ℕ be the set of natural numbers including 0, ℚ the set of rational numbers, andℝ the set of real numbers. The set ℕ (resp. ℝ ) is the set of positive natural (resp.
real) numbers. The 𝑑 product of ℝ, ℝ × ℝ × … × ℝ, is denoted by ℝ . Note that ℝ
is exactly the same as ℝ. A point 𝑥 in the space ℝ is denoted by(𝑥 , 𝑥 , … , 𝑥 )

or

𝑥
⎡ ⎤
⎢𝑥 ⎥ .
⎢⋮⎥⎣𝑥 ⎦For any 𝑑 pairs of real numbers 𝑝 and 𝑞 with 𝑝 < 𝑞 (𝑖 = 1, … , 𝑑), the set ofelements (𝑥 , 𝑥 , … , 𝑥 ) of ℝ such that 𝑝 < 𝑥 < 𝑞 for all 𝑖 ∈ {1, 2, … , 𝑑} is called
an open interval, written by
open interval(𝑝 , 𝑞 ) × (𝑝 , 𝑞 ) × … × (𝑝 , 𝑞 );the set of elements (𝑥 , 𝑥 , … , 𝑥 ) of ℝ such that 𝑝 ≤ 𝑥 ≤ 𝑞 for all 𝑖 ∈ {1, 2, … , 𝑑}is called a closed interval, written byclosed interval[𝑝 , 𝑞 ] × [𝑝 , 𝑞 ] × … × [𝑝 , 𝑞 ].
Let 𝑋 be a set. A metric on 𝑋 is a function

metric𝔡∶𝑋×𝑋→ℝ
such that, for all 𝑥, 𝑦, 𝑧 ∈ 𝑋,
𝔡(𝑥, 𝑦) ≥ 0(non-negativity),𝔡(𝑥, 𝑦) = 0 if and only if 𝑥 = 𝑦𝔡(𝑥, 𝑦) = 𝔡(𝑦, 𝑥)(identity),(symmetry),𝔡(𝑥, 𝑧) ≤ 𝔡(𝑥, 𝑦) + 𝔡(𝑦, 𝑧)(triangle inequality).

A pair (𝑋, 𝔡) denotes a metric space 𝑋 together with 𝔡, that is, for all pairs of ele- metric spacements 𝑥, 𝑦 ∈ 𝑋, the distance between them is de ined by the metric 𝔡. One of themost popular metric spaces is the Euclidean space, where 𝑋 = ℝ and the metric Euclidean space𝔡 is the Euclidean metric 𝑑E such that
Euclidean metric𝑑E (𝑥, 𝑦) ≔(𝑥 − 𝑦 )

137

138

MATHEMATICAL BACKGROUND

Minkowski metric
𝐿 metric

for all 𝑥, 𝑦 ∈ ℝ with 𝑥 = (𝑥 , … , 𝑥 ) and 𝑦 = (𝑦 , … , 𝑦 ). Generally, the Minkowski
metric, or 𝐿 metric, is de ined as/𝑥 −𝑦𝑑 (𝑥, 𝑦) ≔Manhattan metric
Chebyshev metricMaximum metricball

If 𝑝 = 1, this metric coincides with the Manhattan metric, and if 𝑝 = 2, it with the
Euclidean metric. We also use the case 𝑝 = ∞ (e.g., Subsection 4.3.2), called the
Chebyshev metric of Maximum metric, where𝑑 (𝑥, 𝑦) ≔maxouter point
exterior
boundary point
boundary
closure𝑥 −𝑦 .

∈{ , ,…, }In the following, we consider the 𝑑-dimensional Euclidean space ℝ equippedwith the Euclidean metric 𝑑E . For a point 𝑎 ∈ ℝ and 𝜀 ∈ ℝ , the ball 𝐵(𝑎, 𝜀) of
radius 𝜀 centered at 𝑎 is de ined by
𝐵(𝑎, 𝜀) ≔ 𝑥 ∈ ℝ

inner point, neighborhood
open kernel.

𝑑E (𝑎, 𝑥) < 𝜀 .

For a subset 𝑆 ⊆ ℝ and its element 𝑎 ∈ 𝑆, if there exists 𝜀 such that 𝐵(𝑎, 𝜀) ⊆ 𝑆,
𝑎 is called an inner point of 𝑆, and 𝑆 is called a neighborhood of 𝑎. The set of inner∘
points of 𝑆 is said as the open kernel of 𝑆, denoted by 𝑆 . An inner point of ℝ ⧵ 𝑆 isan outer point of 𝑆. The set of outer points of 𝑆 is called the exterior of 𝑆, denoted
by 𝑆 . Moreover, a point in 𝑆 which is neither an inner point nor an outer point is∘said as a boundary point. The set of boundary points ℝ ⧵ (𝑆 ∪ 𝑆 ) is the boundaryof 𝑆, denoted by 𝑆 .
∘For a subset 𝑆 ⊆ ℝ , the set 𝑆 ∪ 𝑆 is called the closure of 𝑆, denoted by 𝑆. In
general,∘𝑆 ⊆𝑆⊆𝑆
holds. In particular, if∘

𝑆=𝑆 ,open set

we call 𝑆 an open set, and if
𝑆 = 𝑆,

closed set

we call it a closed set.
Let 𝑆 be a subset of ℝ . If for every arbitrary collection {𝑂 } ∈ of open setswith 𝐼 ⊆ ℕ such that
𝑆⊆

𝑂,
∈

there exists a inite subset {𝑖 , 𝑖 , … , 𝑖 } of 𝐼 such that𝑆⊆𝑂 ∪𝑂 ∪…∪𝑂 ,A.2 TOPOLOGY AND METRIC SPACE
then 𝑆 is called a compact set. The set of compact sets of ℝ is denoted by 𝒦, and compact set∗
the set of nonempty compact sets of ℝ is denoted by 𝒦 .
For a subset 𝑆 ⊆ ℝ , if 𝑆 ⊆ 𝐵(𝑎, 𝜀) for some ball 𝐵(𝑎, 𝜀), 𝑆 is said as bounded. In bounded
the Euclidean space ℝ , we can apply the well-known Heine–Borel theorem: For a Heine–Borel theorem
subset 𝑆 ⊆ ℝ , 𝑆 is compact if and only if 𝑆 is closed and bounded.
Let us consider a sequence (𝑥 ) ∈ℕ over 𝑋 in a metric space (𝑋, 𝔡). If for arbitrary 𝜀 ∈ ℝ there exists a natural number 𝑛 such that𝔡(𝑥 , 𝑥 ) < 𝜀
for all 𝑚, 𝑛 with 𝑚 > 𝑛 and 𝑛 > 𝑛 , (𝑥 ) ∈ℕ is called a Cauchy sequence. If every Cauchy sequence
Cauchy sequence has a limit, the metric space (𝑋, 𝔡) is said as complete.
complete
In a metric space (𝑋, 𝔡), the diameter of a nonempty subset 𝑆 is de ined bydiameter|𝑆| ≔ sup { 𝔡(𝑥, 𝑦) 𝑥, 𝑦 ∈ 𝑆 }for all 𝑥, 𝑦 ∈ 𝑆. A set 𝑆 is countable if there is a bijection from ℕ to 𝑆. A set 𝒰 is a countablecover of 𝑆 ⊆ 𝑆 if 𝒰 is countable and
cover
𝑈,

𝑆⊆
∈𝒰

and 𝒰 is a 𝛿-cover of 𝑋 if 𝒰 is a cover of 𝑋 and |𝑈| ≤ 𝛿 for all 𝑈 ∈ 𝒰.

𝛿-cover139

Symbols
Sets and Topological Spaces
ℕ
ℕ
ℚ
ℝ
ℝ
𝑑
ℝ
∗
𝒦
(𝑎, 𝑏)[𝑎, 𝑏]
ℐ
𝐾, 𝐿
ℱ
#𝑋
𝑑E
𝑑H

The set of natural numbers including 0
The set of positive natural numbers; i.e., ℕ = ℕ ⧵ {0}The set of rational numbers
The set of real numbers
The set of positive real numbers
The number of dimensions (𝑑 ∈ ℕ )𝑑-dimensional Euclidean space
The set of igures (nonempty compact subsets of ℝ )The open interval { 𝑥 ∈ ℝ ∣ 𝑎 < 𝑥 < 𝑏 }The closed interval { 𝑥 ∈ ℝ ∣ 𝑎 ≤ 𝑥 ≤ 𝑏 }The unit interval [0, 1] × … × [0, 1]
Figures (nonempty compact sets)
Set of Figures
The number of elements in set 𝑋
The Euclidean metric, cf. Section 2.2
The Hausdorff metric, cf. Section 2.2

Fractal Geometry and Dimensions
CT
𝐶
𝐂𝐓
ℌ
dimH
dimB
dimS
dimVC

Contraction for real numbers, cf. Section 2.2
Finite set of contractions, cf. Section 2.2
Contraction for igures, cf. Section 2.2
The Hausdorff measure, cf. Subsection 2.5.1
The Hausdorff dimension, cf. Subsection 2.5.1
The box-counting dimension, cf. Subsection 2.5.1
The similarity dimension, cf. Subsection 2.5.1
The VC dimension, cf. Subsection 2.5.1

StringsΣΣ∗ΣΣAlphabet
The set of inite sequences whose length are 𝑑;i.e., Σ = {𝑎 𝑎 … 𝑎 ∣ 𝑎 ∈ Σ}
The set of inite sequences
The set of inite sequences without the empty string 𝜆SYMBOLSΣ𝜆𝑢, 𝑣, 𝑤
𝑤⊑𝑝
↑𝑤
↑𝑊
⟨⋅⟩
|𝑤|
|𝑊|
diam(𝑘)𝑝, 𝑞
𝑉, 𝑊The set of in inite sequences (Cantor space)
The empty string
Finite sequences
𝑤 is a pre ix of 𝑝 (𝑤 ⊏ 𝑝 is 𝑤 ⊑ 𝑝 and 𝑤 ≠ 𝑝)
The set {𝑝 ∈ Σ ∣ 𝑤 ⊏ 𝑝}
The set {𝑝 ∈ Σ ∣ 𝑤 ⊑ 𝑝 for some 𝑤 ∈ 𝑊}, cf. Section 3.2
The tupling function;i.e., ⟨ 𝑝 , 𝑝 , … , 𝑝 ⟩ ≔ 𝑝 𝑝 … 𝑝 𝑝 𝑝 … 𝑝 𝑝 𝑝 … 𝑝 … , cf. Section 2.2
The length of 𝑤∗
If 𝑤 = ⟨ 𝑤 , … , 𝑤 ⟩ ∈ (Σ ) , |𝑤| = |𝑤 | = ⋯ = |𝑤 |The size of 𝑊 de ined by ∑ ∈ |𝑤|, cf. Section 3.2
The diameter of the set 𝜌(𝑤) with |𝑤| = 𝑘;i.e., diam(𝑘) = √𝑑 ⋅ 2 , cf. Section 2.2
In inite sequences
Set of inite or in inite sequences

Computable Analysis
𝜌
𝜉, 𝜁

𝜉≤𝜁
𝜉≡𝜁
𝜈ℚ
𝜈𝒬
𝜑
𝛽
𝜑
𝜑
𝜏

Binary representation, cf. Section 2.2
Representation;i.e., a mapping from inite or in inite sequences to some objects,
cf. Subsection 2.6.1
𝜉 is reducible to 𝜁, cf. Subsection 2.6.1
𝜉 is equivalent to 𝜁, cf. Subsection 2.6.1
Representation for rational numbers, cf. Subsection 2.6.1
Representation for inite sets of rational numbers, cf. Subsection 2.6.1
EmbeddingBase, cf. De inition 3.3
Base-𝛽 embedding, cf. De inition 3.3
Gray code embedding, cf. De inition 4.7
Cantor topology, cf. De inition 3.1

Learning Theory
ℋ
ℋ
𝐻
ℎ
𝜅
M
𝜎
Pos(𝐾)Pos (𝐾)
Neg(𝐾)The hypothesis space (the set of inite sets of inite sequences),
cf. Section 2.2
The set { 𝐻 ∈ ℋ ∣ #𝐻 ∈ 𝑁 }, cf. SubSection 2.3.1
Hypothesis, cf. Section 2.2
Classi ier of hypothesis 𝐻, cf. Section 2.2
The mapping from hypotheses to igures, cf. Equation (2.7)
Learner, cf. Section 2.2
Presentation (informant or text), cf. Section 2.2
The set of inite sequences of positive examples of 𝐾;
i.e., {𝑤 ∣ 𝜌(𝑤) ∩ 𝐾 ≠ ∅}, cf. Section 2.2
The set {𝑤 ∈ Pos(𝐾) ∣ |𝑤| = 𝑘}, cf. Subsection 2.3.1
The set of inite sequences of negative examples of 𝐾;
i.e., {𝑤 ∣ 𝜌(𝑤) ∩ 𝐾 = ∅}, cf. Section 2.2

141

142

SYMBOLS

Databases𝜏 = (𝐻, 𝑋)
set(𝑋)
ℎ
𝑥, 𝑦
𝑥(ℎ)
𝑥|
|𝜏|
⊥
𝑛
𝑑
Dom(ℎ)

Table, pair of header 𝐻 and body 𝑋
The set of tuples of body 𝑋
Feature (element in 𝐻)
Tuple
Value of 𝑥 for attribute ℎ ∈ 𝐻
Projection of 𝑥 on 𝐽 ⊆ 𝐻
Number of tuples
Missing value
Number of data (objects)
Number of features
Domain of the feature ℎClustering
𝐶, 𝐷
𝒞, 𝒟
𝐾
𝑘
𝑁
𝑙

Cluster (set of objects)
Partition (set of clusters)
Number of clusters
Discretization level
Noise parameter
Distance parameter

Classiﬁcation and Ranking
𝜆
ℒ
Λ(𝑥)
Γ(𝐺)
𝑅
𝜔(𝑅)
lPref(𝜆)≺∗≺CR(≺, ≺∗ )CP(≺)
mPref (𝜆 ∣ 𝜏)
mPref (𝜆 ∣ 𝜏)Label
The domain of labels, cf. Subsection 6.2.4
Single Label (Chapter 6) or set of labels (Chapter 7) of
object (tuple) 𝑥, cf. Subsection 6.2.3 and Subsection 7.1.1
Set of labeled objects in 𝐺, cf. Subsection 6.2.3
Classi ication rule (pair of set of attributes and label),cf. Subsection 6.2.3
Weight of classi ication rule 𝑅, cf. Subsection 6.2.3
𝑙-preference of label 𝜆, cf. De inition 6.8
True partial order, cf. Subsection 6.3.1
Predicted partial order, cf. Subsection 6.3.1
Correctness of ≺, cf. Subsection 6.3.1
Completeness of ≺, cf. Subsection 6.3.1
𝑚-preference of label 𝜆 at level 𝑘 for tuple 𝑦 with respect to 𝜏,
cf. Subsection 7.1.1
𝑚-preference of label 𝜆 for tuple 𝑦 with respect to 𝜏
(abbreviated as 𝜓 (𝜆) if 𝜏 is understood from context),
cf. Subsection 7.1.1

Formal Concept Analysis
𝐺
𝑀

The set of objects, cf. Subsection 6.2.1
The set of attributes, cf. Subsection 6.2.1

SYMBOLS𝐼(𝐺, 𝑀, 𝐼)
𝑔
𝑚
ℎ.𝑚
′′
𝔅(𝐺, 𝑀, 𝐼)
𝔅 (𝜏)𝑘Binary relation between 𝐺 and 𝑀, cf. Subsection 6.2.1
Context, cf. Subsection 6.2.1
Object, identi ied with tuple, cf. Subsection 6.2.1
Attribute, cf. Subsection 6.2.1
Quali ied attribute generated from feature ℎ, cf. Subsection 6.2.1
Closure operator, cf. Subsection 6.2.2
Concept lattice, cf. Subsection 6.2.2
Concept lattice generated from table 𝜏 at discretization level 𝑘,cf. Section 7.1
Maximum discretization level, cf. De inition 7.7

143Bibliography
C. C. Aggarwal and P. S. Yu. On classi ication of high-cardinality data streams.
In Proceedings of 2010 SIAM International Conference on Data Mining, pages
802–813, 2010.
C. C. Aggarwal, J. Han, J. Wang, and P. S. Yu. On demand classi ication of data
streams. In Proceedings of the 10th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pages 503–508, 2004.
D. Angluin. Inductive inference of formal languages from positive data. Information
and Control, 45(2):117–135, 1980.
D. Angluin. Inference of reversible languages. Journal of the ACM, 29(3):741–765,
1982.
K. Apsītis, S. Arikawa, R. Freivalds, E. Hirowatari, and C. H. Smith. On the inductive
inference of recursive real-valued functions. Theoretical Computer Science, 219(1–2):3–12, 1999.
B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues indata stream systems. In Proceedings of the 21st ACM SIGMOD-SIGACT-SIGART
Symposium on Principles of Database Systems, pages 1–16, 2002.
D. C. Baird. Experimentation: An Introduction to Measurement Theory and Experiment Design. Benjamin Cummings, 3 edition, 1994.
P. J. Ballester and J. B. O. Mitchell. A machine learning approach to predicting protein–ligand binding af inity with applications to molecular docking. Bioinformatics, 26(9):1169–1175, 2010.
M. F. Barnsley. Fractals Everywhere. Morgan Kaufmann, 2 edition, 1993.
Y. M. Barzdin. Inductive inference of automata, languages and programs (inRussian). In Proceedings of the International Congress of Mathematicians, volume 2, pages 455–460, 1974.
E. B. Baum and D. Haussler. What size net gives valid generalization? Neural computation, 1(1):151–160, 1989.
G. A. Beer. Topologies on Closed and Closed Convex Sets, volume 268 of Mathematics
and Its Applications. Kluwer Academic Publishers, 1993.
S. Ben-David and E. Dichterman. Learning with restricted focus of attention. Journal of Computer and System Sciences, 56(3):277–298, 1998.

BIBLIOGRAPHY
P. Berkhin. A survey of clustering data mining techniques. Grouping Multidimensional Data, pages 25–71, 2006.
G. Beslon, D. P. Parsons, J. M. Peña, C. Rigotti, and Y. Sanchez-Dehesa. From digital
genetics to knowledge discovery: Perspectives in genetic network understanding. Intelligent Data Analysis, 14(2):173–191, 2010.
C. Bishop. Pattern Recognition and Machine Learning (Information Science andStatistics). Springer, 2007.
V. G. Blinova, D. A. Dobrynin, V. K. Finn, S. O. Kuznetsov, and E. S. Pankratova.
Toxicology analysis by means of the JSM-method. Bioinformatics, 19(10):1201–1207, 2003.
L. Blum and M. Blum. Toward a mathematical theory of inductive inference. Information and Control, 28(2):125–155, 1975.
A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the
Vapnik-Chervonenkis dimension. Journal of the ACM, 36(4):929–965, 1989.
V. Brattka and G. Presser. Computability on subsets of metric spaces. TheoreticalComputer Science, 305(1-3):43–76, 2003.
V. Brattka and K. Weihrauch. Computability on subsets of Euclidean space I: Closed
and compact subsets. Theoretical Computer Science, 219(1-2):65–93, 1999.
G. Brock, V. Pihur, S. Datta, and S. Datta. clValid: An R package for cluster validation.
Journal of Statistical Software, 25(4):1–22, 2008.
J. R. Büchi. On a decision method in restricted second order arithmetic. In Proceedings of International Congress on Logic, Methodology and Philosophy of Science,
pages 1–12, 1960.
F. Chang, W. Qiu, R. H. Zamar, R. Lazarus, and X. Wang. clues: An R package for nonparametric clustering based on local shrinking. Journal of Statistical Software,
33(4):1–16, 2010.
V. Chaoji, M. A. Hasan, S. Salem, and M. J. Zaki. SPARCL: An effective and ef icient algorithm for mining arbitrary shape-based clusters. Knowledge and Information
Systems, 21(2):201–229, 2009.
V. Chaoji, G. Li, H. Yildirim, and M. J. Zaki. ABACUS: Mining arbitrary shaped clusters
from large datasets based on backbone identi ication. In Proceedings of 2011
SIAM International Conference on Data Mining, pages 295–306, 2011.
O. Chapelle, B. Schölkopf, and A. Zien, editors. Semi-Supervised Learning. MIT Press,
2006. URL http://www.kyb.tuebingen.mpg.de/ssl-book.
H. Cheng, Z. Liu, and J. Yang. Sparsity induced similarity measure for label propagation. In 12th IEEE International Conference on Computer Vision, pages 317–324,
2009.
W. Cheng, M. Rademaker, B. De Baets, and E. Hüllermeier. Predicting partial orders: Ranking with abstention. In J. Balcázar, F. Bonchi, A. Gionis, and M. Sebag,
editors, Machine Learning and Knowledge Discovery in Databases, volume 6321
of Lecture Notes in Computer Science, pages 215–230. Springer, 2010.

145

146

BIBLIOGRAPHY
R. Cilibrasi and P. M. B. Vitányi. Clustering by compression. IEEE Transactions on
Information Theory, 51(4):1523–1545, 2005.
W. D. Cornell, P. Cieplak, C. I. Bayly, I. R. Gould, K. M. Merz, D. M. Ferguson, D. C.
Spellmeyer, T. Fox, J. W. Caldwell, and P. A. Kollman. A second generation force
ield for the simulation of proteins, nucleic acids, and organic molecules. Journalof the American Chemical Society, 117(19):5179–5197, 1995.
R. Dara, S. C. Kremer, and D. A. Stacey. Clustering unlabeled data with SOMs improves classi ication of labeled real-world data. In Proceedings of the 2002 International Joint Conference on Neural Networks, volume 3, pages 2237–2242,
2002.
C. J. Date. An Introduction to Database Systems. Addison Wesley, 8 edition, 2003.
B. A. Davey and H. A. Priestley. Introduction to lattices and order. Cambridge University Press, 2 edition, 2002.
M. de Brecht. Topological and Algebraic Aspects of Algorithmic Learning Theory.
PhD thesis, Graduate School of Informatics, Kyoto University, 2010.
M. de Brecht and A. Yamamoto. Σ -admissible representations. In Proceedingsof the 6th International Conference on Computability and Complexity in Analysis,
2009.
M. de Brecht and A. Yamamoto. Topological properties of concept spaces (full version). Information and Computation, 208:327–340, 2010.
C. De La Higuera and J.-C. Janodet. Inference of 𝜔-languages from pre ixes. In
N. Abe, R. Khardon, and T. Zeugmann, editors, Algorithmic Learning Theory, volume 2225 of Lecture Notes in Computer Science, pages 364–377. Springer, 2001.
S. E. Decatur and R. Gennaro. On learning from noisy and incomplete examples.
In Proceedings of the 8th Annual Conference on Computational Learning Theory,
pages 353–360, 1995.
A. Demiriz, K. P. Bennett, and M. J. Embrechts. Semi-supervised clustering using
genetic algorithms. In Proceedings of Arti icial Neural Networks in Engineering,
pages 809–814, 1999.
J. Dieudonné. Foundations of Modern Analysis. Academic Press, 1960.
P. Domingos and G. Hulten. Mining high-speed data streams. In Proceedings of
the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining, pages 71–80, 2000.
A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A general lower bound on
the number of examples needed for learning. Information and Computation, 82(3):247–261, 1989.
T. Elomaa and J. Rousu. Necessary and suf icient pre-processing in numerical range
discretization. Knowledge and Information Systems, 5(2):162–182, 2003.

BIBLIOGRAPHY
M. Ester, H. P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd
International Conference on Knowledge Discovery and Data Mining, volume 96,
pages 226–231, 1996.
K. Falconer. Fractal Geometry: Mathematical Foundations and Applications. Wiley,
2003.
U. M. Fayyad and K. B. Irani. Multi-interval discretization of continuous-valued
attributes for classi ication learning. In Proceedings of the 13th International
Joint Conference on Arti icial Intelligence, pages 1022–1029, 1993.
H. Federer. Geometric Measure Theory. Springer, 1996.
R. Fisher. Statistical Methods for Research Workers. Oliver and Boyd, 1925.
R. Fisher. Statistical Methods and Scienti ic Inference. Oliver and Boyd, 1956.
A. Frank and A. Asuncion. UCI machine learning repository, 2010. URL http://archive.ics.uci.edu/ml.
R. Freivalds and C. H. Smith. On the role of procrastination in machine learning.
Information and Computation, 107(2):237–271, 1993.
N. Friedman, M. Goldszmidt, and T. J. Lee. Bayesian network classi ication with
continuous attributes: Getting the best of both discretization and parametricitting. In Proceedings of the 15th International Conference on Machine Learning,
pages 179–187, 1998.
J. Fürnkranz and E. Hüllermeier, editors. Preference learning. Springer, 2010.
M. M. Gaber, A. Zaslavsky, and S. Krishnaswamy. Mining data streams: A review.
SIGMOD Record, 34:18–26, 2005.
J. Gama and P. Kosina. Learning decision rules from data streams. In Proceedings of the 22nd International Joint Conference on Arti icial Intelligence, pages
1255–1260, 2011.
J. Gama and C. Pinto. Discretization from data streams: applications to histogramsand data mining. In Proceedings of the 21st Annual ACM Symposium on Applied
Computing, pages 23–27, 2006.
B. Ganter and S. Kuznetsov. Formalizing hypotheses with concepts. In B. Ganter and G. W. Mineau, editors, Conceptual Structures: Logical, Linguistic, andComputational Issues, volume 1867 of Lecture Notes in Computer Science, pages
342–356. Springer, 2000.
B. Ganter and S. Kuznetsov. Hypotheses and version spaces. In A. de Moor, W. Lex,
and B. Ganter, editors, Conceptual Structures for Knowledge Creation and Communication, volume 2746 of Lecture Notes in Computer Science, pages 83–95.
Springer, 2003.
B. Ganter and R. Wille.
Springer, 1998.

Formal Concept Analysis: Mathematical Foundations.

147

148

BIBLIOGRAPHY
H. Garcia-Molina, J. D. Ullman, and J. Widom. Database systems: The complete book.
Prentice Hall Press, 2008.
H. Gohlke, M. Hendlich, and G. Klebe. Knowledge-based scoring function to predict protein-ligand interactions1. Journal of molecular biology, 295(2):337–356,
2000.
E. M. Gold. Limiting recursion. The Journal of Symbolic Logic, 30(1):28–48, 1965.
E. M. Gold. Language identi ication in the limit. Information and Control, 10(5):447–474, 1967.
S. A. Goldman, S. S. Kwek, and S. D. Scott. Learning from examples with unspeci ied
attribute values. Information and Computation, 180(2):82–100, 2003.
L. Goodman and W. Kruskal.
Springer, 1979.

Measures of Association for Cross Classi ications.

S. Guha, R. Rastogi, and K. Shim. CURE: An ef icient clustering algorithm for large
databases. Information Systems, 26(1):35–58, 1998.
M. Halkidi, Y. Batistakis, and M. Vazirgiannis. On clustering validation techniques.
Journal of Intelligent Information Systems, 17(2):107–145, 2001.
J. Han and M. Kamber. Data Mining. Morgan Kaufmann, 2 edition, 2006.
J. Han, M. Kamber, and A. K. H. Tung. Spatial clustering methods in data mining: A
survey. In H. J. Miller and J. Han, editors, Geographic Data Mining and Knowledge
Discovery, Research Monographs in GIS. Taylor and Francis, 2001.
J. Handl, J. Knowles, and D. B. Kell. Computational cluster validation in postgenomic data analysis. Bioinformatics, 21(15):3201, 2005.
A. Hinneburg and D. A. Keim. An ef icient approach to clustering in large multimedia databases with noise. In Proceedings of the 4th International Conference on
Knowledge Discovery and Data Mining, pages 58–65, 1998.
E. Hirowatari and S. Arikawa. Inferability of recursive real-valued functions. In
M. Li and A. Maruoka, editors, Algorithmic Learning Theory, volume 1316 of Lecture Notes in Computer Science, pages 18–31. Springer, 1997.
E. Hirowatari and S. Arikawa. A comparison of identi ication criteria for inductive
inference of recursive real-valued functions. Theoretical Computer Science, 268(2):351–366, 2001.
E. Hirowatari, K. Hirata, T. Miyahara, and S. Arikawa. Criteria for inductive inference with mind changes and anomalies of recursive real-valued functions. IEICE
Transactions on Information and Systems, 86(2):219–227, 2003.
E. Hirowatari, K. Hirata, T. Miyahara, and S. Arikawa. Refutability and reliability
for inductive inference of recursive real-valued functions. IPSJ Digital Courier,
1:141–152, 2005.

BIBLIOGRAPHY
E. Hirowatari, K. Hirata, and T. Miyahara. Prediction of recursive real-valued functions from inite examples. In T. Washio, A. Sakurai, K. Nakajima, H. Takeda,
S. Tojo, and M. Yokoo, editors, New Frontiers in Arti icial Intelligence, volume4012 of Lecture Notes in Computer Science, pages 224–234. Springer, 2006.
J. E. Hopcroft and J. D. Ullman. Introduction to Automata Theory, Languages, andComputation. Addison Wesley Publishing Company, 1979.
N. Huang, C. Kalyanaraman, K. Bernacki, and M. P. Jacobson. Molecular mechanics methods for predicting protein–ligand binding. Physical Chemistry Chemical
Physics, 8(44):5166–5177, 2006.
L. Hubert and P. Arabie. Comparing partitions. Journal of Classi ication, 2(1):
193–218, 1985.
R. Huey, G. M. Morris, A. J. Olson, and D. S. Goodsell. A semiempirical free energy
force ield with charge-based desolvation. Journal of computational chemistry,
28(6):1145–1152, 2007.
E. Hülermeier, J. Fünkranz, W. Cheng, and K. Brinker. Label ranking by learning
pairwise preferences. Arti icial Intelligence, 172(16–17):1897–1916, 2008.
G. Hulten and P. Domingos. VFML – a toolkit for mining high-speed time-changing
data streams. 2003. URL http://www.cs.washington.edu/dm/vfml/.
D. P. Huttenlocher, G. A. Klanderman, and W. A. Rucklidge. Comparing images using the Hausdorff distance. IEEE Transactions on pattern analysis and machine
intelligence, pages 850–863, 1993.
T. Hwang and R. Kuang. A heterogeneous label propagation algorithm for disease gene discovery. In SIAM International Conference on Data Mining, pages
583–594, 2010.
A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: A review. ACM Computing
Surveys, 31(3):264–323, 1999a.
S. Jain. Hypothesis spaces for learning. Information and Computation, 209(3):513–527, 2011.
S. Jain, D. Osherson, J. S. Royer, and A. Sharma. Systems That Learn. The MIT Press,
2 edition, 1999b.
S. Jain, E. Kinber, R. Wiehagen, and T. Zeugmann. Learning recursive functions
refutably. In N. Abe, R. Khardon, and T. Zeugmann, editors, Algorithmic Learning Theory, volume 2225 of Lecture Notes in Computer Science, pages 283–298,
2001.
S. Jain, Q. Luo, P. Semukhin, and F. Stephan. Uncountable automatic classes andlearning. Theoretical Computer Science, 412(19):1805–1820, 2011.
K. P. Jantke. Monotonic and non-monotonic inductive inference. New Generation
Computing, 8(4):349–360, 1991.
R. Jaschke, A. Hotho, C. Schmitz, B. Ganter, and G. Stumme. TRIAS–An algorithm
for mining iceberg tri-lattices. In Proceedings of the 6th IEEE International Conference on Data Mining, pages 907–911, 2006.

149

150

BIBLIOGRAPHY
D. H. Johnson. The insigni icance of statistical signi icance testing. The journal of
wildlife management, 63(3):763–772, 1999.
A. Karatzoglou, A. Smola, K. Hornik, and A. Zeileis. kernlab–an S4 package for kernel methods in R. Journal of Statistical Software, 11(9):1–20, 2004.
G. Karypis, H. Eui-Hong, and V. Kumar. CHAMELEON: Hierarchical clustering using
dynamic modeling. Computer, 32(8):68–75, 1999.
M. Kaytoue, S. O. Kuznetsov, and A. Napoli. Revisiting numerical pattern mining
with formal concept analysis. In Proceedings of the 22nd International Joint Conference on Arti icial Intelligence, pages 1342–1347, 2011a.
M. Kaytoue, S. O. Kuznetsov, A. Napoli, and S. Duplessis. Mining gene expression
data with pattern structures in formal concept analysis. Information Sciences,181:1989–2001, 2011b.
M. J. Kearns and U. V. Vazirani. An Introduction to Computational Learning Theory.
The MIT Press, 1994.
A. S. Kechris. Classical Descriptive Set Theory. Springer, 1995.
E. Keogh, S. Lonardi, C. Ratanamahatana, L. Wei, S.-H. Lee, and J. Handley.
Compression-based data mining of sequential data. Data Mining and Knowledge
Discovery, 14:99–129, 2007.
R. Khardon and D. Roth. Learning to reason with a restricted view. Machine Learning, 35(2):95–116, 1999.
E. Kinber. Monotonicity versus ef iciency for learning languages from texts. In
Algorithmic Learning Theory, volume 872 of Lecture Notes in Computer Science,
pages 395–406. Springer, 1994.
R. D. King, S. H. Muggleton, A. Srinivasan, and M. J. E. Sternberg. Structure-activity
relationships derived by machine learning: The use of atoms and their bond connectivities to predict mutagenicity by inductive logic programming. Proceedingsof the National Academy of Sciences, 93(1):438–442, 1996.
D. E. Knuth. The Art of Computer Programming, Volume 4, Fascicle 2: Generating All
Tuples and Permutations. Addison-Wesley Professional, 2005.
S. Kobayashi. Approximate identi ication, inite elasticity and lattice structure of
hypothesis space. Technical Report CSIM 96-04, Department of Computer Science and Information Mathematics, The University of Electro-Communications,
1996.
S. Kok and P. Domingos. Learning Markov logic network structure via hypergraphlifting. In Proceedings of the 26th International Conference on Machine Learning,
pages 505–512, 2009.
P. Kontkanen and P. Myllymäki. An empirical comparison of NML clustering algorithms. In Proceedings of Information Theory and Statistical Learning, pages
125–131, 2008.

BIBLIOGRAPHY
P. Kontkanen, P. Myllymäki, T. Silander, and H. Tirri. A bayesian approach to discretization. In Proceedings of the European Symposium on Intelligent Techniques,
pages 265–268, 1997.
P. Kontkanen, P. Myllymäki, W. Buntine, J. Rissanen, and H. Tirri. An MDL framework for data clustering. In P. Grünwald, I. J. Myung, and M. Pitt, editors,
Advances in Minimum Description Length: Theory and Applications. MIT Press,
2005.
S. Kullback and R. A. Leibler. On information and suf iciency. The Annals of Mathematical Statistics, 22(1):79–86, 1951.
S. O. Kuznetsov. Machine learning and formal concept analysis. In P. Eklund, editor, Concept Lattices, volume 2961 of Lecture Notes in Computer Science, pages
287–312. Springer, 2004.
S. O. Kuznetsov and M. V. Samokhin. Learning closed sets of labeled graphs
for chemical applications. In S. Kramer and B. Pfahringer, editors, Inductive
Logic Programming, volume 3625 of Lecture Notes in Computer Science, pages
190–208. Springer, 2005.
S. Lange and T. Zeugmann. Monotonic versus non-monotonic language learning.
In Nonmonotonic and Inductive Logic, volume 659 of Lecture Notes in Computer
Science, pages 254–269. Springer, 1993.
S. Lange and T. Zeugmann. Characterization of language learning front informant
under various monotonicity constraints. Journal of Experimental & Theoretical
Arti icial Intelligence, 6(1):73–94, 1994.
S. Lange, T. Zeugmann, and S. Zilles. Learning indexed families of recursive languages from positive data: A survey. Theoretical Computer Science, 397(1–3):
194–232, 2008.
M. Li, J. H. Badger, X. Chen, S. Kwong, P. Kearney, and H. Zhang. An informationbased sequence distance and its application to whole mitochondrial genome
phylogeny. Bioinformatics, 17(2):149–154, 2001.
M. Li, X. Chen, X. Li, B. Ma, and P. Vitányi. The similarity metric. In Proceedings ofthe 14th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 863–872,
2003.
J. Lin, E. Keogh, S. Lonardi, and B. Chiu. A symbolic representation of time series, with implications for streaming algorithms. In Proceedings of the 8th ACM
SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery,
pages 1–11, 2003.
J. Lin, E. Keogh, L. Wei, and S. Lonardi. Experiencing SAX: a novel symbolic representation of time series. Data mining and knowledge discovery, 15(2):107–144,
2007.
F. T. Liu, K. M. Ting, and Z. H. Zhou. Isolation forest. In Proceedings of 8th IEEE
International Conference on Data Mining, pages 413–422, 2008.
H. Liu, F. Hussain, C. L. Tan, and M. Dash. Discretization: An enabling technique.
Data Mining and Knowledge Discovery, 6(4):393–423, 2002.

151

152

BIBLIOGRAPHY
P. M. Long and L. Tan. PAC learning axis-aligned rectangles with respect to productdistributions from multiple-instance examples. Machine Learning, 30(1):7–21,
1998.
J. MacQueen. Some methods for classi ication and analysis of multivariate observations. In Proceedings of the 5th Berkeley symposium on mathematical statistics
and probability, volume 1, pages 281–297, 1967.
M. Maechler, P. Rousseeuw, A. Struyf, and M. Hubert. Cluster analysis basics andextensions, 2005.
K. Makino and T. Uno. New algorithms for enumerating all maximal cliques. In Algorithm Theory – SWAT 2004, volume 3111 of Lecture Notes in Computer Science,
pages 260–272. Springer, 2004.
B. B. Mandelbrot. The Fractal Geometry of Nature. W. H. Freeman, 1982.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural
images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of 8th International Conference on Computer Vision, volume 2, pages 416–423, 2001.
W. Merkle and F. Stephan. Refuting learning revisited. Theoretical Computer Science, 298(1):145–177, 2003.
L. Michael. Partial observability and learnability. Arti icial Intelligence, 174(11):639–669, 2010.
L. Michael. Missing information impediments to learnability. In 24th Annual Conference on Learning Theory, pages 1–2, 2011.
S. Minato. Overview of erato minato project: The art of discrete structure manipulation between science and engineering. New Generation Computing, 29(2):223–228, 2011.
E. Minicozzi. Some natural properties of strong-identi ication in inductive inference. Theoretical Computer Science, 2(3):345–360, 1976.
N. Moitessier, P. Englebienne, D. Lee, J. Lawandi, and C. R. Corbeil. Towards thedevelopment of universal, fast and highly accurate docking/scoring methods: along way to go. British journal of pharmacology, 153(S1):S7–S26, 2008.
T. Motoki, T. Shinohara, and K. Wright. The correct de inition of inite elasticity:
Corrigendum to identi ication of unions. In Proceedings of the 4th Annual Workshop on Computational Learning Theory, page 375, 1991.
Y. Mukouchi and S. Arikawa. Towards a mathematical theory of machine discovery
from facts. Theoretical Computer Science, 137(1):53–84, 1995.
Y. Mukouchi and M. Sato. Refutable language learning with a neighbor system.
Theoretical Computer Science, 298(1):89–110, 2003.
N. Müller. The iRRAM: Exact arithmetic in C++. In J. Blanck, V. Brattka, and
P. Hertling, editors, Computability and Complexity in Analysis, volume 2064 of
Lecture Notes in Computer Science, pages 222–252. Springer, 2001.

BIBLIOGRAPHY
S. K. Murthy. Automatic construction of decision trees from data: A multidisciplinary survey. Data Mining and Knowledge Discovery, 2(4):345–389, 1998.
J. Neyman and E. S. Pearson. On the use and interpretation of certain test criteria
for purposes of statistical inference: Part I. Biometrika, 20(1):175–240, 1928.
J. Neyman and E. S. Pearson. On the problem of the most ef icient tests of statistical
hypotheses. Philosophical Transactions of the Royal Society of London. Series A,
231:289–337, 1933.
T. Ogita, S. M. Rump, and S. Oishi. Accurate sum and dot product. SIAM Journal on
Scienti ic Computing, 26:1955–1988, 2005.
S. Oishi. Why research on numerical computation with result veri ication? (inJapanese). Fundamentals Review, 2(2):9–19, 2008.
N. Pasquier, Y. Bastide, R. Taouil, and L. Lakhal. Ef icient mining of association rules
using closed itemset lattices. Information Systems, 24(1):25–46, 1999.
D. Perrin and J.-É. Pin. In inite words. Elsevier, 2004.
M. Plantevit, A. Laurent, D. Laurent, M. Teisseire, and Y. W. Choong. Mining multidimensional and multilevel sequential patterns. ACM Transactions on Knowledge
Discovery from Data, 4(1):4–37, 2010.
W. Qiu and H. Joe. Generation of random clusters with speci ied degree of separation. Journal of Classi ication, 23:315–334, 2006.
J. R. Quinlan. C4.5: programs for machine learning. Morgan Kaufmann, 1993.
J. R. Quinlan. Improved use of continuous attributes in C4.5. Journal of Arti icial
Intelligence Research, 4:77–90, 1996.
R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, 2011. URL http://www.
R-project.org.
W. S. Rasband. ImageJ. U. S. National Institutes of Health, Bethesda, Maryland, USA,
1997–2011. URL http://imagej.nih.gov/ij/.
B. D. Ripley. Pattern Recognition and Neural Networks. Cambridge University Press,
1996.
F. Rosenblatt. The perceptron: A probabilistic model for information storage andorganization in the brain. Psychological Review, 65(6):386–408, 1958.
P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation ofcluster analysis. Journal of Computational and Applied Mathematics, 20:53–65,
1987.
A. Sakurai. Inductive inference of formal languages from positive data enumerated
primitive-recursively. In Algorithmic Learning Theory, pages 73–83. JSAI, 1991.
J. Saquer and S. Jitender. Using closed itemsets for discovering representative association rules. In Z. Ras and S. Ohsuga, editors, Foundations of Intelligent Systems,
volume 1932 of Lecture Notes in Computer Science, pages 495–504. Springer,
2010.

153

154

BIBLIOGRAPHY
E. Schikuta and M. Erhart. The BANG-clustering system: Grid-based data analysis.
In Proceedings of Advances in Intelligent Data Analysis Reasoning about Data, volume 1280 of Lecture Notes in Computer Science, pages 513–524. Springer, 1997.
M. Schröder. Extended admissibility.
519–538, 2002a.

Theoretical Computer Science, 284(2):M. Schröder. Admissible representations for continuous computations. PhD thesis,
dem Fachbereich Informatik, der FernUniversit ̈at – Gesamthochschule in Hagen, 2002b.
E. Y. Shapiro. Inductive inference of theories from facts. Technical report, Department of Computer Science, Yale University, 1981.
E. Y. Shapiro. Algorithmic Program Debugging. The MIT Press, 1983.
J. L. Sharman, C. P. Mpamhanga, M. Spedding, P. Germain, B. Staels, C. Dacquet,
V. Laudet, A. J. Harmar, and NC-IUPHAR. IUPHAR-DB: New receptors and tools
for easy searching and visualization of pharmacological data. Nucleic Acids Research, 39:D534–D538, 2011. Database Issue.
G. Sheikholeslami, S. Chatterjee, and A. Zhang. WaveCluster: A multi-resolution
clustering approach for very large spatial databases. In Proceedings of the 24th
International Conference on Very Large Data Bases, pages 428–439, 1998.
D. A. Simovici and C. Djeraba. Mathematical Tools for Data Mining: Set Theory,
Partial Orders, Combinatorics. Springer, 2008.
M. Skubacz and J. Hollmén. Quantization of continuous input variables for binaryclassi ication. In Intelligent Data Engineering and Automated Learning — IDEAL
2000. Data Mining, Financial Engineering, and Intelligent Agents, volume 1983of Lecture Notes in Computer Science, pages 42–47. Springer, 2000.
M. B. Smyth. Topology. In S. Abramsky, D. M. Gabbay, and T. S. E. Maibaum, editors, Handbook of Logic in Computer Science, volume 1, pages 641–761. Oxford
University Press, 1992.
N. R. Tavana and K. Weihrauch. Turing machines on represented sets, a model ofcomputation for analysis. Logical Methods in Computer Science, 7(2):1–21, 2011.
K. M. Ting and J. R. Wells. Multi-dimensional mass estimation and mass-based
clustering. In Proceedings of 10th IEEE International Conference on Data Mining,
pages 511–520, 2010.
B. Trakhtenbrot and Y. M. Barzdin. Konetschnyje awtomaty (powedenie i sintez),
1970. English Translation: Finite automata-behavior and synthesis, Fundamental Studies in Computer Science 1, 1975.
H. Tsuiki. Real number computation through Gray code embedding. TheoreticalComputer Science, 284(2):467–485, 2002.
A. M. Turing. On computable numbers, with the application to the entscheidungsproblem. Proceedings of the London Mathematical Society, 1(42):230–265,
1937.

BIBLIOGRAPHY
T. Uno, M. Kiyomi, and H. Arimura. LCM ver. 3: Collaboration of array, bitmap andpre ix tree for frequent itemset mining. In Proceedings of the 1st International
Workshop on Open Source Data Mining: Frequent Pattern Mining Implementations, pages 77–86, 2005.
L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
P. Valtchev, R. Missaoui, and R. Godin. Formal concept analysis for knowledge
discovery and data mining: The new challenges. In P. Eklund, editor, Concept
Lattices, volume 2961 of Lecture Notes in Computer Science, pages 352–371.
Springer, 2004.
V. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequenciesof events to their probabilities. Theory of Probability and its Applications, 16(2):264–280, 1971.
V. Vapnik and A. Sterin. On structural risk minimization or overall risk in a problem of pattern recognition. Automation and Remote Control, 10(3):1495–1503,
1977.
S. Vembu and T. Gärtner. Label ranking algorithms: A survey. In Preference Learning, pages 45–64. Springer, 2010.
F. Wang and C. Zhang. Label propagation through linear neighborhoods. In
Proceedings of the 23rd international conference on Machine learning, pages
985–992, 2006.
W. Wang, J. Yang, and R. Muntz. STING: A statistical information grid approach tospatial data mining. In Proceedings of the 23rd International Conference on Very
Large Data Bases, pages 186–195, 1997.
K. Weihrauch. Computable Analysis: An Introduction. Springer, 2000.
K. Weihrauch. The computable multi-functions on multi-represented sets areclosed under programming. Journal of Universal Computer Science, 14(6):801–844, 2008.
K. Weihrauch and T. Grubba. Elementary computable topology. Journal of Universal
Computer Science, 15(6):1381–1422, 2009.
R. Wiehagen. A thesis in inductive inference. In J. Dix, K. P. Jantke, and P. H. Schmitt,editors, Nonmonotonic and Inductive Logic, volume 543 of Lecture Notes in Computer Science, pages 184–207. Springer, 1991.
R. Wille. Restructuring lattice theory: An approach based on hierarchies of concepts. In Ordered Sets, pages 445–470. D. Reidel Publishing Company, 1982. This
article is included in Formal Concept Analysis, LNCS 5548, 314–339, Springer(2009).
K. Wright. Identi ication of unions of languages drawn from an identi iable class.
In Proceedings of the 2nd Annual Workshop on Computational Learning Theory,
pages 328–333, 1989.

155

156

BIBLIOGRAPHY
M. J. Zaki. Generating non-redundant association rules. In Proceedings of the 6th
ACM SIGKDD international conference on Knowledge discovery and data mining,
pages 34–43, 2000.
T. Zeugmann and S. Zilles. Learning recursive functions: A survey. TheoreticalComputer Science, 397(1-3):4–56, 2008.
T. Zeugmann, S. Lange, and S. Kapur. Characterizations of monotonic and dual
monotonic language learning. Information and Computation, 120(2):155–173,
1995.
Y. Zhang, B. Feng, and Y. Xue. A new search results clustering algorithm based onformal concept analysis. In Proceedings of 5th International Conference on Fuzzy
Systems and Knowledge Discovery, pages 356–360. IEEE, 2008.
C. Zhao, W. Shi, and Y. Deng. A new Hausdorff distance for image matching. Pattern
Recognition Letters, 26(5):581–586, 2005.
X. Zhu and A. B. Goldberg. Introduction to semi-supervised learning. Morgan and
Claypool Publishers, 2009.

Publications by the Author
Books
[B1]

小林茂夫, 杉山麿人.: 生命科学研究に成功するための統計法ノート,
講談社, 2009.

Journal Papers
[J1]
[J2]

Sugiyama, M., Yamamoto, A.: Semi-Supervised Learning on Closed Set
Lattices, Intelligent Data Analysis, IOS Press, 17(3), 2013, to appear.
Sugiyama, M., Imajo, K., Otaki, K., Yamamoto, A.: Semi-Supervised Ligand
Finding Using Formal Concept Analysis, IPSJ Transactions on Mathematical Modeling and Its Applications (TOM), The Information Processing Society of Japan (IPSJ), accepted.

Peer-reviewed Conference Proceedings
[P1]

[P2]

[P3]

[P4]

Sugiyama, M., Hirowatari, E., Tsuiki, H., Yamamoto, A.: Learning Figures with the Hausdorff Metric by Self-similar Sets (extended abstract),
In Proceedings of 6th Workshop on Learning with Logics and Logics for
Learning (LLLL2009), pp. 27–34, Kyoto, Japan, Jul. 6–7, 2009.
Sugiyama, M., Hirowatari, E., Tsuiki, H., Yamamoto, A.: Learning Figureswith the Hausdorff Metric by Fractals, Hutter, M. and Stephan, F. andVovk, V. and Zeugmann, T. (eds.), Algorithmic Learning Theory, LNCS
6331, pp. 315–329, Springer, 2010 (Proceedings of 21th International
Conference on Algorithmic Learning Theory (ALT 2010), Canberra, Australia, Oct. 6–8, 2010)
Sugiyama, M., Yamamoto, A.: The Coding Divergence for Measuring theComplexity of Separating Two Sets, In Proceedings of 2nd Asian Conference on Machine Learning (ACML2010), JMLR Workshop and Conference
Proceedings, vol. 13, pp. 127–143, Tokyo, Japan, Nov. 8–10, 2010
Sugiyama, M., Yamamoto, A.: Fast Clustering Based on the Gray-Code (extended abstract), In Proceedings of 7th Workshop on Learning with Logics and Logics for Learning (LLLL2011), p. 42, Osaka, Japan, Mar. 29–30,
2011

158

PUBLICATIONS BY THE AUTHOR
[P5]

[P6]

[P7]

[P8]

Sugiyama, M., Yamamoto, A.: Semi-Supervised Learning for Mixed-Type
Data via Formal Concept Analysis, Andrews, S., Polovina, S., Hill, R.,
Akhgar, B. (eds.), Conceptual Structures for Discovering Knowledge, LNCS
6828, pp. 284–297, Springer, 2011 (Proceedings of the 19th International Conference on Conceptual Structures (ICCS2011), Derby, UK, Jul.
25–29, 2011)
Sugiyama, M., Yamamoto, A.: The Minimum Code Length for Clustering
Using the Gray Code, Gunopulos, D. and Hofmann, T. and Malerba, D. andVazirgiannis, M. (eds.), Machine Learning and Knowledge Discovery in
Databases, LNCS 6913, pp. 365–380, Springer, 2011 (Proceedings of theEuropean Conference on Machine Learning and Principles and Practice ofKnowledge Discovery in Databases (ECML PKDD 2011), Athens, Greece,
Sep. 5–9, 2011)
Sugiyama, M., Yoshioka, T., Yamamoto, A.: High-throughput Data Stream
Classi ication on Trees, In Proceedings of Second Workshop on Algorithms
for Large-Scale Information Processing in Knowledge Discovery (ALSIP
2011), Kagawa, Japan, Dec. 1–2, 2011
Sugiyama, M., Yamamoto, A.: A Fast and Flexible Clustering Algorithm
Using Binary Discretization, In Proceedings of the 2011 IEEE International Conference on Data Mining (ICDM 2011), pp. 1212–1217, Vancouver, Canada, Dec. 11–14, 2011

Conference Proceedings[C1]

Sugiyama, M., Imajo, K., Otaki, K., Yamamoto, A.: Discovering Ligands
for TRP Ion Channels Using Formal Concept Analysis, In Proceedings ofthe 21st International Conference on Inductive Logic Programming (ILP
2011), Windsor Great Park, UK, Jul.31–Aug.3, 2011.

Index(𝜉, 𝜁)-computable, 37
bounded, 139(𝜉, 𝜁)-realization, 37
box-counting dimension, 32
𝐿 metric, 138
bucket sort, 83
F C-I -learning, 24
Cantor space, 47
F C-T -learning, 24
Cantor topology, 47
F E E -I -learning, 29
Cauchy sequence, 139
F E E -T -learning, 29
characteristic function, 9
F E -I -learning, 21
Chebyshev metric, 138
F E -T -learning, 21
classi ication, 45
F R E -I -learning, 27classi ication rule, 103, 124
F R E -T -learning, 27classi ier, 9, 17
F R E -I -learning, 25
closed, 17, 102
F R E -T -learning, 25
0closed interval, 137
Σ2 -admissible representation, 39
closed set, 138
𝛿-cover, 139closed set lattice, 96, 103, 120
𝛿-neighborhood, 16
closure, 138
𝜔-automaton, 13
coding divergence, 44, 49, 50
𝜉-computable, 37
compact set, 139
𝑑-dimensional base-𝛽 embedding, 49
complete, 139𝑙-neighborhood at level 𝑘, 80
completeness, 109
𝑙-preference, 108
compression, 62
l-consistent, 104
computability, 36
m-consistency, 124
computable, 17m-consistent, 124concept, 102
m-preference, 126
m-preference at discretization level k, 124 concept lattice, 102, 120
conceptual scaling, 97
s-dimensional Hausdorff measure, 31
consistent, 20, 49
consistent learning, 50
admissible representation, 37
context, 99
af inity, 121
continuum, 2anti-monotonicity, 125
contraction, 14ball, 138
contractivity factor, 32
base-𝛽 embedding, 48
correctness, 109
binary classi ication, 9correspondence, 136
binary discretization, 76
countable, 139
binary embedding, 49
cover, 139
binary encoding, 2data stream, 57
binary representation, 2, 10, 13
datum, 1
binary-coding divergence, 45, 50
density-connected, 82
boundary, 138
boundary point, 138
density-reachable, 82

160

INDEX
diameter, 139
directed coding divergence, 50
directly density-reachable, 82
directly reachable at level 𝑘, 80
discretization, 2, 8, 79, 90
discretization level, 101
distance parameter, 80
domain, 136

informant, 20
inner point, 138
intent, 102
inverse, 136

effective, 52
effective computing, 3effectivity, 28
embedding, 47, 62
Eps-neighborhood, 81
equivalent, 36
error, 1
Euclidean metric, 137
Euclidean space, 137
example, 19
extent, 102
exterior, 138label, 104
language, 14
lazy learner, 51
learner, 20
learning in the limit, 21
level-𝑘 partition, 66, 80
level-k example, 29
ligand, 119

false negative, 20
false positive, 20
FCA, 96, 120feature vector, 9, 119
igure, 10
inite learning, 39
inite tell-tale set, 23
Formal Concept Analysis, 96, 120
fractals, 11function, 137
Galois connection, 102
generalization error, 16, 28
generator, 15
Gold-style learning model, 9graph, 136
Gray code, 63
Gray code embedding, 69
Hausdorff dimension, 31
Hausdorff metric, 11, 16
Heine–Borel theorem, 139
hypothesis, 15
hypothesis space, 15

JSM-method, 97
Kolmogorov complexity, 62Manhattan metric, 138
mapping, 137
mass distribution principle, 33
Maximum metric, 138
MCL, 62
MDL, 62
metric, 137
metric space, 137
micrometer, 1minimum code length, 62
minimum description length, 62
minimum open set, 45
Minkowski metric, 138
mixed-type data, 96, 120
model, 49monotonicity, 19, 53
multi-label classi ication, 119naïve Cauchy representation, 39
negative control, 44
negative example, 11, 19
neighborhood, 138
noise, 81
noise parameter, 68, 81
normal enumeration, 22

one shot learning, 39
open interval, 137
open kernel, 138
identi ication in the limit, 9open set, 45, 49, 138
IFS (Iterated Function System), 11
open set condition, 32
image, 136
indexed family of recursive concepts, 19 outer point, 138

INDEX
PAC learning model, 8partial function, 136
positive control, 44
positive example, 10, 19
preference, 97, 119, 123preference learning, 119
pre ix, 3
pre ix closed set, 11
presentation, 19
Probably Approximately Correct learning model, 8
qualify, 99
radix sort, 86
range, 136reachable at level 𝑘, 80
real number, 1
receptor, 119representation, 10, 36, 47
self-similar set, 11, 14
semi-supervised learning, 96, 119
shape-based clustering, 76
Sierpiński triangle, 15
similarity dimension, 32
source, 136
spatial clustering, 76
statistical hypothesis test, 44
stream, 3
target, 136
total function, 137trie, 57
true negative, 20
true positive, 20
TTE, 3tuple, 119tupling function, 49
Turing machine, 3
type I error, 20
type II error, 20
Type-2 machine, 3, 36, 52
Type-2 Theory of Effectivity, 3
Valiant-style learning model, 8
VC dimension, 9, 32
Vietoris topology, 17, 37

161