Abstract of thesis entitled“Modularity Beyond Compositionality”
Submitted by

Weixin Zhang
for the Degree of Doctor of Philosophy
at The University of Hong Kongin April 2021
Modern software becomes increasingly complex and rapid-evolving. The cost of developing and maintaining software systems is extremely high. To reduce the engineering
eﬀort involved in developing and maintaining software systems, modularity plays an important role. A modular way to develop software will subdivide a software system into abunch of smaller components. Each of those components can then be of a manageable
size, can be developed in parallel with isolated functionality assigned to speciﬁc programmers. The components should be extensible so that new functionality can be added
easily without aﬀecting other existing components. However, such a modular extensiblestyle of component-based software development is not widely applicable in practice due
to inadequate support from the modularization techniques in programming languages.
This thesis aims at investigating techniques on programming languages that support
modular extensible software development from three perspectives: plain design patterns,
metaprogramming-based design patterns, and novel language designs. Plain design patterns can be directly applied in existing programming languages. Existing languages havean advantage that programmers are already familiar with, allowing a wider application of
the modularization technique without a steep learning curve. With the help of metaprogramming, the boilerplate code associated with the design patterns can be automated.
However, existing languages are sometimes too restrictive in terms of syntax and semantics, which might not be expressive or concise enough for certain cases. For such cases,
new programming languages (features) suit better.
The ﬁrst part of the thesis focuses on plain design patterns. We connect shallow
embeddings to Object-Oriented Programming (OOP) via procedural abstraction. We argue
that common OOP mechanisms increase the modularity and reuse of shallow EDSLs. We
make our arguments by using Gibbons and Wu’s examples, where procedural abstraction
is used in Haskell to model a simple shallow EDSL. We recode that EDSL in Scala andwith an improved OO-inspired Haskell encoding. We further illustrate our approach witha case study on refactoring a deep external SQL query processor to make it more modular,
shallow, and embedded.
The second part of the thesis turns to metaprogramming-based design patterns. We
present Castor, a Scala framework for programming with extensible, generative visitors.
Castor has several advantages over previous approaches including its support for patternmatching, type-safe interpreters, imperative style visitors, and graphs. The applicability

of Castor is shown by several examples and two case studies on modularizing interpretersfrom the “Types and Programming Languages” book and UML activity diagrams.
The last part of the thesis is on novel language designs. We propose a programmingstyle called Compositional Programming. We introduce four key concepts for Compositional Programming: compositional interfaces, compositional traits, method patterns, andnested trait composition. Altogether these concepts allow us to naturally solve challenges
such as the Expression Problem, model attribute-grammar-like programs, and generally
deal with modular programs with complex dependencies. We present a language design,
called CP, which is proved to be type-safe, together with several examples and three casestudies.
(483 words)

ii

The University of Hong Kong

Modularity Beyond Compositionality

A thesis submitted in partial fulﬁllment of the requirements
for the Degree of Doctor of Philosophy
at The University of Hong Kong
Weixin Zhang

April 2021

Declaration
I declare that this thesis represents my own work, except where due acknowledgment
is made, and that it has not been previously included in a thesis, dissertation or reportsubmitted to this University or to any other institution for a degree, diploma or otherqualiﬁcations.

.........................
Weixin Zhang
April 2021

Acknowledgements
First of all, I owe sincere thanks to my brilliant advisor Dr. Bruno C. d. S. Oliveira fora long time. This thesis would never be done without his continuous help and guidance.
Bruno is such a great role model of a researcher whose enthusiasm for research will havea lifelong inﬂuence on me.
It was a long and special journey at HKU. Along the way, I received a lot of help frommany people. The beginning was not easy for me both ﬁnancially and mentally. ThankProf. T.H. Tse for generously supporting me as a research assistant during part of myMPhil period. Thank Bruno for allowing me to switch to another research project when
I got stuck. Thank staﬀs from the general oﬃce of Department of Computer Science for
their help, in particular: Priscilla Chung, Olive Hui, and Maria Lam.
I would like to thank the members of HKU PL Group for the insightful discussions andconstructive comments on my work: Xuan Bi, Haoyuan Zhang, Yanlin Wang, Yanpeng
Yang, Tomas Tauber, Zhiyuan Shi, Ningning Xie, Huang Li, Jinxu Zhao, Xuejing Huang,
Yaoda Zhou, Baber Rehman, Yaozhu Sun, to name a few.
I would also like to thank other friends in Hong Kong: Qiuling Zeng, Yanmin Zhao,
Xuhui Jia, Rui Zhou, Kai Han, Yu Liu, Ningyu Liu, and Kan Wu.
I am grateful to live in an era where MOOCs are free to everyone. Thank Prof. Dan
Grossman, Prof. Martin Odersky, and Prof. Gregor Kiczales for their fantastic MOOCs
that stimulated my interest in programming languages.
When I doubted myself and felt uncertain about the direction after a PhD, Dr. Meng
Wang encouraged me and oﬀered me a postdoc position. Thank Meng for being an excellent host when I was a stranger in Bristol. I am fortunate to have Dr. Tĳs van der Stormand Dr. Marco Servetto been my referees. I appreciate the fruitful discussions with themat several conferences.
Thank my parents and family members for their endless love and support for me.
Finally, I hope the COVID-19 pandemic will end and the world will back to normal
soon!

April 2021
Weixin Zhangv

Contents

Declarationiii

Acknowledgementsv

List of Figuresxi

List of Tablesxiii

1 Introduction

1

1.1 Why Modularity Matters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2 Problem Statement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.2.1 The Expression Problem . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.2.2 Dependent Operations . . . . . . . . . . . . . . . . . . . . . . . . . .

4

1.2.3 Binary and Producer Operations. . . . . . . . . . . . . . . . . . . .

4

1.3 Approaches to Modular Extensibility . . . . . . . . . . . . . . . . . . . . . .

5

1.3.1 Syntactic Modularity Approaches . . . . . . . . . . . . . . . . . . . .

5

1.3.2 Semantic Modularity Approaches . . . . . . . . . . . . . . . . . . . .

6

1.4 Contributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

1.5 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

92 Preliminaries

11

2.1 The Extensible Interpreter Pattern . . . . . . . . . . . . . . . . . . . . . . .

11

2.2 The Visitor Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

2.3 The Cake Pattern. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

2.4 Object Algebras. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

2.4.1 Limitations of Object Algebras . . . . . . . . . . . . . . . . . . . . . .

16

2.5 Disjoint Intersection Types . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

2.6 SEDEL and First-Class Traits . . . . . . . . . . . . . . . . . . . . . . . . . .

22

3 Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

25

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

3.2 Shallow Object-Oriented Programming . . . . . . . . . . . . . . . . . . . . .

27

3.2.1 Scans: A DSL for Parallel Preﬁx Circuits . . . . . . . . . . . . . . . .

27

3.2.2 Shallow Embeddings and OOP

28. . . . . . . . . . . . . . . . . . . . .

3.3 Multiple Interpretations in Shallow Embeddings. . . . . . . . . . . . . . .

30
vii

CONTENTS

3.3.1 Simple Multiple Interpretations . . . . . . . . . . . . . . . . . . . . .

30

3.3.2 Dependent Interpretations . . . . . . . . . . . . . . . . . . . . . . . .

31

3.3.3 Context-sensitive Interpretations . . . . . . . . . . . . . . . . . . . .

32

3.3.4 An Alternative Encoding of Modular Interpretations . . . . . . . . . .

34

3.3.5 Modular Language Constructs . . . . . . . . . . . . . . . . . . . . . .

35

3.3.6 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

3.4 Modular interpretations in Haskell . . . . . . . . . . . . . . . . . . . . . . .

36

3.4.1 Revisiting Scans. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37

3.4.2 Modular Dependent Interpretations . . . . . . . . . . . . . . . . . . .

37

3.4.3 Modular terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39

3.4.4 Comparing Modular Implementations Using Scala and Haskell . . . .

40

3.5 Modular Terms in Scala . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

40

3.6 Case Study: A Shallow EDSL for SQL Queries . . . . . . . . . . . . . . . . .

42

3.6.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.6.2 Embedded Syntax

42. . . . . . . . . . . . . . . . . . . . . . . . . . . .

43

3.6.3 A Relational Algebra Compiler . . . . . . . . . . . . . . . . . . . . . .

44

3.6.4 Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

46

3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

46

4 Castor: Programming with Extensible Generative Visitors

49

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49

4.2 Open Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51

4.2.1 Desirable Properties of Open Pattern Matching . . . . . . . . . . . . .

52

4.2.2 Running Example: Arith . . . . . . . . . . . . . . . . . . . . . . . . .

52

4.2.3 The Visitor Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . .

53

4.2.4 Sealed Case Classes . . . . . . . . . . . . . . . . . . . . . . . . . . .

56

4.2.5 Open Case Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . .

58

4.2.6 Partial Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60

4.2.7 Extensible Visitors . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61

4.2.8 EVF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65

4.2.9 Castor

67. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.3 Hierarchical Datatypes. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

70

4.3.1 Flat Datatypes versus Hierarchical Datatypes . . . . . . . . . . . . .

70

4.3.2 Explicit Delegations. . . . . . . . . . . . . . . . . . . . . . . . . . .

71

4.3.3 Default Visitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72

4.4 GADTs and Well-Typed EDSLs . . . . . . . . . . . . . . . . . . . . . . . . .

73

4.4.1 GADTs and Well-Typed Terms . . . . . . . . . . . . . . . . . . . . . .

73

4.4.2 Well-Typed Big-Step Evaluator. . . . . . . . . . . . . . . . . . . . .

74

4.4.3 Well-Typed Small-Step Evaluator . . . . . . . . . . . . . . . . . . . .

76

4.4.4 Extension: Higher-Order Abstract Syntax for Name Binding . . . . .

76

4.5 Graphs and Imperative Visitors . . . . . . . . . . . . . . . . . . . . . . . . .

77

4.5.1 The Diﬃculties in Modeling Graphs . . . . . . . . . . . . . . . . . . .

78

4.5.2 FSM in Castor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

79

viii

CONTENTS

4.5.3 Language Composition and Memoized Traversals . . . . . . . . . . .

81

4.6 Formalized Code Generation . . . . . . . . . . . . . . . . . . . . . . . . . .

82

4.6.1 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

82

4.6.2 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

4.6.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

4.7 Case Study I: Types and Programming Languages. . . . . . . . . . . . . .

83

4.7.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

85

4.7.2 Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

85

4.8 Case Study II: UML Activity Diagrams . . . . . . . . . . . . . . . . . . . . .

88

4.8.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

88

4.8.2 Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

90

4.9 Design Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

92

4.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

93

5 Compositional Programming95

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95

5.2 An Overview of Compositional Programming . . . . . . . . . . . . . . . . . .

98

5.2.1 The Expression Problem with Compositional Programming . . . . . .

98

5.2.2 Dependencies and S-Attributed Grammars . . . . . . . . . . . . . . . 103
5.3 Parametric Polymorphism and L-Attributed Grammars . . . . . . . . . . . . 105
5.3.1 Contexts and Modular Components . . . . . . . . . . . . . . . . . . . 106
5.3.2 Polymorphic Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.3.3 L-Attributed Grammars
5.4 Formalization. . . . . . . . . . . . . . . . . . . . . . . . . 109. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

5.4.1 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4.2 An Informal Introduction to the Elaboration . . . . . . . . . . . . . . 113
5.4.3 Static Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.4.4 Formal Elaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.4.5 Type Safety and Coherence. . . . . . . . . . . . . . . . . . . . . . . 122

5.5 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.5.1 Scans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.5.2 Mini Interpreter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.5.3 C0 Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6 Related Work131

6.1 Design Patterns for Modular Extensibility . . . . . . . . . . . . . . . . . . . 131
6.2 Modular Pattern Matching. . . . . . . . . . . . . . . . . . . . . . . . . . . 132

6.3 Language Designs for Modular Extensibility . . . . . . . . . . . . . . . . . . 133
6.4 Metaprogramming for Modular Extensibility . . . . . . . . . . . . . . . . . . 137
ix

CONTENTS

7 Future Work

139

7.1 Plain Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.2 Castor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.3 CP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8 Conclusion

143

Bibliography

155

Appendices

157

A Full Type System of CP

159

B Metatheory of CP

165

xList of Figures

1.1

Object-oriented programming style . . . . . . . . . . . . . . . . . . . . . .

3

1.2

Functional programming style . . . . . . . . . . . . . . . . . . . . . . . . .

3

2.1

Fi+ syntax extened with (recursive) let bidings. . . . . . . . . . . . . . . . .

20

2.2

Fi+ subtyping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

2.3

F + typing rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

2.4

F + disjointness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

3.1

The grammar of Scans. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

3.2

The Brent-Kung circuit of width 4.

. . . . . . . . . . . . . . . . . . . . . .

27

3.3

Circuit interpretation in Scala. . . . . . . . . . . . . . . . . . . . . . . . . .

29

4.1

The syntax and semantics of Arith. . . . . . . . . . . . . . . . . . . . . . .

53

4.2

Hierarchical representation of Arith terms. . . . . . . . . . . . . . . . . . .

71

4.3

Class diagram of FSM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

78

4.4

A state machine for controlling a door. . . . . . . . . . . . . . . . . . . . .

78

4.5

Castor syntax. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

82

4.6

Castor transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

84

4.7

Simpliﬁed language/feature dependency graph. . . . . . . . . . . . . . . .

85

4.8

Performance evaluation of TAPL interpreters. . . . . . . . . . . . . . . . . .

87

4.9

Performance evaluation of Arith. . . . . . . . . . . . . . . . . . . . . . . .

87

4.10

Metamodel of UML Activity Diagrams, an excerpt adapted after the TTC’15

iidocument [Mayerhofer and Wimmer, 2015].

. . . . . . . . . . . . . . . . .

89

4.11

Refactored implementation. . . . . . . . . . . . . . . . . . . . . . . . . . .

91

5.1

Syntax of CP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5.2

The relation of relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

5.3

Selected typing rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

5.4

Selected sort transformation and type expansion rules. . . . . . . . . . . . 119

5.5

Subtyping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

5.6

Top-like types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

xi

List of Tables

3.1

Language features needed for modular interpretations: Scala vs. Haskell. .

40

3.2

SLOC for original (Deep) and refactored (Shallow) versions. . . . . . . . . .

46

4.1

Pattern matching support comparison:= good, G#= neutral, #= bad. . . .

70

4.2

SLOC evaluation of TAPL interpreters . . . . . . . . . . . . . . . . . . . . .

86

4.3

Performance evaluation in milliseconds.

92

5.1

Diﬀerent kinds of dependencies used in the mini interpreter. . . . . . . . . 126

5.2

Source lines of code for the three implementations of the C0 compiler. . . . 129. . . . . . . . . . . . . . . . . . .

xiii

Chapter

1

Introduction

1.1 Why Modularity Matters
Ever since the software crisis [Naur, 1968], modern software systems have become
increasingly complex, large-scale and rapid-evolving. The cost of developing and maintaining software systems is extremely high. To reduce the engineering eﬀort involved indeveloping and maintaining software systems, plays an important role in software development. A modular way to develop software will subdivide a software system into a bunchof smaller components. Each of those components can then be of a manageable size, canbe developed in parallel with isolated functionality assigned to speciﬁc programmers.
The functionality of each component is exposed by a well-deﬁned interface with which
other components communicate. Interacting through interfaces ensures loose coupling
between the components and allows multiple implementations of the same interface tocoexist and be readily swapped. The components should be extensible so that new functionality can be added easily without aﬀecting other existing components. Developed in amodular way, these components can further serve as reusable units for developing software systems that have similar functionality, forming a software product line [Pohl et al.,2005]. Yet, such a modular style of component-based software development [Heineman
and Councill, 2001] is not widely applicable in practice due to the inadequate support
from the modularization techniques in programming languages.
To enable the software development style described above, a proper modularization
technique should meet the following requirements:• Extensibility in both dimensions: New data variants and new operations can be added
to the system;• Strong static type safety: The mismatch between variants and operations can bestatically caught by the type system;• No modiﬁcation or duplication: Existing components can neither be modiﬁed nor
duplicated;• Separate compilation and type-checking: Existing components should neither be
type-checked nor compiled again when adding extensions;1

Chapter 1. Introduction• Independent extensibility: Independently developed extensions can be combined forjoint use;
• Modular dependencies: Components can depend on other components and suchdependencies can be expressed modularly.
The ﬁrst four requirements come from the well-known modularity challenge – the Expression Problem (EP) [Wadler, 1998]. The EP examines the type-safe extensibility of programming languages, where extensions should be done without rewriting existing code
and without using type-unsafe features (such as casts or reﬂection). Solving the EP isimportant for several reasons. Firstly, the two dimensions of extensibility (data variants
and operations) capture one essential aspect of software evolution. Secondly, strong typesafety avoids runtime errors and ensures the safety of component compositions. Thirdly,
without duplication, the code is easier to maintain. Modiﬁcation is not even possible
when the source code may not be available such as distributed in binary forms. Lastly,
the cost of compilation and type-checking is high for complex software systems. Separate
compilation and type-checking boost the development process. The ﬁfth requirement is
added by Zenger and Odersky [2005], which addresses the compositionality. Compositionality is important for allowing separately developed components to be merged in theﬁnal system. We further propose the sixth requirement that assesses the dependability.
Dependability is critical because realistic programs will depend on some other program
parts. A common case of program dependencies is using library functions or functions
deﬁned in other parts of the program. It is important to have modular dependencies because the weaker dependencies between program parts are, the more modular a program
is.

1.2

Problem Statement

Small canonical modularity problems, such as the EP, illustrate some of the basic
issues: the dilemma between choosing one kind of extensibility over another one in mostprogramming languages. However, there are more modularity challenges not covered by
the EP, but also very important in practice. In particular, an important issue and a focusof this thesis are how to express dependencies modularly. Therefore, extending the EP,
we set up a concrete problem for examining the ability of programming languages on
modularity.

1.2.1 The Expression Problem
The initial setting of the EP is about how to create a very simple form of expressions (such as numeric literals, addition, and multiplication), and operations over thoseexpressions (such as evaluation and pretty-printing) in a modular way.
Conventional Object-Oriented Programming (OOP) and functional programming (FP)cannot solve the EP since they typically excel at only one dimension of extensibility. We
now illustrate the dilemma in OOP and FP using the Scala language [Odersky et al., 2004]
since it supports both object-oriented and functional paradigms.
2

1.2 Problem Statement

trait Exp {def eval:

Int}
class Lit(n: Int) extends Exp {def eval= n}class Add(e1: Exp, e2: Exp) extends Exp {def eval= e1.eval + e2.eval}// Adding new variants is easy!
class Mul(e1: Exp, e2: Exp) extends Exp {def eval = e1.eval * e2.eval}

Figure 1.1: Object-oriented programming stylesealed abstract class Exp
case class Lit(n: Int) extends Exp
case class Add(e1: Exp, e2: Exp) extends Exp
def eval(e: Exp): Int = e match {case Lit(n)=> ncase Add(e1, e2) => eval(e1) + eval(e2)}// Adding new operations is easy!
def print(e: Exp): String = e match {case Lit(n)=> n.toStringcase Add(e1, e2) => "(" + print(e1) + "+" + print(e2) + ")"}Figure 1.2: Functional programming style

Object-oriented programming

Figure 1.1 gives an OOP implementation of the simple

expression language. Expressions are modeled as a class hierarchy, and operations on
the datatype are methods implemented throughout the hierarchy. It is quite easy to add
more data variants (such as multiplication) by creating new subclasses of Exp. However,adding new operations (such as pretty-printing) is diﬃcult because every class needs
amendments for new methods. Such amendments violate the open-closed principle in
OOP [Meyer, 1988].

Functional programming

The situation is exactly the opposite when it comes to FP.

Figure 1.2 gives an FP-style implementation of the simple expression language. Expressions are modeled using an algebraic data type (a case class hierarchy in Scala) andoperations over expressions are deﬁned as pattern matching functions. It is easy to add
new operations just by deﬁning new pattern matching functions. However, adding data
variants becomes diﬃcult since it requires modiﬁcation to every function for new patterns.
3Chapter 1. Introduction

1.2.2 Dependent Operations
An important modularity challenge not covered by the EP is about dependencies [Oliveira
et al., 2013; Rendel et al., 2014; Zhang and Oliveira, 2017]. The following code snippet
shows a simple form of dependency, where one operation depends on another operation:
def dprint(e: Exp): String = e match {case Lit(n)=> n.toStringcase Add(e1, e2) =>if (eval(e1) == 0) dprint(e2) // dependency on evalelse "(" + dprint(e1) + "+" + dprint(e2) + ")"}dprint depends on eval: if the ﬁrst operand of an addition expression evaluates to zero,

only its second operand will be printed. dprint expresses such dependency by directly
referring to eval. Such a way of declaring dependency is not modular because it ties
dprint to a ﬁxed implementation of evaluation. How to express the dependency modularly

imposes new challenges to modularization techniques.

1.2.3 Binary and Producer Operations
Binary and producer operations [Bruce et al., 1995] are also challenges not coveredby the EP. Binary operations take two parameters of a datatype while producer operations return values of a datatype. Binary and producer operations are problematic in amodularity setting because the datatype they refer to might be extended.
Binary operations

A typical example of binary operations is structural equality. Struc-tural equality further examines the ability to destruct data structures. An FP-style implementation of structural equality is given below:
def equals(e1: Exp, e2: Exp): Boolean = (e1,e2) match {case (Lit(v1), Lit(v2))=> v1 == v2

case (Add(e1,e2), Add(e3,e4)) => equals(e1,e3) && equals(e2,e4)
case _=> false}
equals compares whether two expressions are structurally the same by pattern matchingon the two expressions simultaneously. However, implementing equals in an OOP-styleis tricker:
trait Exp {def equals(that: Exp): Boolean}
class Lit(val n: Int) extends Exp {def equals(that: Exp) =if (that.isInstanceOf[Lit])n == that.asInstanceOf[Lit].n
else
false
}
class Add(val e1: Exp, val e2: Exp) extends Exp {def equals(that: Exp) =4

1.3 Approaches to Modular Extensibility

if (that.isInstanceOf[Add]) {val add = that.asInstanceOf[Add]e1.equals(add.e1) && e2.equals(add.e2)
}
else
false
}

The implementation is verbose and type-unsafe since type-tests and type-casts are used
for recognizing the concrete representation of Exp.
Producer operations

Besides referring to datatypes, producer operations may also callthe constructors for rebuilding data structures, which is another challenge for modularity.
An instance of producer operation is given below:
def evalToLit(e: Exp): Exp = e match {case Add(Lit(n1),Lit(n2)) => Lit(n1+n2)
case Add(e1,e2)
case _=> evalToLit(Add(evalToLit(e1),evalToLit(e2)))=> e}evalToLit is another version of evaluation that recursively rewrites an expression into aliteral, where constructors Lit and Add are called for rewriting the expression.

1.3 Approaches to Modular Extensibility
A lot of work has been done on modular extensibility in both software engineeringand programming languages communities over the past several decades. We classify
the approaches to modular extensibility into two categories: syntactic modularity approaches and semantic modularity approaches. Syntactic modularity approaches allow
the code to be separately deﬁned and then concatenated as a whole for type-checking
and compilation. There are some drawbacks of syntactic approaches. Existing code has
to be type-checked and compiled again together with the newly added code, which is
time-consuming especially for complex software systems. Moreover, the error messages
are reported not on the original code but on the generated code, which may be hardto understand. Compared to syntactic modularity approaches, semantic modularity approaches preserve separate compilation and modular typechecking so that the problems
abovementioned can be avoided.

1.3.1 Syntactic Modularity Approaches
Software development/Programming paradigms

Many software development/program-ming paradigms have been proposed for tackling diﬀerent aspects of modularity problems. Well-known approaches like subject-oriented programming [Harrison and Ossher,1993], component-based software development [Heineman and Councill, 2001], multidimensional separation of concerns [Tarr et al., 1999], aspect-oriented programming [Kiczales et al., 1997] and feature-oriented programming [Prehofer, 1997] oﬀer diﬀerent perspectives on how to decompose programs modularly and deal with challenges such as
5

Chapter 1. Introduction

cross-cutting concerns. These approaches can be classiﬁed as syntactic modularity approaches because the composition mechnism provided by these approaches is typicallyrealized using metaprogramming.

1.3.2 Semantic Modularity Approaches
Programming languages

Programming languages provide language constructs for mod-ularization. For instance, many languages support some notion of modules [MacQueen,1984] that can group various kinds of deﬁnitions and functions and can be separately
compiled. At a smaller scale, most OOP languages support classes which can also be
separately deﬁned, compiled, and reused by subclassing. Suﬀering from the diamondproblem, traditional classes support only single inheritance, restricting the reusability.
Mixins [Bracha and Cook, 1990] handle the diamond problem through linearization where
conﬂicts between mixins are implicitly resolved according to the composition order, allowing a form of multiple inheritance. However, a disadvantage of such an order-based
resolution mechanism is that selecting conﬂicting implementations from multiple sources
becomes impossible. Without dealing with state, traits [Schärli et al., 2003] serve as
lightweight reusable units, where conﬂicts must be explicitly resolved by the programmer.
Ideas such as virtual classes [Madsen and Moller-Pedersen, 1989; Ernst et al., 2006]
and family polymorphism [Ernst, 2001], extend the idea of virtual methods to classes.
Thus, classes and constructors can themselves be virtual, weakening the dependencies toclasses and constructors. Family polymorphism provides powerful forms of reuse, which
can solve tricky modularity problems, such as the EP, naturally [Ernst, 2004; Ernst et al.,2006]. Nevertheless, family polymorphism still uses inheritance as a primary mechanism
to express dependencies. Similarly to (regular) classes, the use of inheritance in family
polymorphism sometimes creates more coupling than necessary between sub- and superclasses/families. Furthermore, the type systems and semantics for calculi with virtual
classes are quite complex and often require advanced features such as forms of dependent
types [Ernst et al., 2006; Nystrom et al., 2006, 2004].
Other work has focused on more general language features – such as generics, higherkinded types [Moors et al., 2008], virtual types [Thorup, 1997] – which can also help withvarious modularity problems.

Design patterns

Although language features specialized for modularity are very power-

ful, they are typically not available in mainstream programming languages. Much of themore recent work on type-safe extensibility focus on design patterns [Gamma et al., 1994]
that are applicable to mainstream languages with modest type system features. Traditional design patterns, in particular the Interpreter pattern and the Visitor pattern, support only one dimension of extensibility. Improving on traditional design patterns, newdesign patterns such as Extensible Visitors [Torgersen, 2004; Oliveira, 2009; Hofer andOstermann, 2010; Zhang and Oliveira, 2017], Extensible Interpreters [Torgersen, 2004;
Wang and Oliveira, 2016], Object Algebras [Oliveira and Cook, 2012], Finally Tagless in6

1.4 Contributions

terpreters [Carette et al., 2009] or Polymorphic Embeddings [Hofer et al., 2008] are capableof solving the EP.
Essentially these techniques are closely related. The foundation for a lot of that work
comes from functional programming and type-theoretic encodings of datatypes [Church,
1936; Scott, 1963]. In particular, the work by Hinze [2006] was the precursor for thosetechniques. In his work Hinze employed so-called Church [1936] and Scott [1963]
encodings of datatypes to model generic programming libraries. Later Oliveira et al.
[2006a] showed that variants of those techniques have wider applications and solve theEP. These ideas were picked up by Carette et al. [2009] to enable tagless interpreters,
while also beneﬁting from the extensibility properties of the techniques. Carette et al.’s[2009] work popularized those applications of the techniques as the nowadays so-called
Finally Tagless style. Soon after Hofer et al. [2008] proposed Polymorphic Embeddings inScala, highly inspired by the Finally Tagless style in languages like Haskell and OCaml.
In parallel with the work on Finally Tagless and Polymorphic Embeddings the connections of those techniques to the Visitor pattern in OOP were further explored, building
on observations between the relationship between type-theoretic encodings of datatypes
and visitors by Buchlovsky and Thielecke [2006]. That work showed that Church andScott encodings of datatypes correspond to two variants of the Visitor pattern called,respectively, Internal and External visitors. Later on Oliveira and Cook [2012] showeda simpliﬁed version of Internal Visitors called Object Algebras, which could solve the EPeven in languages like Java.
While Internal Visitors, Object Algebras, Finally Tagless or Polymorphic Embeddings
can all be traced back to Church encodings, there has been much less work on techniques that are based on Scott encodings. Scott encodings are more powerful, as they
allow a (generally) recursive programming style. In contrast, Church encodings rely on a
programming style that is akin to programming with folds in functional programming [Gibbons, 2003]. In general, Scott encodings require more sophisticated type system features,
which is one reason why they have seen less adoption. In particular recursive types arenecessary, which also brings up extra complications due to the interaction of recursive
types and subtyping.
However, due to the lack of proper language support, these design patterns often
result in heavily parametrized and boilerplate code. Moreover, the lack of suﬃciently
powerful composition mechanisms makes dealing with dependencies hard in such design
patterns [Zhang and Oliveira, 2017; Oliveira et al., 2013].

1.4 Contributions
This thesis aims at investigating techniques for modular extensibility in existing programming languages as well as new programming languages (features) that, to a large
extent, meet the requirements summarized in Section 1.1. Existing languages have an
advantage that programmers are already familiar with, allowing a wider application of themodularization technique without a steep learning curve. However, existing languages are
sometimes too restrictive in terms of syntax and semantics, which might not be expres7

Chapter 1. Introduction

sive or concise enough for certain cases. For such cases, new programming languages(features) suit better. Therefore, it is meaningful to investigate modularization techniquesin both existing and new programming languages. In summary, solutions proposed inthis thesis are classiﬁed in three diﬀerent styles:
1. Plain design patterns, which can be directly applied in existing programming languages;
2. Metaprogramming-based design patterns, where the bolierplate code incurred bydesign patterns is further automated;
3. Novel language designs, which allows specialized syntax and semantics designed
for advanced modularity.
Shallow EDSLs and OOP

The ﬁrst part of the thesis focuses on plain design patterns.

The modularity issue also occurs in embedded domain-speciﬁc languages (EDSLs). The
two approaches to EDSLs, shallow embeddng and deep embedding, complement witheach other in terms of extensibility: adding new variants is easy in shallow embeddings
while adding new interpretations is easy in deep embeddings. This makes it hard forprogrammers to choose between the two embeddings when the extension dimension is
unknown in advance. Gibbons and Wu [2014] already discussed the relationship between
shallow EDSLs and procedural abstraction, while Cook [2009] discussed the connection
between procedural abstraction and OOP. We make the transitive step by connecting
shallow EDSLs directly to OOP via procedural abstraction. The knowledge about thisrelationship enables us to improve on implementation techniques for EDSLs. Common
OOP mechanisms (including subtyping, inheritance, and type-reﬁnement) increase themodularity and reuse of shallow EDSLs when compared to classical procedural abstraction by enabling a simple way to express multiple, possibly dependent, interpretations. We
make our arguments by using Gibbons and Wu’s examples, where procedural abstraction is used in Haskell to model a simple shallow EDSL. We recode that EDSL in Scalausing the Extensible Interpreter pattern [Wang and Oliveira, 2016] and with an improved
OO-inspired Haskell encoding. We further illustrate our approach with a case study on
refactoring a deep external SQL query processor [Rompf and Amin, 2015] to make it more
modular, shallow, and embedded.
The Castor framework

The second part of the thesis turns to metaprogramming-based

design patterns. We develop Castor, a Scala framework for programming with extensible,
generative visitors. Castor has several advantages over previous approaches. Firstly,
Castor comes with support for (type-safe) pattern matching to complement its visitorswith a concise notation to express operations. Secondly, Castor supports type-safe interpreters (à la Finally Tagless), but with additional support for pattern matching and agenerally recursive style. Thirdly, Castor enables many operations to be deﬁned using
an imperative style, which is signiﬁcantly more performant than a functional style (especially in the JVM platform). Finally, functional techniques usually only support tree8

1.5 Organization

structures well, but graph structures are poorly supported. Castor supports type-safe
extensible programming on graph structures. The key to Castor’s usability is the useof annotations to automatically generate large amounts of boilerplate code to simplify
programming with extensible visitors. To illustrate the applicability of Castor we present
several applications and two case studies. The ﬁrst case study compares the ability ofCastor for modularizing the interpreters from the “Types and Programming Languages”
book [Pierce, 2002] with previous modularization work. The second case study on UML
activity diagrams illustrates the imperative aspects of Castor, as well as its support forhierarchical datatypes and graphs.
Compositional Programming

In the last part of the thesis, we concentrate on novel lan-

guage designs. We propose a programming style called Compositional Programming thatencourages: 1) references to methods (or functions), constructors and types to be virtualor abstract; and 2) dependencies to other program components to be expressed in termsof interfaces, instead of concrete implementations. There are four key concepts in Compositional Programming. Compositional interfaces extend conventional OOP interfaces
to allow the speciﬁcation of the signatures of constructors, and can be parametrized by
sorts (which abstract over concrete data types). Compositional traits extend (ﬁrst-class)traits to allow not only the deﬁnition of (virtual) methods but also the deﬁnition of virtual
constructors. Method patterns provide a lightweight syntax to deﬁne implementations fornested traits, which arise from virtual constructors. Finally, a powerful form of nested
trait composition is used to compose compositional traits. Nested trait composition plays
a similar role to traditional class inheritance, but it generalizes to the composition ofnested traits. Thus it enables a form of inheritance of whole hierarchies, similar to theforms of composition found in family polymorphism. Altogether these concepts allow us tonaturally solve challenges such as the EP, model attribute-grammar-like programs, andgenerally deal with modular programs with complex dependencies. We present several
examples and three case studies. Our ﬁrst case study is on the design of an EDSL for
circuits [Hinze, 2004; Gibbons and Wu, 2014]. This EDSL is interesting because it hasvarious extensions that can be modularly deﬁned, as well as various dependencies between components. Our second case study is a mini interpreter, which is larger and canbe extended in several ways. The last case study is an implementation of the C0 compiler,
inspired by the work of Rendel et al. [2014]. In this case study, various extensions can beformulated as attributes, and those attributes contain non-trivial dependencies to otherattributes. Finally, we present a small calculus that captures the essence of CP. This
calculus is shown to be type-safe via an elaboration to the Fi+ calculus [Bi et al., 2019],
which is a recently proposed calculus that supports disjoint intersection types [Oliveira
et al., 2016], disjoint polymorphism [Alpuim et al., 2017] and nested composition [Bi et al.,2018].

1.5 Organization
The rest of this thesis is organized as follows:9

Chapter 1. Introduction• Chapter 2 gives the necessary background of the thesis;• Chapter 3 connects shallow embeddings to OOP and show how OOP features improve on the modularity of shallow EDSLs;• Chapter 4 presents the Castor framework and various applications of Castor;• Chapter 5 presents the Compositional Programming style and the CP language designed for it;• Chapter 6 discusses the related work;• Chapter 7 discusses the future work;• Chapter 8 concludes the thesis.
This thesis is based on the previous publications by the author [Zhang and Oliveira,2018, 2019, 2020; Zhang et al., 2021] and the initial idea of Chapter 3 comes from theauthor’s MPhil thesis [Zhang, 2017].

10

Chapter2

Preliminaries

This chapter gives the preliminaries of the thesis. The techniques reviewed in thischapter, including the extensible Interpreter pattern [Wang and Oliveira, 2016], theVisitor pattern [Gamma et al., 1994], the Cake pattern [Odersky and Zenger, 2005],
Object Algebras [Oliveira and Cook, 2012; Oliveira et al., 2013], disjoint intersection
types [Oliveira et al., 2016; Alpuim et al., 2017; Bi et al., 2018, 2019] and SEDEL [Bi andOliveira, 2018], are either directly used in or improved by this thesis.

2.1 The Extensible Interpreter Pattern
The Interpreter pattern [Gamma et al., 1994], as its name suggests, is often usedin implementing interpreters for programming languages. The OOP code shown in Figure 1.1 essentially follows the conventional Interpreter pattern, which is easy to add newvariants but hard to add new operations. In fact, as shown by Wang and Oliveira [2016],
it is possible to make the Interpreter extensible with common OOP features such as
subtyping, inheritance and type-reﬁnement. The code shown in Figure 1.1 can be revisedusing the extensible interpreter pattern:trait Exp {def eval: Int}trait Lit extends Exp {val n: Int
def eval = n}
trait Add extends Exp {val e1, e2: Exp
def eval = e1.eval + e2.eval}

where traits are used instead of classes in implementing Exp.
Then dependent printing method can be modularly added:
trait ExpExt extends Exp { // subtyping
def print: String}
trait LitExt extends Lit with ExpExt { // inheritancedef print = n.toString}

11

Chapter 2. Preliminaries

trait AddExt extends Add with ExpExt { // inheritanceval e1, e2: ExpExt // type-refinement
def print =if (e1.eval == 0) e2.print
else "(" + e1.print + "+" + e2.print + ")"}Another hierarchy is deﬁned, where the root ExpExt is a subtype of Exp, Lit and Add are
inherited in implementing the extended hierarchy and ﬁelds of type Exp are reﬁned to
ExpExt for allowing invocations on both eval and print.

We can construct expressions using the extended hierarchy:
val e = new AddExt {val e1 = new LitExt { val n = 0 }val e2 = new LitExt { val n = 1 }}println(e.print) // "1"

2.2

The Visitor Pattern

The Visitor pattern [Gamma et al., 1994] is often used interchangeably with the Interpreter pattern in implementing interpreters/compilers. The Visitor pattern is closely
related to the functional approach shown in Figure 1.2, which is easy to add new operations but hard to add new variants. Let us implement the expression language step bystep using the Visitor pattern.
Class hierarchy

The expressions are modeled as an ordinary class hierarchy with anaccept method implemented throughout the hierarchy.
trait Exp {def accept[E](v: ExpVisit[E]): E}
class Lit(val n: Int) extends Exp {def accept[E](v: ExpVisit[E]) = v.lit(this)}class Add(val e1: Exp, val e2: Exp) extends Exp {def accept[E](v: ExpVisit[E]): E = v.add(this)}The accept method is implemented by calling the corresponding visit method exposed by
an ExpVisit object.
Visitor interfaces

ExpVisit is the visitor interface whose deﬁnition is as follows:

trait ExpVisit[E] {def lit(x: Lit): E
def add(x: Add): E}

Each subclass of Exp has a corresponding visit method, which takes an instance of thatclass and returns E.
122.3 The Cake Pattern

Concrete visitors

Then the evaluation operation is deﬁned as a concrete visitor that

implements the visitor interface ExpVisit:object eval extends ExpVisit[Int] {def lit(x: Lit) = x.n
def add(x: Add) = x.e1.accept(this) + x.e2.accept(this)}The concrete visitor eval implements ExpVisit by instantiating the type parameter E as
Int and deﬁning lit and add accordingly. To apply the visitor recursively to the inner

expressions, this is passed as an argument to accept.
Adding new operations is done by deﬁning new concrete visitors:
object dprint extends ExpVisit[String] {def lit(x: Lit) = x.n.toString
def add(x: Add) =if (x.e1.accept(eval) == 0) x.e2.accept(this)else "("+ x.e1.accept(this) + "+" + x.e2.accept(this) + ")"}eval is directly used by passing eval as argument to accept calls.

We can construct an expression and print it like this:val e = new Add(new Lit(0), new Lit(1))
println(e.accept(dprint)) // "1"

Unfortunately, without modifying the existing code, it is hard to add a new subclasssuch as Mul to the Exp hierarchy because there is no corresponding visit method exposed
by ExpVisit for implementing its accept method.

2.3 The Cake Pattern
It is possible to make the conventional Visitor pattern extensible. The key is to decouple the class hierarchy from a speciﬁc visitor interface. As shown by Zenger andOdersky [2005], the fundamental ingredients of the Cake pattern [Odersky and Zenger,2005], namely abstract type members (virtual types), self-type annotations and mixin composition, provide a solution. We adapt their extensible imperative visitor encoding into afunctional one and apply it in modeling the expression language:
trait Base {type ExpV[E] <: ExpVisit[E]trait Exp {def accept[E](v: ExpV[E]): E}
trait ExpVisit[E] {def lit(x: Lit): E
def add(x: Add): E}
class Lit(val n: Int) extends Exp {def accept[E](v: ExpV[E]) = v.lit(this)}class Add(val e1: Exp, val e2: Exp) extends Exp {def accept[E](v: ExpV[E]) = v.add(this)13

Chapter 2. Preliminaries}
trait Eval extends ExpVisit[Int] { _: ExpV[Int] =>def lit(x: Lit) = x.n
def add(x: Add) = x.e1.accept(this) + x.e2.accept(this)}trait DPrint extends ExpVisit[String] { _: ExpV[String] =>def lit(x: Lit) = x.n.toString
def add(x: Add) =if (x.e1.accept(eval) == 0) x.e2.accept(this)else "(" + x.e1.accept(this) + "+" + x.e2.accept(this) + ")"}val eval: ExpV[Int]}

There are several changes to the implementation using conventional visitors. Firstly, thedeﬁnitions are put into a trait Base. Secondly, a higher-kinded, bounded abstract typemember ExpV is introduced for decoupling the Exp hierarchy from the visitor interface
ExpVisit. ExpV is constrained as a subtype of ExpVisit so that visit methods declared by
ExpVisit can still be invoked when implementing the accept methods. Thirdly, visitors aredeﬁned as traits rather than objects. To pass this as an argument for recursive accept
calls, the visitor has a self-type annotation as ExpV. Lastly, to express the dependency on
evaluation, a ﬁeld eval of type Eval is introduced.
Extensions can be done by extending Base:trait Extension extends Base {type ExpV[E] <: ExpVisit[E]class Mul(val e1: Exp, val e2: Exp) extends Exp {def accept[E](v: ExpV[E]) = v.mul(this)}trait ExpVisit[E] extends super.ExpVisit[E] {def mul(x: Mul): E}
trait Eval extends ExpVisit[Int] with super.Eval { _: ExpV[Int] =>def mul(x: Mul) = x.e1.accept(this) * x.e2.accept(this)}trait DPrint extends ExpVisit[String] with super.DPrint { _: ExpV[String] =>def mul(x: Mul) = x.e1.accept(this) + "*" + x.e2.accept(this)}}A new subclass of Exp, Mul, is modularly added to the hierarchy. To implement the accept
method of Mul, ExpVisit is extended with a visit method mul and the ExpV is covariantly
reﬁned to the extended ExpVisit. Existing visitors, Eval and DPrint, are extended with an
implementation of mul.
Extension needs to be instantiated before use:
object Extension extends Extension {type ExpV[E] = ExpVisit[E]object eval extends Eval
object dprint extends DPrint}

where ExpV binds to ExpVisit and each visitor has a lowercase instantiated object.
14

2.4 Object Algebras

Finally we can construct an expression and apply operations to it:
import Extension._val e = new Mul(new Lit(2), new Add(new Lit(0), new Lit(1)))
println(e.accept(dprint))

2.4 Object Algebras
Object Algebras [Oliveira and Cook, 2012] are a well-known OOP solution to the EP.
The abstract syntax of the expression language is described by an Object Algebra interface(a Scala trait):
trait ExpAlg[Exp] {def Lit: Int=> Expdef Add: (Exp,Exp) => Exp}

Essentially ExpAlg is an abstract factory, where the type parameter E captures the expression type and capitalized methods are factory methods returning variants of expressions.
Operations over the expressions are concrete factories that implement the ExpAlg interface. For example, evaluation can be deﬁned as:
trait IEval {def eval: Int}trait Eval extends ExpAlg[IEval] {def Lit = n => new IEval {def eval = n}
def Add = (e1,e2) => new IEval {def eval = e1.eval + e2.eval}}
object eval extends Eval
Eval implements ExpAlg by instantiating E as IEval and returning an instance of IEval foreach factory method accordingly.
Adding new operations

It is easy to add new operations by reimplementing the ExpAlg

interface. For example, printing is deﬁned in a way similar to evaluation:
trait IPrint {def print: String}
trait Print extends ExpAlg[IPrint] {def Lit = n => new IPrint {def print = n.toString}
def Add = (e1,e2) => new IPrint {def print = "(" ++ e1.print ++ "+" ++ e2.print ++ ")"}}object print extends Print

where Print instantiates E as IPrint and implements each factory method accordingly.
15

Chapter 2. Preliminaries

Adding new variants

It is also easy to add new variants by extending the Object Al-

gebra interface with new factory methods. For example, multiplications are modularly
introduced as follows:
trait MulAlg[Exp] extends ExpAlg[Exp] {def Mul: (Exp,Exp) => Exp}

where MulAlg extends ExpAlg with a new constructor Mul. Existing operations such as Eval
can be modularly reuse in extensions:trait EvalMul extends MulAlg[IEval] with Eval {def Mul = (e1,e2) => new IEval {def eval = e1.eval * e2.eval}}

where EvalMul inherits Eval and complements the deﬁnition for Mul only.

Modular terms

Now we show how to modularly construct a simple addition expression:

def exp[E](f: ExpAlg[E]) =f.Add(f.Lit(0),f.Lit(1))

The generic function exp builds an addition expression via the constructors exposed by
the abstract algebra f Concrete expressions can be obtained by calling exp with concrete
algebras such as eval and print:
println(exp(eval).eval)println(exp(print).print)

By supplying Eval and Print, the expression can be respectively evaluated and printed.

2.4.1 Limitations of Object Algebras
Object Algebras, in their basic form, have several limitations in terms of compositionality and dependability. To address these issues, Oliveira et al. [2013] generalize Object
Algebras and employ Scala’s support for intersection types. We discuss the limitations
one by one and show how generalized Object Algebras try to address these limitations.

Object Algebra combinators

The ﬁrst issue is that there is no proper composition mech-anism for Object Algebras. In the previous section, the expression is indeed constructed
twice respectively for evaluating and printing. A more eﬀcient way is to compose Eval
and Print into a single algebra so that the expression can be constructed only once forboth evaluating and printing. A workaround is to use pair-based Object Algebra combinators [Oliveira and Cook, 2012], which require explicit projections and hence are inconvenient for use. Fortunately, explicit projections can be eliminated with the help ofintersection types. In Scala, the type A with B denotes the intersection of two types A andB, where A with B is a subtype of both A and B. The intersection-based Object Algebra

combinator for ExpAlg can be deﬁned as:
16

2.4 Object Algebras

trait ExpMerge[A,B] extends ExpAlg[A with B] {val alg1: ExpAlg[A]val alg2: ExpAlg[B]def lift(x: A, y: B) : A with B
def Lit = n =>lift(alg1.Lit(n),alg2.Lit(n))
def Add = (e1,e2) =>lift(alg1.Add(e1, e2),alg2.Add(e1, e2))}
ExpMerge captures the two algebras to be composed as ﬁelds alg1 and alg2.

ExpMerge

implements ExpAlg by instantiating the type parameter as A with B and deﬁning eachfactory method by ﬁrstly calling the corresponding factory method respectively deﬁned on
alg1 and alg2 then merging the results via the lift method. Concrete composition is done

by implementing ExpMerge with alg1, alg2 and lift deﬁned. For example, the compositionof Eval and Print is done like this:object evalPrint extends ExpMerge[IEval,IPrint] {val alg1 = evalval alg2 = print
def lift(x: IEval, y: IPrint) = new IEval with IPrint {def eval = x.eval
def print = y.print}}

With the composed algebra EvalPrint, the expression can be constructed only once:val e = exp(evalPrint)
println(e.print + "=" + e.eval)

Unfortunately, the composition is still done in a cubersome way. Specialized combinators are needed for each Object Algebra interface and a lot of boilerplate code for eachcomposition. Workarounds are further proposed such as using generic combinators andreﬂection [Oliveira et al., 2013] or a combination of macros and implicits [Rendel et al.,2014].
Dependent operations

The second issue is that it is hard to model dependent oper-

ations. With conventional Object Algebras, the dependent operation has to be deﬁned
together with what it depends on. Recall the pretty-printer that depends on the evaluation discussed in Section 4.1. It is deﬁned as follows:
trait DPrint extends ExpAlg[IEval with IPrint] {def Lit = n => new IEval with IPrint {def eval = n
def print = n.toString}
def Add = (e1,e2) => new IEval with IPrint {def eval = e1.eval + e2.eval
def print = "(" ++ e1.print ++ "+" ++ e2.print ++ ")"}}17

Chapter 2. Preliminaries

The type parameter Exp is instantiated as IEval with IPrint for allowing both eval andprint invocations on expressions. Such an implementation is non-modular because theimplementation of Eval is repeated inside DPrint and DPrint is tightly coupled with aparticular implementation of evaluation.
Generalized Object Algebras

To account for dependencies modularly, a generalizationof Object Algebras has been proposed by Oliveira et al. [2013]. The expression Object
Algebra interface is generalized by distinguishing negative (input) and positive (output)
occurrences of the expression type. For example, ExpAlg can be generalized in the following
way:trait GExpAlg[Exp,OExp] {def Lit: Int=> OExp

def Add: (Exp,Exp) => OExp}

where an additional type parameter OExp is introduced for capturing positive (output)
occurrences of the expression type. Here, the return type positions of the two factory
methods are positive and hence are replaced by OExp. The ordinary Object Algebra interface can be restored by making Exp and OExp consistent, which is convenient for deﬁning
algebras without dependencies:
type ExpAlg[Exp] = GExpAlg[Exp,Exp]

By doing this, evaluation and printing algebras can be deﬁned as before. Furthermore,
dependent printing can be modularly deﬁned with generalized Object Algebras:
trait DPrint extends GExpAlg[IEval with IPrint,IPrint] {def Lit = n => new IPrint {def print = n.toString}
def Add = (e1,e2) => new IPrint {def print = if (e1.eval == 0) e2.print
else "(" ++ e1.print ++ "+" ++ ")"}}The dependency on evaluation is expressed by instantiating the input type as IEval
with IPrint and the output type as IPrint. The dependency is modular because DPrint

does not depend on a particular implementation of evaluation. The dependency can be
fulﬁlled by composing DPrint with any other algebra that implements ExpAlg[IEval] suchas Eval. However this requires a generalized Object Algebra combinator for performing
such composition.
Summary

The EP illustrates some fundamental diﬃculties of writing modular code incurrent programming languages. Techniques such as Object Algebras provide a solution
for such problems, but they have their own limitations. These limitations partly arise
from the lack of programming language support. In particular:• Unconventional programming style: The programming style required to programwith Object Algebras is quite unconventional compared to standard OOP code. For
18

2.5 Disjoint Intersection Types

instance, since constructors are avoided, all objects must be constructed relative
to an Object Algebra (or factory), similarly to the method exp. Moreover, the code
required for programming with Object Algebras is somewhat verbose.
• No built-in composition: Scala (or other OOP languages) have built-in support
for a form of inheritance, which provides a mechanism to compose code. However,such OOP languages do not support the composition of Object Algebras. Composing
Object Algebras in such languages is possible, but requires the explicit deﬁnition ofcomposition operators, which have to be deﬁned for each Object Algebra interface.
• No built-in support for modular dependencies: Dependencies are quite common
in programming. All realistic software will involve multiple forms of dependencies. However, using simple Object Algebras forces dependencies to be written ina non-modular way. Writing modular dependencies is possible with a generalization of Object Algebra interfaces and specialized composition operators. However,such composition operators require a lot of boilerplate code, generalized Object Algebras require a careful manual encoding that distinguishes positive and negative
occurrences, and the programming style involved in such code is generally quite
heavyweight and unconventional.

2.5 Disjoint Intersection Types
Intersection types are useful to model modular programs, as we have seen in theprevious section. In particular, the Object Algebra combinators use intersection types.
However, a missing feature in Scala for programming with intersection types is a merge
operator. Without this operator, it is sometimes necessary to simulate a merge operator
in Scala using meta-programming or reﬂection [Oliveira et al., 2013; Rendel et al., 2014],
which results in convoluted error messages, performance penalties and, more generally,lack of modular type-checking.
Recent developments on disjoint intersection types [Oliveira et al., 2016] provide an alternative approach that supports a native merge operator. The λi calculus [Oliveira et al.,2016] was the ﬁrst calculus with disjoint intersection types and addressed the incoherence problem of intersection types with a merge operator [Dunﬁeld, 2014] by introducing
the notion of disjointness. The Fi calculus [Alpuim et al., 2017] extends λi with a formof parametric polymorphism called disjoint polymorphism, where type parameters can beconstrained to be disjoint with a speciﬁc type. On the other hand, λi+ [Bi et al., 2018]
extends λi with BCD-style distributive subtyping, enabling nested composition. The Fi+
calculus [Bi et al., 2019] combines disjoint intersection types, disjoint polymorphism, andnested composition, enabling all the foundational ingredients for Compositional Programming.
We review Fi+ only because it is the most comprehensive calculus on disjoint intersection types.
19

Chapter 2. Preliminaries

F
F|
Term contexts Γ FType contexts ∆ F
TypesExpressions

τeInt | α | > | ⊥ | τ1 → τ2 | ∀(α ∗ τ1 ).τ2 | τ1 & τ2 | {l : τ }i | x | > | λx.e | e1 e2 | Λ(α ∗ τ ).e | e τ | e1 , , e2 | {l = e } | e.l

let x : τ = e1 in e2• | Γ, x : τ• | ∆, α ∗ τFigure 2.1: Fi+ syntax extened with (recursive) let bidings.
τ1 <: τ2TS-trans

TS-reflτ1 <: τ2

τ <: τ

TS-topτ2 <: τ3

τ <: >

τ1 <: τ3
TS-andTS-andrτ1 <: τ2

τ1 & τ2 <: τ2TS-bot⊥ <: τ

τ3 <: τ1TS-distRcd{l : τ1 } <: {l : τ2 }

τ2 <: τ4

τ1 → τ2 <: τ3 → τ4TS-forall> <: ∀(α ∗ >).>

τ1 <: τ2

TS-arrτ1 <: τ3

τ1 <: τ2 & τ3

TS-topAll

TS-rcdτ2 <: τ4

τ3 <: τ1∀(α ∗ τ1 ).τ2 <: ∀(α ∗ τ3 ).τ4

{l : τ1 } & {l : τ2 } <: {l : τ1 & τ2 }TS-topArr> <: > → >TS-andl

τ1 & τ2 <: τ1TS-topRcd> <: {l : >}TS-distArr(τ1 → τ2 ) & (τ1 → τ3 ) <: τ1 → (τ2 & τ3 )TS-distAll∀(α ∗ τ1 ).τ2 & ∀(α ∗ τ1 ).τ3 <: ∀(α ∗ τ1 ).τ2 & τ3

Figure 2.2: Fi+ subtyping.

SyntaxFigure 2.1 gives the syntax of Fi+ . Metavariable τ ranges over types. Types

include integers Int, type variables α, the top type >, the bottom type ⊥, arrows τ1 →
τ2 , disjoint quantiﬁcation ∀(α ∗ τ1 ).τ2 , intersections τ1 & τ2 , and single-ﬁeld record types{l : τ }. Metavariable e ranges over expressions. Expressions include integer literals i,term variables x, the top value >, lambda abstractions λx.e, term applications e1 e2 ,type abstractions Λ(α ∗ τ ).e with α constrained to be disjoint with τ, type applicationse τ, merges e1 , , e2 , single-ﬁeld records {l = e }, record projections e.l and (recursive) let
expressions let x : τ = e1 in e2 .
Subtyping

Figure 2.2 shows the subtyping rules of the form τ1 <: τ2 . The subtyping

relation is reﬂective (TS-refl) and transitive (S-trans). Rules for top types (TS-top), bottom
types (TS-bot), function types (TS-arr) and record types (TS-rcd) are standard. The threerules on intersection types (TS-andL, TS-andR and TS-and) state that τ1 & τ2 is the greatest
lower bound for τ1 and τ2 . The BCD-style distributive rules [Barendregt et al., 1983] (TSdistArr, TS-distRcd and TS-distAll) are particularly interesting, since they enable nested
composition.

TypingFigure 2.3 gives the bidirectional type system employed by Fi+ : under the con-texts ∆ and Γ, the inference mode (⇒) synthesizes a type τ from an expression e while the20

2.5 Disjoint Intersection Types∆; Γ ` e ⇒ τ
TT-top`∆TT-nat`∆∆`Γ∆; Γ ` > ⇒ >TT-var∆`Γ`∆∆; Γ ` i ⇒ Int

TT-app∆; Γ ` e1 ⇒ τ1 → τ2

(x : A) ∈ Γ∆; Γ ` x ⇒ A

TT-merge∆; Γ ` e2 ⇐ τ1∆`Γ∆; Γ ` e1 ⇒ τ1∆; Γ ` e1 e2 ⇒ τ2∆; Γ ` e2 ⇒ τ2

∆ ` τ1 ∗ τ2∆; Γ ` e1 , , e2 ⇒ τ1 & τ2

TT-annoTT-rcdTT-proj∆; Γ ` e : τ ⇒ τ∆; Γ ` {l = e } ⇒ {l : τ }∆; Γ ` e.l ⇒ τ∆; Γ ` e ⇐ τ∆; Γ ` e ⇒ τTT-tabs∆, α ∗ τ1 ; Γ ` e ⇒ τ2∆; Γ ` Λ(α ∗ τ1 ).e ⇒ ∀(α ∗ τ1 ).τ2
TT-let∆; Γ, x : A ` e1 ⇐ A∆; Γ ` e ⇒ {l : τ }TT-tapp∆; Γ ` e ⇒ ∀(α ∗ τ2 ).τ3

∆ ` τ1 ∗ τ2∆; Γ ` e τ1 ⇒ [τ1 /α ]τ3∆; Γ, x : A ` e2 ⇒ B∆; Γ ` let x : A = e1 in e2 ⇒ B∆; Γ ` e ⇐ τTT-abs∆ ` τ1∆; Γ, x : τ1 ` e ⇐ τ2∆; Γ ` λx.e ⇐ τ1 → τ2

TT-sub∆; Γ ` e ⇒ τ2

τ2 <: τ1∆; Γ ` e ⇐ τ1

Figure 2.3: Fi+ typing rules.

checking mode (⇐) checks the type of an expression against τ. ` ∆ and ∆ ` Γ ensure thewell-formedness of contexts. The rule TT-merge infers the merge of two expressions as
an intersection type if their types are disjoint. The rule TT-tapp additionally checks thatthe supplied type τ1 is disjoint with the constraining type τ2 .

Disjointness

The disjointness judgement (∆ ` τ1 ∗ τ2 ), shown in Figure 2.4, ensures thatthe merge of τ1 and τ2 is conﬂict-free. The disjointness judgement further relies on thedeﬁnition of top-like types (eτ d) and a disjoint axiom (τ1 ∗ax τ2 ). Top-like types are types
isomorphic to >, which was a concept ﬁrstly introduced by Barendregt et al. [1983] andthen employed by Fi+ and its precursors [Oliveira et al., 2016; Alpuim et al., 2017; Bi
et al., 2019] for proving coherence. An important property of top-like types is that they
are disjoint to any other types Alpuim et al. [2017]. Furthermore, the deﬁnition of top-liketypes is crucial for deﬁning disjointness and the inclusion of types such as Int → > inthe class of top-like types, which is important to ensure the disjointness of function types
and in turn enables merges with multiple functions. A more detailed discussion can befound in work by Huang and Oliveira [2020].
21

Chapter 2. Preliminaries∆ ` τ1 ∗ τ2
TD-topL

TD-topR

TD-arr∆ ` τ1 ∗ τ2

∆ ` τ1 ∗ τ2

∆ ` τ1 → τ2 ∗ τ3 → τ4

eτ2 d

eτ1 d

TD-andR∆ ` τ1 ∗ τ2

∆ ` τ1 ∗ τ3

∆ ` τ1 ∗ τ3

TD-rcdEq

TD-rcdNeq∆ ` {l : τ1 } ∗ {l : τ2 }∆ ` {l1 : τ1 } ∗ {l2 : τ2 }l1 , l2

TD-forall

TD-tvarR

TD-tvarL(α ∗ τ1 ) ∈ ∆τ1 <: τ2∆ ` α ∗ τ2

∆ ` τ2 ∗ τ3

∆ ` τ1 & τ2 ∗ τ3

∆ ` τ1 ∗ τ2

∆ ` τ1 ∗ τ2 & τ3
(α ∗ τ1 ) ∈ ∆TD-andL∆ ` τ2 ∗ τ4∆, α ∗ τ1 & τ3 ` τ2 ∗ τ4

τ1 <: τ2∆ ` τ2 ∗ α∆ ` ∀(α ∗ τ1 ).τ2 ∗ ∀(α ∗ τ3 ).τ4TD-ax

τ1 ∗ax τ2∆ ` τ1 ∗ τ2eτ d
TL-andeτ1 dTL-tope>deτ2 d

TL-arr

TL-rcdTL-alleτ1 → τ2 de{l : τ }de∀(α ∗ τ1 ).τ2 deτ2 deτ1 & τ2 d

eτ d

eτ2 d

Figure 2.4: Fi+ disjointness.
Semantics

The semantics of Fi+ is given by elaborating to Fco , a variant of System F

extended with products and explicit coercions. Details of Fco are beyond the scope of thisthesis. We refer interested readers to the original Fi+ paper [Bi et al., 2019].

2.6

SEDEL and First-Class Traits

As a core calculus, Fi+ lacks higher-level programming abstractions to make programming convenient. SEDEL [Bi and Oliveira, 2018] is a surface language built upon Fi that
supports ﬁrst-class traits. That is, unlike Scala traits, SEDEL traits are expressions thatcan be passed to a function, assigned to a variable or returned as values. Furthermore,they can be composed to achieve a form of multiple inheritance. Such composition isenabled by the merge operator of Fi .
Trait expressions and self-types

Like Scala traits, SEDEL traits can be annotatedwith self-types for expressing the dependency on some methods that are implemented byother traits. For example, we can have two mutually dependent traits that respectively
implement the methods for testing whether a number is even or odd:
type Even = { isEven : Int -> Bool };type Odd= { isOdd: Int -> Bool };even = trait [self: Odd] => {22

2.6 SEDEL and First-Class Traits

isEven (n : Int) = if n == 0 then true else self.isOdd (n - 1)
} : Even;
odd = trait [self: Even] => {isOdd (n : Int) = if n == 0 then false else self.isEven (n - 1)
} : Odd;
Even and Odd are type aliases respectively bound to the record types declaring isEvenand isOdd. By annotating the self-type as [self: Odd], even can call isOdd via self for
implementing isEven in its body. The Even annotation in the end of the trait expression
makes sure that isEven has been implemented.
Trait types

In the previous example, the type of even is Trait[Odd,Even] and the typeof odd is Trait[Even,Odd]. Trait types in SEDEL distinguish between the required and theprovided interface. The required interface describes the methods that the trait needs forproviding its functionality, playing a similar role to abstract methods in other OOP languages. Meanwhile, the provided interface describes the functionality that the trait oﬀers.
For the case of Trait[Odd,Even], Odd is the required interface while Even is the provided
interface. When nothing is required, we can just write Trait[A] instead of Trait[Top,A].
Trait instantiations

A trait can be instantiated into an object (a record) using a new

expression only when its required interface is met. For example, the following attempt toinstantiate even fails:
new[Even] even -- Type Error!

Here the object’s type, Even, must be explicitly speciﬁed inside [] of the new expression. The
above instantiation fails because the required interface of even, Odd, is neither implemented
by the trait even nor its parents. Nevertheless, even can be instantiated together with odd
since the two traits implement each other’s required interface:evenOdd = new[Even & Odd] even & odd; -- OK!
main = evenOdd.isEven 2--> true

The two traits are instantiated into a single object of type Even & Odd that implements bothisEven and isOdd methods.

First-class traits, disjoint polymorphism, and dynamic inheritanceAn example that

diﬀerentiates SEDEL’s traits from Scala’s traits is:
combine A [B * A] (x : Trait[A]) (y : Trait[B]) = trait inherits x & y => {};The function combine takes two traits, x and y, and returns a trait that inherits both x andy. The deﬁnition of combine illustrates three key features of SEDEL: disjoint polymorphism,ﬁrst-class traits and dynamic inheritance. Firstly, combine is a polymorphic function andthe notation [B * A], means that the type parameter B is disjoint with A. This constraint
ensures that inheriting x and y simultaneously will have no conﬂicts. Secondly, traits
are passed as arguments (x and y) and a trait is the return value of combine, showingthat traits are ﬁrst-class values. Thirdly, note that what the trait expression inherits (xand y) are parameters of combine, which are statically unknown. Such kind of dynamic
23

Chapter 2. Preliminaries

inheritance is not possible in conventional statically typed OOP languages (like Scala orJava), where classes must be statically known, and they cannot be passed as arguments.
Resolving conﬂicts

Trait composition (or inheritance) in SEDEL follows the traditional

trait model [Schärli et al., 2003] where two traits cannot have conﬂicts for composition to
be successful. A beneﬁt of the trait model is that composition is commutative and associative, which ensures that the order of composition is irrelevant. In the implementationof SEDEL, trait composition is encoded in terms of the merge operator, which is itself
commutative and associative and does not allow for overlapping (or conﬂicting) values [Biet al., 2018]. In the case that two traits being composed have conﬂicts, those conﬂicts
must be explicitly resolved in SEDEL. Suppose that we have two traits t1 and t2 that
contain conﬂicting ﬁelds:
t1 = trait => { f = 1; g = "a" };t2 = trait => { f = 2; g = "b" };For a trait that would like to inherit from both t1 and t2, the conﬂicts must be explicitly
resolved. Otherwise, a type error will be reported saying that t1 and t2 are not disjoint.
One way to resolve the conﬂicts is:
t3 = trait [self: Top] inherits t1 \ {f: Int} & t2 \ {g: String} => {override f = super.f + (t1 ^ self).f};

The conﬂicts are resolved by using the exclusion operator (\) to remove f and g respectively
from t1 and t2. Together with a self-type annotation [self: Top], the excluded methods
from parents can still be accessed via the forwarding operator (^) [Bi and Oliveira, 2018].
Here, the overridden f sums up the inherited f from t2 via super and the excluded f fromt1 via the forwarding expression. We can construct an object from t3 to test that f actually

returns 3:
main = (new[{f: Int; g: String}] t3).f

24

--> 3

Chapter3

Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

In this chapter, we focus on using plain design patterns to enhance the modularity of
programs. We ﬁrst connect shallow embeddings to OOP and argue that OOP mechanisms
improve the modularity of shallow embeddings. We make our argument by employing the
Extensible Interpreter pattern and Object Algebras in modularizing several embedded
domain-speciﬁc languages (EDSLs).

3.1 Introduction
Since Hudak’s seminal paper [Hudak, 1998] on EDSLs, existing languages have beenused to directly encode DSLs.

Two common approaches to EDSLs are the so-called

shallow and deep embeddings. Deep embeddings emphasize a syntax-ﬁrst approach: theabstract syntax is deﬁned ﬁrst using a data type, and then interpretations of the abstract
syntax follow. The role of interpretations in deep embeddings is to map syntactic values
into semantic values in a semantic domain. Shallow embeddings emphasize a semanticsﬁrst approach, where a semantic domain is deﬁned ﬁrst. In the shallow approach, theoperations of the EDSLs are interpreted directly into the semantic domain. Therefore
there is no data type representing uninterpreted abstract syntax.
The trade-oﬀs between shallow and deep embeddings have been widely discussed[Svenningsson and Axelsson, 2012; Jovanovic et al., 2014]. Deep embeddings enabletransformations on the abstract syntax tree (AST), and multiple interpretations are easy to
implement. Shallow embeddings enforce the property of compositionality by construction
and are easily extended with new EDSL syntax. Such discussions lead to a generally
accepted belief that it is hard to support multiple interpretations [Svenningsson andAxelsson, 2012] and AST transformations in shallow embeddings.
Compositionality is considered a sign of good language design, and it is one of thehallmarks of denotational semantics. Compositionality means that a denotation (or interpretation) of a language is constructed from the denotation of its parts. Compositionality
leads to a modular semantics, where adding new language constructs does not requirechanges in the semantics of existing constructs. Because compositionality oﬀers a guideline for good language design, Erwig and Walkingshaw [2012] argue that a semantics-ﬁrst
25

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

approach to EDSLs is superior to a syntax-ﬁrst approach. Shallow embeddings ﬁt wellwith such a semantics-driven approach. Nevertheless, the limitations of shallow embeddings compared to deep embeddings can deter their use.
This chapter shows that, given adequate language support, having multiple modular
interpretations in shallow DSLs is not only possible but simple. Therefore we aim to
debunk the belief that multiple interpretations are hard to model with shallow embeddings. Several previous authors [Gibbons and Wu, 2014; Erwig and Walkingshaw, 2012]
already observed that, by using products and projections, multiple interpretations canbe supported with a cumbersome and often non-modular encoding. Moreover, it is also
known that multiple interpretations without dependencies on other interpretations are
modularized easily using variants of Church encodings [Gibbons and Wu, 2014; Caretteet al., 2009; Oliveira and Cook, 2012]. We show that a solution for multiple interpretations, including dependencies, is encodable naturally when the host language combines
functional features with common OO features, such as subtyping, inheritance, and typereﬁnement.
At the center of this chapter is Reynolds’ [1978] idea of procedural abstraction, whichenables us to relate shallow embeddings and OOP directly. With procedural abstraction,
data is characterized by the operations that are performed over it. This chapter builds on
two independently observed connections to procedural abstraction:
Gibbons and Wu [2014]/ Procedural Abstraction oShallow Embeddings o

Cook [2009]/ OOP

The ﬁrst connection is between procedural abstraction and shallow embeddings. As Gibbons and Wu [2014] state, “it was probably known to Reynolds, who contrasted deep
embeddings (user deﬁned types) and shallow (procedural data structures)”. Gibbons andWu noted the connection between shallow embeddings and procedural abstractions, although they did not go into much detail. The second connection is between OOP andprocedural abstraction, which was discussed in depth by Cook [2009].
We make our arguments concrete using Gibbons and Wu’s [2014] examples, where
procedural abstraction is used in Haskell to model a simple shallow EDSL. We recodethat EDSL in Scala using Wang and Oliveira’s [2016] extensible interpreter pattern. The
resulting Scala version has modularity advantages over the Haskell version, due to the useof subtyping, inheritance, and type-reﬁnement. In particular, the Scala code can easily
express modular interpretations that may not only depend on themselves but also dependon other modular interpretations, leading to our motto: beyond simple compositionality.
While Haskell does not natively support subtyping, inheritance, and type-reﬁnement,
its powerful and expressive type system is suﬃcient to encode similar features. Therefore we can port back to Haskell some of the ideas used in the Scala solution using
an improved Haskell encoding that has similar (and sometimes even better) beneﬁts interms of modularity. In essence, in the Haskell solution we encode a form of subtypingon pairs using type classes. This is useful to avoid explicit projections, that clutter theoriginal Haskell solution. Inheritance is encoded by explicitly delegating interpretations
using Haskell superclasses. Finally, type-reﬁnement is simulated using the subtyping
typeclass to introduce subtyping constraints.
26

3.2 Shallow Object-Oriented Programming

hcircuit i F id hpositive number i| fan hpositive number i| hcircuit i beside hcircuit i| hcircuit i above hcircuit i| stretch hpositive numbersi hcircuit i| ( hcircuit i )Figure 3.1: The grammar of Scans.

(fan 2 beside fan 2) above(stretch 2 2 fan 2) above(id 1 beside fan 2 beside id 1)

Figure 3.2: The Brent-Kung circuit of width 4.

While the techniques are still cumbersome for AST transformations, yielding eﬃcient
shallow EDSLs is still possible via staging [Rompf and Odersky, 2010; Carette et al.,2009]. By removing the limitation of multiple interpretations, we enlarge the applicabilityof shallow embeddings. A concrete example is our case study, which refactors an external
SQL query processor that employs deep embedding techniques [Rompf and Amin, 2015]
into a shallow EDSL. The refactored implementation allows both new (possibly dependent)
interpretations and new constructs to be introduced modularly without sacriﬁcing theperformance.
The complete code for all examples and case study is available at:https://github.com/wxzh/shallow-dsl

3.2 Shallow Object-Oriented Programming
This section shows how OOP and shallow embeddings are related via procedural abstraction. We use the same DSL presented by Gibbons and Wu [2014] as a running
example. We ﬁrst give the original shallow embedded implementation in Haskell, andrewrite it towards an “OOP style”. Then translating the program into a functional OOP
language like Scala becomes straightforward.

3.2.1 Scans: A DSL for Parallel Preﬁx Circuits
Scans [Hinze, 2004] is a DSL for describing parallel preﬁx circuits. Given an associative
binary operator •, the preﬁx sum of a non-empty sequence x1 , x2 , ..., xn is x1 , x1 • x2 , ..., x1 •x2 • ... • xn . Such computation can be performed in parallel for a parallel preﬁx circuit.
Parallel preﬁx circuits have many applications, including binary addition and sorting27

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

algorithms. The grammar of Scans is given in Figure 3.1. Scans has ﬁve constructs: twoprimitives (id and fan) and three combinators (beside, above and stretch). Their meanings
are: id n contains n parallel wires; fan n has n parallel wires with the leftmost wire
connected to all other wires from top to bottom; c1 beside c2 joins two circuits c1 and c2
horizontally; c1 above c2 combines two circuits of the same width vertically; stretch ns c
inserts wires into the circuit c so that the i th wire of c is stretched to a position of ns1 + ... +nsi , resulting in a new circuit of width by summing up ns. Figure 3.2 visualizes a circuit
constructed using all these ﬁve constructs. The structure of this circuit is explained asfollows. The whole circuit is vertically composed by three sub-circuits: the top sub-circuit
is a two 2-fans put side by side; the middle sub-circuit is a 2-fan stretched by inserting awire on the left-hand side of its ﬁrst and second wire; the bottom sub-circuit is a 2-fan in
the middle of two 1-ids.

3.2.2 Shallow Embeddings and OOP
Shallow embeddings deﬁne a language directly by encoding its semantics using procedural abstraction. In the case of Scans, a shallow embedded implementation (in Haskell)
conforms to the following types:

type Circuit = ... -- the operations we wish to support for circuits
id:: Int → Circuit
fan:: Int → Circuit
beside:: Circuit → Circuit → Circuit
above:: Circuit → Circuit → Circuit
stretch:: [Int ] → Circuit → Circuit
The type Circuit, representing the semantic domain, is to be ﬁlled with a concrete typeaccording to the semantics. Each construct is declared as a function that produces a

Circuit. Suppose that the semantics of Scans calculates the width of a circuit. The
deﬁnitions are:type Circuit = Int
id n
=n
fan n
=n
beside c1 c2 = c1 + c2
above c1 c2 = c1
stretch ns c = sum ns
For this interpretation, the Haskell domain is simply Int. This means that we will get
the width immediately after the construction of a circuit. Note that the Int domain forwidth is a degenerate case of procedural abstraction: Int can be viewed as a no argument
function. In Haskell, due to laziness, Int is a good representation. In a call-by-value
language, a no-argument function () → Int is more appropriate to deal correctly withpotential control-ﬂow language constructs.
Now we are able to construct the circuit in Figure 3.2 using these deﬁnitions:28

3.2 Shallow Object-Oriented Programming// object interface

trait Circuit1 {def width : Int }// concrete implementations

trait Id1 extends Circuit1 {val n : Int
def width = n}
trait Fan1 extends Circuit1 {val n : Int
def width = n}

trait Beside1 extends Circuit1 {val c1 , c2 : Circuit1
def width = c1 .width + c2 .width}trait Above1 extends Circuit1 {val c1 , c2 : Circuit1
def width = c1 .width}trait Stretch1 extends Circuit1 {val ns : List [Int ]; val c : Circuit1
def width = ns.sum}

Figure 3.3: Circuit interpretation in Scala.

> (fan 2 ‘beside‘ fan 2) ‘above‘| stretch [2, 2] (fan 2) ‘above‘| (id 1 ‘beside‘ fan 2 ‘beside‘ id 1)
4

Towards OOP

An isomorphic encoding of width is given below, where a record with one

ﬁeld captures the domain and is declared as a newtype:newtype Circuit1 = Circuit1 {width1 :: Int }id1 n= Circuit1 {width1 = n}fan1 n= Circuit1 {width1 = n}beside1 c1 c2= Circuit1 {width1 = width1 c1 + width1 c2 }above1 c1 c2= Circuit1 {width1 = width1 c1 }stretch1 ns c
= Circuit1 {width1 = sum ns}
The implementation is still shallow because Haskell’s newtype does not add any operational behavior to the program. Hence the two programs are eﬀectively the same. However,having ﬁelds makes the program look more like an OO program.
Porting to Scala

Indeed, we can easily translate the program from Haskell to Scala, as

shown in Figure 3.3. The idea is to map Haskell’s record types into an object interface(modeled as a trait in Scala) Circuit1 , and Haskell’s ﬁeld declarations become method
declarations. Object interfaces make the connection to procedural abstraction clear: data
is modeled by the operations that can be performed over it. Each case in the semantic
function corresponds to a concrete implementation of Circuit1 , where function parameters
are captured as immutable ﬁelds.
This implementation is essentially how we would model Scans with an OOP languagein the ﬁrst place. A minor diﬀerence is the use of traits instead of classes in implementing

Circuit1 . Although a class deﬁnition like
29

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

class Id1 (n : Int) extends Circuit1 {def width = n}
is more common, some modularity oﬀered by the trait version (e.g. mixin composition) is
lost. To use this Scala implementation in a manner similar to the Haskell implementation,
we need some smart constructors for creating objects conveniently:

def id (x : Int)= new Id1
{val n = x}
def fan (x : Int)= new Fan1
{val n = x}
def beside (x : Circuit1 , y : Circuit1 ) = new Beside1 {val c1 = x; val c2 = y}def above (x : Circuit1 , y : Circuit1 ) = new Above1 {val c1 = x; val c2 = y}def stretch (x : Circuit1 , xs : Int∗) = new Stretch1 {val ns = xs.toList; val c = x}
Now we are able to construct the circuit shown in Figure 3.2 in Scala:val circuit = above (beside (fan (2), fan (2)),
above (stretch (fan (2), 2, 2),
beside (beside (id (1), fan (2)), id (1))))
Finally, calling circuit.width will return 4 as expected.
As this example illustrates, shallow embeddings and straightforward OO programming are closely related. The syntax of the Scala code is not as concise as the Haskell
version due to some extra verbosity caused by trait declarations and smart constructors.
Nevertheless, the code is still quite compact and elegant, and the Scala implementation
has advantages in terms of modularity, as we shall see next.

3.3

Multiple Interpretations in Shallow Embeddings

An often stated limitation of shallow embeddings is that multiple interpretations are
diﬃcult. Gibbons and Wu [2014] work around this problem by using tuples. However,their encoding needs to modify the original code and thus is non-modular. This section
illustrates how various types of interpretations can be modularly deﬁned using standard
OOP mechanisms, and compares the result with Gibbons and Wu’s Haskell implementations.

3.3.1 Simple Multiple Interpretations
A single interpretation may not be enough for realistic DSLs. For example, besides

width, we may want to have another interpretation that calculates the depth of a circuitin Scans.
Multiple interpretations in Haskell

Here is Gibbons and Wu’s [2014] solution:

type Circuit2 = (Int, Int)
id2 n
= (n, 0)fan2 n= (n, 1)
above2 c1 c2 = (width c1 , depth c1 + depth c2 )30

3.3 Multiple Interpretations in Shallow Embeddings

beside2 c1 c2 = (width c1 + width c2 , depth c1 ‘max‘ depth c2 )stretch2 ns c = (sum ns, depth c)width = fst
depth = snd
A tuple is used to accommodate multiple interpretations, and each interpretation is deﬁned as a projection on the tuple. However, this solution is not modular because it relies
on deﬁning the two interpretations (width and depth) simultaneously. It is not possible toreuse the independently deﬁned width interpretation in Section 3.2.2. Whenever a newinterpretation is needed (e.g. depth), the original code has to be revised: the arity of thetuple must be incremented and the new interpretation has to be appended to each case.
Multiple interpretations in Scala

In contrast, a Scala solution allows new interpreta-

tions to be introduced in a modular way:trait Circuit2 extends Circuit1 {def depth : Int } // subtyping
trait Id2 extends Id1 with Circuit2 {def depth = 0}
trait Fan2 extends Fan1 with Circuit2 {def depth = 1}
trait Above2 extends Above1 with Circuit2 { // inheritanceoverride val c1 , c2 : Circuit2// covariant type-reﬁnement
def depth = c1 .depth + c2 .depth}
trait Beside2 extends Beside1 with Circuit2 {override val c1 , c2 : Circuit2
def depth = Math.max (c1 .depth, c2 .depth)}
trait Stretch2 extends Stretch1 with Circuit2 {override val c : Circuit2
def depth = c.depth}
The encoding relies on three OOP abstraction mechanisms: inheritance, subtyping,
and type-reﬁnement. Speciﬁcally, Circuit2 is a subtype of Circuit1 , which extends thesemantic domain with a depth method. Concrete cases, for instance Above2 , implement

Circuit2 by inheriting Above1 and implementing depth. Also, ﬁelds of type Circuit1 arecovariantly reﬁned as type Circuit2 to allow depth invocations. Importantly, all deﬁnitions
for width in Section 3.2.2 are modularly reused here.
3.3.2 Dependent Interpretations
Dependent interpretations are a generalization of multiple interpretations. A dependentinterpretation does not only depend on itself but also on other interpretations, which goesbeyond simple compositional interpretations. An instance of dependent interpretation is

wellSized, which checks whether a circuit is constructed correctly. The interpretation of
wellSized is dependent because combinators like above use width in their deﬁnitions.
31

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

Dependent interpretations in Haskell

In the Haskell solution given by Gibbons andWu [2014], dependent interpretations are again deﬁned with tuples in a non-modularway:type Circuit3 = (Int, Bool)id3 n= (n, True)
fan3 n= (n, True)
above3 c1 c2 = (width c1 , wellSized c1 ∧ wellSized c2 ∧ width c1 ≡ width c2 )beside3 c1 c2 = (width c1 + width c2 , wellSized c1 ∧ wellSized c2 )stretch3 ns c = (sum ns, wellSized c ∧ length ns ≡ width c)
wellSized = snd
where width is called in the deﬁnition of wellSized for above3 and stretch3 .
Dependent interpretations in Scala

Once again, it is easy to model dependent inter-

pretation with a simple OO approach:trait Circuit3 extends Circuit1 {def wellSized : Boolean} // dependency declaration
trait Id3 extends Id1 with Circuit3 {def wellSized = true}
trait Fan3 extends Fan1 with Circuit3 {def wellSized = true}
trait Above3 extends Above1 with Circuit3 {override val c1 , c2 : Circuit3
def wellSized =c1 .wellSized ∧ c2 .wellSized ∧ c1 .width ≡ c2 .width // dependency usage}
trait Beside3 extends Beside1 with Circuit3 {override val c1 , c2 : Circuit3
def wellSized = c1 .wellSized ∧ c2 .wellSized}trait Stretch3 extends Stretch1 with Circuit3 {override val c : Circuit3
def wellSized = c.wellSized ∧ ns.length ≡ c.width// dependency usage}
Note that width and wellSized are deﬁned separately. Essentially, it is suﬃcient to deﬁne

wellSized while knowing only the signature of width in the object interface. In the deﬁnitionof Above3 , for example, it is possible not only to call wellSized, but also width.
3.3.3 Context-sensitive Interpretations
Interpretations may rely on some context. Consider an interpretation that simpliﬁes
the representation of a circuit. A circuit can be divided horizontally into layers. Eachlayer can be represented as a sequence of pairs (i, j), denoting the connection from wire i
to wire j. For instance, the circuit shown in Figure 3.2 has the following layout:[[(0, 1), (2, 3)], [(1, 3)], [(1, 2)]]32

3.3 Multiple Interpretations in Shallow Embeddings

The combinator stretch and beside will change the layout of a circuit. For example, if
two circuits are put side by side, all the indices of the right circuit will be increased by
the width of the left circuit. Hence the interpretation layout is also dependent, relying
on itself as well as width. An intuitive implementation of layout performs these changes
immediately to the aﬀected circuit. A more eﬃcient implementation accumulates thesechanges and applies them all at once. Therefore, an accumulating parameter is used toachieve this goal, which makes layout context-sensitive.
Context-sensitive interpretations in Haskell

The following Haskell code implements(non-modular) layout:

type Circuit4 = (Int, (Int → Int) → [[(Int, Int)]])id4 n= (n, λf → [ ])fan4 n= (n, λf → [[(f 0, f j) | j ← [1 . . n − 1]]])
above4 c1 c2 = (width c1 , λf → layout c1 f ++ layout c2 f )beside4 c1 c2 = (width c1 + width c2 ,λf → lzw (++) (layout c1 f ) (layout c2 (f ◦ (width c1 +))))
stretch4 ns c = (sum ns, λf → layout c (f ◦ pred ◦ (scanl1 (+) ns!!)))lzw:: (a → a → a) → [a] → [a] → [a]lzw f [ ] ys
= ys
lzw f xs [ ]= xs
lzw f (x : xs) (y : ys) = f x y : lzw f xs ys
layout = snd
The domain of layout is a function type (Int → Int) → [[(Int, Int)]], which takes a transformation on wires and produces a layout. An anonymous function is hence deﬁned foreach case, where f is the accumulating parameter. Note that f is accumulated in beside4
and stretch4 through function composition, propagated in above4 , and ﬁnally applied towire connections in fan4 . An auxiliary deﬁnition lzw (stands for “long zip with”) zips twolists by applying the binary operator to elements of the same index and appending theremaining elements from the longer list to the resulting list. By calling layout on a circuitand supplying an identity function as the initial value of the accumulating parameter, we
will get the layout.
Context-sensitive interpretations in Scala

Context-sensitive interpretations in Scala

are unproblematic as well:trait Circuit4 extends Circuit1 {def layout (f : Int ⇒ Int) : List [List [(Int, Int)]]}trait Id4 extends Id1 with Circuit4 {def layout (f : Int ⇒ Int) = List ()}trait Fan4 extends Fan1 with Circuit4 {def layout (f : Int ⇒ Int) = List (for (i ← List.range (1, n)) yield (f (0), f (i)))}
trait Above4 extends Above1 with Circuit4 {override val c1 , c2 : Circuit4
33

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

def layout (f : Int ⇒ Int) = c1 .layout (f ) ++ c2 .layout (f )}trait Beside4 extends Beside1 with Circuit4 {override val c1 , c2 : Circuit4
def layout (f : Int ⇒ Int) =lzw (c1 .layout (f ), c2 .layout (f .compose (c1 .width + ))) ( ++ )}trait Stretch4 extends Stretch1 with Circuit4 {override val c : Circuit4
def layout (f : Int ⇒ Int) = {val vs = ns.scanLeft (0) ( + ).tailc.layout (f .compose (vs ( ) − 1))}}
def lzw [A] (xs : List [A], ys : List [A]) (f :(A, A) ⇒ A) : List [A] = (xs, ys) match {⇒ yscase (Nil, )case ( , Nil)
⇒ xscase (x :: xs, y :: ys) ⇒ f (x, y) :: lzw (xs, ys) (f )}The Scala version captures contexts as method arguments and the implementation oflayout is a direct translation from the Haskell version. There are some minor syntax
diﬀerences that need explanations. Firstly, in Fan4 , a for comprehension is used for
producing a list of connections. Secondly, for simplicity, anonymous functions are created
without a parameter list. For example, inside Beside4 , c1 .width +i ⇒ c1 .width + i, where the placeholder

is a shorthand forplays the role of the named parameter i. Thirdly,

function composition is achieved through the compose method deﬁned on function values,
which has a reverse composition order as opposed to ◦ in Haskell.

Fourthly, lzw is

implemented as a curried function, where the binary operator f is moved to the end as aseparate parameter list for facilitating type inference.

3.3.4 An Alternative Encoding of Modular Interpretations
There is an alternative encoding of modular interpretations in Scala. For example, thewellSized interpretation can be re-deﬁned like this:trait Circuit3 extends Circuit1 {def wellSized : Boolean}
trait Id3 extends Circuit3 {def wellSized = true}...

trait Stretch3 extends Circuit3 {val c : Circuit3 ; val ns : List [Int ]def wellSized = c.wellSized ∧ ns.length ≡ c.width34

3.3 Multiple Interpretations in Shallow Embeddings}
where a concrete case like Id 3 does not inherit Id 1 and leaves the width method unimplemented. Then, an extra step to combine wellSized and width is needed:

trait Id13 extends Id1 with Id3...

trait Stretch13 extends Stretch1 with Stretch3
Compared to the previous encoding, this encoding is more modular because it decouples wellSized with a particular implementation of width. However, more boilerplateis needed for combining interpretations. Moreover, it requires some support for multipleinheritance, which restricts the encoding itself from being applied to a wider range of OO
languages.

3.3.5 Modular Language Constructs
Besides new interpretations, new language constructs may be needed when a DSL
evolves. For example, in the case of Scans, we may want a rstretch (right stretch) combinator which is similar to the stretch combinator but stretches a circuit oppositely.
New constructs in Haskell

Shallow embeddings make the addition of rstretch easy by

deﬁning a new function:

rstretch:: [Int ] → Circuit4 → Circuit4
rstretch ns c = stretch4 (1 : init ns) c ‘beside4 ‘ id4 (last ns − 1)
rstretch happens to be syntactic sugar over existing constructs. For non-sugar constructs,
a new function that implements all supported interpretations is needed.
New constructs in Scala

Such simplicity of adding new constructs is retained in Scala.

Diﬀerently from the Haskell approach, there is a clear distinction between syntactic sugar
and ordinary constructs in Scala.
In Scala, syntactic sugar is deﬁned as a smart constructor upon other smart constructors:

def rstretch (ns : List [Int ], c : Circuit4 ) = stretch (1 :: ns.init, beside (c, id (ns.last − 1)))On the other hand, adding ordinary constructs is done by deﬁning a new trait that implements Circuit4 . If we treated rstretch as an ordinary construct, its deﬁnition wouldbe:trait RStretch extends Stretch4 {override def layout (f : Int ⇒ Int) = {val vs = ns.scanLeft (ns.last − 1) ( + ).initc.layout (f .compose (vs ( )))35

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality}}
Such an implementation of RStretch illustrates another strength of the Scala approach
regarding modularity. Note that RStretch does not implement Circuit4 directly. Instead, it
inherits Stretch4 and overrides the layout deﬁnition so as to reuse other interpretations aswell as ﬁeld declarations from Stretch4 . Inheritance and method overriding enable partial
reuse of an existing language construct implementation, which is particularly useful for
deﬁning specialized constructs.

3.3.6 Discussion
Gibbons and Wu claim that in shallow embeddings new language constructs are easy
to add, but new interpretations are hard. It is possible to deﬁne multiple interpretations
via tuples, “but this is still a bit clumsy: it entails revising existing code each time a newinterpretation is added, and wide tuples generally lack good language support” [Gibbons
and Wu, 2014]. In other words, Haskell’s approach based on tuples is essentially nonmodular. However, as our Scala code shows, using OOP mechanisms both language
constructs and interpretations are easy to add in shallow embeddings. Moreover, dependent interpretations are possible too, which enables interpretations that may dependon other modular interpretations and go beyond simple compositionality. The key pointis that procedural abstraction combined with OOP features (subtyping, inheritance, andtype-reﬁnement) adds expressiveness over traditional procedural abstraction.
One worthy point about the Scala solution presented so far is that it is straightforward
using OOP mechanisms, it uses only simple types, and dependent interpretations are nota problem. Gibbons and Wu do discuss a number of more advanced techniques [Caretteet al., 2009; Swierstra, 2008] that can solve some of the modularity problems. In theirpaper, they show how to support modular depth and width (corresponding to Section 3.3.1)
using the Finally Tagless [Carette et al., 2009] approach. This is possible because depth
and width are non-dependent. However they do not show how to modularize wellSized norlayout (corresponding to Section 3.3.2 and 3.3.3, respectively). In Section 3.4 we revisit
such Finally Tagless encoding and improve it to allow dependent interpretations, inspiredby the OO solution presented in this section.

3.4

Modular interpretations in Haskell

Modular interpretations are also possible in Haskell via a variant of Church encodingsthat uses type classes. The original technique is due to Hinze [2006] and was shown tobe modular and extensible by Oliveira et al. [2006a]. It has since been popularized under
the name Finally Tagless [Carette et al., 2009] in the context of EDSLs. The idea is to use
a type class to abstract over the signatures of constructs and deﬁne interpretations as
instances of that type class. This section recodes the Scans example and compares thetwo modular implementations in Haskell and Scala.
363.4 Modular interpretations in Haskell

3.4.1 Revisiting Scans
Here is the type class deﬁned for Scans:class Circuit c whereid:: Int → cfan:: Int → c
above :: c → c → cbeside :: c → c → cstretch :: [Int ] → c → c
The signatures are the same as what Section 3.2.2 shows except that the semantic domain
is captured by a type parameter c. Interpretations such as width are then deﬁned as
instances of Circuit:newtype Width = Width {width :: Int }instance Circuit Width where
id n= Width n
fan n= Width nabove c1 c2 = Width (width c1 )beside c1 c2 = Width (width c1 + width c2 )stretch ns c = Width (sum ns)
where c is instantiated as a record type Width. Instantiating the type parameter as Widthrather than Int avoids the conﬂict with the depth interpretation which also produces integers.
Multiple interpretations

Adding the depth interpretation can now be done in a modular

manner similar to width:

newtype Depth = Depth {depth :: Int }instance Circuit Depth whereid n= Depth 0
fan n= Depth 1
above c1 c2 = Depth (depth c1 + depth c2 )beside c1 c2 = Depth (depth c1 ‘max‘ depth c2 )stretch ns c = Depth (depth c)

3.4.2 Modular Dependent Interpretations
Adding a modular dependent interpretation like wellSized is more challenging in the
Finally Tagless approach. However, inspired by the OO approach we can try to mimic
the OO mechanisms in Haskell to obtain similar beneﬁts in Haskell. In what follows we
explain how to encode subtyping, inheritance, and type-reﬁnement in Haskell and howthat encoding enables additional modularity beneﬁts in Haskell.
37

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

SubtypingIn the Scala solution subtyping avoids the explicit projections that are neededin the Haskell solution presented in Section 3.3. We can obtain a similar beneﬁt in Haskell
by encoding a subtyping relation on tuples in Haskell. We use the following type class,which was introduced by Bahr and Hvitved [2011], to express a subtyping relation on
tuples:class a ≺ b where
prj :: a → b
instance a ≺ a where
prj x = x
instance (a, b) ≺ a where
prj = fst
instance (b ≺ c) ⇒ (a, b) ≺ c where
prj = prj ◦ snd
In essence a type a is a subtype of a type b (expressed as a ≺ b) if a has the same or moretuple components as the type b. This subtyping relation is closely related to the elaboration interpretation of intersection types proposed by Dunﬁeld [2014], where Dunﬁeld’s
merge operator corresponds (via elaboration) to the tuple constructor and projections are
implicit and type-driven. The function prj simulates up-casting, which converts a valueof type a to a value of type b. The three overlapping instances deﬁne the behavior of the
projection function by searching for the type being projected in a compound type.
Modular wellSized and encodings of inheritance and type-reﬁnement

Now, deﬁning

wellSized modularly becomes possible:
newtype WellSized = WellSized {wellSized :: Bool}
instance (Circuit c, c ≺ Width) ⇒ Circuit (WellSized, c) where
id n= (WellSized True, id n)
fan n= (WellSized True, fan n)
above c1 c2 = (WellSized (gwellSized c1 ∧ gwellSized c2 ∧ gwidth c1 ≡ gwidth c2 ), above (prj c1 ) (prj c2 ))beside c1 c2 = (WellSized (gwellSized c1 ∧ gwellSized c2 ), beside (prj c1 ) (prj c2 ))stretch ns c = (WellSized (gwellSized c ∧ length ns ≡ gwidth c), stretch ns (prj c))
gwidth :: (c ≺ Width) ⇒ c → Int
gwidth = width ◦ prj
gwellSized :: (c ≺ WellSized) ⇒ c → BoolgwellSized = wellSized ◦ prj
Essentially, dependent interpretations are still deﬁned using tuples. The dependency on

width is expressed by constraining the type parameter as c ≺ Width. Such constraint
allows us to simulate the type-reﬁnement of ﬁelds in the Scala solution. Although theimplementation is modular, it requires some boilerplate. The reuse of width interpretation
38

3.4 Modular interpretations in Haskell

is achieved via delegation, where prj needs to be called on each subcircuit. Such explicit
delegation simulates the inheritance employed in the Scala solution.

Also, auxiliary

deﬁnitions gwidth and gwellSized are necessary for projecting the desired interpretations
from the constrained type parameter.

3.4.3 Modular terms
As new interpretations may be added later, a problem is how to construct the termthat can be interpreted by those new interpretations without reconstruction of the AST
for each interpretation. We show how to do this for the circuit shown in Figure 3.2:

circuit :: Circuit c ⇒ ccircuit = (fan 2 ‘beside‘ fan 2) ‘above‘
stretch [2, 2] (fan 2) ‘above‘(id 1 ‘beside‘ fan 2 ‘beside‘ id 1)
Here, circuit is a generic circuit that is not tied to any interpretation. When interpretingcircuit, its type must be instantiated:> width (circuit :: Width)4> depth (circuit :: Depth)

3> gwellSized (circuit :: (WellSized, Width))True
At user-site, circuit must be annotated with the target semantic domain so that an appropriate type class instance for interpretation can be chosen.

Syntax extensions

This solution also allows us to modularly extend Scans with morelanguage constructs such as rstretch:class Circuit c ⇒ ExtendedCircuit c where
rstretch :: [Int ] → c → c
Existing interpretations can be modularly extended to handle rstretch:

instance ExtendedCircuit Width where
rstretch = stretch
Existing circuits can also be reused for constructing circuits in extended Scans:

circuit2 :: ExtendedCircuit c ⇒ ccircuit2 = rstretch [2, 2, 2, 2] circuit

39

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

Table 3.1: Language features needed for modular interpretations: Scala vs. Haskell.
Goal
Multiple interpretation
Interpretation reuse
Dependency declaration

Scala

Haskell

Trait & Type-reﬁnement
Inheritance
Subtyping / InheritanceType class
Delegation
Tuples & Type constraints

3.4.4 Comparing Modular Implementations Using Scala and Haskell
Although both the Scala and Haskell solutions are able to model modular dependentinterpretations, they use a diﬀerent set of language features. Table 3.1 compares thelanguage features needed by Scala and Haskell. The Scala approach relies on built-infeatures. In particular, subtyping, inheritance (mixin composition) and type-reﬁnement
are all built-in. This makes it quite natural to program the solutions in Scala, withouteven needing any form of parametric polymorphism. In contrast, the Haskell solution does
not have such built-in support for OO features. Subtyping and type-reﬁnement need to
be encoded/simulated using parametric polymorphism and type classes. Inheritance issimulated by explicit delegations. The Haskell encoding is arguably conceptually morediﬃcult to understand and use, but it is still quite simple. One interesting feature that is
supported in Haskell is the ability to encode modular terms. This relies on the fact thatthe constructors are overloaded. The Scala solution presented so far does not allow suchoverloading, so code using constructors is tied with a speciﬁc interpretation. In the next
section we will see a ﬁnal reﬁnement of the Scala solution that enables modular terms,
also by using overloaded constructors.

3.5

Modular Terms in Scala

One advantage of the Finally Tagless approach over our Scala approach presented so
far is that terms can be constructed modularly without tying those terms to any interpretation. Modular terms are also possible by combining our Scala approach with Object
Algebras [Oliveira and Cook, 2012], which employ a technique similar to Finally Taglessin the context of OOP. Diﬀerently from the Haskell solution presented in Section 3.4,
the Scala approach only employs parametric polymorphism to overload the constructors.
Both inheritance and type-reﬁnement do not need to be simulated or encoded.

Object Algebra interface

To capture the generic interface of the constructors we deﬁne

an abstract factory (or Object Algebra interface) for circuits similar to the type classversion shown in Section 3.4.1:

trait Circuit [C ] {def id (x : Int) : C
def fan (x : Int) : C
def above (x : C, y : C) : C
def beside (x : C, y : C) : C
40

3.5 Modular Terms in Scaladef stretch (x : C, xs : Int∗) : C}
which exposes factory methods for each circuit construct supported by Scans.

Abstract terms

Modular terms can be constructed via the abstract factory. For exam-

ple, the circuit shown in Figure 3.2 is built as:def circuit [C ] (f : Circuit [C ]) =f .above (f .beside (f .fan (2), f .fan (2)),
f .above (f .stretch (f .fan (2), 2, 2),
f .beside (f .beside (f .id (1), f .fan (2)), f .id (1))))
Similarly, circuit is a generic method that takes a Circuit instance and builds a circuit
through that instance. With Scala the deﬁnition of circuit can be even simpler: we can
avoid preﬁxing “f .” everywhere by importing f . Nevertheless, the deﬁnition shown here ismore language-independent.

Object Algebras

We need concrete factories (Object Algebras) that implement Circuit toactually invoke circuit. Here is a concrete factory that produces Circuit1 :trait Factory1 extends Circuit [Circuit1 ] {def id (x : Int)= new Id1
{val n = x}
def fan (x : Int)= new Fan1
{val n = x}
def above (x : Circuit1 , y : Circuit1 ) = new Above1 {val c1 = x; val c2 = y}def beside (x : Circuit1 , y : Circuit1 ) = new Beside1 {val c1 = x; val c2 = y}def stretch (x : Circuit1 , xs : Int∗) = new Stretch1 {val ns = xs.toList; val c = x}}where the body is identical to the smart constructors presented in Section 3.2.2. Concrete
factories for other circuit implementations can be deﬁned in a similar way by instantiating
the type parameter Circuit accordingly:

trait Factory4 extends Circuit [Circuit4 ] { ... }Concrete terms

By supplying concrete factories to abstract terms, we obtain concreteterms that can be interpreted diﬀerently:

circuit (new Factory1 { }).width// 4
circuit (new Factory4 { }).layout {x ⇒ x}// List(List((0,1),(2,3)),List((1,3)),List((1,2)))41

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

Modular extensions

Both factories and terms can be modularly reused when the DSL

is extended with new language constructs. To support right stretch for Scans, we ﬁrst
extend the abstract factory with new factory methods:

trait ExtendedCircuit [C ] extends Circuit [C ] {def rstretch (x : C, xs : Int∗) : C}
We can also build extended concrete factories upon existing concrete factories:

trait ExtendedFactory4 extends ExtendedCircuit [Circuit4 ] with Factory4 {def rstretch (x : Circuit4 , xs : Int∗) = new RStretch {val c = x; val ns = xs.toList }}Furthermore, previously deﬁned terms can be reused in constructing extended terms:

def circuit2 [C ] (f : ExtendedCircuit [C ]) = f .rstretch (circuit (f ), 2, 2, 2, 2)

3.6

Case Study: A Shallow EDSL for SQL Queries

A common motivation for using deep embeddings is performance. Deep embeddings
enable complex AST transformations, which is useful to implement optimizations that
improve the performance. An alternative way to obtain performance is to use staging
frameworks, such as Lightweight Modular Staging (LMS) [Rompf and Odersky, 2010]. As
illustrated by Rompf and Amin [2015] staging can preclude the need for AST transformations for a realistic query DSL. To further illustrate the applicability of shallow OO
embeddings, we refactored Rompf and Amin’s [2015]’s deep, external DSL implementation to make it more modular, shallow and embedded. The shallow DSL retains theperformance of the original deep DSL by generating the same code.

3.6.1 Overview
SQL is the best-known DSL for data queries. Rompf and Amin [2015] present a
SQL query processor implementation in Scala. Their implementation is an external DSL,
which ﬁrst parses a SQL query into a relational algebra AST and then executes the queryin terms of that AST. Based on the LMS framework [Rompf and Odersky, 2010], the SQL
compilers are nearly as simple as an interpreter while having performance comparable
to hand-written code. The implementation uses deep embedding techniques such asalgebraic data types (case classes in Scala) and pattern matching for representing andinterpreting ASTs. These techniques are a natural choice as multiple interpretations areneeded for supporting diﬀerent backends. But problems arise when the implementation
evolves with new language constructs. All existing interpretations have to be modiﬁed for
dealing with these new cases, suﬀering from the Expression Problem.
We refactored Rompf and Amin [2015]’s implementation into a shallow EDSL for thefollowing reasons. Firstly, multiple interpretations are no longer a problem for our shallow embedding technique. Secondly, the original implementation contains no hand-coded423.6 Case Study: A Shallow EDSL for SQL Queries

AST transformations. Thirdly, it is common to embed SQL into a general purpose language.
To illustrate our shallow EDSL, suppose there is a data ﬁle talks.csv that contains alist of talks with time, title and room. We can write several sample queries on this ﬁlewith our EDSL. A simple query that lists all items in talks.csv is:def q0 = FROM ("talks.csv")
Another query that ﬁnds all talks at 9 am with their room and title selected is:

def q1 = q0 WHERE ‘time === "09:00 AM" SELECT (‘room, ‘title)
Yet another relatively complex query to ﬁnd all conﬂicting talks that happen at the sametime in the same room with diﬀerent titles is:

def q2 = q0 SELECT (‘time, ‘room, ‘title AS ‘title1 ) JOIN
(q0 SELECT (‘time, ‘room, ‘title AS ‘title2 )) WHERE‘title1 <> ‘title2
Compared to an external implementation, our embedded implementation has the beneﬁtof reusing the mechanisms provided by the host language for free. As illustrated by the
sample queries above, we are able to reuse common subqueries (q0 ) in building complex queries (q1 and q2 ). This improves the readability and modularity of the embedded
programs.

3.6.2 Embedded Syntax
Thanks to the good support for EDSLs in Scala, we can precisely model the syntax
of SQL. The syntax of our EDSL is close to that of LINQ [Meĳer et al., 2006], whereselect is an optional, terminating clause of a query. We employ well-established OO
and Scala techniques to simulate the syntax of SQL queries in our shallow EDSL implementation. Speciﬁcally, we use the Pimp My Library pattern [Odersky, 2006] for liftingﬁeld names and literals implicitly. For the syntax of combinators such as where andjoin, we adopt a ﬂuent interface style. Fluent interfaces enable writing something like
“FROM (...).WHERE (...).SELECT (...)”. Scala’s inﬁx notation further omits “.” in method
chains. Other famous embedded SQL implementations in OOP such as LINQ [Meĳer et al.,2006] also adopt similar techniques in designing their syntax. The syntax is implemented
in a pluggable way, in the sense that the semantics is decoupled from the syntax. Detailsof the syntax implementation are beyond the scope of this thesis. The interested reader
can consult the companion code.
Beneath the surface syntax, a relational algebra operator structure is constructed.
For example, we will get the following operator structure for q1 :Project (Schema ("room", "title"),Filter (Eq (Field ("time"), Value ("09:00 AM")),
Scan ("talks.csv")))

43

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

3.6.3 A Relational Algebra Compiler
A SQL query can be represented by a relational algebra expression. The basic interfaceof operators is modeled as follows:

trait Operator {def resultSchema : Schema
def execOp (yld : Record ⇒ Unit) : Unit}
Two interpretations, resultSchema and execOp, need to be implemented for each concrete
operator: the former collects a schema for projection; the latter executes actions to therecords of the table. Very much like the interpretation layout discussed in Section 3.3.3,

execOp is both context-sensitive and dependent: it takes a callback yld and accumulates
what the operator does to records into yld and uses resultSchema in displaying execution
results. In our implementation execOp is indeed introduced as an extension just likelayout. Here we merge the two interpretations for conciseness of presentation. Some coreconcrete relational algebra operators are given below:

trait Project extends Operator {val out, in : Schema; val op : Operator
def resultSchema = out
def execOp (yld : Record ⇒ Unit) = op.execOp {rec ⇒ yld (Record (rec (in ), out))}}
trait Join extends Operator {val op1 , op2 : Operator
def resultSchema = op1 .resultSchema ++ op2 .resultSchema
def execOp (yld : Record ⇒ Unit) =op1 .execOp {rec1 ⇒op2 .execOp {rec2 ⇒val keys = rec1 .schema intersect rec2 .schema
if (rec1 (keys) ≡ rec2 (keys))yld (Record (rec1 .ﬁelds ++ rec2 .ﬁelds, rec1 .schema ++ rec2 .schema))}}}trait Filter extends Operator {val pred : Predicate; val op : Operator
def resultSchema = op.resultSchema
def execOp (yld : Record ⇒ Unit) = op.execOp {rec ⇒ if (pred.eval (rec)) yld (rec)}}Project rearranges the ﬁelds of a record; Join matches a record against another andcombines the two records if their common ﬁelds share the same values; Filter keeps arecord only when it meets a certain predicate. There are also two utility operators, Print
44

3.6 Case Study: A Shallow EDSL for SQL Queries

and Scan, for processing inputs and outputs, whose deﬁnitions are omitted for spacereasons.
From an interpreter to a compiler

The query processor presented so far is elegant

but unfortunately slow. To achieve better performance, Rompf and Amin extend the SQL
processor in various ways. One direction is to turn the slow query interpreter into a fastquery compiler by generating specialized low-level code for a given query. With the help of
the LMS framework, this task becomes rather easy. LMS provides a type constructor Rep
for annotating computations that are to be performed in the next stage. The signature of
the staged execOp is:def execOp (yld : Record ⇒ Rep [Unit ]) : Rep [Unit ]where Unit is lifted as Rep [Unit ] for delaying the actions on records to the generated code.
Two staged versions of execOp are introduced for generating Scala and C code respectively.
By using the technique presented in Section 3.3, they are added modularly with existing
interpretations such as resultSchema reused. The implementation of staged execOp is
similar to the unstaged counterpart except for minor API diﬀerences between staged andunstaged types. Hence the simplicity of the implementation remains. At the same time,
dramatic speedups are obtained by switching from interpretation to compilation.
Language extensions

Rompf and Amin also extend the query processor with two new

language constructs, hash joins and aggregates. The introduction of these constructs isdone in a modular manner with our approach:trait Group extends Operator {val keys, agg : Schema; val op : Operator
def resultSchema = keys ++ agg
def execOp (yld : Record ⇒ Unit) { ... }}trait HashJoin extends Join {override def execOp (yld : Record ⇒ Unit) = {val keys = op1 .resultSchema intersect op2 .resultSchema
val hm = new HashMapBuﬀer (keys, op1 .resultSchema)
op1 .execOp {rec1 ⇒hm (rec1 (keys)) += rec1 .ﬁelds}
op2 .execOp {rec2 ⇒hm (rec2 (keys)) foreach {rec1 ⇒yld (Record (rec1 .ﬁelds ++ rec2 .ﬁelds, rec1 .schema ++ rec2 .schema))
}
}
}
}
45

Chapter 3. Shallow EDSLs and Object-Oriented Programming: Beyond Simple Compositionality

Table 3.2: SLOC for original (Deep) and refactored (Shallow) versions.
Sourcequery_unstaged
query_staged
query_optc

Functionality

Deep

Shallow

SQL interpreter
SQL to Scala compiler
SQL to C compiler

83
179
245

98
194
262

Group supports SQL’s group by clause, which partitions records and sums up speciﬁed
ﬁelds from the composed operator. HashJoin is a replacement for Join, which uses ahash-based implementation instead of naive nested loops. With inheritance and method
overriding, we are able to reuse the ﬁeld declarations and other interpretations from Join.

3.6.4 Evaluation
We evaluate our refactored shallow implementation with respect to the original deep
implementation. Both implementations of the DSL (the original and our refactored version) generate the same code, thus the performance of the two implementations is similar. Hence we compare the two implementations only in terms of the source lines ofcode (SLOC). We exclude the code related to surface syntax for the fairness of comparison because our refactored version uses embedded syntax whereas the original uses aparser. As seen in Table 3.2, our shallow approach takes a dozen more lines of code thanthe original deep approach for each version of SQL processor. The SLOC expansion is
attributed to the fact that functional decomposition (case classes) is more compact thanobject-oriented decomposition in Scala. Nevertheless, our shallow approach makes it
easier to add new language constructs.

3.7

Conclusion

This chapter reveals the close correspondence between OOP and shallow embeddings:
the essence of both is procedural abstraction. It also showed how OOP increases themodularity of shallow EDSLs. OOP abstractions, including subtyping, inheritance, andtype-reﬁnement, bring extra modularity to traditional procedural abstraction. As a result,multiple interpretations are allowed to co-exist in shallow embeddings. Moreover, themultiple interpretations can be dependent: an interpretation can depend not only on itself
but also on other modular interpretations. Thus the approach presented here allows us togo beyond simple compositionality, where interpretations can only depend on themselves.
It has always been a hard choice between shallow and deep embeddings when designing an EDSL: there are some tradeoﬀs between the two styles. Deep embeddings tradesome simplicity and the ability to add new language constructs for some extra power. This
extra power enables multiple interpretations, as well as complex AST transformations. As
this chapter shows, in languages with OOP mechanisms, multiple (possibly dependent)
interpretations are still easy to do with shallow embeddings and the full beneﬁts of an
extended form of compositionality still apply. Therefore the motivation to employ deep
embeddings becomes weaker than before and mostly reduced to the need for AST trans46

3.7 Conclusion

formations. Prior work on the Finally Tagless [Kiselyov, 2012] and Object Algebras [Zhang
et al., 2015] approaches already show that AST transformations are still possible in thosestyles. However this requires some extra machinery, and the line between shallow anddeep embeddings becomes quite blurry at that point.
Finally, this work shows a combination of two previously studied solutions to theExpression Problem in OO: the extensible interpreter pattern proposed by Wang andOliveira [2016] and Object Algebras [Oliveira and Cook, 2012]. The combination exploits
the advantages of each of the approaches to overcome the limitations of each approach individually. In the original approach by Wang and Oliveira [2016] modular terms are hardto model, whereas with Object Algebras a diﬃculty is modeling modular dependent operations. A closely related technique is employed by Cazzola and Vacchi [2016], although
in the context of external DSLs. Their technique is slightly diﬀerent with respect to theextensible interpreter pattern. Essentially while our approach is purely based on subtyping and type-reﬁnement, they use generic types instead to simulate the type-reﬁnement.
While the focus of our work is embedded DSLs, the techniques discussed here are usefulfor other applications, including external DSLs as Cazzola and Vacchi [2016] show.

47

Chapter

4

Castor: Programming with Extensible Generative Visitors

In this chapter, we turn to metaprogramming-based design patterns. With metaprogramming, the boilerplate associated with design patterns can be largely eliminated. This
allows us to employ more powerful design patterns in solving more challenging modularity issues such as binary and producer operations without worrying too much about thecomplexity brought by these advanced design patterns. We design the Castor frameworkfor programming with extensible generative visitors. Castor supports both functional
and imperative style of visitors, pattern matching, type-safe interpreters and graphs. The
applicability of Castor is demonstrated by two case studies.

4.1 Introduction
This chapter presents Castor: an extensible and expressive Scala visitor framework.
Unlike previous work, Castor aims to support not only a functional style but also animperative programming style with visitors. Castor visitors bring several advantages
over existing approaches:
Concise notation

Programming with the Visitor pattern is typically associated with a

lot of boilerplate code. Extensible Visitors [Zenger and Odersky, 2005; Oliveira, 2009;
Hofer and Ostermann, 2010; Zhang and Oliveira, 2017] make the situation even worse
due to the heavy use of sophisticated type system features. Although previous workon EVF [Zhang and Oliveira, 2017] alleviated the burden of programmers by generating
boilerplate code related to visitors and traversals, it is restricted by Java’s syntax andannotation processor. Castor improves on EVF by employing Scala’s concise syntax and
Scalameta1 to simplify client code. Unlike Java’s annotation processor, we can directlytransform the client code with Scalameta, hiding the boilerplate and sophisticated typesystem features from users.
Pattern matching support

Castor comes with support for (type-safe) pattern match-

ing to complement its visitors with a concise notation to express operations. We identify
1

http://scalameta.org

49

Chapter 4. Castor: Programming with Extensible Generative Visitors

several desirable properties for pattern matching in an OOP context and show how existing approaches are lacking some of these properties (Section 4.2). We argue that thetraditional semantics of pattern matching, which is based on the order of patterns andadopted by many approaches, conﬂicts with the openness of data structures. Therefore
we suggest that a more restricted, top-level pattern matching model, where the order ofpatterns is irrelevant. To compensate for the absence of ordered patterns we propose
a complementary mechanism for case analysis with defaults, which can be used whennested or multiple case analysis is needed.

GADT-Style deﬁnitions

Castor supports type-safe interpreters (à la Finally Tagless),but with additional support for pattern matching and a generally recursive style. While
Finally Tagless interpreters are nowadays widely used by programmers in multiple languages (including Haskell and Scala), they must be written in fold-like style. Supporting
operations that require nested patterns, or simply depend on other operations is quite
cumbersome (although workarounds exist [Kiselyov, 2012]), especially if modularity is tobe preserved. In contrast, Castor can support those features naturally.
Hierarchical datatypes

Functional datatypes are typically ﬂat where variants have no

relationships with each other. Object-oriented style datatypes, on the other hand, canbe hierarchical [Millstein et al., 2004] where datatype constructors can be reﬁned bymore speciﬁc constructors. Hierarchical datatypes facilitate reuse since the subtyping
relationship allows the semantics deﬁned for supertypes to be reused in subtypes. Castor
exploits OOP features and employs subtyping to model hierarchical datatypes.

Imperative traversals

Castor enables many operations to be deﬁned using an imper-

ative style, which is signiﬁcantly more performant than a functional style (especially in
the JVM platform). Both functional and imperative visitors [Buchlovsky and Thielecke,2006] written with Castor are fully extensible and can later support more variants modularly. Imperative visitors enable imperative style traversals that instead of returning a
new Abstract Syntax Tree (AST), modify an existing AST in-place.

Graph structures

Finally functional techniques usually only support tree structures

well, but graph structures are poorly supported. Castor supports type-safe extensible
programming on graph structures. Compared to trees, graphs are a more general data
structure that have many important applications. In the domain of compilers, abstract
semantic graphs can be used for representing shared subexpressions, which facilitate
optimizations like common subexpression elimination.
In summary, this chapter makes the following contributions:• Extensible pattern matching with modular external visitors: We evaluate existing approaches to pattern matching in an OOP context (Section 4.2). We show how
to incoorporate extensible (or open) pattern matching support on modular external
visitors, which allows Castor to deﬁne non-trivial pattern matching operations.
50

4.2 Open Pattern Matching• Support for hierarchical datatypes: Besides ﬂat datatypes that are typically modeled in functional languages, we show how OOP style hierarchical datatypes is
supported in Castor (Section 4.3).
• Support for GADTs: We show how to use Castor’s support for GADTs in building
well-typed interpreters (Section 4.4), which would be quite diﬃcult to model in a
Finally Tagless style.
• Imperative style modular external visitors: We show how to deﬁne imperative
style modular external visitors in Castor (Section 4.5).
• Support for graph structures: We show how to do type-safe extensible programming on graph structures, which generalize the typical tree structures in functional
programming (Section 4.5).
• The Castor framework: We present a novel encoding for modular pattern matching based on extensible visitors (Section 4.2.7). The encoding is automated using
metaprogramming and the transformation is formalized (Section 4.6).
• Case studies: We conduct two case studies to illustrate the eﬀectiveness of Castor.
The ﬁrst case study on TAPL interpreters (Section 4.7) demonstrates functional
aspects of Castor, while the second one on UML activity diagrams (Section 4.8)
demonstrates the object-oriented aspects of Castor.
Source code for Castor and case studies is available at:https://github.com/wxzh/Castor

4.2 Open Pattern Matching
Pattern matching is a pervasive and useful feature in functional languages, e.g.
ML [Milner et al., 1997] and Haskell [Jones, 2003], for processing data structures conveniently. Data structures are ﬁrstly modeled using algebraic datatypes and then processed
through pattern matching. On the other hand, OOP uses class hierarchies instead of algebraic datatypes to model data structures. Still, the same need for processing data
structures also exists in OOP. However, there are important diﬀerences between data
structures modeled with algebraic datatypes and class hierarchies. Algebraic datatypes
are typically closed, having a ﬁxed set of variants. In contrast, class hierarchies are open,
allowing the addition of new variants. A closed set of variants facilitates exhaustivenesschecking of patterns but sacriﬁces the ability to add new variants. OO class hierarchies do support the addition of new variants, but without mechanisms similar to patternmatching, some programs are unwieldy and cumbersome to write. In this section, we ﬁrst
characterize four desirable properties of pattern matching in the context of OOP. We thenreview some of the existing pattern matching approaches in OOP and discuss why theyfall in short of the desirable properties. This section ends with an overview of Castor andan evaluation summary on the presented approaches.
51

Chapter 4. Castor: Programming with Extensible Generative Visitors

4.2.1 Desirable Properties of Open Pattern Matching
We identify the following desirable properties for pattern matching in an OOP context:• Conciseness. Patterns should be described concisely with potential support for
wildcards, deep patterns, and guards.
• Exhaustiveness. Patterns should be exhaustive to avoid runtime matching failure.
The exhaustiveness of patterns should be statically veriﬁed by the compiler and themissing cases should be reported if patterns are incomplete.
• Extensibility. Datatypes should be extensible in the sense that new data variants can be added while existing operations can be reused without modiﬁcation or
recompilation.
• Composability. Patterns should be composable so that complex patterns can bebuilt from smaller pieces.

When composing overlapped patterns, programmers

should be warned about possible redundancies.
Using these properties as criteria, we next evaluate pattern matching approaches in
OOP. We show that many widely used approaches lack some of these properties. We argue
that a problem is that many approaches try to closely follow the traditional semantics ofpattern matching, which assumes a closed set of variants. Under a closed set of variants,
it is natural to use the order of patterns to prioritize some patterns over the others.
However, when the set of variants is not predeﬁned a priori then relying on some orderingof patterns is problematic, especially if separate compilation and modular type-checking
are to be preserved. Nonetheless, many OO approaches, which try to support both anextensible set of variants and pattern matching, still try to use the order of patterns
to deﬁne the semantics. Unfortunately, this makes it hard to support other desirable
properties such as exhaustiveness or composability.

4.2.2 Running Example: Arith
To facilitate our discussion, a running example from TAPL [Pierce, 2002]—an untyped,
arithmetic language called Arith—is used throughout this chapter. The syntax and semantics of Arith are formalized in Figure 4.1. Our goal is to model the syntax andsemantics of Arith in a concise and modular manner.
Arith has the following syntactic forms: zero, successor, predecessor, true, false,
conditional and zero test. The deﬁnition nv identiﬁes 0 and successive application of
succ to 0 as numeric values. The operational semantics of Arith is given in small-stepstyle, with a set of reduction rules specifying how a term can be rewritten in one step.
Repeatedly applying these rules will eventually evaluate a term to a value. There might bemultiple rules deﬁned on a single syntactic form. For instance, rules PredZero, PredSucc
and Pred are all deﬁned on a predecessor term. How pred t is going to be evaluated in thenext step is determined by the shape of the inner term t: if t is 0, then PredZero will be52

4.2 Open Pattern Matching

t
nv

::=
::=

0 | succ t | pred t | true | false | if t then t else t | iszero t
0 | succ nv

t1 → t100succ t1 → succ t1

pred 0 → 0 PredZero

t1 → t100pred t1 → pred t1

Pred

pred (succ nv1 ) → nv1 PredSucc

if true then t2 else t3 → t2

t1 → t100
if t1 then t2 else t3 → if t1 then t2 else t3

if false then t2 else t3 → t3

t1 → t10
iszero 0 → trueiszero (succ nv1 ) → false0iszero t1 → iszero t1

Figure 4.1: The syntax and semantics of Arith.
applied; if t is a successor application to a numeric value, then PredSucc will be applied;otherwise pred will be applied.
Arith is a good example for assessing the four properties because: 1) The small-stepstyle semantics is best expressed with a concise nested case analysis on terms; 2) Arithis, in fact, a uniﬁcation of two sublanguages, Nat (zero, successor and predecessor) and
Bool (true, false, and conditional) plus an extension (zero test). Ideally, Nat and Boolshould be separately deﬁned and modularly reused.

4.2.3 The Visitor Pattern
The Visitor design pattern [Gamma et al., 1994] is frequently used to implement
interpreters or compilers because of its ability to add new interpretations or compiler
phases without modifying the class hierarchy. Let us implement the Arith language
using the Visitor pattern step by step. The implementation is written in Scala without
using any Scala-speciﬁc features and can be easily mapped to other OOP languages like
C++ or Java.
Abstract syntax

The abstract syntax of Arith is modeled by the following class hierar-chy:
abstract class Tm {def accept[A](v: TmVisit[A]): A}
class TmZero() extends Tm {def accept[A](v: TmVisit[A]) = v.tmZero(this)}class TmSucc(val t: Tm) extends Tm {def accept[A](v: TmVisit[A]) = v.tmSucc(this)}class TmPred(val t: Tm) extends Tm {def accept[A](v: TmVisit[A]) = v.tmPred(this)53

Chapter 4. Castor: Programming with Extensible Generative Visitors}class TmTrue() extends Tm {def accept[A](v: TmVisit[A]): A = v.tmTrue(this)}class TmFalse extends Tm {def accept[A](v: TmVisit[A]): A = v.tmFalse(this)}class TmIf(val t1: Tm, val t2: Tm, val t3: Tm) extends Tm {def accept[A](v: TmVisit[A]): A = v.tmIf(this)}class TmIsZero(val t: Tm) extends Tm {def accept[A](v: TmVisit[A]): A = v.tmIsZero(this)}The abstract class Tm represents the datatype of terms, and syntactic constructs of terms
are subclasses of Tm. A generic accept method is deﬁned throughout the class hierarchy,
which is implemented by invoking the corresponding lowercase visit method exposed by
TmVisit.

Visitor interface

TmVisit is the visitor interface that declares all the visit methods re-quired by accept implementations. Its deﬁnition is given below:
trait TmVisit[A] {def tmZero(x: TmZero): A
def tmSucc(x: TmSucc): A
def tmPred(x: TmPred): A
def tmTrue(x: TmTrue): A
def tmFalse(x: TmFalse): A
def tmIf(x: TmIf): A
def tmIsZero(x: TmIsZero): A}
TmVisit is parameterized by A for abstracting over the return type of visit methods. Eachvisit method takes an instance of its corresponding class and returns a value of A.
Concrete visitors

Operations over Tm are concrete visitors that implement the visitor

interface TmVisit. The numeric value checker is deﬁned like this:class Nv extends TmVisit[Boolean] {def tmZero(x: TmZero) = true
def tmSucc(x: TmSucc)= x.t.accept(this)def tmPred(x: TmPred) = false
def tmTrue(x: TmTrue) = false
def tmFalse(x: TmFalse) = false
def tmIf(x: TmIf) = false
def tmIsZero(x: TmIsZero) = false}
Nv implements TmVisit by instantiating the type parameter A as Boolean and giving animplementation to each visit method. Here, the interesting cases are tmZero and tmSucc.
For the former, a true is returned; for the latter, we call _.t.accept(this) for recursively
applying Nv to check the inner term. The remaining cases are not numeric values thus
return false.
544.2 Open Pattern Matching

With Nv deﬁned, we can now implement the small-step evaluation visitor:
class Eval1 extends TmVisit[Tm] {//val eval1 = thisval nv = new Nv// Dependency on the visitor itself// Dependency on another visitor

def tmZero(x: TmZero) = throw NoRuleApplies
def tmSucc(x: TmSucc) = new TmSucc(x.t.accept(this))def tmPred(x: TmPred) = x.t.accept(new TmVisit[Tm] {def tmZero(y: TmZero) = y// PredZero

def tmSucc (y: TmSucc) =if (y.t.accept(nv)) y.t// PredSucc

else new TmPred(y.t.accept(eval1))// Preddef tmPred(y: TmPred) = new TmPred(y.accept(eval1))// Preddef tmTrue(y: TmTrue) = new TmPred(y.accept(eval1))// Preddef tmFalse(y: TmFalse) = new TmPred(y.accept(eval1))// Preddef tmIf(y: TmIf) = new TmPred(y.accept(eval1))// Preddef tmIsZero(y: TmIsZero) = new TmPred(y.accept(eval1)) // Pred})def tmTrue(x: TmTrue) = throw NoRuleApplies
def tmFalse(x: TmFalse)= throw NoRuleApplies

def tmIf(x: TmIf) = x.t1.accept(new TmVisit[Tm] {def tmTrue(y: TmTrue) = x.t2
def tmFalse(y: TmFalse) = x.t3
def tmZero(y: TmZero) = new TmIf(y.accept(eval1),x.t2,x.t3)
def tmSucc(y: TmSucc) = new TmIf(y.accept(eval1),x.t2,x.t3)
def tmPred(y: TmPred) = new TmIf(y.accept(eval1),x.t2,x.t3)
def tmIf(y: TmIf) = new TmIf(y.accept(eval1),x.t2,x.t3)
def tmIsZero(y: TmIsZero) = new TmIf(y.accept(eval1),x.t2,x.t3)})
def tmIsZero(x: TmIsZero) = x.t.accept(new TmVisit[Tm] {def tmZero(y: TmZero) = new TmTrue
def tmSucc (y: TmSucc) =if (y.t.accept(nv)) new TmFalse
else new TmIsZero(y.accept(eval1))
def tmPred(y: TmPred) = new TmIsZero(y.accept(eval1))
def tmTrue(y: TmTrue) = new TmIsZero(y.accept(eval1))
def tmFalse(y: TmFalse) = new TmIsZero(y.accept(eval1))
def tmIf(y: TmIf) = new TmIsZero(y.accept(eval1))
def tmIsZero(y: TmIsZero) = new TmIsZero(y.accept(eval1))})}

The small-step evaluator rewrites a term to another thus A is instantiated as Tm. Since
primitive cases are already values, we simply throw a NoRuleApplies exception for tmZero,tmTrue and tmFalse. Deﬁning the case for tmSucc is easy too: we construct a new successor

with its inner term rewritten by eval1. In contrast, deﬁning tmPred, tmIf and tmIsZero
is trickier because they all have multiple rules. Take tmPred for example. As a visitor
recognizes only one level representation of a term, it is insuﬃcient to encode rules that
require nested case analysis. To further reveal the shape of the inner term, anonymous
visitors are created. Rules like PredSucc can then be speciﬁed inside the tmSucc method of
the inner visitor. Moreover, the inner visitor of tmPred depends on both Eval1 and Nv. These
dependencies are expressed by the ﬁelds eval1 and nv, which are instantiated as visitor
instances. Then we can pass eval1 or nv as an argument to the accept method for using
55

Chapter 4. Castor: Programming with Extensible Generative Visitors

the dependency. Notice that the Pred rule is repeated 6 times. Similar situations also
happen in tmIf and tmIsZero, making the overall implementation of Eval1 quite lengthy.
Client code

We can write some tests for our implementation of Arith:// iszero (if false then true else pred (succ 0))val tm = new TmIsZero(new TmIf(new TmFalse,new TmTrue,new TmPred(new TmSucc(newTmZero))))val eval1 = new Eval1
val tm1 = tm.accept(eval1)// iszero (pred (succ 0))val tm2 = tm1.accept(eval1) // iszero 0
val tm3 = tm2.accept(eval1) // 0

where we construct a term using all syntactic forms of the Arith language and evaluate
it step by step using eval1. The evaluation result of each step is shown in the commentson the right hand side.
Discussion of the approach

The conventional Visitor pattern has been criticized forits verbosity and inextensibility [Meyer and Arnout, 2006; Pati and Hill, 2014], which
are manifested in the implementation of Arith. Programming with the Visitor pattern is
associated with a lot of infrastructure code, including the visitor interface, the class hierarchy, etc. Writing such infrastructure manually is tedious and error-prone, especiallywhen there are many classes involved. Such verbosity restricts the usage Visitor pattern,
as Martin [2002] wrote:“Often, something that can be solved with a Visitor can also be solved bysomething simpler.”Moreover, the Visitor pattern suﬀers from the Expression Problem [Wadler, 1998]: it iseasy to add new operations by deﬁning new visitors (as illustrated by nv and eval1) but
hard to add new variants. The reason is that Tm and TmVisit are tightly coupled. When trying to add new subclasses to the Tm hierarchy, it is not possible to implement their accept
methods because there exist no corresponding visit methods in TmVisit. A non-solution
is to modify TmVisit with new visit methods. As a consequence, all existing concrete implementations of TmVisit have to be modiﬁed in order to account for those variants. Thisviolates the “no modiﬁcation on existing code” principle of the Expression Problem. Modiﬁcation is even impossible if the source code is unavailable. As a result, Nat and Boolcannot be separated from Arith. Thus, the whole implementation is neither extensible
nor composable. Nevertheless, the exhaustiveness on visit methods is guaranteed sincea class cannot contain any abstract methods.

4.2.4 Sealed Case Classes
The Visitor pattern is often used as a poor man’s approach to pattern matchingin OO languages. Fortunately, Scala [Odersky et al., 2004] is a language that uniﬁesfunctional and OO paradigms and supports pattern matching natively via case classes/extractors [Emir et al., 2007]. Case classes can be either open or sealed. Sealed case56

4.2 Open Pattern Matchingclasses are close to algebraic datatypes in functional languages, which have a ﬁxed set ofvariants.
Representing the Tm hierarchy using sealed case classes looks like this:sealed trait Tmcase object TmZero extends Tmcase class TmSucc(t: Tm) extends Tmcase class TmPred(t: Tm) extends Tmcase object TmTrue extends Tmcase object TmFalse extends Tmcase class TmIf(t1: Tm, t2: Tm, t3: Tm) extends Tmcase class TmIsZero(t: Tm) extends Tm

The diﬀerences are that Tm is a sealed trait and variants of Tm are additionally marked
as case. Also, no-argument variants are Scala’s singleton objects and ﬁelds of case classes
are by default val.
The case keyword triggers the Scala compiler to automatically inject methods into aclass, including a constructor method (apply) and an extractor method (unapply). The
injected constructor method simpliﬁes creating objects from case classes. For example,
a successor application to zero can be constructed via TmSucc(TmZero). Conversely, theinjected extractor enables tearing down an object via pattern matching.
The numeric value checker can be deﬁned by pattern matching on the term:
def nv(t: Tm): Boolean = t match {case TmZero => truecase TmSucc(t1) => nv(t1)case _ => false}

The term t is matched sequentially against a series of patterns (case clauses). For example,
TmSucc(TmZero) will be handled by the second case clause of nv, which recursively invokes
nv on its subterm t1 (which is TmZero). Then, TmTrue will be matched by the ﬁrst case clausewith a true returned eventually. A wildcard pattern (_) is used in the last case clause forhandling boring cases altogether.
The strength of pattern matching shines in encoding the small-step semantics:
def eval1(t: Tm): Tm = t match {case TmSucc(t1) => TmSucc(eval1(t1))
case TmPred(TmZero) => TmZero// PredZerocase TmPred(TmSucc(t1)) if nv(t1) => t1 // PredSucc
case TmPred(t1) => TmPred(eval1(t1))
case TmIf(TmTrue,t2,_) => t2// Predcase TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(eval1(t1),t2,t3)
case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t1)) if nv(t1) => TmFalse
case TmIsZero(t1) => TmIsZero(eval1(t1))
case _ => throw NoRuleApplies}

With the help of pattern matching, the overall deﬁnition is a direct mapping from theformalization shown in Figure 4.1. There is a one-to-one correspondence between the57

Chapter 4. Castor: Programming with Extensible Generative Visitors

rules and the case clauses. For example, PredSucc is concisely described by a deep
pattern (TmPred(TmSucc(t1))) with a guard (if nv(t1)) and Pred is captured only once by
TmPred(t1).

Client code

The client code is also more natural and compact than that in visitors:

val tm = TmIsZero(TmIf(TmFalse,TmTrue,TmPred(TmSucc(TmZero))))val tm1 = eval1(tm)// iszero (pred (succ 0))val tm2 = eval1(tm1) // iszero 0
val tm3 = eval1(tm2) // 0

where new clauses are no longer needed.
Discussion of the approach

The Arith implementation using sealed case classes isvery concise. Moreover, sealed case classes facilitate exhaustiveness checking on patterns
since all variants are statically known. If we forgot to write the wildcard pattern in nv, theScala compiler would warn us that a case clause for TmPred is missing. An exception iseval1, whose exhaustiveness is not checked by the compiler due to the use of guards. The

reason is that a guard might call some function whose execution result is only known atruntime, making the reachability of that pattern diﬃcult to decide statically. The price to
pay for exhaustiveness is the inability to add new variants of Tm in separate ﬁles. Thus,
like the visitor version, the implementation is neither extensible nor composable.

4.2.5 Open Case Classes
While the implementation using sealed case classes is concise, it is not modular
because Arith is still deﬁned as a whole. To separate out Nat and Bool, we turn to open
case classes by trading exhaustiveness checking for the ability to add new variants inseparate ﬁles. To make up for the loss of exhaustiveness, Zenger and Odersky’s [2001]
idea of Extensible Algebraic Datatypes with Defaults (EADDs) can be applied. The key
idea is to always use a default in each operation to handle variants that are not explicitly
mentioned. The existence of a default makes operations extensible, as variants added later
will be automatically subsumed by that default. If the extended variants have behavior
diﬀerent from the default, we can deﬁne a new operation that deals with the extended
variants and delegates to the old operation.
We ﬁrst remove the sealed constraint on Tm and specify the default behavior of eval1
inside a trait Term:
trait Term {trait Tm
def eval1(t: Tm): Tm = throw NoRuleApplies}

Then, Nat can be deﬁned as an extended trait for Term:
trait Nat extends Term {case object TmZero extends Tmcase class TmSucc(t: Tm) extends Tmcase class TmPred(t: Tm) extends Tm

58

4.2 Open Pattern Matchingdef nv(t: Tm): Boolean = t match {case TmZero => truecase TmSucc(t1) => nv(t1)case _ => false}
override def eval1(t: Tm): Tm = t match {case TmSucc(t1) => TmSucc(eval1(t1))
case TmPred(TmZero) => TmZero// PredZerocase TmPred(TmSucc(t1)) if nv(t1) => t1 // PredSucc
case TmPred(t1) => TmPred(eval1(t1))
case _ => super.eval1(t)// Pred}}Nat introduces TmZero, TmSucc and TmPred as variants of Tm. nv is deﬁned in the old way.
eval1 is overridden with case clauses for TmSucc and TmPred, and TmZero is dealt by Term’s
eval1 via a super call.

Similarly, Bool is deﬁned as another trait that extends Tm with its own variants andeval1:
trait Bool extends Term {case object TmTrue extends Tmcase object TmFalse extends Tmcase class TmIf(t1: Tm,t2: Tm,t3: Tm) extends Tm
override def eval1(t: Tm): Tm = t match {case TmIf(TmTrue,t2,_) => t2
case TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(eval1(t1),t2,t3)
case _ => super.eval1(t)}}Finally, Arith can be deﬁned as a uniﬁcation of Nat and Bool implementations:trait Arith extends Nat with Bool {case class TmIsZero(t: Tm) extends Tm
override def eval1(t: Tm) = t match {case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t1)) if nv(t1) => TmFalse
case TmIsZero(t1) => TmIsZero(eval1(t1))
case TmZero => super[Nat].eval1(t)case _: TmSucc => super[Nat].eval1(t)case _: TmPred => super[Nat].eval1(t)case _ => super[Bool].eval1(t)}}Scala’s mixin composition allows Arith to extend both Nat and Bool. The deﬁnition nv
inherited from Nat works well in Arith, as it happens to have a very good default that
automatically ﬁts for the new cases. For instance, calling nv(TmFalse) returns false as
expected. However, overridding eval1 becomes problematic. We cannot simply complement the cases for TmIsZero and handle all the inherited cases at once since both Nat andBool are extended. Instead we have to separate the inherited cases using typecases anddelegate appropriately to either Nat or Bool via super calls.
59

Chapter 4. Castor: Programming with Extensible Generative Visitors

Discussion of the approach

Combining open case classes with EADDs brings extensi-

bility. This idea works well for linear extensions (such as Nat and Bool) but not so wellfor non-linear extensions like Arith. As shown by eval1 in Arith, composing non-linear
extensions is tedious and error-prone. Without any assistance from the Scala compiler
during this process, it is rather easy to make mistakes like forgetting to delegate a caseor delegating a case to a wrong parent. Moreover, the exhaustiveness checking on caseclauses is lost. Although in the spirit of EADDs case clauses should always end with awildcard that ensures exhaustiveness, it is not enforced by the Scala compiler.

4.2.6 Partial Functions
To ease the composition of Nat and Bool, one may consider Scala’s PartialFunction.
PartialFunction provides an orElse method for composing partial functions. orElse tries

the composed partial functions sequentially until no MatchError is raised.
The open case class version of Arith can be adapted to a partial function version witha few changes. First, eval1 in Term should be declared as a partial function:
def eval1: PartialFunction[Tm,Tm]

Second, wildcards cannot be used in implementing eval1 anymore because they will
shadow other partial functions to be composed. For example, eval1 in Bool is rewritten as:
override def eval1 {case TmIf(TmTrue,t2,_) => t2
case TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(eval1(t1),t2,t3)
case TmTrue => throw NoRuleApplies
case TmFalse => throw NoRuleApplies}

An instance of PartialFunction[Tm,Tm] is constructed using the anonymous function syntax with the argument Tm being directly pattern matched. The wildcard pattern is replacedby two constructor patterns TmTrue and TmFalse with identical right hand side, losing someconvenience. Nevertheless, partial functions make the composition work more smoothly,
avoiding the problems caused by the open case classes approach:override def eval1 = super[Nat].eval1 orElse super[Bool].eval1 orElse {case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t1)) if nv(t1) => TmFalse
case TmIsZero(t1) => TmIsZero(eval1(t1))}
eval1 is overridden by chaining eval1 from Nat and Bool as well as a new partial function

for the zero test using the orElse combinator.

Discussion of the approach

Although combining open case classes with partial func-

tions makes the composition smoother, it is still not fully satisfactory. The orElse combinator is left-biased, thus the composition order determines the composed semantics. That60

4.2 Open Pattern Matchingis, f orElse g is not equivalent to g orElse f, if f and g are two overlapped partial functions (i.e. containing case clauses with identical left hand side but diﬀerent right handside). When composing such overlapped partial functions, orElse gives no warning. Also,
the semantics of the overlapped patterns are all from either f or g, depending on whichcomes ﬁrst. It is not possible to have a mixed semantics for overlapped patterns (e.g. picking case A from f and case B from g when both f and g deﬁne case A and case B), which
restricts the reusability of partial functions. Lastly, partial functions rely on exception
handling, which has a negative impact on performance.

4.2.7 Extensible Visitors
Essentially what makes pattern matching hard to be extended or composed is theorder-sensitive semantics of pattern matching and wildcard patterns that cover both
known and unknown variants. We think it is useful to distinguish between top-level (shallow) patterns and nested (deep) patterns. Top-level patterns should be order-insensitiveand partitioned into multiple deﬁnitions so that they can be easily composed. We canachieve this by combining open case classes with extensible visitors [Zenger and Odersky,
2005; Oliveira, 2009; Hofer and Ostermann, 2010; Zhang and Oliveira, 2017].
The Arith implementation is organized in a way similar to the open case classes
approach. Let us start with Term:
trait Term {type TmV <: TmVisit
trait Tm { def accept(v: TmV): v.OTm }trait TmVisit { _: TmV =>type OTm
def apply(t: Tm) = t.accept(this)}trait TmDefault extends TmVisit { _: TmV =>def tm: Tm => OTm}
trait Eval1 extends TmDefault { _: TmV =>type OTm = Tm
def tm = _ => throw NoRuleApplies}
val eval1: Eval1}

Instead of using TmVisit in declaring the accept method, we use an abstract type member
TmV and constrain it to be a subtype of TmVisit. This enables invocations on the methods

declared inside TmVisit, but at the same time, decouples Tm from TmVisit. The upperbound of the return type of the visit methods is also captured by an abstract type rather
than a type parameter for avoiding reinstantiation in inherited visitors. Accordingly, thereturn type of accept is now a path dependent type v.OTm. A syntactic sugar method apply
is deﬁned inside TmVisit for enabling v(x) as a shorthand of x.accept(v), where x andv are instances of Tm and TmVisit, respectively. To pass this as an argument of accept

in implementing apply, we state that TmVisit is of type TmV using a self-type annotation.
To mimic wildcards, we use default visitors [Nordberg III, 1996]. But unlike wildcards,
61

Chapter 4. Castor: Programming with Extensible Generative Visitors

default visitors only deal with known variants. TmDefault is the default visitor interface,which extends TmVisit with a generic tm method for specifying the default behavior. Eval1
is a default visitor thus it extends TmDefault, speciﬁes the output type OTm as Tm and implements tm. Each concrete visitor has a companion val declaration for allowing themselves
to be used in other visitors.
The encoding makes more sense with the implementation of Nat given:
trait Nat extends Term {type TmV <: TmVisit
case object TmZero extends Tm {def accept(v: TmV): v.OTm = v.tmZero}case class TmSucc(t: Tm) extends Tm {def accept(v: TmV): v.OTm = v.tmSucc(this)}case class TmPred(t: Tm) extends Tm {def accept(v: TmV): v.OTm = v.tmPred(this)}trait TmVisit extends super.TmVisit { _: TmV =>def tmZero: OTm
def tmSucc: TmSucc => OTm
def tmPred: TmPred => OTm}
trait TmDefault extends TmVisit with super.TmDefault { _: TmV =>def tmZero = tm(TmZero)def tmSucc = tm(_)
def tmPred = tm(_)}
def nv(t: Tm): Boolean = t match {case TmZero => truecase TmSucc(t1) => nv(t1)case _ => false}
trait Eval1 extends TmDefault with super.Eval1 { _: TmV =>override def tmSucc = x => TmSucc(this(x.t))
override def tmPred = {case TmPred(TmZero) => TmZerocase TmPred(TmSucc(t)) if nv(t) => tcase TmPred(t) => TmPred(this(t))}}}
Tm is extended with several case classes/objects. Correspondingly TmVisit is extended

with new visit methods and TmV is covariantly reﬁned as the subtype of the extended
TmVisit. Visit methods are declared using Scala’s functions instead of ordinary methods

for two reasons. First, the argument type (e.g. TmSucc) has already been revealed by themethod name (tmSucc) and can be inferred by the Scala compiler without losing information. Second, ﬁrst-class functions facilitate pattern matching on the argument. These
two advantages result in a concise deﬁnition of Eval1, where the type of x is omitted anda value of TmPred => Tm is constructed by pattern matching. Unlike conventional visitors,
nested case analysis is much simpliﬁed via (nested) pattern matching rather than aux62

4.2 Open Pattern Matchingiliary visitors. For example, when a predecessor term is processed by Eval1, it will berecognized and dispatched to the tmPred method. Then the TmPred object is matched by
the case clauses. As these are case clauses, deep patterns and guards can be used. To
restore the convenience of wildcards for top-level patterns, TmDefault is used, which implements visit methods by delegating to tm. Notice that Eval1 is deﬁned as a trait insteadof a class for enabling mixin composition. By extending both TmDefault and super.Eval1,
Eval1 only needs to override interesting cases.

The numeric value checker is deﬁned as a method rather than a visitor. This isbecause, as we have discussed, nv is a good candidate for applying EADDs. Of course,
nv can be deﬁned as a default visitor like Eval1. But whenever Nat is extended with newterms, the deﬁnition of nv has to be reﬁned by composing Nv with the extended TmDefault.
Bool is deﬁned in a similar manner:trait Bool extends Term {type TmV <: TmVisit
trait TmVisit extends super.TmVisit { _: TmV =>def tmTrue: OTm
def tmFalse: OTm
def tmIf: TmIf => OTm}
trait TmDefault extends TmVisit with super.TmDefault { _: TmV =>def tmTrue = tm(TmTrue)def tmFalse = tm(TmFalse)
def tmIf = tm}
case object TmTrue extends Tm {override def accept(v: TmV) = v.tmTrue}case object TmFalse extends Tm {override def accept(v: TmV) = v.tmFalse}
case class TmIf(t1: Tm, t2: Tm, t3: Tm) extends Tm {override def accept(v: TmV) = v.tmIf(this)}trait Eval1 extends TmDefault with super.Eval1 { _: TmV =>override def tmIf = {case TmIf(TmTrue,t2,_) => t2
case TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(this(t1), t2, t3)}}}With case clauses partitioned into visit methods according to their top-level pattern,
unifying Nat and Bool becomes easy via Scala’s mixin composition:
trait Arith extends Nat with Bool {type TmV <: TmVisit
case class TmIsZero(t: Tm) extends Tm {override def accept(v: TmV) = v.tmIsZero(this)}trait TmVisit extends super[Nat].TmVisit
with super[Bool].TmVisit { _: TmV =>63

Chapter 4. Castor: Programming with Extensible Generative Visitors

def tmIsZero: TmIsZero => OTm}
trait TmDefault extends TmVisit with super[Nat].TmDefault
with super[Bool].TmDefault { _: TmV =>def tmIsZero = tm}
trait Eval1 extends TmVisit with super[Nat].Eval1
with super[Bool].Eval1 { _: TmV =>def tmIsZero = {case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t)) if nv(t) => TmFalse
case TmIsZero(t) => TmIsZero(this(t))}}}

Deﬁning Eval1 for Arith only needs to inherit Eval1 deﬁnitions from Nat and Bool andcomplement the tmIsZero method. Since tmIsZero is an interesting case, Eval1 extends
TmVisit rather than TmDefault.

Instantiation

Components deﬁned in this way cannot be directly used in client code.

An additional step to instantiate traits into objects is required. Instantiating Arith, forexample, is done like this:object Arith extends Arith {type TmV = TmVisit
object eval1 extends Eval1}

The companion object Arith binds the abstract type TmV to its corresponding the visitor
interface TmVisit. The eval1 declaration is met by a singleton object that extends Eval1. If
Eval1 does not implement all the visit methods, the object creation fails, with the missing

methods reported.
Client code

Now we can use the companion object Arith in client code:

import Arith._val tm = TmIsZero(TmIf(TmFalse,TmTrue,TmPred(TmSucc(TmZero))))val tm1 = eval1(tm)// iszero (pred (succ 0))val tm2 = eval1(tm1) // iszero 0
val tm3 = eval1(tm2) // 0

By importing Arith, the constructors and visitors deﬁned inside Arith are in scope. With
the syntactic sugar deﬁned for visitors, a term can be constructed and evaluated identically to the case class version.
Discussion of the approachWith the powerful extensible visitor encoding, the Arith

implementation is made both extensible and composable. However, extensible visitors
are even more verbose than conventional ones. The use of traits in implementing visitors
brings composability but, at the same time, requires extra instantiation code. Anotherdownside of using traits is that the exhaustiveness checking on visit methods is deferred64

4.2 Open Pattern Matching

to the instantiation stage. Moreover, the encoding relies on advanced features of Scala,making it less accessible to novice Scala programmers.

4.2.8 EVF
Programming with visitors can be greatly simpliﬁed with the associated infrastructure
automatically generated. This idea has been adopted in our previous work on EVF [Zhang
and Oliveira, 2017], which employs Java annotation processors for generating extensible
visitor infrastructure.
EVF uses Object Algebra interfaces [Oliveira and Cook, 2012] to describe the abstract
syntax:@Visitor interface TmAlg<Tm> {Tm TmZero();Tm TmSucc(Tm t);
Tm TmPred(Tm t);}

where the type parameter Tm represents the datatype and capitalized methods that return Tm represent variants of Tm. Annotated as @Visitor, TmAlg will be recognized andprocessed by EVF. Then the infrastructure for TmAlg will be generated, including a classhierarchy, a visitor interface and various default visitors. Based on the generated visitor
infrastructure, we are able to implement Nv:
interface Nv<Tm> extends TmAlgDefault<Tm,Boolean> {@Override default Zero<Boolean> m() {return () -> false;}
default Boolean TmZero() {return true;}default Boolean TmSucc(Tm t) {return visitTm(t);}}
Nv is deﬁned as an interface with visit methods implemented using default methods forretaining composability. The Java extensible visitor encoding adopted by EVF is, however,
not as powerful as the Scala one shown in Section 4.2.7, which does not support modular ASTs. Whenever an annotated Object Algebra interface gets extended, a new classhierarchy is generated. Thus, we cannot refer to a concrete datatype directly in visitorssince this will make them inextensible. Instead, datatypes are kept abstract in visitors.
To traverse an abstract datatype like Tm, visitTm is called. visitTm is a method exposedby the generated visitor interface, similar to apply shown in Section 4.2.7. TmAlgDefault isthe default visitor similar to TmDefault, where the default behavior is speciﬁed inside m().
Deﬁning Eval1 is tricker:
interface Eval1<Tm> extends TmAlgDefault<Tm,Tm>, tm.Eval1<Tm> {TmAlgMatcher<Tm,Tm> matcher(); // Dependency for nested case analysis
TmAlg<Tm> f();// Dependency for AST reconstruction

Nv<Tm> nv();// Dependency for another visitor

65Chapter 4. Castor: Programming with Extensible Generative Visitors@default Tm TmPred(Tm t) {return matcher().TmZero(() -> t).TmSucc(t1 -> nv().visitTm(t1) ? t1 : TmPred(visitTm(t))).otherwise(() -> f().TmPred(visitTm(t))).visitTm(t);}default Tm TmSucc(Tm t) {return f().TmSucc(visitTm(t));}}

There are three dependencies declared using abstract methods. Firstly, since Java does
not support native pattern matching, the matcher dependency is convenient for constructing anonymous visitors. matcher returns an instance of the generated TmAlgMatcher interface, which provides ﬂuent setters for deﬁning visit methods via Java 8’s lambdas. The
otherwise setter mimics the wildcard pattern. Secondly, the reconstruction of a term isdone via an abstract factory f of type TmAlg<Tm>. Lastly, the abstract method nv expresses
the dependency on the visitor Nv.
Bool is implemented similarly in another package bool, whose deﬁnition is omitted.
The implementation of Arith is more interesting, which is shown below:@Visitor interface TmAlg<Tm> extends nat.TmAlg<Tm>, bool.TmAlg<Tm> {Tm TmIsZero(Tm t);}interface Eval1<Tm> extends GTmAlg<Tm,Tm>,bool.Eval1<Tm>,nat.Eval1<Tm> {TmAlgMatcher<Tm,Tm> matcher(); // Dependency refinement
TmAlg<Tm> f();// Dependency refinement

default Tm TmIsZero(Tm t) {return matcher().TmZero(() -> f().TmTrue()).TmSucc(t1 -> nv(t1) ? f().TmFalse() : f().TmIsZero(visitTm(t))).otherwise(() -> f().TmIsZero(visitTm(t))).visitTm(t);}}interface Nv<Tm> extends TmAlgDefault<Tm,Boolean>, nat.Nv<Tm> {}Nat and Bool implementations are merged using Java 8’ multiple interface inheritance.
Despite complementing TmIsZero, return types of dependencies are covariantly reﬁned forallowing TmIsZero calls. Since Nv is implemented as a visitor, it needs to be reﬁned as well.
Instantiation

Instantiating interfaces into classes for creating objects is also required:static class NvImpl implements Nv<CTm>, TmAlgVisitor<Boolean> {}static class Eval1Impl implements Eval1<CTm>, TmAlgVisitor<CTm> {public TmAlg<CTm> f() { return f; }public TmAlgMatcher<CTm,CTm> matcher() {return new TmAlgMatcherImpl<>();}
public Nv<CTm> nv() { return nv; }}static TmAlgFactory f = new TmAlgFactory();66

4.2 Open Pattern Matchingstatic NvImpl nv = new NvImpl();static Eval1Impl eval1 = new Eval1Impl();

The interfaces are instantiated into classes with a suﬃx Impl. Eval1Impl, for example,
implements Eval1 by: 1) instantiating Tm as the generated datatype CTm; 2) inheriting thegenerated TmAlgVisitor for a visitTm implementation; 3) fullﬁlling the dependencies using
TmAlgFactory, TmAlgMatcherImpl and NvImpl respectively.

Client code

The term is constructed via the factory object f and can be evaluated likethis:
CTm tm = f.TmIsZero(f.TmIf(f.TmFalse(),f.TmTrue(),f.TmPred(f.TmSucc(f.TmZero()))));
eval1.visitTm(eval1.visitTm(eval1.visitTm(tm)));Discussion of the approach

EVF simpliﬁes programming with visitors through code

generation. It further addresses the extensibility issue by adopting extensible visitors.
Restricted by Java, nested case analysis in EVF is done by means of anonymous visitors,
which is not as expressive and concise as pattern matching in Scala. To enable composability, EVF visitors are deﬁned using Java 8’s interfaces with default methods—inthe same spirit of using traits in Scala. Consequently, the exhaustiveness checking on
the top-level visit methods is lost in visitor deﬁnition site and is delayed to the visitor instantiation site. Nevertheless, the exhaustiveness on the visit methods of the anonymous
visitors is guaranteed because the otherwise setter must be called when constructing ananonymous visitor.

4.2.9 Castor
Highly inspired by EVF, Castor is a Scala framework designed for programming withgenerative, extensible visitors. Castor improves on EVF in two aspects. First, Castor
adopts a more powerful Scala extensible visitor encoding presented in Section 4.2.7 that
additionally enables pattern matching, GADTs, hierarchical datatypes, graphs, etc. Second, Castor employs Scalameta for annotation processing, which allows not only generating new code based on the annotated code but also modifying the annotated code
itself. These extra abilities together result in more concise and expressive visitor code
than that in EVF. We next give a modular implementation of Arith using Castor, which
has a one-to-one correspondence with the code shown in Section 4.2.7.
Let us start with the root component Term:@family trait Term {@adt trait Tm@default(Tm) trait Eval1 {type OTm = Tm
def tm = _ => throw NoRuleApplies}}

Several Castor’s annotations are employed: @family denotes a Castor’s component; @adt
denotes a datatype; @default(Tm) denotes a default visitor on Tm. Compared to the Term
67

Chapter 4. Castor: Programming with Extensible Generative Visitors

deﬁnition shown in Section 4.2.7, the deﬁnition here is much simpliﬁed. The accept
declaration, the type member TmV, the visitor interface TmVisit and the default visitor
TmDefault are all generated by analyzing the @adt deﬁnition of Tm. Similarly, Castor adds

the extends clause, the self type annotation and the corresponding val declaration for
Eval1 by the annotation @default(Tm).

Deﬁning Nat is also much simpliﬁed:@family trait Nat extends Term {@adt trait Tm extends super.Tm {case object TmZerocase class TmSucc(t: Tm)case class TmPred(t: Tm)}
def nv(t: Tm): Boolean = t match {case TmZero => truecase TmSucc(t1) => nv(t1)case _ => false}@default(Tm) trait Eval1 extends super.Eval1 {override def tmSucc = x => TmSucc(this(x.t))
override def tmPred = {case TmPred(TmZero) => TmZerocase TmPred(TmSucc(t)) if nv(t) => tcase TmPred(t) => TmPred(this(t))}}}

Variants of Tm are declared inside Tm. Castor will pull them outside of Tm and automaticallycomplement the extends clause and the accept method deﬁnition. Since new variants of
Tm are introduced, Castor will add the extended TmVisit, TmDefault and reﬁned TmV to Nat.

Similarly, Bool can be deﬁned as follows:@family trait Bool extends Term {@adt trait Tm extends super.Tm {case object TmTruecase object TmFalse
case class TmIf(t1: Tm, t2: Tm, t3: Tm)}@default(Tm) trait Eval1 extends super.Eval1 {override def tmIf = {case TmIf(TmTrue,t2,_) => t2
case TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(this(t1),t2,t3)}}}

The code below ﬁnishes the Arith implementation:@family trait Arith extends Nat with Bool {@adt trait Tm extends super[Nat].Tm with super[Bool].Tm {case class TmIsZero(t: Tm)}@visit(Tm) trait Eval1 extends super[Nat].Eval1

684.2 Open Pattern Matchingwith super[Bool].Eval1 {def tmIsZero = {case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t)) if nv(t) => TmFalse
case TmIsZero(t) => TmIsZero(this(t))}}}

Since the TmIsZero is an interesting case for Eval1, @visit annotation is used, which
denotes an ordinary visitor. Thus, Eval1 extends TmVisit after transformation.
Client code

A @family trait can be directly imported in client code since Castor auto-

matically generates a companion object for it:
import Arith._val tm= TmIsZero(TmIf(TmFalse,TmTrue,TmPred(TmSucc(TmZero))))val tm1 = eval1(tm)// iszero (pred (succ 0))val tm2 = eval1(tm1) // iszero 0
val tm3 = eval1(tm2) // 0

which is identical to the client code for Scala extensible visitors shown in Section 4.2.7.
Discussion of the approach

We discuss how Castor addresses the four properties:• Conciseness. By employing Scala’s concise syntax and metaprogramming, Castor greatly simpliﬁes the deﬁnition and usage of visitors. In particular, the needfor auxiliary visitors in performing deep case analysis is now replaced by patternmatching via case clauses. The concept of visitors is even made transparent to theend-user, making the framework more user-friendly.
• Exhaustiveness. The exhaustiveness of patterns in Castor consists of two parts.
The exhaustiveness of visit methods is checked by the Scala compiler when generating companion objects. For nested patterns using case clauses, a default must be
provided. However, this default is neither statically enforced by Scala nor Castor.
Note, however, that with specialized language support it is possible to enforce thatnested patterns always provide a default. This is precisely what EADDs [Zenger andOdersky, 2001] do.
• Extensibility. As illustrated by Nat, Bool and Arith, we can extend the datatypewith new variants and operations, modularly. Such extensibility is enabled by theunderlying extensible visitor encoding.
• Composability. Castor obtains composability via Scala’s mixin composition, as
illustrated by Arith. Unlike partial functions, which silently compose overlapped
patterns, composing overlapped patterns in Castor will trigger compilation errors
because they are conﬂicting methods from diﬀerent traits. The error message will
indicate the source of conﬂicts and we are free to select an implementation in resolving the conﬂict. The composition order does not matter as well.
69

Chapter 4. Castor: Programming with Extensible Generative Visitors

Table 4.1: Pattern matching support comparison:

ConcisenessConventional visitors
Sealed case classes
Open case classes
Partial functions
Extensible visitors
EVF
Castor*Exhaustiveness= good, G#= neutral, #= bad.

Extensibility

Composability

#
#

#
#
#
G
#

#

#
G
#

#
#
#
G
#
G
#*

Castor only gets half score on exhaustiveness because for nested case analysis Scalacannot enforce a default. In a language-based approach nested case analysis should
always require a default, thus fully supporting exhaustiveness.

Table 4.1 summarizes the evaluation on pattern matching approaches abovementioned
in terms of conciseness, exhaustiveness, extensibility, and composability.

Castor iscompared favorably in terms of the four properties among the approaches.

4.3

Hierarchical Datatypes

Traditional functional style datatypes are ﬂat: variants have no relationships among
each other. In contrast, object-oriented style datatypes (i.e. data structures modeled asclass hierarchies) can be hierarchical: a variant can extend intermediate datatypes and/or
an existing variant. In other words, while OO style class hierarchies can be arbitrarily
deep, typical functional datatypes would correspond to a hierarchy where the depth isalways one.
Hierarchical datatypes facilitate reuse. The subtyping relation allows the semantics
deﬁned for supertypes to be reused in subtypes. Castor supports both styles of datatypes.
In this section, we illustrate Castor’s support for hierarchical datatypes by revising theArith language. Another form of hierarchical datatypes will be shown in Section 4.5,
where a new variant is introduced by reﬁning an existing variant. Moreover, the case study
on UML Activity Diagrams Section 4.8 further illustrates the application of hierarchicaldatatypes.

4.3.1 Flat Datatypes versus Hierarchical Datatypes
Terms of the Arith language shown in Section 4.2 are represented as a ﬂat datatype,where all the variants directly extend the root datatype Tm. In fact, terms can be organized
in a hierarchical manner according to their types and arities. Figure 4.2 visualizes thehierarchical representation of terms and the following code materializes it using Castor:@adt trait Tm {trait TmNullary
trait TmUnary { val t: Tm }trait TmTernary { val t1, t2, t3: Tm }trait TmNat extends TmNullary
trait TmBool extends TmNullary704.3 Hierarchical Datatypes

Tm
TmNullary
TmNat

TmTernary

TmBoolTmZero TmTrue

TmIf

TmFalseTmUnary

TmNat2Nat

TmNat2Bool

TmSucc TmPred

TmIsZeroFigure 4.2: Hierarchical representation of Arith terms.

trait TmNat2Nat extends TmUnary
trait TmNat2Bool extends TmUnarycase object TmZero extends TmNat
case class TmSucc(t: Tm) extends TmNat2Nat
case class TmPred(t: Tm) extends TmNat2Nat
case object TmTrue extends TmBool
case object TmFalse extends TmBool
case class TmIf(t1: Tm, t2: Tm, t3: Tm) extends TmTernary
case class TmIsZero(t: Tm) extends TmNat2Bool}The hierarchy becomes multi-layered, where several intermediate datatypes are introduced and case classes/objects do not directly extend the root but an intermediate
datatype. Traits in the second layer (TmNullary, TmUnary and TmTernary) classify terms according to their arities. Based on arities, traits in the third layer (TmNat, TmBool, TmNat2Nat,
TmNat2Bool) further classify terms according to their types. Concrete case classes/objects

are in the fourth layer that extend a corresponding intermediate datatypes. For example,
both TmSucc and TmPred extend TmNat2Nat.

4.3.2 Explicit Delegations
Now we illustrate the advantages of hierarchical datatypes. Suppose we would like todeﬁne a printer for Arith that prints out a term using an S-expression like format. For
example, TmIsZero(TmIf(TmFalse,TmTrue,TmPred(TmSucc(TmZero)))) is printed as "(iszero(if false true (pred (succ 0))))". With terms being classiﬁed according to their arities,

the printer can be modularized:@visit(Tm) trait Print {type OTm = String
def tmUnary(x: TmUnary, op: String) = "(" + op + " " + this(x.t) + ")"def tmSucc = tmUnary(_,"succ")
def tmPred = tmUnary(_,"pred")def tmIsZero = tmUnary(_,"iszero")def tmZero = "0"
def tmTrue = "true"
def tmFalse = "false"
def tmIf = x =>71

Chapter 4. Castor: Programming with Extensible Generative Visitors"(if " + this(x.t1) + " " + this(x.t2) + " " + this(x.t3) + ")"}Since all unary terms (TmSucc, TmPred and TmIsZero) are printed in the same way except forthe operator, we deﬁne an auxiliary method tmUnary. Taking a TmUnary instance and an
operator string as arguments, tmUnary puts the parentheses around the operator and theprinted inner term of TmUnary. Then, tmSucc, tmPred and tmIsZero are implemented just by
calling tmUnary with their respective instances and operator strings.

4.3.3 Default Visitors
The previous example has shown how to enhance the modularity through explicit
delegations. When subtypes share the same behavior with their supertypes, the explicit
delegations can be eliminated with the help of the generated default visitor. Currently, theArith language presented allows ill-typed terms such as TmPred(TmTrue) to be constructed.
To rule out these ill-typed terms, typechecking is needed. Some of the terms share typing
rules: TmTrue and TmFalse; TmSucc and TmPred. With Castor’s default visitor, we can avoid
duplication of typing rules:@adt trait Ty {case object TyNat
case object TyBool}@default(Tm) trait Typeof {type OTm = Option[Ty]override def tmBool = _ => Some(TyBool)
override def tmNat = _ => Some(TyNat)
override def tmNat2Nat = x => this(x.t) match {case Some(TyNat) => Some(TyNat)case _ => None}override def tmNat2Bool = x => this(x.t) match {case Some(TyNat) => Some(TyBool)case _ => None}override def tmIf = x => (this(x.t1),this(x.t2),this(x.t3)) match {case (Some(TyBool),ty1,ty2) if ty1 == ty2 => this(x.t2)
case _ => None}
def tm = _ => None}

Like Tm, Ty is a datatype for representing types, where TyNat and TyBool are two concrete
types. A visitor Typeof is deﬁned for typechecking terms. The output type of Typeof is
Option[Ty], indicating that if a term is well-typed, some type will be returned; otherwise aNone will be returned. Except for TmIf, typing rules are deﬁned on intermediate datatypes.

For example, tmNat2Nat is overridden, which checks whether its inner term is of type TyNat
and returns TyNat if so. tmSucc and tmPred are implicitly implemented by the inheriteddefault visitor, whose deﬁnition is given below:
trait TmDefault extends TmVisit { _: TmV =>72

4.4 GADTs and Well-Typed EDSLs

def tm: Tm => OTm
def tmNullary = (x: TmNullary) => tm(x)def tmUnary = (x: TmUnary) => tm(x)def tmTernary = (x: TmTernary) => tm(x)def tmNat = (x: TmNat) => tmNullary(x)def tmBool = (x: TmBool) => tmNullary(x)def tmNat2Nat = (x: TmNat2Nat) => tmUnary(x)def tmNat2Bool = (x: TmNat2Bool) => tmUnary(x)def tmZero = tmNat(TmZero)def tmSucc = tmNat2Nat(_)
def tmPred = tmNat2Nat(_)
def tmTrue = tmBool(TmTrue)def tmFalse = tmBool(TmFalse)
def tmIf = tmTernary(_)
def tmIsZero = tmNat2Bool(_)}

We can see that the default visitor extends the visitor interface with visit methods forintermediate datatypes and each visit method is implemented by delegating to its direct
parent’s visit method.

4.4 GADTs and Well-Typed EDSLs
In this section, we show the support for generalized algebraic data types (GADTs) [Xiet al., 2003] in Castor. GADTs allow not only datatypes to be parameterized but also
well-formedness constraints to be expressed in constructors. GADTs are widely used forbuilding well-typed domain-speciﬁc languages (EDSLs), which exploit the type system of
the host language to typecheck the terms of the EDSL. Popular approaches to EDSLs
like Finally Tagless [Carette et al., 2009] can provide an encoding of GADTs and provide
modularity as well. However, the encoding employed by Finally Tagless is based on
Church encodings. Unfortunately, this makes it hard to model several operations that
require nested patterns or operations with dependencies. The interested reader is referred
to Section 2 and 3 of the EVF paper [Zhang and Oliveira, 2017] for a detailed discussion
on the issue of Church encodings. We show that just as Finally Tagless encodings,
modularity is supported, and like GADTs nested pattern matching and dependencies areeasy to do as well.

4.4.1 GADTs and Well-Typed Terms
We have shown how to rule out ill-typed terms using a type-checking algorithm in
Section 4.3.3. A better solution, however, is to prevent such terms from being constructedin the ﬁrst place. This is possible through representing Arith terms using a GADT-style:@family trait GArith {@adt trait Tm[A] {case object TmZero extends Tm[Int]case class TmSucc(t: Tm[Int]) extends Tm[Int]case class TmPred(t: Tm[Int]) extends Tm[Int]case object TmTrue extends Tm[Boolean]73

Chapter 4. Castor: Programming with Extensible Generative Visitors

case object TmFalse extends Tm[Boolean]case class TmIf[A](t1: Tm[Boolean], t2: Tm[A], t3: Tm[A])extends Tm[A]case class TmIsZero(t: Tm[Int]) extends Tm[Boolean]}}
Tm is now parameterized by a type parameter A. When declaring variants of Tm, the extends

clause cannot be omitted anymore since Castor does not know how to instantiate A.
Notice that A is instantiated diﬀerently as Int or Boolean for expressing well-formedness
constraints. For example, TmIsZero requires its subterm t of type Tm[Int]. Consequently,
one cannot supply a term of type Tm[Boolean] constructed from TmTrue, TmFalse or TmIsZero
to TmIsZero. Therefore, ill-formed terms are statically rejected by the Scala type system:
TmIsZero(TmZero) // Accepted!
TmIsZero(TmTrue) // Rejected!

4.4.2 Well-Typed Big-Step Evaluator
As opposed to small-step semantics, big-step semantics immediately evaluates a valid
term to a value. In the case of Arith, a term can either be evaluated to an integer or aboolean value. Without GADTs, implementing a big-step evaluator for Arith is tedious:@family @adts(Tm) @ops(Eval1) trait EvalArith extends Arith {@adt trait Value {case class IntValue(v: Int)case class BoolValue(v: Boolean)}@visit(Tm) trait Eval {type OTm = Valuedef tmZero = IntValue(0)def tmSucc = x => this(x.t) match {case IntValue(n) => IntValue(n+1)
case _ => throw NoRuleApplies}
def tmPred = x => this(x.t) match {case IntValue(n) => IntValue(n-1)
case _ => throw NoRuleApplies}
def tmTrue = BoolValue(true)
def tmFalse = BoolValue(false)
def tmIf = x => this(x.t1) match {case BoolValue(true) => this(x.t2)
case BoolValue(false) => this(x.t3)
case _ => throw NoRuleApplies}
def tmIsZero = x => this(x.t) match {case IntValue(0) => BoolValue(true)case IntValue(_) => BoolValue(false)
case _ => throw NoRuleApplies
}
}
}

74

4.4 GADTs and Well-Typed EDSLs

EvalArith illustrates the operation extensibility of Castor, which does not introduce any

new variants of Tm but a new visitor Eval on Tm. Auxiliary annotations @adts and @ops
provide inherited datatypes and operations for Castor to generate the companion object.
Such an implementation suﬀers from the so-called tag problem [Carette et al., 2009]:
to accommodate diﬀerent evaluation result types, an open datatype Value is deﬁned foraccommodating integers, booleans and many other evaluation result types that might be
added in the future. The two variants IntValue and BoolValue are introduced for wrapping
integers and boolean values, respectively. Pattern matching is used for unwrapping theevaluation results from inner terms. A defensive wildcard is needed for dealing with
ill-typed terms. We can see that the tagging overhead is high.
Fortunately, we can avoid the tag problem with the help of Castor’s GADTs. The
extensible visitor encoding for GADTs is slightly diﬀerent from the one presented in Section 4.2.7, which additionally take the type information carried by terms into account.
For instance, the visitor interface generated for Tm[A] is listed below:
trait TmVisit { _: TmV =>type OTm[A]def apply[A](x: Tm[A]) = x.accept(this)def tmZero: OTm[Int]def tmSucc: TmSucc => OTm[Int]def tmPred: TmPred => OTm[Int]def tmTrue: OTm[Boolean]
def tmFalse: OTm[Boolean]
def tmIf[A]: TmIf[A] => OTm[A]def tmIsZero: TmIsZero => OTm[Boolean]}

Each visit method now returns a value of a higher-kinded type OTm[A], where A is instantiated consistently with how it is instantiated in the extends clause. For example, tmZero isof type OTm[Int] while tmTrue is of type OTm[Boolean]. Then, a well-typed big-step evaluatorcan be made tagless:@family @adts(Tm) trait EvalGArith extends GArith {@visit(Tm) trait Eval {type OTm[A] = A
def tmZero = 0
def tmSucc = x => this(x.t) + 1
def tmPred = x => this(x.t) - 1
def tmTrue = true
def tmFalse = false
def tmIf[A] = x => if (this(x.t1)) this(x.t2) else this(x.t3)
def tmIsZero = x => this(x.t) == 0}}With the output type speciﬁed as A, the visit method returns a value of the type carriedby the term. For example, visit methods tmZero and tmTrue return Int and Boolean values
respectively. Moreover, this Eval implementation remains retroactive when terms of new
types (such as Tm[Float]) are introduced.
Here are some terms that have diﬀerent evaluation result types.
75

Chapter 4. Castor: Programming with Extensible Generative Visitors

import EvalGArith._eval(TmSucc(TmZero))// 1eval(TmIsZero(TmZero)) // true

4.4.3 Well-Typed Small-Step Evaluator
Well-typed big-step evaluators can be deﬁned with Finally Tagless in an equally simple
manner. What distinguishes Castor from Finally Tagless is the ability to deﬁne small-stepevaluators in an easy way. The need for deep patterns and the dependency on a numericvalue checker causes immediate trouble for Finally Tagless. Although workarounds may
be possible for some of the issues, they are cumbersome and require signiﬁcant amountsof boilerplate code [Kiselyov, 2012]. In contrast, encoding small-step semantics in a
GADT-style with Castor is unproblematic:@family @adts(Tm) trait Eval1Arith extends GArith {def nv[A](t: Tm[A]): Boolean = t match {case TmZero => truecase TmSucc(t1) => nv(t1)case _ => false}@default(Tm) trait Eval1 {type OTm[A] = Tm[A]def tm[A] = x => throw NoRuleApplies
override def tmIf[A] = {case TmIf(TmTrue,t2,_) => t2
case TmIf(TmFalse,_,t3) => t3
case TmIf(t1,t2,t3) => TmIf(this(t1),t2,t3)}
override def tmIsZero = {case TmIsZero(TmZero) => TmTruecase TmIsZero(TmSucc(t)) if nv(t) => TmFalse
case TmIsZero(t) => TmIsZero(this(t))}... // Other cases are the same as before}}

The instantiation of the output type guarantees that the small-step evaluator is typepreserving. That is, the type carried by a term remains the same after one step of evaluation. For example, calling eval1 on TmZero will never return TmTrue no matter how Eval1 is
implemented. The actual deﬁnition of Eval1 is almost the same as before except that nv,
tm and tmIf become generic. Still, the ability to do nested pattern matching and to call nv

in Eval1 is preserved.

4.4.4 Extension: Higher-Order Abstract Syntax for Name Binding
A recurring problem in designing EDSLs is how to deal with binders. For example, inlambda calculus, operations involved with names like α-equivalence and capture-avoiding
substitution are non-trivial to deﬁne. Higher-order abstract syntax (HOAS) [Pfenning andElliott, 1988] avoids these problems through reusing the binding mechanisms provided
76

4.5 Graphs and Imperative Visitors

by the host language. The following code shows how to extend Arith with simply-typed
lambda calculus modularly:@family trait HOAS extends EvalGArith {@adt trait Tm[A] extends super.Tm[A] {case class TmVar[A](v: A) extends Tm[A]case class TmAbs[A,B](f: Tm[A] => Tm[B]) extends Tm[A => B]
case class TmApp[A,B](t1: Tm[A => B], t2: Tm[A]) extends Tm[B]}@visit(Tm) trait Eval extends super.Eval {def tmVar[A] = _.v
def tmAbs[A,B] = x => y => this(x.f(TmVar(y)))
def tmApp[A,B] = x => this(x.t1)(this(x.t2))}}

Three new forms of terms are introduced: lifters (TmVar), lambda abstractions (TmAbs) andapplications (TmApp). Of particular interest is TmAbs, which constructs a term of type Tm[A=> B] from a Scala lambda function Tm[A] => Tm[B] and thus is higher-order.

Correspondingly, Eval is extended with three new visit method implementations. tmVar
simply extracts the value out of the lifter. tmAbs is trickier since it returns a value of typeA => B. A lambda function is hence created, which takes y of type A and lifts it into Tm[A]using TmVar, then applies x.f to the lifted term for computing a Tm[B] and ﬁnally does arecursive call to evaluate Tm[B] into B. tmApp recursively evaluates t1 and t2, which returns
the value A => B and A respectively. Then it applies A => B to A for getting a value of B.
Here is an example that illustrates the use of HOAS:
import HOAS._
eval(TmApp(TmAbs((t: Tm[Int]) => TmSucc(TmSucc(t))), TmZero)) // 2

We ﬁrst create an abstraction term that applies successor twice to the argument t andthen apply it to constant zero. Note that the type of t is explicitly speciﬁed because Scala’s
type system is not powerful enough to infer the type of TmAbs without the type annotation.

4.5 Graphs and Imperative Visitors
Examples presented so far are all functional visitors (i.e. computation is done via
returning values) on immutable trees. In fact, Castor also supports imperative visitors(i.e. computation is done via side eﬀects) and the data structure can be a mutable graph.
Imperative computation is, in some cases, more eﬃcient than the functional counterpart
regarding time and memory. Compared to trees, graphs are a more general data structurethat have many important applications. For instance, in the domain of compilers, abstract semantic graphs can be used for representing shared subexpressions, facilitating
optimizations like common subexpression elimination. In this section, we show how tomodel graphs and imperative visitors with Castor.
77

Chapter 4. Castor: Programming with Extensible Generative Visitors

Statestates *transitions

name: String

Machine*targetTransition
event: String

Figure 4.3: Class diagram of FSM.

close

OpenedClosedopen
unlocklock

LockedFigure 4.4: A state machine for controlling a door.

4.5.1 The Diﬃculties in Modeling Graphs
Modeling graphs modularly is non-trivial in approaches such as Object Algebras.
Consider modeling a Finite State Machine (FSM) language. Figure 4.3 shows a UML class
diagram for the FSM language. A Machine consists of some States. Each State has a nameand a number of Transitions. A Transition is triggered by an event, taking one State
to another. Concretely, Figure 4.4 shows a simple state machine for controlling a door,
which has three states (opened, closed and locked) and four transitions (close, open,
unlock and lock). From Figure 4.4 we can see that this state machine forms a graph,
where we can go back and forth from one state to another along with the transitions.
A failed attempt with Object Algebras

Let us try to model the FSM language with

Object Algebras [Oliveira and Cook, 2012]. Describing the FSM language using an Object
Algebra interface is unproblematic:
trait FSM[M,S,T] {def machine(states: List[S]): M
def state(name: String, trans: List[T]): S
def trans(event: String, target: S): T}

where M, S, T and their variants are captured as type parameters and factory methods
respectively. However, constructing a graph using this representation is hard because
Object Algebras support only immutable tree structures that are built bottom up. Here isa failed attempt on modeling the door state machine:// Forward reference error!
def door[M,S,T](f: FSM[M,S,T]) = {78

4.5 Graphs and Imperative Visitors

val close: T = f.trans("close",closed)
val open: T = f.trans("open",opened)
val lock: T = f.trans("lock",locked)
val unlock: T = f.trans("unlock",closed)
val opened: S = f.state("opened", List(close))val closed: S = f.state("closed", List(open,lock))val locked: S = f.state("locked", List(unlock))f.machine(List(opened,closed,locked))}

A forward reference error will always occur no matter how we arrange these statements.
The reason is that there is no proper way to decouple the cyclic references between states
and transitions.

4.5.2 FSM in Castor
Fortunately, modeling the FSM language using Castor is not a problem:@family trait FSM {@adt trait M {val states = ListBuffer[S]()class Machine}@adt trait S {val trans = ListBuffer[T]()var name: Stringclass State(var name: String)}@adt trait T {class Trans(val event: String, var target: S)}@visit(M,S,T) trait Print {type OM = String
type OS = OM
type OT = OM
def machine = _.states.map{this(_)}.mkString("\n")def state = s => s.trans.map{this(_)}.mkString(s.name+":\n","\n","")
def trans = t => t.event + " -> " + t.target.name}@visit(M,S,T) trait Step {type OM = String => Unit
type OS = OM
type OT = OM
var res: S = nulldef machine = m => event => m.states.foreach{this(_)(event)}
def state = s => event => s.trans.foreach{this(_)(event)}
def trans = t => event => if (event == t.event) res = t.target}}

The actual class hierarchies of the FSM language are slightly diﬀerent from what Figure 4.3 shows. Each class in the UML diagram is deﬁned inside an @adt trait for allowing
potential variant extensions. Fields are either declared as var or val for enabling/disabling
mutability.
79Chapter 4. Castor: Programming with Extensible Generative Visitors

Combined visitors

There are two visitors deﬁned for the FSM language, namely Print

and Step. Both of them are combined visitors that apply to transitions, states, and machines. Such a combined implementation is much more compact than deﬁning threemutually dependent visitors with distinct names. Annotated as @visit(M,S,T), Print instantiates the output types OM, OS, OT consistently as String and implements three visit
methods machine, state and trans altogether. Concretely, methods machine and state map
Print to the substructures and concatenate the results with a newline. For trans, weshould not call this on the target state otherwise it will not terminate. Instead, we printout the name ﬁeld on the target state only.
Imperative visitors

The Step visitor captures the small-step execution semantics of

FSM. Given an event, it goes through the structure for ﬁnding out the transition triggeredby that event and returning the state that transition points to. Note that Step is, at thesame time, an imperative visitor. Step instantiates the output types as String => Unit andupdates the ﬁeld res to the found target transition. If res is still null after traversal, thenno such transition exists.
Now we are able to model the state machine that controls doors like this:import FSM._
val door = new Machine
val opened = new State("Opened")
val closed = new State("Closed")
val locked = new State("Locked")val open = new Trans("open",opened)val close = new Trans("close",closed)
val lock = new Trans("lock",locked)
val unlock = new Trans("unlock",closed)
door.states += (opened,closed,locked)
opened.trans += close
closed.trans += (open,lock)locked.trans += unlock

The graph is constructed in a conventional OOP style. Unlike Object Algebras, the structure is built top down. To decouple cyclic references, the declaration and initializationof the variables are separated. This is possible in Castor because variants are concreteclasses provided with setters whereas in Object Algebras they are abstract types without
concrete representations.
Calling print(door) produces the following output:
Opened:close -> Closed
Closed:open -> Opened
lock -> Locked
Locked:unlock -> Closed

Some tests on Step are:80

4.5 Graphs and Imperative Visitors

step(door)("open")
println(step.res.name) // "Opened"
step.res = null// Reset to nullstep(door)("close")println(step.res.name) // "Closed"

Imperative visitors should be used more carefully. In the case of Step, its ﬁeld res needs
to be reset to null afterwards. Otherwise, the result may be wrong next time we call step.

4.5.3 Language Composition and Memoized Traversals
Consider unifying FSM and Arith.

The uniﬁcation happens when a new kind oftransition called guarded transitions is introduced. A guarded transition additionallycontains a boolean term and is triggered not only by the event but also by the evaluation
result of that term. Combining FSM with the GADT version of Arith is given below:@family @adts(Tm,F,S) @ops(Eval)
trait GuardedFSM extends FSM with EvalArith {@adt trait T extends super[FSM].T {class GuardedTrans(event: String, target: State, val tm: Tm[Boolean])extends Trans(event, target)}@visit(M,S,T) trait Print extends super[FSM].Print {def guardedTrans = t => trans(t) + " when " + t.tm.toString}@visit(F,S,T) trait Step extends super[FSM].Step {def guardedTrans = t => event => if (eval(t.tm)) trans(t)(event)}@visit(S,T) trait Reachable {type OS = Unit
type OT = Unitval reached = collection.mutable.Set[S]()def state = s =>if (!reached.contains(s)) {reached += ss.trans.foreach(this(_))}
def trans = t => this(t.target)
def guardedTrans = t => if (eval(t.tm)) this(t.target)}}

Class GuardedTrans extends Trans with an additional ﬁeld tm of type Tm[Boolean]. To handle
GuardedTrans, Print and Step are extended with an implementation of guardedTrans method.

Having GuardedTrans as a subtype of Trans, we are able to partially reuse the semantics ofTrans for GuardedTrans via passing t to the inherited trans method.

Memoized traversals

Naively traversing a graph might be ineﬃcient because the same

object may be traversed multiple times. In the worst case, the traversal may not eventerminate if not dealt with carefully. A better approach is to memoize the results oftraversed objects and fetch the cached result when an object is traversed again. Reachable
81Chapter 4. Castor: Programming with Extensible Generative Visitors

Fam ::=@family @adts(D ) @ops(V ) trait F extends F {Adt Vis}

Adt

::=

@adt trait D [X ] extends super[F ].D [X ] {Ctr }

Ctr

::=

class C [X ] extends C [T ] with ? D [T ]

|
Vis
T

(

(

)

)

object C extends C [T ] with ? D [T ]| trait D [X ] extends D [T ]()::= @ default | visit (D ) trait V extends super[F ].V::= X | D [T ] | Int | T =>T
Figure 4.5: Castor syntax.

is a combined imperative visitor that ﬁnds out all reachable states for the given state. The
reachable states are collected in a reached ﬁeld, which is initialized as an empty mutableset. Reachable employs memoized depth-ﬁrst search, which ﬁrst checks whether the state
has already been traversed. If not, the state is added to reached and the recursion goesto the states its transitions lead to. Similarly, memoization can be applied to functional
visitors by changing reached to a mutable map.
We can build a guarded door controller by changing the import statement and how
lock is initialized:
val lock = new GuardedTrans("lock",locked,TmFalse)

Now, an opened door can no longer be locked because the guard evaluates to false:
reachable(open)
println(reachable.reached.size) // 2

By setting the expression to TmTrue, the door can be locked again:
lock.tm = TmTruereachable.clear // Reset to empty
reachable(open)
println(reachable.reached.size) // 3

4.6

Formalized Code Generation

In previous sections, we have shown code written with Castor and its corresponding
generated code. In this section, we formally describe the valid Scala programs acceptedby Castor and the transformation scheme.

4.6.1 Syntax
Figure 5.1 describes valid Scala programs accepted by Castor. Uppercase metavariables range over capitalized names. A is written as a shorthand for a potentiallyempty sequence A1 • . . . • An , where • denotes with, comma or semicolon depending on
the context.

(). . . ? denotes that . . . is optional. For brevity, we ignore the syntax thatis irrelevant to the transformation, such as the case modiﬁer, constructors, ﬁelds, andmethods. These parts are kept unchanged after transformation.
82

4.7 Case Study I: Types and Programming Languages

4.6.2 Transformation
Figure 4.6 formalizes the transformation. We use semantic brackets (J·K) in deﬁning
the transformation rules and angle brackets (<>) for processing sequences. The transformation is given by pattern matching on the concrete syntax and is quite straightforward.
One can see that processing the Arith implementation in Castor (cf. Section 5.2) throughFigure 4.6 will get back the extensible visitor implementation (cf. Section 4.2.7).
Here we brieﬂy discuss some interesting cases. A trait is recognized as a base case
if it extends nothing. Base cases have extra declarations such as accept declaration for
datatypes or val declaration for visitors. Variants declared using class, trait or object
are treated diﬀerently. objects and classes have their corresponding visit methods in thevisitor interface while visit methods for traits only exist in the default visitor. The extends
clause for @adt is used in inferring the extends clause for concrete visitors.

4.6.3 Implementation
Castor employs Scalameta [Burmako, 2017] (version 1.8.0), a modern Scala metaprogramming library, for analyzing and generating the code. The actual implementation
closely follows the formalization. After parsing, the Scala source program is represented
as an AST. We ﬁrst check the validity of that AST with errors like annotating @adt not on a
trait reported. We then generate code by analyzing the AST. Next, we build the AST withcode injected. Finally, the AST is typechecked by the Scala compiler. During the process,
Scala’s quasiquotes are used, which allow us to analyze and rebuild the AST conveniently
via the concrete syntax.

4.7 Case Study I: Types and Programming Languages
In this section, we present a case study on modularizing the interpreters in TAPL [Pierce,2002]. The Arith language and its variations are directly from or greatly inspired by the
TAPL case study. TAPL are a good benchmark for examining Castor’s capabilities of open
pattern matching and modular dependencies. The reason is that core data structures of
TAPL interpreters, types and terms, are modeled using algebraic datatypes; operations
over types and terms are deﬁned via pattern matching. There are a few operations that require nested patterns: small-step semantics, type equality, and subtyping relations. They
all come with a default. The data structures and associated operations should be modular
as new language features are introduced and combined. However, without proper support
for modular pattern matching, the original implementation duplicates code for featuresthat could be shared. With Castor and techniques shown in Section 5.2, we are able torefactor the non-modular implementation into a modular manner. Our evaluation showsthat the refactored version signiﬁcantly reduces the SLOC compared to a non-modular
implementation found online. However, at the moment, improved modularity does comeat some performance penalty.
83

Chapter 4. Castor: Programming with Extensible Generative Visitors

J@family @adts(D ) @ops(V ) trait F extends F {Adt Vis}K =trait F extends F {JAdt K JVisK}
object F extends F {htype D V = D Visit | D ∈ D ∪ Adt ihobject v extends V | V ∈ V ∪ Visi}

J@adt trait D [X ] {Ctr }K =type D V <:D Visit
trait D [X ] {def accept(v:D V): v.OD [X ]}JCtr K
trait D Visit { _:D V =>type OD [X ]def apply[X ](x:D [X ]) = x.accept(this)JCtr Kvisit}
trait D Default extends D Visit { _:D V =>def d [X ]:D [X ] => OD [X ]JCtr Kdefault}

J@adt trait D extends super[F ].D {Ctr }K =type D V <:D Visit

JCtr K

trait D Visit extends super[F ].D Visit { _:D V =>JCtr Kvisit }trait D Default extends D Visit with super[F ].D Default {_:D V => JCtr Kdefault }Jclass C[X ] . . .K = class C[X ] . . . {override def accept(v:D V) = v.c (this)}
Jobject C . . .K = object C . . . {override def accept(v:D V) = v.c }JCtr K = Ctr()
Jclass C[X ] extends . . . with ? D [T ]Kvisit = def c [X ]:C => OD [T ]()
Jobject C extends . . . with ? D [T ]Kvisit = def c : OD [T ]JCtr Kvisit = ∅Jclass C1 [X ] extends C2 [T ] . . .Kdefault = def c1 [X ]= x => c2 (x)Jobject C1 extends C2 [T ] . . .Kdefault = def c1 = c2 (C1 )Jtrait D1 [X ] extends D2 [T ] . . .Kdefault = def d1 = (x:D1 [X ]) => d2 (x)()
J@ default | visit (D ) trait V K =()
trait V extends D Default | Visit { _:D V=> . . . }val v : V()
J@ default | visit (D ) trait V extends super[F ].V K =()
trait V extends D Default | Visit with super[F ].V { _:D V=> . . . }JX K = hJX K | X ∈ X i
Figure 4.6: Castor transformation.

84

4.7 Case Study I: Types and Programming Languages

LEGEND

FullSub

FullSimple

Original language

MoreExt

Extracted feature
Dependency

TyArithArith

FullError

RcdSubBot

Extension

FullUntyped
TyNat

TyBool

Nat

Bool

Bot

Variant

SimpleBool

TyRcd

TyStr

TyLet

Top

RecordStr

Untyped

Let

Typed

Type

VarAppTerm

Figure 4.7: Simpliﬁed language/feature dependency graph.

4.7.1 Overview
An existing Scala implementation of TAPL2 strictly follows the original OCaml version,
which uses sealed case classes and pattern matching. The ﬁrst ten languages (arith, untyped, fulluntyped, tyarith, simplebool, fullsimple, fullerror, bot, rcdsubbot and fullsub) are
our candidates for refactoring. Each language implementation consists of 4 ﬁles: parser,
syntax, core and demo. These languages cover various features including arithmetic,lambda calculus, records, ﬁxpoints, error handling, subtyping, etc. Features are shared
among these ten languages. However, such featuring sharing is achieved via duplicating
code, causing problems like:• Inconsistent deﬁnitions. Lambdas are printed as "lambda" in all languages except
for untyped, where lambdas are printed as "\".
• Feature leaks. Features introduced in the latter part of the book (e.g., System F)leak to previous language implementations such as fullsimple.
Our refactoring focuses on syntax and core where datatypes and associated operations are deﬁned. Figure 4.7 gives a simpliﬁed high-level overview of the refactored implementation. The candidate languages are represented as gray boxes whereas extracted
features/sub-languages are represented as white boxes. From Figure 4.7 we can see thatthe interactions between languages (revealed by the arrows) are quite intense. Take Arithfor example, it is a sublanguage for tyarith, fulluntyped, fullerror, fullsimple and fullsub.
Unfortunately, without proper modularization techniques, the original implementation
repeats the deﬁnition of arith at least ﬁve times. In the refactored implementation writtenwith Castor, however, arith is deﬁned only once and modularly reused in other places.

4.7.2 Evaluation
We evaluate Castor by answering the following questions:2

https://github.com/ilya-klyuchnikov/tapl-scala85

Chapter 4. Castor: Programming with Extensible Generative Visitors

Table 4.2: SLOC evaluation of TAPL interpreters
Extracted
boolextensionstr
letmoreext
nat
recordtop
typed
varappvariant
misc

Castor
71
24
42
48
112
85
117
79
82
40
136
212

EVF
98
34
55
47
106
103
198
86
138
65
161
172

Languagearith
untypedfulluntyped
tyarithsimplebool
fullsimple
fullerrorbot
rcdsubbot
fullsub

Castor
31
40
18
22
24
24
68
40
30
57

EVF
33
46
47
26
38
83
105
61
39
116

Scala
106
124
256
157
212
619
396
190
257
618

Total

1402

1857

2935

• Q1. Is Castor eﬀective in reducing SLOC?
• Q2. How does Castor compare to EVF?
• Q3. How much performance penalty does Castor incur?
Q1

Table 4.2 reports the SLOC comparison results. With all the features/sublanguages

extracted, implementing a candidate language with Castor is merely done by composing
features/sublanguages. Therefore, the more features/sublanguages the candidate language uses, the more code Castor reduces. Compared to the non-modular Scala implementation, for a simple language like arith, the reduction rate3 is 71%; for a feature-richlanguage like fullsimple, the reduction rate can be up to 96%. Overall, Castor reduces
over half of the total SLOC with respect to the non-modular version.

Q2

Table 4.2 also compares Castor with EVF [Zhang and Oliveira, 2017]. Castor re-duces over 400 SLOC compared to EVF. As we have shown in Section 4.2, the reduction
comes from the native support for pattern matching, generated dependency declarations,etc. More importantly, the instantiation burden for EVF is heavy if there are a lot ofvisitors and the dependencies are complex. In contrast, Castor completely removes theinstantiation burden by generating companion objects automatically.

Q3

To measure the performance, we randomly generate 10,000 terms for each language

and calculate the average evaluation time for 10 runs. The ScalaMeter4 microbenchmark
framework is used for performance measurements. The benchmark programs are compiled using Scala 2.12.7, JDK version 1.8.0_211 and are executed on a MacBook Pro with
2.3 GHz quad-core Intel Core i5 processor with 8 GB memory. Figure 4.8 compares theexecution time in milliseconds. From the ﬁgure we can see that Castor implementations
3
4

Reduction rate =Scala SLOC − Castor SLOC
Scala SLOC

http://scalameter.github.io

86

× 100%

4.7 Case Study I: Types and Programming Languages

Evaluation time (ms)
arith

62

untypedCastor

83.6
248.2

126.6

fulluntyped
tyarith

65.2

355.3

120.3
97.3

simplebool

570.7

160.6

fullsimple

124.4

bot

133.4

fullerror
rcdsubbot
fullsub

Scala

357
316.2
272.7

89.6

341.2

135.4

488.6

124.8

0

150

300

450

600

Figure 4.8: Performance evaluation of TAPL interpreters.

Evaluation time (ms)Conventional visitor
Sealed case class
Open case classPartial function
Castor

68.9
62
69.9
84.1
83.6
0

22.5

45

67.5

90

Figure 4.9: Performance evaluation of Arith.
have a 1.35x (arith) to 3.92x (fullsub) slowdown with respect to the corresponding nonmodular Scala implementations. The more features a modular implementation combines,
the more signiﬁcant the slowdown is. Figure 4.9 further compares the performance of the
Scala Arith implementations discussed in Section 4.2. Obviously, modular implementations are slower than non-modular implementations. With the underlying optimizations,the implementation based on sealed case classes is faster than the implementation basedon conventional visitors.
We believe that the performance penalty is mainly caused by method dispatching. A
modular implementation typically has a complex inheritance hierarchy. Dispatching ona case needs to go across that hierarchy. Thus, the more complex the hierarchy is, theworse the performance is. Another source of performance penalty might be the use offunctions instead of normal methods in visitors. Of course, more rigorous benchmarks
need to be conducted to verify our guesses. One possible way to boost the performanceis to turn TAPL interpreters into compilers via staging using the LMS framework [Rompf
and Odersky, 2010]. This is currently not possible because LMS and Scalameta areincompatible in terms of the Scala compiler versions.
Threats to validity

There are two major threats to the validity of our evaluation. The

ﬁrst threat is that measuring conciseness by counting SLOC may not be fair especiallywhen diﬀerent languages are used. We mitigate this threat by making the code style and87

Chapter 4. Castor: Programming with Extensible Generative Visitors

the maximum character-per-line consistent for each implementation. The second threat
is the representativeness of the TAPL interpreters. They are small languages for teaching
purposes. It might still be questionable whether Castor scale to model larger languages
that are actually used in practice. Nevertheless, TAPL interpreters have already covered
a lot of core features that are available in mainstream languages.

4.8

Case Study II: UML Activity Diagrams

In Section 4.7, we have evaluated the functional aspects of Castor. In this section, we
evaluate the imperative aspects of Castor. To do so, we conduct another case study on
a subset of the UML activity diagrams, which can be seen as a richer language than the
FSM language discussed in Section 4.5. This case study examines hierarchical datatypes,
imperative visitors and graphs.

4.8.1 Overview
An execution model of UML activity diagrams has been proposed as one of the challenges of the Transformation Tool Contest (TTC’15).

Metamodel

Figure 4.10 shows the metamodel of UML activity diagrams, where Namedenotes abstract classes and Name denotes concrete classes. An Activity object represents an instance of a UML activity diagram, which contains a sequence of ActivityNodesand ActivityEdges. ExecutableNode and ControlNode are two intermediate types of ActivityNodefor classifying nodes that perform actions or control the ﬂow. There are several concrete nodes. InitialNode and ActivityFinalNode are the start/end of activity diagrams;
DecisionNode and MergeNode are the start/end of alternative branches; ForkNode and JoinNode

are the start/end of concurrent branches. On the other hand, OpaqueAction sequentially
executes a sequence of Expressions. ActivityNodes are connected by ActivityEdges. Similar to GuardedTrans discussed in Section 4.5.3, a ControlFlow is a specialized ActivityEdge,
which is guarded by the current BooleanValue stored in a BooleanVariable. Expressions arealso organized in a hierarchical way according to their types (Boolean or Integer) and thenumber of operands (Unary or Binary).

Goal and challenges

The goal is to extend this simpliﬁed metamodel of UML activity

diagrams with the dynamic execution semantics. The semantics is deﬁned by performing transitions on activity nodes step by step using an imperative style. Several runtime
concepts need to be introduced. Adding these runtime concepts poses two modularity challenges: operation extensions and ﬁeld extensions. One example of an operation
extension is execute, which is added to the Expression hierarchy for executing the calculation. One example of a ﬁeld extension is a mutable boolean value running, which is added
to ActivityNode for distinguishing triggered nodes from others.
884.8 Case Study II: UML Activity Diagrams

edges
ControlFlow

ActivityEdge

guard: BooleanVariable

outgoing *** incoming

source 1

1 targetActivity
activity

nodes

ActivityNode

1*

ControlNode

ExecutableNode

InitialNode

ActionForkNode

FinalNode

JoinNode

MergeNode
Value

OpaqueAction

DecisionNode0..1 initialValue

ActivityFinalNode
expressions*

Expression<<enumeration>>
BooleanUnaryOperator
NOT

IntegerValue

BooleanValue

value: Intvalue: Boolean

BooleanExpression
operator

assignee

Variable

BooleanUnaryExpression<<enumeration>>
BooleanBinaryOperator
ANDOR

name: String**
inputsoperand
BooleanVariable

operator
BooleanBinaryExpression<<enumeration>>
IntegerCalculationOperator

operand1

ADD
SUBTRACT

operand2

operand1
IntegerExpression
operator

IntegerVariable
operand2

assignee

IntegerCalculationExpression<<enumeration>>
IntegerComparisonOperator
SMALLERSMALLER_EQUALSEQUALSGREATER_EQUALSGREATER

localsoperator

IntegerComparisonExpression

Figure 4.10: Metamodel of UML Activity Diagrams, an excerpt adapted after the TTC’15
document [Mayerhofer and Wimmer, 2015].

89

Chapter 4. Castor: Programming with Extensible Generative Visitors

Reference implementation

The reference implementation5 is written in Java with

EMF [Steinberg et al., 2008]. The metamodel is described in Ecore from which Java
interfaces are generated. Then semantics is encoded by deﬁning classes that implement
those interfaces using the Interpreter pattern [Gamma et al., 1994]. The reference is
non-modular because the Interpreter pattern facilitates adding new classes but lacks theability to add new operations. Therefore, the reference implementation has to anticipate
the operations on the metamodel. Moreover, consistent with what Figure 4.10 shows, operators were modeled as enumerations and recognized using switch-case clauses in Java,
which are closed for extensions.
Refactored implementation

Our refactoring only focuses on the metamodel and thesemantics parts. Since the original implementation is written in Java, we ﬁrst port it into
Scala and then refactor it using Castor. Figure 4.11 gives an overview of the refactored
implementation, which consists of four Castor components. Concretely, we make thefollowing changes to the ported implementation for increasing modularity:
1. Separate metamodel and operations. With Castor, we do not need to foresee theoperations on the metamodel since operations can be modularly added afterwards.
Thus, the refactored implementation separates the metamodel and operations upon
it respectively in *Model and *Lang.
2. Expression language as an independently reusable component. Values, variables and expressions are essentially a sublanguage independent of the UML activity diagrams. Instead of deﬁning the expression sublanguage together with UML
activity diagrams within a single @family component, we extract its metamodel into
ExpModel and its semantics into ExpLang and let UmlModel and UmlLang extend
them respectively. This allows the expression sublanguage to be reused or extended
individually.
3. Overridden methods as visitors. Methods that are overridden in the subclasses
are rewritten as visitors, such as isReady and fire on ActivityNode and execute on
Expression. Since only a few cases of isReady and fire are overridden whereas every

case of execute is overridden, we use the default visitor (annotated as @default) for
the former and the ordinary visitor (annotated as @visit) for the latter. For nonoverridden methods, we move them out of a class and use an explicit argument tocapture this.
4. Operators as open datatypes. Operators are refactored as @adt hierarchies andtheir semantics are given by visitors for enabling extensions. This allows new kindsof operators such as multiplication to be added later.

4.8.2 Evaluation
We evaluate Castor by answering the following questions:5

https://github.com/moliz/moliz.ttc2015

90

4.8 Case Study II: UML Activity Diagrams

ExpModelExpLang

UmlModelUmlLang
Figure 4.11: Refactored implementation.

• Q1. Does the refactoring preserve the behavior of the ported implementation?
• Q2. Can Castor solve the modularity challenges?
• Q3. How does the refactoring aﬀect the SLOC?
• Q4. Is the performance overhead reasonable?

Q1

To make sure that our refactoring does not aﬀect the correctness of the implemen-tation, we ran the test suite provided by the TTC’15 document. The test suite contains
6 small activity diagrams where all kinds of ActivityNodes and Expressions are covered.
The refactored implementation passes all the tests in the test suite. This gives us someconﬁdence that the refactored implementation preserves the behavior of the ported implementation.

Q2

For the operation extension challenge, the answer is yes. Operations are added by

deﬁning new visitors, which are fully modular. However, Castor does not address theﬁeld extension challenge very well. With the current version of Castor, we cannot extend
existing classes with additional ﬁelds while keeping their names. The workaround is to introduce subclasses of diﬀerent names. For example, if we want to extend ActivityNode with
a ﬁeld called running, we have to deﬁne a new class called RuntimeActivityNode that extends
ActivityNode with running. The drawback is that RuntimeActivityNode and ActivityNode co-exist and all existing operations need to be modiﬁed for handling RuntimeActivityNode.
It is possible to have an alternative design for Castor, which does not introduce a newname while accomplishing ﬁeld extensions in Castor. However, this brings some othercomplications. Such alternative design is discussed in Section 4.9.

Q3

The SLOC of the ported version and the refactored version are 489 and 411 respec-tively. Surprisingly, the refactoring brings extra modularity while reducing the SLOC.
One reason is that in the ported version, methods are ﬁrst declared in traits and thenimplemented in classes while the refactored version needs no prior declarations. Anotherreason is that by properly using Castor’s default visitors and combined visitors, some def91

Chapter 4. Castor: Programming with Extensible Generative Visitors

Table 4.3: Performance evaluation in milliseconds.

Name
test1
test2
test3

Description
1000 sequential actions
100 parallel branches each with 10 actions
Similar to test2 with a variable increased

Interpreter
22.1
20.7
22.8

Castor
56.6
39.8
39.9

initions can be shortened. For example, Execute in the refactored version is a combined
visitor for Expression and 4 operators.
Q4

We reuse the test suite provided by the TTC’15 document, which includes 3 large

activity diagrams for measuring the performance. Table 4.3 gives a simple description foreach test case and the average execution time for 10 runs (measured in milliseconds) for
the two implementations. The benchmark is executed using the same machine speciﬁedin Section 4.7. The Castor’s implementation is around 2 to 3 times slower than the
non-modular ported implementation. These results are similar to the results we get in
Section 4.7 and further conﬁrm that Castor’s modular implementation introduces anacceptable performance penalty.
Threats to validity

One threat to the validity of the evaluation is that the test suite isvery small and might not be able to ﬁnd out bugs that are introduced by refactoring. Also,
directly comparing a Castor’s implementation with respect to the reference implementation may be unfair since diﬀerent programming languages are used. To exclude suchlanguage-wise factor on evaluation, we compared to the ported Scala implementation. As
our focus is on the semantics part, irrelevant code like parsing is ignored.

4.9

Design Options

In this section, we brieﬂy discuss other design options and their compromises.
Nested patterns

There is an alternative way of writing nested patterns. For example,

tmIf can be rewritten in the following way:override def tmIf = x => x.t1 match {case TmTrue => x.t2
case TmFalse => x.t3
case t1 => TmIf(this(t1),x.t2,x.t3)}

Instead of directly pattern matching on an TmIf object, we capture it ﬁrst using a variable x and then explicitly match on its subterm t1. For the case of tmIf, this alternative
implementation is arguably less intuitive than the version we presented in Section 5.2.
Nevertheless, this approach comes in handy when: 1) the object being matched contains
a lot of ﬁelds and most of them are not interesting in nested patterns; 2) there are a lotof case clauses for nested patterns and repeating the top-level pattern in each case clause
becomes tedious.
924.10 Conclusion

Specialized visitors

Programming with visitors can be simpliﬁed using specialized visi-

tors. The default visitors generated by Castor (annotated as @default) are an instance. In
fact, there are more such specialized visitors. For example, visitors can be combined withvisitor combinators [Visser, 2001]; boilerplate for querying and transforming data structures can be eliminated by traversal templates [Zhang and Oliveira, 2017]. Essentially,
these specialized visitors can also be generated by Castor. Currently, only default visitors
are generated because 1) in our experience they are most frequently used; 2) generating allother infrequently used specialized visitors increases the time of code generation and thesize of generated code. Ideally, specialized visitors should be generated by need. Limited
by current Scalameta, this is impossible for the moment.
Reﬁnable variants

As our visitor encoding shows, the key to extensibility is capturingconcrete types with bounded type members for allowing future reﬁnements. The sameidea can also be applied to variants, where the visitor method signature refers to a typemember instead of a class name. By doing this, we are able to extend that class withadditional ﬁelds seamlessly by covariantly reﬁning the type member to the new class. An
application of reﬁnable variants would be guarded transitions discussed in Section 4.5.3:
class Trans(event: String, to: State, var tm: Tm[Boolean] = TmTrue)
extends super.Trans(event, to)Instead of adding a new variant called GuardedTrans, we reﬁne the existing Trans. The
beneﬁt is that existing visitors that do not concern about the additional parameter tm canbe unchanged. In contrast, for the case of GuardedTrans, we have to update all existing
visitors with an implementation of guardedTrans. However, the downside of supporting
reﬁnable variants in Castor is that it brings more book-keeping burden on variants forthe user. We consider the price to pay is higher than the beneﬁt it brings.

4.10 Conclusion
In this chapter, we have presented Castor, a Scala framework for programming withextensible, generative visitors using simple annotations. Visitors written with Castor are
type-safe, concise, exhaustive, extensible and composable. Moreover, both functional andimperative style visitors are supported. We have shown how to use Castor in designinga better pattern matching mechanism in an OO context, developing modular well-typed
EDSLs, doing extensible programming on graphs, etc. The eﬀectiveness of Castor isvalidated by our case studies on TAPL interpreters and UML activity diagrams.

93

Chapter

5

Compositional Programming

In this chapter, we shift our focus on novel language design for modularity. We
propose a new statically typed modular programming style called Compositional Programming. Compositional Programming oﬀers an alternative way to model data structuresthat diﬀers from both algebraic datatypes in functional programming and conventional
OOP class hierarchies. We introduce four key concepts for Compositional Programming:
compositional interfaces, compositional traits, method patterns and nested trait composition. Altogether these concepts allow us to naturally solve challenges such as the EP,
model attribute-grammar-like programs and generally deal with modular programs withcomplex dependencies. We present a language design called CP, which is proved to be
type-safe, together with several examples and three case studies.

5.1 Introduction
An important aspect of programming is how to deﬁne data structures and the operations over those data structures. Diﬀerent language designs oﬀer diﬀerent mechanisms
for this purpose. OOP languages model data structures using class hierarchies and techniques such as the Composite pattern [Gamma et al., 1994]. Typically, there is an interface that speciﬁes all the operations (methods) of interest for the data structure. Multiple
classes implement diﬀerent types of nodes in the data structure, supporting all the operations in the interface. Many functional languages employ algebraic datatypes [Burstall
et al., 1981] to model data structures, and use functions (typically deﬁned by patternmatching) to model the operations over those data structures. As widely acknowledgedby the EP [Wadler, 1998], both algebraic datatypes and class hierarchies have modularity
problems. With algebraic datatypes and pattern matching, adding new operations is easy,
but adding constructors is hard.
This chapter presents a new statically typed modular programming style called Compositional Programming. In Compositional Programming, there is no EP: it is easy to get
extensibility in two dimensions (i.e. it is both easy to add new constructors, as well as
new operations). Compositional Programming oﬀers an alternative way to model data
structures that diﬀers from both algebraic datatypes in functional programming and conventional OOP class hierarchies. The key ideas of Compositional Programming are implemented in a new programming language called CP. For example, the code for modeling95

Chapter 5. Compositional Programmingarithmetic expressions can be expressed in CP as:
type ExpSig<Exp> = {-- Compositional interface for expressions with a sort Exp

Lit : Int -> Exp;-- Constructor (returns the sort Exp)

Add : Exp -> Exp -> Exp;-- Constructor (returns the sort Exp)};
type Eval = { eval : Int };-- Concrete type for evaluation (instantiates Exp)

evalNum = trait implements ExpSig<Eval> => {(Litn).eval = n;

(Add e1 e2).eval = e1.eval + e2.eval;-- Compositional trait for evaluation-- Definition using a method pattern-- Definition using a method pattern};

In the CP code above, the deﬁnition ExpSig<Exp> (a compositional interface) plays asimilar role to the algebraic datatype in functional languages. The special type parameter
Exp in ExpSig<Exp> is called a sort, and models types that represent datatypes in CP. Allconstructors in CP must have a return type which is a sort. Like in functional languages,in CP adding new operations is easy using a special form of traits [Schärli et al., 2003].
Unlike functional programming, where adding new constructors is diﬃcult and nonmodular, in CP the addition of new constructors is also easy. We will later illustrate themodularity of CP in detail. In essence, there are four new key concepts in CP:• Compositional interfaces can be viewed as an extension of traditional OOP interfaces, such as those found in languages like Java or Scala. In addition to declaring
method signatures, compositional interfaces also allow the speciﬁcation of constructor signatures. In turn, this enables programming the construction of objects against
an interface, instead of a concrete implementation. Compositional interfaces can beparametrized by sorts, which abstract over concrete types, and are used to determine which kind of object is built.
• Compositional traits extend traditional traits [Schärli et al., 2003] and ﬁrst-classtraits [Bi and Oliveira, 2018]. Compositional traits allow not only the deﬁnitionof (virtual) methods but also the deﬁnition of virtual constructors. They also allow
nested traits, which are used to support trait families. Trait families are akin toclass families in family polymorphism [Ernst, 2001].
• Method patterns, such as (Lit n).eval, provide a lightweight syntax to deﬁnemethod implementations for nested traits, which arise from virtual constructors.
This enables compact method deﬁnitions for trait families, which resemble programs deﬁned by pattern matching in functional languages, and programs writtenwith attributes in attribute grammars [Knuth, 1968, 1990].
• Nested trait composition is the mechanism used to compose compositional traits.
The foundations for this mechanism, originate from nested composition, which
has been investigated in recent calculi with disjoint intersection types and polymorphism [Oliveira et al., 2016; Alpuim et al., 2017; Bi et al., 2018, 2019]. Weshow how nested composition can smoothly be integrated into compositional traits.
Nested trait composition plays a similar role to traditional class inheritance, but
96

5.1 Introduction

generalizes to the composition of nested traits. Thus it enables a form of inheritance of whole hierarchies, similar to the forms of composition found in family
polymorphism. Nested trait composition is associative and commutative [Bi et al.,2018], just like the composition for traditional traits [Schärli et al., 2003].
Altogether, these concepts allow us to solve various challenges naturally. For instance,
they enable a very natural and simple solution to the EP. More interestingly, Compositional Programming can deal with harder modularity challenges, such as modeling interesting classes of attribute grammars, which often contain non-trivial dependencies. Such
attribute-grammar-like programs can be expressed in a statically safe way and without
giving up the modularity beneﬁts of attribute grammars. More generally, Compositional
Programming oﬀers a range of mechanisms to deal with modular programs with various
complex forms of dependencies.
The CP language is inspired by the SEDEL language [Bi and Oliveira, 2018]. The
main novelties over SEDEL are the four compositional programming mechanisms listed
above: compositional interfaces, compositional traits, method patterns, and nested trait
composition. Compositional Programming is partly inspired by generalized Object Algebras [Oliveira et al., 2013], but the built-in language mechanisms make modular programming natural, fully statically typed, and without boilerplate code and excessive
parametrization. We present several examples and three case studies in CP. We also
introduce a technique called polymorphic contexts to deal with components that require
some form of context in a modular way. In turn, polymorphic contexts are helpful to model
L-attributed grammars. Our ﬁrst case study is on the design of an Embedded DomainSpeciﬁc Language (EDSL) for circuits [Hinze, 2004; Gibbons and Wu, 2014]. This EDSL
is interesting because it has various extensions that can be modularly deﬁned, as well as
various dependencies between components. Our second case study is a mini interpreter,
which is a larger study and can be extended in several ways. The last case study is
an implementation of the C0 compiler, inspired by the work of Rendel et al. [2014]. In
this case study, various extensions can be formulated as attributes and those attributes
contain non-trivial dependencies to other attributes.
Finally, we present a small calculus that captures the essence of CP. This calculus isshown to be type-safe via an elaboration to the Fi+ calculus [Bi et al., 2019], which is a
recently proposed calculus that supports disjoint intersection types [Oliveira et al., 2016],
disjoint polymorphism [Alpuim et al., 2017] and nested composition [Bi et al., 2018].
In summary, the contributions of this chapter are:• Compositional Programming: We propose a new programming style that encourages weaker dependencies and increases the modularity of programs. Compositional Programming eliminates the EP and can deal with modular programs withcomplex dependencies.
• A language design for Compositional Programming: We present a concrete language design in the form of the CP calculus. The semantics of this calculus is givenby elaboration to the Fi+ calculus and we prove the type safety of the elaboration.
97

Chapter 5. Compositional Programming• Attribute Grammars in CP: We show that CP is powerful enough to implement
programs with attributes that are expressible in attribute grammars [Knuth, 1968,
1990] in a concise and fully statically type-checked manner. Our technique is partly
inspired by the encoding of Rendel et al. [2014], but CP avoids explicit deﬁnitions ofcomposition operators, which are necessary in Rendel et al. [2014]’s encoding.
• Polymorphic contexts: We introduce a simple technique that combines disjoint
polymorphism and Compositional Programming to allow for modular contexts inmodular components.
• Implementation, case studies, and examples: We have an implementation of
CP. We present several examples and three case studies in CP. Altogether, theseexamples and case studies illustrate how to naturally solve challenges such as the
EP or modeling attribute-grammar-like programs. The implementation and casestudies can be found in:
https://github.com/wxzh/CP

5.2

An Overview of Compositional Programming

This section presents an overview of Compositional Programming. We start by introducing the basic mechanisms of Compositional Programming with the EP. We thenmove on to harder modularity issues, such as various forms of dependencies, which arise
in modular programs that compute various forms of attributes. All the CP programs
presented here can run in our implementation of CP.

5.2.1 The Expression Problem with Compositional Programming
In Compositional Programming, there is no EP: it is both easy to add new variants,
as well as new operations. Indeed, an explicit goal of Compositional Programming is that
programmers do not need to face the tension of choosing one dimension for extensibility.
Therefore, Compositional Programming oﬀers an alternative way to model data structuresthat diﬀers from both algebraic datatypes and conventional OOP class hierarchies. Because of such fundamental diﬀerences, and the more modular programming style, we
can think of Compositional Programming as an alternative programming paradigm We
introduce the mechanisms used in the programming language CP by solving the EP next.
Compositional interfaces and sorts

In CP, we use a compositional interface to declarea datatype. Compositional interfaces generalize conventional OOP interfaces: we candeﬁne not only the signatures of methods but also those of constructors, which is notpossible in conventional OOP languages like Java. The compositional interface for basicarithmetic expressions is:
type ExpSig<Exp> = {Lit : Int -> Exp;
Add : Exp -> Exp -> Exp;98

5.2 An Overview of Compositional Programming};

There are two kinds of expressions for now: literals and addition. The type parameter Exp
wrapped in angle brackets is called a sort of the compositional interface, working as thetype of both kinds of expressions. Lit and Add are the signatures of the constructors forarithmetic expressions. In CP, constructors always start with a capital letter, while all
methods start with a lowercase letter. The return type of a constructor must always be asort, and there can be arbitrarily many sorts in a compositional interface.
Note that sorts in compositional interfaces are handled diﬀerently from normal typeparameters, which is why they have a special syntax of angle brackets around them.
In essence, compositional interfaces are based on several ideas related to generalized
Object Algebras (see 2.4.1), and the special treatment for sorts involves, among otherthings, distinguishing uses of positive and negative occurrences of the type variables.
The semantics of sorts is discussed in detail in Section 5.4. Nonetheless, these subtle
semantic diﬀerences are essentially only relevant for programs with dependencies, such as
those presented in Section 5.2.2. For now, it suﬃces to think of sorts as type parameters.
Analogy with algebraic datatypes

Compositional interfaces deﬁning only constructors

play a similar role to algebraic datatypes in functional programming.

Like algebraic

datatypes, they can be used to deﬁne data structures by specifying the name of the data
structure and the signatures of the constructors. For example, a Haskell counterpart ofthe compositional interface above is:data Exp where
Lit :: Int -> Exp
Add :: Exp -> Exp -> Exp

The sort Exp of the compositional interface corresponds to the name of the datatype,whereas in both cases the constructors essentially specify the same information: thesignatures of the constructors.
Compositional traits

The unit of code reuse in CP is a generalization of (ﬁrst-class)traits [Schärli et al., 2003; Bi and Oliveira, 2018]. The generalization stems from the factthat we can deﬁne not only method implementations but also (virtual) trait constructors
within a trait. In CP, the implementation of the eval operation for the basic form of
expressions can be done in a trait:
type Eval = { eval : Int };evalNum = trait implements ExpSig<Eval> => {(Litn).eval = n;

(Add e1 e2).eval = e1.eval + e2.eval;};This trait implements the compositional interface ExpSig with the sort instantiated as
Eval, corresponding to its actual operation. Eval is another example of a compositional

interface, which in this case, because it has no sorts, just degenerates into a conventional OOP-style interface with a method eval returning Int. Within the trait evalNum, we
implement the eval method for Lit and Add to evaluate the arithmetic expressions.
99

Chapter 5. Compositional Programming

Method patterns

The method pattern (Lit n).eval is a lightweight syntax used to deﬁne

the eval method within the trait (which implements the interface Eval) that the constructor
Lit returns. In CP, method patterns are used to make trait deﬁnitions concise. Method

patterns allow deﬁnitions that resembles functional programming deﬁnitions by patternmatching, or attributes in attribute grammars. An alternative way of deﬁning constructors would explicitly use ﬁrst-class traits, which is essentially what method patterns aredesugared to:evalNum = trait implements ExpSig<Eval> => {Lit

n = trait => { eval = n; };Add e1 e2 = trait => { eval = e1.eval + e2.eval; };};Nested traits and trait families

As shown above, method patterns inside a trait are

essentially deﬁning nested traits. The outer trait, evalNum, is called a trait family. The
terminology trait family is borrowed from family polymorphism [Ernst, 2001]. In family
polymorphism, a family is a class that contains nested virtual classes. In CP, a trait family
is a trait that contains nested virtual traits.

Virtual constructors

One signiﬁcant diﬀerence from most existing languages is that

CP’s constructors are virtual. In languages like Java, there are virtual methods, whose
concrete implementation is unknown at the time a class is deﬁned. In this way, the references to methods are loose and determined only when objects are instantiated. However,languages like Java do not support virtual constructors or virtual classes [Madsen andMoller-Pedersen, 1989]. A virtual constructor is not bound to a speciﬁc implementation ofa trait. Instead, it conforms to the signature in the compositional interface. With virtual
constructors, it is possible to create a trait that constructs an expression without sticking
to a particular implementation of the compositional interface:
expAdd Exp = trait [self : ExpSig<Exp>] => {test = new Add (new Lit 4) (new Lit 8);};

In CP, term parameters start with a lowercase letter while type parameters are capitalized,
so Exp is a type parameter, serving as the sort of ExpSig. Thus expAdd contains an abstract
expression that is not associated with any concrete method implementations.

Self-type annotations

The expAdd trait above is parameterized by Exp and speciﬁes its

self-type as ExpSig<Exp>. Such self-type annotations are similar to those in Scala [Odersky
et al., 2004], which enable a modular way of injecting dependencies on other operations/constructors. The self-type annotation of expAdd expresses that expAdd must be mergedwith some trait that concretely implements ExpSig for instantiation. By specifying theself-type, the constructors declared inside ExpSig, i.e. Lit and Add, are directly available
for building an expression named test. Section 5.2.2 will discuss additional mechanisms
in CP to deal with other forms of dependencies.
100

5.2 An Overview of Compositional Programming

Extensibility: adding new operations

Like in functional programming with algebraic

datatypes and pattern matching, adding a new operation is trivial in CP. We can justcreate an independent trait family that implements the compositional interface:type Print = { print : String };printNum = trait implements ExpSig<Print> => {(Litn).print = n.toString;(Add e1 e2).print = "(" ++ e1.print ++ "+" ++ e2.print ++ ")";};The trait printNum implements ExpSig<Print> and deﬁnes Lit and Add using method patterns, in a way similar to evalNum. It modularly supports pretty-printing for expressions.
Nested trait composition

At the heart of Compositional Programming is a powerfulcomposition mechanism called nested composition [Bi et al., 2018]. Nested composition
allows the composition of multiple trait families. This mechanism can be viewed as aform of multiple (trait) inheritance, but with the added ability to recursively compose
nested traits automatically. Thus it provides composition mechanisms that are similar tothe ones found in languages with family polymorphism. Like trait composition [Schärli
et al., 2003], nested composition in CP is associative and commutative [Bi et al., 2018].
Furthermore, following the trait model and SEDEL, conﬂicts arising during composition
are rejected. In CP, which is statically typed, such conﬂicts are statically rejected as typeerrors, following a similar approach to trait composition in SEDEL. Conﬂict resolution in
CP can be done using method overriding or an explicit exclusion operator (like in manymodels of traits).
In CP, nested composition is performed by the merge operator ,, [Dunﬁeld, 2014]. For
example, to compose the trait families evalNum and printNum with expAdd we can write:e = new evalNum ,, printNum ,, expAdd @(Eval&Print);

Note that the merge operator ,, binds tighter than new (but application and type application still bind tighter than ,,). The type application expAdd @(Eval&Print) makes
the generic trait expAdd concrete by instantiating the type parameter Exp as the argument Eval&Print. Since the self-type annotation of expAdd has been instantiated as
ExpSig<Eval&Print>, in order to meet this self-type requirement, the trait has to be mergedwith some trait that simultaneously implements the eval and print operations for theconstructors exposed by ExpSig. The requirement is met by merging expAdd @(Eval&Print)
with evalNum and printNum. Thus, the merged trait is successfully instantiated into an
object using a new expression.
It is useful to take a moment and understand why the trait composition performed for e and the method calls in the expression above pass typechecking and work as expected.

Note that the types of evalNum,

printNumand expAdd @(Eval&Print) are, respectively, Trait[ExpSig<Eval>], Trait[ExpSig<Print>]
and Trait[ExpSig<Eval&Print>,{test:Eval&Print}].

Their merge is then of the type

Trait[ExpSig<Eval&Print>,ExpSig<Eval>&ExpSig<Print>&{test:Eval&Print}].

The merged

trait type is an instantiatable trait type (i.e. the provided type is a subtype of the required type) because ExpSig<Eval>&ExpSig<Print> is a subtype of ExpSig<Eval&Print> in CP.
101

Chapter 5. Compositional Programming

In essence, the subtyping relation employed by CP supports distributivity of intersections
over other constructs [Bi et al., 2018]. In CP, intersections can distribute over function
types, records, and trait types. Through the object e, we can both evaluate and print theaddition expression:e.test.print ++ " is " ++ e.test.eval.toString

Extensibility: adding new variants--> "(4+8) is 12"

Finally, we show how to add multiplications to

the language modularly. Note that this is where Compositional Programming diﬀers fromalgebraic datatypes and deﬁnitions by pattern matching, where such extensions cannot
be modularly added. We ﬁrst deﬁne a compositional interface that extends ExpSig with a
Mul constructor:
type MulSig<Exp> extends ExpSig<Exp> = {Mul : Exp -> Exp -> Exp;};We then implement evaluation and printing by deﬁning two trait families evalMul andprintMul that respectively inherit evalNum and printNum and complement a deﬁnition for
Mul:
evalMul = trait implements MulSig<Eval> inherits evalNum => {(Mul e1 e2).eval = e1.eval * e2.eval;};printMul = trait implements MulSig<Print> inherits printNum => {(Mul e1 e2).print = "(" ++ e1.print ++ "*" ++ e2.print ++ ")";};

Without editing any existing code, we modularly add new data variants. Note that here weuse a new keyword inherits. In CP, inheritance is based on the merge operator but givena more convenient syntax that resembles conventional OOP. Besides nested composition,
inherits additionally allows ﬁelds deﬁned in the parent trait to be overridden but still canbe used via super calls in the trait body. With super calls and inheritance, we can, forinstance, construct a slightly more complex expression:
expMul Exp = trait [self : MulSig<Exp>] inherits expAdd @Exp => {override test = new Mul super.test (new Lit 4);};

The trait expMul inherits expAdd, reﬁnes the self-type to MulSig and overrides the test
ﬁeld. The overridden expression reuses the inherited expression via super.test. Finally,
the object e’ supports all three constructors together with the evaluation and printing
operations.

Summary

For the basic EP, there are already many solutions in the literature, including

some in mainstream programming languages [Garrigue, 2000; Torgersen, 2004; Oliveira
et al., 2006a; Zenger and Odersky, 2005; Swierstra, 2008; Oliveira and Cook, 2012]. One
interesting aspect of the solution presented here is that it is quite elegant and natural,
while solutions in mainstream languages tend to have signiﬁcant amounts of boilerplate102

5.2 An Overview of Compositional Programming

code, or are written in highly parametrized code that is not programmer-friendly. Additionally, the more interesting aspect of Compositional Programming is its wide support
for various forms of modular code with dependencies, which is shown next.

5.2.2 Dependencies and S-Attributed Grammars
In the original EP by Wadler [1998] there are very few dependencies. In particular,
the operations (eval and print) depend only on themselves, but they do not depend onother operations. Matters become signiﬁcantly more complicated in the presence of moreadvanced forms of dependencies, and very few existing solutions to the EP have eﬀective
mechanisms to deal with dependencies in a modular way. Two primary mechanisms areused to deal with dependencies in CP:• Compositional interface type reﬁnement: When a trait implements some compositional interface, it can reﬁne the sort types in input positions. This allows the childnodes in a data structure to assume some functionality that is not implemented in
the enclosing trait, but will eventually be part of the ﬁnal composition later.
• Self-type annotations: Like in Scala, the types of self-references can be speciﬁed/reﬁned. This enables the self-references to assume functionality that will be
implemented by a diﬀerent trait that will eventually be composed with the current
trait. In the context of trait families, there are two kinds of self-references: object
self-references and family self-references [Oliveira et al., 2013]. The trait expAdd
in Section 5.2.1 already illustrates family self-references, so in what follows weillustrate only object self-references.
We use examples inspired from S-attributed grammars [Knuth, 1968] and the work from Rendel et al. [2014] to illustrate how Compositional Programming addresses programs with
diﬀerent kinds of dependencies in a modular way. S-attributed grammars deal with synthesized attributes, which are computed from the children. Compositional Programming
also allows dependencies on self. The simplest form of dependencies is dependencies on
the same attribute of children, which occurred several times in Section 5.2.1. Therefore,
we focus on two kinds of non-trivial dependencies on other synthesized attributes, which
hereinafter we call child dependencies and self dependencies.

Child dependencies

Child dependencies occur when an attribute depends on othersynthesized attributes of the children. Here is the printChild example from Section 4.1 inCP:
printChild = trait implements ExpSig<Eval % Print> => {(Litn).print = n.toString;(Add e1 e2).print = if e2.eval == 0 then e1.print
else "(" ++ e1.print ++ "+" ++ e2.print ++ ")";};

103

Chapter 5. Compositional Programming

Here (Add e1 e2).print depends on e2.eval. However, there is no implementation of eval in
the trait printChild. To make this child dependency feasible, we use compositional interface type reﬁnement: we change the concrete interface being implemented to ExpSig<Eval% Print> (instead of just using ExpSig<Print>). This syntax means that the input type forthe expressions in the Add constructor (and other constructors, if any, referring to thesort Exp) is Eval&Print, while the output type is still Print. By doing this, we enable thechild nodes of Exp to depend on an attribute whose implementation is modularly deﬁned
somewhere else. Two examples of trait instantiations are:new printChild ,, expAdd @Print-- Type Error!

new printChild ,, evalNum ,, expAdd @(Print&Eval)-- OK!

This ﬁrst instantiation attempt fails type-checking because it depends on eval, which is
missing. The second one works since we merge printChild with the trait evalNum (which
implements Eval). Importantly, instead of evalNum, we could have used any implementation with the same type.
Contrast with inheritance

It is useful to compare the previous example with the morecommon OOP approach based on inheritance:
printInh = trait implements ExpSig<Eval&Print> inherits evalNum => {(Litn).print = n.toString;(Add e1 e2).print = if e2.eval == 0 then e1.print
else "(" ++ e1.print ++ "+" ++ e2.print ++ ")";};

The ﬁrst thing to notice is that the code in the trait body of printIhn is exactly the same asthat in printChild. Conforming to the compositional interface instantiated with Eval&Print,
printInh essentially implements both eval and print instead of only print, because theevalNum trait family is inherited. Although both approaches work, printInh is tightly cou-

pled with the particular implementation of evaluation coming from evalNum. In contrast,
printChild declares a weaker dependency on the abstract interface of Eval. We can delay

the combination with a concrete implementation of Eval until the instantiation phase. In
short, printChild allows for weak child dependencies, which are not coupled with a particular implementation, while preserving strong static type safety. More generally, mostuses of inheritance in CP can be converted into code that has weaker dependencies.
Self dependencies

A second interesting case is dependencies on other synthesized at-tributes of the self-reference. In the following example, the attribute print depends on
self.eval:printSelf = trait implements ExpSig<Eval % Print> => {(Litn).print = n.toString;(Add e1 e2 [self:Eval]).print = if self.eval == 0 then "0"else "(" ++ e1.print ++ "+" ++ e2.print ++ ")";};

To deal with this dependency without sticking to a particular implementation of eval, we
add an (object) self-type annotation [self:Eval] to use self.eval in Add. Note that we104

5.3 Parametric Polymorphism and L-Attributed Grammars

also need to change the sort instantiation to ExpSig<Eval % Print> like the child dependencies, in order to change the self-type of the returning trait correspondingly. The static
type-checker will check whether the trait is later merged with another trait that implements ExpSig<Eval>. With no compromises on type safety, CP enables modular weak self
dependencies on other attributes.

Mutual dependencies

Finally, a more general form of dependencies is mutual depen-

dencies, which happen when two attributes are inter-deﬁned, i.e., they depend on eachother. Mutual dependencies can involve both child and self dependencies, as illustrated
in the following example:
type PrintAux = { printAux : String };printMutual = trait implements ExpSig<PrintAux % Print> => {(Litn).print = n.toString;(Add e1 e2).print = e1.printAux ++ "+" ++ e2.printAux;};
printAux = trait implements ExpSig<Print % PrintAux> => {(Litn [self:Print]).printAux = self.print;(Add e1 e2 [self:Print]).printAux = "(" ++ self.print ++ ")";};The two trait families printMutual and printAux cooperate to omit the outermost parentheses in pretty-printing. We can see that (Add e1 e2).print depends on the printAux while
printAux depends on print, thus print and printAux are mutually dependent. CP handles

such mutual dependencies modularly. We can combine the traits and use them as before:(new printMutual ,, printAux ,, expAdd @(Print&PrintAux)).test.print

Summary--> "4+8"

Many attribute grammar systems allow the modular deﬁnition of attributes,

but this is usually done at the cost of modular type-safety. The compositional mechanisms
in CP retain the ability of modularly deﬁning attributes from S-attributed grammars oreven attributes from self-references, but in a statically type-safe setting. Moreover, theimplementations of the attributes are changeable in the ﬁnal composition. For example,
the aforementioned traits printNum, printChild, printSelf, and printMutual all implement
the print method. A programmer can freely pick his favorite implementation to combinewith other attributes, such as eval.

5.3 Parametric Polymorphism and L-Attributed Grammars
In Section 5.2, we introduced the basic mechanisms for Compositional Programmingand illustrated how CP deals with dependencies. One feature that practically all modern
languages support is some form of parametric polymorphism (or generics in OOP languages). A reasonable question to ask is how Compositional Programming interacts withparametric polymorphism. Furthermore, one may wonder if the combination of parametric polymorphism and Compositional Programming enables novel techniques that are
useful for programming.
105

Chapter 5. Compositional Programming

In this section, we explore this question. CP has full support for a form of parametric polymorphism called disjoint polymorphism [Alpuim et al., 2017]. We introduce
a novel technique called polymorphic contexts, which addresses the problem of diﬀerent
modular components requiring diﬀerent kinds of contextual information. The techniqueprovides encapsulation of contexts: modular components have access to the contextual
information they require but do not have access to contextual information used by othercomponents. This technique is useful to model inherited attributes, where it is possible toaccess attributes from parents or siblings. To achieve this, a context should be attachedto pass attributes from top to bottom. There is a close relationship between contextual
evaluation and L-attributed grammars [Knuth, 1968; Rendel et al., 2014].

5.3.1 Contexts and Modular Components
There are plenty of scenarios where we need to add contexts to our code. One of themost common examples is variable binding in an interpreter. Recall that, in Section 5.2.1,
we deﬁned the type of eval as Int. But this interface is not suitable for an expression
language with variable binding. A naive ﬁx is to modify the existing code and add a context
to all the trait families:
type Eval = { eval : EnvN -> Int };evalNum = trait implements ExpSig<Eval> => {(Litn).eval (env:EnvN) = n;

(Add e1 e2).eval (env:EnvN) = e1.eval env + e2.eval env;};
evalVar = trait implements VarSig<Eval> => {(Let s e1 e2).eval (env:EnvN) = e2.eval (insert @Int s (e1.eval env) env);(Vars).eval (env:EnvN) = lookup @Int s env;};

Note that we add an EnvN parameter to eval. Since CP’s support for type inference is still
limited, while n, e1 and e2 do not require type annotations, we have to annotate env with
EnvN here. EnvN is a map from String to Int, serving as the variable environment, while
insert and lookup are auxiliary functions on maps. For the Let expression in evalVar, weevaluate e1 and insert the evaluation result into env before evaluating e2 (the body of the
Let expression). For the Var expression, we look up the variable name to get its value.

Although evalNum does not need the context for evaluating arithmetic expressions, we still
have to pass the context to the recursive calls to make env available everywhere.
So far, it seems that everything works well. But what if we add a new context? For
example, many interpreters need to pre-deﬁne some primitive functions, which are called
intrinsics. Therefore, we should add an intrinsic environment to store these intrinsic functions. Just like Common Lisp, functions and values do not share the same namespace inthis expression language, so these two environments are independent of each other. We
need a second parameter for eval, which requires modifying all existing code again:
type Eval = { eval : EnvN -> EnvF -> Int };evalNum = trait implements ExpSig<Eval> => {(Litn).eval (envN:EnvN) (envF:EnvF) = n;

(Add e1 e2).eval (envN:EnvN) (envF:EnvF) = e1.eval envN envF + e2.eval envN envF;1065.3 Parametric Polymorphism and L-Attributed Grammars};
evalVar = trait implements VarSig<Eval> => {(Let s e1 e2).eval (envN:EnvN) (envF:EnvF) =e2.eval (insert @Int s (e1.eval envN envF) envN) envF;(Vars).eval (envN:EnvN) (envF:EnvF) = lookup @Int s envN;};
evalFunc = trait implements FuncSig<Eval> => {(LetF s f e).eval (envN:EnvN) (envF:EnvF) = e.eval envN (insert @Func s f envF);
(AppF

s

e).eval (envN:EnvN) (envF:EnvF) = (lookup @Func s envF) (e.eval envN

envF);};Such an approach to adding contextual information has two main problems:• It is highly non-modular: Every time a new context is needed, all the existing code
has to be modiﬁed. What is worse, we cannot easily modify the type of context if
the previous deﬁnitions are from a library. In other words, the library author has toanticipate what kind of contexts will be needed in the future, which is impossible.
• It does not encapsulate contexts: Interfaces of contexts are fully exposed, even if
they are not directly used. For example, Let and Var do not touch envF, while LetF
and AppF do not touch envD. Such unnecessary exposure may lead to unexpected
modiﬁcations to the contexts which ought to be hidden to avoid exploitation bymalicious code.

5.3.2 Polymorphic Contexts
To address the two problems identiﬁed in the previous section, we propose a techniquethat relies on disjoint polymorphism, intersection types and nested composition.

This

technique enables modular, encapsulated contexts for modular components. Since wecannot anticipate what a context will evolve to in the future, our idea is to make contexts
subject to change using parametric polymorphism.
Evaluation with a polymorphic context

Instead of creating an interface for evaluation

with speciﬁc contexts, we can parametrize the type of the context:
type Eval Context = { eval : Context -> Int };Here Context is a type parameter.

In essence Eval becomes a variant of the Reader

Monad [Wadler, 1992], which is commonly used in functional programming. Similarly
to Monads, some initial planning is necessary to adapt the existing code to use polymorphic contexts:
evalNum Context = trait implements ExpSig<Eval Context> => {(Litn).eval (ctx:Context) = n;

(Add e1 e2).eval (ctx:Context) = e1.eval ctx + e2.eval ctx;};

The trait evalNum has a type parameter Context, which is used as the type of the context
in evaluation. Importantly, when implementing evalNum, the only thing one can do with
ctx is to pass it to the children’s call on eval since ctx is the only value of type Context

107Chapter 5. Compositional Programmingin scope. In other words, parametric polymorphism enforces the encapsulation of thecontext and ensures that the correct context is passed to the evaluation of the children.
To illustrate how CP enforces encapsulation of the context, suppose that we try to extract
information from the polymorphic context, for example, by trying to look up a variable inthe context:
evalNum Context = trait implements ExpSig<Eval Context> => {(Lit n).eval (ctx:Context) = lookup @Int "foobar" ctx;-- Type Error!

-- (Add e1 e2).eval ... is omitted}This code fails to type check because the type of ctx (i.e. Context) is not a subtype of EnvN,
and there is no dynamic casting in CP that can change its type to EnvN. In short, if thepolymorphic context is completely polymorphic (i.e. its type is just a type variable) thenthere is not much that can be done with the context except passing it down to recursive
calls.
To instantiate evalNum, which requires no context on its own, we can specify Context
as Top and pass () (the canonical top value) to eval:(new evalNum @Top ,, expAdd @(Eval Top)).add.eval ()--> 12

While the code requires an initial modiﬁcation to be adapted to polymorphic contexts,
no additional changes are necessary when future contexts are added to the program.
Moreover, a nice quality of the code is that it is written in a direct style. Using moresophisticated solutions, such as Monads, would give additional expressive power, but
often at the cost of writing code in a diﬀerent style (for instance, with the monadic donotation).
Adding components with contexts

Let us revisit the variable binding example. In

order to add constructs that deal with variables and binders, a context with an envD ﬁeld
is needed:e.eval ({ envF = insert @Func s f ctx.envF } ,, ctx:Context);

Now, we have to write the code for a trait that deals with the evaluation of variables andbinders. But what should be the type of context in this case? Using a fully polymorphic
context, as we did for literal and addition expressions, will not work, because we needto extract and update information from the environment. Furthermore, using the type
CtxN directly as the type of the environment is too speciﬁc, because it forces the contexts

to contain exactly CtxN and nothing else. This would prevent modular context evolution.
The answer is to use a context with the intersection type CtxN&Context:
evalVar (Context * CtxN) = trait implements VarSig<Eval (CtxN&Context)> => {(Let s e1 e2).eval (ctx:CtxN&Context) =e2.eval ({ envN = insert @Int s (e1.eval ctx) ctx.envN } ,, ctx:Context);(Vars).eval (ctx:CtxN&Context) = lookup @Int s ctx.envN;};
Context is a type variable disjoint with CtxN (expressed as the annotation Context * CtxN).

This is an example of disjoint polymorphism [Alpuim et al., 2017], which is supported
108

5.3 Parametric Polymorphism and L-Attributed Grammars

in CP. The disjointness constraint ensures that when the type variable is instantiated toa concrete type, that type cannot share a common supertype with CtxN. By using thetype CtxN&Context as the type of context, we ensure that we can access the environment
while being oblivious of other information in the context.

Thus the context remains

partly polymorphic and adaptable to future extensions while retaining encapsulation andensuring that the other information in Context cannot be altered.
A record update problem

The variable environment should be updated during theevaluation of the Let expression, whereas any other information in the context should
be retained as well. Note that, the type of {envD = insert ...} is CtxN, which does notmatch CtxN&Context. So we have to merge it with (ctx:Context) to get back the Context
part. This upcasting is possible because Context is a supertype of CtxN&Context. The
disjoint constraint also ensures that the merge passes type-checking. The code illustrates
that in CP we can do a polymorphic record update [Cardelli and Mitchell, 1991], which
is a notorious problem in many calculi with polymorphic records. For instance, it iswell-known that F<: with records (and many other calculi with bounded quantiﬁcation)cannot solve the polymorphic record update problem. There are only a few calculi withpolymorphic records and subtyping that can deal with this problem [Cardelli and Mitchell,
1991; Cardelli, 1994; Poll, 1997]. CP and its core foundation Fi+ are among them.
A second component with a context

To support intrinsic functions, we need a second

environment EnvF, and a corresponding trait family:
type CtxF = { envF : EnvF };evalFunc (Context * CtxF) = trait implements FuncSig<Eval (CtxF&Context)> => {(LetF s f e).eval (ctx:CtxF&Context) =e.eval ({ envF = insert @Func s f ctx.envF } ,, ctx:Context);
(AppF

s

e).eval (ctx:CtxF&Context) = (lookup @Func s ctx.envF) (e.eval ctx);};

Similarly, a polymorphic record update is needed for the LetF expression. Although thesepolymorphic trait families are deﬁned separately, we can still merge them together using
nested composition:
expPoly Exp = trait [self : ExpSig<Exp>&VarSig<Exp>&FuncSig<Exp>] => {test = new LetF "f" (\(x:Int) -> x * x)(new Let "x" (new Lit 9) (new AppF "f" (new Var "x")));};
e = new evalNum @(CtxN&CtxF) ,, evalVar @CtxF ,, evalFunc @CtxN ,,expPoly @(Eval (CtxN&CtxF));
e.test.eval { envN = empty @Int, envF = empty @Func }--> 81

During the composition of e, the context types used by other trait families are passed as
type arguments to make the ﬁnal context consistent.

5.3.3 L-Attributed Grammars
The previous examples only access attributes from parents. It is also possible forinherited attributes to depend on siblings. There is a superset of the aforementioned
109

Chapter 5. Compositional Programming

S-attributed grammars called L-attributed grammars [Knuth, 1968], where inherited attributes depend on parents and left siblings. In such grammars, attributes can be easily
evaluated by a left-to-right depth-ﬁrst traversal.
To illustrate L-attributed grammars, we take pretty-printing as an example. We use
a position number to represent each terminal node in this pretty-printing function. The
position number is determined by the pre-order traversal of the syntax tree.

Beforecomputing it, we need an auxiliary attribute called cnt that calculates the total numberof nodes in the current subtree:
type Cnt = { cnt : Int };cnt = trait implements ExpSig<Cnt> => {(Litn).cnt = 1;

(Add e1 e2).cnt = e1.cnt + e2.cnt + 1;};

With the help of cnt, we can compute that e1 .pos = e0 .pos + 1 and e2 .pos = e0 .pos + e1 .cnt + 1,
where e0 is the parent node of e1 and e2 . The latter equation is a typical example of Lattributed grammars because the attribute depends on both its parent (e0 .pos) and left
sibling (e1 .cnt). In our encoding, such computation is done on the parent side and theresult is passed down using polymorphic contexts:
type Pos = { pos : Int };type InhPos = { pos1 : Pos -> Int; pos2 : Pos -> Cnt -> Int };type PrintPos Ctx = { print : Pos&Ctx -> String };printPos (Ctx * Pos) = trait [self : InhPos] implements ExpSig<Cnt % PrintPos Ctx> =>{(Litn).print (inh:Pos&Ctx) = "{" ++ inh.pos.toString ++ "}";

(Add e1 e2).print (inh:Pos&Ctx) ="(" ++ e1.print ({ pos = pos1 inh } ,, inh:Ctx) ++ "+" ++e2.print ({ pos = pos2 inh e1 } ,, inh:Ctx) ++ ")";
pos1 (e0:Pos) = e0.pos + 1;
pos2 (e0:Pos) (e1:Cnt) = e0.pos + e1.cnt + 1;};

To compute the inherited attribute pos, we introduce two auxiliary functions in InhPos.
They are implemented in accordance with the equations stated before. The self-type annotation [self:InhPos] is added for (Add e1 e2).print to access these two functions. Just
like previous environments in interpreters, pos serves as a part of the context parameterof print. Since there is a child dependency on cnt, the sort is instantiated as <Cnt %PrintPos Ctx>. The position number is calculated and passed down via print calls in Add,

while inh.pos is ﬁnally used in Lit.
With all of the traits ready, we can compose an expression and call the print function
like before:
exp Exp = trait [self : ExpSig<Exp>] => {test = new Add (new Add (new Lit 1) (new Lit 2)) (new Add (new Lit 3) (new Lit 4));};e = new cnt ,, printPos @Top ,, exp @(Cnt & PrintPos Top);e.test.print { pos = 0 }--> (({2}+{3})+({5}+{6}))Since no other contexts need to be mixed together, we just pass Top as the type argumentof printPos. In the last invocation on print, we set the initial context to {pos = 0}. This110

5.4 Formalizationmeans that the root node of the whole syntax tree is marked as 0, then the parent of theleft subtree is 1 and the leaves are 2 and 3. Similarly, the parent of the right subtree is
4 and the leaves are 5 and 6. We ﬁnally print the position numbers of leaf nodes, as thecode shows above.
In fact, our encoding does not only allow L-attributed grammars, any inherited andsynthesized attributes can be implemented by contextual evaluation. Nevertheless, Lattributed grammars correspond to a one-pass traversal and ensure termination, so they
are the most common usage of attribute grammars and a good example for polymorphic
contexts.

Summary

With polymorphic contexts, L-attribute-grammar-like programs are expressedin a statically safe way. Such programs are not uncommon in real-world applications. Interpreters or other operations over ASTs are typical applications where non-trivial formsof attributes can occur. We have shown how variable and intrinsic environments aresupported in the previous examples. Besides the two, more contexts may emerge when alanguage evolves, such as dynamic scoping, mutable parameters, and error handling.
There are three advantages of our approach to polymorphic contexts: 1) it enforces therecursive calls to take the full context as an argument; 2) the polymorphic portion of thecontext cannot be ﬁddled with, i.e., the only thing one can do is to pass it unchanged; 3)
nonetheless, polymorphic contexts can still be reﬁned for particular uses and expose justthe right amount of information while hiding the remaining information. Polymorphic
contexts, as well as the ability to express complex forms of attributes, are a valuablesupplement to the modularity of Compositional Programming.

5.4 Formalization
This section presents the syntax and semantics of CP. The syntax of CP extends
SEDEL [Bi and Oliveira, 2018] with constructs for Compositional Programming (i.e. compositional interfaces, compositional traits, method patterns, and nested trait composition). The semantics of CP is given by elaborating to a call-by-name formulation of Fi+ [Biet al., 2019], a typed calculus combining disjoint intersection types and polymorphismwith BCD-style subtyping [Barendregt et al., 1983]. Nested trait composition is built on
top of Fi+ ’s nested composition [Bi et al., 2018]. We prove that the elaboration to Fi+ is
type-safe and coherent.

5.4.1 Syntax
Figure 5.1 gives the syntax of CP. A program is a sequence of declarations followed by
an expression.

Declarations

There are two kinds of declarations: type declarations and term declara-

tions. A type declaration type X hα i extends A = B is used for two purposes: introducing
type aliases and declaring compositional interfaces. To simplify the formalization, we
111

Chapter 5. Compositional Programming

ProgramDeclarations
Term declarations
Types

P
D
M
A, B

Sorts
Expressions

S
E

Term contexts
Type contexts
Sort contexts

Γ
∆
Σ

F
F
F
F
|
F
F
|
|
F
F
F

D; P | E
M | type X hα i extends A = B
x = E | (L x : A [self : B]).l = E
Int | α | > | ⊥ | A → B | ∀(α ∗ A).B | A & B | {l : A}
Trait[A, B] | X hS i
A |A%Bi | x | > | λx.E | E1 E2 | Λ(α ∗ A).E | E @A | E1 , , E2 | {M } | E.l
E : A | let x : A = E1 in E2 | open E1 in E2 | new E | E1 ˆE2
trait[self : A] implements B inherits E1 => E2• | Γ, x : A• | ∆, α ∗ A | ∆, X hα, i 7→ A
• | Σ, α 7→
Figure 5.1: Syntax of CP.

do not formalize type declarations with conventional type parameters (i.e. we only consider type declarations with sorts). However, adding type parameters can be done instandard ways and we have such declarations in our implementation. The type (or type
constructor) X is parametrized with a sequence of sorts hα i and can optionally extend
another type. There are two forms of term declarations: x = E is for simple variable bindings; (L x : A [self : B]).l = E is a method pattern serving as syntactic sugar for deﬁning
single-ﬁeld traits conveniently.

Types

Metavariables A and B range over types. Types include integers Int, type variables

α, the top type >, the bottom type ⊥, arrows A → B, disjoint quantiﬁcation ∀(α ∗ A).B,
intersections A & B, single-ﬁeld record types {l : A} (multi-ﬁeld record types are syntax
sugar for intersections of multiple single-ﬁeld record types) , trait types Trait[A, B], andtype aliases with sorts instantiated X hSi, where each sort can be either instantiated usinga type or a pair of types.

Expressions

Metavariable E ranges over expressions. Expressions include integer lit-

erals i, term variables x, the top value >, lambda abstractions λx.E, term applications E1 E2 , type abstractions Λ(α ∗ A).E, type applications E @A, merges E1 , , E2 ,multi-ﬁeld records {M }, record projections E.l, annotated expressions E : A, and (recursive) let bindings let x : A = E1 in E2 . There are also a few trait related constructs.
trait[self : A] implements B inherits E1 => E2 speciﬁes a an explicit self reference of typeA, type B to implement, an explicit self reference of type A, an inherited trait expression E1
and a body expression E2 . The new E construct instantiates a trait expression. E1 ˆE2 is
the forwarding expression inherited from SEDEL [Bi and Oliveira, 2018]. Inspired by MLlike modules [MacQueen, 1984], open E1 in E2 is a new construct for directly accessing
ﬁelds from a record without explicit projections.
112

5.4 Formalization

5.4.2 An Informal Introduction to the Elaboration
The semantics of CP is deﬁned by elaborating to Fi+ extended with recursive let bindings. Elaborating a CP program into a Fi+ expression takes several steps, including desugaring, sort transformation, type expansion, etc. The elaboration builds on two ideas fromthe literature: the denotational model of inheritance by Cook and Palsberg [1989], andgeneralized Object Algebras [Oliveira et al., 2013]. To better understand the elaboration,
let us revisit some of the examples presented in Section 5.2 and show their concrete elaborations into CP code using more atomic features, such as record types and records, andrecursive let bindings.

Elaborating compositional interfaces and sorts.

Firstly, the compositional interface

ExpSig:type ExpSig<Exp> = {Lit : Int -> Exp;
Add : Exp -> Exp -> Exp;};is translated to an equivalent type using only type parameters, and the sort Exp is eliminated:
type ExpSig Exp OExp ={ Lit : Double -> Trait[Exp,OExp] } &{ Add : Exp -> Exp -> Trait[Exp,OExp] };The sort Exp is represented by two type parameters, Exp and OExp, for respectively capturing the negative and positive occurrences of Exp. Negative occurrences of Exp (at input
positions) are kept unchanged while positive occurrences of Exp (at output positions) arechanged to OExp. For Add, the two parameters of type Exp are negative. Therefore, only the
return type of Add is transformed. Since Add is a constructor, positive occurrences of Exp
are further translated to a trait type Trait[Exp,OExp]. As a type synonym, ExpSig and its
right-hand side are put into the type context that tracks type declarations, among otherthings. In essence, this transformation is inspired by generalized Object Algebra interfaces [Oliveira et al., 2013] (see also 2.4.1), and the distinction of positive and negative
occurrences is helpful for expressing dependencies. If we just wanted to model modular programs without dependencies, then distinguishing between positive and negative
occurrences would not be necessary.
Similarly, the extended compositional interface MulSig<Exp>:
type MulSig<Exp> extends ExpSig<Exp> = {Mul : Exp -> Exp -> Exp;};is translated to a type equivalent to:
type MulSig Exp OExp ={ Lit : Double -> Trait[Exp,OExp] } &{ Add : Exp -> Exp -> Trait[Exp,OExp] } &{ Mul : Exp -> Exp -> Trait[Exp,OExp] };113

Chapter 5. Compositional Programmingwhere the type appearing in the extends clause is expanded by looking up the type context
and intersecting that type with the type on the original right-hand side.
Elaborating traits.

The trait family evalNum implements ExpSig:

evalNum = trait implements ExpSig<Eval> => {(Litn).eval = n;

(Add e1 e2).eval = e1.eval + e2.eval;};

which is desugared to:evalNum = trait [self: Top] implements ExpSig Eval Eval => open self in{ Lit = \(n: Double) -> trait => { eval = n } } ,,{ Add = \(e1: Eval) -> \(e2: Eval) -> trait => { eval = e1.eval + e2.eval} };The sort instantiation <Eval> indicates that there are no dependencies. Therefore, bothtype parameters (Exp and OExp) in the elaborated code are instantiated as Eval. Since
no self-type is speciﬁed, the self-reference has type Top by default. We can ignore theopen expression for the moment, since it has no eﬀect in this case, as the self type isTop. Method patterns are desugared to functions returning traits and multi-ﬁeld records

are desugared to a merge of multiple single-ﬁeld records. After type-checking, evalNum isfurther elaborated into a Fi+ expression:
let evalNum = \(self: Top) ->{ Lit = \(n: Int) -> \(self: Top) -> { eval = n } } ,,{ Add = \(e1: Eval) -> \(e2: Eval) -> \(self: Top) -> { eval = e1.eval + e2.eval } }in ...

The term declaration is elaborated to a let expression and the trait expressions are elaborated to functions. Since no self-types are speciﬁed, the self parameters are all of typeTop.

Elaborating child dependencies.

printChild also implements ExpSig but contains childdependencies:
printChild = trait implements ExpSig<Eval % Print> => {(Litn).print = n.toString;(Add e1 e2).print = if e2.eval == 0 then e1.print
else "(" ++ e1.print ++ "+" ++ e2.print ++ ")";};

This trait is desugared to:printChild = trait [self: Top] implements ExpSig (Eval&Print) Print => open self in{ Lit (n: Double) = trait => { print = n.toString } } ,,{ Add (e1: Eval&Print) (e2: Eval&Print) ={ print = if e2.eval == 0 then e1.print
else "(" ++ e1.print ++ "+" ++ e2.print ++ ")" } };Since printChild instantiates the sort as <Eval % Print>, Exp and OExp are respectively
instantiated as Eval&Print (i.e. the intersection of the two types in <Eval % Print>) and
Print (i.e. the second type in <Eval % Print>). Through some type inference, the argumentswith expression types in the constructors (such as e1 and e2) are of type Eval&Print. This114

5.4 Formalization

enables eval to be called on e1 and e2 without an implementation of that operation in thecurrent trait. In short, printChild is elaborated to an expression similar to evalNum exceptthat expressions arguments are of diﬀerent type (Eval&Print) from the output. Importantly,
for the modular deﬁnition of printChild to work it is crucial to use diﬀerent types in the
instantiation of Exp for input (negative) positions and output (positive) positions.
Elaborating self-type annotations

On the other hand, traits with explicit self-type

annotations are desugared diﬀerently. For instance, expAdd:
expAdd Exp = trait [self : ExpSig<Exp>] => {test = new Add (new Lit 4) (new Lit 8);};

is desugared to:
expAdd = /\Exp. trait [self: ExpSig Exp Exp] => open self in {test = new Add (new Lit 4) (new Lit 8);};

The original trait body is wrapped into an open expression so that the constructors/methods exposed by the self-type are directly in scope. Further elaboration into Fi+ resultsin:
let expAdd =/\Exp. \(self : { Lit : Int -> Exp -> Exp } & { Add : Exp -> Exp -> Exp -> Exp }) ->let Add = self.Add
in let Lit = self.Lit
in { test = letrec self : Exp = Add (letrec self : Exp = Lit 4 self in self)(letrec self : Exp = Lit 8 self in self)
self
in self }in ...

The type alias used in specifying the self-type is expanded. The type translation rewrites
trait types as function types. The open self expression is elaborated into a series of letbindings, one for each record label. The new expressions are elaborated to lazy ﬁxedpoint of self, following Cook and Palsberg denotational model of inheritance [Cook andPalsberg, 1989].
Elaborating inheritance and overriding

We now show the elaboration of a trait that

additionally uses inherits and override. An instance is expMul:
expMul Exp = trait [self : MulSig<Exp>] inherits expAdd @Exp => {override test = new Mul super.test (new Lit 4);};

which is elaborated as:
let expMul = /\ Exp. \(self : { Lit : Int -> Exp -> Exp } &{ Add : Exp -> Exp -> Exp -> Exp } &{ Mul : Exp -> Exp -> Exp -> Exp }) ->let super = (expAdd Exp) selfin (super : Top) ,,let Add = self.Add

115

Chapter 5. Compositional Programmingin let Lit = self.Lit
in let Mul = self.Mul
in { test = letrec self : Exp = Mul super.test(letrec self : Exp = Lit 4 self in self)
self
in self }in ...

The inherited trait expression (expAdd) is elaborated as a function and applied to the selfreference and the result is bound to super. Then super is merged with the body of thetrait. Since test is overridden, it should be excluded from super; otherwise a conﬂict will
occur when merging super with the body. The exclusion of test from super is done byinjecting a type annotation Top.

Elaborating nested composition of traits

Finally, we show how the merge of traits iselaborated using (new evalNum ,, expAdd @Eval).test.eval as an example:
letrec oself : { Lit : Int -> Top -> {eval : Int} } &{ Add : {eval : Int} -> {eval : Int} -> Top -> {eval : Int} } &{ test : {eval : Int} } =\(iself : { Lit : Int -> {eval : Int} -> {eval : Int} } &{ Add : {eval : Int} -> {eval : Int} -> {eval : Int} -> {eval : Int} })-> (evalNum iself) ,, ((expAdd {eval : Int} iself) oself)in oself.test.eval

where we have renamed the outer and inner self as oself and iself for distinction.
evalNum ,, expAdd @Eval is elaborated to a function that returns the merge of respec-tively applying evalNum and expAdd @Eval, which are already elaborated as functions, toiself.

Since the merged trait is of type Trait[ExpSig<Eval>,ExpSig<Eval>&{test:Eval}],iself has the elaborated type of ExpSig<Eval> while oself has the elaborated type of
ExpSig<Eval>&{test:Eval}.

5.4.3 Static Semantics
Overview

Figure 5.2 gives an overview of all the relations involved in the elaboration,

where → denotes the transformation ﬂow and d denotes the dependencies between therelations. Firstly, patterns and multi-ﬁeld records are desugared by J·K. Then the program
P is elaborated into a Fi+ expression e by ∆; Γ ` P ⇒ A ; e. During the elaboration process, some transformations and checks are performed: type synonyms and compositional
interfaces X hSi are expanded by ∆, Σ ` A ⇒ B; the right-hand side of a type declaration istransformed by Σ `cP A ⇒ B for distinguishing positive and negative occurrences of sorts;
the subtyping relation between two types is checked by A <: B; the disjointness of two
types is ensured by ∆ ` A ∗ B. Both subtyping and disjointness checking rely on top-liketypes (eAd). The latter further relies on disjointness axioms (A ∗ax B).
116

5.4 Formalization

Σ `cp A ⇒ B∆, Σ ` A ⇒ B
Type expansion

J·K
Desugaring

A ∗ax B
Disjointness axioms|·|

Type translation
Sort transformation
i S
S
6
O
l
l
S S
ll
S S

l
l

/

∆; Γ ` P ⇒ A ; e
Type-directed elaboration

o_ _ _ _ _ _∆`A∗BDisjointness_ _ _ _/

_ _ _ _ _ _/A <: B
Subtyping




eAd
Top-like types

Figure 5.2: The relation of relations.

Desugaring

The core desugaring rules are:J{M1 , . . . , Mn }K= J{M1 }K , , . . . , , J{Mn }K
def
J(L x : A [self : B]).l = E K=
L = λx : A.Jtrait[self : B] implements > inherits > => {l = E }K
def
Jtrait[self : A] implements B inherits E1 => E2 K=
trait[self : A] implements B inherits JE1 K => open self in JE2 K
def

The ﬁrst rule desugars a multi-ﬁeld record into an intersection of singleton records. The
second rule desugars a method pattern into an ordinary variable binding to a functionthat returns a single-ﬁeld trait. The last rule implicitly opens self for the trait body sothat members declared by the self-type can be directly accessed without preﬁxing self.

Type checking

Figure 5.3 shows the selected typing rules. The gray parts could be

ignored for the moment. They will be discussed later in Section 5.4.4. The type systemof CP is bidirectional: under the contexts ∆ and Γ, the inference mode (⇒) synthesizes
a type A while the checking mode (⇐) checks against A. A lot of the rules are presented
previously in the literature [Bi et al., 2019], thus we discuss only the novel ones, whichmostly relate to traits and declarations.
The rule T-tyDecl adds a type alias to the type context ∆. The right-hand side, typeA, is expanded and transformed before it is added to ∆ for type-checking the remaining
program. Each sort α has a fresh companion variable(corresponding to OExp in theexamples) for distinguishing negative and positive occurrences of sorts.

The rule T-

tmDecl deals with term declarations. The bound expression, E, is inferred with a type A.
Then, x of type A is added to the term context Γ for type-checking the remaining program.
The rule T-trait is the most complicated one since multiple types and expressions
are involved and several validity checks are performed. It ﬁrstly expands the self type A
and the type to implement B as A1 and B1 . Then self is added to Γ in type-checking theinherited expression E1 and the body E2 . The inherited expression E1 is valid only when it
is of a trait type Trait[A2 , B2 ] and the requirement A2 is met by A1 . After validity checking,super can be added to Γ in type-checking the body E2 . The type of body should be disjoint

117Chapter 5. Compositional Programming∆; Γ ` P ⇒ A ; e
T-tyDecl

fresh∆; α 7→` B ⇒ B1

α 7→

`+

false

∆; α 7→ ` A ⇒ A1
B1 ⇒ B2∆, X hα, i 7→ A1 & B2 ; Γ ` P ⇒ C ; e∆; Γ ` type X hα i extends A = B; P ⇒ C ; e
T-tmDecl∆; Γ ` E ⇒ A ; e1∆; Γ, x : A ` P ⇒ B ; e2∆; Γ ` x = E ; P ⇒ B ; let x : |A| = e1 in e2∆; Γ ` E ⇒ A ; e (Inference)
T-trait∆; • ` A ⇒ A1∆; • ` B ⇒ B1∆; Γ, self : A1 ` E1 ⇒ Trait[A2 , B2 ] ; e1
A1 <: A2∆; Γ, self : A1 , super : B2 ` E2 ⇒ C ; e2
C ∗ B2
C & B2 <: B1∆; Γ ` trait[self : A] implements B inherits E1 => E2 ⇒ Trait[A1 , C & B2 ]; λ(self : |A1 |). let super = e1 self in e2 , , superT-mergeTrait∆; Γ ` E1 ⇒ Trait[A1 , B1 ] ; e1∆; Γ ` E2 ⇒ Trait[A2 , B2 ] ; e2

∆ ` B1 ∗ B2∆; Γ ` E1 , , E2 ⇒ Trait[A1 & A2 , B1 & B2 ] ; λ(self : |A1 & A2 |). e1 self , , e2 self
T-new∆; Γ ` E ⇒ Trait[A, B] ; e

B <: A∆; Γ ` new E ⇒ B ; let self : |B| = e self in self
T-open∆; Γ ` E1 ⇒ {li : Ai } ; e1∆; Γ, li : Ai ` E2 ⇒ e2 ; B∆; Γ ` open E1 in E2 ⇒ B ; let li : |A| = e1 .li in e2
Figure 5.3: Selected typing rules.

to the type of the inherited expression E1 (C ∗ B2 ). Meanwhile the body and the inheritedexpression should together implement what B1 speciﬁes (C & B2 <: B1 ). The rule T-new
instantiates a trait. The expression E should be of type Trait[A, B] and the requirement
A should be met by B. The rule T-open collects the record types from the inferred typeof E1 and adds every label type pair to the term context in inferring E2 . Besides a rulefor ordinary merges, T-mergeTrait is a novel rule specially for the merge of two traits.
T-mergeTrait says that if the two expressions are of type Trait[A1 , B1 ] and Trait[A2 , B2 ]and B1 and B2 are disjoint, then the merged expression is of type Trait[A1 & A2 , B1 & B2 ].
Inferring the merged traits as a trait type rather than an intersection type brings several
advantages, which will be discussed in Chapter 6.

Sort transformation

Figure 5.4 shows the sort transformation, which is a phase that

replaces positive occurrences of a sort appearing in a type declaration with some othertype. Sort transformation is illustrated in the previous section by elaborating the righthand side of ExpSig and MulSig. From the names we cannot distinguish sorts from type118

5.4 Formalization

Σ `cp A ⇒ B
TR-rcdTR-positive

TR-ctrPositiveΣ `falseα⇒+Σ `trueα ⇒ Trait[α, ]
+

α 7→

∈Σ

α 7→

TR-arrΣ `cﬂip(p) A ⇒ A1(l )
Σ `isCapitalized
A⇒B
p

∈Σ

Σ `cp {l : A} ⇒ {l : B}

TR-traitΣ `cp B ⇒ B1

Σ `cﬂip(p) A ⇒ A1

Σ `cp A → B ⇒ A1 → B1

Σ `cp B ⇒ B1

Σ `cp Trait[A, B] ⇒ Trait[A1 , B1 ]∆, Σ ` A ⇒ B
E-tvarα∗A ∈∆

E-sig

X hα, i 7→ C ∈ ∆∆, Σ ` α ⇒ αΣ ` S ⇒ hA, Bi∆, Σ ` X hSi ⇒ [A/α, B/ ]CΣ ` S ⇒ hA, Bi
E-sort1Sort

α 7→

∈Σ

E-sort1

E-sort2Σ ` A ⇒ hA, AiΣ ` α ⇒ hα, iΣ ` A%B ⇒ hA & B, Bi

Figure 5.4: Selected sort transformation and type expansion rules.

parameters and therefore a sort context Σ is needed. How a sort is transformed is further
determined by two conditions: p and c. The condition p tracks the positions where sorts
appear: + indicates a positive position and − indicates a negative position. Positive andnegative positions are treated diﬀerently: negative positions are kept unchanged while
positive positions are transformed. Speciﬁcally, for a function type A → B, p is ﬂippedin processing A but unchanged in processing B (rule TR-arr). The same applies for
Trait[A, B] (rule TR-trait). The boolean value c bookmarks whether the type appears

inside a constructor. By default false, c is set to be true when the label of a record iscapitalized (rule TR-rcd). According to p and c, rules TR-positive and TR-ctrPositivetransform the sort α toType expansion

and Trait[α, ] respectively.

Type expansion plays two roles: eliminating type aliases and ensuring

the well-formedness of the types. It is used, for example, when elaborating the extends
clause on MulSig and implements clauses on evalNum and printChild. Figure 5.4 also shows
the relevant type expansion rules. The rule E-tvar checks whether the type variable isin the type context. The rule E-sig eliminates X hSi by looking up X in the type context ∆and substituting the negative and positive occurrences of sorts (α and ). An instantiatedsort S is expanded to a pair of types for substituting α andcases for expanding S: if S is a sort, then α andhas only one type, both α andrespectively. There are three

are kept abstract (E-sort1Sort); if S

are substituted by that type (E-sort1); otherwise, α andare substituted by the intersection of the two types (A & B) and the second type (B) fromthe pair (E-sort2).
119Chapter 5. Compositional Programming

A <: B
S-trans

S-reflA <: BA <: AA <:eBd

A <: C

S-rcd

S-bot

S-topLike

B <: CA <: B

A & B <: B

S-arr

A <: CB1 <: A1A2 <: B2

A1 → A2 <: B1 → B2

A <: B & C

S-forallA & B <: A{l : A} <: {l : B}

S-and

S-andrS-andlA <: B⊥ <: A

S-trait

B1 <: B2A2 <: A1A2 <: A1∀(α ∗ A1 ).B1 <: ∀(α ∗ A2 ).B2
S-distArr(A → B) & (A → C) <: A → B & C
S-distRcd

B1 <: B2

Trait[A1 , B1 ] <: Trait[A2 , B2 ]S-distTrait
Trait[A, B] & Trait[A, C] <: Trait[A, B & C]
S-distAll{l : A} & {l : B} <: {l : A & B}∀(α ∗ A).B & ∀(α ∗ A).C <: ∀(α ∗ A).B & C
Figure 5.5: Subtyping.

eAd
TL-tope>dTL-andeAdeB deA & B d

TL-arr

TL-rcdTL-all

TL-traiteA → Bde{l : A}de∀(α ∗ A).Bd

eTrait[A, B]d

eBd

eA d

eBd

eB d

Figure 5.6: Top-like types

Subtyping

Figure 5.5 shows the subtyping rules.

Most rules come from multiple

sources in previous work [Barendregt et al., 1983; Bi and Oliveira, 2018; Bi et al., 2018,2019]. The main novelty is the rule that deals with distributivity of trait types (rules
S-distTrait),which essentially follows the TL-arr rule for functions (which is inspired by
BCD subtyping Barendregt et al. [1983]). The rule S-distTrait distributes intersections
over the Trait type constructor. In the original work on ﬁrst-class traits, the elaboration
targeted the Fi calculus, which is a precursor of Fi+ without distributivity rules. Thus,
that work did not have distributivity for traits. The novel distributivity rules for traits are
essential for achieving the nested composition of traits (and trait families). The trait rules
and other distributive rules allow, for example, ExpSig<Eval>&ExpSig<Print> to be subtypeof ExpSig<Eval&Print>. The rule S-topLike is also novel. It states that any type is a subtypeof a top-like type.

Top-like types

As discussed in Section 2.5, top-like types are types isomorphic to >(i.e. both sub- and supertypes of >). We add a new rule TL-trait for trait types, whichstates that a trait type is top-like when its provided interface is top-like.
120

5.4 Formalization

Disjointness

The disjointness judgment detects the conﬂicts when merging two expres-sions of type A and B. These rules are omitted here since they are merely a combinationof the rules from Fi+ (which can be found in Figure 2.4) and SEDEL. Interested readers
can refer to Appendix A for the full disjointness rules of CP.

5.4.4 Formal Elaboration
The dynamic semantics of CP is given by an elaboration to Fi+ (cf. Section 2.5), which is
our target language. The semantics and metatheory (including type-safety and coherence)of Fi+ have been studied in previous work [Bi et al., 2019]. The semantics of Fi+ is given byelaborating to Fco , a variant of System F extended with products and explicit coercions.
In the original paper by Bi et al. [2019], Fco has a call-by-value semantics. Since Fi+ is
deﬁne by elaboration to Fco it inherits the call-by-value semantics of Fco . Here we assume
a call-by-name variant Fco , which we expect to be type-sound. We expect this result to bestraightforward since Fco is just a minor variant of System F and System F is known to be
type-sound in both call-by-value and call-by-name. The translations from CP to Fi+ andthen to Fco are unaﬀected by the choice of evaluation strategy in Fco , and simply inherit
the evaluation strategy from Fco . As we have mentioned earlier, we also assume lazyrecursive let bindings, which are not present in Fi+ . Lazy recursive let bindings are in fact
the main motivation to switch to a call-by-name semantics, since they are necessary forthe encodings of self references. Although we expect the proof of type-soundness of callby-name Fco and coherence of call-by-name Fi+ to easily hold, we leave such validation
for future work.
Fi+ is a subset of CP excluding declarations and trait related constructs. Therefore,
elaborating the shared constructs is straightforward, and only elaborating constructs speciﬁc to CP requires some explanations. Let us focus on the the elaborated Fi+ expressions( gray parts) shown in Figure 5.3. Intuitively, T-tmDecl elaborates a term declaration
into a let expression, as illustrated by the elaborations on evalNum, printChild and expAdd
in Section 5.4.2. The gray parts of T-trait and T-new are inherited from SEDEL, which
follows Cook and Palsberg [1989]’s denotational semantics of inheritance. Concretely, a
trait expression is elaborated to a function that takes self as an argument, binds the application result of e1 self to super and returns a merge of the body (e2 ) and super. T-trait isalso illustrated by the elaborations on evalNum, printChild and expAdd. As shown by expAdd,
T-new elaborates the expression into a lazy ﬁxed-point of self. T-mergeTrait elaborates
the merge of two traits into a function that takes self as an argument and returns themerge of applying the two elaborated traits e1 and e2 to self. Note that the type of the selfargument for the merged trait is an intersection of the two self types from the two traits
being merged. Elaboration on evalNum ,, expAdd @Eval illustrates T-mergeTrait. T-open
elaborates an open expression into a sequence of let expressions, one for each ﬁeld fromthe record e2 . These let expressions bind labels to their projections so that e2 can directly
access all the ﬁelds from e1 without explicit projections. The elaboration on expAdd is a
showcase of T-open.
Finally note that CP’s types also need to be translated to Fi+ ’s types, because in Fi+
121

Chapter 5. Compositional Programming

there is no Trait[A, B] construct. All types, except Trait[A, B] have straightforward translations. Trait types are translated to function types in Fi+ (following SEDEL) with the rule

|Trait[A, B]| = |A| → |B|.
5.4.5 Type Safety and Coherence
The elaboration of CP into Fi+ is type-safe and coherent. We summarize the key results
here. Detailed proofs and other auxiliary lemmas can be found in Appendix B.
The ﬁrst result is that elaboration is type-safe. To prove this result we need a few
results for some of the auxiliary relations, which are shown next.
Lemma 5.1 [Well-formedness preservation] If ∆, Σ ` A ⇒ B then |∆| ` |B|.

□

Proof. By induction on the derivation of the judgment.
Lemma 5.2 [Disjointness axiom preservation] If A ∗ax B then |A| ∗ax |B|.
Proof. Note that |Trait[A, B]| = |A| → |B|; the rest are immediate.

□

Lemma 5.3 [Subtyping preservation] If A <: B then |A| <: |B|.

□

Proof. By structural induction on the subtyping judgment.
Lemma 5.4 [Disjointness preservation] If ∆ ` A ∗ B then |∆| ` |A| ∗ |B|.

□

Proof. By structural induction on the disjointness judgment.
Then the main type-safety theorem is:
Theorem 5.1 [Type-safety] We have that:• If ∆; Γ ` P ⇒ A ; e then |∆|; |Γ| ` e ⇒ |A|.

□

Proof. By structural induction on the typing judgment.
The second theorem is the coherence of the elaboration:
Theorem 5.2 [Coherence] Each well-typed CP program has a unique elaboration.

Proof. For each elaboration rule, the elaborated Fi+ expression in the conclusion is uniquelydetermined by the elaborated Fi+ expressions in the premises. By the coherence propertyof Fi+ , we conclude that each well-typed CP program has a unique elaboration. Therefore□

CP is coherent.
Additional Properties

There are more properties proved in the Fi+ paper, including thedecidability of the type system. These properties should easily hold for CP by extending
the proofs with a case for trait types. The cases for trait types are essentially similar
to the cases of function types (trait types are actually encoded as function types in theelaboration to Fi+ ), so the proof extensions should be straightforward.

One thing to

notice is that the subtyping relation presented in this paper is non-algorithmic due to theexistence of a transitive rule (S-trans). An algorithmic variant of the subtyping relationis shown by Bi et al. [2019]. For proving decidability, we would need to use an extended
version of such algorithmic subtyping with trait types.
122

5.5 Case Studies

5.5 Case Studies
To further demonstrate the applicability of CP, we conducted three case studies. The
ﬁrst case study is Scans, a small DSL for describing parallel circuits originally proposedby Hinze [2004] and further studied by Gibbons and Wu [2014]. The second one is a miniinterpreter, which integrates and extends the examples in Section 5.2 and Section 5.3.
The last case study is an implementation of the C0 compiler, inspired by the work ofRendel et al. [2014], which compiles a subset of C to JVM instructions. In all of the threecase studies, the need for dealing with extensibility and complex dependencies arises.

5.5.1 Scans
Recall Scans introduced in Section 3.2.1. It is an interesting case study because
implementing Scans modularly requires an approach not only to solving the EP but also
capable of expressing dependencies. Besides the ones shown in Chapter 3, there are
already a few implementations written in diﬀerent languages [Gibbons and Wu, 2014;
Zhang and Oliveira, 2019; Bi et al., 2019]. Still, these implementations are not fully
satisfying. We compare our implementation with respect to these implementations. Note,however, that there are no dependencies on self-references in this case study. Thus thiscase study does not exercise such a form of dependencies.

Scans in CP

The techniques shown in Section 5.2 are used in modularizing Scans with

CP. The syntax of Scans is captured by a compositional interface:type CircuitSig<Circuit> = {Identity : Int -> Circuit;
Fan: Int -> Circuit;

Above: Circuit -> Circuit -> Circuit;

Beside: Circuit -> Circuit -> Circuit;

Stretch: (List Int) -> Circuit -> Circuit;};

Operations are modeled as trait families that concretely implement the compositional
interface. For example, the implementation of wellSized is given below:
type WellSized = { wS : Bool };wellSized = trait implements CircuitSig<Width,WellSized> => {(Identityn).wS = true;(Fann).wS = true;(Abovec1 c2).wS = c1.wS && c2.wS && c1.width == c2.width;(Beside c1 c2).wS = c1.wS && c2.wS;(Stretch ns c).wS = c.wS && length ns == c.width;};

The trait family wellSized implements CircuitSig<Width%WellSized>, indicating that it depends on another trait family of CircuitSig that implements Width. As discussed in Section 5.2.2, such dependency is weak and allows us to, for example, call width on c inStretch.

123

Chapter 5. Compositional Programming

New variants

The new constructor RStretch is added by extending the compositional

interface:type NCircuitSig<Circuit> extends CircuitSig<Circuit> = {RStretch : (List Int) -> Circuit -> Circuit};

Accordingly, existing trait families are extended with a case for RStretch, for example:
nWellSized = trait implements NCircuitSig<Width,WellSized> inherits wellSized => {(RStretch ns c).wS = c.wS && length ns == c.width;};

Similarly, nWidth, nDepth, and nLayout extend their respective trait families.
Finally, we obtain the full implementation of Scans by composing all the operations
as well as a generic trait that constructs a modular circuit:scans = new nWidth,, nDepth ,, nWellSized ,, nLayout ,,circuit @(Width & Depth & WellSized & Layout);

Scans in Haskell

Gibbons and Wu [2014] implement Scans as a shallow EDSL in Haskell(cf. Section 3.2 and Section 3.3). Variants of Scans are modeled as functions, thus adding
new variants is simple through deﬁning new functions. However, multiple (possibly dependent) operations are deﬁned using tuples. Such an implementation is not modular
because whenever a new operation is needed, existing code has to be modiﬁed for accommodating new operations. A follow-up Haskell implementation (cf. Section 3.4) employsa technique commonly known as Finally Tagless Carette et al. [2009], together with a
type-class based encoding of subtyping on tuples, to modularize operation extensions.
In essence, such an approach simulates subtyping and inheritance for enabling modular
composition of the operations. However, this comes at the cost of boilerplate code andextra complexity due to additional parametrization that is needed to make the encoding
work. Moreover, explicit delegations are required for deﬁning dependent operations in the
Haskell approach, making the code cumbersome to write. In contrast, CP avoids thoseissues by having built-in language support for nested composition, and compositional
interfaces/traits and method patterns make the code quite easy to write.
Scans in Scala

Chapter 3 presents a modular solution in Scala by combining the ex-tensible Interpreter pattern [Wang and Oliveira, 2016] and Object Algebras [Oliveira andCook, 2012]. Scans is modeled by a hierarchy of traits, where the root is an interface
describing the operations a circuit supports while other traits concretely implement thatinterface. Adding new operations is done by deﬁning another trait hierarchy that implements the extended interface and inherits existing traits. Through covariantly reﬁning thecircuit ﬁelds to the type of the extended interface, previously deﬁned operations can beused as dependencies. Although such an implementation is modular, the extensions arelinearly added, and the dependencies are strong due to the use of inheritance to express
dependencies. Alternatively, the new trait hierarchy can be separately deﬁned without
inheriting existing ones. This weakens the dependencies but requires some boilerplate forgluing the hierarchies using mixin composition. Unlike CP where the composition is done
124

5.5 Case Studies

once in the family level, every trait in Scala needs to be composed because Scala lacks
support for nested composition. Another drawback is that Scala’s constructors are notvirtual. Directly calling new on constructors for creating objects results in non-modular
code. Therefore, Object Algebras [Oliveira and Cook, 2012] are used for abstracting overthe constructor calls, resulting in more boilerplate.
Scans in Fi+

Bi et al. [2019] modularize Scans directly in Fi+ using extensible records.

Note that, because Scans do not have self-reference dependencies, there is no strict needfor using traits, which are not directly supported in Fi+ . The syntax of Scans is deﬁned
similarly to CircuitSig except that Circuit is captured as a type parameter rather than asort. The operation extensibility is achieved by deﬁning new record instances while thevariant extensibility is achieved by the intersection types and the merge operator. A key
diﬀerence is how dependent operations are deﬁned:
wellSized = {identity (n : Int) = { wS = true },fan(n : Int) = { wS = true },above (c1 : WellSized&Width) (c2 : WellSized&Width) ={ wS = c1.wS && c2.wS && c1.width == c2.width },beside (c1 : WellSized) (c2 : WellSized) = { wS

= c1.wS && c2.wS },stretch (ns : List Int) (c : WellSized&Width) ={ wS = c.wS && length ns == c.width }};Note that wellSized is not given a type. Instead, it deﬁnes a record with various ﬁelds
modeling constructors, and all the “constructor” arguments are explicitly annotated. Thisillustrates a crucial diﬀerence to the CP solution: while in CP wellSized (and other operations) implement a proper interface (via implements), in the Fi+ encoding that is not
the case. The dependency on width is loosely expressed by reﬁning the circuit type as
WellSized&Width. Such dependency is repeated several times in above and stretch. The

lack of a proper interface when implementing operations allows for a relatively compact
solution in Fi+ , but it has some important disadvantages. By implementing an interface
in CP, we can ensure various things at the point of the operation deﬁnition. For instance,
CP checks that: we implement all constructors, and all the constructors are deﬁned withthe right number of parameters and types for the parameters. In Fi+ such checks are notdone when deﬁning operations, which is very error-prone. Errors like forgetting to implement a constructor or implementing a constructor with the wrong number of parametersor the wrong parameter types cannot be checked at the deﬁnition point, but are delayed
to the composition point.
Evaluation

Besides a qualitative analysis of the aforementioned modular implementa-

tions, we further evaluate them in terms of source lines of code (SLOC). To make thecomparison fair, we have adapted their implementations to ensure that all the implementations are written in a similar programming style and provide the same functionalities.
The SLOC for the modular implementations in Haskell, Scala, Fi+ and CP are 87, 129, 72
and 70 respectively. CP’s solution is the most compact one among the four implementa125

Chapter 5. Compositional Programming

Table 5.1: Diﬀerent kinds of dependencies used in the mini interpreter.
Operation
Dependency
Child dependencies
Self dependencies
Mutual dependencies
Inherited attributes

eval

print

print(aux)

✓
✓
✓

✓

log

✓

tions, while also being the most modular one.

5.5.2 Mini Interpreter
Overview

The second case study is a mini interpreter for an expression language. Thiscase study is larger (around 700 SLOC) and more comprehensive than the previous one.
Besides the EP and simple dependencies, it covers more forms of dependencies. In particular, self dependencies occur in this case study. Furthermore, the case study contains
several uses of S-attributed grammars, polymorphic contexts, as well as multiple sorts.
The expression language consists of various sublanguages, including numeric andboolean literals, arithmetic expressions, logical expressions, comparisons, if-then-else
branches, variable bindings, function closures, and function applications. The supported
operations include a few variants of evaluation, pretty-printing, and logging. The sublanguages are separately deﬁned as features [Prehofer, 1997], using diﬀerent compositional
interfaces and trait families. Through nested composition, these features can be arbitrarily combined to form a product line of interpreters [Pohl et al., 2005].

Dependencies

The operations on the expression language contain non-trivial depen-

dencies. Table 5.1 summarizes the diﬀerent kinds of dependencies used in the miniinterpreter. With techniques shown in Section 5.2.2, these dependencies are expressedin a modular way. For example, log is a simple form of logging, which shows the printing and evaluation results of an expression for debugging purposes. Here is a simpliﬁed
logging implementation for numeric expressions:logNum = trait implements NumSig<Eval&Print % Log> => {(Add e1 e2 [self:Eval&Print]).log = self.print ++ " is " ++ self.eval.toString;-- other constructors are omitted};

To express self dependencies on eval and print, we annotate Add’s self-type as Eval&Print.

Polymorphic contexts

There are four diﬀerent kinds of contexts for evaluation in total:

an empty context (Top); a map from strings to numbers for evaluating variable bindings;
a map for dynamic scoping; an environment for intrinsic functions. Using techniques
shown in Section 5.3.2, we model these contexts as polymorphic contexts to make thecode with contexts modular and evolvable.
1265.5 Case Studies

Multi-sorted languages

Examples presented in the chapter so far are all based on asingle-sorted expression language. Essentially, CP allows compositional interfaces to beparameterized by multiple sorts. This ability is demonstrated by the following code excerpt
extracted from this case study:
type CmpSig<Boolean,Numeric> = {Eq: Numeric -> Numeric -> Boolean;Cmp : Numeric -> Numeric -> Numeric;-- other constructors are omitted};
CmpSig models equality and three-way comparison (also known as the spaceship operator),

which returns 0 if the two operands are equal, 1 if the left operand is greater, or -1 otherwise. They take Numeric arguments and construct Boolean and Numeric traits respectively.
Notice that CmpSig is developed as an independent feature. It can later be combined withother independently developed features such as numeric expressions:
expCmp N B = trait [self : NumSig<N>&CmpSig<B,N>] => {cmp = new Cmp (new Add (new Lit 1) (new Lit 2)) (new Lit 3);};

In cmp, we construct an expression with the new constructor Cmp, as well as Lit and Add
which are independently developed before.

5.5.3 C0 Compiler
Overview

The C0 compiler was originally an educational one-pass compiler developed

for the compilation course at Aarhus University. Rendel et al. [2014] translated thiscompiler into their encoding of Object Algebras, whereas we will present this case study
in our approach of Compositional Programming. Then we will make a comparison with
the implementation by Rendel et al. [2014], as well as the original non-modular version.
C0 selected a subset of the C programming language, consisting of only integer types,
arithmetic, bitwise, and comparison operations, a few control ﬂow statements, functions,and basic I/O. In other words, it is also a multi-sorted language, whose interfaces areparameterized by Program, Function, Statement, Expression, etc. A C0 program consists offunction declarations and deﬁnitions, which will be compiled into Java bytecode. The
original implementation was written in Java and later reimplemented in Scala using Object Algebras by Rendel et al. [2014]. Both implementations include a bytecode generator
from AST nodes to JVM instructions as well as a recursive descent parser. Since the CP
language is currently a research prototype and does not support I/O or complex string
manipulation, we eliminate the parsing phase from this case study. Lexical analysis is
not the core part of C0 and will not aﬀect the validity of our evaluation.
Chained attributes

We have shown various forms of dependencies in terms of S-

attributed grammars in Section 5.2.2 and introduced polymorphic contexts to tackle Lattributed grammars in Section 5.3.3. However, there are still other kinds of dependencies
in attribute grammars. In the C0 compiler, an attribute can depend on both its children
and its parent or siblings. For example, consider an attribute that counts the number of127Chapter 5. Compositional Programming

leaf nodes (terminal symbols) occurring to the left or in the subtree of the current node.
The attribute equations together with the production rules of Lit and Add are listed below:
E0 → E1 "+" E2
E→n

{Lit}

E.counts = E.counti + 1

(5.1)

{Add}

E1 .counti = E0 .counti(5.2)

E2 .counti = E1 .counts(5.3)

E0 .counts = E2 .counts(5.4)

Note that count has two roles: the subscript i stands for inherited attributes while s stands
for synthesized ones.
For terminal symbols, we just add one to the inherited number, as Equation 5.1
shows. For nonterminal symbols, there are three attribute evaluation rules: Equation 5.2
is a trivial inherited attribute depending on its parent; Equation 5.3 is more interesting
because it depends on its left sibling; Equation 5.4 is a trivial synthesized attributedepending on its child. These three equations compose a traversal of the syntax tree:
E1 .counti inherits from its parent E0 .counti and then does its own computation on the left
subtree to obtain E1 .counts ; E2 .counti inherits from its left sibling and then traverses theright subtree to obtain E2 .counts ; ﬁnally E0 .counts synthesizes the attribute from its child.
Such a tree traversal reveals an interesting class of attributes called chained attributes [Kastens and Waite, 1994]. A chained attribute is both inherited and synthesized. If we regard inherited attributes as Reader Monads and synthesized as Writer,then chained attributes correspond to State Monads.
In this case study, HasVariables and HasFunctions are chained attributes. They areused to do bookkeeping for variable and function declarations. Like inherited attributes,
we model them with polymorphic contexts, where the inherited part serves as the context
of the corresponding synthesized part:type HasVariables = { variables : Map };-- inherited

type ChainedVariables Ctx = { variables : HasVariables&Ctx -> Map };-- synthesized

If the chained attribute depends on other attributes, the context can be easily extended as
we described in Section 5.3.2. Here is an example of extending the context with HasOffset:
type ParamSig<Parameter> = { Param : String -> Parameter };parameterVariables (Ctx * (HasVariables&HasOffset)) =trait implements ParamSig<ChainedVariables (HasOffset&Ctx)> => {(Param id).variables (inh : HasOffset&HasVariables&Ctx) (name : String) =if name == idthen inh.offset
else inh.variables name;};

The constructor Param is used to declare a parameter within a function, and the trait
implements the chained attribute of the variable environment. (Param id).variables will
update the previous environment inh.variables with a new mapping from the given identiﬁer to the current oﬀset which works as the variable index. Note that, although thesetwo variables have the same name, the former is a chained attribute while the latter is128

5.6 Conclusion

Table 5.2: Source lines of code for the three implementations of the C0 compiler.
Java (Aarhus University)
Entangled Compiler(Tokenizer excluded)

SLOC
235

Bytecode (Reformatted)
Main
Total

25
14
274

Scala (Rendel et al. [2014])
Generic
Trees, Signatures and Combinators
Composition and Assembly
Attribute Interfaces
Algebra Implementations
Bytecode Prelude
Main
Total

SLOC
140
558
101
32
191
25
5
1,052

CP
Maybe Algebra
Compositional Interfaces

SLOC
12
32

Attribute Interfaces
Trait Implementations
Bytecode Prelude
Main Example
Total

8
216
25
21
314

an inherited attribute. Such a delegation forms a tree traversal to do bookkeeping forparameter declarations.

Comparision

The code statistics of the aforementioned three C0 implementations areshown in Table 5.2. The original Java implementation inlines semantic actions into thehandwritten parser. For the sake of fairness, lexical analysis related lines are not counted
and bytecode prelude are reformatted in the same style as the other two. Although theoriginal code is slightly shorter than CP, it is highly entangled and hinders modularityand extensibility.
To modularize the original C0 implementation, Rendel et al. [2014] use an extended
form of generalized Object Algebras to model attribute grammars in Scala. It allows Lattributed grammars with diﬀerent kinds of dependencies to be modularly deﬁned. Dueto the lack of the proper composition mechanisms in Scala, the attributes cannot beeasily composed, and specialized composition operators have to be explicitly deﬁned.
Such boilerplate code largely accounts for the SLOC reported in “Trees, Signatures andCombinators”. In addition, “Composition and Assembly” is the handwritten code to deal
with various dependencies. Their “Attribute Interfaces” and “Algebra Implementations”
are counterparts of our “Attribute Interfaces” and “Trait Implementations”.
Compared to CP, Rendel et al. [2014]’s approach is signiﬁcantly more verbose (about
3.5x SLOC). In CP, we do not need to write boilerplate code for trait composition and only
a few lines of “Compositional Interfaces” are necessary. A workaround they propose to
avoid such boilerplate code is to employ meta-programming for generating the specialized
signatures and combinators. However, their source code will not be type-checked before
macro expansion. Compilation errors will be reported in terms of generated code, which
could be confusing for programmers and make it hard to debug. Nevertheless, their assembly mechanism, which relies on specialized combinators that have to be handwritten,
encodes the pattern of chained attributes and simpliﬁes the algebra implementations. Our
approach of polymorphic contexts is a little more complicated than theirs, but does not
require any specialized combinators and works smoothly with nested trait composition.

5.6 Conclusion
We have presented key concepts of Compositional Programming together with a language design and implementation called CP. CP’s support for compositional interfaces,
129Chapter 5. Compositional Programmingcompositional traits, virtual constructors, and method patterns enables a programmingstyle that allows programs with non-trivial dependencies to be modularized. The applicability of CP is demonstrated by various examples and case studies. The calculus that
captures the essence of CP is proved to be type-safe.
We envision Compositional Programming as an alternative programming style to what
is currently oﬀered in both functional programming and object-oriented programming.
Compositional Programming borrows many ideas from both paradigms.

The style ofCompositional Programming presented in this chapter is essentially a purely functional
programming style, and draws inspiration from languages like Haskell. A purely functional style has beneﬁts in terms of reasoning about programs and it also simpliﬁes
some issues related to the composition of code. Multiple inheritance in the presence ofmutable state is a notorious problem, for instance. On the other hand, Compositional
Programming also borrows ideas from object-oriented programming, namely by employing
subtyping and nested composition (which is closely related to inheritance). This mix ofideas, together with some new ideas, results in a language that supports highly modular
programs in a natural way.

130

Chapter

6

Related Work

There is a large body of literature on modular extensibility. This chapter only covers
the closely related work.

6.1 Design Patterns for Modular Extensibility
This section discusses work on design patterns for solving the EP, which relates to all
the thesis.

Polymorphic variants

OCaml supports polymorphic variants [Garrigue, 1998]. Un-

like traditional variants, polymorphic variant constructors are deﬁned individually andare not tied to a particular datatype. Garrigue [Garrigue, 2000] presents a solution to
the Expression Problem using polymorphic variants. To correctly deal with recursive
calls, open recursion and an explicit ﬁxed-point operator must be used properly. Otherwise, the recursion may go to the original function rather than the extended one. This
causes additional work for the programmer, especially when the operation has complex
dependencies. In contrast, Castor handles open recursion easily through OO dynamic
dispatching, reducing the burden of programmers signiﬁcantly.

Data types à la carte (DTC)

DTC [Swierstra, 2008] encodes composable datatypes using

existing features of Haskell. The idea is to express extensible datatypes as a ﬁxpoint of
co-products of functors. While it is possible to deﬁne operations that have dependencies
or require nested pattern matching with DTC, the encoding becomes complicated andneeds signiﬁcant machinery. There is some follow-up work that tries to equip DTC withadditional power. Bahr and Hvitved [2011] extend DTC with GADTs [Xi et al., 2003] andautomatically generates boilerplate using Template Haskell [Sheard and Jones, 2002].
Oliveira et al. [2015] use list-of-functors instead of co-products to better simulate OOP
features including subtyping, inheritance, and overriding.

Algebraic signatures and Object Algebras

Readers familiar with algebraic signatures [Gut-tag and Horning, 1978] used in algebraic speciﬁcation languages may observe some similarities to compositional interfaces.

Algebraic signatures also allow the deﬁnition of

constructors. However, the semantics of constructors in compositional interfaces diﬀers
131

Chapter 6. Related Work

from those in conventional algebraic signatures. The key diﬀerence is that the positive
and negative occurrences of sorts are distinguished in CP, which is important to support an advanced form of modular dependencies but not well-supported with algebraic
signatures.
Oliveira and Cook [2012] explored an encoding of algebraic signatures in OOP languages, yielding a solution to the EP called Object Algebras.

Other design patterns,

such as Polymorphic Embeddings [Hofer et al., 2008] and Finally Tagless [Carette et al.,2009] use similar encodings. Such encodings originate from the work by Hinze [2006]
and Oliveira et al. [2006b]. Hinze [2006] showed how to model Church encodings of
GADTs [Cheney and Hinze, 2002] using Haskell type classes. Oliveira et al. [2006b] thenshowed that such type class based encoding can also solve the EP. Object Algebras use
parametric interfaces and classes to represent and implement the algebraic signatures,
respectively. As discussed in Section 2.4.1, Object Algebras, in their basic form, are hard
to be composed and express dependencies. These limitations are later addressed in Scalaby means of generalized Object Algebras and intersection types, together with specialized
combinators [Oliveira et al., 2013; Rendel et al., 2014]. However, the Scala approaches
based on reﬂection or meta-programming have important drawbacks. In contrast, CP has
built-in language support for sorts and nested composition, which is more convenient touse and avoids the use of reﬂection or meta-programming techniques.

6.2

Modular Pattern MatchingThis section discusses previous work on modular forms of pattern matching, which
relates to open pattern matching in Chapter 4 and method patterns in Chapter 5.
Open datatypes and open functions

To solve the Expression Problem, Löh and Hinze[2006] propose to extend Haskell with open datatypes and open functions. Diﬀerent fromclassic closed datatypes and closed functions, the open counterparts decentralize the definition of datatypes and functions and there is a mechanism that reassembles the pieces
into a complete deﬁnition. To avoid unanticipated captures caused by classic ﬁrst-ﬁt
pattern matching, a best-ﬁt scheme is proposed, which rearranges patterns according totheir speciﬁcness rather than the order (e.g. wildcards are least speciﬁc). However open
datatypes and open functions are not supported in standard Haskell and more importantly, they do not support separate compilation: all source ﬁles of variants belonging to
the same datatype must be available for code generation.
Object-Oriented pattern matching

There are many attempts to bring notions similar

to pattern matching into OOP. Multimethods [Chambers, 1992; Clifton et al., 2000] allow
a series of methods of the same signature to co-exist. The dispatching for these methods
additionally takes the runtime type of arguments into consideration so that the most
speciﬁc method is selected. Pattern matching on multiple arguments can be simulatedwith multimethods. However, it is unclear how to do deep patterns with multimethods.
Also, multimethods signiﬁcantly complicate the type system. As we have discussed in132

6.3 Language Designs for Modular Extensibility

Section 4.2, case classes in Scala [Odersky et al., 2004] provide an interesting blend
between algebraic datatypes and class hierarchies. Sealed case classes are very much
like classical algebraic datatypes, and facilitate exhaustiveness checking at the cost of aclosed (non-extensible) set of variants. Open case classes support pattern matching forclass hierarchies, which can modularly add new variants. However no exhaustivenesschecking is possible for open case classes. Besides case classes, extractors [Emir et al.,2007] are another alternative pattern matching mechanism in Scala. An extractor is acompanion object with a user-deﬁned unapply method that speciﬁes how to tear down that
object. Unlike case classes whose unapply method is automated and hidden, extractors
are ﬂexible, independent of classes but verbose.

There are also proposals to extend

mainstream languages with pattern matching such as Java. JMatch [Liu and Myers,
2003] extends Java with pattern matching using modal abstraction. JMatch methods
additionally have backward modes that can compute the arguments from a given result,serving as patterns.

Follow-up work [Isradisaikul and Myers, 2013] extends JMatchwith exhaustiveness and totality checking on patterns in the presence of subtyping andinheritance. However, it requires a non-trivial language design with the help of an SMT
solver. More recent OO languages like Newspeak [Geller et al., 2010] and Grace [Homeret al., 2012] are designed with ﬁrst-class pattern matching, where patterns are objects
and can easily be combined. To the best of our knowledge, none of these approaches fully
meet the desirable properties summarized in Section 4.2.1.

6.3 Language Designs for Modular Extensibility
This section discusses features designed for modular extensibility, which relates to
Chapter 4 and Chapter 5.
Mainstream statically typed OOP and virtual methods

Virtual methods in OOP pro-vide a way to weaken method dependencies. In a virtual method call, such as this.m() ina method of a class A, the call does not necessarily refer to the implementation of m() in
A. Instead, it may refer to a later implementation in a subclass of A. The choice of the im-

plementation of this.m() depends on which subclass of A is used to instantiate the object
on which this.m() is called. However, virtual methods alone are insuﬃcient to weaken
other kinds of dependencies. Most programming languages tend to have static references
to both constructors and types. For instance, if we refer to a constructor in Java, say
new A(), then the constructor (unlike the method this.m()) will always refer to the same

constructor of class A. Such static dependencies create a tight coupling between the useof the constructor and the class A, and make programs less modular than they ought tobe. Moreover, most statically typed OOP languages use (static) inheritance pervasively.
Inheritance often creates more coupling than needed between method implementations
in subclasses and method implementations in the superclass. In a subclass declaration,such as class A extends B {...}, B must be some concrete class, with (possibly) someconcrete method implementations. In other words, inheritance cannot be parametrized,
and we cannot program against the interface of the superclass: we must program against
133

Chapter 6. Related Work

some concrete class implementation.
CP adopts virtual methods, while also supporting virtual constructors to avoid staticreferences to constructors. Static references to types, which would typically arise fromconstructors, are avoided by using sorts. Moreover, in CP most uses of inheritance canbe replaced by code with weaker dependencies (see discussion in Section 5.2.2), thus
avoiding the coupling problems introduced by inheritance. Altogether this leads to codethat is more modular and has weaker dependencies than in conventional statically typed
OOP languages.
Mixins and traits

Single inheritance supported by many class-based OOP languages is

insuﬃcient for code reuse. On the other hand, multiple inheritance is hard to do correctly
due to the existence of the diamond problem. Mixins [Bracha and Cook, 1990] provide one
form of multiple inheritance. The Jigsaw framework [Bracha, 1992] formalizes mixins andprovides a set of operators on mixins. There are other formalizations of mixins proposedfor diﬀerent languages such as ML-like languages [Ancona and Zucca, 2002; Duggan
and Sourelis, 1996] and Java-like languages [Flatt et al., 1998; Lagorio et al., 2009].
Traits [Schärli et al., 2003] are an alternative to mixins. The fundamental diﬀerence
between traits and mixins is the way of dealing with conﬂicts when composing multiple
traits/mixins. Mixins implicitly resolve the conﬂicts according to the composition orderwhereas the programmer must explicitly resolve the conﬂicts for traits. The trait model
avoids unexpected errors caused by the wrong choice of implementation through implicit
resolution. Furthermore, it makes trait composition associative and commutative, andthe order of traits being composed does not aﬀect semantics (all permutations have thesame behavior). This is in contrast with mixins, where composition is order-sensitive.
Typically classes, mixins and traits in statically typed languages (such as Scala) aresecond-class and do not support dynamic inheritance. Dynamic languages like JavaScript
can encode quite general forms of mixins and support dynamic inheritance. However,type-checking dynamic inheritance is hard.

There is little work on typing ﬁrst-classclasses/mixins/traits. Takikawa et al. [2012]’s ﬁrst-class classes in Typed Racket, Leeet al. [2015]’s tagged objects and Bi and Oliveira [2018]’s ﬁrst-class traits are three notableworks, which support such features in statically typed languages. Our work follows theﬁrst-class traits model by Bi and Oliveira [2018] and traits in CP are ﬁrst-class, statically
typed and support dynamic inheritance.
First-class traits and disjoint intersection types

The theoretical foundation of CP isbuilt on calculi supporting disjoint intersection types [Oliveira et al., 2016]. In particular,
the semantics of CP is given by an elaboration to Fi+ [Bi et al., 2019]. The Fi+ calculus
is the most advanced calculus in the line of work of disjoint intersection types. The λi
calculus [Oliveira et al., 2016] was the ﬁrst calculus with disjoint intersection types, andaddressed the incoherence problem of intersection types with a merge operator [Dunﬁeld,
2014] by introducing the notion of disjointness. The Fi calculus [Alpuim et al., 2017]
extends λi with disjoint polymorphism. λi+ [Bi et al., 2018] extends λi with BCD-style
distributive subtyping, enabling nested composition. The Fi+ calculus [Bi et al., 2019]1346.3 Language Designs for Modular Extensibility

combines disjoint intersection types, disjoint polymorphism, and nested composition, enabling all the foundational ingredients for Compositional Programming. However Fi+ isstill a core calculus, and has no support for compositional interfaces, compositional traits,
method patterns and direct support for nested trait composition. These mechanisms, together with the mechanisms that enable weak dependencies, are all novel to the language
design of CP.
Built upon Fi , SEDEL [Bi and Oliveira, 2018] is a statically-typed language that supports ﬁrst-class traits and dynamic inheritance. The design of CP extends the design of
SEDEL, and in particular it extends the design of ﬁrst-class traits of SEDEL. However
SEDEL does not support Compositional Programming. In addition to the new features
specially designed for Compositional Programming, CP further takes the advantage ofthe unrestricted intersections brought by Fi+ in improving the trait model proposed by
SEDEL. In particular, the design of CP allows simpler typing rules compared to SEDEL.
For example the typing rule for deﬁning traits in SEDEL is:Γ, self : B ` Ei ⇒ Trait[Bi , Ci ] ; eiΓ, self : B, super : C1 & .. & Cn ` {lj = Ej0
B <: Bii ∈1..nΓ ` C1 & .. & Cn & CΓ ` trait[self : B] inherits Ei

i ∈1..nj∈1..m}⇒C;e

C1 & .. & Cn & C <: A

{lj = Ej0λ(self : |B|). let super = (ei self)i ∈1..nj∈1..mi ∈1..n} : A ⇒ Trait[B, A] ;in super , , eThis rule, among others, is clearly more complicated than its counterpart (T-trait) in CP.
The complexity is mainly caused by dealing with expression sequences: every expression
needs to be translated and validated. In contrast, CP processes only one expression
thanks to the newly introduced rule mergeTrait. mergeTrait checks the disjointness of
two traits and infers their merge as a trait type rather than an intersection type, thus
reducing the complexity and duplication. Another important beneﬁt of the CP design isimproved support for type inference. In SEDEL, a type must be provided to instantiate
a trait, but this type is inferred in CP. Moreover, parameters of a method pattern inside
a trait can omit types in CP if they are declared by the type speciﬁed in the implements
clause. This is quite handy for deﬁning trait families.

Virtual classes and family polymorphism

Ideas such as virtual classes [Madsen andMoller-Pedersen, 1989; Ernst et al., 2006] and family polymorphism [Ernst, 2001], extend
the idea of virtual methods to classes. Thus, classes and constructors can themselves bevirtual, weakening the dependencies to classes and constructors. Virtual classes were ﬁrst
introduced in the Beta language [Madsen et al., 1993]. Beta supports only single, static
inheritance. The composition of Beta programs is done through the fragment system[Knudsen et al., 1994]. The gbeta language [Ernst, 1999] extends Beta with mixins and,more importantly, supports family polymorphism [Ernst, 2001]. Family polymorphism isa powerful mechanism for extensible and composable software design, which can solve
the EP [Ernst, 2004]. Clarke et al. [2007] classify family polymorphism approaches into
object family approaches and class family approaches. In an object family approach,
135Chapter 6. Related Work

nested classes are attributes of objects of the family class. Some examples of object
family approaches are Beta, gbeta, CaesarJ [Aracic et al., 2006] and Newspeak [Bracha
et al., 2010]. Whereas in a class family approach, nested classes are attributes of thefamily class. Class family approaches include Concord [Jolly et al., 2004], .FJ [Igarashi
et al., 2005], Jx [Nystrom et al., 2004], J& [Nystrom et al., 2006] and Familia [Zhang andMyers, 2017]. There are also hybrid approaches like Tribe [Clarke et al., 2007]. Object
family approaches are typically more expressive but require a more complex dependent
type system, e.g. vc [Ernst et al., 2006]. The closest approach is Familia [Zhang andMyers, 2017], which also supports subtype polymorphism, family polymorphism, andparametric polymorphism but does not support the merge operator.
One diﬀerence between CP and the family polymorphism systems is that in those systems, inheritance is still used as a primary mechanism to express dependencies. Similarly
to (regular) classes, the use of inheritance in family polymorphism sometimes creates
more coupling than necessary between sub- and super-classes/families. In contrast,
such dependencies can be weakened via CP’s support for self-references and compositional interface type reﬁnement, leading to more modular programs. Another diﬀerenceis that conﬂicts are often implicitly resolved based on some order in those systems (e.g.,gbeta uses the composition order and Jx uses the dispatch order). In contrast, CP adopts
an approach where conﬂicts are explicitly resolved, following the trait model [Schärli et al.,2003].

ML modules and Scala

The design of CP is partly inspired by ML module systems [Mac-Queen, 1984]. Analogously to compositional interfaces and trait families in CP, ML signatures and structures can be used to specify and implement the constructors. However,unlike CP, ML modules are neither extensible nor ﬁrst-class. There are many proposals
to extend the ML modules with additional expressiveness, such as making modules ﬁrstclass [Russo, 2000] or recursive [Russo, 2001]. Together with other features, ML modules
can also be used in solving the EP [Nakata and Garrigue, 2006].
CP is also inﬂuenced by Scala [Odersky et al., 2004], where features such as intersection types, traits, and self-types are shared. Scala’s traits are not ﬁrst-class and do
not support dynamic inheritance and nested composition. Nevertheless, Scala supports
virtual types [Madsen and Moller-Pedersen, 1989; Igarashi and Pierce, 1999], which canbe used for simulating family polymorphism but in a much more verbose way [Odersky
and Zenger, 2005]. Sorts in CP are modeled in a way similar to type parameters. Another option is to use virtual types. The strengths and weaknesses of type parametersand virtual types are summarized by Bruce et al. [1998]: type parameters are ﬂexible
in composing and decomposing types while virtual types are good at specifying mutually
recursive families of classes whose relationships are preserved in the derived family. Typeparameters are a natural choice for CP since the underlying Fi+ calculus supports disjoint
polymorphism. In future work, we would like to explore a design with virtual types.
1366.4 Metaprogramming for Modular Extensibility

6.4 Metaprogramming for Modular Extensibility
This section discusses approaches that use metaprogramming to improve modularity,
which relates to Chapter 4 and Chapter 5.
SOP, MDSoC, AOP, and FOP

Subject-oriented programming (SOP) [Harrison and Os-

sher, 1993], multi-dimensional separation of concerns (MDSoC) [Tarr et al., 1999], aspectoriented programming (AOP) [Kiczales et al., 1997], and feature-oriented programming(FOP) [Prehofer, 1997] are software paradigms that share a similar vision of separationof concerns: i.e., separating a program into diﬀerent parts so that each part addresses a
separate concern. Since in those paradigms, a complete program has been separated, acomposition mechanism to assemble the parts back together is necessary. Typically SOP,
MDSoC, AOP and FOP employ a meta-programming approach to software composition.
Such meta-programming approaches are usually an extension to an existing programming language, such as Hyper/J and AspectJ for Java. A source-to-source compiler will
combine the separated aspects before producing plain Java code. Quite often many of
those tools do not fully support modular type-checking or separate compilation.
In contrast, Compositional Programming is a language-based approach, with both aclearly deﬁned static and dynamic semantics. The merge operator provides the composition mechanism in Compositional Programming. What distinguishes the elaboration
adopted by CP from general meta-programming is that the elaboration is completely transparent for a programmer: 1) Type-checking is done directly in the source language, wheretype errors (and other well-formedness errors) in programs are reported in terms of thesource rather than the target; 2) Type-checking is modular: each deﬁnition can be typechecked with only its implementation and type signatures of the dependencies. Worthnoting is that the style of elaboration employed by CP to give the semantics to the language
is also adopted by other languages. Most notably the GHC Haskell compiler elaborates
the source language (Haskell) into a small core language [Sulzmann et al., 2007]. Like CP,
all type-checking is done at the source level and the elaboration process is transparent toHaskell programmers. In contrast, in many approaches that employ meta-programming,
often there is no source-level type-checking or even some more basic error checking likesyntax well-formedness. Consequently, no modular type-checking is oﬀered and errors
are reported on the generated program, which are hard to understand.
Language workbenches

Language workbenches [Fowler, 2005; Erdweg et al., 2013]

have been proposed for reducing the engineering eﬀort involved in software language development. Modularity is an important concern in language workbenches for allowing
existing language components to be reused in developing new languages [Combemale
et al., 2018]. Traditionally most of the work on language workbenches has focused onsyntatic modularity approaches. More semantic modularity aspects such as separate compilation and modular typechecking are not well addressed. However, more recent workon language workbenches has started to incorporate semantic modularity techniques.
We compare our work next, to the language workbenches that employ semantic modu137

Chapter 6. Related Worklarization techniques. With Neverlang [Vacchi and Cazzola, 2015], users do not directly
program with visitors. Instead, they have to use a DSL and learn speciﬁc concepts suchas slice and roles. MontiCore [Heim et al., 2016] generates the visitor infrastructure fromits grammar speciﬁcation. To address the extensibility issue, MontiCore overrides theaccept method and uses casts for choosing the right visitor for extended variants, thus

is not type-safe. Also, MontiCore supports imperative style visitors only. Alex [Leducet al., 2018] also provides a form of semantic modularity based on the Revisitor pattern [Leduc et al., 2017], which can be viewed as a combination of Object Algebras andWalkabout [Palsberg and Jay, 1998]. By moving the dispatching method from the classhierarchy to the visitor interface, the Revisitor pattern can work for legacy class hierarchies that do not anticipate the usage of visitors. However, the dispatching method
generated by Alex is implemented using casts and has to be modiﬁed whenever new variants are added, thus is neither modular nor type-safe. Castor fully supports semantic
modularity and allows users to do the development using their familiar language with a
few annotations. For the moment, Castor still lacks much of the functionality for various
other aspects of language implementations that are covered by language workbenches.
Nevertheless, the modularization techniques employed by Castor could be useful in thecontext of language workbenches to improve reuse and type-safety of language components, in the same way that visitors are used in Neverlang and Revisitors are used in
Alex.

138

Chapter

7

Future Work

The three approaches proposed in this thesis have their own strengths and weaknesses
and complement each other.

This chapter discusses the limitations of the proposedsolutions and possible directions for future work.

7.1 Plain Design Patterns
In Chapter 3, we show how to modularize shallow EDSLs using a combination of two
design patterns, Extensible Interpreters and Object Algebras. Such an approach has two
major limitations. Firstly, it is problematic to model binary and producer operations using
the Extensible Interpreter pattern since the AST type evolves in extensions. Although
the signature of producer operations can be updated through covariant reﬁnement, theimplementations have to be duplicated. For binary operations, it is worse because it is
not possible to reﬁne the signature since AST types occur in contravariant positions. As
a result, the new version of that binary operation must coexist with the old one, which
is non-modular and error-prone. Secondly, the combination of Extensible Interpretersand Object Algebras brings more expressiveness and modularity but at the same timecomplicates the encoding. One direction of future work is to employ metaprogramming toeliminate some of the boilerplate, just like what we have done in Castor. Essentially, thetrait hierarchies can be generated by analyzing the Object Algebra interface or vice versa.
For example, from the following Object Algebra interface:trait CircuitAlg[Circuit] {def Id(n: Int): Circuit
def Fan(n: Int): Circuit
def Beside(c1: Circuit, c2: Circuit): Circuit
def Above(c1: Circuit, c2: Circuit): Circuit
def Stretch(ns: List[Int], c: Circuit): Circuit}

we can generate the hierarchy below:
trait Circuit {}trait Id extends Circuit {val n: Int}trait Fan extends Circuit {139

Chapter 7. Future Workval n: Int}trait Besides extends Circuit {val c1, c2: Circuit}
trait Above extends Circuit {val c1, c2: Circuit}
trait Stretch extends Circuit {val ns: List[Int]; val c: Circuit}

We can see a close connection between the two code snippets, where the type parameterof the Object Algebra interface becomes the root of the hierarchy and each factory method
has a trait deﬁnition with arguments captured as ﬁelds. With the hierarchy generated,
the programmer only needs to ﬁll in the methods in each trait. Similarly, more code canbe generated such as the factory for creating objects from the trait hierarchy. However,with metaprogramming, the beniﬁts of a plain design pattern are lost. Therefore, anotherdirection is to seek powerful yet simple design patterns for modularity.

7.2

Castor

While Castor is practical and serves the purpose of programming with visitors, there
are important drawbacks on such a meta-programming, library-based approach:• Unnecessary annotations. With the current version of Scalameta, we are not able
to get information from annotated parents. If parents’ information were accessible,
the inherited datatypes and visitors could be analyzed and @adts and @ops annotations could be eliminated.
• Boilerplate for nested composition. Lacking of parents’ information also disallows
automatically composing nested members. Consequently, the composition has tobe repeated for each member of the family, which is quite tedious.
• Imprecise error messages. As Castor modiﬁes the annotated programs, what
the compiler reports are errors on the modiﬁed program rather than the original
program. Reasoning about the error messages becomes harder as they are mispositioned and require some understanding of the generated code.
One direction of future work is to further simplify programming with Castor, Assuming that there is suﬃcient support from the underlying metaprogramming library, theimplementation of the Arith language can be simpliﬁed as:@family trait Arith extends Nat with Bool {@adt trait Tm {case class TmIsZero(t: Tm)}@visit(Tm) trait Eval1 {def tmIsZero = {case TmIsZero(TmZero) => TmTrue

140

7.2 Castor

case TmIsZero(TmSucc(t)) if nv(t) => TmFalse
case TmIsZero(t) => TmIsZero(this(t))}}}

where the extends clause is expressed only once at the top level and extends clauses fornested members such as super[Nat].Tm with super[Bool].Tm are inferred. Still, the syntax
and semantics of Scala cannot be changed to enforce certain restrictions such as enforcing
a default for nested patterns.
Another direction of future work is to integrate the idea of Castor into CP that additionally supports open datatypes and open pattern matching. Here, we sketch out how
the code for implementing Arith would be like in the extended CP language:
type Term = {data Tm; -- Open datatypeeval1 : Tm -> Tm;};
type Nat extends Term = {TmZero : Tm;-- Constructor for Tm

TmSucc : Tm -> Tm; -- Constructor for Tm
TmPred : Tm -> Tm; -- Constructor for Tm
nv : Tm -> Boolean;};nat = trait implements Nat => {TmZero.nv = true;(TmSucc t).nv = t.nv;_.nv = false; -- Local wildcard(TmSucc t).eval1 = TmSucc t.eval1;(TmPred TmZero).eval1 = TmZero;(TmPred (TmSucc t)).eval1 if t.nv = t;
(TmPred t).eval1 = TmPred t.eval1;_.eval1 = throw NoRuleApplies;};-- The code for bool omitted
type Arith extends Nat & Bool = {TmIsZero : Tm -> Tm;};arith = trait implements Arith inherits nat ,, bool => {(TmIsZero TmZero).eval1 = TmTrue;(TmIsZero (TmSucc t)).eval1 if t.nv = TmFalse;(TmIsZero t).eval1 = TmIsZero t.eval1;};

Without the restriction of existing languages, the syntax and semantics can be much more
ﬂexible and coherent. Both top-level (shallow) and nested (deep) patterns are written usinga uniﬁed syntax and patterns with distinct top-level constructors are by default orderinsensitive. Moreover, we can enforce that nested patterns must come with a default case.
As we shall discuss later, supporting such a design in CP is still challenging.
141

Chapter 7. Future Work

7.3

CP

As a prototype design for Compositional Programming, CP is still far from being afully-ﬂedged programming language for real-world software development. Some possible
directions for future work include the addition of recursive types and type constructors,
mutable state for modeling imperative objects, and improvements on type-inference.
In our view, the addition of recursive types is most pressing as there are many use
cases for such signatures. For example, with recursive types, we can model binary methods [Bruce et al., 1995], or operations that return the same type that is being processed.
For example, with recursive types, we should be able to model the double operation described by Zenger and Odersky [2005]. This operation doubles the literals in an arithmetic
expression, where each constructor implements the following interface:type Dbl = mu Exp. { double : Exp };Exp is captured as a recursive type. On the other hand, supporting type constructors

allows us to model, for example, the compositional interface of streams adapted from
Biboudis et al. [2015]’s work:
type StreamSig<F : Type -> Type> = {Source : forall T. Array T -> F T;
Map : forall T R. (T -> R) -> F T -> F R;
FlatMap : forall T R. (T -> F R) -> F T -> F R;
Filter : (T -> Bool) -> F T -> F T;};

where the sort F is a type constructor (i.e. a function on types). Extending CP with recursive types and type constructors is non-trivial. The ﬁrst step is to study how recursive
types and type constructors interact with disjoint intersection types and other features ofCP.
Although the programming style of CP is functional, a natural question is whether
the ideas of Compositional Programming can be adapted into a programming model withimperative objects. There are several challenges here. One of them is to see how mutablestate can be integrated into calculi with disjoint intersection types and a merge operator.
A starting point in this direction is the work by Blaauwbroek [2017], which studies the
addition of mutable references into calculus with intersection types and a merge operator.
Another general challenge is that, once imperative objects are supported, we must face
the issues of multiple inheritance in the presence of mutable state, which is a well-known
source of problems.
Finally, CP has some support for type inference, such as inferring the constructor
parameters from the implements clause. However, this support is rather limited. In particular, uses of polymorphic deﬁnitions must explicitly pass all type arguments. It will
be interesting to investigate local type inference [Pierce and Turner, 2000] to infer someof those arguments and improve the convenience of using polymorphic deﬁnitions. A
more ambitious direction would be looking into MLsub [Dolan and Mycroft, 2017] and see
whether it would be possible to adapt or extend MLsub type inference to CP. The work by
van den Berg [2020] is a starting point in this direction.
142

Chapter

8

Conclusion

Supporting modular extensible software development is a longstanding problem. In
this thesis, we have presented diﬀerent approaches to modular extensible software development from the programming languages perspective. The proposed solutions can solve
not only canonical modularity problems such as the Expression Problem but also model
programs with non-trivial dependencies. In particular, we have explored:• Plain design patterns, which can be directly applied to existing programming languages. We have shown that with a combination of two design patterns, the Extensible Interpreter pattern and Object Algebras, shallow EDSLs can be made more
modular.
• Metaprogramming-based design patterns, which further eliminate boilerplate incurred by design patterns. The proposed Castor framework employs annotations toallow us to program with extensible visitors without the pain of writing boilerplatecode.
• Novel language designs, which allow specialized syntax and semantics designed formodularity. With such ﬂexibility, we present CP, a language design for Compositional Programming.
Although there is still a gap between the current solutions and the ideal way of modular
extensible software development, it is a meaningful step forward. We hope this thesis cangive programmers and language designers some insights on how to write modular extensible code and how to support modular extensible code development from the programming
language perspective.

143

BibliographyJoão Alpuim, Bruno C d S Oliveira, and Zhiyuan Shi. 2017. Disjoint polymorphism. In
European Symposium on Programming. Springer, 1–28.
Davide Ancona and Elena Zucca. 2002. A calculus of module systems. Journal of functional programming 12, 2 (2002), 91–132.
Ivica Aracic, Vaidas Gasiunas, Mira Mezini, and Klaus Ostermann. 2006. An overview
of CaesarJ. In Transactions on Aspect-Oriented Software Development I. Springer, 135–
173.
Patrick Bahr and Tom Hvitved. 2011. Compositional Data Types. In Proceedings of theSeventh ACM SIGPLAN Workshop on Generic Programming (WGP ’11). 83–94.

https://doi.org/10.1145/2036918.2036930

Henk Barendregt, Mario Coppo, and Mariangiola Dezani-Ciancaglini. 1983.

A ﬁlterlambda model and the completeness of type assignment 1. The journal of symbolic
logic 48, 4 (1983), 931–940.
Xuan Bi and Bruno C d S Oliveira. 2018. Typed ﬁrst-class traits. In 32nd European
Conference on Object-Oriented Programming (ECOOP 2018). Schloss Dagstuhl-LeibnizZentrum fuer Informatik.
Xuan Bi, Bruno C d S Oliveira, and Tom Schrĳvers. 2018. The essence of nested composition. In 32nd European Conference on Object-Oriented Programming (ECOOP 2018).
Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
Xuan Bi, Ningning Xie, Bruno C d S Oliveira, and Tom Schrĳvers. 2019. Distributive
Disjoint Polymorphism for Compositional Programming. In European Symposium on
Programming. Springer, 381–409.
Aggelos Biboudis, Nick Palladinos, George Fourtounis, and Yannis Smaragdakis. 2015.
Streams a la carte:

Extensible pipelines with object algebras. In 29th European

Conference on Object-Oriented Programming (ECOOP 2015). Schloss Dagstuhl-LeibnizZentrum fuer Informatik.
Lasse Blaauwbroek. 2017. On the Interaction Between Unrestricted Union and Intersection
Types and Computational Eﬀects. Master’s thesis. Technical University Eindhoven.
Gilad Bracha. 1992. The programming language jigsaw: mixins, modularity and multiple
inheritance. Ph.D. Dissertation. Dept. of Computer Science, University of Utah.
145

BIBLIOGRAPHY

Gilad Bracha and William Cook. 1990. Mixin-based Inheritance. In Proceedings of theEuropean Conference on Object-oriented Programming on Object-oriented Programming
Systems, Languages, and Applications (OOPSLA/ECOOP ’90).
Gilad Bracha, Peter Von Der Ahé, Vassili Bykov, Yaron Kashai, William Maddox, and Eliot
Miranda. 2010. Modules as objects in newspeak. In European Conference on ObjectOriented Programming. Springer, 405–428.
Kim Bruce, Luca Cardelli, Giuseppe Castagna, Hopkins Objects Group, Gary T Leavens,and Benjamin Pierce. 1995. On binary methods. Theory and Practice of Object Systems
1, 3 (1995), 221–242.
Kim Bruce, Martin Odersky, and Philip Wadler. 1998. A statically safe alternative to
virtual types. In European Conference on Object-Oriented Programming.
Peter Buchlovsky and Hayo Thielecke. 2006. A Type-theoretic Reconstruction of the
Visitor Pattern. Electron. Notes Theor. Comput. Sci. 155 (May 2006), 309–329.

https://doi.org/10.1016/j.entcs.2005.11.061

Eugene Burmako. 2017. Uniﬁcation of Compile-Time and Runtime Metaprogramming in
Scala. Ph.D. Dissertation. EPFL.
R. M. Burstall, D. B. MacQueen, and D. T. Sannella. 1981. HOPE: An experimental
applicative Language. Technical Report CSR-62-80. Computer Science Dept, Univ. of
Edinburgh.
Luca Cardelli. 1994. Extensible Records in a Pure Calculus of Subtyping. MIT Press,
Cambridge, MA, USA.
Luca Cardelli and John Mitchell. 1991. Operations on Records. Mathematical Structures in
Computer Science 1 (1991), 3–48. Also in ; available as DEC Systems Research Center
Research Report #48, August, 1989, and in the proceedings of MFPS ’89, Springer
LNCS volume 442.
Jacques Carette, Oleg Kiselyov, and Chung-chieh Shan. 2009. Finally Tagless, Partially
Evaluated: Tagless Staged Interpreters for Simpler Typed Languages. Journal of Functional Programming 19, 05 (2009), 509–543. https://doi.org/10.1017/S0956796809007205
Walter Cazzola and Edoardo Vacchi. 2016. Language Components for Modular DSLs
Using Traits. Computer Languages, Systems & Structures 45 (2016), 16–34.

https://doi.org/10.1016/j.cl.2015.12.001

Craig Chambers. 1992. Object-Oriented Multi-Methods in Cecil. In Proceedings of theEuropean Conference on Object-Oriented Programming (ECOOP 92).

https://doi.org/

10.5555/646150.679216

James Cheney and Ralf Hinze. 2002. A lightweight implementation of generics and dynamics, Manuel M.T. Chakravarty (Ed.). ACM, New York, NY, USA, 90–104.
//doi.org/10.1145/581690.581698

146

https:

BIBLIOGRAPHY

Alonzo Church. 1936. An unsolvable problem of elementary number theory. American
journal of mathematics 58, 2 (1936), 345–363.
Dave Clarke, Sophia Drossopoulou, James Noble, and Tobias Wrigstad. 2007. Tribe:a simple virtual class calculus. In Proceedings of the 6th international conference on
Aspect-oriented software development. 121–134.
Curtis Clifton, Gary T Leavens, Craig Chambers, and Todd Millstein. 2000. MultiJava:
Modular open classes and symmetric multiple dispatch for Java. In ACM Sigplan Notices, Vol. 35. ACM, 130–145.
Benoit Combemale, Jörg Kienzle, Gunter Mussbacher, Olivier Barais, Erwan Bousse, Walter Cazzola, Philippe Collet, Thomas Degueule, Robert Heinrich, Jean-Marc Jézéquel,
et al. 2018. Concern-oriented language development (cold): Fostering reuse in language
engineering. Computer Languages, Systems & Structures 54 (2018), 139–155.
William Cook and Jens Palsberg. 1989. A Denotational Semantics of Inheritance and Its
Correctness. In Proceedings on Object-Oriented Programming Systems, Languages andApplications (OOPSLA 89). ACM, New York, NY, USA, 433443. https://doi.org/10.1145/
74877.74922

William R. Cook. 2009. On Understanding Data Abstraction, Revisited. In Proceedings ofthe 24th ACM SIGPLAN Conference on Object Oriented Programming Systems Languages
and Applications (OOPSLA ’09). 557–572. https://doi.org/10.1145/1639949.1640133
Stephen Dolan and Alan Mycroft. 2017. Polymorphism, Subtyping, and Type Inferencein MLsub (POPL 2017). ACM, New York, NY, USA, 60–72.

https://doi.org/10.1145/

3009837.3009882

Dominic Duggan and Constantinos Sourelis. 1996. Mixin modules. ACM SIGPLAN Notices
31, 6 (1996), 262–273.
Joshua Dunﬁeld. 2014. Elaborating Intersection and Union Types. Journal of Functional
Programming 24, 2-3 (2014), 133–165. https://doi.org/10.1017/S0956796813000270
Burak Emir, Martin Odersky, and John Williams. 2007. Matching objects with patterns.
In European Conference on Object-Oriented Programming.
Sebastian Erdweg, Tĳs Van Der Storm, Markus Völter, Meinte Boersma, Remi Bosman,William R Cook, Albert Gerritsen, Angelo Hulshout, Steven Kelly, Alex Loh, et al. 2013.
The state of the art in language workbenches. In International Conference on Software
Language Engineering.
Erik Ernst. 1999. gbeta-a Language with Virtual Attributes, Block Structure, and Propagating, Dynamic Inheritance. Ph.D. Dissertation. University of Aarhus.
Erik Ernst. 2001. Family Polymorphism. In Proceedings of the 15th European Conference
on Object-Oriented Programming (ECOOP ’01).
147

BIBLIOGRAPHY

Erik Ernst. 2004. The expression problem, Scandinavian style. On Mechanisms For
Specialization (2004), 27.
Erik Ernst, Klaus Ostermann, and William R. Cook. 2006. A Virtual Class Calculus.
In Conference Record of the 33rd ACM SIGPLAN-SIGACT Symposium on Principles ofProgramming Languages (POPL ’06).
Martin Erwig and Eric Walkingshaw. 2012. Semantics-Driven DSL Design. In Formal andPractical Aspects of Domain-Speciﬁc Languages: Recent Developments. 56–80. https://doi.org/10.4018/978-1-4666-2092-6.ch003

Matthew Flatt, Shriram Krishnamurthi, and Matthias Felleisen. 1998. Classes and mixins. In Proceedings of the 25th ACM SIGPLAN-SIGACT symposium on Principles of programming languages. 171–183.
Martin Fowler. 2005. Language workbenches: The killer-app for domain speciﬁc languages. http://martinfowler.com/articles/languageWorkbench.html
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
Jacques Garrigue. 1998. Programming with polymorphic variants. In ML Workshop.
Jacques Garrigue. 2000. Code reuse through polymorphic variants. In Workshop on
Foundations of Software Engineering.
Felix Geller, Robert Hirschfeld, and Gilad Bracha. 2010. Pattern Matching for an objectoriented and dynamically typed programming language. Number 36. Universitätsverlag
Potsdam.
Jeremy Gibbons. 2003. Origami Programming. 41–60. http://www.comlab.ox.ac.uk/oucl/
work/jeremy.gibbons/publications/origami.pdf

Jeremy Gibbons and Nicolas Wu. 2014.

Folding Domain-Speciﬁc Languages: Deepand Shallow Embeddings (Functional Pearl). In Proceedings of the 19th ACM SIGPLAN International Conference on Functional Programming (ICFP ’14). 339–347. https://doi.org/10.1145/2628136.2628138

John V. Guttag and James J. Horning. 1978. The algebraic speciﬁcation of abstract datatypes. Acta informatica 10, 1 (1978), 27–52.
William Harrison and Harold Ossher. 1993. Subject-oriented programming: a critique ofpure objects. In Proceedings of the eighth annual conference on Object-oriented programming systems, languages, and applications. 411–428.
Robert Heim, Pedram Mir Seyed Nazari, Bernhard Rumpe, and Andreas Wortmann. 2016.
Compositional Language Engineering Using Generated, Extensible, Static Type-Safe
Visitors. In European Conference on Modelling Foundations and Applications.
148

BIBLIOGRAPHY

George T. Heineman and William T. Councill (Eds.). 2001. Component-Based Software
Engineering: Putting the Pieces Together. (2001).
Ralf Hinze. 2004. An Algebra of Scans. In Mathematics of Program Construction. 186–210.
https://doi.org/10.1007/978-3-540-27764-4_11

Ralf Hinze. 2006. Generics for the Masses. Journal of Functional Programming 16, 4-5(2006), 451–483. https://doi.org/10.1017/S0956796806006022
Christian Hofer and Klaus Ostermann. 2010. Modular Domain-speciﬁc Language Components in Scala. In Proceedings of the Ninth International Conference on Generative
Programming and Component Engineering (GPCE ’10).
Christian Hofer, Klaus Ostermann, Tillmann Rendel, and Adriaan Moors. 2008. Polymorphic Embedding of Dsls. In Proceedings of the 7th International Conference on Generative
Programming and Component Engineering (GPCE ’08).
Michael Homer, James Noble, Kim B. Bruce, Andrew P. Black, and David J. Pearce.
2012. Patterns As Objects in Grace. In Proceedings of the 8th Symposium on Dynamic
Languages (DLS ’12). New York, NY, USA, 17–28.
Xuejing Huang and Bruno C. d. S. Oliveira. 2020. A Type-Directed Operational Semantics
For a Calculus with a Merge Operator. In 34th European Conference on Object-Oriented
Programming (ECOOP 2020) (Leibniz International Proceedings in Informatics (LIPIcs)),
Vol. 166. https://doi.org/10.4230/LIPIcs.ECOOP.2020.26
Paul Hudak. 1998. Modular Domain Speciﬁc Languages and Tools. In Proceedings. Fifth
International Conference on Software Reuse. 134–142.

https://doi.org/10.1109/ICSR.

1998.685738

Atsushi Igarashi and Benjamin C Pierce. 1999. Foundations for virtual types. In European
Conference on Object-Oriented Programming. Springer, 161–185.
Atsushi Igarashi, Chieri Saito, and Mirko Viroli. 2005. Lightweight family polymorphism.
In Asian Symposium on Programming Languages and Systems. Springer, 161–177.
Chinawat Isradisaikul and Andrew C. Myers. 2013.

Reconciling Exhaustive Pattern

Matching with Objects. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13).
Paul Jolly, Sophia Drossopoulou, Christopher Anderson, and Klaus Ostermann. 2004.
Simple dependent types: Concord. In ECOOP Workshop on Formal Techniques for Java
Programs (FTfJP).
Simon Peyton Jones. 2003. Haskell 98 language and libraries: the revised report. Cambridge University Press.
149

BIBLIOGRAPHY

Vojin Jovanovic, Amir Shaikhha, Sandro Stucki, Vladimir Nikolaev, Christoph Koch, andMartin Odersky. 2014. Yin-Yang: Concealing the Deep Embedding of DSLs. In Proceedings of the 2014 International Conference on Generative Programming: Concepts andExperiences (GPCE 2014). 73–82. https://doi.org/10.1145/2658761.2658771
Uwe Kastens and William M. Waite. 1994. Modularity and reusability in attribute grammars. Acta Informatica 31, 7 (1994), 601–627.
Gregor Kiczales, John Lamping, Anurag Mendhekar, Chris Maeda, Cristina Lopes, JeanMarc Loingtier, and John Irwin. 1997. Aspect-oriented programming. In European
conference on object-oriented programming. Springer, 220–242.
Oleg Kiselyov. 2012. Typed Tagless Final Interpreters. In Generic and Indexed Programming. 130–174. https://doi.org/10.1007/978-3-642-32202-0_3
Jorgen Lindskov Knudsen, Boris Magnusson, Mats Lofgren, and Ole L Madsen. 1994.
Object Oriented Software Development Environments: The Mjolner Approach. PrenticeHall, Inc.
Donald E. Knuth. 1968. Semantics of Context-Free Languages. Math. Sys. Theory 2, 2
(1968), 127–145.
Donald E. Knuth. 1990. The Genesis of Attribute Grammars. In WAGA. 1–12.
Giovanni Lagorio, Marco Servetto, and Elena Zucca. 2009. Featherweight jigsaw: A minimal core calculus for modular composition of classes. In European Conference on ObjectOriented Programming. Springer, 244–268.
Manuel Leduc, Thomas Degueule, and Benoit Combemale. 2018. Modular language composition for the masses. In Proceedings of the 11th ACM SIGPLAN International Conference on Software Language Engineering.
Manuel Leduc, Thomas Degueule, Benoit Combemale, Tĳs Van Der Storm, and Olivier
Barais. 2017. Revisiting visitors for modular extension of executable DSMLs. In 2017
ACM/IEEE 20th International Conference on Model Driven Engineering Languages andSystems (MODELS).
Joseph Lee, Jonathan Aldrich, Troy Shaw, and Alex Potanin. 2015. A theory of tagged
objects. In 29th European Conference on Object-Oriented Programming (ECOOP 2015).
Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
Jed Liu and Andrew C Myers. 2003. JMatch: Iterable abstract pattern matching for Java.
In PADL.
Andres Löh and Ralf Hinze. 2006. Open data types and open functions. In Proceedings of
the 8th ACM SIGPLAN international conference on Principles and practice of declarative
programming.
150

BIBLIOGRAPHYDavid MacQueen. 1984. Modules for standard ML. In Proceedings of the 1984 ACM
Symposium on LISP and functional programming. 198–207.
Ole Lehrmann Madsen and Birger Moller-Pedersen. 1989. Virtual classes: A powerful mechanism in object-oriented programming. In Conference proceedings on Objectoriented programming systems, languages and applications. 397–406. https://doi.org/
10.1145/74877.74919

Ole Lehrmann Madsen, Birger Møller-Pedersen, and Kristen Nygaard. 1993.

Object-oriented programming in the BETA programming language. Addison-Wesley.
Robert C. Martin. 2002. The Principles, Patterns, and Practices of Agile Software Development. Prentice Hall.
Tanja Mayerhofer and Manuel Wimmer. 2015. The TTC 2015 Model Execution Case.. In
TTC@ STAF. 2–18.
Erik Meĳer, Brian Beckman, and Gavin Bierman. 2006. LINQ: Reconciling Object, Relations and XML in the .NET Framework. In Proceedings of the 2006 ACM SIGMODInternational Conference on Management of Data (SIGMOD ’06). 706–706.

https://doi.org/10.1145/1142473.1142552

Bertrand Meyer. 1988. Object-Oriented Software Construction. Prentice Hall.
Bertrand Meyer and Karine Arnout. 2006. Componentization: the Visitor example. Computer 39, 7 (2006), 23–30.
Todd Millstein, Colin Bleckner, and Craig Chambers. 2004. Modular Typechecking for
Hierarchically Extensible Datatypes and Functions. ACM Trans. Program. Lang. Syst.
26, 5 (Sept. 2004).
Robin Milner, Mads Tofte, Robert Harper, and David Macqueen. 1997. The Deﬁnition ofStandard ML-Revised. (1997).
Adriaan Moors, Frank Piessens, and Martin Odersky. 2008. Generics of a Higher Kind.
In Proceedings of the 23rd ACM SIGPLAN Conference on Object-oriented Programming
Systems Languages and Applications (OOPSLA ’08). https://doi.org/10.1145/1449764.
1449798

Keiko Nakata and Jacques Garrigue. 2006. Recursive modules for programming. ACM
SIGPLAN Notices 41, 9 (2006), 74–86.
Peter Naur. 1968. Software engineering-report on a conference sponsored by the NATO
Science Committee Garimisch, Germany. http://homepages. cs. ncl. ac. uk/brian. randell/NATO/nato1968. PDF (1968).
Martin E Nordberg III. 1996. Variations on the visitor pattern. In PLoP96 Writers Workshop, Vol. 154.
151

BIBLIOGRAPHYNathaniel Nystrom, Stephen Chong, and Andrew C Myers. 2004. Scalable extensibility
via nested inheritance. In Proceedings of the 19th annual ACM SIGPLAN conference on
Object-oriented programming, systems, languages, and applications. 99–115.
Nathaniel Nystrom, Xin Qi, and Andrew C Myers. 2006. J& nested intersection for scalable
software composition. ACM SIGPLAN Notices 41, 10 (2006), 21–36.
Martin Odersky. 2006. Pimp My Library.

http://www.artima.com/weblogs/viewpost.jsp?

thread=179766 Last accessed on 2019-01-29.

Martin Odersky, Philippe Altherr, Vincent Cremet, Burak Emir, Sebastian Maneth,
Stéphane Micheloud, Nikolay Mihaylov, Michel Schinz, Erik Stenman, and Matthias
Zenger. 2004. An overview of the Scala programming language. Technical Report.
Martin Odersky and Matthias Zenger. 2005. Scalable Component Abstractions. In Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming,
Systems, Languages, and Applications (OOPSLA 2005).
Bruno C. d. S. Oliveira. 2009. Modular Visitor Components. In Proceedings of the 23rd
European Conference on Object-Oriented Programming.
Bruno C. d. S. Oliveira and William R. Cook. 2012. Extensibility for the Masses: Practical
Extensibility with Object Algebras. In Proceedings of the 26th European Conference on
Object-Oriented Programming (ECOOP ’12).

https://doi.org/10.1007/978-3-642-31057-

7_2

Bruno C. d. S. Oliveira, Ralf Hinze, and Andres Löh. 2006a. Extensible and modular
generics for the masses. Trends in Functional Programming 7 (2006), 199–216.
Bruno C. d. S. Oliveira, Ralf Hinze, and Andres Löh. 2006b. Extensible and Modular
Generics for the Masses. In Trends in Functional Programming. 199–216.
Bruno C. d. S. Oliveira, Shin-Cheng Mu, and Shu-Hung You. 2015. Modular Reiﬁable
Matching: A List-of-functors Approach to Two-level Types. In Proceedings of the 2015
ACM SIGPLAN Symposium on Haskell (Haskell ’15).
Bruno C d S Oliveira, Zhiyuan Shi, and João Alpuim. 2016. Disjoint intersection types. In
Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming. 364–377.
Bruno C. d. S. Oliveira, Tĳs van der Storm, Alex Loh, and William R. Cook. 2013. FeatureOriented Programming with Object Algebras. In Proceedings of the 27th European Conference on Object-Oriented Programming. https://doi.org/10.1007/978-3-642-39038-8_2
Jens Palsberg and C. Barry Jay. 1998. The Essence of the Visitor Pattern. In Proceedingsof the 22nd International Computer Software and Applications Conference.
Tanumoy Pati and James H Hill. 2014. A survey report of enhancements to the visitor
software design pattern. Software: Practice and Experience 44, 6 (2014), 699–733.
152

BIBLIOGRAPHYFrank Pfenning and Conal Elliott. 1988. Higher-order Abstract Syntax. In Proceedings of
the ACM SIGPLAN 1988 Conference on Programming Language Design and Implementation (PLDI ’88). https://doi.org/10.1145/53990.54010
Benjamin C Pierce. 2002. Types and programming languages. MIT press.
Benjamin C. Pierce and David N. Turner. 2000. Local Type Inference. ACM Trans. Program.
Lang. Syst. 22, 1 (Jan. 2000), 44.
Klaus Pohl, Günter Böckle, and Frank J van Der Linden. 2005. Software product line engineering: foundations, principles and techniques. Springer Science & Business Media.
Erik Poll. 1997. System F with Width-Subtyping and Record Updating. In Proceedings of
the Third International Symposium on Theoretical Aspects of Computer Software (TACS
1997). Springer-Verlag, Berlin, Heidelberg.
Christian Prehofer. 1997. Feature-oriented programming: A fresh look at objects. In
European Conference on Object-Oriented Programming. Springer, 419–443.
Tillmann Rendel, Jonathan Immanuel Brachthäuser, and Klaus Ostermann. 2014. From
Object Algebras to Attribute Grammars. In Proceedings of the 2014 ACM International
Conference on Object-Oriented Programming Systems Languages and Applications.
John C. Reynolds. 1978. User-deﬁned Types and Procedural Data Structures as Complementary Approaches to Type Abstraction. 309–317.

https://doi.org/10.1007/978-1-

4612-6315-9_22

Tiark Rompf and Nada Amin. 2015. Functional Pearl: A SQL to C Compiler in 500 Linesof Code. In Proceedings of the 20th ACM SIGPLAN International Conference on Functional
Programming (ICFP 2015). 2–9. https://doi.org/10.1145/2784731.2784760
Tiark Rompf and Martin Odersky. 2010. Lightweight Modular Staging: A Pragmatic Approach to Runtime Code Generation and Compiled DSLs. In Proceedings of the Ninth
International Conference on Generative Programming and Component Engineering (GPCE’10). 127–136. https://doi.org/10.1145/1868294.1868314
Claudio V Russo. 2000. First-class structures for Standard ML. In European Symposiumon Programming. Springer, 336–350.
Claudio V Russo. 2001. Recursive structures for Standard ML. In Proceedings of the sixth
ACM SIGPLAN international conference on Functional programming. 50–61.
Nathanael Schärli, Stéphane Ducasse, Oscar Nierstrasz, and Andrew P Black. 2003.
Traits: Composable units of behaviour. In European Conference on Object-Oriented
Programming. Springer, 248–274.
DS Scott. 1963. A system of functional abstraction. Unpublished manuscript (1963).
Tim Sheard and Simon Peyton Jones. 2002. Template meta-programming for Haskell. In
Proceedings of the 2002 ACM SIGPLAN workshop on Haskell.
153

BIBLIOGRAPHY

Dave Steinberg, Frank Budinsky, Ed Merks, and Marcelo Paternostro. 2008. EMF: eclipse
modeling framework. Pearson Education.
M. Sulzmann, M. M. T. Chakravarty, S. L. Peyton-Jones, and K. Donnelly. 2007. System
F with type equality coercions. In TLDI.
Josef Svenningsson and Emil Axelsson. 2012. Combining Deep and Shallow Embeddingfor EDSL. In Trends in Functional Programming. 21–36.

https://doi.org/10.1007/978-3-642-40447-4_2

Wouter Swierstra. 2008. Data Types à la Carte. Journal of Functional Programming 18,
04 (2008), 423–436. https://doi.org/10.1017/S0956796808006758
Asumu Takikawa, T Stephen Strickland, Christos Dimoulas, Sam Tobin-Hochstadt, andMatthias Felleisen. 2012. Gradual typing for ﬁrst-class classes. In Proceedings of the
ACM international conference on Object oriented programming systems languages andapplications. 793–810.
Peri Tarr, Harold Ossher, William Harrison, and Stanley M Sutton. 1999. N degrees
of separation: Multi-dimensional separation of concerns. In Proceedings of the 1999
International Conference on Software Engineering. IEEE, 107–119.
Kresten Krab Thorup. 1997. Genericity in Java with virtual types. In European Conference
on Object-Oriented Programming.
Mads Torgersen. 2004. The expression problem revisited. In European Conference on
Object-Oriented Programming.
Edoardo Vacchi and Walter Cazzola. 2015. Neverlang: A framework for feature-oriented
language development. Computer Languages, Systems & Structures 43 (2015), 1–40.
Birthe van den Berg. 2020. ICFP: G: Type Inference for Disjoint Intersection Types.
Joost Visser. 2001. Visitor Combination and Traversal Control. In Proceedings of the 16th
ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, andApplications (OOPSLA ’01). https://doi.org/10.1145/504282.504302
Philip Wadler. 1992. The essence of functional programming. 19th POPL (Jan. 1992),
1–14.
Philip Wadler. 1998. The Expression Problem. (Nov. 1998).

Note to Java Genericity

mailing list.
Yanlin Wang and Bruno C. d. S. Oliveira. 2016. The Expression Problem, Trivially!. In
Proceedings of the 15th International Conference on Modularity (MODULARITY 2016).
37–41. https://doi.org/10.1145/2889443.2889448
Hongwei Xi, Chiyan Chen, and Gang Chen. 2003. Guarded Recursive Datatype Constructors. In Proceedings of the 30th ACM SIGPLAN-SIGACT Symposium on Principles ofProgramming Languages (POPL ’03).
154

BIBLIOGRAPHY

Matthias Zenger and Martin Odersky. 2001. Extensible Algebraic Datatypes with Defaults. In Proceedings of the Sixth ACM SIGPLAN International Conference on Functional
Programming.
Mathhias Zenger and Martin Odersky. 2005. Independently Extensible Solutions to theExpression Problem. In Foundations of Object-Oriented Languages (FOOL).
Haoyuan Zhang, Zewei Chu, Bruno C. d. S. Oliveira, and Tĳs van der Storm. 2015.
Scrap Your Boilerplate with Object Algebras. In Proceedings of the 2015 ACM SIGPLAN
International Conference on Object-Oriented Programming, Systems, Languages, andApplications (OOPSLA 2015). 127–146. https://doi.org/10.1145/2814270.2814279
Weixin Zhang. 2017. Extensible Domain-Speciﬁc Languages in Object-Oriented Programming. HKU Theses Online (HKUTO) (2017).
Weixin Zhang and Bruno C.d.S. Oliveira. 2020. Castor: Programming with extensible
generative visitors. Science of Computer Programming 193 (2020), 102449.

https://doi.org/10.1016/j.scico.2020.102449

Weixin Zhang and Bruno C. d. S. Oliveira. 2017. EVF: An Extensible and Expressive
Visitor Framework for Programming Language Reuse. In 31st European Conference on
Object-Oriented Programming. https://doi.org/10.4230/LIPIcs.ECOOP.2017.29
Weixin Zhang and Bruno C. d. S. Oliveira. 2018. Pattern Matching in an Open World. In
Proceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. https://doi.org/10.1145/3278122.3278124
Weixin Zhang and Bruno C. d. S. Oliveira. 2019. Shallow EDSLs and Object-Oriented
Programming: Beyond Simple Compositionality. The Art, Science, and Engineering ofProgramming 3, 3 (2019), 1–25.
Weixin Zhang, Yaozhu Sun, and Bruno C. d. S. Oliveira. 2021. Compositional Programming. ACM Transactions on Programming Languages and Systems (2021), to appear.
Yizhou Zhang and Andrew C Myers. 2017. Familia: unifying interfaces, type classes, andfamily polymorphism. Proceedings of the ACM on Programming Languages 1, OOPSLA(2017), 1–31.

155

Appendices157

Appendix

A

Full Type System of CP∆, Σ ` A ⇒ B
E-tvar

E-top

E-bot

E-int

∆, Σ ` > ⇒ >∆, Σ ` ⊥ ⇒ ⊥∆, Σ ` Int ⇒ Int

E-sig

X hα, i 7→ C ∈ ∆E-arrΣ ` S ⇒ hA, Bi∆, Σ ` A ⇒ A1∆, Σ ` X hSi ⇒ [A/α, B/ ]CE-and∆ , Σ ` A ⇒ A1

E-trait∆ , Σ ` A ⇒ A1∆, Σ ` A & B ⇒ A1 & B1∆, Σ ` A ⇒ A1∆, Σ ` { l : A } ⇒ { l : B }∆, Σ ` B ⇒ B1∆, Σ ` B ⇒ B1∆, Σ ` Trait[A, B] ⇒ Trait[A1 , B1 ]E-all∆, Σ ` A ⇒ B∆, Σ ` α ⇒ α∆, Σ ` A → B ⇒ A1 → B1∆, Σ ` B ⇒ B1

E-rcd

α∗A ∈∆∆, α ∗ A1 ` B ⇒ B1∆, Σ ` ∀(α ∗ A).B ⇒ ∀(α ∗ A1 ).B1

Σ ` S ⇒ hA, Bi
E-sort1Sort

α 7→

∈Σ

Σ ` α ⇒ hα, iE-sort1

E-sort2Σ ` A ⇒ hA, AiΣ ` A%B ⇒ hA & B, Bi
159

Σ `cp A ⇒ B
TR-positive

TR-topTR-botTR-intΣ `cp > ⇒ >Σ `cp ⊥ ⇒ ⊥Σ `cp Int ⇒ Int

TR-ctrPositive

α 7→

TR-arrΣ `cﬂip(p) A ⇒ A1

Σ `cp A ⇒ A1

Σ `cp {l : A} ⇒ {l : B}

TR-traitΣ `cp B ⇒ B1

Σ `cﬂip(p) A ⇒ A1

Σ `cp A → B ⇒ A1 → B1
TR-andΣ `falseα⇒+(l )Σ `isCapitalizedA⇒BpΣ `cp α ⇒ αΣ `trueα ⇒ Trait[α, ]+∈ΣTR-rcdTR-tvar∈Σ

α 7→

Σ `cp B ⇒ B1

Σ `cp Trait[A, B] ⇒ Trait[A1 , B1 ]TR-allΣ `cp B ⇒ B1

Σ `cp A ⇒ A1

Σ `cp A & B ⇒ A1 & B1

Σ\α `cp B ⇒ B1

Σ `cp ∀(α ∗ A).B ⇒ ∀(α ∗ A1 ).B1

A <: B
S-trans

S-reflA <: B

B <: CA <: A
A <: C

S-rcd

S-topLike

S-bot

A <:eBd⊥ <: A

B1 <: A1

A & B <: B

S-and

A2 <: B2

A <: BA1 → A2 <: B1 → B2

S-forallA & B <: A{l : A} <: {l : B}

S-arr

S-andrS-andlA <: BA <: CA <: B & C

S-trait

B1 <: B2A2 <: A1A2 <: A1∀(α ∗ A1 ).B1 <: ∀(α ∗ A2 ).B2

B1 <: B2

Trait[A1 , B1 ] <: Trait[A2 , B2 ]S-distArr

S-distTrait(A → B) & (A → C) <: A → B & C

Trait[A, B] & Trait[A, C] <: Trait[A, B & C]

S-distRcd

S-distAll{l : A} & {l : B} <: {l : A & B}∀(α ∗ A).B & ∀(α ∗ A).C <: ∀(α ∗ A).B & C

eAd
TL-tope>dTL-andeAdeB deA & B d

TL-arr

TL-rcdTL-all

TL-traiteA → Bde{l : A}de∀(α ∗ A).Bd

eTrait[A, B]d

eBd

eA d

eBd

eB d

∆`A∗B
D-topL

D-topR

D-arr

∆`A∗B

∆`A∗B

∆ ` A1 → A2 ∗ B1 → B2

eA d

eBd

D-andL∆ ` A2 ∗ B2

∆ ` A1 ∗ B∆ ` A1 & A2 ∗ B

D-rcdEq

D-andR∆ ` A ∗ B1

∆ ` A ∗ B2

∆ ` A ∗ B1 & B2

D-rcdNeq∆`A∗B

l1 , l2∆ ` { l : A } ∗ {l : B }∆ ` {l1 : A} ∗ {l2 : B}
D-forallD-tvarR

D-tvarL(α ∗ A ) ∈ ∆(α ∗ A ) ∈ ∆A <: B∆`α∗B
D-trait∆ ` A2 ∗ B∆, α ∗ A1 & A2 ` B1 ∗ B2

A <: B∆`B∗α∆ ` ∀(α ∗ A1 ).B1 ∗ ∀(α ∗ A2 ).B2
D-traitArr∆ ` B1 ∗ B2

∆ ` B1 ∗ B2

∆ ` Trait[A1 , B1 ] ∗ Trait[A2 , B2 ]∆ ` Trait[A1 , B1 ] ∗ A2 → B2

D-arrTrait

D-ax∆ ` A1 → B1 ∗ Trait[A2 , B2 ]∆`A∗B

∆ ` B1 ∗ B2

A ∗ax BA ∗ax B

Dax-intArr

Dax-intRcd

Dax-intAll

Dax-intTrait

Int ∗ax A1 → A2

Int ∗ax {l : A}

Int ∗ax ∀(α ∗ B1 ).B2

Int ∗ax Trait[A, B]

Dax-arrAll

Dax-arrRcd

Dax-arrTrait

A1 → A2 ∗ax ∀(α ∗ B1 ).B2

A1 → A2 ∗ax {l : B}

A1 → A2 ∗ax Trait[A, B]

Dax-arrAll

Dax-arrTrait∀(α ∗ A1 ).A2 ∗ax {l : B}∀(α ∗ A1 ).A2 ∗ax Trait[A, B]

Note: For each A ∗ax B we have a symmetric rule B ∗ax A.

∆; Γ ` P ⇒ A ; e
T-tyDecl

fresh∆; α 7→` B ⇒ B1

α 7→

`+

false

∆; α 7→ ` A ⇒ A1
B1 ⇒ B2∆, X hα, i 7→ A1 & B2 ; Γ ` P ⇒ C ; e∆; Γ ` type X hα i extends A = B; P ⇒ C ; e
T-tmDecl∆; Γ ` E ⇒ A ; e1∆; Γ, x : A ` P ⇒ B ; e2∆; Γ ` x = E ; P ⇒ B ; let x : |A| = e1 in e2∆; Γ ` E ⇒ A ; e
T-var

T-top

T-nat

∆; Γ ` > ⇒ > ; >∆; Γ ` i ⇒ Int ; i

T-app∆; Γ ` E1 ⇒ A1 → A2 ; e1(x : A ) ∈ Γ∆; Γ ` x ⇐ A ; x∆; Γ ` E2 ⇐ A1 ; e2∆; Γ ` E1 E2 ⇒ A2 ; e1 e2
T-anno∆; • ` A ⇒ B

T-rcd∆; Γ ` E ⇐ B ; e∆; Γ ` E ⇒ A ; e∆; Γ ` E : A ⇒ B ; e : |B|
T-tabsT-proj∆; Γ ` E ⇒ {l : A} ; e∆; • ` A ⇒ A1∆; Γ ` E.l ⇒ A ; e
T-tapp∆; Γ ` {l = E } ⇒ {l : A} ; e∆ , α ∗ A1 ; Γ ` E ⇒ B ; e∆; Γ ` Λ(α ∗ A).E ⇒ ∀(α ∗ |A1 |).B ; e∆; • ` A ⇒ A1∆; Γ ` E ⇒ ∀(α ∗ B).C ; e∆ ` A1 ∗ B∆; Γ ` E @A ⇒ [A1 /α ]C ; e |A1 |T-let∆; • ` A ⇒ A1∆; Γ, x : A1 ` E1 ⇐ A1 ; e1∆; Γ, x : A1 ` E2 ⇒ B ; e2∆; Γ ` let x : A = E1 in E2 ⇒ B ; let x : |A1 | = e1 in e2
T-mergeTrait∆; Γ ` E1 ⇒ Trait[A1 , B1 ] ; e1∆; Γ ` E2 ⇒ Trait[A2 , B2 ] ; e2

∆ ` B1 ∗ B2∆; Γ ` E1 , , E2 ⇒ Trait[A1 & A2 , B1 & B2 ] ; λ(self : |A1 & A2 |). e1 self , , e2 self
T-merge∆; Γ ` E1 ⇒ A1 ; e1∆; Γ ` E2 ⇒ A2 ; e2

∆ ` A1 ∗ A2∆; Γ ` E1 , , E2 ⇒ A1 & A2 ; e1 , , e2
T-new∆; Γ ` E ⇒ Trait[A, B] ; e

B <: A∆; Γ ` new E ⇒ B ; let self : |B| = e self in self
T-open∆; Γ ` E1 ⇒ {li : Ai } ; e1∆; Γ, li : Ai ` E2 ⇒ e2 ; B∆; Γ ` open E1 in E2 ⇒ B ; let x = e1 in let li : |A| = x.li in e2

T-trait∆; • ` A ⇒ A1∆; • ` B ⇒ B1∆; Γ, self : A1 ` E1 ⇒ Trait[A2 , B2 ] ; e1
A1 <: A2∆; Γ, self : A1 , super : B2 ` E2 ⇒ C ; e2
C ∗ B2
C & B2 <: B1∆; Γ ` trait[self : A] implements B inherits E1 => E2 ⇒ Trait[A1 , C & B2 ]; λ(self : |A1 |). let super = e1 self in e2 , , super
T-forward∆; Γ ` E1 ⇒ Trait[A, B] ; e1∆; Γ ` E2 ⇐ A ; e2∆; Γ ` E1 ˆE2 ⇒ B ; e1 e2∆; Γ ` E ⇐ A ; e
T-abs

T-sub

∆; Γ, x : A ` E ⇐ B ; e∆; Γ ` E ⇒ B ; e∆; Γ ` λx.E ⇐ A → B ; λx.E

B <: A∆; Γ ` E ⇐ A ; e|Trait[A, B]| = |A| → |B||A → B | = |A | → | B ||A & B| = |A| & |B||{l : A}| = {l : |A|}|∀(α ∗ A).B| = ∀(α ∗ |A|).|B||A| = AAppendix

B

Metatheory of CP

Lemma 2.5 [Well-formedness preservation] If ∆, Σ ` A ⇒ B then |∆| ` |B|.
Proof. By simple induction on the derivation of the judgment.

□

Lemma 2.6 [Disjointness axiom preservation] If A ∗ax B then |A| ∗ax |B|.
Proof. Note that |Trait[A, B]| = |A| → |B|; the rest are immediate.

□

Lemma 2.7 [Subtyping preservation] If A <: B then |A| <: |B|.
Proof. Most of them are just Fi+ subtyping. We only show the rule S-trait
S-traitA2 <: A1

B1 <: B2

Trait[A1 , B1 ] <: Trait[A2 , B2 ]|A2 | <: |A1 ||B1 | <: |B2 ||A1 | → |B1 | <: |A2 | → |B2 |By i.h.
By i.h.
By TS-arr□Lemma 2.8 [Disjointness preservation] If ∆ ` A ∗ B then |∆| ` |A| ∗ |B|.
Proof. By induction on the derivation of the judgment.
• D-topL, D-topR, and D-rcdNeq are immediate.
•D-tvarL(α ∗ A ) ∈ ∆A <: B∆`α∗B|A | < : |B |a∗A ∈∆a ∗ |A| ∈ |∆||∆| ` α ∗ |B|

By Lemma 2.7
Given
Above
By TD-tvarL165

•D-tvarR(α ∗ A) ∈ ∆A <: B∆`B∗α|A| <: |B|a∗A ∈∆a ∗ |A| ∈ |∆||∆| ` |B| ∗ α

By Lemma 2.7
Given
Above
By TD-tvarR•
D-forall∆, α ∗ A1 & A2 ` B1 ∗ B2

∆ ` ∀(α ∗ A1 ).B1 ∗ ∀(α ∗ A2 ).B2|∆|, α ∗ |A1 | & |A2 | ` |B1 | ∗ |B2 ||∆| ` ∀(α ∗ |A1 |).B1 ∗ ∀(α ∗ |A2 |).|B2 |By i.h.
By TD-forall•
D-rcdEq∆`A∗B∆ ` { l : A } ∗ {l : B }|∆| ` |A| ∗ |B||∆| ` {l : |A|} ∗ {l : |B|}

By i.h.
By TD-rcdEq

•
D-arr

∆ ` A2 ∗ B2

∆ ` A1 → A2 ∗ B1 → B2|∆| ` |A2 | ∗ |B2 ||∆| ` |A1 | → |A2 | ∗ |B1 | → |B2 |By i.h.
By TD-arr•
D-andL∆ ` A1 ∗ B∆ ` A2 ∗ B∆ ` A1 & A2 ∗ B|∆| ` |A1 | ∗ |B||∆| ` |A2 | ∗ |B||∆| ` |A1 | & |A2 | ∗ |B|

By i.h.
By i.h.
By TD-andL

•
D-andR∆ ` A ∗ B1

∆ ` A ∗ B2

∆ ` A ∗ B1 & B2|∆| ` |A| ∗ |B1 ||∆| ` |A| ∗ |B2 ||∆| ` |A| ∗ |B1 | & |B2 |By i.h.
By i.h.
By TD-andR•
D-trait∆ ` B1 ∗ B2

∆ ` Trait[A1 , B1 ] ∗ Trait[A2 , B2 ]|∆| ` |B1 | ∗ |B2 ||∆| ` |A1 | → |B1 | ∗ |A2 | → |B2 |By i.h.
By TD-arr•
D-traitArr∆ ` B1 ∗ B2

∆ ` Trait[A1 , B1 ] ∗ A2 → B2|∆| ` |B1 | ∗ |B2 ||∆| ` |A1 | → |B1 | ∗ |A2 | → |B2 |By i.h.
By TD-arr•
D-arrTrait∆ ` B1 ∗ B2

∆ ` A1 → B1 ∗ Trait[A2 , B2 ]|∆| ` |B1 | ∗ |B2 ||∆| ` |A1 | → |B1 | ∗ |A2 | → |B2 |By i.h.
By TD-arr•
D-ax

A ∗ax B∆`A∗B
|A| ∗ax |B||∆| ` |A| ∗ |B|

By Lemma 2.6
By TD-ax□
Theorem 2.3 [Type-safety] We have that:• If ∆; Γ ` P ⇒ A ; e then |∆|; |Γ| ` e ⇒ |A|.
• If ∆; Γ ` E ⇒ A ; e then |∆|; |Γ| ` e ⇒ |A|.

• If ∆; Γ ` E ⇐ A ; e then |∆|; |Γ| ` e ⇐ |A|.
Proof. By induction on the typing judgment.
•
T-tyDecl

fresh
α 7→

`+

false

∆; α 7→ ` A ⇒ A1∆; α 7→ ` B ⇒ B1
B1 ⇒ B2∆, X hα, i 7→ A1 & B2 ; Γ ` P ⇒ C ; e∆; Γ ` type X hα i extends A = B; P ⇒ C ; e|∆|; |Γ| ` e ⇒ |C|

By i.h.

•T-tmDecl∆; Γ ` E ⇒ A ; e1∆; Γ, x : A ` P ⇒ B ; e2∆; Γ ` x = E ; P ⇒ B ; let x : |A| = e1 in e2|∆|; |Γ| ` e1 ⇒ |A||∆|; |Γ|, x : |A| ` e2 ⇒ |B||∆|; |Γ| ` let x : |A| = e1 in e2 ⇒ |B|

By i.h.
By i.h.
By TT-let• T-top, T-nat, and T-var are immediate.
•
T-app∆; Γ ` E1 ⇒ A1 → A2 ; e1∆; Γ ` E2 ⇐ A1 ; e2∆; Γ ` E1 E2 ⇒ A2 ; e1 e2|∆|; |Γ| ` e1 ⇒ |A1 | → |A2 ||∆|; |Γ| ` e2 ⇐ |A2 ||∆|; |Γ| ` e1 e2 ⇒ |A2 |By i.h.
By i.h.
By TT-app•
T-anno∆; • ` A ⇒ B∆; Γ ` E ⇐ B ; e∆; Γ ` E : A ⇒ B ; e : |B||∆|; |Γ| ` e ⇐ |B||∆|; |Γ| ` e : |B| ⇒ |B|

By i.h.
By TT-anno

•
T-rcd

∆; Γ ` E ⇒ A ; e∆; Γ ` {l = E } ⇒ {l : A} ; e|∆|; |Γ| ` e ⇒ |A||∆|; |Γ| ` {l = e } ⇒ {l : |A|}

By i.h.
By TT-rcd•
T-proj∆; Γ ` E ⇒ {l : A} ; e∆; Γ ` E.l ⇒ A ; e|∆|; |Γ| ` e ⇒ {l : |A|}|∆|; |Γ| ` e.l ⇒ |A|

By i.h.
By TT-proj•
T-tabs∆; • ` A ⇒ A1

∆ , α ∗ A1 ; Γ ` E ⇒ B ; e∆; Γ ` Λ(α ∗ A).E ⇒ ∀(α ∗ |A1 |).B ; e|∆| ` |A1 ||∆|; |Γ|, α ∗ |A| ` e ⇒ |B||∆|; |Γ| ` Λ(α ∗ |A|). e ⇒ ∀(α ∗ |A|). |B|

By Lemma 2.5
By i.h.
By TT-tabs•
T-tapp∆; • ` A ⇒ A1∆; Γ ` E ⇒ ∀(α ∗ B).C ; e∆ ` A1 ∗ B∆; Γ ` E @A ⇒ [A1 /α ]C ; e |A1 ||∆| ` |A1 ||∆|; |Γ| ` e ⇒ ∀(α ∗ |B|).|C||∆|; |Γ| ` |A1 | ∗ |B||∆|; |Γ| ` e |A| ⇒ [|A1 |/α ]|C|By Lemma 2.5
By i.h.
By Lemma 2.8
By TT-tapp

•
T-let

∆; • ` A ⇒ A1∆; Γ, x : A1 ` E1 ⇐ A1 ; e1∆; Γ, x : A1 ` E2 ⇒ B ; e2∆; Γ ` let x : A = E1 in E2 ⇒ B ; let x : |A1 | = e1 in e2|∆| ` |A1 ||∆|; |Γ|, x : |A1 | ` e1 ⇐ |A||∆|; |Γ|, x : |A1 | ` e2 ⇒ |B||∆|; |Γ| ` let x : |A1 | = E1 in E2 ⇒ |B|

By Lemma 2.5
By i.h.
By i.h.
By TT-let•T-mergeTrait∆; Γ ` E1 ⇒ Trait[A1 , B1 ] ; e1∆; Γ ` E2 ⇒ Trait[A2 , B2 ] ; e2

∆ ` B1 ∗ B2∆; Γ ` E1 , , E2 ⇒ Trait[A1 & A2 , B1 & B2 ] ; λ(self : |A1 & A2 |). e1 self , , e2 self|∆|; |Γ| ` e1 ⇒ |A1 → B1 ||∆|; |Γ| ` e2 ⇒ |A2 → B2 ||∆|; |Γ|, self : |A1 & A2 | ` self ⇒ |A1 & A2 |By i.h.
By i.h.
By TT-var

|A1 & A2 | <: |A1 ||∆|; |Γ| ` self ⇐ |A1 ||∆|; |Γ| ` e1 self ⇒ |B1 ||A1 & A2 | <: |A2 ||∆|; |Γ| ` self ⇐ |A2 ||∆|; |Γ| ` e2 self ⇒ |B2 ||∆|; |Γ| ` e1 self , , e2 self ⇒ |B1 & B2 ||∆|; |Γ| ` λ(self : |A1 & A2 |). e1 self , , e2 self ⇒ |A1 & A2 | → |B1 & B2 |By TS-andL
By TT-sub
By TT-app
By TS-andR
By TT-sub
By TT-app
By TT-merge
By TT-abs•
T-merge∆; Γ ` E1 ⇒ A1 ; e1∆; Γ ` E2 ⇒ A2 ; e2

∆ ` A1 ∗ A2∆; Γ ` E1 , , E2 ⇒ A1 & A2 ; e1 , , e2|∆|; |Γ| ` e1 ⇒ |A||∆|; |Γ| ` e2 ⇒ |B||∆|; |Γ| ` |A| ∗ |B||∆|; |Γ| ` e1 , , e2 ⇒ |A| & |B|

By i.h.
By i.h.
By Lemma 2.8
By TT-merge

•
T-new

∆; Γ ` E ⇒ Trait[A, B] ; e

B <: A∆; Γ ` new E ⇒ B ; let self : |B| = e self in self|∆|; |Γ| ` e ⇒ |A| → |B||B| <: |A||∆|; |Γ|, self : |B| ` self ⇒ |B||∆|; |Γ|, self : |B| ` self ⇐ |A||∆|; |Γ|, self : |B| ` e self ⇐ |B||∆|; |Γ| ` let self : |B| = e self in self ⇒ |B|

By i.h.
By Lemma 2.7
By TT-var
By TT-sub
By TT-app
By TT-let•T-open∆; Γ ` E1 ⇒ {li : Ai } ; e1∆; Γ, li : Ai ` E2 ⇒ e2 ; B∆; Γ ` open E1 in E2 ⇒ B ; let x = e1 in let li : |A| = x.li in e2|∆|; |Γ| ` e1 ⇒ |{li : Ai }||∆|; |Γ| ` e1 .li ⇒ |Ai ||∆|; |Γ|` let li : |Ai | = e1 .li in e2 ⇒ |B|

By i.h.
By TT-proj
By TT-let•
T-trait∆; • ` A ⇒ A1∆; • ` B ⇒ B1∆; Γ, self : A1 ` E1 ⇒ Trait[A2 , B2 ] ; e1
A1 <: A2∆; Γ, self : A1 , super : B2 ` E2 ⇒ C ; e2
C ∗ B2
C & B2 <: B1∆; Γ ` trait[self : A] implements B inherits E1 => E2 ⇒ Trait[A1 , C & B2 ]; λ(self : |A1 |). let super = e1 self in e2 , , super|∆| ` |A1 ||∆| ` |B1 ||∆|; |Γ|, self : |A1 | ` e1 ⇒ |A2 | → |B2 ||A1 | <: |A2 ||∆|; |Γ|, self : |A1 |, super : |B2 | ` e2 ⇒ |C||∆|; |Γ|, self : |A1 | ` self ⇒ |A1 ||∆|; |Γ|, self : |A1 | ` self ⇐ |B1 ||∆|; |Γ|, self : |A1 | ` e1 self ⇒ |B2 ||∆|; |Γ|, self : |A1 |, super : |B2 | ` e2 , , super ⇒ |C| & |B2 ||∆|; |Γ| ` let super = e1 self in e2 , , super ⇒ |C| & |B2 ||∆|; |Γ| ` λ(self : |A1 |). let super = e1 self in e2 , , super ⇒ |A1 | → (|C| & |B2 |)•T-forward∆; Γ ` E1 ⇒ Trait[A, B] ; e1∆; Γ ` E2 ⇐ A ; e2∆; Γ ` E1 ˆE2 ⇒ B ; e1 e2|∆|; |Γ| ` e1 ⇒ |A| → |B||∆|; |Γ| ` e2 ⇐ |A||∆|; |Γ| ` e1 e2

By i.h.
By i.h.
By TT-app•
T-abs∆; Γ, x : A ` E ⇐ B ; e∆; Γ ` λx.E ⇐ A → B ; λx.E|∆|; |Γ|, x : |A| ` e ⇐ |B||∆|; |Γ| ` λ(x : e ). ⇐ |A| → |B|

By i.h.
By TT-abs•
T-sub∆; Γ ` E ⇒ B ; e

B <: A∆; Γ ` E ⇐ A ; e|∆|; |Γ| ` e ⇒ |B||B | < : |A ||∆|; |Γ| ` e ⇐ |B|

By i.h.
By Lemma 2.7
By TT-sub

By Lemma 2.5
By Lemma 2.5
By i.h.
By Lemma 2.7
By i.h.
By TT-var
By TT-sub
By TT-tapp
By TT-merge
By TT-let
By TT-abs□