Telling a programmer there’s already a library to do X is like telling a songwriter there’s
already a song about love.
Pete Cordell

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

1

tc::json::parser

Extract some JSON members
40

Time in ms

30

fully validating pull parser
fast, O(1) memory usage
implicitly define schema

20

10

0
tc

RapidJSON
nlohmannsimdjson
Boost.JSON

Library

github.com/think-cell/think-cell-library

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

2

(JSON) Parser design

What is a parser?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

3

Javascript Object Notation (JSON)

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

4

Javascript Object Notation (JSON)

Idea: Human-readable serialization of Javascript objects.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

5

Javascript Values
Primitive Values:
null, undefined

Boolean: true, false
Strings: "hello", 'world'
Numbers: 0, -17, 3.1415, 1e10

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

6

Javascript Values
Primitive Values:
null, undefined

Boolean: true, false
Strings: "hello", 'world'
Numbers: 0, -17, 3.1415, 1e10

Objects: Collection of properties (key-value pairs)
Custom objects: {id: 42, name: "Jonathan Müller", speaker: true}
Arrays (keys are indices): [5, "abc", undefined]
Functions…

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

6

JSON
“Subset” of Javascript literals.
{"id": 42,
"name": "Jonathan Müller","speaker": true,"talks": [{"title": "Express your expectations","subtitle": "A fast, compliant JSON pull parser..."}]}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

7

JSON grammar

value = 'null' | 'true' | 'false' | number | string | array | object

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

8

JSON grammar

value = 'null' | 'true' | 'false' | number | string | array | object
string = '"' characters-or-escape '"'Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

8

JSON grammar

value = 'null' | 'true' | 'false' | number | string | array | object
string = '"' characters-or-escape '"'number = '-'? digits ('.' digits)? (('e'|'E') ('+'|'-')? digits)?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

8

JSON grammar

value = 'null' | 'true' | 'false' | number | string | array | object
string = '"' characters-or-escape '"'number = '-'? digits ('.' digits)? (('e'|'E') ('+'|'-')? digits)?
array = '[' element-list? ']'element-list = value (',' element-list)?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

8

JSON grammar

value = 'null' | 'true' | 'false' | number | string | array | object
string = '"' characters-or-escape '"'number = '-'? digits ('.' digits)? (('e'|'E') ('+'|'-')? digits)?
array = '[' element-list? ']'element-list = value (',' element-list)?
object = '{' member-list? '}'member-list = member (',' member-list)?
member = string ':' value

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

8

JSON parser (skipper)
void value(const char* str) {if (skip_if(str, "null") || skip_if(str, "true") || skip_if("false")) {}else if (*str == '-' || isdigit(*str)) number(str);
else if (*str == '"') string(str);
else if (*str == '[') array(str);
else if (*str == '{') object(str);
else error();}void object(const char* str) {skip(str, '{');
while (!skip(str, '}')) {string(str); skip(str, ":");
value(str); skip_if(str, ",");}}
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

9JSON parsing in Javascript{ "id": 42, "name": "Jonathan…", "speaker": true, "talks": […

] }let object = JSON.parse(json)
console.log(object.name)console.log(object.talks[0].subtitle)
console.log(object.foo)
Jonathan Müller
A fast, compliant JSON pull parser for writing robust applications
undefined

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

10JSON parsing in C++

auto parse_json(std::ranges::input_range auto json)-> ???;

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

11JSON DOM parser

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

12JSON DOM parser

Idea: Represent a JSON value in C++.
auto parse_json(std::ranges::input_range auto json)-> std::expected<json_value, error>;Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

13

Mapping JSON values to C++

Null: std::nullptr_t

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool
String: std::string

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool
String: std::string
Number: std::int64_t, std::uint64_t, double, or big ints?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool
String: std::string
Number: std::int64_t, std::uint64_t, double, or big ints?
Array: std::vector<json_value>

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool
String: std::string
Number: std::int64_t, std::uint64_t, double, or big ints?
Array: std::vector<json_value>
Object: [std/boost]::unordered_map<std::string, json_value>

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Mapping JSON values to C++

Null: std::nullptr_t
Boolean: bool
String: std::string
Number: std::int64_t, std::uint64_t, double, or big ints?
Array: std::vector<json_value>
Object: [std/boost]::unordered_map<std::string, json_value>

using json_value = std::variant<std::nullptr_t, bool, …>;

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

14

Boost.JSON API

boost::json::value parse(std::string_view json); // null on syntax error

This library focuses on a common and popular use-case: parsing and serializing to andfrom a container called value which holds JSON types. […] The value container isdesigned to be well suited as a vocabulary type appropriate for use in public interfaces
and libraries, allowing them to be composed.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

15

Boost.JSON API usage{ "id": 42, "name": "Jonathan…", "speaker": true, "talks": […

] }auto object = boost::json::parse(json).as_object();
std::print("{}\n", object.at("name"));std::print("{}\n", object.at("talks").as_array()[0].as_object().at("subtitle"));std::print("{}\n", object.at("foo"));Jonathan Müller
A fast, compliant JSON pull parser for writing robust applications
Uncaught exception.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

16

Problems with JSON DOM parsers

json_value is stringly typed,

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

17

Problems with JSON DOM parsers

json_value is stringly typed,

which requires dynamic memory allocation,

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

17

Problems with JSON DOM parsers

json_value is stringly typed,

which requires dynamic memory allocation,
manual error handling of the JSON content,

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

17

Problems with JSON DOM parsers

json_value is stringly typed,

which requires dynamic memory allocation,
manual error handling of the JSON content,
and is completely unnecessary in most situations!

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

17Actual data structure
struct conference_attendee{
std::string name;bool is_speaker = false;
std::vector<conference_talk> talks;…
};
struct conference_talk{
std::string title, subtitle;…
};

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

18Full usage
auto parse_json(std::ranges::input_range auto json)-> std::expected<json_value, error>{// provided by JSON library}
auto parse_conference_attendee(const json_value& json)-> std::expected<conference_attendee, error>{// hand-written by author of conference_attendee}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

19

JSON DOM is (mostly) useless!

Useful in two situations:

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

20

JSON DOM is (mostly) useless!

Useful in two situations:1

You implement generic algorithms over arbitrary JSON without a compile-time known schema(e.g. jq).

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

20

JSON DOM is (mostly) useless!

Useful in two situations:1

You implement generic algorithms over arbitrary JSON without a compile-time known schema(e.g. jq).

2

You need to do multiple passes over the JSON content and the parser is slow.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

20

JSON SAX parser

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

21

JSON SAX parser

Idea: Parser invokes handler.
auto parse_json(std::ranges::input_range auto json, json_handler& handler)-> std::optional<error>;Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

22

Parser generates a sequence of events
object_begin,{

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id")"id":

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,key("name"), string("Jonathan…"),"name": "Jonathan Müller",

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,key("name"), string("Jonathan…"),"name": "Jonathan Müller",key("speaker"), bool(true),"speaker": true,

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,key("name"), string("Jonathan…"),"name": "Jonathan Müller",key("speaker"), bool(true),"speaker": true,key("talks"), array_begin,"talks": [Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,key("name"), string("Jonathan…"),"name": "Jonathan Müller",key("speaker"), bool(true),"speaker": true,key("talks"), array_begin,"talks": [object_begin,{

key("title"), string("…")"title": "Express…",

key("subtitle"), string("…")"subtitle": "A…"

object_end}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

23

Parser generates a sequence of events
object_begin,{

key("id"), number(42),"id": 42,key("name"), string("Jonathan…"),"name": "Jonathan Müller",key("speaker"), bool(true),"speaker": true,key("talks"), array_begin,"talks": [object_begin,{

key("title"), string("…")"title": "Express…",

key("subtitle"), string("…")"subtitle": "A…"

object_end}]}

Jonathan Müller — @foonathan

array_end
object_end

Express your expectations

CppCon2023-10-03

23

JSON handler
struct json_handler{
virtual void on_null() {}virtual void on_bool(bool v) {}virtual void on_number(std::int64_t v) {}virtual void on_number(std::uint64_t v) {}virtual void on_number(double v) {}virtual void on_string(std::string_view str) {}virtual void on_array_begin() {}virtual void on_array_end() {}virtual void on_object_begin() {}virtual void on_key(std::string_view key) {}virtual void on_object_end() {}};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

24

RapidJSON SAX API

class Reader{
public:template <typename InputStream, typename Handler>bool Parse(InputStream& is, Handler& handler);};

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

25

SAX API usage
struct print_name_and_subtitle final : json_handler{
std::vector<std::string_view> object_stack;std::string_view last_key;
void on_string(std::string_view str) override {if (object_stack == {""} && last_key == "name")
std::print("{}\n", str);
else if (object_stack == {"", "talks"} && last_key == "subtitle")
std::print("{}\n", str);}
void on_object_begin() override { object_stack.push_back(last_key); }void on_object_end() override { object_stack.pop_back(); }void on_key(std::string_view key) override { last_key = key; }};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

26

Problems with JSON SAX parsers

They're incredibly annoying to use!
Awkward to return results
Context blindness
State machine

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

27

Range Algorithms

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

28

Goal

Input: std::vector<int>
1

Remove odd integers.

2

Square remaining integers.

Output: First square that is divisible by 3.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

29

1. Hand-written implementation
std::optional<int> compute(const std::vector<int>& vec){for (auto i : vec){
if (i % 2 != 0)
continue;
auto square = i * i;if (square % 3 == 0)
return square;}
return std::nullopt;}
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-0330

1. Hand-written implementation

Pro: Intuitive assembly.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

31

1. Hand-written implementation

Pro: Intuitive assembly.
Pro: Lazy computation.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

31

1. Hand-written implementation

Pro: Intuitive assembly.
Pro: Lazy computation.
Con: Not composable.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

31

2. C++98 algorithms

Idea: Identify atomic, reusable algorithm parts.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

32

2. C++98 algorithms
std::optional<int> compute(const std::vector<int>& vec){
auto copy = vec;
std::erase_if(copy, [](int i) { return i % 2 != 0; });std::transform(copy.begin(), copy.end(), copy.begin(),[](int i) { return i * i; });auto iter = std::find_if(copy.begin(), copy.end(),[](int square) { return square % 3 == 0; });return iter == copy.end() ? std::nullopt : std::optional(*iter);}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

33

2. C++98 algorithms

Pro: Composition of smaller building blocks.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

34

2. C++98 algorithms

Pro: Composition of smaller building blocks.
Con: Intermediate allocations.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

34

2. C++98 algorithms

Pro: Composition of smaller building blocks.
Con: Intermediate allocations.
Con: Eager computation.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

34

3. Internal iteration

Idea: Don’t store intermediate results, immediately pass them to caller.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

35

3. Internal iteration
template <typename Rng, typename Sink>
void for_each(const Rng& rng, Sink sink){
if constexpr (std::ranges::input_range<Rng>){for (const auto& x : rng) try {sink(x);} catch (break_exception) {}
}
else
{
try {
rng(sink);} catch (break_exception) {}}}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

36

3. Internal iteration

template <typename Rng, typename Fn>
auto filter(const Rng& rng, Fn predicate){
return [=](auto& sink) {for_each(rng, [&](const auto& x) {if (predicate(x))
sink(x);});};}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

37

3. Internal iteration

template <typename Rng, typename Fn>
auto transform(const Rng& rng, Fn fn){
return [=](auto& sink) {for_each(rng, [&](const auto& x) {sink(fn(x));
});
};
}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

38

3. Internal iteration
std::optional<int> compute(const std::vector<int>& vec){
std::optional<int> result;auto even_squares = transform(
filter(vec, [](int i) { return i % 2 == 0; }),[](int i) { return i * i; });for_each(even_squares, [](int square) { // can't use find_if
if (square % 3 == 0) {result = square;throw break_exception{};}});return result;}
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

39

3. Internal iteration

Pro: Composition of smaller building blocks.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

40

3. Internal iteration

Pro: Composition of smaller building blocks.
Pro: Lazy computation.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

40

3. Internal iteration

Pro: Composition of smaller building blocks.
Pro: Lazy computation.
Con: Can’t use native for, range algorithms.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

40

3. Internal iteration

Pro: Composition of smaller building blocks.
Pro: Lazy computation.
Con: Can’t use native for, range algorithms.
Con: Short-circuit more complicated (exceptions, error channel)

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

40

4. C++20 iterator ranges

Idea: Lazily compute value when requested.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

41

4. C++20 iterator ranges
template <typename Rng, typename Predicate>struct filter_view {Rng _base;Predicate _pred;
struct iterator {filter_view* _parent;
ranges::iterator_t<Rng> _cur;
auto& operator*() const { return *_cu; }iterator& operator++() {do {++_cur;} while (_cur != _parent->end() && !_parent->_pred(*_cur));}};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

42

4. C++20 iterator ranges
template <typename Rng, typename Fn>struct transform_view {Rng _base;
Fn _fn;struct iterator {transform_view* _parent;
ranges::iterator_t<Rng> _cur;decltype(auto) operator*() const { return _parent->_fn(*_cur); }iterator& operator++() { ++_cur; }};};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

43

4. C++20 iterator ranges
std::optional<int> compute(const std::vector<int>& vec){
auto even_squares = vec| stdv::filter([](int i) { return i % 2 == 0; })| stdv::transform([](int i) { return i * i; });auto iter = stdr::find_if(even_squares,[](int square) { return square % 3 == 0; });return iter == even_squares.end()? std::nullopt : std::optional(*iter);}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

44

Aside: Customizable range return in think-cell algorithms

std::optional<int> compute(const std::vector<int>& vec){
auto even_squares = vec| stdv::filter([](int i) { return i % 2 == 0; })| stdv::transform([](int i) { return i * i; });return tc::find_first_if<tc::return_value_or_none>(even_squares,[](int square) { return square % 3 == 0; });}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

45

4. C++20 iterator ranges

Pro: Composition of smaller building blocks.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

46

4. C++20 iterator ranges

Pro: Composition of smaller building blocks.
Pro: Lazy computation.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

46

4. C++20 iterator ranges

Pro: Composition of smaller building blocks.
Pro: Lazy computation.
Pro: Can use native for, range algorithms.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

46

What does this have to do with JSON parsing?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

47

What does this have to do with JSON parsing?

1

Hand-written implementation

Jonathan Müller — @foonathan

1

Express your expectations

Hand-written parser/serializer

CppCon2023-10-03

47

What does this have to do with JSON parsing?

1
2

Hand-written implementation
C++98 algorithms

Jonathan Müller — @foonathan

12

Express your expectations

Hand-written parser/serializer
JSON DOM parser

CppCon2023-10-03

47

What does this have to do with JSON parsing?

1
2
3

Hand-written implementation
C++98 algorithms
Internal iteration

Jonathan Müller — @foonathan

123

Express your expectations

Hand-written parser/serializer
JSON DOM parser
JSON SAX parser

CppCon2023-10-03

47

What does this have to do with JSON parsing?

1
2
3
4

Hand-written implementation
C++98 algorithms
Internal iteration
C++20 iterator ranges

Jonathan Müller — @foonathan

1
2
3
4

Express your expectations

Hand-written parser/serializer
JSON DOM parser
JSON SAX parser???

CppCon2023-10-03

47

Missing JSON parser abstraction

What is the equivalent of iterator ranges for parsing?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

48

Push-model: Iteration and JSON parser
Push model: Algorithm computes data and invokes handler.
struct range_sink

struct handler{{
void on_null();void operator()(auto const& x);

void on_bool(bool v);
void on_number(std::int64_t v);
void on_string(string_view str);
…
};

};

Algorithm is in control.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

49

Pull-model: Iteration and JSON parser

Pull model: Algorithm computes next data when requested by user.
struct iterator{
T& operator*() const;???

iterator& operator++();
bool operator==(sentinel) const;};

User is in control.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

50

Pull-model: Iteration and JSON parser

Pull model: Algorithm computes next data when requested by user.
struct iterator{???

std::optional<T> item();};

User is in control.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

50

JSON pull parser

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

51

JSON pull parser

Idea: Parser is driven by user.
template <typename Rng, typename ErrorHandler>
struct tc::json::parser{…
};

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

52

Pulling primitive values

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{bool null();
std::optional<bool> boolean();};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

53

Pulling primitive valuesbool parser::null() {if (skip_if(cur, end, "null")) {return true;} else {return false;}}
std::optional<bool> parser::boolean() {if (skip_if(cur, end, "true") {return true;} elseif (skip_if(cur, end, "false")) {return false;} else {return std::nullopt;}}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

54

Pulling strings

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto string() -> std::optional<???>;};

If ", eagerly advance until closing ", and return … what?

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

55

Pulling strings

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto string() -> std::optional<???>;};

If ", eagerly advance until closing ", and return … what?
std::string_view/std::ranges::subrange: can’t handle escape sequences

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

55

Pulling strings

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto string() -> std::optional<???>;};

If ", eagerly advance until closing ", and return … what?
std::string_view/std::ranges::subrange: can’t handle escape sequences
std::string: allocation and copy

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

55

Pulling strings

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto string() -> std::optional<???>;};

If ", eagerly advance until closing ", and return … what?
std::string_view/std::ranges::subrange: can’t handle escape sequences
std::string: allocation and copy
tc::json::decode_adaptor: lazily decode escape sequences as you iterate

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

55

Pulling strings"Hello\nWorld!"
if (auto string = p.string()) {tc::for_each(*string, [](char c) {std::print("{}", c);
});
}
Hello
World!

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

56

Pulling numbers

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto number() -> std::optional<decode_adaptor>; // as string
template <typename T>
auto number() -> std::optional<T>: // as number};

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

57

Pulling numbers
Get as raw characters:
if (auto number = p.number()) {tc::for_each(*number, [](char c) {…
});}Or as specific type:if (std::optional<int> number = p.number<int>()) {…
}if (std::optional<float> number = p.number<float>()) {…
}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

58

Pulling arrays

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{bool array();// skip [bool element(); // skip ] or ,};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

59

Pulling arrays

if (p.array()) {while (p.element()) {if (auto str = p.string()) {…
} else … {…
}}}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

60

Pulling objects

template <typename Rng, typename ErrorHandler>
struct tc::json::parser{bool object();// skip {

std::optional<decode_adaptor> key(); // skip } or , and key};

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

61

Pulling objects

if (p.object()) {while (auto key = p.key()) {if (auto value = p.string()) {…
} else … {…
}}}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

62

Using a pull parser to validate JSON
void parser::skip_value() {if (this->null()) {return;} else if (this->boolean()) {return;} else … {…
} else if (this->object()) {while (this->key()) {this->skip_value();}}}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

63

Using a pull parser to create a DOM
json_value parse_value(tc::json::parser& p) {if (p.null()) {return nullptr;} else if (auto boolean = p.boolean()) {return *boolean;} else … {…
} else if (p.object()) {json_object result;while (auto key = p.key()) {result[*key] = parse_value(p);}return result;}}
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

64Using a pull parser to implement a SAX parser
void parse_value(tc::json::parser& p, json_handler& handler) {if (p.null()) {handler.on_null();} else if (auto boolean = p.boolean()) {handler.on_bool(*boolean);} else … {…
} else if (p.object()) {handler.on_object_begin();while (auto key = p.key()) {handler.on_key(*key);parse_value(p, handler);}handler.on_object_end();}}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

65

Pull parser

Lowest level of abstraction.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

66

Pull parser

Lowest level of abstraction.
Real advantage: When you know what to expect!

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

66

The expect_ API
template <typename Rng, typename ErrorHandler>
struct tc::json::parser{
auto expect_null()-> void;

auto expect_boolean() -> bool;
auto expect_string()-> decode_adaptor;

auto expect_number()-> decode_adaptor;

template <typename T>
auto expect_number()-> T;

void expect_array();
void expect_element();
void expect_array_end();void expect_object();};
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

67Error Handlingstruct error_handler{
void char_expected(auto const& input, auto pos, tc::char_ascii char);void null_expected(auto const& input, auto pos);void boolean_expected(auto const& input, auto pos);void number_expected(auto const& input, auto pos);…
};

Throw an exception or assert.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

68Pull parser API usage
p.expect_object();

bjjPMEcvq

while (auto key = p.key()) {if (tc::equal(*key, "name")) {std::print("{}\n", p.expect_string());} else if (tc::equal(*key, "talks")) {p.expect_array(); p.expect_element();
p.expect_object();
while (auto key = p.key()) {if (tc::equal(*key, "subtitle"))std::print("{}\n", p.expect_string());
else
p.skip_value();}p.expect_array_end();} else {p.skip_value();
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

69

JSON pull vs SAX parser

Pull

SAX

Easy to return results
Full context available
Straight-forward code

Awkward to return results
Context blindness
State machine

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

70

JSON pull vs SAX parser

Pull

SAX

Easy to return results
Full context available
Straight-forward code

Awkward to return results
Context blindness
State machine

Current API is still a bit annoying to use!

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

70

Pull parser algorithms

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

71

Array ranges
hz3hWzsn9[1, 2, 3]Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

72

Array ranges
hz3hWzsn9[1, 2, 3]auto ints = p.expect_array([&] {return p.expect_number<int>();});
auto vec1 = ints | std::ranges::to<std::vector<int>>;// C++23

auto vec2 = tc::explicit_cast<std::vector<int>>(ints); // think-cell

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

72

Array ranges: Coroutine implementation

template <typename Func>
std::generator<std::invoke_result_t<Func>> parser::expect_array(Func func){
this->expect_array();
while (this->element()) {co_yield func();}}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

73

Array ranges: Internal iteration

template <typename Func>
auto parser::expect_array(Func func){
this->expect_array();
return [=](auto sink) {while (this->element()) {sink(func());}};}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

74

Object parsing: Expect key?
{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

75

Object parsing: Expect key?
{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;p.expect_object();
p.expect_key("id"); p.skip_value();p.expect_key("name"); tc::cont_assign(result.name, p.expect_string());
if (p.key("speaker")) result.is_speaker = p.expect_boolean();p.expect_key("talks");
tc::cont_assign(result.talks, p.expect_array([&] { … });Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

75

Object parsing: Expect key?
{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;!

p.expect_object();
p.expect_key("id"); p.skip_value();

is
der

ary
r
t
i
b

ar

or
r
e
b
em

p.expect_key("name"); tc::cont_assign(result.name, p.expect_string());

N
JSO

m

if (p.key("speaker")) result.is_speaker = p.expect_boolean();p.expect_key("talks");
tc::cont_assign(result.talks, p.expect_array([&] { … });Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

75

Object: Order insensitive key parsing{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;p.expect_object();
while (auto key = p.key()) {if (tc::equal(*key, "name")) {tc::cont_assign(result.name, p.expect_string());} else if (tc::equal(*key, "speaker")) {…
} else if (tc::equal(*key, "talks")) {…
} else {p.skip_value();}}
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

76

Object: Declarative order insensitive key parsing
5dKrcnvGh{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;p.expect_object(
tc::json::required(tc::named<"name">([&] {tc::cont_assign(result.name, p.expect_string());})),);
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

77

Object: Declarative order insensitive key parsing
5dKrcnvGh{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;p.expect_object(
tc::json::required(tc::named<"name">([&] {tc::cont_assign(result.name, p.expect_string());})),
tc::json::optional(tc::named<"speaker">([&] {result.is_speaker = p.expect_boolean();})),);
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

77

Object: Declarative order insensitive key parsing
5dKrcnvGh{ "id": 42, "name": "Jonathan Müller", "speaker": true, "talks": [ … ] }conference_attendee result;p.expect_object(
tc::json::required(tc::named<"name">([&] {tc::cont_assign(result.name, p.expect_string());})),
tc::json::optional(tc::named<"speaker">([&] {result.is_speaker = p.expect_boolean();})),
tc::json::required(tc::named<"talks">([&] {tc::cont_assign(result.talks, p.expect_array([&] { … }));})));
Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

77

Object: Different types{ "value": { "number": 42 } }{ "value": { "string": "abc" } }p.expect_object(tc::json::required(tc::named<"value">([&] {p.expect_object(tc::json::required(
tc::named<"number">([&] { … = p.expect_number(); }),tc::named<"string">([&] { … = p.expect_string(); }),
…
))
})));

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

78

Object: Order insensitive key parsing implementation

template <typename ... Groups>
void parser::expect_object(Groups... groups){
this->expect_object();
while (auto key = this->key()) {if (!(groups.parse(*this, *key) || ...))
this->skip_value();}(groups.finalize(*this), ...);}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

79

Object: Order insensitive key parsing implementation
template <typename ... Keys>
struct required_group {std::tuple<Keys...> keys;
bool parsed = false;bool parse(parser& p, auto const& key_name) {return tc::any_of(keys, [&](auto const& key) {if (!tc::equal(key_name, key.name())) return false;
if (!tc::change(parsed, true)) { /* error */ }key.value();return true;});}
void finalize(parser& p) {if (!parsed) { /* error */ }}};Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

80

Convenience: Single member object{ "data": [1, 2, …] }auto values = p.expect_single_member_object(
tc::json::required(tc::named<"data">([&]{
return p.expect_array([&]{
return p.expect_number<int>();}) | stdr::to<std::vector<int>>;})));

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

81Many more algorithms possible

template <typename Func>
std::optional<std::invoke_result_t<Func>> value_or_null(parser& p, Func f){
if (p.null())
return std::nullopt;
else
return f();}Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

82

Many more algorithms possible
template <std::size_t N, typename Func>
std::array<…, N> expect_array_of_size(parser& p, Func f){// invoke f() for each element of the array and collect result}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

83Many more algorithms possible
template <std::size_t N, typename Func>
std::array<…, N> expect_array_of_size(parser& p, Func f){// invoke f() for each element of the array and collect result}
template <std::size_t N, typename Func>
std::tuple<…> expect_tuple(parser& p, Func f){// invoke f(std::integral_constant<std::size_t, Idx>{})// for each element of the array and collect result}

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

83Potential Future: Reflection?

struct conference_attendee{
std::string name;[[tc::json("speaker")]] bool is_speaker = false;
std::vector<conference_talk> talks;};
auto attendee = tc::json::read<conference_attendee>(p);Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

84

(JSON) Parser design

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

85What is a range algorithm?

A range algorithm takes an input range and produces asequence of output values.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

86What is a range algorithm?

A range algorithm takes an input range and produces asequence of output values.
Eager algorithm : Collect all output values in a container.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

86What is a range algorithm?

A range algorithm takes an input range and produces asequence of output values.
Eager algorithm : Collect all output values in a container.
Internal iteration : Invoke callback for each value.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

86What is a range algorithm?

A range algorithm takes an input range and produces asequence of output values.
Eager algorithm : Collect all output values in a container.
Internal iteration : Invoke callback for each value.
Iterator range : User requests new values.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

86What is a parser?

A parser takes an input range and produces a sequence ofparse events.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

87What is a parser?

A parser takes an input range and produces a sequence ofparse events.
DOM/AST: Collect parse events in a container.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

87What is a parser?

A parser takes an input range and produces a sequence ofparse events.
DOM/AST: Collect parse events in a container.
SAX/Push parser: Invoke a callback for each event.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

87What is a parser?

A parser takes an input range and produces a sequence ofparse events.
DOM/AST: Collect parse events in a container.
SAX/Push parser: Invoke a callback for each event.
Pull parser: User drives parsing.

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

87

Conclusion
github.com/think-cell/think-cell-library
JSON pull parser
XML pull parser
better ranges, algorithms, string handling, utilities…

We're hiring: think-cell.com/cppcon

jonathanmueller.dev/talk/think-cell-json@foonathan@fosstodon.org
youtube.com/@foonathan

Jonathan Müller — @foonathan

Express your expectations

CppCon2023-10-03

88