How To Make a New Programming Language
By: Ken Shibata
Tags: Parser, Lexer, Programming Languages, Compilers, Go

v1.1: Changed some wording and added more comprehensive definitions

In this article, we’ll be making a new, simple toy programming language and a compiler for it. This toy language, Cytoplasm, will probably have no practical use, but it is a good one to start learning about and implementing compilers, parsers, and programming languages. Our language will be fully procedural, and only support floats/integers as the only data structure.

Cytoplasm will only be able to call functions (just like in math: sin(180) ) and use numbers (integers and rational numbers):

out(add(1,2)) 3

But, we’ll make the compiler (a program to change the above code into machine code, there will be a detailed definiton later) to be expandable, so you can implement your own functions like sin , cos , and tan .

The compiler’s source code is available here.

Why do I need to learn this?

You will probably never make a programming language, but making your own (or remixing one) is a shorter way to learn about them instead of reading a 700 page textbook (although the amount learned will be smaller and more shallow). Getting a shallow understanding of parsers and compiler will be useful when programming in general, and also helps explain a lot about language design. For example, why do most modern languages use reserved words (words which cannot be variables like if ), like Python and JavaScript? This is because having reserved words makes parsing, compiling, and optimizing easier (since values are determined at compile time instead of runtime). Having some knowledge about it can answer these types of questions.

Quick Details

The implementation shown in this article will support basic I/O and basic mathematical functions, such as in , out , add , multiply , modulo , pow , and round .

Parsers, Lexers, & Compilers

What are lexers? Lexers essentially break up input into pieces called tokens, which can be read by the parser, which interprets the tokens into a tree structure, called an AST. Compilers are programs that convert a program in one type to another type, without losing its meaning.

(this is Python)

Source: print('Python is dynamically typed.')

↓ Lexer

Tokens: id(print) open_paren string('Python is dynamically typed.') close_paren

↓ Parser

AST: function_call(print,args('Python is dynamically typed.'))

↓ Compiler

Bytecode: ??????????????????? (not real bytecode)

Compilers, don’t have to convert to bytecode ( javac , CPython) or binary (GCC, LLVM (usually)). It simply means a program to convert a program into another form (language) without changing it’s meaning. That means that a compiler from our language can still be called a compiler, but it’s better to call it a front end or a transpiler, subsets of compilers.

(this is simplified Go)

fmt.Println("Hello! 1+1 is ", 1+1)

↓ Front End

println("Hello! 1+1 is " + string(1+1))

↓ Optimizer (and others)

println("Hello! 1+1 is 2")

↓ Back End

?????????????????????????? (not real machine code)

The Implementation

We’re going to use software called participle for our parser and lexer, and we’re going to make our own compiler/transpiler to Go. parser/ast.go defines the AST template for the parser and regex patterns for the lexer.

Regex is a language used to describe patterns in text like “match all words starting with a” would be a.* . a matches “a”, and .* means any character . , and matching it zero or more times * .

Ident stands for identifier which matches strings like add . This becomes [a-zA-z][a-zA-Z0-9]* in regex: [a-zA-Z] matches any character of the alphabet, and [a-zA-Z0-9]* matches any character in the alphabet plus digits, zero or more times (there is a more succinct way of writing this, but this is more straightforward). Num matches any number with or without a decimal point, and Comma matches a comma. OParen and CParen matches opening and closing parentheses, used in functions: add(1,2) .

// (edited for clarity)

var lexer = lexer.Must(lexer.Regexp(

// (omitted)

`|(?P<Ident>[a-zA-Z][a-zA-Z0-9]*)` +

`|(?P<Num>\d+(?:\.\d+)?|\d+)` +

`|(?P<Comma>,)` +

`|(?P<OParen>\()` +

`|(?P<CParen>\))`,

))

Once we use participle.Build(&AST{},participle.Lexer(lexer)) and parser.Parse(src, ast) , we got our AST! Now, we need to feed this into a compiler to print out Go (a programming language) code. This is done in compiler/main.go and main.go . The compiler will output the compiled Go code, and the program will run using a Go interpreter (for the sake of simplicity, not performance). Here is an example of a cube root calculator:

out(pow(in(),divide(1/3)))

As a mathematical expression:

let x represent the input

x1/3

When we execute this program, this will output: