Programming

Research

Group

A

PRETTIER

COMPILER-COMPILER

GENERATING

HIGHER

ORDER

PARSERS

IN

C

P.T.

Breuer

J.P.

Bowen

Oxford

University

Computing

Laboratory

Keble

Road

Oxford

A

PREttier

Compiler-Compiler

Generating

Higher

Order

Parsers

in

C

P.T.

Breuer

J.P.

Boweny

Oxford

University

Computing

Laboratory

Programming

Research

Group

Keble

Road

Oxford

UK

Abstract

Top-down

-LRB-

LL

-RRB-

parsers

are

easy

to

express

in

functional

programming

languages

but

the

elegant

functional

programming

model

can

also

serve

as

an

exact

prototype

for

a

real

implementation

of

the

technology

in

ansi

C

The

result

is

a

compiler-compiler

that

takes

unlimited

lookahead

and

backtracking

the

extended

BNF

notation

and

parameterized

grammars

with

-LRB-

higher

order

-RRB-

metaparameters

to

the

world

of

C

programming

The

generated

code

is

standard

ansi

C

and

is

plug

compatible

with

lex-generated

lexical

analyzers

prepared

for

the

unix

yacc

compiler-compiler

In

contrast

to

yacc

however

the

generated

code

is

modular

and

thus

allows

parts

of

scripts

to

be

compiled

separately

and

linked

in

incrementally

but

it

remains

efficient

as

is

demonstrated

by

the

example

occam

parser

treated

in

depth

here

Discussion

No

medium

to

large-scale

software

engineering

project

nowadays

seems

to

be

without

its

proprietary

language

and

the

generic

unix

utility

yacc

-LSB-

-RSB-

is

usually

the

tool

chosen

to

turn

the

language

design

from

an

abstraction

to

a

working

reality

This

is

not

only

because

it

genuinely

does

permit

the

implementation

-LRB-

and

therefore

later

maintenance

and

version

control

-RRB-

to

be

carried

out

at

a

higher

level

but

because

it

has

traditionally

been

seen

as

conferring

some

degree

of

respectability

upon

the

final

product

Visiting

Research

Fellow

British

Telecom

Laboratories

Martlesham

Heath

Ipswich

UK

Email

<Peter.Breuer@comlab.ox.ac.uk>

yFunded

by

the

UK

Information

Engineering

Directorate

SAFEMOS

project

-LRB-

-RRB-

Email

<Jonathan.Bowen@comlab.ox.ac.uk>

On

the

face

of

it

yacc

offers

attractive

programming

options

too

it

implements

a

variety

of

the

BNF

grammar

specification

language

and

compiles

definition

scripts

into

a

C

program

-LSB-

-RSB-

which

implements

a

finite

state

automaton

thus

converting

specification

into

implementation

via

wellunderstood

theory

-LSB-

-RSB-

In

practice

however

the

utilities

drawbacks

may

be

more

obvious

than

its

attractions

The

supported

BNF

is

an

extremely

impoverished

variety

for

example

every

BNF

construct

having

to

be

expanded

out

by

hand

into

the

basic

sequential

and

alternation

components

before

it

can

be

incorporated

into

a

yacc

script

which

obscures

the

specification

and

makes

maintenance

difficult

This

is

true

too

of

more

modern

implementations

of

yacc

and

also

bison

-LRB-

a

similar

widely

distributed

utility

available

from

the

GNU

Software

Foundation

which

use

improved

internal

algorithms

-LRB-

see

-LSB-

-RSB-

for

example

-RRB-

but

possess

essentially

the

same

scripting

language

and

functionality

One

approach

to

improving

on

yacc

to

parse

nonLR

-LRB-

k

-RRB-

grammars

requiring

infinite-lookahead

would

be

to

modify

yacc

itself

This

is

the

approach

taken

in

-LSB-

-RSB-

in

order

to

handle

C++

-LSB-

-RSB-

adequately

by

allowing

dynamic

stack

allocation

but

it

is

not

very

elegant

Even

the

original

author

of

yacc

has

produced

a

prototype

tool

y

to

handle

C++

using

the

notion

of

attribute

grammars

-LSB-

-RSB-

Another

approach

as

described

in

this

paper

is

to

produce

a

considerably

more

versatile

tool

from

scratch

that

is

designed

to

accept

higher

level

grammar

descriptions

More

modern

languages

such

as

C++

which

require

more

complicated

parsing

than

older

languages

may

be

the

key

to

the

widespread

acceptance

of

improved

parser

technology

although

both

issues

drive

each

other

include

ccx

h

define

INIT

V

-LRB-

-RRB-

define

STEP

V

-LRB-

-RRB-

V

-LRB-

-RRB-

define

TOTL

printf

-LRB-

d

terms

OKnnn

Next

terms

are

d

d

nn

V

-LRB-

-RRB-

a

b

-RRB-

MAIN

-LRB-

fibs

-RRB-

ffl

fibs

INIT

fib

-LRB-

-RRB-

fibs

ffl

fib

-LRB-

a

b

-RRB-

number

-LRB-

a

-RRB-

h

i

STEP

fib

-LRB-

b

a

b

-RRB-

j

h

i

h

i

TOTL

ffl

number

-LRB-

n

-RRB-

digit

-LRB-

n

-RRB-

j

digit

-LRB-

HEAD

-LRB-

n

-RRB-

-RRB-

number

-LRB-

TAIL

-LRB-

n

-RRB-

-RRB-

ffl

digit

-LRB-

n

-RRB-

hn

i

Figure

-LRB-

a

-RRB-

A

precc

parser

which

accepts

only

the

Fibonacci

sequence

as

input

and

pinpoints

errors

to

within

a

character

-LRB-

HEAD

and

TAIL

are

C

macros

-RRB-

terms

OK

Next

terms

are

-LRB-

line

error

-RRB-

failed

parse

probable

error

at

Figure

-LRB-

b

-RRB-

Two

input

lines

and

responses

Declarative

language

issues

In

a

sense

yacc

does

not

correctly

implement

the

fragment

of

the

BNF

syntax

that

it

allows

expression

of

Only

those

definition

scripts

can

be

implemented

that

do

not

implicitly

require

the

automaton

to

look

ahead

more

than

one

token

-LRB-

-RRB-

but

they

can

still

be

specified

In

effect

this

means

that

yacc

does

not

support

declarative

programming

because

the

user

has

to

be

aware

of

the

limitations

of

the

implementation

when

designing

the

specification

Yacc

does

report

some

violations

of

the

restriction

as

shift/reduce

clashes

at

compile-time

-LRB-

or

shift/shift

or

reduce/reduce

-RRB-

which

amounts

to

a

static

check

for

implementability

and

it

may

be

argued

that

this

restricts

the

range

of

expressible

constructs

to

those

that

can

be

implemented

but

the

result

is

still

that

the

declarations

in

the

script

have

to

be

checked

against

the

implementation

semantics

-LRB-

which

is

not

context

free

-RRB-

before

they

can

be

passed

Moreover

the

conflict

reports

refer

to

the

underlying

automaton

and

can

be

extremely

difficult

to

relate

to

what

might

be

wrong

with

the

script

itself

Frequently

some

conflicts

have

to

be

accepted

in

the

final

script

and

since

these

are

resolved

in

the

favour

of

shifts

by

default

this

means

that

extra

-LRB-

and

unspecified

-RRB-

productions

are

introduced

A

yacc

script

with

shift/reduce

conflicts

will

always

implement

a

parser

that

is

looser

than

the

specification

It

can

therefore

be

very

frustrating

for

engineers

hoping

to

use

the

compiler-compiler

for

rapid

prototyping

because

time

has

to

be

spent

debugging

the

automaton

as

well

as

the

script

itself

For

example

a

yacc

script

for

a

comma-separated

sequence

of

x

's

phooey

x

j

x

phooey

may

generate

a

shift/reduce

conflict

report

if

any

other

production

rule

contains

phooey

because

yacc

does

not

know

whether

to

begin

looking

for

an

x

to

follow

the

comma

-LRB-

shift

for

more

phooey

-RRB-

or

to

jump

with

what

it

already

has

into

the

other

production

-LRB-

reduce

-RRB-

The

problem

is

essentially

a

lack

of

referential

transparency

the

same

phooey

construction

can

evoke

different

effects

in

different

scripts

Nowadays

interest

in

the

software

community

has

shifted

towards

syntax-directed

techniques

which

produce

high-quality

software

that

has

few

errors

and

is

easy

to

maintain

-LSB-

-RSB-

and

these

are

precisely

the

issues

which

are

addressed

by

precc

-LSB-

-RSB-

and

the

very

areas

in

which

yacc

technology

shows

its

weakest

aspect

Non-functional

considerations

Some

non-functional

aspects

of

the

yacc

utility

lead

to

further

problems

when

changes

to

a

design

are

required

whether

for

debugging

purposes

or

as

genuine

developments

First

of

all

yacc

scripts

have

to

be

a

single

unit

and

a

change

to

any

part

requires

the

whole

script

to

be

recompiled

again

This

can

be

a

slow

process

because

the

original

unix

versions

of

yacc

are

not

fast

and

even

if

modern

algorithms

for

LALR

-LRB-

-RRB-

parsing

have

speeded

up

some

yacc

implementations

-LSB-

-RSB-

the

monolithic

output

still

has

to

be

sent

to

the

C

compiler

for

conversion

to

native

machine

code

which

can

also

be

a

slow

process

-LRB-

the

compiled

C

code

is

always

a

very

large

file

and

the

C

compiler

is

not

especially

well

suited

to

the

compilation

of

that

kind

of

code

containing

as

it

does

a

very

large

loop

and

switch/case

statement

-RRB-

There

can

therefore

be

a

long

turn-around

time

built

into

the

basic

development

cycle

under

yacc

Secondly

the

resulting

executable

may

also

be

very

large

particularly

under

older

versions

of

yacc

because

of

the

nature

of

the

automaton

and

can

cause

swapping

problems

-LRB-

or

worse

-RRB-

in

the

runtime

environment

which

result

in

a

poor

perceived

performance

for

the

application

as

a

whole

This

can

be

more

than

disappointing

for

a

software

project

which

has

invested

many

programmer-years

of

effort

in

the

application

and

devises

the

front

end

for

the

application

in

the

industry-standard

style

a

yacc-generated

automaton

Malign

influences

It

may

be

observed

that

lex

and

yacc

have

together

acted

to

limit

the

style

of

modern

computer

languages

to

those

that

can

be

handled

easily

by

the

pair

Conforming

languages

have

to

have

syntactic

constructs

that

are

well

signposted

fore

and

aft

with

distinctive

tokens

and

it

is

noticeable

that

languages

invented

before

the

utilities

became

popular

are

not

parsable

by

them

because

they

lack

this

design

Neither

cobol

nor

fortran

can

be

parsed

by

technology

-LRB-

in

practice

this

means

that

a

prior

pass

is

required

to

add

in

scoping

constructs

and

distinctive

signals

which

can

make

a

second

pass

successful

-RRB-

In

fortran

for

example

the

scope

of

the

logical

conditional

if

-LRB-

-RRB-

is

a

single

following

statement

but

a

following

then

makes

multiple

statements

permissible

before

an

endif

terminates

the

scope

Logically

the

then

comes

too

late

in

the

stream

to

signal

to

a

parser

that

everything

is

really

alright

after

all

It

is

possible

to

overcome

the

difficulties

by

setting

flags

as

a

side-effect

but

the

parser

specification

rapidly

loses

its

comprehensibility

as

the

declarative

content

dwindles

Moreover

it

is

possible

in

fortran

to

declare

variables

with

the

same

identifiers

as

keywords

which

makes

keyword

identification

at

the

lexical

stage

impossible

and

raises

obvious

difficulties

for

a

lex/yacc

generated

parser

-LRB-

the

solution

is

to

modify

the

lexer

to

use

lookup

tables

which

are

modified

by

the

parser

-RRB-

And

in

cobol

the

scope

of

record

structure

declarations

is

only

determined

by

numbering

-LRB-

and

conventionally

by

indentation

-RRB-

which

leads

to

equal

difficulties

A

declarative

approach

Developments

in

declarative

languages

such

as

Prolog

-LSB-

-RSB-

and

ML

-LSB-

-RSB-

have

led

to

the

incorporation

of

parser

-LRB-

that

is

compiler

-RRB-

suites

into

the

associated

development

environment

and

these

allow

more

expressive

use

of

BNF-style

descriptions

than

yacc

and

thus

allow

more

languages

to

be

parsed

precisely

When

these

declarative

environments

are

compilable

the

results

from

their

built-in

compilercompiler

facilities

are

very

acceptable

relative

to

yacc

non-functional

overheads

and

much

more

acceptable

from

the

point

of

view

of

the

convenience

and

succinctness

and

accuracy

of

the

definitional

language

itself

Indeed

the

op

-LRB-

-RRB-

pseudo-clause

of

Prolog

allows

enough

flexibility

in

the

definition

of

pre/in/post-fix

operators

together

with

their

associativity

and

precedence

for

an

abstract

tree

input

to

a

compiler

to

be

sufficiently

readable

that

it

is

usable

directly

at

least

for

rapid-prototyping

purposes

-LSB-

-RSB-

In

a

functional

language

one

may

expect

to

express

the

comma-separated

list

of

x

's

given

as

a

yacc

script

above

for

example

as

phooey

x

literal

phooey

x

given

appropriate

definitions

of

the

operators

literal

and

and

this

definition

will

be

absolutely

independent

of

whatever

other

rules

are

defined

because

of

the

referential

transparency

of

functional

languages

and

one

may

expect

to

write

such

a

definition

yet

more

succinctly

using

an

equivalent

of

the

common

BNF

construct

xg

phooey

-LRB-

literal

x

-RRB-

having

made

the

obvious

abstraction

of

from

the

body

of

the

original

phooey

-LRB-

separator

separated

-RRB-

separated

separator

-LRB-

separator

separated

-RRB-

separated

-LRB-

This

is

more

cumbersomely

expressed

than

it

might

be

but

is

presented

this

way

to

provide

an

exact

comparison

with

the

yacc

construction

-RRB-

However

higher

order

language

suites

usually

compile

into

native

machine

code

with

a

consequent

lack

of

portability

between

different

architectures

and

usually

carry

the

considerable

overhead

of

at

least

some

part

of

the

development

environment

with

them

into

the

final

product

Even

if

the

overhead

is

not

considerable

in

terms

of

machine

loading

then

it

still

may

be

considerable

in

financial

terms

and

there

is

always

some

nagging

doubt

about

the

effectiveness

of

the

essentially

recursive

code

no

matter

how

good

the

optimization

is

because

of

the

number

of

layers

of

translation

involved

Therefore

a

niche

remains

to

be

filled

by

a

dedicated

standalone

and

portable

compiler-compiler

targetted

at

ansi

C

-LSB-

-RSB-

which

corrects

some

of

yacc

's

infelicities

and

introduces

no

more

of

its

own

This

is

what

is

claimed

for

the

precc

compiler-compiler

So

how

is

it

done

Design

and

implementation

The

secret

-LRB-

though

hardly

a

secret

to

many

functional

programmers

-RRB-

behind

the

declarative

compiler-compiler

suites

is

top-down

recursive

descent

parsing

-LSB-

-RSB-

and

LL

-LRB-

-RRB-

infinite

look-ahead

technology

Heavily

recursive

in

execution

the

parser

definitions

can

nevertheless

be

made

as

elegant

as

one

cares

to

make

them

In

a

functional

language

all

that

has

to

be

done

is

to

implement

the

higher

order

combinators

that

make

compound

parsers

out

of

simpler

component

parsers

The

combinators

correspond

to

the

operators

and

separators

in

BNF

notation

so

the

gap

between

the

specification

language

-LRB-

BNF

-RRB-

and

the

implementation

as

a

working

parser

can

be

tiny

-LCB-

especially

if

the

functional

language

allows

the

direct

binding

of

the

appropriate

mixfix

symbols

to

these

higher

order

functions

This

method

is

set

out

with

memorable

clarity

by

Jon

Fairbairn

in

-LSB-

-RSB-

but

runtime

considerations

do

not

figure

heavily

in

the

account

The

precc

design

sprang

from

the

belief

that

the

same

method

of

writing

parsers

and

compilers

would

be

feasible

in

C

and

would

result

in

much

more

efficient

practical

parsers

than

possible

in

a

functional

programming

environment

Interestingly

the

method

indeed

turns

out

to

transfer

quite

easily

and

naturally

and

the

generated

C

code

follows

the

form

of

the

intended

grammar

closely

providing

a

natural

modularity

Note

that

precc

as

designed

can

not

use

regular

algebra

and

define

an

automaton

like

yacc

's

because

-LRB-

-RRB-

are

allowed

in

precc

scripts

and

-LRB-

-RRB-

it

permits

context

dependent

grammars

in

that

each

grammar

clause

may

be

parameterised

and

-LRB-

-RRB-

the

parameters

may

even

be

other

nonterminals

or

terminals

which

are

then

implanted

dynamically

into

the

clause

But

the

intention

was

always

to

investigate

whether

the

use

of

a

more

powerful

definitional

language

could

compensate

by

its

greater

succinctness

for

the

extra

speed

of

an

automaton

following

example

defines

a

conditional

parse

of

gum

if

condition

bar

holds

else

a

parse

of

dee

ffl

foo

-RRB-

bar

-LRB-

gum

j

dee

Efficiency

Modularity

and

higher

order

definitions

were

intended

to

compensate

for

the

deficiencies

in

execution

speed

with

respect

to

yacc

After

all

most

of

the

effort

in

writing

languages

does

go

into

maintenance

of

one

kind

or

another

so

it

is

reasonable

to

trade

off

execution

speed

against

maintenance

time

especially

as

speed

is

a

function

of

hardware

nowadays

but

surprisingly

precc-generated

executables

turned

out

to

run

at

least

as

fast

as

those

generated

by

yacc

There

is

no

disadvantage

discernible

for

the

parsers

that

have

been

constructed

-LRB-

in

fact

a

considerable

advantage

is

seen

-RRB-

It

may

be

that

LL

parsers

have

got

a

bad

press

from

old

technology

and

that

nowadays

their

capabilities

ought

to

be

re-evaluated

With

hindsight

this

can

be

attributed

to

the

fact

that

C

compilers

implement

subroutine

calls

and

therefore

recursions

very

efficiently

nowadays

Presumably

this

was

not

altogether

the

case

at

the

time

that

lex

and

yacc

were

becoming

popular

which

is

curious

because

unix

environments

ought

to

provide

an

environment

in

which

large

call-stacks

are

invisible

both

to

the

user

and

the

programmer

being

handled

by

the

operating

system

through

swapping

and

virtual

memory

if

necessary

The

space

overhead

claimed

by

precc

is

not

even

very

great

of

call-stack

is

required

to

process

nested

constructions

in

occam

-LSB-

-RSB-

-LRB-

and

is

the

maximum

in

occam

because

there

are

eighty

characters

to

a

line

and

each

nesting

must

be

indented

by

at

least

two

characters

from

the

last

-RRB-

corresponding

to

about

stacked

function

calls

from

the

kernel

Precc

executables

also

turn

out

to

be

smaller

than

yacc-generated

ones

with

no

sign

of

swapping

problems

Again

with

hindsight

this

can

be

attributed

to

the

higher-order

and

modular

nature

of

precc

output

code

which

ensures

that

fewer

subroutines

are

necessary

and

that

subroutines

have

small

bodies

which

never

approach

a

memory

page

or

even

a

memory

cache

in

size

Moreover

the

functional

model

followed

by

precc

offers

a

way

out

of

the

debugging

morass

of

yacc

's

shift/reduce

errors

all

BNF

constructs

are

implemented

correctly

in

the

model

and

therefore

they

are

implemented

correctly

in

precc

so

there

can

be

no

implementation

conflicts

to

report

Precc

validly

refines

the

recursion

equations

which

hold

in

the

functional

programming

model

-LRB-

there

is

some

extra

strictness

introduced

but

not

in

the

higher-order

combinators

themselves

-RRB-

In

one

practical

matter

too

precc

seems

more

amenable

than

yacc

It

is

easier

by

far

to

debug

simply

because

the

C

code

produced

both

follows

the

structure

of

the

BNF

source

closely

and

is

conserved

through

re-compilations

Local

changes

in

the

source

force

only

local

changes

in

the

generated

C

code

-LCB-

another

aspect

of

modularity

so

breakpoints

and

the

like

can

be

relocated

quickly

Development

The

initial

development

effort

furnished

a

definition

of

precc

in

its

own

language

which

then

made

further

development

easy

after

the

initial

bootstrap

-LRB-

thereby

hangs

a

tale

first

a

rewrite

engine

was

constructed

using

the

Brief

editor

for

PC

's

macro

language

and

loaded

with

rewrite

rules

which

constructed

the

first

C

code

for

precc

from

its

BNF

and

that

gave

the

first

self-compiling

executable

-RRB-

Improvements

in

the

definition

alone

were

found

to

allow

an

order

of

magnitude

improvement

in

precc

's

own

execution

speed

-LRB-

this

does

not

affect

the

speed

of

parsers

and

compilers

built

by

precc

but

it

illustrates

the

advantages

offered

by

a

clear

and

succinct

definitional

language

-RRB-

and

precc

is

now

known

to

be

of

linear

complexity

with

respect

to

input

definition

scripts

It

runs

at

thousands

of

lines

of

input

per

second

on

standard

platforms

such

as

Sun

and

workstations

and

the

output

code

will

compile

under

any

ansi-compliant

C

compiler

Moreover

the

output

is

indeed

completely

modular

one

may

split

definition

scripts

any

which

way

pass

them

through

precc

separately

run

the

C

compiler

on

them

separately

and

then

link

together

the

object

modules

In

this

respect

precc

fulfils

its

design

goals

it

reduces

the

turn-around

time

in

language

development

both

because

it

requires

only

local

recompilations

when

changes

are

made

and

also

through

the

expressive

power

of

its

definitional

language

It

has

been

used

to

make

full

scale

parsers

and

scanners

for

cobol

-LSB-

-RSB-

and

to

implement

the

programming

language

Uniform

-LSB-

-RSB-

The

definition

scripts

for

these

languages

comprise

between

one

and

two

thousand

lines

involving

between

one

and

two

hundred

parser

definitions

The

number

of

keywords

alone

in

cobol

is

over

one

hundred

and

the

resulting

executables

are

efficient

both

in

time

and

in

space

The

worries

over

the

efficiency

of

this

kind

of

technology

seem

to

have

been

dissolved

by

the

passage

of

years

Higher

order

features

In

addition

to

all

the

above

precc

turns

out

to

provide

truly

higher

order

programming

facilities

in

the

C

runtime

environment

This

is

a

byproduct

of

C

's

non-strict

typing

restrictions

which

means

that

functions

-LRB-

such

as

other

parsers

-RRB-

can

be

freely

passed

along

with

other

parameters

to

the

parsers

which

precc

builds

Functional

parameters

are

passed

as

integer-sized

addresses

of

functions

and

re-interpreted

as

functions

upon

receipt

by

the

callee

But

the

result

is

that

precc

quite

transparently

supports

a

higher

order

macro

language

and

frequently

used

grammar

constructions

can

be

defined

as

precc

macros

in

order

to

improve

the

clarity

and

maintainability

of

a

script

For

example

the

precc

code

for

the

functional

abstraction

-LRB-

i.e.

a

separated

list

-RRB-

can

be

precisely

-LRB-

separator

separated

-RRB-

separated

separator

-LRB-

separator

separated

-RRB-

j

separated

although

-LRB-

separator

separated

-RRB-

separated

f

separator

separated

g

is

a

rather

more

efficient

version

In

both

of

these

separator

and

separated

are

other

parser

identifiers

which

is

what

makes

the

language

higher-order

In

the

C

code

that

is

generated

separator

and

separated

are

the

addresses

of

functions

Literacy

and

compatibility

Other

features

include

the

permitted

embedding

of

C

code

and

C

preprocessor

macros

or

instructions

anywhere

in

a

precc

script

with

the

resulting

mix

of

advantages

from

precc

's

own

referentially

transparent

higher-order

functions

and

the

referentially

opaque

C

ones

In

general

precc

scripts

are

literate

programs

in

the

sense

of

Knuth

-LSB-

-RSB-

in

that

only

certain

parts

of

the

script

are

visible

to

the

utility

itself

and

these

visible

parts

can

therefore

be

embedded

in

ordinary

text

-LRB-

i.e.

literate

prose

-RRB-

or

other

computer

language

instructions

typically

C

function

definitions

and

preprocessor

directives

Precc

also

provides

a

simple

hook

for

tokenising

pre-filters

such

as

the

unix

lex

-LSB-

-RSB-

utility

to

which

it

presents

an

interface

that

is

intended

to

be

plug

compatible

with

yacc

's

Most

lexers

aimed

at

a

yacc

should

be

usable

as

they

stand

with

precc

Most

yacc

scripts

can

be

converted

to

precc

scripts

too

There

are

some

difficulties

with

yacc

's

precedence

operator

but

operator

precedences

can

be

encoded

explicitly

Disadvantages

What

disadvantages

precc

does

have

include

-LRB-

-RRB-

at

runtime

its

use

of

the

C

stack

-LCB-

which

can

grow

large

in

deeply

structured

and

recursive

grammar

descriptions

and

which

is

a

limitation

-LRB-

of

C

-RRB-

under

some

memory

models

on

restricted

architectures

such

as

an

PC

with

its

segment

limit

-LRB-

stack

growth

can

be

restricted

by

the

use

of

cuts

in

the

definition

script

however

-RRB-

-LCB-

and

-LRB-

-RRB-

in

scripts

an

as

yet

inadequate

facility

for

defining

local

-LRB-

volatile

-RRB-

data

structures

for

use

as

attached

synthetic

attributes

or

in

the

actions

attached

to

precc

scripts

but

this

is

also

an

inheritance

from

the

compatibility

with

yacc

-LRB-

see

next

section

for

a

fuller

explanation

of

what

should

be

done

-RRB-

Theory

The

model

that

precc

follows

holds

that

a

compiler

is

simply

a

modified

parser

and

a

parser

is

a

function

which

consumes

an

input

stream

of

tokens

returning

a

boolean

value

to

indicate

whether

or

not

the

input

satisfies

it

plus

the

unconsumed

portion

of

the

input

PARSER

-LSB-

TOKEN

-RSB-

-LRB-

-LSB-

TOKEN

-RSB-

BOOLEAN

-RRB-

and

the

fundamental

idea

behind

the

C

implementation

is

that

the

side-effecting

part

of

the

functionality

-LCB-

the

consuming

of

the

input

stream

-LCB-

really

may

be

implemented

as

a

side-effect

in

C

typedef

BOOLEAN

PARSER

-LRB-

-RRB-

and

the

side-effect

is

always

to

move

the

global

pointer

pstr

to

a

new

position

in

the

input

stream

-LRB-

if

the

parse

returns

the

SUCCESS

boolean

value

-RRB-

A

raw

stream

can

not

be

rewound

by

moving

the

pointer

but

precc

buffers

input

in

order

that

rewinding

may

be

correctly

by

fallback

to

an

earlier

value

of

the

pointer

pstr

and

thus

remains

faithful

to

the

functional

representation

above

As

stated

above

the

functional

model

is

based

on

an

exposition

by

Jon

Fairbairn

in

-LSB-

-RSB-

To

implement

a

BNF

description

as

a

parser

it

suffices

to

know

how

to

-LRB-

-RRB-

implement

the

description

of

a

sequence

a

b

when

actions

are

executed

-LRB-

where

the

instruction

occurs

in

the

specification

-RRB-

is

the

historical

stream

input

jettisoned

because

accurate

backtracking

is

no

longer

possible

-LCB-

the

design

justification

is

that

unknown

side-effects

may

have

been

triggered

which

are

certainly

impossible

to

recall

so

it

is

not

worth

recalling

anything

from

now

on

as

the

parser

which

accepts

the

input

which

a

accepts

followed

by

the

input

which

b

accepts

given

that

one

knows

how

to

implement

the

descriptions

a

and

b

correctly

as

parsers

which

accept

precisely

what

the

descriptions

a

and

b

specify

and

-LRB-

-RRB-

do

the

same

for

the

description

of

an

or

parse

a

j

b

which

accepts

input

which

a

accepts

or

if

that

fails

accepts

input

which

b

accepts

We

can

improve

upon

the

original

exposition

-LRB-

not

in

clarity

but

-RRB-

through

the

use

of

modern

developments

in

the

application

of

monadic

constructions

-LSB-

-RSB-

-LSB-

-RSB-

has

noticed

this

connection

but

this

paper

expands

on

the

idea

Booleans

really

are

a

trivial

exception-type

disjoint

sum

BOOLEAN

E

in

which

the

exception

value

is

False

and

the

single

ordinary

result

is

True

We

need

one

exception

value

only

E

but

writing

the

disjoint

sum

values

as

inl

-LRB-

-RRB-

and

inr

-LRB-

-RRB-

instead

of

False

and

True

makes

it

easier

to

see

the

monad

structure

Fixing

E

as

the

singleton

type

here

we

define

the

auto-rewinding

parsers

of

type

X

as

the

type

P

-LRB-

X

-RRB-

-LSB-

TOKEN

-RSB-

-LRB-

-LSB-

TOKEN

-RSB-

E

X

-RRB-

satisfying

the

equation

p

ts

-LRB-

inl

e

-RRB-

-RRB-

ts

which

says

that

parsers

which

fail

to

parse

their

input

rewind

the

input

to

the

state

in

which

they

found

it

The

simple

PARSER

type

is

then

the

general

type

specialized

to

return

a

unique

value

inr

-LRB-

-RRB-

-LRB-

i.e.

True

-RRB-

on

success

PARSER

P

-LRB-

-RRB-

Note

that

the

general

type

which

has

an

equational

restriction

is

isomorphic

to

the

type

of

side

effects

or

exceptions

which

does

not

and

is

therefore

mathematically

more

tractable

-LSB-

TOKEN

-RSB-

-LRB-

E

-LRB-

-LSB-

TOKEN

-RSB-

X

-RRB-

-RRB-

but

the

isomorphism

is

artificial

in

practice

sideeffects

are

unavoidable

even

if

they

are

trivial

so

the

type

P

-LRB-

X

-RRB-

is

the

truer

model

for

the

real

situation

Two

parsers

which

follow

on

from

each

other

can

be

put

together

with

an

operator

which

mimics

sequential

composition

p

q

This

operator

has

to

run

p

but

save

the

input

stream

beforehand

so

that

if

p

does

not

raise

an

exception

but

q

does

then

it

can

rewind

the

stream

to

the

initial

state

as

required

by

the

constraining

equation

of

the

parser

type

The

auto-rewind

from

q

alone

would

only

recover

the

output

state

from

p

not

the

input

-LRB-

p

q

-RRB-

ts

let

-LRB-

ra

-RRB-

p

ts

in

case

ra

in

inl

e

-LRB-

ts

inl

e

-RRB-

inr

u

let

-LRB-

rb

-RRB-

q

in

case

rb

in

inl

f

-LRB-

ts

inl

f

-RRB-

inr

v

-LRB-

inr

v

-RRB-

esac

esac

The

operator

is

precisely

what

is

required

in

order

to

implement

the

and

parse

combination

of

two

parsers

If

p

implements

specification

a

and

q

implements

specification

b

the

p

q

implements

a

b

The

operator

has

type

P

-LRB-

X

-RRB-

P

-LRB-

Y

-RRB-

P

-LRB-

Y

-RRB-

in

general

but

PARSER

PARSER

PARSER

in

particular

when

X

Y

and

precc

's

kernel

library

duplicates

the

functionality

faithfully

by

implementing

the

C

prototype

BOOLEAN

-LRB-

PARSER

PARSER

-RRB-

replacing

explicit

references

to

the

stream

states

ts

and

with

real

side-effects

on

the

stream

pointer

the

global

variable

pstr

The

C

code

is

remarkably

close

to

the

functional

specification

BOOLEAN

-LRB-

PARSER

p

PARSER

q

-RRB-

f

void

saved

pstr

static

BOOLEAN

tok

if

GOOD

-LRB-

tok

=p

-LRB-

-RRB-

-RRB-

if

GOOD

-LRB-

tok

q

-LRB-

-RRB-

-RRB-

return

tok

else

pstr

saved

return

tok

g

P

-LRB-

-RRB-

type

type

X

P

-LRB-

X

-RRB-

P

-LRB-

X

-RRB-

-LRB-

X

P

-LRB-

Y

-RRB-

-RRB-

P

-LRB-

Y

-RRB-

-LRB-

p?Q

-RRB-

R

p

x

-LRB-

Q

x?R

-RRB-

-LRB-

-RRB-

x?Q

Q

x

-LRB-

-RRB-

p

p

-LRB-

-RRB-

Figure

The

generic

monad

type

constructor

operators

and

laws

and

requires

only

a

suitable

macro

for

GOOD

to

complete

it

The

real

code

in

the

kernel

is

not

much

more

complicated

than

this

and

differs

essentially

only

in

that

it

saves

a

bigger

frame

and

also

performs

some

stack

manipulations

itself

and

handles

the

cases

when

there

are

parameters

to

the

parsers

p

q

It

is

simple

to

prove

associativity

for

-LRB-

p

q

-RRB-

r

p

-LRB-

q

r

-RRB-

but

unnecessary

because

is

a

trivial

case

of

the

more

general

monadic

star

operator

which

takes

the

result

of

p

into

account

instead

of

running

q

regardless

A

monad

is

a

typed

algebraic

structure

generalizing

-LRB-

untyped

-RRB-

semi

groups

with

identity

and

the

laws

are

shown

in

Figure

There

is

one

type

constructor

P

which

casts

to

the

abstract

data

type

monad

of

X

possessing

two

operators

infix

and

unary

A

trivial

example

of

a

monad

is

provided

by

the

triple

P

-LRB-

o

-RRB-

o

x

x

a?f

f

-LRB-

a

-RRB-

A

good

example

is

the

monad

of

lists

P

-LRB-

o

-RRB-

-LSB-

o

-RSB-

in

which

is

the

singleton

embedding

x

-LSB-

x

-RSB-

and

a?f

is

the

map

of

function

f

onto

the

list

a

followed

by

the

flattening

of

the

resulting

list

of

lists

-LSB-

-RSB-

listsf

f

-LRB-

-RRB-

f

-LRB-

-RRB-

P

-LRB-

X

-RRB-

the

general

auto-rewinding

parser

type

is

a

monad

with

the

operational

semantics

shown

in

Figure

The

monad

expression

p

x

Q

x

is

equivalent

at

run

time

to

a

parser

p

Q

x

where

the

x

is

the

success

value

delivered

by

parser

p

as

inr

x

if

it

succeeded

Because

the

value

x

is

unknowable

before

running

is

not

expressible

in

terms

of

But

the

operator

can

be

expressed

in

terms

of

the

monad

and

the

monad

associativity

law

-LRB-

-RRB-

then

reduces

to

associativity

for

Set

to

be

P

-LRB-

X

-RRB-

fp

-LSB-

TOKEN

-RSB-

-LRB-

-LSB-

TOKEN

-RSB-

E

X

-RRB-

j

p

ts

-LRB-

inl

e

-RRB-

-RRB-

ts

-LRB-

p?Q

-RRB-

ts

let

-LRB-

ra

-RRB-

p

ts

in

case

ra

in

inl

e

-LRB-

ts

inl

e

-RRB-

inr

x

let

-LRB-

rb

-RRB-

Q

x

in

case

rb

in

inl

f

-LRB-

ts

inl

f

-RRB-

inr

v

-LRB-

inr

v

-RRB-

esac

esac

x

ts

-LRB-

ts

inr

x

-RRB-

Figure

The

monad

of

auto-rewinding

parsers

p

q

p?const

q

and

it

may

be

confirmed

that

this

definition

reduces

to

the

directly

defined

semantics

Similarly

the

precc

library

function

implements

the

a

j

b

specification

and

in

monadic

terms

the

monad

is

over

a

semi-group

a

j

b

corresponds

to

an

addition

of

parsers

p

q

within

the

type

P

-LRB-

X

-RRB-

where

p

corresponds

to

a

and

q

to

b

The

addition

p?q

has

the

semantics

of

a

compound

parser

which

tries

each

of

p

and

q

in

turn

and

accepts

the

first

-LRB-

in

this

order

-RRB-

to

succeed

The

autorewinding

properties

of

p

return

the

input

stream

to

the

initial

state

before

trying

q

if

p

raises

an

exception

The

monad

-LRB-

and

hence

-RRB-

distributes

over

the

addition

and

there

is

a

zero

too

with

which

also

interacts

well

The

precise

extra

laws

which

hold

in

an

abstract

monad

over

a

semi-group

are

shown

in

Figure

and

the

semantics

of

the

plus

and

zero

in

the

monad

of

auto-rewinding

parsers

are

shown

in

Figure

and

it

may

be

verified

that

the

given

semantics

satisfies

the

laws

Note

that

although

has

to

be

associative

it

is

not

necessarily

commutative

and

that

is

the

case

for

the

parser

monad

The

laws

state

that

distributes

over

and

and

the

latter

two

comprise

a

semi-group

The

C

code

in

the

kernel

is

again

close

to

the

functional

specification

P

-LRB-

X

-RRB-

P

-LRB-

X

-RRB-

P

-LRB-

X

-RRB-

P

-LRB-

X

-RRB-

a

-LRB-

b

c

-RRB-

-LRB-

a

b

-RRB-

c

-LRB-

-RRB-

a

a

a

-LRB-

-RRB-

-LRB-

p

q

-RRB-

R

p?R

q?R

-LRB-

-RRB-

p

x

-LRB-

Q

x

R

x

-RRB-

p?Q

p?R

-LRB-

-RRB-

Q

-LRB-

-RRB-

p?const

-LRB-

-RRB-

Figure

Extra

laws

for

a

monad

over

a

semigroup

BOOLEAN

-LRB-

PARSER

p

PARSER

q

-RRB-

f

static

BOOLEAN

tok

if

GOOD

-LRB-

tok

=p

-LRB-

-RRB-

-RRB-

return

tok

return

q

-LRB-

-RRB-

g

and

relies

in

the

same

way

on

the

rewinding

of

the

parse

stream

by

p

when

it

fails

All

the

basic

monad

operators

can

be

represented

in

C

and

are

present

in

preccs

kernel

Note

that

the

and

functions

which

represent

and

respectively

are

essentially

higher

order

functions

and

that

C

supports

this

functionality

one

may

pass

the

addresses

of

functions

-LRB-

rather

than

their

machine

code

-RRB-

whenever

a

datum

of

type

PARSER

is

required

-LRB-

p

q

-RRB-

ts

let

-LRB-

ra

-RRB-

p

ts

in

case

ra

in

inl

e

q

ts

inr

v

-LRB-

inr

v

-RRB-

esac

ts

-LRB-

ts

inl

-LRB-

-RRB-

-RRB-

Figure

The

parser

semantics

for

the

semi-group

operators

As

remarked

above

the

description

is

complicated

further

in

the

real

implementation

by

the

presence

of

parameters

to

grammar

descriptions

which

requires

that

the

and

combinator

implementations

in

C

also

accept

arguments

which

correspond

to

those

parameters

This

is

because

the

parameters

can

not

be

supplied

to

the

parser

arguments

without

evaluating

them

in

C

making

the

compounded

parser

stricter

than

the

functional

description

given

here

Compilers

Now

compilers

and

interpreters

differ

from

parsers

in

that

they

invoke

actions

which

have

effects

on

some

external

state

-LRB-

such

as

the

lines

written

into

a

file

of

generated

object

code

or

the

image

on

a

display

screen

-RRB-

One

way

of

extending

the

functional

model

to

treat

compilers

sets

them

to

be

the

type

COMPILER

-LRB-

STATE

-LSB-

TOKEN

-RSB-

-RRB-

-LRB-

STATE

-LSB-

TOKEN

-RSB-

BOOLEAN

-RRB-

and

these

can

be

implemented

in

C

by

making

the

side-effects

on

state

and

the

input

stream

into

real

side-effects

which

means

that

compilers

have

the

same

C

type

as

parsers

because

the

extra

effect

on

STATE

can

not

be

made

explicit

We

define

an

action

to

be

a

side-effecting

function

in

the

functional

model

ACTION

STATE

STATE

and

implement

these

in

C

as

functions

which

take

no

arguments

and

return

no

results

typedef

void

ACTION

-LRB-

-RRB-

Strictly

this

prototype

permits

some

arguments

to

the

function

call

but

that

is

exactly

what

is

required

in

order

to

support

the

translation

of

parameterized

BNF

definitions

so

the

looseness

of

the

type

is

fortuitous

A

typically

higher-order

way

of

building

a

compiler

is

to

attach

an

action

to

a

parser

-LCB-

letting

the

action

be

executed

precisely

when

the

parse

succeeds

attach

-LRB-

PARSER

ACTION

-RRB-

COMPILER

attach

-LRB-

p

f

-RRB-

-LRB-

w

ts

-RRB-

let

-LRB-

ts

rp

-RRB-

p

ts

in

case

rp

in

inl

e

-LRB-

w

ts

inl

e

-RRB-

inr

v

-LRB-

f

w

ts

inr

v

-RRB-

esac

and

this

method

can

be

duplicated

in

C

using

a

higher

order

function

with

prototype

BOOLEAN

attach

-LRB-

PARSER

ACTION

-RRB-

which

moves

the

side-effects

out

of

the

parameter

list

and

implements

them

directly

One

important

complication

arises

however

Whereas

the

input

stream

can

be

rewound

in

the

real

world

if

the

parser

parameter

fails

side-effects

can

not

be

Therefore

it

is

important

that

if

the

parser

succeeds

instead

of

triggering

the

action

immediately

attach

should

simply

place

the

address

of

its

action

parameter

onto

a

fifo

queue

for

later

execution

The

queue

is

emptied

whenever

a

cut

command

is

reached

in

the

grammar

script

-LRB-

or

at

the

end

of

stream

input

if

the

parse

is

successful

whichever

occurs

first

-RRB-

So

the

C

code

includes

a

PUSHACTION

command

instead

of

a

direct

execution

of

the

action

BOOLEAN

attach

-LRB-

PARSER

p

ACTION

a

-RRB-

f

if

GOOD

-LRB-

p

-LRB-

-RRB-

-RRB-

f

PUSHACTION

-LRB-

a

-RRB-

return

SUCCESS

g

return

FAILURE

g

Again

C

passes

addresses

of

functions

as

parameters

not

code

so

there

must

be

a

C

function

declaration

which

corresponds

to

the

action

parameter

to

the

attach

routine

and

precc

builds

the

C

code

for

it

explicitly

When

the

form

a

code

is

encountered

in

a

script

-LCB-

which

means

execute

code

if

parser

a

succeeds

-LCB-

the

action

static

void

-LRB-

-RRB-

f

code

g

is

written

into

the

generated

code

making

the

code

accessible

as

a

C

function

And

it

is

followed

by

the

parser

definition

static

BOOLEAN

-LRB-

-RRB-

f

PARSER

a

ACTION

return

-LRB-

attach

-LRB-

a

-RRB-

-RRB-

g

which

will

push

the

function

onto

the

action

stack

for

later

execution

provided

that

parser

a

succeeds

Further

aspects

of

the

action

semantics

are

considered

in

the

next

sections

In

view

of

the

possibilities

afforded

by

the

monadic

construction

this

method

of

building

a

compiler

by

attaching

side-effecting

actions

is

not

ideal

but

it

is

what

precc

was

developed

with

At

the

time

of

the

design

the

monadic

insight

was

not

available

and

future

developments

will

have

to

take

it

into

account

The

monadic

view

shows

that

it

is

good

design

to

allow

each

parser

to

return

a

more

generic

value

than

just

True

or

False

but

the

present

precc

design

is

restricted

to

just

these

two

It

has

the

advantage

of

a

certain

space

and

time

efficiency

because

in

practice

only

very

few

parsers

will

carry

a

value

that

is

inspected

by

any

other

parser

so

it

might

be

wasteful

to

provide

for

a

value

from

every

parser

as

at

present

attributes

notionally

attached

to

parsers

are

catered

for

by

special

actions

described

in

the

next

section

which

manipulate

a

secondary

runtime

stack

on

which

they

must

be

placed

at

parse

time

It

should

be

noted

that

precc

properly

interprets

all

inherited

attribute

grammar

descriptions

This

simply

means

that

it

allows

parameters

to

all

the

parsers/compilers

it

It

properly

interprets

all

synthetic

attribute

grammar

descriptions

too

in

the

same

way

as

yacc

does

through

the

special

-LRB-

attribute-manipulating

-RRB-

actions

Precc

's

limitation

in

theory

as

it

is

at

present

is

that

it

can

not

feed

back

the

synthetic

attributes

as

inherited

attributes

to

sub-grammars

-LRB-

data

flow

in

the

other

direction

is

perfectly

feasible

and

indeed

usual

in

precc

definition

scripts

-RRB-

This

is

because

the

synthesised

attributes

are

constructed

by

actions

and

the

actions

are

triggered

after

a

parse

so

can

not

feed

back

into

the

parse

itself

The

situation

will

change

in

later

releases

that

implement

the

monadic

model

more

fully

Parsing

n

input

tokens

causes

precc

to

stack

at

most

n

actions

for

later

execution

along

with

some

instructions

for

the

manipulation

of

the

runtime

attribute

stack

Some

of

these

actions

are

trivial

-LRB-

for

example

on

successfully

matching

a

literal

h

x

i

say

the

token

x

is

by

default

scheduled

for

attachment

to

the

BNF

expression

h

x

i

as

its

synthesised

attribute

and

this

is

achieved

by

stacking

the

action

which

will

place

x

on

the

runtime

value

stack

in

the

correct

position

-RRB-

but

the

nett

effect

is

that

precc

builds

a

program

of

size

that

is

linear

in

its

uncut

input

stream

length

It

also

necessarily

stacks

C

calls

on

the

C

call

stack

to

a

depth

again

linearly

-LRB-

directly

-RRB-

bounded

by

the

input

stream

but

the

depth

is

usually

much

less

than

the

theoretical

maximum

especially

if

non-recursive

constructions

are

favoured

in

the

script

over

recursive

ones

is

a

practical

limit

to

the

number

of

parameters

allowed

in

the

precc

library

routines

but

it

should

not

be

a

problem

in

practice

Up

to

eight

parameters

are

allowed

at

present

More

on

actions

The

horrors

of

the

present

implementation

of

synthetic

attributes

deserve

revelation

if

only

because

as

remarked

above

precc

follows

the

yacc

style

faithfully

synthetic

attributes

are

referenced

and

assigned

in

the

actions

attached

to

the

grammar

clauses

This

method

of

dealing

with

synthetic

attributes

is

not

ideal

for

various

reasons

and

the

next

release

of

precc

will

use

the

neater

method

based

on

the

monadic

theory

set

out

above

where

attributes

are

returned

as

success

values

from

the

parse

of

the

form

inr

x

But

the

present

method

is

compatible

for

yacc

users

which

may

be

deemed

one

advantage

It

also

has

the

second

advantage

that

backtracking

a

parse

destroys

the

putative

synthetic

attributes

very

neatly

as

a

side-effect

of

removing

all

pending

actions

and

this

has

been

helpful

to

the

author

at

least

Actions

are

pieces

of

C

code

-LRB-

terminated

by

a

semi-colon

-RRB-

and

placed

between

a

pair

of

colons

-LRB-

:'-RRB-

in

the

grammar

definition

script

For

example

addexpr

expr

h

i

expr

is

not

unreasonable

The

is

the

result

value

attached

to

the

whole

expression

is

the

value

attached

to

the

leftmost

term

and

is

that

attached

to

the

rightmost

term

The

method

of

locating

attributes

on

a

run-time

value

stack

is

essentially

exactly

the

same

as

yacc

's

and

it

also

allows

one

to

incorporate

the

same

incomprehensible

trick

of

pulling

values

off

the

stack

when

they

are

notionally

further

to

the

left

than

the

current

expression

using

or

even

lower

references

Side-effecting

actions

can

be

sequenced

as

follows

-LRB-

recall

that

no

side-effect

takes

place

until

after

the

parse

by

which

time

precc

has

built

an

internal

list

of

actions

to

execute

which

corresponds

exactly

to

the

successful

trace

through

the

grammar

-RRB-

The

specification

@abc

a

b

c

printf

-LRB-

D

-RRB-

@a

h

a

i

printf

-LRB-

A

-RRB-

@b

h

b

i

printf

-LRB-

B

-RRB-

@c

h

c

i

printf

-LRB-

C

-RRB-

will

upon

receiving

input

abc

generate

the

C

program

equivalent

printf

-LRB-

A

-RRB-

printf

-LRB-

B

-RRB-

printf

-LRB-

C

-RRB-

printf

-LRB-

D

-RRB-

to

be

executed

later

Thus

actions

attached

to

a

sequence

expression

may

be

thought

of

as

occurring

immediately

after

the

actions

attached

to

sub-expressions

and

so

on

down

A

worked

example

As

an

examination

of

the

practicality

and

usability

of

precc

the

language

occam

has

provided

a

readymade

test

-LSB-

-RSB-

Occam

's

razor

Entia

non

sunt

multiplicanda

praeter

necessitatatem

No

more

things

should

be

presumed

to

exist

than

are

absolutely

necessary

reminds

us

that

a

thing

which

can

be

done

without

ought

not

to

be

used

Fortunately

it

seems

that

precc

may

be

needed

in

order

to

build

a

parser

for

occam

because

it

is

certainly

not

an

easy

language

for

LALR

-LRB-

-RRB-

based

parsers

to

handle

In

particular

it

requires

the

lexical

indentation

of

constructs

from

the

left

hand

edge

to

be

recognised

by

the

parser

since

this

often

delimits

their

scope

in

place

of

the

perhaps

more

familiar

BEGIN/END

delimiters

of

Pascal

and

other

block

structured

languages

-LRB-

or

-LCB-

and

-RCB-

of

C

-RRB-

As

a

result

knowledge

of

the

current

indentation

may

be

needed

at

many

points

during

parsing

and

backtracking

to

an

alternative

interpretation

may

be

necessary

as

the

scope

of

-LRB-

possibly

several

-RRB-

constructs

is

ended

For

example

the

two

programs

below

from

-LSB-

-RSB-

differ

only

in

the

indentation

of

one

line

but

the

program

on

the

left

yields

y

while

that

on

the

right

results

in

y

SEQ

SEQ

x

y

z

x

y

z

WHILE

x

WHILE

x

SEQ

SEQ

z

z

x

z

z

x

x

x

y

y

x

y

y

x

The

scope

of

the

second

SEQ

is

terminated

by

the

offside

-LRB-

less

indented

-RRB-

statement

occam

could

theoretically

be

handled

by

a

LR

-LRB-

-RRB-

grammar

with

a

token

lookahead

since

the

indentation

may

be

limited

to

columns

-LRB-

spaces

per

indentation

-RRB-

So

yacc

could

handle

all

practical

occam

programs

once

the

LR

-LRB-

-RRB-

specification

is

expanded

out

into

LR

-LRB-

-RRB-

form

However

this

would

be

very

wasteful

of

space

for

the

automaton

's

tables

much

of

which

would

be

duplicated

The

use

of

purely

synthetic

attribute

grammars

is

in

any

case

slightly

problematic

because

several

syntactic

constructs

may

be

terminated

by

a

single

lexical

group

For

example

in

SEQ

SEQ

SEQ

x

y

y

z

the

group

y

z

belongs

to

the

outermost

SEQ

by

virtue

of

its

indentation

position

which

terminates

the

two

innermost

SEQ

constructs

so

it

is

not

clear

what

token

type

it

should

present

A

prior

pass

may

be

necessary

to

insert

BEGIN

and

END

tokens

which

can

serve

as

an

unambiguous

delineation

of

the

block

structure

but

this

amounts

to

admitting

that

yacc-type

technology

is

not

well-suited

to

this

kind

of

parsing

problem

A

realistic

parser

for

Occam

This

section

presents

a

precc

parser

for

a

significant

subset

of

occam

and

demonstrates

how

closely

the

parser

can

be

related

to

the

BNF-style

semiformal

syntax

provided

in

Appendix

G

of

-LSB-

-RSB-

a

sample

of

which

is

given

in

Figure

The

indentation

of

constructs

is

specified

by

presenting

the

BNF

description

in

indented

form

where

appropriate

The

corresponding

specification

in

precc

is

shown

in

Figure

This

is

one

of

the

more

complicated

and

therefore

more

interesting

parts

of

the

specification

as

far

as

parsing

is

concerned

The

precc

specification

differs

in

several

respects

Firstly

and

most

obviously

the

majority

of

the

constructions

are

parameterized

They

take

the

parameter

n

which

denotes

the

absolute

indentation

of

the

concrete

representation

so

that

conditional

-LRB-

-RRB-

denotes

an

IF

construct

with

the

initial

I

in

column

Where

the

concrete

specification

denotes

extra

indentation

by

spacing

the

precc

specification

uses

an

incremented

parameter

instead

Thus

choices

-LRB-

n

-RRB-

is

included

where

the

concrete

syntax

instead

has

choices

two

spaces

further

to

the

right

than

the

governing

IF

keyword

Secondly

the

precc

specification

employs

special

matches

for

the

end

of

line

Although

precc

can

match

the

literal

ascii

line

feed

control

character

directly

the

default

lexical

analyzer

which

comes

process

SKIP

j

STOP

j

construction

j

construction

sequence

j

conditional

j

loop

j

conditional

IF

f

choices

g

choice

guarded.choice

j

conditional

guarded.choice

boolean

process

loop

WHILE

boolean

process

Figure

Sample

from

occam

syntax

summary

with

precc

generates

the

special

zero

token

at

an

end

of

line

and

it

is

this

that

is

matched

by

the

and

constructs

-LRB-

see

below

-RRB-

Though

precc

can

use

lexers

generated

by

the

lex

utility

directly

in

exactly

the

same

way

as

yacc

does

no

special

lexical

analysis

is

required

for

the

purposes

of

this

trial

and

the

default

lexer

can

be

used

This

lexer

passes

every

character

unchanged

except

that

it

substitutes

a

line

feed

control

character

with

the

special

zero

token

The

rationale

behind

the

design

of

the

default

lexer

is

that

end

of

line

frequently

marks

a

natural

break

point

at

which

special

action

must

be

taken

and

yacc-compatible

lexers

conventionally

signal

special

conditions

with

the

zero

token

Actions

which

check

for

end

of

file

conditions

can

be

attached

to

the

match

for

the

zero

token

The

is

a

match

for

end

of

line

which

can

be

backtracked

over

and

the

is

a

match

for

one

which

can

not

It

incorporates

the

special

cut

symbol

which

disables

backtracking

It

is

best

to

include

as

many

cuts

as

possible

in

a

precc

script

because

in

the

event

of

an

error

being

detected

in

the

parsed

stream

they

prevent

exponential

searches

through

all

the

possible

alternative

parses

The

cut

is

one

of

the

innovative

implementational

aspects

of

precc

in

comparison

with

other

LL

-LRB-

-RRB-

parsers

the

use

of

the

C

call

stack

is

another

It

is

certain

that

an

IF

begins

a

conditional

statement

and

nothing

else

so

a

cut

may

be

inserted

after

the

IF

but

it

is

conceivable

that

it

may

be

necessary

to

backtrack

over

the

newline

following

a

boolean

expression

in

case

the

following

stream

does

not

match

a

clause

-LRB-

n

-RRB-

after

all

In

fact

this

is

probably

never

going

to

be

the

case

but

it

is

best

to

allow

for

such

a

possibility

so

a

is

needed

at

this

point

Thirdly

the

precc

specification

uses

the

square

brackets

syntax

-LSB-

foo

-RSB-

to

indicate

an

optional

inclusion

where

the

concrete

syntax

uses

curly

brackets

instead

This

is

a

trivial

difference

-LCB-

precc

reserves

the

curly

brackets

for

use

as

grouping

parentheses

Fourthly

where

the

concrete

specification

has

process

the

precc

specification

has

clause

-LRB-

n

-RRB-

instead

The

n

indicates

extra

indentation

to

the

right

but

the

clause

-LRB-

n

-RRB-

construct

has

been

introduced

to

stand

for

indent

-LRB-

n

process

-RRB-

which

might

equally

well

be

placed

directly

in

the

specification

However

giving

a

name

to

the

construction

makes

the

specification

more

maintainable

because

it

eliminates

repetition

and

narrows

the

range

over

which

changes

may

have

to

be

made

at

some

future

point

It

also

makes

the

resulting

code

more

compact

at

the

cost

of

a

tiny

loss

in

runtime

performance

which

may

even

be

optimized

away

by

the

C

compiler

-LRB-

Experiments

have

shown

that

function

calls

with

simple

bodies

-LCB-

which

this

construction

induces

-LCB-

are

usually

copied

into

inline

code

by

optimizing

C

compilers

-RRB-

All

this

is

possible

because

of

precc

's

declarative

semantics

An

expression

may

be

substituted

by

a

name

for

the

expression

anywhere

in

the

text

without

altering

the

semantics

Why

does

the

precc

specification

need

an

indent

-LRB-

n

process

-RRB-

construct

at

all

when

the

concrete

syntax

includes

nothing

of

the

sort

It

is

because

the

n

spaces

which

appear

to

the

left

must

be

remembered

The

concrete

syntax

specification

drops

these

from

the

representation

of

succeeding

lines

showing

only

the

relative

indentation

Inserting

them

explicitly

gives

the

concrete

specification

the

shape

process

-LRB-

n

-RRB-

SKIP

j

STOP

j

construction

-LRB-

n

-RRB-

j

construction

-LRB-

n

-RRB-

sequence

-LRB-

n

-RRB-

j

conditional

-LRB-

n

-RRB-

j

loop

-LRB-

n

-RRB-

j

conditional

-LRB-

n

-RRB-

IF

-LSB-

choices

-LRB-

n

-RRB-

-RSB-

choice

-LRB-

n

-RRB-

-LSB-

specifications

-LRB-

n

-RRB-

-RSB-

indent

-LRB-

n

chosen

-RRB-

chosen

-LRB-

n

-RRB-

guardedchoice

-LRB-

n

-RRB-

j

conditional

-LRB-

n

-RRB-

guardedchoice

-LRB-

n

-RRB-

boolean

clause

-LRB-

n

-RRB-

loop

-LRB-

n

-RRB-

WHILE

somespace

boolean

clause

-LRB-

n

-RRB-

clause

-LRB-

n

-RRB-

indent

-LRB-

n

process

-RRB-

Figure

Corresponding

precc

specification

guardedchoice

-LRB-

n

-RRB-

guarded

choice

boolean

t

t

t

t

-LCB-

z

-RCB-

blank

n

t

t

process

process

-LRB-

n

-RRB-

and

the

precc

guardedchoice

-LRB-

n

-RRB-

definition

is

to

represent

the

construction

starting

with

a

boolean

in

column

n

The

full

precc

specification

across

the

two

lines

has

to

include

the

n

blank

spaces

before

the

process

part

guardedchoice

-LRB-

n

-RRB-

boolean

blank

-LRB-

n

-RRB-

process

-LRB-

n

-RRB-

and

the

whole

lower

line

of

this

definition

is

replaced

in

the

actual

script

by

clause

-LRB-

n

-RRB-

We

may

define

the

higher

order

construction

indent

-LRB-

n

p

-RRB-

blank

n

p

-LRB-

n

-RRB-

because

the

pattern

is

frequently

reused

in

the

occam

specification

Note

that

blank

n

is

the

precc

syntax

for

blank

repeated

n

times

-LRB-

and

a

without

a

following

expression

means

repeated

zero

or

more

times

-RRB-

One

example

of

the

use

of

indent

is

in

the

definition

of

clause

-LRB-

n

-RRB-

clause

-LRB-

n

-RRB-

indent

-LRB-

n

process

-RRB-

The

script

can

be

made

even

more

flexible

if

required

The

precc

parser

can

allow

process

to

start

beyond

the

mandatory

two

spaces

farther

to

the

right

and

can

compute

any

indentations

to

come

relative

to

the

actual

position

in

which

process

starts

-LRB-

although

this

flexibility

is

disallowed

by

the

standard

occam

language

perhaps

because

of

the

extra

complexity

of

handling

it

-RRB-

This

may

be

achieved

by

including

indented

-LRB-

n

p

-RRB-

instead

of

p

-LRB-

n

-RRB-

and

defining

the

indented

grammars

so

that

they

detect

any

extra

blanks

and

incorporate

them

into

the

indentation

count

indented

-LRB-

n

p

-RRB-

blank

indented

-LRB-

n

p

-RRB-

j

p

-LRB-

n

-RRB-

indent

-LRB-

n

p

-RRB-

blank

n

indented

-LRB-

n

p

-RRB-

There

is

some

inefficiency

introduced

here

because

a

malformed

p

following

some

blanks

may

be

-LRB-

partially

-RRB-

scanned

twice

or

more

but

it

may

be

considered

worthwhile

because

of

the

extra

flexibility

in

the

acceptable

syntax

that

it

affords

There

are

also

good

reasons

for

inserting

a

test

into

the

specification

to

ensure

that

n

does

not

grow

too

large

indented

-LRB-

n

p

-RRB-

blank

-RRB-

n

-LRB-

indented

-LRB-

n

p

-RRB-

This

will

prevent

arbitrary

indentations

which

are

not

allowed

by

the

occam

folding

editor

in

any

case

Nearly

all

the

recursion

in

the

parser

is

introduced

by

this

one

construction

Note

that

the

out-turned

parentheses

which

denote

a

parse-time

test

may

contain

any

valid

C

expression

returning

a

one-orzero

integer

result

following

the

C

convention

for

boolean

values

There

is

a

danger

of

destroying

the

declarative

semantics

here

since

C

expressions

may

be

used

to

implement

side-effects

but

this

possibility

will

only

rarely

be

required

since

the

cost

in

terms

of

maintainability

is

great

Side-effects

may

also

be

incorporated

into

parameters

since

these

may

be

arbitrary

C

expressions

too

But

side-effects

in

actions

-LRB-

see

below

-RRB-

attached

to

the

parse

can

have

only

limited

effect

on

the

parse

itself

because

precc

discharges

actions

after

parsing

not

during

at

the

cut

points

marked

in

the

specification

script

The

concrete

syntax

and

the

precc

specification

differ

in

one

final

point

The

precc

specification

allows

each

choice

case

in

an

IF

statement

to

be

preceded

by

an

optional

set

of

specifications

In

fact

the

concrete

syntax

specifies

this

too

but

elsewhere

with

the

recursive

alternate

production

choice

specification

choice

In

this

respect

precc

is

deficient

All

the

production

rules

for

a

given

target

must

appear

together

as

alternates

or

equivalent

in

the

one

definition

and

can

not

be

scattered

across

a

script

or

across

several

scripts

This

is

a

necessary

requirement

for

modularity

and

is

good

practice

anyway

Tokenization

The

precc

parser

specification

for

occam

includes

a

token

module

which

acts

as

the

lexical

analyzer

to

the

rest

of

the

parser

There

is

no

need

to

use

precc

as

its

own

lexer

but

there

is

no

disadvantage

in

doing

so

either

The

module

consists

of

eight

definitions

of

various

sorts

of

white

space

one

metaproduction

which

helps

with

the

recognition

of

keywords

and

the

keywords

themselves

This

module

illustrates

the

remaining

parts

of

precc

's

basic

language

quite

well

and

so

the

module

is

shown

in

Figure

All

the

other

terminals

like

digit

were

defined

in

the

literal

section

of

the

concrete

syntax

and

appear

in

the

literal.y

precc

specification

file

associated

with

a

literal.c

C

program

and

literal.o

object

module

The

first

point

to

note

about

this

literate

script

is

that

any

line

which

does

not

begin

with

an

is

passed

through

as

unchanged

C

code

Thus

C

preprocessor

directives

may

be

used

to

include

files

The

occam.h

file

contains

the

definitions

of

the

C

macros

EMPTY

which

tests

for

an

empty

string

HD

which

returns

the

head

of

a

non-empty

string

and

TL

which

returns

the

tail

of

a

string

Secondly

the

C

macros

are

quite

valid

in

parameters

to

the

grammatical

rules

or

as

a

boolean

test

-LRB-

inside

the

out-turned

parentheses

-RRB-

But

C

macros

and

C

code

have

a

more

important

use

in

the

actions

which

one

can

attach

to

a

parser

The

actions

appear

between

colons

in

the

rules

and

occur

at

runtime

in

sequence

corresponding

to

their

position

just

as

yacc

attached

actions

do

Actions

refer

to

the

synthetic

attributes

attached

to

each

terminal

and

non-terminal

with

the

n

formals

The

synthetic

attribute

for

the

whole

rule

may

be

assigned

to

as

the

formal

but

no

assignments

are

made

here

because

this

parser

was

only

intended

to

print

a

tree

representation

properly

scoped

with

the

block

structure

For

example

the

loweralpha

action

outputs

the

character

which

is

held

as

the

value

using

the

C

printf

-LRB-

c

-RRB-

library

call

and

the

action

occurs

at

runtime

at

a

point

where

printing

the

character

is

appropriate

Precc

has

to

dive

into

the

C

expressions

in

order

to

translate

n

references

so

it

is

not

possible

to

hide

them

within

C

macros

but

otherwise

etc

may

be

treated

like

-LRB-

volatile

-RRB-

C

variables

of

integer

type

The

character

value

in

was

designated

by

the

default

lexer

at

parse

time

which

placed

it

in

the

yylval

variable

as

precc

requested

the

token

This

is

the

standard

mode

for

lexers

generated

by

lex

and

the

precc

default

lexer

conforms

to

this

de

facto

standard

even

though

it

is

noticeably

less

efficient

than

it

might

be

-LRB-

the

fast

way

to

pass

tokens

and

values

to

precc

is

to

map

the

input

stream

onto

the

area

beginning

at

the

yybuffer

address

-LCB-

precc

will

read

from

this

area

when

it

can

-RRB-

Precc

then

places

an

instruction

into

the

stack

of

pending

action

that

later

loads

a

matched

token

onto

the

runtime

stack

The

expression

-LRB-

islower

-RRB-

is

a

predicate

It

tests

the

incoming

tokens

using

the

C

islower

-LRB-

-RRB-

call

In

this

case

islower

is

a

standard

C

library

routine

but

it

could

be

a

userdefined

function

A

token

that

satisfies

the

predicate

will

have

its

yylval

value

scanned

and

attached

as

the

attribute

of

the

-LRB-

islower

-RRB-

expression

By

means

of

customized

predicates

in

simple

combinations

precc

can

handle

arbitrary

amounts

of

tokenising

itself

without

the

need

for

a

separate

tool

like

lex

Precc

syntax

An

outline

of

the

precc

language

is

shown

in

Figure

using

an

unsophisticated

precc

script

that

may

be

also

be

suitable

for

bootstrapping

using

a

simpler

parser

generator

White

space

and

comments

may

be

assumed

to

have

been

already

removed

by

the

tokenizer

and

lines

not

beginning

with

the

symbol

to

have

been

discarded

The

C

rvalue

in

the

description

of

predicate

is

intended

to

denote

either

include

occam.h

key

-LRB-

word

-RRB-

-RRB-

EMPTY

-LRB-

word

-RRB-

-LRB-

j

h

HD

-LRB-

word

-RRB-

i

key

-LRB-

TL

-LRB-

word

-RRB-

-RRB-

blank

quietblank

PRINTSPACE

quietblank

h

i

whitespace

quietblank

somespace

quietblank

whitespace

name

loweralpha

digit

loweralpha

-LRB-

islower

-RRB-

printf

-LRB-

c

-RRB-

IF

key

-LRB-

IF

-RRB-

Figure

The

token

definitions

and

actions

a

function

name

or

a

function

name

followed

by

a

bracketed

list

of

C

expressions

-LCB-

a

function

application

like

foo

-LRB-

x

y

z

-RRB-

Precc

will

invoke

the

function

with

the

current

token

as

an

extra

parameter

-LCB-

in

this

example

foo

-LRB-

x

y

z

tok

-RRB-

-LCB-

and

expects

a

boolean

value

to

be

returned

Note

that

this

syntax

specification

in

particular

could

be

significantly

strengthened

by

using

precc

's

context-dependent

definitions

Each

precc

declaration

potentially

introduces

an

environment

list

of

formal

variables

which

may

appear

in

the

expression

to

the

right

of

the

equals

sign

For

example

the

formals

introduced

in

the

useful

macro

-LRB-

this

could

also

be

used

in

the

script

with

some

gain

in

maintainability

-RRB-

sep

list

-LRB-

s

p

-RRB-

p

f

s

p

g

are

the

separator

s

and

the

parser

p

If

the

list

of

formals

are

stored

in

a

structure

then

every

part

of

the

definition

of

the

expression

parser

can

be

parameterized

by

it

with

the

aim

of

restricting

the

expression

's

variables

to

those

declared

sequent

-LRB-

env

-RRB-

-LRB-

env

-RRB-

-LSB-

postop

-LRB-

env

-RRB-

-RSB-

-LRB-

env

-RRB-

h

h

i

C

expression

-LRB-

env

-RRB-

h

i

i

j

where

C

expression

-LRB-

env

-RRB-

parses

the

putative

C

expression

in

detail

and

rejects

it

if

its

undeclared

variables

are

not

a

subset

of

the

formals

in

the

environment

Precc

really

does

analyze

C

expressions

for

their

variables

although

not

explicitly

for

the

purpose

just

suggested

C

expressions

in

actions

must

be

searched

for

the

n

references

to

synthetic

attributes

and

these

pseudo-variables

are

then

replaced

by

valid

C

code

Precc

also

has

to

know

about

C

expressions

in

order

to

determine

where

they

end

The

syntax

described

in

the

figure

is

highly

ambiguous

The

following

is

perfectly

legal

precc

syntax

-LRB-

a

sequence

of

two

conditions

surrounding

a

parser

b

-RRB-

for

example

a

-RRB-

foo

-LRB-

TRUE

-RRB-

-LRB-

b

-RRB-

-LRB-

bar

-RRB-

-LRB-

but

it

is

only

prevented

from

being

interpreted

as

two

conditions

surrounding

a

parser

TRUE

a

-RRB-

foo

-LRB-

TRUE

-RRB-

-LRB-

b

-RRB-

-LRB-

bar

-RRB-

-LRB-

by

recognition

of

foo

-LRB-

TRUE

-RRB-

as

the

maximal

initial

segment

of

the

parse

interpretable

as

a

C

expression

It

can

be

seem

that

extra

care

is

needed

when

using

C

macros

Implementation

Precc

interprets

BNF

scripts

as

C

code

and

the

functions

which

appear

in

the

generated

C

code

are

the

and

functions

which

have

been

discussed

in

prior

sections

The

precise

syntax

of

precc

scripts

is

given

in

Figure

but

it

suffices

to

take

a

less

developed

syntax

in

order

to

explain

the

script

declaration

declaration

lvalue

h

i

expression

lvalue

parserid

-LSB-

h

-LRB-

i

formals

h

-RRB-

i

-RSB-

formals

varid

f

h

i

varid

g

expression

alternate

f

h

j'

i

alternate

g

alternate

sequent

sequent

-LSB-

postop

-RSB-

h

h

i

C

expression

h

i

i

literal

j

h

i

i

C

expression

h

h

i

not-a

literal

j

h

-RRB-

i

C

expression

h

-LRB-

i

condition

j

h

-LRB-

i

C

rvalue

h

-RRB-

i

predicate

j

h

i

C

code

h

i

action

j

h

-LSB-

i

expression

h

-RSB-

i

option

j

h

-RSB-

i

expression

h

-LSB-

i

phantom

j

h

f

i

expression

h

g

i

bracket

j

h

i

any

token

j

h

i

EOL

j

h

i

cut

j

rvalue

atomic

postop

f

h

i

j

h

i

g

-LSB-

C

expression

-RSB-

rvalue

parserid

-LSB-

h

-LRB-

i

actuals

h

-RRB-

i

-RSB-

actuals

C

expression

f

h

i

C

expression

g

Figure

Precc

syntax

summary

interpretation

Each

production

rule

is

treated

essentially

independently

-LRB-

with

the

exceptions

that

a

global

counter

is

incremented

with

each

clause

for

the

purpose

of

generating

unique

function

names

and

a

count

of

input

lines

and

similar

other

information

is

also

maintained

and

passed

on

from

clause

to

clause

-RRB-

Assume

that

clauses

have

the

simplified

form

declaration

lvalue

h

i

expression

expression

alternate

-LSB-

h

j'

i

expression

-RSB-

alternate

sequent

alternate

j

empty

sequent

rvalue

and

for

even

greater

simplicity

assume

that

lvalue

and

rvalue

are

terminals

with

the

form

name

-LRB-

args

-RRB-

This

syntax

essentially

corresponds

to

yacc

scripts

except

that

parameters

are

allowed

Each

clause

and

component

will

carry

an

attribute

a

structure

with

two

parts

VALUE

hname

fnid

args

-LSB-

varid

-RSB-

i

and

the

value

attached

to

a

terminal

name

-LRB-

args

-RRB-

will

be

the

record

hname

argsi

In

the

real

implementation

the

value

is

simply

a

string

itself

borrowed

from

a

run-time

heap

and

garbage-collected

after

parsing

each

rule

The

split

into

the

two

components

is

marked

by

the

position

of

the

first

-LRB-

in

the

string

There

is

a

large

but

finite

limit

on

the

string

heap

size

but

it

may

be

adjusted

by

the

user

in

a

command

line

switch

to

precc

and

is

not

likely

to

be

exceeded

since

garbage

collection

takes

place

between

parses

of

each

new

declaration

Then

precc

prints

out

C

code

as

a

side-effect

while

parsing

the

syntax

The

action

attached

to

the

declaration

definition

for

example

is

define

P

DCL

-LRB-

x

y

-RRB-

n

printf

-LRB-

BOOLEAN

sfnn

x

-RRB-

n

printf

-LRB-

PARSER

s

nn

name

-LRB-

y

-RRB-

-RRB-

n

printf

-LRB-

return

s

nngnn

y

-RRB-

declaration

lvalue

h

i

expression

environment

args

-LRB-

-RRB-

P

DCL

-LRB-

-RRB-

in

the

yacc-style

action

syntax

The

effect

is

such

that

when

the

input

a

-LRB-

x

y

-RRB-

b

-LRB-

y

-RRB-

is

parsed

then

the

C

code

BOOLEAN

a

-LRB-

x

y

-RRB-

f

PARSER

b

return

b

-LRB-

y

-RRB-

g

is

printed

But

it

is

easiest

to

represent

the

transformation

using

an

abstract

syntax

declaration

Decl

lvalue

expression

expression

Expr

-LSB-

alternate

-RSB-

alternate

Alt

-LSB-

sequent

-RSB-

sequent

Seqt

rvalue

lvalue

rvalue

VALUE

and

a

corresponding

set

of

interpreters

D

-LSB-

-LSB-

-RSB-

-RSB-

declaration

int

C

code

E

-LSB-

-LSB-

-RSB-

-RSB-

expression

int

-LSB-

varid

-RSB-

-LRB-

VALUE

C

code

-RRB-

A

-LSB-

-LSB-

-RSB-

-RSB-

alternate

int

-LSB-

varid

-RSB-

-LRB-

VALUE

C

code

-RRB-

S

-LSB-

-LSB-

-RSB-

-RSB-

sequent

int

-LSB-

varid

-RSB-

-LRB-

VALUE

C

code

-RRB-

which

construct

C

code

as

a

side

effect

whilst

primarily

returning

a

VALUE

structure

All

these

interpreters

take

the

count

of

unique

names

as

an

argument

and

all

except

D

-LSB-

-LSB-

-RSB-

-RSB-

take

a

list

of

variable

names

-LRB-

the

environment

-RRB-

too

D

-LSB-

-LSB-

-RSB-

-RSB-

constructs

the

environment

list

taking

it

to

be

the

formal

arguments

in

the

lvalue

of

a

declaration

The

others

will

use

the

environment

as

the

list

of

arguments

to

a

newly

invented

function

name

uniq

n

but

the

terminal

lvalues

and

rvalues

always

come

with

their

own

arguments

The

complete

semantics

of

the

set

of

interpreters

is

shown

in

Figure

and

should

be

sufficient

to

enable

the

reader

to

reproduce

precc

in

a

functional

programming

language

given

that

the

printout

macros

P

MT

P

SEQ

P

ALT

follow

the

same

pattern

as

the

P

DCL

macro

above

The

printout

function

P

SEQ

for

example

is

responsible

for

converting

the

parser

expression

a

b

to

the

code

static

BOOLEAN

hiddenfn

-LRB-

x

y

-RRB-

-LCB-

PARSER

a

b

return

-LRB-

a

b

-RRB-

-RCB-

given

the

unique

name

hiddenfn

and

the

environment

x

y

and

the

VALUEs

a

-LRB-

-RRB-

and

b

-LRB-

-RRB-

The

name

hiddenfn

will

in

any

case

be

private

to

the

module

-LRB-

because

of

the

static

keyword

-RRB-

and

is

in

the

D

-LSB-

-LSB-

Decl

x

y

-RSB-

-RSB-

n

py

P

DCL

x

vy

-LRB-

vy

py

-RRB-

E

-LSB-

-LSB-

y

-RSB-

-RSB-

n

x.args

E

-LSB-

-LSB-

Expr

-LSB-

-RSB-

-RSB-

-RSB-

n

e

-LRB-

-LRB-

zero

-LSB-

-RSB-

-RRB-

P

MT

-RRB-

E

-LSB-

-LSB-

Expr

-LRB-

a

b

-RRB-

-RSB-

-RSB-

n

e

-LRB-

vu

pa

pb

P

ALT

vu

va

vb

-RRB-

vu

-LRB-

uniq

n

e

-RRB-

-LRB-

va

pa

-RRB-

A

-LSB-

-LSB-

a

-RSB-

-RSB-

-LRB-

n

-RRB-

-LRB-

vb

pb

-RRB-

E

-LSB-

-LSB-

Expr

b

-RSB-

-RSB-

-LRB-

n

-RRB-

A

-LSB-

-LSB-

Alt

-LSB-

-RSB-

-RSB-

-RSB-

n

e

-LRB-

-LRB-

zero

-LSB-

-RSB-

-RRB-

P

MT

-RRB-

A

-LSB-

-LSB-

Alt

-LRB-

a

b

-RRB-

-RSB-

-RSB-

n

e

-LRB-

vu

pa

pb

P

SEQ

vu

va

vb

-RRB-

vu

-LRB-

uniq

n

e

-RRB-

-LRB-

va

pa

-RRB-

S

-LSB-

-LSB-

a

-RSB-

-RSB-

-LRB-

n

-RRB-

-LRB-

vb

pb

-RRB-

A

-LSB-

-LSB-

Alt

b

-RSB-

-RSB-

-LRB-

n

-RRB-

S

-LSB-

-LSB-

Seq

x

-RSB-

-RSB-

n

e

-LRB-

x

-RRB-

Figure

Interpreters

for

precc

's

abstract

syntax

producing

C

code

output

real

implementation

of

such

a

form

as

to

be

unlikely

ever

to

be

chosen

deliberately

by

a

programmer

The

similar

printout

functions

P

ALT

and

P

MT

generate

code

appropriate

to

the

alternation

of

parsers

and

an

empty

parser

definition

respectively

This

strategy

provides

code

that

compiles

to

an

addressable

function

for

every

scripted

subexpression

A

trace/refusal

semantics

Whilst

the

C

implementation

of

the

specifications

is

always

subject

to

the

vagaries

of

the

C

compiler

the

functional

specification

of

the

precc

parsers

and

combinators

provides

a

suitable

basis

for

verification

That

a

parser

has

been

verified

means

that

it

has

been

proved

to

act

in

a

specified

way

and

to

that

end

we

define

a

trace/refusal

semantics

for

parsers

This

provides

the

language

with

which

to

discuss

the

behaviour

of

parsers

Accordingly

let

a

trace

-LRB-

a

b

r

-RRB-

of

a

parser

p

consist

of

the

absorbed

part

of

the

input

token

stream

-LRB-

a

finite

sequence

of

tokens

-RRB-

the

unabsorbed

part

released

to

the

output

stream

-LRB-

also

a

finite

sequence

of

tokens

-RRB-

and

the

result

value

and

let

Tr

-LRB-

p

-RRB-

be

the

set

of

these

traces

Tr

P

-LRB-

X

-RRB-

P

-LRB-

-LSB-

tok

-RSB-

-LSB-

tok

-RSB-

X

-RRB-

Tr

-LRB-

p

-RRB-

f

-LRB-

a

b

r

-RRB-

j

p

-LRB-

a

b

-RRB-

-LRB-

b

inr

r

-RRB-

g

-LRB-

-RRB-

The

treatment

here

will

ignore

r

because

only

the

serial

composition

a

b

and

the

alternation

a

b

of

parsers

is

considered

not

the

more

general

monadic

composition

a

r

b

But

the

result

value

is

necessary

in

the

general

treatment

Similarly

we

define

a

refusal

-LRB-

a

e

-RRB-

for

a

parser

p

to

be

a

finite

sequence

of

tokens

a

which

causes

an

error

signal

e

The

stream

is

automatically

rewound

afterwards

Re

P

-LRB-

X

-RRB-

P

-LRB-

-LSB-

tok

-RSB-

E

-RRB-

Re

-LRB-

p

-RRB-

f

-LRB-

a

e

-RRB-

j

p

a

-LRB-

a

inl

e

-RRB-

g

-LRB-

-RRB-

The

set

of

traces

and

the

set

of

refusals

together

determine

the

parser

behaviour

uniquely

provided

-LRB-

-RRB-

consideration

is

restricted

to

those

parsers

in

the

parser

type

which

only

absorb

-LRB-

some

or

-RRB-

input

tokens

and

leave

the

remainder

on

the

output

stream

and

-LRB-

-RRB-

the

assumption

is

made

that

parsers

at

least

produce

a

pair

as

output

even

when

they

hang

-LRB-

i.e.

-LRB-

-RRB-

is

the

bottom

element

of

the

output

type

-LRB-

-LSB-

tok

-RSB-

E

X

-RRB-

-RRB-

Both

assumptions

are

true

of

all

the

atomic

parsers

in

precc

's

repertoire

and

are

preserved

by

all

the

parser

combinators

so

it

is

reasonable

It

is

also

perfectly

feasible

to

generalize

the

definition

of

trace

and

refusal

in

order

to

make

them

applicable

to

parsers

which

disobey

these

rules

These

include

miraculous

parsers

-LRB-

parsers

which

may

mutilate

the

output

stream

by

pushing

extra

tokens

for

example

or

perform

output-dependent

parsing

-RRB-

and

non-auto-rewinding

parsers

These

restrictions

if

adhered

to

do

not

however

preclude

parsers

from

examining

the

output

stream

It

is

perfectly

feasible

that

a

parser

should

perform

actions

which

are

dependent

on

the

tokens

to

come

In

practice

the

parser

merely

has

to

look

ahead

a

little

Theorem

The

following

deductions

are

sound

w.r.t.

the

defined

semantics

of

auto-rewinding

parsers

and

the

least

fixed

point

semantics

of

recursion

equations

-LRB-

a

b

c

x

-RRB-

Tr

-LRB-

p

-RRB-

-LRB-

b

c

y

-RRB-

Tr

-LRB-

q

-RRB-

-LRB-

a

b

c

y

-RRB-

Tr

-LRB-

p

q

-RRB-

-LRB-

a

b

x

-RRB-

-LRB-

p

-RRB-

-LRB-

a

b

x

-RRB-

-LRB-

p

q

-RRB-

-LRB-

a

e

-RRB-

-LRB-

p

-RRB-

-LRB-

a

b

y

-RRB-

-LRB-

q

-RRB-

-LRB-

a

b

y

-RRB-

-LRB-

p

q

-RRB-

-LRB-

a

e

-RRB-

-LRB-

p

-RRB-

-LRB-

a

e

-RRB-

-LRB-

p

q

-RRB-

-LRB-

a

b

x

-RRB-

-LRB-

p

-RRB-

-LRB-

b

f

-RRB-

-LRB-

q

-RRB-

-LRB-

a

f

-RRB-

-LRB-

p

q

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

p

-RRB-

-LRB-

a

f

-RRB-

Re

-LRB-

q

-RRB-

-LRB-

a

f

-RRB-

Re

-LRB-

p

q

-RRB-

-LRB-

a

b

x

-RRB-

Tr

-LRB-

Fn

-LSB-

-RSB-

-RRB-

-LRB-

a

b

x

-RRB-

Tr

-LRB-

p

F

-LSB-

p

-RSB-

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

Fn

-LSB-

-RSB-

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

p

F

-LSB-

p

-RSB-

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

-RRB-

-LRB-

-LSB-

-RSB-

a

x

-RRB-

Tr

-LRB-

x

-RRB-

Proof

By

the

definitions

of

and

and

the

other

parser

combinators

For

example

consider

the

proof

that

the

singleton

token

stream

-LSB-

a

-RSB-

is

refused

by

the

compound

parser

hai

hai

where

hai

is

the

parser

that

matches

the

input

token

a

only

with

functionality

hai

-LRB-

a

ts

-RRB-

-LRB-

ts

inr

-LRB-

-RRB-

-RRB-

hai

ts

-LRB-

ts

inl

e

-RRB-

otherwise

and

hai

is

defined

by

the

recursion

equation

hai

a

hai

-LRB-

-RRB-

In

this

case

F

-LSB-

x

-RSB-

is

the

expression

-LRB-

-RRB-

and

it

suffices

to

consider

-LSB-

-RSB-

an

expression

equivalent

to

-LRB-

-RRB-

-LRB-

-RRB-

by

the

monad

laws

To

prove

that

-LSB-

a

-RSB-

is

refused

by

hai

hai

it

suffices

to

use

the

instantiated

rule

for

refusals

of

a

sequential

composition

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

Tr

-LRB-

hai

-RRB-

-LRB-

-LSB-

-RSB-

e

-RRB-

Re

-LRB-

hai

-RRB-

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-RRB-

Re

-LRB-

hai

hai

-RRB-

The

second

antecedent

follows

immediately

from

the

definition

of

hai

above

and

the

first

can

be

proved

by

showing

that

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

is

a

trace

of

the

-LRB-

-RRB-

alternate

within

-LSB-

-RSB-

-LSB-

a

-RSB-

having

been

refused

by

the

alternate

For

the

former

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

Tr

-LRB-

hai

-RRB-

-LRB-

-LSB-

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

Tr

-LRB-

eta

-LRB-

-RRB-

-RRB-

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

Tr

-LRB-

-LRB-

-RRB-

-RRB-

It

remains

to

show

that

-LSB-

a

-RSB-

is

refused

by

The

rule

to

use

is

-LRB-

-LSB-

a

-RSB-

-LSB-

-RSB-

-LRB-

-RRB-

-RRB-

Tr

-LRB-

hai

-RRB-

-LRB-

-LSB-

-RSB-

e

-RRB-

Re

-LRB-

hai

-RRB-

-LRB-

-LSB-

a

-RSB-

e

-RRB-

Re

-LRB-

-RRB-

so

-LSB-

a

-RSB-

is

refused

by

and

hence

by

How

do

such

proofs

tie

up

with

the

parser

syntax

used

by

precc

The

syntax

is

implemented

by

precc

as

parser

functions

in

which

p

q

becomes

p

q

and

pjq

becomes

p

q

so

that

when

one

asks

if

input

stream

a

b

is

accepted

by

the

parser

constructed

by

parser

description

p

one

is

really

asking

is

-LRB-

a

b

x

-RRB-

in

the

trace

of

p

And

that

question

is

answered

by

the

axiomatic

semantics

given

above

Allowing

the

use

of

parameters

certainly

makes

the

deduction

system

incomplete

with

respect

to

the

intended

model

however

because

one

can

specify

the

Turing

machine

grammar

a

specification

which

takes

a

parameter

-LRB-

l

c

r

-RRB-

representing

the

tape

of

a

Turing

machine

The

l

represents

the

tape

to

the

left

of

the

read/write

head

the

r

the

tape

to

the

right

and

c

the

symbol

under

the

head

The

grammar

contains

at

most

one

element

IWillStop

and

that

only

when

the

input

tape

-LRB-

l

c

r

-RRB-

corresponds

to

a

position

from

which

a

Turing

machine

will

eventually

halt

T

-LRB-

l

c

r

-RRB-

-RRB-

c

MoveLeft

-LRB-

T

-LRB-

tl

l

hd

l

c

r

-RRB-

j

j

-RRB-

c

Stop

-LRB-

hIWillStopi

Asking

whether

the

singleton

input

token

IWillStop

is

accepted

by

the

grammar

means

asking

whether

-LRB-

-LSB-

IWillStop

-RSB-

-LSB-

-RSB-

x

-RRB-

is

in

the

trace

and

this

is

equivalent

to

asking

whether

or

not

a

Turing

machine

will

stop

so

it

can

not

be

algorithmically

decided

Moreover

there

may

not

always

be

a

sequence

of

proof

steps

which

shows

that

the

trace

is

not

accepted

when

it

is

not

This

example

is

valid

whenever

-LRB-

rules

for

-RRB-

the

literal

match

hai

and

the

conditional

-RRB-

v

-LRB-

are

included

in

the

deduction

system

even

if

only

non-recursive

arithmetic

operations

and

logical

comparisons

are

permitted

-LRB-

-LSB-

a

-RSB-

b

x

-RRB-

Tr

-LRB-

hai

-RRB-

a

-LRB-

-LRB-

b

-RRB-

e

-RRB-

Re

-LRB-

hai

-RRB-

-LRB-

-LSB-

-RSB-

e

-RRB-

Re

-LRB-

hai

-RRB-

v

-LRB-

-LSB-

-RSB-

b

x

-RRB-

Tr

-LRB-

-RRB-

v

-LRB-

-RRB-

v

-LRB-

a

e

-RRB-

Re

-LRB-

-RRB-

v

-LRB-

-RRB-

but

it

is

altogether

another

question

if

one

can

provide

any

reasonable

fixed

set

of

atomic

parsers

such

as

the

hai

and

-RRB-

v

-LRB-

here

confine

oneself

to

the

parser

combinators

and

expect

the

associated

deduction

system

to

be

complete

for

assertions

of

the

form

-LRB-

a

b

x

-RRB-

Tr

-LRB-

p

-RRB-

and

-LRB-

a

e

-RRB-

Re

-LRB-

p

-RRB-

It

is

certainly

not

true

that

the

axioms

given

force

parsers

to

be

deterministic

functions

since

there

is

no

axiom

which

states

that

refusals

and

traces

must

be

disjoint

nor

that

parsers

must

return

a

unique

result

from

a

given

trace

but

this

is

not

an

essential

objection

What

is

the

relationship

with

CSP

-LSB-

-RSB-

which

also

has

a

trace/refusal

semantics

Clearly

a

CSP

program

can

be

viewed

as

a

parser

specification

for

tokens

which

are

really

events

on

a

universal

timestream

but

a

CSP

program

can

not

look

ahead

as

a

parser

can

In

this

respect

-LRB-

terminating

-RRB-

CSP

programs

have

representations

as

parsers

which

accept

a

trace

only

on

the

basis

of

those

tokens

already

absorbed

-LRB-

a

b

x

-RRB-

Tr

-LRB-

p

-RRB-

-RRB-

-LRB-

a

x

-RRB-

Tr

-LRB-

p

-RRB-

for

any

choice

of

continuation

in

particular

the

empty

continuation

-LSB-

-RSB-

So

-LRB-

a

b

x

-RRB-

Tr

-LRB-

p

-RRB-

iff

-LRB-

a

-LSB-

-RSB-

x

-RRB-

Tr

-LRB-

p

-RRB-

and

all

the

information

in

the

trace

is

contained

in

the

absorbed

tokens

The

sequential

and

asymmetric

disjunctive

-LRB-

try

a

else

b

-RRB-

compositions

of

terminating

CSP

programs

maps

across

to

the

sequential

composition

and

alternation

of

parsers

respectively

The

zero

parser

is

the

image

of

the

abort

CSP

program

The

parallel

concurrent

composition

of

CSP

programs

is

not

atomic

in

this

setting

It

can

be

modelled

as

the

sequential

composition

-RSB-

p

-LSB-

q

of

parsers

where

the

-RSB-

-LSB-

operator

is

the

precc

phantom

which

tries

a

parser

out

and

then

backtracks

on

success

-LRB-

or

failure

of

course

but

that

always

happens

-RRB-

-RSB-

p

-LSB-

ts

-LRB-

ts

x

-RRB-

where

-LRB-

x

-RRB-

p

ts

with

rules

-LRB-

a

b

x

-RRB-

Tr

-LRB-

p

-RRB-

-LRB-

-LSB-

-RSB-

a

b

x

-RRB-

Tr

-LRB-

-RSB-

p

-LSB-

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

p

-RRB-

-LRB-

a

e

-RRB-

Re

-LRB-

-RSB-

p

-LSB-

-RRB-

For

a

token

stream

to

be

accepted

by

the

-RSB-

p

-LSB-

q

combination

it

has

to

be

accepted

by

both

p

and

q

and

for

it

to

be

refused

it

has

to

be

refused

by

one

or

both

The

interpretation

of

recursion

is

also

different

from

CSP

The

parser

here

is

the

empty

set

program

in

CSP

with

no

traces

or

refusals

and

the

fixed

point

given

here

is

the

least

fixed

point

above

it

In

contrast

CSP

uses

the

chaos

parser

which

has

all

traces

and

refusals

as

the

basis

for

recursion

and

chooses

the

greatest

fixed

point

below

it

This

is

usually

not

a

deterministic

function

but

certainly

its

traces

and

refusals

are

always

supersets

of

the

traces

and

refusals

of

the

fixed

point

given

here

Note

that

the

trace/refusal

axiomatic

semantics

here

provides

a

suitable

basis

for

an

implementation

of

precc

as

a

logic

program

A

query

to

a

parser

implementation

would

then

yield

a

list

of

possible

traces

-LRB-

or

refusals

-RRB-

for

the

parser

Trials

Comparative

tests

were

run

between

precc

and

GNU

bison

-LRB-

on

a

PC

under

MSDOS

-RRB-

and

yacc

-LRB-

on

a

HP

workstation

running

HP-unix

-RRB-

A

parser

for

the

small

language

was

compiled

by

all

compiler-compilers

using

as

a

basis

a

yacc

following

the

definition

given

in

-LSB-

-RSB-

Although

many

other

languages

have

been

treated

by

precc

they

have

for

the

most

part

been

without

yacc

specifications

and

is

the

first

complete

language

for

which

an

existing

-LRB-

and

working

-RRB-

yacc

specification

has

been

used

as

a

basis

for

a

precc

specification

As

few

changes

as

possible

were

made

-LRB-

the

resulting

specification

is

better

tuned

to

yacc

than

to

precc

-RRB-

and

all

parsers

used

the

same

token

lexer

compiled

by

the

fast

lex

utility

flex

Both

yacc

and

bison

were

able

to

read

and

compile

the

original

yacc

script

which

therefore

appears

to

be

fairly

standard

All

the

compiled

parsers

differ

only

slightly

in

behaviour

On

a

suite

of

Oberon

test

scripts

supplied

with

the

original

yacc

specification

all

are

accepted

by

the

bison

and

yacc

parser

and

are

accepted

and

rejected

by

the

precc

parser

One

of

the

rejected

scripts

-LRB-

-RRB-

definitely

contains

an

error

CONST

x

y

Simultaneous

constant

declarations

are

not

allowed

in

Oberon

In

the

second

rejected

script

-LRB-

-RRB-

precc

parser

the

constructions

a

-LRB-

a

-RRB-

c

-LRB-

x

-RRB-

a

-LRB-

a

-RRB-

c

-LSB-

x

y

-RSB-

supplied

by

T.

Channon

-LRB-

tchannon@black.demon.co.uk

-RRB-

Yacc

specification

originally

by

Stephen

J.

Bevan

-LRB-

bevan@cs.man.ac.uk

-RRB-

and

copyright

c

Department

of

Computer

Science

University

of

Manchester

are

the

cause

a

-LRB-

a

-RRB-

c

ought

to

be

accepted

-LRB-

a

record

reference

off

a

function

result

-RRB-

and

so

also

ought

a

c

-LRB-

x

-RRB-

-LRB-

a

qualified

function

call

-RRB-

and

a

c

-LSB-

x

y

-RSB-

-LRB-

a

qualified

array

reference

-RRB-

but

mixtures

ought

to

have

been

rejected

as

nonsense

The

third

script

-LRB-

-RRB-

contains

the

error

x

f

-LCB-

z

-RCB-

invalid

set

element

The

reason

for

the

acceptances

by

the

yacc

and

bison

parsers

is

certainly

the

shift/reduce

conflicts

reported

in

the

script

The

CONST

acceptance

is

caused

by

a

shift

into

the

list

state

when

a

list

is

not

specified

by

the

script

and

in

the

second

script

too

it

seems

that

the

yacc

parser

is

shifting

where

the

script

wants

reduce

with

the

result

that

an

invalid

construction

is

wrongly

accepted

The

translated

precc

script

does

not

pick

up

this

side-effect

and

hence

-LRB-

correctly

-RRB-

rejects

the

unspecified

constructions

The

set

element

error

is

also

the

kind

of

problem

that

might

be

expected

from

a

shift

instead

of

a

reduce

There

are

states

in

the

yacc/bison

compilation

and

intermediate

functions

in

the

precc

compilation

-LRB-

a

measure

of

the

additional

complexity

introduced

by

precc

over

and

above

that

in

the

script

itself

-RRB-

GNU

bison

's

code

is

hand

crafted

and

precc

's

is

generated

from

a

specification

but

precc

appears

to

be

faster

and

certainly

could

be

speeded

up

by

removing

redundancies

from

the

generated

code

and

buffering

I/O

and

using

a

lexical

tokenizer

-LRB-

bison

runs

one

of

the

fastest

LALR

algorithms

-RRB-

As

a

test

of

the

compiled

executable

the

test

scripts

were

passed

through

the

parsers

in

sequence

There

are

a

total

of

lines

in

the

scripts

Bison/yacc

code

is

noticeably

quicker

to

run

but

the

specification

was

written

for

bison

not

precc

and

the

translated

script

for

precc

contains

forced

backtracks

-LRB-

i.e.

some

poorly

crafted

productions

-RRB-

In

particular

the

three

error

scripts

where

errors

are

detected

will

have

slowed

the

precc

parser

down

through

the

extra

backtracking

searches

for

alternate

parses

as

well

as

generating

more

output

error

messages

In

conclusion

in

the

tests

precc

appeared

to

be

about

faster

at

compiling

and

the

compiled

parsers

about

slower

than

yacc

or

bison

equivalents

but

the

factors

discussed

above

tended

to

bias

the

results

in

favour

of

the

yacc

implementations

which

in

any

case

failed

to

detect

several

deliberate

errors

in

the

test

scripts

Future

developments

It

is

planned

to

extend

precc

to

take

full

advantage

of

the

monad

algebra

set

out

in

this

paper

in

order

to

implement

synthetic

attributes

Rather

than

the

present

yacc-style

actions

as

in

the

following

typical

script

sum

summand

h

i

summand

in

which

an

attached

action

manipulates

a

runtime

stack

of

values

the

current

research

prototype

uses

the

monadic

operation

to

implement

a

script

in

the

following

style

sum

summandnx

h

i

summandny

x

y

where

the

attached

attributes

appear

as

nx

constructs

The

new

strategy

moves

manipulation

of

synthetic

attributes

back

into

parse-time

when

the

manipulation

of

inherited

attributes

-LRB-

parameters

-RRB-

also

takes

place

thus

making

data

flow

possible

in

both

directions

between

synthetic

and

inherited

attributes

instead

of

as

at

present

only

from

inherited

to

synthetic

attributes

The

compiled

C

code

has

the

functionality

of

sum

summand

x

-LRB-

literal

-RRB-

instead

of

sum

summand

-LRB-

literal

-RRB-

but

it

still

has

not

been

decided

exactly

what

syntax

to

use

to

assign

the

synthetic

attributes

at

parsetime

which

would

eliminate

the

need

for

a

run-time

stack

entirely

It

may

be

best

to

allow

each

parser

sequence

to

be

terminated

by

an

optional

C

expression

as

in

sum

summandnx

h

i

summandny

x

y

but

there

is

some

danger

of

confusion

between

precc

's

alternation

operator

symbol

j'

which

may

follow

and

the

C

bitwise-or

operator

which

may

appear

in

the

C

expression

It

is

also

intended

to

take

advantage

of

the

exception

type

mechanism

to

provide

better

control

over

error

actions

The

treatment

in

this

paper

has

focussed

on

the

choice

E

which

forces

a

unique

error

signal

inl

-LRB-

-RRB-

in

the

exception

type

E+X

But

the

type

E

can

be

different

and

allowing

many

different

signals

makes

a

number

of

error

actions

possible

Environment

MSDOS

HP

HP

-LCB-

UX

Utility

precc

bison

precc

yacc

Source

oberon

y

oberon

y

Size

-LRB-

bytes

-RRB-

Output

y

tab.c

y.tab.c

Size

-LRB-

bytes

-RRB-

Compile

time

-LRB-

secs

-RRB-

Shift/reduces

Run

time

-LRB-

secs

-RRB-

Errors

detected

Table

Comparative

figures

for

compilation

and

execution

-LRB-

on

a

script

test

suite

-RRB-

of

by

precc

and

GNU

bison

and

unix

yacc

-LRB-

Timings

include

system

and

execution

time

-RRB-

At

present

the

default

action

on

error

is

to

do

nothing

but

rewind

the

incoming

token

stream

and

this

choice

is

preserved

by

parser

combinators

like

But

it

is

possible

to

parameterize

each

parser

with

a

custom

error

handler

and

thus

obtain

a

more

flexible

response

At

the

implementation

level

in

C

it

is

simple

-LCB-

an

error

value

return

should

raise

a

signal

that

invokes

a

unique

handler

The

syntax

currently

implemented

in

the

research

prototype

allocates

handlers

only

for

named

parsers

rather

than

every

parser

constructed

-LRB-

there

is

a

parser

corresponding

to

every

subexpression

-RRB-

fsumerrg

sum

summandnx

h

i

summandny

x

y

HANDLER

sumerr

-LRB-

-RRB-

f

g

and

handlers

in

reality

also

have

more

information

available

to

them

than

the

description

of

autorewinding

parsers

given

here

permits

The

maximum

depth

of

penetration

into

the

parse

stream

is

recorded

for

example

so

handlers

know

what

tokens

a

parser

has

seen

This

information

is

useful

because

it

appears

that

the

real

error

from

a

human

standpoint

is

indeed

found

close

to

the

farthest

successfully

parsed

token

This

may

be

an

artifact

of

the

way

that

grammar

specification

scripts

are

usually

written

but

it

appears

reasonable

that

in

the

absence

of

other

information

an

error

should

be

deemed

to

have

occurred

at

the

first

point

after

the

longest

initial

section

of

the

parse

stream

to

still

have

made

sense

For

example

the

parser

in

Figure

accepts

only

the

Fibonacci

sequence

as

input

and

errors

are

detected

at

the

first

digit

within

a

number

that

fails

to

match

up

with

the

expected

series

Because

handlers

have

access

to

information

about

what

a

parser

has

seen

their

behaviour

appears

miraculous

Miraculous

parsers

have

properties

which

are

not

always

implementable

within

the

given

type

constraints

The

analogy

with

CSP

noted

here

has

also

encouraged

us

to

incorporate

the

CSP

hiding

construct

within

precc

In

the

parsing

context

the

a

-LRB-

p

-RRB-

syntax

means

that

parser

a

sees

the

input

stream

of

tokens

only

after

it

has

been

filtered

by

predicate

p

This

is

useful

for

example

for

ridding

the

input

stream

of

white

space

The

syntax

may

eventually

be

extended

to

allow

more

than

the

two

levels

of

parsing

that

this

mechanism

explicitly

allows

Summary

Precc

is

a

compiler-compiler

based

on

a

powerful

higher

order

-LRB-

monadic

-RRB-

programming

model

and

produces

modular

and

highly

portable

and

maintainable

C

code

All

the

standard

BNF

constructs

are

built

into

the

grammar

definition

language

and

more

may

be

defined

using

precc

's

own

macros

The

utility

has

been

used

to

generate

parsers

and

compilers

for

both

commercially

available

-LRB-

cobol

-LSB-

-RSB-

-RRB-

and

experimental

-LRB-

Uniform

-LSB-

-RSB-

-RRB-

languages

and

the

treatment

of

occam

-LSB-

-RSB-

has

been

described

in

this

paper

A

comparative

study

with

yacc

-LSB-

-RSB-

and

GNU

's

bison

clone

has

been

undertaken

using

-LSB-

-RSB-

as

the

input

grammar

Precc

also

uses

itself

to

parse

its

own

input

specification

and

compile

its

own

code

Precc

has

been

designed

to

lift

some

of

the

programming

and

implementational

restrictions

imposed

by

compiler-compilers

in

the

style

of

the

unix

yacc

utility

The

design

notably

provides

infinite

lookahead

and

backtracking

in

place

of

yacc

's

one-token

lookahead

and

a

declarative

semantics

Scripts

can

contain

arbitrarily

complex

BNF

expressions

and

these

may

take

parameters

When

the

parameters

are

other

grammar

descriptions

the

scripted

clause

behaves

like

a

referentially

transparent

higher

order

macro

which

helps

achieve

clearer

coding

and

maintainability

Scripts

may

in

any

case

be

divided

up

into

independent

modules

and

compiled

separately

and

linked

together

independently

in

order

to

cater

for

improved

change

management

References

-LSB-

-RSB-

A.

Y.

Aho

The

theory

of

parsing

translation

compiling

New

Jersey

USA

Vol

-LCB-

-LSB-

-RSB-

A.

V.

Aho

and

J.

D.

Ullman

Principles

of

compiler

design

Addison-Wesley

Publishing

Company

-LSB-

-RSB-

J.

P.

Bowen

From

programs

to

object

code

using

logic

and

logic

programming

in

R.

Giegerich

and

S.

L.

Graham

-LRB-

eds

-RRB-

Code

Generation

-LCB-

Concepts

Tools

Techniques

Proc

International

Workshop

on

Code

Generation

Dagstuhl

Germany

-LCB-

May

SpringerVerlag

Workshops

in

Computing

pp.

-LCB-

-LSB-

-RSB-

J.

P.

Bowen

and

P.

T.

Breuer

Occam

's

Razor

the

cutting

edge

of

parser

technology

in

Proc

TOULOUSE

Fifth

International

Conference

on

Software

Engineering

and

its

Applications

Toulouse

France

-LCB-

December

-LSB-

-RSB-

P.

T.

Breuer

A

PREttier

Compiler-Compiler

higher

order

programming

in

C

in

Proc

TOULOUSE

Fifth

International

Conference

on

Software

Engineering

and

its

Applications

Toulouse

France

-LCB-

December

-LSB-

-RSB-

W.

F.

Clocksin

and

C.

S.

Mellish

Programming

in

Prolog

Springer-Verlag

edition

-LSB-

-RSB-

J.

Fairbairn

Making

form

follow

function

An

exercise

in

functional

programming

style

Software

Practice

and

Experience

-LRB-

-RRB-

pp.

-LCB-

-LSB-

-RSB-

C.A.

R.

Hoare

Communicating

Sequential

Processes

Prentice

Hall

International

Series

in

Computer

Science

-LSB-

-RSB-

G.

Hutton

Higher-order

functions

for

parsing

Journal

of

Functional

Programming

To

appear

-LSB-

-RSB-

INMOS

Limited

Occam

reference

manual

Prentice

Hall

International

Series

in

Computer

Science

-LSB-

-RSB-

S.

C.

Johnson

and

M.

E.

Lesk

Language

development

tools

The

Bell

System

Technical

Journal

-LRB-

-RRB-

part

pp.

-LCB-

July/August

-LSB-

-RSB-

S.

C.

Johnson

Yacc

meets

C++

in

UNIX

around

the

World

Proc

Spring

EUUG

Conference

pp.

-LCB-

-LSB-

-RSB-

B.

W.

Kernighan

and

D.M.

Ritchie

The

C

prgramming

language

edition

PrenticeHall

Software

Series

-LSB-

-RSB-

D.E.

Knuth

Literate

programming

The

Computer

Journal

-LRB-

-RRB-

pp.

-LCB-

May

-LSB-

-RSB-

B.

B.

Kristensen

and

O.

L.

Madsen

Methods

for

computing

LALR

-LRB-

K

-RRB-

lookahead

ACM

TOPLAS

-LRB-

-RRB-

pp.

-LCB-

-LSB-

-RSB-

J.

L.

Lewi

K.

de

Blaminck

I.

van

Horebeek

and

E.

Steegmans

Software

development

by

LL

-LRB-

-RRB-

syntax

description

John

Wiley

Sons

Ltd

-LSB-

-RSB-

G.H.

Merrill

Parsing

non-LR

-LRB-

k

-RRB-

grammars

with

yacc

SAS

Institute

Inc.

SAS

Campus

Drive

Cary

NC

USA

Submitted

to

Software

Practice

and

Experience

-LSB-

-RSB-

P.

Milner

and

M.

Tofte

Commentary

on

Standard

ML

MIT

Press

-LSB-

-RSB-

H.

Mossenbock

and

N.

Wirth

The

Programming

Language

Institut

fur

Computersysteme

ETH

Zurich

Switzerland

January

-LSB-

-RSB-

J.

C.H.

Park

and

K.

M.

Choe

Remarks

on

recent

algorithms

for

LALR

lookahead

sets

ACM

Sigplan

Notices

-LRB-

-RRB-

pp.

April

-LSB-

-RSB-

A.

Parkin

COBOL

for

students

Edward

Arnold

London

-LSB-

-RSB-

S.

Sippo

Parsing

theory

Vol

EATCS

Monographs

on

Theoretical

Computer

Science

Springer-Verlag

-LSB-

-RSB-

C.

Stanley-Smith

and

A.

Cahill

UNIFORM

A

language

geared

to

system

description

and

transformation

University

of

Limerick

Ireland

-LSB-

-RSB-

B.

Stroustrup

The

C++

programming

language

edition

Addison-Wesley

-LSB-

-RSB-

P.

Wadler

Comprehending

monads

in

G.

Kahn

-LRB-

ed

-RRB-

LISP

and

functional

programming

Proc

ACM

Conference

ACM

Press

New

York

USA

pp.

-LCB-

-LSB-

-RSB-

D.

Weber-Wulff

Proven

correct

front-end

specification

in

B.

von

Karger

-LRB-

ed

-RRB-

Compiler

Development

chapter

Vol

ESPRIT

BRA

Provably

Correct

Systems

ProCoS

Draft

Final

Deliverable

October

-LSB-

-RSB-

L.M.

Wegner

On

parsing

two-level

grammars

Acta

Informatica

pp.

-LCB-

FTP

access

Precc

is

available

via

anonymous

FTP

access

from

the

Internet

host

ftp.comlab.ox.ac.uk

-LRB-

-RRB-

The

relevant

files

are

Z

-LRB-

in

compressed

tar

form

-RRB-

for

unix

systems

and

under

the

directory

pub/Programs/preccx

msdos

for

use

on

PC-compatible

computers

under

DOS

The

latter

directory

contains

scripts

for

several

different

languages

There

is

a

nominal

charge

for

supply

on

a

floppy

disk

instead

compiler-compiler
infinite-lookahead
top-down parsing