I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The method self.predict(i) applies the prediction operation to the state $Q_i$:
If $\langle A \rightarrow \beta \bullet C \delta, k \rangle \in Q_j$, then
the parser tries to recognize $C\delta$ after having read mString[:j+1].  To this end
it has to parse $C$ in the string mString[j+1:].
Therefore, if $C \rightarrow \gamma$ is a rule of our grammar,
we add the Earley item $\langle C \rightarrow \bullet \gamma, j\rangle$ to the set $Q_j$:
$$ \langle A \rightarrow \beta \bullet C \delta, k\rangle \in Q_j
       \wedge (C \rightarrow \gamma) \in R
       \;\rightarrow\;
       Q_j := Q_j \cup\bigl{ \langle C \rightarrow \bullet\gamma, j\rangle\bigr}.
$$
As the right hand side $\gamma$ might start with a variable, the function uses a fix point iteration
until no more Earley items are added to $Q_j$.
Can you write Python code for it?

def predict(self, i):
    change = False
    added  = True
    Qi     = self.mStateList[i]
    while added:
        added = False
        newQi = set()
        for item in Qi:
            c = item.nextVar()
            if c != None:
                for rule in self.mGrammar.mRules:
                    if c == rule[0]:
                        newQi.add(EarleyItem(c, (), rule[1:], i))
        if not (newQi <= Qi):
            change = True
            added  = True
            print("prediction:")
            for newItem in newQi:
                if newItem not in Qi:
                    print(f'{newItem} added to Q{i}')
            self.mStateList[i] |= newQi
            Qi = self.mStateList[i]
    return change

EarleyParser.predict = predict
del predict


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
We start by generating both scanner and parser.
Can you write Python code for it?

!antlr4 -Dlanguage=Python3 Grammar.g4

from GrammarLexer  import GrammarLexer
from GrammarParser import GrammarParser
import antlr4


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
We use Antlr to develop a parser for this Grammar.
The pure grammar to parse this type of grammar is stored in
the file Pure.g4.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter-Matching.ipynb".
The first task is:
An Interpreter for a Simple Programming Language
In this notebook we develop an interpreter for a small programming language.
The grammar for this language is stored in the file Pure.g4.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter-Matching.ipynb".
The first task is:
The function execute_tuple takes two arguments:
- Statement_List is a list of statements,
- Values is a dictionary assigning integer values to variable names.
The function executes the statements in Statement_List.  If an assignment statement is executed,
the dictionary Values is updated.
Can you write Python code for it?

def execute_tuple(Statement_List, Values={}):
    for stmnt in Statement_List:
        execute(stmnt, Values)


I am working on the file "ANTLR4-Python/Interpreter/Interpreter-Matching.ipynb".
The first task is:
The function main takes one parameter file.  This parameter is a string specifying the name of file containing a program.
The function reads this program and executes it.
Can you write Python code for it?

def main(file):
    with open(file, 'r') as handle:
        program_text = handle.read()
    input_stream  = antlr4.InputStream(program_text)
    lexer         = SimpleLexer(input_stream)
    token_stream  = antlr4.CommonTokenStream(lexer)
    parser        = SimpleParser(token_stream)
    result        = parser.program()
    Statements    = result.stmnt_list
    ast           = tuple2dot(Statements)
    print(Statements)
    display(ast)
    ast.render('ast', view=True)
    execute_tuple(Statements)


I am working on the file "ANTLR4-Python/Interpreter/Interpreter-Matching.ipynb".
The first task is:
The parser shown above will transform the program sum.sl into the nested tuple stored in the file sum.ast.
Can you write Python code for it?

!type sum.ast

!cat sum.ast

!antlr4 -Dlanguage=Python3 Simple.g4

from SimpleLexer  import SimpleLexer
from SimpleParser import SimpleParser
import antlr4

%run ../AST-2-Dot.ipynb


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
An Interpreter for a Simple Programming Language
In this notebook we develop an interpreter for a small programming language.
The grammar for this language is stored in the file Pure.g4.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The file Simple.g4 contains a parser for the language described by the grammar Pure.g4. This parser returns
an abstract syntax tree.  This tree is represented as a nested tuple.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The function evaluate takes two arguments:
- expr is a logical expression or an arithmetic expression,
- Values is a dictionary assigning integer values to variable names.
The function evaluates the given expression and returns this value.
Can you write Python code for it?

def evaluate(expr, Values):
    if isinstance(expr, int):
        return expr
    if isinstance(expr, str):
        return Values[expr]
    op = expr[0]
    if op == 'read':
        return int(input('Please enter a natural number:'))
    if op == '==':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) == evaluate(rhs, Values)
    if op == '<':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) < evaluate(rhs, Values)
    if op == '+':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) + evaluate(rhs, Values)
    if op == '-':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) - evaluate(rhs, Values)
    if op == '*':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) * evaluate(rhs, Values)
    if op == '/':
        _, lhs, rhs = expr
        return evaluate(lhs, Values) / evaluate(rhs, Values)
    assert False, f'{expr} unexpected'

!type sum.sl

!cat sum.sl

main('sum.sl')

!type factorial.sl

!cat factorial.sl

main('factorial.sl')

!del *.py *.tokens *.interp
!del *.pdf
!del ast

!rmdir /Q /S __pycache__

!dir /B

!rm *.py *.tokens *.interp
!rm ast
!rm -r __pycache__/
!rm *.pdf

!ls


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The function execute takes two arguments:
- stmnt is a statement,
- Values is a dictionary assigning integer values to variable names.
The function executes the statements in Statement_List.  If an assignment statement is executed,
the dictionary Values is updated.
Can you write Python code for it?

L = [1,2,3,4,5]
a, b, *R = L
a, b, R

def execute(stmnt, Values):
    op = stmnt[0]
    if stmnt == 'program':
        pass
    elif op == ':=':
        _, var, value = stmnt
        Values[var] = evaluate(value, Values)
    elif op == 'read':
        _, var = stmnt
        Values[var] = int(input())
    elif op == 'print':
        _, expr = stmnt
        print(evaluate(expr, Values))
    elif op == 'if':
        _, test, *SL = stmnt
        if evaluate(test, Values):
            execute_tuple(SL, Values)
    elif op == 'while':
        _, test, *SL = stmnt
        while evaluate(test, Values):
            execute_tuple(SL, Values)
    else:
        assert False, f'{stmnt} unexpected'


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The function execute takes two arguments:
- stmnt is a statement,
- Values is a dictionary assigning values to variable names.
The function executes the statements in Statement_List.  If an assignment statement is executed,
the dictionary Values is updated.
The following trick can be used to split a list into its components.
Can you write Python code for it?

L = [1, 2, 3, 4, 5]
a, b, *R = L
a, b, R

def execute(stmnt, Values):
    op = stmnt[0]
    if stmnt == 'program':
        pass
    elif op == ':=':
        _, var, value = stmnt
        Values[var] = evaluate(value, Values)
    elif op == 'print':
        _, expr = stmnt
        print(evaluate(expr, Values))
    elif op == 'if':
        _, test, *SL = stmnt
        if evaluate(test, Values):
            execute_tuple(SL, Values)
    elif op == 'while':
        _, test, *SL = stmnt
        while evaluate(test, Values):
            execute_tuple(SL, Values)
    else:
        assert False, f'{stmnt} unexpected'


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The function main takes one parameter file.  This parameter is a string specifying a program file.
The function reads the program contained in this file and executes it.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The grammar shown above does only contain skip actions.  The corrsponding grammar that is enriched with actions is stored in the file Simple.g4.
An example program that conforms to this grammar is stored in the file sum.sl.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Interpreter/Interpreter.ipynb".
The first task is:
The parser shown above will transform the program sum.sl into the nested tuple stored in the file sum.ast.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Extended Marked Rules
The class ExtendedMarkedRule stores a single marked rule of the form
$$ v \rightarrow \alpha \bullet \beta : L $$
where the variable $v$ is stored in the member variable mVariable, while $\alpha$ and $\beta$ are stored in the variables mAlphaand mBeta respectively.  The set of follow tokens $L$ is stored in the variable mFollow. These variables are assumed to contain tuples of grammar symbols.  A grammar symbol is either
- a variable,
- a token, or
- a literal, i.e. a string enclosed in single quotes.
Later, we need to maintain sets of marked rules to represent states.  Therefore, we have to define the methods __eq__, __ne__, and __hash__.
Can you write Python code for it?

class ExtendedMarkedRule():
    def __init__(self, variable, alpha, beta, follow):
        self.mVariable = variable
        self.mAlpha    = alpha
        self.mBeta     = beta
        self.mFollow   = follow

    def __eq__(self, other):
        return isinstance(other, ExtendedMarkedRule) and \
               self.mVariable == other.mVariable     and \
               self.mAlpha    == other.mAlpha        and \
               self.mBeta     == other.mBeta         and \
               self.mFollow   == other.mFollow

    def __ne__(self, other):
        return not self.__eq__(other)

    def __hash__(self):
        return hash(self.mVariable) + \
               hash(self.mAlpha)    + \
               hash(self.mBeta)     + \
               hash(self.mFollow)

    def __repr__(self):
        alphaStr  = ' '.join(self.mAlpha)
        betaStr   = ' '.join(self.mBeta)
        if len(self.mFollow) > 1:
            followStr = '{' + ','.join(self.mFollow) + '}'
        else:
            followStr = ','.join(self.mFollow)
        return f'{self.mVariable} → {alphaStr} • {betaStr}: {followStr}'


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a Grammar, the function compute_tables computes
- the sets First(v) and Follow(v) for every variable v,
- the set of all states of the LR-Parser,
- the action table, and
- the goto table.
Given a grammar g,
- the set g.mFirst is a dictionary such that g.mFirst[a] = First[a] and
- the set g.mFollow is a dictionary such that g.mFollow[a] = Follow[a] for all variables a.
Can you write Python code for it?

def compute_tables(self):
    self.mFirst  = initialize_dictionary(self.mVariables)
    self.mFollow = initialize_dictionary(self.mVariables)
    self.compute_first()
    self.compute_follow()
    self.compute_rule_names()
    self.all_states()
    self.compute_action_table()
    self.compute_goto_table()

Grammar.compute_tables = compute_tables
del compute_tables


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a grammar symbol name, which is either a variable, a token, or a literal, the function is_var checks whether name is a variable.  The function can distinguish variable names from tokens and literals because variable names consist only of lower case letters, while tokens are all uppercase and literals start with the character "'".
Can you write Python code for it?

def is_var(name):
    return name[0] != "'" and name.islower()


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a list Rules of GrammarRules, the function collect_variables(Rules) returns the set of all variables occuring in Rules.
Can you write Python code for it?

def collect_variables(Rules):
    Variables = set()
    for rule in Rules:
        Variables.add(rule.mVariable)
        for item in rule.mBody:
            if is_var(item):
                Variables.add(item)
    return Variables


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a set Rules of GrammarRules, the function collect_tokens(Rules) returns the set of all tokens and literals occuring in Rules.
Can you write Python code for it?

def collect_tokens(Rules):
    Tokens = set()
    for rule in Rules:
        for item in rule.mBody:
            if not is_var(item):
                Tokens.add(item)
    return Tokens


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a set of extended marked rules $\mathcal{M}$ and a grammar symbol $X$, the function $\texttt{goto}(\mathcal{M}, X)$
is defined as follows:
$$\texttt{goto}(\mathcal{M}, X) := \texttt{closure}\Bigl( \bigl{
   a \rightarrow \beta\, X \bullet \delta:L \bigm| (a \rightarrow \beta \bullet X\, \delta:L) \in \mathcal{M}
   \bigr} \Bigr).
$$
Can you write Python code for it?

def goto(self, Marked_Rules, x):
    Result = set()
    for mr in Marked_Rules:
        if mr.symbol_after_dot() == x:
            Result.add(mr.move_dot())
    return combine_rules(self.cmp_closure(Result))

Grammar.goto = goto
del goto


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a set of extended marked rules M, the function combine_rule combines those extended marked ruless that have the same core:  If
$$ a \rightarrow \beta \bullet \gamma : L $$
is an extended marked rule, then its core is defined as the marked rule
$$ a \rightarrow \beta \bullet \gamma. $$
If $a \rightarrow \beta \bullet \gamma : L_1$ and $a \rightarrow \beta \bullet \gamma : L_2$ are two extended marked rules, then they can be combined into the rule
$$ a \rightarrow \beta \bullet \gamma : L_1\cup L_2 $$
Can you write Python code for it?

def combine_rules(M):
    Result = set()
    Core   = set()
    for emr1 in M:
        Follow = set()
        core1 = emr1.to_marked_rule()
        if core1 in Core:
            continue
        Core.add(core1)
        for emr2 in M:
            core2 = emr2.to_marked_rule()
            if core1 == core2:
                Follow |= emr2.mFollow
        new_emr = ExtendedMarkedRule(core1.mVariable, core1.mAlpha, core1.mBeta, frozenset(Follow))
        Result.add(new_emr)
    return frozenset(Result)


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a string name, which is either a variable, a token, or a literal, the function is_var checks whether name is a variable.  The function can distinguish variable names from tokens and literals because variable names consist only of lower case letters, while tokens are all uppercase and literals start with the character "'".
Can you write Python code for it?


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given a tuple of variables and tokens alpha, the function first_list(alpha) computes the function $\texttt{FirstList}(\alpha)$ that has been defined above.  If alpha is nullable, then the result will contain the empty string $\varepsilon = \texttt{''}$.
Can you write Python code for it?

def first_list(self, alpha):
    if len(alpha) == 0:
        return { '' }
    elif is_var(alpha[0]):
        v, *r = alpha
        return eps_union(self.mFirst[v], self.first_list(r))
    else:
        t = alpha[0]
        return { t }

Grammar.first_list = first_list
del first_list


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given an augmented grammar $G = \langle V,T,R\cup{\widehat{s} \rightarrow s\,\$}, \widehat{s}\rangle$
and a variable $a$, the set of tokens that might follow $a$ is defined as:
$$\texttt{Follow}(a) :=
 \bigl{ t \in \widehat{T} \,\bigm|\, \exists \beta,\gamma \in (V \cup \widehat{T})^:
                           \widehat{s} \Rightarrow^ \beta \,a\, t\, \gamma
  \bigr}.
$$
The function compute_follow computes the sets $\texttt{Follow}(a)$ for all variables $a$ via a fixed-point iteration.
Can you write Python code for it?

def compute_follow(self):
    self.mFollow[self.mStart] = { '$' }
    change = True
    while change:
        change = False
        for rule in self.mRules:
            a, body = rule.mVariable, rule.mBody
            for i in range(len(body)):
                if is_var(body[i]):
                    yi        = body[i]
                    Tail      = self.first_list(body[i+1:])
                    firstTail = eps_union(Tail, self.mFollow[a])
                    if not (firstTail <= self.mFollow[yi]):
                        change = True
                        self.mFollow[yi] |= firstTail
    print('Follow sets (note that "$" denotes the end of file):');
    for v in self.mVariables:
        print(f'Follow({v}) = {self.mFollow[v]}')

Grammar.compute_follow = compute_follow
del compute_follow


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given an extended marked rule self of the form
$$ c \rightarrow \alpha \bullet X\, \delta: L, $$
the function symbol_after_dot returns the symbol $X$. If there is no symbol after the $\bullet$, the method returns None.
Can you write Python code for it?

def symbol_after_dot(self):
    if len(self.mBeta) > 0:
        return self.mBeta[0]
    return None

ExtendedMarkedRule.symbol_after_dot = symbol_after_dot
del symbol_after_dot


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given an extended marked rule self, the function is_complete checks, whether the extended marked rule self has the form
$$ c \rightarrow \alpha\; \bullet: L,$$
i.e. it checks, whether the $\bullet$ is at the end of the grammar rule.
Can you write Python code for it?

def is_complete(self):
    return len(self.mBeta) == 0

ExtendedMarkedRule.is_complete = is_complete
del is_complete


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Given an extended marked rule, this function returns the variable following the dot.  If there is no variable following the dot, the function returns None.
Can you write Python code for it?

def next_var(self):
    if len(self.mBeta) > 0:
        var = self.mBeta[0]
        if is_var(var):
            return var
    return None

ExtendedMarkedRule.next_var = next_var
del next_var


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
If $\mathcal{M}$ is a set of extended marked rules, then the closure of $\mathcal{M}$ is the smallest set $\mathcal{K}$ such that
we have the following:
- $\mathcal{M} \subseteq \mathcal{K}$,
- If $a \rightarrow \beta \bullet c\, \delta: L$ is a extended marked rule from
  $\mathcal{K}$, $c$ is a variable, and $t\in L$ and if, furthermore,
  $c \rightarrow \gamma$ is a grammar rule,
  then the marked rule $c \rightarrow \bullet \gamma: \texttt{First}(\delta\,t)$
  is an element of $\mathcal{K}$:
  $$(a \rightarrow \beta \bullet c\, \delta) \in \mathcal{K}
         \;\wedge\;
         (c \rightarrow \gamma) \in R
         \;\Rightarrow\; (c \rightarrow \bullet \gamma: \texttt{First}(\delta\,t)) \in \mathcal{K}
  $$
We define $\texttt{closure}(\mathcal{M}) := \mathcal{K}$.  The function cmp_closure computes this closure for a given set of extended marked rules via a fixed-point iteration.
Can you write Python code for it?

def cmp_closure(self, Marked_Rules):
    All_Rules = Marked_Rules
    New_Rules = Marked_Rules
    while True:
        More_Rules = set()
        for rule in New_Rules:
            c = rule.next_var()
            if c == None:
                continue
            delta = rule.mBeta[1:]
            L = rule.mFollow
            for rule in self.mRules:
                head, alpha = rule.mVariable, rule.mBody
                if c == head:
                    newL = frozenset({ x for t in L for x in self.first_list(delta + (t,)) })
                    More_Rules |= { ExtendedMarkedRule(head, (), alpha, newL) }
        if More_Rules <= All_Rules:
            return frozenset(All_Rules)
        New_Rules  = More_Rules - All_Rules
        All_Rules |= New_Rules

Grammar.cmp_closure = cmp_closure
del cmp_closure


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
Implementing an LR-Table-Generator
A Grammar for Grammars
As the goal is to generate an LR-table-generator we first need to implement a parser for context free grammars.
The file arith.g contains an example grammar that describes arithmetic expressions.
Can you write Python code for it?

!type Examples\c-grammar.g

!cat Examples/arith.g


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
LR-Table-Generation
The class Grammar represents a context free grammar.  It stores a list of the GrammarRules of the given grammar.
Each grammar rule of the form
$$ a \rightarrow \beta $$
The start symbol is assumed to be the variable on the left hand side of the first rule. The grammar is augmented with the rule
$$ \widehat{s} \rightarrow s. $$
Here $s$ is the start variable of the given grammar and $\widehat{s}$ is a new variable that is the start variable of the augmented grammar. The symbol $ denotes the end of input.  The non-obvious member variables of the class Grammar have the following interpretation
- mStates is the set of all states of the LR-parser.  These states are sets of extended marked rules.
- mStateNamesis a dictionary assigning names of the form s0, s1, $\cdots$, sn to the states stored in
  mStates.  The functions action and goto will be defined for state names, not for states, because
  otherwise the table representing these functions would become both huge and unreadable.
- mConflicts is a Boolean variable that will be set to true if the table generation discovers
  shift/reduce conflicts or reduce/reduce conflicts.
Can you write Python code for it?

class Grammar():
    def __init__(self, Rules):
        self.mRules      = Rules
        self.mStart      = Rules[0].mVariable
        self.mVariables  = collect_variables(Rules)
        self.mTokens     = collect_tokens(Rules)
        self.mStates     = set()
        self.mStateNames = {}
        self.mConflicts  = False
        self.mVariables.add('ŝ')
        self.mTokens.add('$')
        self.mRules.append(GrammarRule('ŝ', (self.mStart, ))) # augmenting
        self.compute_tables()


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The Class GrammarRule
The class GrammarRule is used to store a single grammar rule.  As we have to use objects of type GrammarRule as keys in a dictionary later, we have to provide the methods __eq__, __ne__, and __hash__.
Can you write Python code for it?

class GrammarRule:
    def __init__(self, variable, body):
        self.mVariable = variable
        self.mBody     = body

    def __eq__(self, other):
        return isinstance(other, GrammarRule)    and \
               self.mVariable == other.mVariable and \
               self.mBody     == other.mBody

    def __ne__(self, other):
        return not self.__eq__(other)

    def __hash__(self):
        return hash(self.__repr__())

    def __repr__(self):
        return f'{self.mVariable} → {" ".join(self.mBody)}'


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The annotated grammar is stored in the file Grammar.g4.
The parser will return a list of grammar rules, where each rule of the form
$$ a \rightarrow \beta $$
is stored as the tuple (a,) + 𝛽.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The arguments S and T of eps_union are sets that contain tokens and, additionally, they might contain the empty string.
Can you write Python code for it?

def eps_union(S, T):
    if '' in S:
        if '' in T:
            return S | T
        return (S - { '' }) | T
    return S


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The class MarkedRule is similar to the class ExtendedMarkedRule but does not have the follow set.
Can you write Python code for it?

class MarkedRule():
    def __init__(self, variable, alpha, beta):
        self.mVariable = variable
        self.mAlpha    = alpha
        self.mBeta     = beta

    def __eq__(self, other):
        return isinstance(other, MarkedRule)     and \
               self.mVariable == other.mVariable and \
               self.mAlpha    == other.mAlpha    and \
               self.mBeta     == other.mBeta

    def __ne__(self, other):
        return not self.__eq__(other)

    def __hash__(self):
        return hash(self.mVariable) + \
               hash(self.mAlpha)    + \
               hash(self.mBeta)

    def __repr__(self):
        alphaStr  = ' '.join(self.mAlpha)
        betaStr   = ' '.join(self.mBeta)
        return f'{self.mVariable} → {alphaStr} • {betaStr}'


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The command below cleans the directory.  If you are running windows, you have to replace rm with del.
Can you write Python code for it?

!del GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rmdir /S /Q __pycache__

!dir /B

!rm GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rm -r __pycache__

!ls


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The following function computes the action table and is defined as follows:
- If $\mathcal{M}$ contains an extended marked rule of the form $a \rightarrow \beta \bullet t\, \delta:L$
  then we have
  $$\texttt{action}(\mathcal{M},t) := \langle \texttt{shift}, \texttt{goto}(\mathcal{M},t) \rangle.$$
- If $\mathcal{M}$ contains an extended marked rule of the form $a \rightarrow \beta\, \bullet:L$ and we have
  $t \in L$, then we define
  $$\texttt{action}(\mathcal{M},t) := \langle \texttt{reduce}, a \rightarrow \beta \rangle$$
- If $\mathcal{M}$ contains the extended marked rule $\widehat{s} \rightarrow s \bullet:{\$}$, then we define
  $$\texttt{action}(\mathcal{M},\$) := \texttt{accept}. $$
- Otherwise, we have
  $$\texttt{action}(\mathcal{M},t) := \texttt{error}. $$
Can you write Python code for it?

def compute_action_table(self):
    self.mActionTable = {}
    print('\nAction Table:')
    for state in self.mStates:
        stateName = self.mStateNames[state]
        actionTable = {}
        # compute shift actions
        for token in self.mTokens:
            newState  = self.goto(state, token)
            if newState != set():
                newName = self.mStateNames[newState]
                actionTable[token] = ('shift', newName)
                self.mActionTable[stateName, token] = ('shift', newName)
                print(f'action("{stateName}", {token}) = ("shift", {newName})')
        # compute reduce actions
        for mr in state:
            if mr.is_complete():
                for token in mr.mFollow:
                    action1 = actionTable.get(token)
                    action2 = ('reduce', mr.to_rule())
                    if action1 == None:
                        actionTable[token] = action2
                        r = self.mRuleNames[mr.to_rule()]
                        self.mActionTable[stateName, token] = ('reduce', r)
                        print(f'action("{stateName}", {token}) = {action2}')
                    elif action1 != action2:
                        self.mConflicts = True
                        print('')
                        print(f'conflict in state {stateName}:')
                        print(f'{stateName} = {state}')
                        print(f'action("{stateName}", {token}) = {action1}')
                        print(f'action("{stateName}", {token}) = {action2}')
                        print('')
        for mr in state:
            if mr == ExtendedMarkedRule('ŝ', (self.mStart,), (), frozenset({'$'})):
                actionTable['$'] = 'accept'
                self.mActionTable[stateName, '$'] = 'accept'
                print(f'action("{stateName}", $) = accept')

Grammar.compute_action_table = compute_action_table
del compute_action_table


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function all_states computes the set of all states of an LR-parser.  The function starts with the state
$$ \texttt{closure}\bigl({ \widehat{s} \rightarrow \bullet s : {\$} }\bigr) $$
and then tries to compute new states by using the function goto.  This computation proceeds via a
fixed-point iteration.  Once all states have been computed, the function assigns names to these states.
This association is stored in the dictionary mStateNames.
Can you write Python code for it?

def all_states(self):
    start_state  = self.cmp_closure({ ExtendedMarkedRule('ŝ', (), (self.mStart,), frozenset({'$'})) })
    start_state  = combine_rules(start_state)
    self.mStates = { start_state }
    New_States   = self.mStates
    while True:
        More_States = set()
        for Rule_Set in New_States:
            for mr in Rule_Set:
                if not mr.is_complete():
                    x = mr.symbol_after_dot()
                    next_state = self.goto(Rule_Set, x)
                    if next_state not in self.mStates and next_state not in More_States:
                        More_States.add(next_state)
                        print('.', end='')
        if len(More_States) == 0:
            break
        New_States = More_States;
        self.mStates |= New_States
        print('\n', len(self.mStates), sep='')
    print("All LR-states:")
    counter = 1
    self.mStateNames[start_state] = 's0'
    print(f's0 = {set(start_state)}')
    for state in self.mStates - { start_state }:
        self.mStateNames[state] = f's{counter}'
        print(f's{counter} = {set(state)}')
        counter += 1

Grammar.all_states = all_states
del all_states


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function compute_first(self) computes the sets $\texttt{First}(c)$ for all variables $c$ and stores them in the dictionary mFirst.  Abstractly, given a variable $c$ the function $\texttt{First}(c)$ is the set of all tokens that can start a string that is derived from $c$:
$$\texttt{First}(\texttt{c}) :=
  \Bigl{ t \in T \Bigm| \exists \gamma \in (V \cup T)^: \texttt{c} \Rightarrow^ t\,\gamma \Bigr}.
$$
The definition of the function $\texttt{First}()$ is extended to strings from $(V \cup T)^$ as follows:
- $\texttt{FirstList}(\varepsilon) = {}$.
- $\texttt{FirstList}(t \beta) = { t }$  if $t \in T$.
- $\texttt{FirstList}(\texttt{a} \beta) = \left{
       \begin{array}[c]{ll}
         \texttt{First}(\texttt{a}) \cup \texttt{FirstList}(\beta) & \mbox{if $\texttt{a} \Rightarrow^ \varepsilon$;} \
         \texttt{First}(\texttt{a})                                & \mbox{otherwise.}
       \end{array}
       \right.
      $
If $\texttt{a}$ is a variable of $G$ and the rules defining $\texttt{a}$ are given as
$$\texttt{a} \rightarrow \alpha_1 \mid \cdots \mid \alpha_n, $$
then we have
$$\texttt{First}(\texttt{a}) = \bigcup\limits_{i=1}^n \texttt{FirstList}(\alpha_i). $$
The dictionary mFirst that stores this function is computed via a fixed point iteration.
Can you write Python code for it?

def compute_first(self):
    change = True
    while change:
        change = False
        for rule in self.mRules:
            a, body = rule.mVariable, rule.mBody
            first_body = self.first_list(body)
            if not (first_body <= self.mFirst[a]):
                change = True
                self.mFirst[a] |= first_body
    print('First sets:')
    for v in self.mVariables:
        print(f'First({v}) = {self.mFirst[v]}')

Grammar.compute_first = compute_first
del compute_first


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function compute_goto_table computes the goto table.
Can you write Python code for it?

def compute_goto_table(self):
    self.mGotoTable = {}
    print('\nGoto Table:')
    for state in self.mStates:
        for var in self.mVariables:
            newState = self.goto(state, var)
            if newState != set():
                stateName = self.mStateNames[state]
                newName   = self.mStateNames[newState]
                self.mGotoTable[stateName, var] = newName
                print(f'goto({stateName}, {var}) = {newName}')

Grammar.compute_goto_table = compute_goto_table
del compute_goto_table

%%time
g = Grammar(grammar)

def strip_quotes(t):
    if t[0] == "'" and t[-1] == "'":
        return t[1:-1]
    return t

def dump_parse_table(self, file):
    with open(file, 'w') as handle:
        handle.write('# Grammar rules:\n')
        for rule in self.mRules:
            rule_name = self.mRuleNames[rule]
            handle.write(f'{rule_name} =("{rule.mVariable}", {rule.mBody})\n')
        handle.write('\n# Action table:\n')
        handle.write('actionTable = {}\n')
        for s, t in self.mActionTable:
            action = self.mActionTable[s, t]
            t = strip_quotes(t)
            if action[0] == 'reduce':
                rule_name = action[1]
                handle.write(f"actionTable['{s}', '{t}'] = ('reduce', {rule_name})\n")
            elif action == 'accept':
                handle.write(f"actionTable['{s}', '{t}'] = 'accept'\n")
            else:
                handle.write(f"actionTable['{s}', '{t}'] = {action}\n")
        handle.write('\n# Goto table:\n')
        handle.write('gotoTable = {}\n')
        for s, v in self.mGotoTable:
            state = self.mGotoTable[s, v]
            handle.write(f"gotoTable['{s}', '{v}'] = '{state}'\n")

Grammar.dump_parse_table = dump_parse_table
del dump_parse_table

g.dump_parse_table('parse-table.py')

!cat parse-table.py


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function compute_rule_names assigns a unique name to each rule of the grammar.  These names are used later
to represent reduce actions in the action table.
Can you write Python code for it?

def compute_rule_names(self):
    self.mRuleNames = {}
    counter = 0
    for rule in self.mRules:
        self.mRuleNames[rule] = 'r' + str(counter)
        counter += 1

Grammar.compute_rule_names = compute_rule_names
del compute_rule_names


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function move_dot(self) transforms an extended marked rule  of the form
$$ c \rightarrow \alpha \bullet X\, \beta: L $$
into an extended marked rule of the form
$$ c \rightarrow \alpha\, X \bullet \beta: L, $$
i.e. the $\bullet$ is moved over the next symbol.  Invocation of this method assumes that there is a symbol
following the $\bullet$.
Can you write Python code for it?

def move_dot(self):
    return ExtendedMarkedRule(self.mVariable,
                              self.mAlpha + (self.mBeta[0],),
                              self.mBeta[1:],
                              self.mFollow)

ExtendedMarkedRule.move_dot = move_dot
del move_dot


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function parse_grammar takes a string filename as its argument and returns the grammar that is stored in the specified file.  The grammar is represented as list of rules.  Each rule is represented as a tuple.  The example below will clarify this structure.
Can you write Python code for it?

def parse_grammar(filename):
    input_stream  = antlr4.FileStream(filename, encoding="utf-8")
    lexer         = GrammarLexer(input_stream)
    token_stream  = antlr4.CommonTokenStream(lexer)
    parser        = GrammarParser(token_stream)
    grammar       = parser.start()
    return [GrammarRule(head, tuple(body)) for head, *body in grammar.g]

grammar = parse_grammar('Examples/c-fragment.g')
grammar


def parse_grammar(filename):
    input_stream  = antlr4.FileStream(filename, encoding="utf-8")
    lexer         = GrammarLexer(input_stream)
    token_stream  = antlr4.CommonTokenStream(lexer)
    parser        = GrammarParser(token_stream)
    grammar       = parser.start()
    return [GrammarRule(head, tuple(body)) for head, *body in grammar.g]

grammar = parse_grammar('Examples/c-grammar.g')
grammar


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function to_rule(self) turns the extended marked rule self into  a GrammarRule, i.e. the extended marked rule
$$ c \rightarrow \alpha \bullet \beta: L $$
is turned into the grammar rule
$$ c \rightarrow \alpha\, \beta. $$
Can you write Python code for it?

def to_rule(self):
    return GrammarRule(self.mVariable, self.mAlpha + self.mBeta)

ExtendedMarkedRule.to_rule = to_rule
del to_rule


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
The function to_rule(self) turns the extended marked rule self into a MarkedRule, i.e. the extended marked rule
$$ c \rightarrow \alpha \bullet \beta: L $$
is turned into the marked rule
$$ c \rightarrow \alpha\bullet \beta. $$
Can you write Python code for it?

def to_marked_rule(self):
    return MarkedRule(self.mVariable, self.mAlpha, self.mBeta)

ExtendedMarkedRule.to_marked_rule = to_marked_rule
del to_marked_rule


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
We start by generating both scanner and parser.
Can you write Python code for it?

!antlr4 -Dlanguage=Python3 Grammar.g4

from GrammarLexer  import GrammarLexer
from GrammarParser import GrammarParser
import antlr4


I am working on the file "ANTLR4-Python/LR-Parser-Generator/LR-Table-Generator.ipynb".
The first task is:
We use Antlr to develop a parser for context free grammars.  The pure grammar used to parse context free grammars is stored in the file Pure.g4.  It is similar to the grammar that we have already used to implement Earley's algorithm, but allows additionally the use of the operator |, so that all grammar rules that define a variable can be combined in one rule.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/LR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
A Shift-Reduce Parser for Arithmetic Expressions
In this notebook we implement a simple recursive descend parser for arithmetic expressions.
This parser will implement the following grammar:
$$
  \begin{eqnarray}
  \mathrm{expr}        & \rightarrow & \mathrm{expr}\;\;\texttt{'+'}\;\;\mathrm{product}   \
                       & \mid        & \mathrm{expr}\;\;\texttt{'-'}\;\;\mathrm{product}   \
                       & \mid        & \mathrm{product}                                    \[0.2cm]
  \mathrm{product}     & \rightarrow & \mathrm{product}\;\;\texttt{''}\;\;\mathrm{factor} \
                       & \mid        & \mathrm{product}\;\;\texttt{'/'}\;\;\mathrm{factor} \
                       & \mid        & \mathrm{factor}                                     \[0.2cm]
  \mathrm{factor}      & \rightarrow & \texttt{'('} \;\;\mathrm{expr} \;\;\texttt{')'}     \
                       & \mid        & \texttt{NUMBER}
  \end{eqnarray*}
$$
Implementing a Scanner
Can you write Python code for it?


I am working on the file "ANTLR4-Python/LR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
Testing
Can you write Python code for it?

def test(s):
    parser = ShiftReduceParser(actionTable, gotoTable)
    TL     = tokenize(s)
    print(f'tokenlist: {TL}\n')
    if parser.parse(TL):
        print('Parse successful!')
    else:
        print('Parse failed!')

test('(1 + 2) * 3')

test('1 * 2 + 3 * (4 - 5) / 2')

test('11+22*(33-44)/(5-10*5/(4-3))')

test('1+2*3-')


I am working on the file "ANTLR4-Python/LR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
The function tokenize transforms the string s into a list of tokens. See below for an example.
Can you write Python code for it?

def tokenize(s):
    '''Transform the string s into a list of tokens.  The string s
       is supposed to represent an arithmetic expression.
    '''
    lexSpec = r'''([ \t]+)        |  # blanks and tabs
                  ([1-9][0-9]*|0) |  # number
                  ([()])          |  # parentheses
                  ([-+*/])        |  # arithmetical operators
                  (.)                # unrecognized character
               '''
    tokenList = re.findall(lexSpec, s, re.VERBOSE)
    result    = []
    for ws, number, parenthesis, operator, error in tokenList:
        if ws:        # skip blanks and tabs
            continue
        elif number:
            result += [ 'NUMBER' ]
        elif parenthesis:
            result += [ parenthesis ]
        elif operator:
            result += [ operator ]
        else:
            result += [ f'ERROR({error})']
    return result

tokenize('1 + 2 * (3 - 4)')

class ShiftReduceParser():
    def __init__(self, actionTable, gotoTable):
        self.mActionTable = actionTable
        self.mGotoTable   = gotoTable

def parse(self, TL):
    index   = 0      # points to next token
    Symbols = []     # stack of symbols
    States  = ['s0'] # stack of states, s0 is start state
    TL     += ['$']
    while True:
        q = States[-1]
        t = TL[index]
        print('Symbols:', ' '.join(Symbols + ['|'] + TL[index:]).strip())
        p = self.mActionTable.get((q, t), 'error')
        if p == 'error':
            return False
        elif p == 'accept':
            return True
        elif p[0] == 'shift':
            s = p[1]
            Symbols += [t]
            States  += [s]
            index   += 1
        elif p[0] == 'reduce':
            head, body = p[1]
            n       = len(body)
            Symbols = Symbols[:-n]
            States  = States [:-n]
            Symbols = Symbols + [head]
            state   = States[-1]
            States += [ self.mGotoTable[state, head] ]

ShiftReduceParser.parse = parse
del parse

%run parse-table.py


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
As the operator ** is not supported by our grammar, we get a syntax error at the
last occurrence of the character * in the given string.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
As there is no syntax error, the string '1 + 2 * 3 - 4' adheres to the specification given by our grammar.
Lets try a string that is not generated by our grammar.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
In order to use the Antlr runtime, we have to import it.
Note that this runtime is not contained in antlr-4.8-complete.jar and needs to be installed separately using the following command:
pip install antlr4-python3-runtime
Can you write Python code for it?


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
Let us clean up the working directory.
Can you write Python code for it?

!ls

!dir /B

!rm *.py *.tokens *.interp *.java *.class
!rm -r __pycache__/

!del *.py *.tokens *.interp *.java *.class /Q
!rmdir __pycache__ /S /Q

!ls -l

!dir /B


!ls
# !dir /B   # For Windows

!rm *.py *.tokens *.interp *.java *.class
!rm -r __pycache__/

# For Windows
#!del *.py *.tokens *.interp *.java *.class /Q
#!rmdir __pycache__ /S /Q

!ls -l
!dir /B   # For Windows


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
Next, we can start the so called TestRig to generate and display the parse tree for a given string.
Can you write Python code for it?

!echo "1+2*3-4*5*(2-3)" | java -cp .:/usr/local/lib/antlr-4.9.2-complete.jar org.antlr.v4.gui.TestRig Expr expr -gui

!echo 1+2*3-4 | java -cp .;C:/Users/Karl/anaconda3/envs/fl/Library/lib/antlr4-4.9.2_1-complete.jar org.antlr.v4.gui.TestRig Expr expr -gui


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
Note that this grammar does not contain any embedded actions.
Hence we cannot compute anything with it.  We will only be able to
check whether a given string is generated by this grammar.  Provided you have stored the file
antlr-4.9-complete.jar in the directory
/usr/local/lib/ we can generate both the scanner and the parser using the following command:
Can you write Python code for it?

!antlr4 -Dlanguage=Python3 Expr.g4

!ls -l

!dir /B


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
Now we can parse a string.  The function parser_string takes the string s as its argument and checks,
whether this string can be parsed as an arithmetic expression.  This is done in five steps:
- The string is converted into an antlr4.InputStream.
- The input stream is converted into a lexer.
- The lexer is converted into an antlr4.CommonTokenStream.
- The token stream is converted into a parser.
- The parser tries to parse with start symbol.
Can you write Python code for it?

def parse_string(string):
    inputStream = antlr4.InputStream(string)
    lexer       = ExprLexer(inputStream)
    tokenStream = antlr4.CommonTokenStream(lexer)
    parser      = ExprParser(tokenStream)
    parser.expr()

parse_string('1 + 2 * 3 - 4')


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
On Unix-like systems it is also convenient to define an alias of the following form in the file .bashrc
that is used to initialize the bash shell:
alias antlr='java -jar /usr/local/lib/antlr-4.8-complete.jar -Dlanguage=Python3'
We can use the system command ls to see which files have been generated by Antlr.
If you are using a windows system you have to use the command dir instead.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
Testing an Antlr Grammar via grun
In order for the examples using Antlr to work,
we first have to install Antlr.  This can be done by executing
the following commands in an Anaconda environment:
conda install -y -c conda-forge antlr4-python3-runtime
conda install -y -c conda-forge antlr
Alternatively, you can download https://www.antlr.org/download/antlr-4.9-complete.jar.  I will assume that this .jarfile is
stored in the directory /usr/local/lib/.  Furthermore, I assume that both a java runtime
and a java compiler are available.  Then you also need to install the python language bindings
using the following command:
pip install antlr4-python3-runtime
Our grammar is stored in the file Expr.g4.  In order to inspect it, we use the command line tool cat.  This will work with MacOs and Linux.  On Windows,
either use the power shell, which understands cat,  or use the command type instead.  The option  -n of cat provides numbered output.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
The files ExprLexer.py and ExprParser.py contain the generated scanner and parser, respectively.
If we want to test the parser in this notebook, we have to import these files.
Can you write Python code for it?

from ExprLexer  import ExprLexer
from ExprParser import ExprParser

import antlr4


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
This command has generated some files for us that contain a both a lexer and a parser.
However, this time these are .java-files.
Can you write Python code for it?


!ls -l *.java
#!dir /B *.java   # For Windows


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
This time we get a lexical error as the character < is not a legal token.
We can also generate a parse tree with our grammar.  However, for this to work Antlr
first has to generate a java parser.  Hence we have to call antlr4 again, but this time with Java as the target language.
Can you write Python code for it?

!java -jar /usr/local/lib/antlr-4.9.2-complete.jar -Dlanguage=Java Expr.g4

!java -jar C:/Users/Karl/anaconda3/envs/fl/Library/lib/antlr4-4.9.2_1-complete.jar -Dlanguage=Java Expr.g4


I am working on the file "ANTLR4-Python/PureExprParser/PureParser.ipynb".
The first task is:
We have to compile the generated .java files.  Below, you might have to change the path to
the file antlr-4.8-complete.jar to make this work.
Can you write Python code for it?

!javac -cp .:/usr/local/lib/antlr-4.8-complete.jar *.java
#!javac -cp .;C:/Users/xxx/antlr-4.8-complete.jar *.java   # For Windows


!javac -cp .:/usr/local/lib/antlr-4.9.2-complete.jar *.java

!javac -cp .;C:/Users/Karl/anaconda3/envs/fl/Library/lib/antlr4-4.9.2_1-complete.jar *.java


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Fun Fact: The invocation of "'return'".islower() returns True.  This is the reason that we have to test that
name does not start with a "'" character because otherwise keywords like 'return' or 'while' appearing in a grammar would be mistaken for variables.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a Grammar, the function compute_tables computes
- the sets First(v) and Follow(v) for every variable v,
- the set of all states of the SLR-Parser,
- the action table, and
- the goto table.
Given a grammar g,
- the set g.mFirst is a dictionary such that g.mFirst[a] = First[a] and
- the set g.mFollow is a dictionary such that g.mFollow[a] = Follow[a] for all variables a.
Can you write Python code for it?

def compute_tables(self):
    self.mFirst  = initialize_dictionary(self.mVariables)
    self.mFollow = initialize_dictionary(self.mVariables)
    self.compute_first()
    self.compute_follow()
    self.compute_rule_names()
    self.all_states()
    self.compute_action_table()
    self.compute_goto_table()

Grammar.compute_tables = compute_tables
del compute_tables


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a list Rules of GrammarRules, the function collect_variables(Rules) returns the set of all variables occuring in Rules.
Can you write Python code for it?

def collect_variables(Rules):
    Variables = set()
    for rule in Rules:
        Variables.add(rule.mVariable)
        for item in rule.mBody:
            if is_var(item):
                Variables.add(item)
    return Variables


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a marked rule of the form
$$ c \rightarrow \alpha \bullet b \delta, $$
this function returns the variable $b$ following the dot.  If there is no variable following the dot, the function returns None.
Can you write Python code for it?

def next_var(self):
    if len(self.mBeta) > 0:
        var = self.mBeta[0]
        if is_var(var):
            return var
    return None

MarkedRule.next_var = next_var
del next_var


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a marked rule self of the form
$$ c \rightarrow \alpha \bullet X\, \delta, $$
the function symbol_after_dot returns the symbol $X$. If there is no symbol after the $\bullet$, the method returns None.
Can you write Python code for it?

def symbol_after_dot(self):
    if len(self.mBeta) > 0:
        return self.mBeta[0]
    return None

MarkedRule.symbol_after_dot = symbol_after_dot
del symbol_after_dot


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a marked rule self, the function is_complete checks, whether the marked rule self has the form
$$ c \rightarrow \alpha\; \bullet,$$
i.e. it checks, whether the $\bullet$ is at the end of the grammar rule.
Can you write Python code for it?

def is_complete(self):
    return len(self.mBeta) == 0

MarkedRule.is_complete = is_complete
del is_complete


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a marked rule, this function returns the variable following the dot.  If there is no variable following the dot, the function returns None.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a set Rules of GrammarRules, the function collect_tokens(Rules) returns the set of all tokens and literals occuring in Rules.
Can you write Python code for it?

def collect_tokens(Rules):
    Tokens = set()
    for rule in Rules:
        for item in rule.mBody:
            if not is_var(item):
                Tokens.add(item)
    return Tokens


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a set of Variables, the function initialize_dictionary returns a dictionary that assigns the empty set to all variables.
Can you write Python code for it?

def initialize_dictionary(Variables):
    return { a: set() for a in Variables }


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a set of marked rules $\mathcal{M}$ and a grammar symbol $X$, the function $\texttt{goto}(\mathcal{M}, X)$
is defined as follows:
$$\texttt{goto}(\mathcal{M}, X) := \texttt{closure}\Bigl( \bigl{
   a \rightarrow \beta\, X \bullet \delta \bigm| (a \rightarrow \beta \bullet X\, \delta) \in \mathcal{M}
   \bigr} \Bigr).
$$
Can you write Python code for it?

def goto(self, Marked_Rules, x):
    Result = set()
    for mr in Marked_Rules:
        if mr.symbol_after_dot() == x:
            Result.add(mr.move_dot())
    return self.cmp_closure(Result)

Grammar.goto = goto
del goto


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a string name, which is either a variable, a token, or a literal, the function is_var checks whether name is a variable.  The function can distinguish variable names from tokens and literals because variable names consist only of lower case letters, while tokens are all uppercase and literals start with the character "'".
Can you write Python code for it?

def is_var(name):
    return name[0] != "'" and name.islower()


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given a tuple of variables and tokens alpha, the function first_list(alpha) computes the function $\texttt{FirstList}(\alpha)$ that has been defined above.  If alpha is nullable, then the result will contain the empty string $\varepsilon = \texttt{''}$.
Can you write Python code for it?

def first_list(self, alpha):
    if len(alpha) == 0:
        return { '' }
    elif is_var(alpha[0]):
        v, *r = alpha
        return eps_union(self.mFirst[v], self.first_list(r))
    else:
        t = alpha[0]
        return { t }

Grammar.first_list = first_list
del first_list


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
Given an augmented grammar $G = \langle V,T,R\cup{\widehat{s} \rightarrow s\,\$}, \widehat{s}\rangle$
and a variable $a$, the set of tokens that might follow $a$ is defined as:
$$\texttt{Follow}(a) :=
 \bigl{ t \in \widehat{T} \,\bigm|\, \exists \beta,\gamma \in (V \cup \widehat{T})^:
                           \widehat{s} \Rightarrow^ \beta \,a\, t\, \gamma
  \bigr}.
$$
The function compute_follow computes the sets $\texttt{Follow}(a)$ for all variables $a$ via a fixed-point iteration.
Can you write Python code for it?

def compute_follow(self):
    self.mFollow[self.mStart] = { '$' }
    change = True
    while change:
        change = False
        for rule in self.mRules:
            a, body = rule.mVariable, rule.mBody
            for i in range(len(body)):
                if is_var(body[i]):
                    yi        = body[i]
                    Tail      = self.first_list(body[i+1:])
                    firstTail = eps_union(Tail, self.mFollow[a])
                    if not (firstTail <= self.mFollow[yi]):
                        change = True
                        self.mFollow[yi] |= firstTail
    print('Follow sets (note that "$" denotes the end of file):');
    for v in self.mVariables:
        print(f'Follow({v}) = {self.mFollow[v]}')

Grammar.compute_follow = compute_follow
del compute_follow


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
If $\mathcal{M}$ is a set of marked rules, then the closure of $\mathcal{M}$ is the smallest set $\mathcal{K}$ such that
we have the following:
- $\mathcal{M} \subseteq \mathcal{K}$,
- If $a \rightarrow \beta \bullet c\, \delta$ is a marked rule from
  $\mathcal{K}$, and $c$ is a variable and if, furthermore,
  $c \rightarrow \gamma$ is a grammar rule,
  then the marked rule $c \rightarrow \bullet \gamma$
  is an element of $\mathcal{K}$:
  $$(a \rightarrow \beta \bullet c\, \delta) \in \mathcal{K}
         \;\wedge\;
         (c \rightarrow \gamma) \in R
         \;\Rightarrow\; (c \rightarrow \bullet \gamma) \in \mathcal{K}
  $$
We define $\texttt{closure}(\mathcal{M}) := \mathcal{K}$.  The function cmp_closure computes this closure for a given set of marked rules via a fixed-point iteration.
Can you write Python code for it?

def cmp_closure(self, Marked_Rules):
    All_Rules = Marked_Rules
    New_Rules = Marked_Rules
    while True:
        More_Rules = set()
        for rule in New_Rules:
            c = rule.next_var()
            if c == None:
                continue
            for rule in self.mRules:
                head, alpha = rule.mVariable, rule.mBody
                if c == head:
                    More_Rules |= { MarkedRule(head, (), alpha) }
        if More_Rules <= All_Rules:
            return frozenset(All_Rules)
        New_Rules  = More_Rules - All_Rules
        All_Rules |= New_Rules

Grammar.cmp_closure = cmp_closure
del cmp_closure


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
SLR-Table-Generation
The class Grammar represents a context free grammar.  It stores a list of the GrammarRules of the given grammar.
Each grammar rule is of the form
$$ a \rightarrow \beta $$
where $\beta$ is a tuple of variables, tokens, and literals.
The start symbol is assumed to be the variable on the left hand side of the first rule. The grammar is augmented with the rule
$$ \widehat{s} \rightarrow s\, \$. $$
Here $s$ is the start variable of the given grammar and $\widehat{s}$ is a new variable that is the start variable of the augmented grammar. The symbol $ denotes the end of input.  The non-obvious member variables of the class Grammar have the following interpretation
- mStates is the set of all states of the SLR-parser.  These states are sets of marked rules.
- mStateNamesis a dictionary assigning names of the form s0, s1, $\cdots$, sn to the states stored in
  mStates.  The functions action and goto will be defined for state names, not for states, because
  otherwise the table representing these functions would become both huge and unreadable.
- mConflicts is a Boolean variable that will be set to true if the table generation discovers
  shift/reduce conflicts or reduce/reduce conflicts.
Can you write Python code for it?

class Grammar():
    def __init__(self, Rules):
        self.mRules      = Rules
        self.mStart      = Rules[0].mVariable
        self.mVariables  = collect_variables(Rules)
        self.mTokens     = collect_tokens(Rules)
        self.mStates     = set()
        self.mStateNames = {}
        self.mConflicts  = False
        self.mVariables.add('ŝ')
        self.mTokens.add('$')
        self.mRules.append(GrammarRule('ŝ', (self.mStart, '$'))) # augmenting
        self.compute_tables()


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The Class GrammarRule
The class GrammarRule is used to store a single grammar rule.  As we have to use objects of type GrammarRule as keys in a dictionary later, we have to provide the methods __eq__, __ne__, and __hash__.
Can you write Python code for it?

class GrammarRule:
    def __init__(self, variable, body):
        self.mVariable = variable
        self.mBody     = body

    def __eq__(self, other):
        return isinstance(other, GrammarRule)    and \
               self.mVariable == other.mVariable and \
               self.mBody     == other.mBody

    def __ne__(self, other):
        return not self.__eq__(other)

    def __hash__(self):
        return hash(self.__repr__())

    def __repr__(self):
        return f'{self.mVariable} → {" ".join(self.mBody)}'


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The annotated grammar is stored in the file Grammar.g4.
The parser will return a list of grammar rules, where each rule of the form
$$ a \rightarrow \beta $$
is stored as the tuple (a,) + 𝛽.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The arguments S and T of eps_union are sets that contain tokens and, additionally, they might contain the empty string.
Can you write Python code for it?

def eps_union(S, T):
    if '' in S:
        if '' in T:
            return S | T
        return (S - { '' }) | T
    return S


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The arguments S and T of eps_union are sets that contain tokens and, additionally, they might contain the empty string.  The specification of eps_union is:
$$ \texttt{eps_union}(S, T) = \left{ \begin{array}{ll}
                                       S          & \mbox{if $\varepsilon \not\in S$} \
                                       S \cup T   & \mbox{if $\varepsilon \in S \wedge \varepsilon \in T$} \
                                       S \cup T - {\varepsilon } & \mbox{if $\varepsilon \in S \wedge \varepsilon \not\in T$}
                                      \end{array}
                              \right.
$$
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The following function computes the action table and is defined as follows:
- If $\mathcal{M}$ contains a marked rule of the form $a \rightarrow \beta \bullet t\, \delta$
  then we have
  $$\texttt{action}(\mathcal{M},t) := \langle \texttt{shift}, \texttt{goto}(\mathcal{M},t) \rangle.$$
- If $\mathcal{M}$ contains a marked rule of the form $a \rightarrow \beta\, \bullet$ and we have
  $t \in \texttt{Follow}(a)$, then we define
  $$\texttt{action}(\mathcal{M},t) := \langle \texttt{reduce}, a \rightarrow \beta \rangle$$
- If $\mathcal{M}$ contains the marked rule $\widehat{s} \rightarrow s \bullet \$ $, then we define
  $$\texttt{action}(\mathcal{M},\$) := \texttt{accept}. $$
- Otherwise, we have
  $$\texttt{action}(\mathcal{M},t) := \texttt{error}. $$
Can you write Python code for it?

def compute_action_table(self):
    self.mActionTable = {}
    print('\nAction Table:')
    for state in self.mStates:
        stateName = self.mStateNames[state]
        actionTable = {}
        # compute shift actions
        for token in self.mTokens:
            if token != '$':
                newState  = self.goto(state, token)
                if newState != set():
                    newName = self.mStateNames[newState]
                    actionTable[token] = ('shift', newName)
                    self.mActionTable[stateName, token] = ('shift', newName)
                    print(f'action("{stateName}", {token}) = ("shift", {newName})')
        # compute reduce actions
        for mr in state:
            if mr.is_complete():
                for token in self.mFollow[mr.mVariable]:
                    action1 = actionTable.get(token)
                    action2 = ('reduce', mr.to_rule())
                    if action1 == None:
                        actionTable[token] = action2
                        r = self.mRuleNames[mr.to_rule()]
                        self.mActionTable[stateName, token] = ('reduce', r)
                        print(f'action("{stateName}", {token}) = {action2}')
                    elif action1 != action2:
                        self.mConflicts = True
                        print('')
                        print(f'conflict in state {stateName}:')
                        print(f'{stateName} = {state}')
                        print(f'action("{stateName}", {token}) = {action1}')
                        print(f'action("{stateName}", {token}) = {action2}')
                        print('')
        for mr in state:
            if mr == MarkedRule('ŝ', (self.mStart,), ('$',)):
                actionTable['$'] = 'accept'
                self.mActionTable[stateName, '$'] = 'accept'
                print(f'action("{stateName}", $) = accept')

Grammar.compute_action_table = compute_action_table
del compute_action_table


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function all_states computes the set of all states of an SLR-parser.  The function starts with the state
$$ \texttt{closure}\bigl({ \widehat{s} \rightarrow \bullet s \, $}\bigr) $$
and then tries to compute new states by using the function goto.  This computation proceeds via a
fixed-point iteration.  Once all states have been computed, the function assigns names to these states.
This association is stored in the dictionary mStateNames.
Can you write Python code for it?

def all_states(self):
    start_state  = self.cmp_closure({ MarkedRule('ŝ', (), (self.mStart, '$')) })
    self.mStates = { start_state }
    New_States   = self.mStates
    while True:
        More_States = set()
        for Rule_Set in New_States:
            for mr in Rule_Set:
                if not mr.is_complete():
                    x = mr.symbol_after_dot()
                    if x != '$':
                        More_States |= { self.goto(Rule_Set, x) }
        if More_States <= self.mStates:
            break
        New_States = More_States - self.mStates;
        self.mStates |= New_States
    print("All SLR-states:")
    counter = 1
    self.mStateNames[start_state] = 's0'
    print(f's0 = {set(start_state)}')
    for state in self.mStates - { start_state }:
        self.mStateNames[state] = f's{counter}'
        print(f's{counter} = {set(state)}')
        counter += 1

Grammar.all_states = all_states
del all_states


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function compute_first(self) computes the sets $\texttt{First}(c)$ for all variables $c$ and stores them in the dictionary mFirst.  Abstractly, given a variable $c$ the function $\texttt{First}(c)$ is the set of all tokens that can start a string that is derived from $c$:
$$\texttt{First}(\texttt{c}) :=
  \Bigl{ t \in T \Bigm| \exists \gamma \in (V \cup T)^: \texttt{c} \Rightarrow^ t\,\gamma \Bigr}.
$$
The definition of the function $\texttt{First}()$ is extended to strings from $(V \cup T)^$ as follows:
- $\texttt{FirstList}(\varepsilon) = {}$.
- $\texttt{FirstList}(t \beta) = { t }$  if $t \in T$.
- $\texttt{FirstList}(\texttt{a} \beta) = \left{
       \begin{array}[c]{ll}
         \texttt{First}(\texttt{a}) \cup \texttt{FirstList}(\beta) & \mbox{if $\texttt{a} \Rightarrow^ \varepsilon$;} \
         \texttt{First}(\texttt{a})                                & \mbox{otherwise.}
       \end{array}
       \right.
      $
If $\texttt{a}$ is a variable of $G$ and the rules defining $\texttt{a}$ are given as
$$\texttt{a} \rightarrow \alpha_1 \mid \cdots \mid \alpha_n, $$
then we have
$$\texttt{First}(\texttt{a}) = \bigcup\limits_{i=1}^n \texttt{FirstList}(\alpha_i). $$
The dictionary mFirst that stores this function is computed via a fixed point iteration.
Can you write Python code for it?

def compute_first(self):
    change = True
    while change:
        change = False
        for rule in self.mRules:
            a, body = rule.mVariable, rule.mBody
            first_body = self.first_list(body)
            if not (first_body <= self.mFirst[a]):
                change = True
                self.mFirst[a] |= first_body
    print('First sets:')
    for v in self.mVariables:
        print(f'First({v}) = {self.mFirst[v]}')

Grammar.compute_first = compute_first
del compute_first


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function compute_goto_table computes the goto table.
Can you write Python code for it?

def compute_goto_table(self):
    self.mGotoTable = {}
    print('\nGoto Table:')
    for state in self.mStates:
        for var in self.mVariables:
            newState = self.goto(state, var)
            if newState != set():
                stateName = self.mStateNames[state]
                newName   = self.mStateNames[newState]
                self.mGotoTable[stateName, var] = newName
                print(f'goto({stateName}, {var}) = {newName}')

Grammar.compute_goto_table = compute_goto_table
del compute_goto_table

grammar

%%time
g = Grammar(grammar)

def strip_quotes(t):
    if t[0] == "'" and t[-1] == "'":
        return t[1:-1]
    return t

def dump_parse_table(self, file):
    with open(file, 'w', encoding="utf-8") as handle:
        handle.write('# Grammar rules:\n')
        for rule in self.mRules:
            rule_name = self.mRuleNames[rule]
            handle.write(f'{rule_name} = ("{rule.mVariable}", {rule.mBody})\n')
        handle.write('\n# Action table:\n')
        handle.write('actionTable = {}\n')
        for s, t in self.mActionTable:
            action = self.mActionTable[s, t]
            t = strip_quotes(t)
            if action[0] == 'reduce':
                rule_name = action[1]
                handle.write(f"actionTable['{s}', '{t}'] = ('reduce', {rule_name})\n")
            elif action == 'accept':
                handle.write(f"actionTable['{s}', '{t}'] = 'accept'\n")
            else:
                handle.write(f"actionTable['{s}', '{t}'] = {action}\n")
        handle.write('\n# Goto table:\n')
        handle.write('gotoTable = {}\n')
        for s, v in self.mGotoTable:
            state = self.mGotoTable[s, v]
            handle.write(f"gotoTable['{s}', '{v}'] = '{state}'\n")

Grammar.dump_parse_table = dump_parse_table
del dump_parse_table

g.dump_parse_table('parse-table.py')

!type parse-table.py

!cat parse-table.py

!del GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rmdir /S /Q __pycache__

!dir /B

!rm GrammarLexer.*...


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function compute_rule_names assigns a unique name to each rule of the grammar.  These names are used later
to represent reduce actions in the action table.
Can you write Python code for it?

def compute_rule_names(self):
    self.mRuleNames = {}
    counter = 0
    for rule in self.mRules:
        self.mRuleNames[rule] = 'r' + str(counter)
        counter += 1

Grammar.compute_rule_names = compute_rule_names
del compute_rule_names


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function move_dot(self) transforms a marked rule  of the form
$$ c \rightarrow \alpha \bullet X\, \beta $$
into a marked rule of the form
$$ c \rightarrow \alpha\, X \bullet \beta, $$
i.e. the $\bullet$ is moved over the next symbol.  Invocation of this method assumes that there is a symbol
following the $\bullet$.
Can you write Python code for it?

def move_dot(self):
    return MarkedRule(self.mVariable,
                      self.mAlpha + (self.mBeta[0],),
                      self.mBeta[1:])

MarkedRule.move_dot = move_dot
del move_dot


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
The function parse_grammar takes a string filename as its argument and returns the grammar that is stored in the specified file.  The grammar is represented as list of rules.  Each rule is represented as a tuple.  The example below will clarify this structure.
Can you write Python code for it?

def parse_grammar(filename):
    input_stream  = antlr4.FileStream(filename, encoding="utf-8")
    lexer         = GrammarLexer(input_stream)
    token_stream  = antlr4.CommonTokenStream(lexer)
    parser        = GrammarParser(token_stream)
    grammar       = parser.start()
    return [GrammarRule(head, tuple(body)) for head, *body in grammar.g]

grammar = parse_grammar('Examples/arith.g')
grammar


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
We start by generating both scanner and parser.
Can you write Python code for it?

!antlr4 -Dlanguage=Python3 Grammar.g4

from GrammarLexer  import GrammarLexer
from GrammarParser import GrammarParser
import antlr4


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/SLR-Table-Generator.ipynb".
The first task is:
We use Antlr to develop a parser for context free grammars.  The pure grammar used to parse context free grammars is stored in the file Pure.g4.  It is similar to the grammar that we have already used to implement Earley's algorithm, but allows additionally the use of the operator |, so that all grammar rules that define a variable can be combined in one rule.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
<b>Exercise</b>: Extending a Shift-Reduce Parser
In this exercise your task is to extend the shift-reduce parser
that has been discussed in the lecture so that it returns an abstract syntax tree.  You should test it with the program sum-for.sl that is given the directory Examples.
Can you write Python code for it?


st, ast = test('Examples/sum-for.sl')
print(st)
print(ast)
display(tuple2dot(st))
display(tuple2dot(ast))


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Exercise 1:  Generate both the action-table and the goto table for this grammar using the notebook SLR-Table-Generator.ipynb.
Implementing a Scanner
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Exercise 2: The function tokenize(s) transforms the string s into a list of tokens.
Given the program sum-for.sl it should produce the list of tokens shown further below.  Note that a number n is stored as a pairs of the form
('NUMBER', n)
and an identifier v is stored as the pair
('ID', v).
You have to take care of keywords like for or while: Syntactically, they are equal to identifiers, but the scanner should <u>not</u> turn them into pairs but rather return them as strings so that the parser does not mistake them for identifiers.
Below is the token list that should be produced from scanning the file sum-for.sl:
['function',
 ('ID', 'sum'),
 '(',
 ('ID', 'n'),
 ')',
 '{',
 ('ID', 's'),
 ':=',
 ('NUMBER', 0),
 ';',
 'for',
 '(',
 ('ID', 'i'),
 ':=',
 ('NUMBER', 1),
 ';',
 ('ID', 'i'),
 '≤',
 ('ID', 'n'),
 '*',
 ('ID', 'n'),
 ';',
 ('ID', 'i'),
 ':=',
 ('ID', 'i'),
 '+',
 ('NUMBER', 1),
 ')',
 '{',
 ('ID', 's'),
 ':=',
 ('ID', 's'),
 '+',
 ('ID', 'i'),
 ';',
 '}',
 'return',
 ('ID', 's'),
 ';',
 '}',
 ('ID', 'print'),
 '(',
 ('ID', 'sum'),
 '(',
 ('NUMBER', 6),
 ')',
 ')',
 ';']
For reference, I have given the old implementation of the function tokenize that has been used in the notebook Shift-Reduce-Parser-Pure.ipynb.  You have to edit this function so that it works with the grammar simple.g.
Can you write Python code for it?

def tokenize(s):
    '''Transform the string s into a list of tokens.  The string s
       is supposed to represent an arithmetic expression.
    '''
    "Edit the code below!"
    lexSpec = r'''([ \t\n]+)      |  # blanks and tabs
                  ([1-9][0-9]*|0) |  # number
                  ([()])          |  # parentheses
                  ([-+*/])        |  # arithmetical operators
                  (.)                # unrecognized character
               '''
    tokenList = re.findall(lexSpec, s, re.VERBOSE)
    result    = []
    for ws, number, parenthesis, operator, error in tokenList:
        if ws:        # skip blanks and tabs
            continue
        elif number:
            result += [ 'NUMBER' ]
        elif parenthesis:
            result += [ parenthesis ]
        elif operator:
            result += [ operator ]
        elif error:
            result += [ f'ERROR({error})']
    return result


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Exercise 3:
The function parse given below is the from the notebook Shift-Reduce-Parser.ipynb. Adapt this function so that it does not just return Trueor False
but rather returns a parse tree as a nested list.  The key idea is that the list Symbols
should now be a list of parse trees and tokens instead of just syntactical variables and tokens, i.e. the syntactical variables should be replaced by their parse trees.
It might be useful to implement an auxilliary function combine_trees that takes a
list of parse trees and combines the into a new parse tree.
Can you write Python code for it?

def parse(self, TL):
    """
    Edit this code so that it returns a parse tree.
    Make use of the auxiliary function combine_trees that you have to
    implement in Exercise 4.
    """
    index   = 0      # points to next token
    Symbols = []     # stack of symbols
    States  = ['s0'] # stack of states, s0 is start state
    TL     += ['$']
    while True:
        q = States[-1]
        t = TL[index]
        print('Symbols:', ' '.join(Symbols + ['|'] + TL[index:]).strip())
        p = self.mActionTable.get((q, t), 'error')
        if p == 'error':
            return False
        elif p == 'accept':
            return True
        elif p[0] == 'shift':
            s = p[1]
            Symbols += [t]
            States  += [s]
            index   += 1
        elif p[0] == 'reduce':
            head, body = p[1]
            n       = len(body)
            if n > 0:
                Symbols = Symbols[:-n]
                States  = States [:-n]
            Symbols = Symbols + [head]
            state   = States[-1]
            States += [ self.mGotoTable[state, head] ]

ShiftReduceParser.parse = parse
del parse


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Exercise 4:
Given a list of tokens and parse trees TL the function combine_trees combines these trees into a new parse tree.  The parse trees are represented as nested tuples.  The data type of a nested tuple is defined recursively:
- A nested tuple is a tuple of the form (Head,) + Body where
  * Head is a string and
  * Body is a tuple of strings, integers, and nested tuples.
When the nested tuple (Head,) + Body is displayed as a tree, Head is used as the label at the root of the tree.  If len(Body) = n, then the root has n children.  These n children are obtained by displaying Body[0], $\cdots$, Body[n-1] as trees.
In order to convert the list of tokens and parse trees into a nested tuple we need a string that can serve as the Head of the parse tree.  The easiest way to to this is to take the first element of TL that is a string because the strings in TL are keywords like for or while or they are operator symbols.  The remaining strings after the first in TL can be discarded.
If there is no string in TL, you can define Head as the empty string.
I suggest a recursive implementation of this function.
The file sum-st.pdf shows the parse tree of the program that is stored in the file sum-for.sl.
Can you write Python code for it?

def combine_trees(TL):
    if len(TL) == 0:
        return ()
    if isinstance(TL, str):
        return (str(TL),)
    Literals = [t for t in TL if     isinstance(t, str)]
    Trees    = [t for t in TL if not isinstance(t, str)]
    if len(Literals) > 0:
        label = Literals[0]
    else:
        label = ''
    result = (label,) + tuple(Trees)
    return result

VoidKeys = { '', '(', ';', 'NUMBER', 'ID' }


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Exercise 5:
The function simplfy_tree(tree) transforms the parse tree tree into an abstract syntax tree.  The parse tree tree is represented as a nested tuple of the form
tree = (head,) + body
The function should simplify the tree as follows:
- If head == '' and body is a tuple of length 2 that starts with an empty string,
  then this tree should be simplified to body[1].
- If head does not contain useful information, for example if head is the empty string
  or an opening parenthesis and, furthermore,  body is a tuple of length 1,
  then this tree should be simplified to body[0].
- By convention, remaining empty Head labels should be replaced by the label '.'
  as this label is traditionally used to construct lists.
I suggest a recursive implementation of this function.
The file sum-ast.pdf shows the abstract syntax tree of the program that is stored in the file sum-for.sl.
Can you write Python code for it?

def simplify_tree(tree):
    if isinstance(tree, int) or isinstance(tree, str):
        return tree
    head, *body = tree
    if body == []:
        return tree
    if head == '' and len(body) == 2 and body[0] == ('',):
        return simplify_tree(body[1])
    if head in VoidKeys and len(body) == 1:
        return simplify_tree(body[0])
    body_simplified = simplify_tree_list(body)
    if head == '(' and len(body) == 2:
        return (body_simplified[0],) + body_simplified[1:]
    if head == '':
        head = '.'
    return (head,) + body_simplified

def simplify_tree_list(TL):
    if TL == []:
        return ()
    tree, *Rest = TL
    return (simplify_tree(tree),) + simplify_tree_list(Rest)


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
Testing
The notebook ../AST-2-Dot.ipynb implements the function tuple2dot(nt) that displays the nested tuple nt as a tree via graphvis.
Can you write Python code for it?

%run ../AST-2-Dot.ipynb

cat -n Examples/sum-for.sl

def test(file):
    with open(file, 'r', encoding='utf-8') as file:
        program = file.read()
    parser = ShiftReduceParser(actionTable, gotoTable)
    TL     = tokenize(program)
    st     = parser.parse(TL)
    ast    = simplify_tree(st)
    return st, ast


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
The cell below tests your tokenizer.  Your task is to compare the output with the output shown above.
Can you write Python code for it?

with open('Examples/sum-for.sl', 'r', encoding='utf-8') as file:
    program = file.read()
tokenize(program)

class ShiftReduceParser():
    def __init__(self, actionTable, gotoTable):
        self.mActionTable = actionTable
        self.mGotoTable   = gotoTable


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
The function parse(self, TL) is called with two arguments:
- self ia an object of class ShiftReduceParser that maintain both an action table
   and a goto table.
- TL is a list of tokens.  Tokens are either
   - literals, i.e. strings enclosed in single quote characters,
   - pairs of the form ('NUMBER', n) where n is a natural number, or
   - the symbol $ denoting the end of input.
Below, it is assumed that parse-table.py is the file that you have created in
Exercise 1.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-AST.ipynb".
The first task is:
The grammar that should be used to parse this program is given in the file
Examples/simple.g.  It is very similar to the grammar that we have developed previously for our interpreter.  I have simplified this grammar at various places to make it more suitable
for the current task.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
A Shift-Reduce Parser for Arithmetic Expressions
In this notebook we implement a simple recursive descend parser for arithmetic expressions.
This parser will implement the following grammar:
$$
  \begin{eqnarray}
  \mathrm{expr}        & \rightarrow & \mathrm{expr}\;\;\texttt{'+'}\;\;\mathrm{product}   \
                       & \mid        & \mathrm{expr}\;\;\texttt{'-'}\;\;\mathrm{product}   \
                       & \mid        & \mathrm{product}                                    \[0.2cm]
  \mathrm{product}     & \rightarrow & \mathrm{product}\;\;\texttt{''}\;\;\mathrm{factor} \
                       & \mid        & \mathrm{product}\;\;\texttt{'/'}\;\;\mathrm{factor} \
                       & \mid        & \mathrm{factor}                                     \[0.2cm]
  \mathrm{factor}      & \rightarrow & \texttt{'('} \;\;\mathrm{expr} \;\;\texttt{')'}     \
                       & \mid        & \texttt{NUMBER}
  \end{eqnarray*}
$$
Implementing a Scanner
Can you write Python code for it?


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
Testing
Can you write Python code for it?

def test(s):
    parser = ShiftReduceParser(actionTable, gotoTable)
    TL     = tokenize(s)
    print(f'tokenlist: {TL}\n')
    if parser.parse(TL):
        print('Parse successful!')
    else:
        print('Parse failed!')

test('(1 + 2) * 3')

test('1 * 2 + 3 * (4 - 5) / 2')

test('11+22*(33-44)/(5-10*5/(4-3))')

test('1+2*3-')


I am working on the file "ANTLR4-Python/SLR-Parser-Generator/Shift-Reduce-Parser-Pure.ipynb".
The first task is:
The function tokenize transforms the string s into a list of tokens. See below for an example.
Can you write Python code for it?

def tokenize(s):
    '''Transform the string s into a list of tokens.  The string s
       is supposed to represent an arithmetic expression.
    '''
    lexSpec = r'''([ \t\n]+)      |  # blanks and tabs
                  ([1-9][0-9]*|0) |  # number
                  ([()])          |  # parentheses
                  ([-+*/])        |  # arithmetical operators
                  (.)                # unrecognized character
               '''
    tokenList = re.findall(lexSpec, s, re.VERBOSE)
    result    = []
    for ws, number, parenthesis, operator, error in tokenList:
        if ws:        # skip blanks and tabs
            continue
        elif number:
            result += [ 'NUMBER' ]
        elif parenthesis:
            result += [ parenthesis ]
        elif operator:
            result += [ operator ]
        elif error:
            result += [ f'ERROR({error})']
    return result

tokenize('1 + 2 * (3 - 4)')

class ShiftReduceParser():
    def __init__(self, actionTable, gotoTable):
        self.mActionTable = actionTable
        self.mGotoTable   = gotoTable

def parse(self, TL):
    index   = 0      # points to next token
    Symbols = []     # stack of symbols
    States  = ['s0'] # stack of states, s0 is start state
    TL     += ['$']
    while True:
        q = States[-1]
        t = TL[index]
        print('Symbols:', ' '.join(Symbols + ['|'] + TL[index:]).strip())
        p = self.mActionTable.get((q, t), 'error')
        if p == 'error':
            return False
        elif p == 'accept':
            return True
        elif p[0] == 'shift':
            s = p[1]
            Symbols += [t]
            States  += [s]
            index   += 1
        elif p[0] == 'reduce':
            head, body = p[1]
            n       = len(body)
            if n > 0:
                Symbols = Symbols[:-n]
                States  = States [:-n]
            Symbols = Symbols + [head]
            state   = States[-1]
            States += [ self.mGotoTable[state, head] ]

ShiftReduceParser.parse =...


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Arecibo burst SNR
Can you write Python code for it?

sp = specAO.reshape(nch1, nint1/16, 16).mean(axis=2)
print(sp.mean(axis=0).max()/sp.mean(axis=0)[:20].std())

tpk_ao = tbary_ao[np.where(specAO.mean(axis=0) == specAO.mean(axis=0).max())][0]
print('%.10f' % tpk_ao)


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Calculate dm delay and topocentric correction for VLA
Can you write Python code for it?

dmt_vla = dmdelay(dm0, st['freq'][-1], 1e4)/(24*3600)
topo_vla = 75.7533100 # seconds, from presto
mjdfrac_vla = st['segmenttimes'][7][0] + intrange[0]*st['inttime']/(24*3600) - mjd0 - dmt_vla + topo_vla/(24*3600)
print('VLA dMJD: {0}'.format(mjdfrac_vla))
print('AO dMJD: {0}'.format(mjdfrac_ao))
print('Diff: {0} s'.format((mjdfrac_ao-mjdfrac_vla)*24*3600))

tbary_vla = np.linspace(mjdfrac_vla, mjdfrac_vla+(dint*st['inttime']/(24*3600)), dint)
freqs_vla = 1e3*st['freq']
extent_vla = (tbary_vla[0], tbary_vla[-1], freqs_vla[0], freqs_vla[-1])

# put them together
fig = pl.figure(figsize=(8,8))
fig.add_subplot(211)
pl.imshow(specVLA.transpose(), interpolation='nearest', origin='bottom', extent=extent_vla, aspect='auto')
fig.add_subplot(212)
pl.imshow(specAO, interpolation='nearest', aspect='auto', extent=extent_ao)

tpk_vla = tbary_vla[np.where(specVLA.mean(axis=1) == specVLA.mean(axis=1).max())][0]
print('%.10f' % tpk_vla)


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Calculate implied spectral index and limits
Can you write Python code for it?

def speclim(s):
    freqs = s[0]
    fluxes = s[1]
    for i in range(len(freqs)-1):
        freqi = freqs[i]
        freqi1 = freqs[i+1]

        # correct for plotting offset
        if freqi <= 1.5:
            freqi = 1.4
        elif freqi1 > 4.2:
            freqi1 = 4.85

        print(freqi, freqi1)
        print(fluxes[i], fluxes[i+1])
        print(np.log10(fluxes[i]/fluxes[i+1])/np.log10(freqi/freqi1))

speclim(s45d)

speclim(s48d)

speclim(s49d)


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Calculate residual sweep and compare to DM model
Can you write Python code for it?

# some residual dm sweep?
print(24*3600*(tbary_ao[np.where(specAO[128:].mean(axis=0) == specAO[128:].mean(axis=0).max())][0] -
               tbary_ao[np.where(specAO[:128].mean(axis=0) == specAO[:128].mean(axis=0).max())][0]))

dmt_ao_delta = dmdelay(dm0, obsfreq*1e-3, st['freq'][-1]) - dmdelay(560, obsfreq*1e-3, st['freq'][-1])
dmt_ao_inter = dmdelay(dm0, obsfreq*1e-3, (obsfreq-500)*1e-3) - dmdelay(560, obsfreq*1e-3, (obsfreq-500)*1e-3)
print(dmt_ao_delta, dmt_ao_inter)


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Define python names for Arecibo header info
Can you write Python code for it?

nch0 = 512
obsfreq = 1780.
df = 1.5625
tint0 = 2.04780080448418e-05
#dm0 = 560.5
dm0 = 553.7  # equivalent to 560.5 in correct scaling

dmt_ao = dmdelay(dm0, obsfreq*1e-3, 1e4)/(24*3600)
bary_ao = 75.7555109 # seconds, from presto
mjd0 = int(57648) # subtract this for convenience. must be an integer mjd
#mjdfrac_ao = (37750 + 0.319370962786796)/(24*3600) - dmt_ao # start time from header
#mjdfrac_ao = (37750 + 0.336629511829426)/(24*3600) - dmt_ao # start time from header, second round
mjdfrac_ao = 57648.43692057008 - dmt_ao - mjd0 + bary_ao/(24*3600)

nint0 = 4096
dt_ao = tint0*nint0


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Make nice Arecibo burst spectrum like the VLA ones
Can you write Python code for it?

fig = pl.figure(figsize=(15,7))
ax = fig.add_subplot(111)
pl.plot(1e-3*freqs_ao[::-1], Sspec, 'k.')
pl.text(0.75, 0.88, '57648, Arecibo', horizontalalignment='left', fontsize=24,
        verticalalignment='center', transform=ax.transAxes)
pl.errorbar(1.600, 1.2*Sspec.max(), yerr=(3./np.sqrt(3.125e6*window*2)), fmt='k.', ecolor='k')
pl.ylim(-0.03, Sspec.max()*1.4)
pl.xlabel('Frequency (GHz)', fontsize=18)
pl.ylabel('Flux density (Jy)', fontsize=18)
xt = pl.setp(ax.get_xticklabels(), fontsize=18)
yt = pl.setp(ax.get_yticklabels(), fontsize=18)
ax.xaxis.set_tick_params(width=4, color='k')
ax.yaxis.set_tick_params(width=4, color='k')
fig.savefig('specAO_57648.pdf', format='pdf')


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Read coherently dedispersed Arecibo dynamic spectrum
Can you write Python code for it?


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Read data with rtpipe and phase it to FRB 121102
Can you write Python code for it?

key = '57648'
read = {}
sdmfile = '16A-496_sb32698778_1_02h00m_000.57648.37452900463.cut/'
scannum = getscannum(sdmfile)
read[key] = read_cut(sdmfile, scannum, 7, npix_max=7400, chans=range(2,256))


I am working on the file "AOVLA_spectrum.ipynb".
The first task is:
Useful functions
Can you write Python code for it?

dmdelay_new = lambda dm, f0, f1: 4.1488e-3*dm*(1./f0**2 - 1./f1**2)  # inconsistent with rtpipe
dmdelay = lambda dm, f0, f1: 4.2e-3*dm*(1./f0**2 - 1./f1**2)  # consistent with rtpipe, but requires scaling by 1.2%


I am working on the file "APS 7.ipynb".
The first task is:
Dessa maneira conclui-se que o T observado está entre os valores de T máximo e mínimo, e portanto, pode se rejeitar a hipótese.
c) Um teste t pode ser usado se a distribuição da variável original for normal. Faça um QQ-Plot para verificar se a amostra é normal, ou seja, se poderíamos ter feito o teste t
Dica: Use stats.probplot(dados, dist='norm', plot=plt) , uma função que faz o QQ Plot de forma imediata.
Can you write Python code for it?

stats.probplot(amostra2, dist='norm', plot=plt)
plt.show()


I am working on the file "APS 7.ipynb".
The first task is:
Questão 1
Você foi contratado para avaliar um novo modelo de taco de golfe. O fabricante quer saber se há evidência de que o coeficiente de restituição médio dos tacos excede 0.82 com nível de significância α=0.05. Em um lançador padronizado de bolas de golfe, foram testados 15 tacos e os coeficientes de restituição medidos foram: 0.8411, 0.8191, 0.8182, 0.8125, 0.8580, 0.8532, 0.8483, 0.8276, 0.8042, 0.8730, 0.8282, 0.8359, 0.8750, 0.7983, 0.8660
Sabe-se por experiência anterior que o coeficiente de restituição tem distribuição normal, mas você não sabe o σ.
Pede-se:
a) Calcule ¯x e s (lembre-se dos graus de liberdade)
Can you write Python code for it?


I am working on the file "APS 7.ipynb".
The first task is:
Questão 3
Algumas peças de polímero produzidas por uma máquina foram submetidas a
um ensaio de tração. As resistências medidas foram (em Kg/cm2):
[ 129.26, 204.49, 116.89, 106.4 , 95.3 , 123.35, 92.3 , 300.02, 264.34, 168.27, 80.02,
117.96, 117.73, 370.47, 462.59, 92.58, 258.52, 82.59]
Não sabemos nada sobre o formato da distribuição da população. Pede-se:
a) Crie um intervalo percentil bootstrap de 95% para a média amostral. Forneça o limite inferior e o limite superior.
Can you write Python code for it?


I am working on the file "APS 7.ipynb".
The first task is:
b) Crie um intervalo percentil bootstrap de 95% para o desvio padrão amostral
Can you write Python code for it?

from math import sqrt
v = []
for i in range(10000):
    b = np.random.choice(amostra3,size = n, replace = True)
    var_amostra3 = sqrt(np.var(b))
    v.append(var_amostra3)

print(pd.Series(v).quantile(al/2))
print(pd.Series(v).quantile(1-(al/2)))


I am working on the file "APS 7.ipynb".
The first task is:
b) Encontre os valores críticos e a região de rejeição
Can you write Python code for it?

alpha = 0.05
maximo = stats.t.ppf(1-(alpha)/2, loc = 0, df = tamanho-1)
print(maximo)
minimo = stats.t.ppf(alpha/2, loc = 0, df = tamanho-1)
print(minimo)


I am working on the file "APS 7.ipynb".
The first task is:
c) Calcule a estatística t de teste para a t padrão usando a relação t_o bs=(¯(x_obs )-μ_0)/(s/√n)
Can you write Python code for it?

mi0 = 0.82
tam = len(amostra)
Tobservado = (x_ - mi0)/(s/sqrt(tam))
print(Tobservado)


I am working on the file "APS5_alunos.ipynb".
The first task is:
Calcule a correlação entre as variáveis Magnitude Error  e Depth
Can you write Python code for it?


I am working on the file "APS5_alunos.ipynb".
The first task is:
Explique o que significa o valor da correlação calculada acima?
A correlação apresentada acima mostra uma espécie de dependência entre as duas variáveis, no caso Magnitude Error e Depth, observando o gráfico mostrado acima os valores são bem distantes, mas é justamente isso e o valor da correlação mostrado, que é baixo, que mostra uma alta dependência entre as duas variáveis, não há grande discrepância entre os valores. O fato de ser negativo justificaria uma reta descrescente.
Q5 - Describe e boxplot
Faça o describe e o boxplot da Latitude e da Longitude. Explique os valores
Can you write Python code for it?

Lat = df["Latitude"].describe()
Long = df["Longitude"].describe()
print(Lat,Long)

df.boxplot(column = ["Latitude","Longitude"])
plt.show()