CVE-2019-19275 (C, published 2019-11-26T15:15Z)
Description: value: typed_ast 1.3.0 and 1.3.1 has an ast_for_arguments out-of-bounds read. An attacker with the ability to cause a Python interpreter to parse Python source (but not necessarily execute it) may be able to crash the interpreter process. This could be a concern, for example, in a web-based service that parses (but does not execute) Python code. (This issue also affected certain Python 3.8.0-alpha prereleases.)
Severity: MEDIUM (CVSS v3 base score: 7.5)
Weakness: CWE-125 \u2014 Out-of-bounds Read
The product reads data past the end, or before the beginning, of the intended buffer.
Repository: https://github.com/python/typed_ast
Files changed:
appveyor.yml
ast27/Python/ast.c
ast3/Custom/typed_ast.c
ast3/Grammar/Grammar
ast3/Include/Python-ast.h
ast3/Include/ast.h
ast3/Include/bitset.h
ast3/Include/compile-ast3.h
ast3/Include/errcode.h
ast3/Include/graminit.h
ast3/Include/grammar.h
ast3/Include/node.h
ast3/Include/parsetok.h
ast3/Include/token.h
ast3/Parser/Python.asdl
ast3/Parser/asdl_c.py
ast3/Parser/grammar1.c
ast3/Parser/parser.h
ast3/Parser/parsetok.c
ast3/Parser/tokenizer.c
ast3/Parser/tokenizer.h
ast3/Python/Python-ast.c
ast3/Python/ast.c
ast3/Python/graminit.c
ast3/tests/test_basics.py
setup.py
tools/Grammar.patch
tools/Python-asdl.patch
tools/asdl_c.patch
tools/ast.patch
tools/find_exported_symbols
tools/parsetok.patch
tools/script
tools/token.patch
tools/tokenizer.patch
tools/update_ast3_asdl
tools/update_ast3_grammar
typed_ast/__init__.py
typed_ast/ast3.py
update_process.md
Commit message: Fully incorporate the code from Python 3.7.2 (#78)

This is a full port, following the recipe in update_process.md. I've also tried to keep the recipe up to date and improved the automation (see tools/script). I haven't cleaned up the commits. As of #77 there are a few tests that sanity-check this (though it's far from a full test suite), and they're run by Travis-CI and AppVeyor.

Vulnerable code:
    - PYTHON: "C:\\Python33"
    - PYTHON: "C:\\Python33-x64"
      DISTUTILS_USE_SDK: 1
        int quote = Py_CHARMASK(*s);
                    char *s = "";
                char *tmp;
#include "compile.h"
    node *n = Ta3Parser_ParseStringObject(s, filename,
                                         &_Ta3Parser_Grammar, start, &err,
                                         &iflags);
# Note:  Changing the grammar specified in this file will most likely
#        require corresponding changes in the parser module
#        (../Modules/parsermodule.c).  If you can't make the changes to
#        that module yourself, please co-ordinate the required changes
#        with someone who can; ask around on python-dev for help.  Fred
#        Drake <fdrake@acm.org> will probably be listening there.

# https://docs.python.org/devguide/grammar.html
comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
        
expr_ty _Ta3_Str(string s, string kind, int lineno, int col_offset, PyArena
                 *arena);
#define BYTE		char
#define BITSPERBYTE	(8*sizeof(BYTE))
#define NBYTES(nbits)	(((nbits) + BITSPERBYTE - 1) / BITSPERBYTE)
#define BIT2BYTE(ibit)	((ibit) / BITSPERBYTE)
#define BIT2SHIFT(ibit)	((ibit) % BITSPERBYTE)
#define BIT2MASK(ibit)	(1 << BIT2SHIFT(ibit))
#define BYTE2BIT(ibyte)	((ibyte) * BITSPERBYTE)
#ifndef Ta3_COMPILE_H
#define Ta3_COMPILE_H

#define Py_func_type_input 342

#endif /* !Ta3_COMPILE_H */
#define E_OK		10	/* No error */
#define E_EOF		11	/* End Of File */
#define E_INTR		12	/* Interrupted */
#define E_TOKEN		13	/* Bad token */
#define E_SYNTAX	14	/* Syntax error */
#define E_NOMEM		15	/* Ran out of memory */
#define E_DONE		16	/* Parsing complete */
#define E_ERROR		17	/* Execution error */
#define E_TABSPACE	18	/* Inconsistent mixing of tabs and spaces */
#define E_OVERFLOW      19	/* Node had too many children */
#define E_TOODEEP	20	/* Too many indentation levels */
#define E_DEDENT	21	/* No matching outer block for dedent */
#define E_DECODE	22	/* Error in decoding into Unicode */
#define E_EOFS		23	/* EOF in triple-quoted string */
#define E_EOLS		24	/* EOL in single-quoted string */
#define E_LINECONT	25	/* Unexpected characters after a line continuation */
#define E_BADSINGLE	27	/* Ill-formed single statement input */
#define comp_for 337
#define comp_if 338
#define encoding_decl 339
#define yield_expr 340
#define yield_arg 341
#define func_type_input 342
#define func_type 343
#define typelist 344
    int		 lb_type;
    char	*lb_str;
#define EMPTY 0		/* Label number 0 is by definition the empty label */
    int		 ll_nlabels;
    label	*ll_label;
    short	a_lbl;		/* Label of this arc */
    short	a_arrow;	/* State where this arc goes to */
    int		 s_narcs;
    arc		*s_arc;		/* Array of arcs */
    int		 s_lower;	/* Lowest label index */
    int		 s_upper;	/* Highest label index */
    int		*s_accel;	/* Accelerator */
    int		 s_accept;	/* Nonzero for accepting state */
    int		 d_type;	/* Non-terminal this represents */
    char	*d_name;	/* For printing */
    int		 d_initial;	/* Initial state */
    int		 d_nstates;
    state	*d_state;	/* Array of states */
    bitset	 d_first;
    int		 g_ndfas;
    dfa		*g_dfa;		/* Array of DFAs */
    labellist	 g_ll;
    int		 g_start;	/* Start symbol of the grammar */
    int		 g_accel;	/* Set if accelerators present */
    short		n_type;
    char		*n_str;
    int			n_lineno;
    int			n_col_offset;
    int			n_nchildren;
    struct _node	*n_child;
extern node *Ta3Node_New(int type);
#define NCH(n)		((n)->n_nchildren)
#define CHILD(n, i)	(&(n)->n_child[i])
#define RCHILD(n, i)	(CHILD(n, NCH(n) + i))
#define TYPE(n)		((n)->n_type)
#define STR(n)		((n)->n_str)
#define PyPARSE_YIELD_IS_KEYWORD	0x0001
#define PyPARSE_DONT_IMPLY_DEDENT	0x0002
#define PyPARSE_WITH_IS_KEYWORD		0x0003
PyAPI_FUNC(void) PyParser_SetError(perrdetail *);
PyAPI_FUNC(void) PyParser_ClearError(perrdetail *);
#define ENDMARKER	0
#define NAME		1
#define NUMBER		2
#define STRING		3
#define NEWLINE		4
#define INDENT		5
#define DEDENT		6
#define LPAR		7
#define RPAR		8
#define LSQB		9
#define RSQB		10
#define COLON		11
#define COMMA		12
#define SEMI		13
#define PLUS		14
#define MINUS		15
#define STAR		16
#define SLASH		17
#define VBAR		18
#define AMPER		19
#define LESS		20
#define GREATER		21
#define EQUAL		22
#define DOT		23
#define PERCENT		24
#define LBRACE		25
#define RBRACE		26
#define EQEQUAL		27
#define NOTEQUAL	28
#define LESSEQUAL	29
#define GREATEREQUAL	30
#define TILDE		31
#define CIRCUMFLEX	32
#define LEFTSHIFT	33
#define RIGHTSHIFT	34
#define DOUBLESTAR	35
#define PLUSEQUAL	36
#define MINEQUAL	37
#define STAREQUAL	38
#define SLASHEQUAL	39
#define PERCENTEQUAL	40
#define AMPEREQUAL	41
#define VBAREQUAL	42
#define CIRCUMFLEXEQUAL	43
#define LEFTSHIFTEQUAL	44
#define RIGHTSHIFTEQUAL	45
#define DOUBLESTAREQUAL	46
#define DOUBLESLASH	47
#define ATEQUAL		50
#define OP		53
#define AWAIT		54
#define ASYNC		55
#define ERRORTOKEN	58
#define N_TOKENS	59
#define NT_OFFSET		256
#define ISTERMINAL(x)		((x) < NT_OFFSET)
#define ISNONTERMINAL(x)	((x) >= NT_OFFSET)
#define ISEOF(x)		((x) == ENDMARKER)
extern const char *_Ta3Parser_TokenNames[]; /* Token names */
         | Str(string s, string kind)
            line = (" " * TABSIZE * depth) + line + "\n"
            self.file.write(line)
        if field.opt:
            check = "exists_not_none(obj, &PyId_%s)" % (field.name,)
            check = "_PyObject_HasAttrId(obj, &PyId_%s)" % (field.name,)
        self.emit("if (%s) {" % (check,), depth, reflow=False)
        self.emit("tmp = _PyObject_GetAttrId(obj, &PyId_%s);" % field.name, depth+1)
        self.emit("if (tmp == NULL) goto failed;", depth+1)
        if field.seq:
            self.emit("%s value;" % ctype, depth+2)
            self.emit("res = obj2ast_%s(PyList_GET_ITEM(tmp, i), &value, arena);" %
            self.emit("asdl_seq_SET(%s, i, value);" % field.name, depth+2)
        self.emit("} else {", depth)
        if not field.opt:
            message = "required field \\\"%s\\\" missing from %s" % (field.name, name)
            format = "PyErr_SetString(PyExc_TypeError, \"%s\");"
            self.emit(format % message, depth+1, reflow=False)
            self.emit("return 1;", depth+1)
        else:
            if self.isNumeric(field):
                self.emit("%s = 0;" % field.name, depth+1)
            elif not self.isSimpleType(field):
                self.emit("%s = NULL;" % field.name, depth+1)
            else:
                raise TypeError("could not determine the default value for %s" % field.name)
static void
    _Py_IDENTIFIER(_fields);
    fields = _PyObject_GetAttrId((PyObject*)Py_TYPE(self), &PyId__fields);
    if (!fields)
        PyErr_Clear();
    if (PyTuple_GET_SIZE(args) > 0) {
        if (numfields != PyTuple_GET_SIZE(args)) {
            PyErr_Format(PyExc_TypeError, "%.400s constructor takes %s"
                         "%zd positional argument%s",
                         Py_TYPE(self)->tp_name,
                         numfields == 0 ? "" : "either 0 or ",
                         numfields, numfields == 1 ? "" : "s");
        for (i = 0; i < PyTuple_GET_SIZE(args); i++) {
            /* cannot be reached when fields is NULL */
            PyObject *name = PySequence_GetItem(fields, i);
            if (!name) {
                res = -1;
                goto cleanup;
            }
            res = PyObject_SetAttr(self, name, PyTuple_GET_ITEM(args, i));
            Py_DECREF(name);
            if (res < 0)
                goto cleanup;
        }
    PyObject *res;
    PyObject *dict = _PyObject_GetAttrId(self, &PyId___dict__);
    if (dict == NULL) {
        if (PyErr_ExceptionMatches(PyExc_AttributeError))
            PyErr_Clear();
        else
            return NULL;
        res = Py_BuildValue("O()O", Py_TYPE(self), dict);
        Py_DECREF(dict);
        return res;
    result = PyObject_CallFunction((PyObject*)&PyType_Type, "s(O){sOss}",
                    type, base, "_fields", fnames, "__module__", "_ast3");
    _Py_IDENTIFIER(_attributes);
        PyDict_SetItemString(d, "_fields", empty_tuple) < 0 ||
        PyDict_SetItemString(d, "_attributes", empty_tuple) < 0) {
static int exists_not_none(PyObject *obj, _Py_Identifier *id)
{
    int isnone;
    PyObject *attr = _PyObject_GetAttrId(obj, id);
    if (!attr) {
        PyErr_Clear();
        return 0;
    }
    isnone = attr == Py_None;
    Py_DECREF(attr);
    return !isnone;
}

        # add parse method to module
        self.emit('PyObject *ast3_parse(PyObject *self, PyObject *args);', 0)
        self.emit('static PyMethodDef ast3_methods[] = {', 0)
        self.emit('{"_parse",  ast3_parse, METH_VARARGS, "Parse string into typed AST."},', 1)
        self.emit('{NULL, NULL, 0, NULL}', 1)
        self.emit('};', 0)

        self.emit("static struct PyModuleDef _astmodule3 = {", 0)
        self.emit('  PyModuleDef_HEAD_INIT, "_ast3", NULL, 0, ast3_methods', 0)
        self.emit('m = PyModule_Create(&_astmodule3);', 1)
        self.emit("Py_INCREF(Py_None);", 2)
        self.emit('return Py_None;', 2)
    if INC_DIR:
        p = "%s/%s-ast.h" % (INC_DIR, mod.name)
        f = open(p, "w")
        f.write(auto_gen_msg)
        f.write('#include "asdl.h"\n\n')
        c = ChainOfVisitors(TypeDefVisitor(f),
                            StructVisitor(f),
                            PrototypeVisitor(f),
                            )
        c.visit(mod)
        f.write("PyObject* Ta3AST_mod2obj(mod_ty t);\n")
        f.write("mod_ty Ta3AST_obj2mod(PyObject* ast, PyArena* arena, int mode);\n")
        f.write("int Ta3AST_Check(PyObject* obj);\n")
        f.close()

    if SRC_DIR:
        p = os.path.join(SRC_DIR, str(mod.name) + "-ast.c")
        f = open(p, "w")
        f.write(auto_gen_msg)
        f.write('#include <stddef.h>\n')
        f.write('\n')
        f.write('#include "Python.h"\n')
        f.write('#include "%s-ast.h"\n' % mod.name)
        f.write('\n')
        f.write("static PyTypeObject AST_type;\n")
        v = ChainOfVisitors(
            PyTypesDeclareVisitor(f),
            PyTypesVisitor(f),
            Obj2ModPrototypeVisitor(f),
            FunctionVisitor(f),
            ObjVisitor(f),
            Obj2ModVisitor(f),
            ASTModuleVisitor(f),
            PartingShots(f),
            )
        v.visit(mod)
        f.close()
    INC_DIR = ''
    SRC_DIR = ''
            INC_DIR = v
            SRC_DIR = v
    if INC_DIR and SRC_DIR:
    assert(0);
    /* NOTREACHED */
	int		 s_state;	/* State in current DFA */
	dfa		*s_dfa;		/* Current DFA */
	struct _node	*s_parent;	/* Where to add next node */
	stackentry	*s_top;		/* Top entry */
	stackentry	 s_base[MAXSTACK];/* Array of stack entries */
					/* NB The stack grows down */
	stack	 	p_stack;	/* Stack of parser states */
	grammar		*p_grammar;	/* Grammar to use */
	node		*p_tree;	/* Top of parse tree */
	unsigned long	p_flags;	/* see co_flags in Include/code.h */
        len = b - a; /* XXX this may compute NULL - NULL */
                err_ret->text = "with Barry as BDFL, use '<>' "
                                "instead of '!='";
        if (a >= tok->line_start)
        else
#if PY_MINOR_VERSION >= 4
#else
// Python 3.3 doesn't have PyAPI_FUNC, but it's not supported on Windows anyway.
char *PyOS_Readline(FILE *, FILE *, char *);
#endif
    tok->altwarning = 1;
    tok->alterror = 1;
    tok->alttabsize = 1;
        bufobj = PyObject_CallObject(tok->decoding_readline, NULL);
        PyObject *bufobj = PyObject_CallObject(readline, NULL);
            buf = PyObject_CallObject(tok->decoding_readline, NULL);
    if (tok->alterror) {
        tok->done = E_TABSPACE;
        tok->cur = tok->inp;
        return 1;
    }
    if (tok->altwarning) {
#ifdef PGEN
        PySys_WriteStderr("inconsistent use of tabs and spaces "
                          "in indentation\n");
#else
        PySys_FormatStderr("%U: inconsistent use of tabs and spaces "
                          "in indentation\n", tok->filename);
#endif
        tok->altwarning = 0;
    }
    return 0;
                col = (col/tok->tabsize + 1) * tok->tabsize;
                altcol = (altcol/tok->alttabsize + 1)
                    * tok->alttabsize;
                    if (indenterror(tok)) {
                        return ERRORTOKEN;
                    }
                    if (indenterror(tok)) {
                        return ERRORTOKEN;
                    }
                    if (indenterror(tok)) {
                        return ERRORTOKEN;
                    }
        /* Process b"", r"", u"", br"" and rb"" */
            if (tok->async_def) {
#if PY_MINOR_VERSION >= 4
#endif
    int altwarning;     /* Issue warning if alternate tabs don't match */
    int alterror;       /* Issue error if alternate tabs don't match */
    int alttabsize;     /* Alternate tab spacing */
extern char * PyTokenizer_RestoreEncoding(struct tok_state* tok,
                                          int len, int *offset);
/* Copyright (c) 2008-2009, Google Inc.
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are
 * met:
 *
 *     * Redistributions of source code must retain the above copyright
 * notice, this list of conditions and the following disclaimer.
 *     * Neither the name of Google Inc. nor the names of its
 * contributors may be used to endorse or promote products derived from
 * this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
 * ---
 * Author: Kostya Serebryany
 */

#ifdef _MSC_VER
# include <windows.h>
#endif

#ifdef __cplusplus
# error "This file should be built as pure C to avoid name mangling"
#endif

#include <stdlib.h>
#include <string.h>

#include "dynamic_annotations.h"

/* Each function is empty and called (via a macro) only in debug mode.
   The arguments are captured by dynamic tools at runtime. */

#if DYNAMIC_ANNOTATIONS_ENABLED == 1

void AnnotateRWLockCreate(const char *file, int line,
                          const volatile void *lock){}
void AnnotateRWLockDestroy(const char *file, int line,
                           const volatile void *lock){}
void AnnotateRWLockAcquired(const char *file, int line,
                            const volatile void *lock, long is_w){}
void AnnotateRWLockReleased(const char *file, int line,
                            const volatile void *lock, long is_w){}
void AnnotateBarrierInit(const char *file, int line,
                         const volatile void *barrier, long count,
                         long reinitialization_allowed) {}
void AnnotateBarrierWaitBefore(const char *file, int line,
                               const volatile void *barrier) {}
void AnnotateBarrierWaitAfter(const char *file, int line,
                              const volatile void *barrier) {}
void AnnotateBarrierDestroy(const char *file, int line,
                            const volatile void *barrier) {}

void AnnotateCondVarWait(const char *file, int line,
                         const volatile void *cv,
                         const volatile void *lock){}
void AnnotateCondVarSignal(const char *file, int line,
                           const volatile void *cv){}
void AnnotateCondVarSignalAll(const char *file, int line,
                              const volatile void *cv){}
void AnnotatePublishMemoryRange(const char *file, int line,
                                const volatile void *address,
                                long size){}
void AnnotateUnpublishMemoryRange(const char *file, int line,
                                  const volatile void *address,
                                  long size){}
void AnnotatePCQCreate(const char *file, int line,
                       const volatile void *pcq){}
void AnnotatePCQDestroy(const char *file, int line,
                        const volatile void *pcq){}
void AnnotatePCQPut(const char *file, int line,
                    const volatile void *pcq){}
void AnnotatePCQGet(const char *file, int line,
                    const volatile void *pcq){}
void AnnotateNewMemory(const char *file, int line,
                       const volatile void *mem,
                       long size){}
void AnnotateExpectRace(const char *file, int line,
                        const volatile void *mem,
                        const char *description){}
void AnnotateBenignRace(const char *file, int line,
                        const volatile void *mem,
                        const char *description){}
void AnnotateBenignRaceSized(const char *file, int line,
                             const volatile void *mem,
                             long size,
                             const char *description) {}
void AnnotateMutexIsUsedAsCondVar(const char *file, int line,
                                  const volatile void *mu){}
void AnnotateTraceMemory(const char *file, int line,
                         const volatile void *arg){}
void AnnotateThreadName(const char *file, int line,
                        const char *name){}
void AnnotateIgnoreReadsBegin(const char *file, int line){}
void AnnotateIgnoreReadsEnd(const char *file, int line){}
void AnnotateIgnoreWritesBegin(const char *file, int line){}
void AnnotateIgnoreWritesEnd(const char *file, int line){}
void AnnotateIgnoreSyncBegin(const char *file, int line){}
void AnnotateIgnoreSyncEnd(const char *file, int line){}
void AnnotateEnableRaceDetection(const char *file, int line, int enable){}
void AnnotateNoOp(const char *file, int line,
                  const volatile void *arg){}
void AnnotateFlushState(const char *file, int line){}

static int GetRunningOnValgrind(void) {
#ifdef RUNNING_ON_VALGRIND
  if (RUNNING_ON_VALGRIND) return 1;
#endif

#ifndef _MSC_VER
  char *running_on_valgrind_str = getenv("RUNNING_ON_VALGRIND");
  if (running_on_valgrind_str) {
    return strcmp(running_on_valgrind_str, "0") != 0;
  }
#else
  /* Visual Studio issues warnings if we use getenv,
   * so we use GetEnvironmentVariableA instead.
   */
  char value[100] = "1";
  int res = GetEnvironmentVariableA("RUNNING_ON_VALGRIND",
                                    value, sizeof(value));
  /* value will remain "1" if res == 0 or res >= sizeof(value). The latter
   * can happen only if the given value is long, in this case it can't be "0".
   */
  if (res > 0 && !strcmp(value, "0"))
    return 1;
#endif
  return 0;
}

/* See the comments in dynamic_annotations.h */
int RunningOnValgrind(void) {
  static volatile int running_on_valgrind = -1;
  /* C doesn't have thread-safe initialization of statics, and we
     don't want to depend on pthread_once here, so hack it. */
  int local_running_on_valgrind = running_on_valgrind;
  if (local_running_on_valgrind == -1)
    running_on_valgrind = local_running_on_valgrind = GetRunningOnValgrind();
  return local_running_on_valgrind;
}

#endif  /* DYNAMIC_ANNOTATIONS_ENABLED == 1 */

/* Computation of FIRST stets */

#include "pgenheaders.h"
#include "grammar.h"
#include "token.h"

extern int Py_DebugFlag;

/* Forward */
static void calcfirstset(grammar *, dfa *);

void
addfirstsets(grammar *g)
{
    int i;
    dfa *d;

    if (Py_DebugFlag)
        printf("Adding FIRST sets ...\n");
    for (i = 0; i < g->g_ndfas; i++) {
        d = &g->g_dfa[i];
        if (d->d_first == NULL)
            calcfirstset(g, d);
    }
}

static void
calcfirstset(grammar *g, dfa *d)
{
    int i, j;
    state *s;
    arc *a;
    int nsyms;
    int *sym;
    int nbits;
    static bitset dummy;
    bitset result;
    int type;
    dfa *d1;
    label *l0;

    if (Py_DebugFlag)
        printf("Calculate FIRST set for '%s'\n", d->d_name);

    if (dummy == NULL)
        dummy = newbitset(1);
    if (d->d_first == dummy) {
        fprintf(stderr, "Left-recursion for '%s'\n", d->d_name);
        return;
    }
    if (d->d_first != NULL) {
        fprintf(stderr, "Re-calculating FIRST set for '%s' ???\n",
            d->d_name);
    }
    d->d_first = dummy;

    l0 = g->g_ll.ll_label;
    nbits = g->g_ll.ll_nlabels;
    result = newbitset(nbits);

    sym = (int *)PyObject_MALLOC(sizeof(int));
    if (sym == NULL)
        Py_FatalError("no mem for new sym in calcfirstset");
    nsyms = 1;
    sym[0] = findlabel(&g->g_ll, d->d_type, (char *)NULL);

    s = &d->d_state[d->d_initial];
    for (i = 0; i < s->s_narcs; i++) {
        a = &s->s_arc[i];
        for (j = 0; j < nsyms; j++) {
            if (sym[j] == a->a_lbl)
                break;
        }
        if (j >= nsyms) { /* New label */
            sym = (int *)PyObject_REALLOC(sym,
                                    sizeof(int) * (nsyms + 1));
            if (sym == NULL)
                Py_FatalError(
                    "no mem to resize sym in calcfirstset");
            sym[nsyms++] = a->a_lbl;
            type = l0[a->a_lbl].lb_type;
            if (ISNONTERMINAL(type)) {
                d1 = Ta3Grammar_FindDFA(g, type);
                if (d1->d_first == dummy) {
                    fprintf(stderr,
                        "Left-recursion below '%s'\n",
                        d->d_name);
                }
                else {
                    if (d1->d_first == NULL)
                        calcfirstset(g, d1);
                    mergebitset(result,
                                d1->d_first, nbits);
                }
            }
            else if (ISTERMINAL(type)) {
                addbit(result, a->a_lbl);
            }
        }
    }
    d->d_first = result;
    if (Py_DebugFlag) {
        printf("FIRST set for '%s': {", d->d_name);
        for (i = 0; i < nbits; i++) {
            if (testbit(result, i))
                printf(" %s", Ta3Grammar_LabelRepr(&l0[i]));
        }
        printf(" }\n");
    }

    PyObject_FREE(sym);
}

/* List a node on a file */

#include "pgenheaders.h"
#include "token.h"
#include "node.h"

/* Forward */
static void list1node(FILE *, node *);
static void listnode(FILE *, node *);

void
PyNode_ListTree(node *n)
{
    listnode(stdout, n);
}

static int level, atbol;

static void
listnode(FILE *fp, node *n)
{
    level = 0;
    atbol = 1;
    list1node(fp, n);
}

static void
list1node(FILE *fp, node *n)
{
    if (n == 0)
        return;
    if (ISNONTERMINAL(TYPE(n))) {
        int i;
        for (i = 0; i < NCH(n); i++)
            list1node(fp, CHILD(n, i));
    }
    else if (ISTERMINAL(TYPE(n))) {
        switch (TYPE(n)) {
        case INDENT:
            ++level;
            break;
        case DEDENT:
            --level;
            break;
        default:
            if (atbol) {
                int i;
                for (i = 0; i < level; ++i)
                    fprintf(fp, "\t");
                atbol = 0;
            }
            if (TYPE(n) == NEWLINE) {
                if (STR(n) != NULL)
                    fprintf(fp, "%s", STR(n));
                fprintf(fp, "\n");
                atbol = 1;
            }
            else
                fprintf(fp, "%s ", STR(n));
            break;
        }
    }
    else
        fprintf(fp, "? ");
}

#include "pgenheaders.h"
#include "metagrammar.h"
#include "grammar.h"
#include "pgen.h"
static arc arcs_0_0[3] = {
    {2, 0},
    {3, 0},
    {4, 1},
};
static arc arcs_0_1[1] = {
    {0, 1},
};
static state states_0[2] = {
    {3, arcs_0_0},
    {1, arcs_0_1},
};
static arc arcs_1_0[1] = {
    {5, 1},
};
static arc arcs_1_1[1] = {
    {6, 2},
};
static arc arcs_1_2[1] = {
    {7, 3},
};
static arc arcs_1_3[1] = {
    {3, 4},
};
static arc arcs_1_4[1] = {
    {0, 4},
};
static state states_1[5] = {
    {1, arcs_1_0},
    {1, arcs_1_1},
    {1, arcs_1_2},
    {1, arcs_1_3},
    {1, arcs_1_4},
};
static arc arcs_2_0[1] = {
    {8, 1},
};
static arc arcs_2_1[2] = {
    {9, 0},
    {0, 1},
};
static state states_2[2] = {
    {1, arcs_2_0},
    {2, arcs_2_1},
};
static arc arcs_3_0[1] = {
    {10, 1},
};
static arc arcs_3_1[2] = {
    {10, 1},
    {0, 1},
};
static state states_3[2] = {
    {1, arcs_3_0},
    {2, arcs_3_1},
};
static arc arcs_4_0[2] = {
    {11, 1},
    {13, 2},
};
static arc arcs_4_1[1] = {
    {7, 3},
};
static arc arcs_4_2[3] = {
    {14, 4},
    {15, 4},
    {0, 2},
};
static arc arcs_4_3[1] = {
    {12, 4},
};
static arc arcs_4_4[1] = {
    {0, 4},
};
static state states_4[5] = {
    {2, arcs_4_0},
    {1, arcs_4_1},
    {3, arcs_4_2},
    {1, arcs_4_3},
    {1, arcs_4_4},
};
static arc arcs_5_0[3] = {
    {5, 1},
    {16, 1},
    {17, 2},
};
static arc arcs_5_1[1] = {
    {0, 1},
};
static arc arcs_5_2[1] = {
    {7, 3},
};
static arc arcs_5_3[1] = {
    {18, 1},
};
static state states_5[4] = {
    {3, arcs_5_0},
    {1, arcs_5_1},
    {1, arcs_5_2},
    {1, arcs_5_3},
};
static dfa dfas[6] = {
    {256, "MSTART", 0, 2, states_0,
     "\070\000\000"},
    {257, "RULE", 0, 5, states_1,
     "\040\000\000"},
    {258, "RHS", 0, 2, states_2,
     "\040\010\003"},
    {259, "ALT", 0, 2, states_3,
     "\040\010\003"},
    {260, "ITEM", 0, 5, states_4,
     "\040\010\003"},
    {261, "ATOM", 0, 4, states_5,
     "\040\000\003"},
};
static label labels[19] = {
    {0, "EMPTY"},
    {256, 0},
    {257, 0},
    {4, 0},
    {0, 0},
    {1, 0},
    {11, 0},
    {258, 0},
    {259, 0},
    {18, 0},
    {260, 0},
    {9, 0},
    {10, 0},
    {261, 0},
    {16, 0},
    {14, 0},
    {3, 0},
    {7, 0},
    {8, 0},
};
static grammar _Ta3Parser_Grammar = {
    6,
    dfas,
    {19, labels},
    256
};

grammar *
meta_grammar(void)
{
    return &_Ta3Parser_Grammar;
}

grammar *
Py_meta_grammar(void)
{
  return meta_grammar();
}
#include "Python.h"

/* snprintf() wrappers.  If the platform has vsnprintf, we use it, else we
   emulate it in a half-hearted way.  Even if the platform has it, we wrap
   it because platforms differ in what vsnprintf does in case the buffer
   is too small:  C99 behavior is to return the number of characters that
   would have been written had the buffer not been too small, and to set
   the last byte of the buffer to \0.  At least MS _vsnprintf returns a
   negative value instead, and fills the entire buffer with non-\0 data.

   The wrappers ensure that str[size-1] is always \0 upon return.

   PyOS_snprintf and PyOS_vsnprintf never write more than size bytes
   (including the trailing '\0') into str.

   If the platform doesn't have vsnprintf, and the buffer size needed to
   avoid truncation exceeds size by more than 512, Python aborts with a
   Py_FatalError.

   Return value (rv):

    When 0 <= rv < size, the output conversion was unexceptional, and
    rv characters were written to str (excluding a trailing \0 byte at
    str[rv]).

    When rv >= size, output conversion was truncated, and a buffer of
    size rv+1 would have been needed to avoid truncation.  str[size-1]
    is \0 in this case.

    When rv < 0, "something bad happened".  str[size-1] is \0 in this
    case too, but the rest of str is unreliable.  It could be that
    an error in format codes was detected by libc, or on platforms
    with a non-C99 vsnprintf simply that the buffer wasn't big enough
    to avoid truncation, or on platforms without any vsnprintf that
    PyMem_Malloc couldn't obtain space for a temp buffer.

   CAUTION:  Unlike C99, str != NULL and size > 0 are required.
*/

int
PyOS_snprintf(char *str, size_t size, const  char  *format, ...)
{
    int rc;
    va_list va;

    va_start(va, format);
    rc = PyOS_vsnprintf(str, size, format, va);
    va_end(va);
    return rc;
}

int
PyOS_vsnprintf(char *str, size_t size, const char  *format, va_list va)
{
    int len;  /* # bytes written, excluding \0 */
#ifdef HAVE_SNPRINTF
#define _PyOS_vsnprintf_EXTRA_SPACE 1
#else
#define _PyOS_vsnprintf_EXTRA_SPACE 512
    char *buffer;
#endif
    assert(str != NULL);
    assert(size > 0);
    assert(format != NULL);
    /* We take a size_t as input but return an int.  Sanity check
     * our input so that it won't cause an overflow in the
     * vsnprintf return value or the buffer malloc size.  */
    if (size > INT_MAX - _PyOS_vsnprintf_EXTRA_SPACE) {
        len = -666;
        goto Done;
    }

#ifdef HAVE_SNPRINTF
    len = vsnprintf(str, size, format, va);
#else
    /* Emulate it. */
    buffer = PyMem_MALLOC(size + _PyOS_vsnprintf_EXTRA_SPACE);
    if (buffer == NULL) {
        len = -666;
        goto Done;
    }

    len = vsprintf(buffer, format, va);
    if (len < 0)
        /* ignore the error */;

    else if ((size_t)len >= size + _PyOS_vsnprintf_EXTRA_SPACE)
        Py_FatalError("Buffer overflow in PyOS_snprintf/PyOS_vsnprintf");

    else {
        const size_t to_copy = (size_t)len < size ?
                                (size_t)len : size - 1;
        assert(to_copy < size);
        memcpy(str, buffer, to_copy);
        str[to_copy] = '\0';
    }
    PyMem_FREE(buffer);
#endif
Done:
    if (size > 0)
        str[size-1] = '\0';
    return len;
#undef _PyOS_vsnprintf_EXTRA_SPACE
}
#include "Python.h"

#include <stdbool.h>


/* Defined in tracemalloc.c */
extern void _PyMem_DumpTraceback(int fd, const void *ptr);


/* Python's malloc wrappers (see pymem.h) */

#undef  uint
#define uint    unsigned int    /* assuming >= 16 bits */

/* Forward declaration */
static void* _PyMem_DebugRawMalloc(void *ctx, size_t size);
static void* _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize);
static void* _PyMem_DebugRawRealloc(void *ctx, void *ptr, size_t size);
static void _PyMem_DebugRawFree(void *ctx, void *p);

static void* _PyMem_DebugMalloc(void *ctx, size_t size);
static void* _PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize);
static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
static void _PyMem_DebugFree(void *ctx, void *p);

static void _PyObject_DebugDumpAddress(const void *p);
static void _PyMem_DebugCheckAddress(char api_id, const void *p);

#if defined(__has_feature)  /* Clang */
 #if __has_feature(address_sanitizer)  /* is ASAN enabled? */
  #define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS \
        __attribute__((no_address_safety_analysis))
 #else
  #define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS
 #endif
#else
 #if defined(__SANITIZE_ADDRESS__)  /* GCC 4.8.x, is ASAN enabled? */
  #define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS \
        __attribute__((no_address_safety_analysis))
 #else
  #define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS
 #endif
#endif

#ifdef WITH_PYMALLOC

#ifdef MS_WINDOWS
#  include <windows.h>
#elif defined(HAVE_MMAP)
#  include <sys/mman.h>
#  ifdef MAP_ANONYMOUS
#    define ARENAS_USE_MMAP
#  endif
#endif

/* Forward declaration */
static void* _PyObject_Malloc(void *ctx, size_t size);
static void* _PyObject_Calloc(void *ctx, size_t nelem, size_t elsize);
static void _PyObject_Free(void *ctx, void *p);
static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
#endif


static void *
_PyMem_RawMalloc(void *ctx, size_t size)
{
    /* PyMem_RawMalloc(0) means malloc(1). Some systems would return NULL
       for malloc(0), which would be treated as an error. Some platforms would
       return a pointer with no memory behind it, which would break pymalloc.
       To solve these problems, allocate an extra byte. */
    if (size == 0)
        size = 1;
    return malloc(size);
}

static void *
_PyMem_RawCalloc(void *ctx, size_t nelem, size_t elsize)
{
    /* PyMem_RawCalloc(0, 0) means calloc(1, 1). Some systems would return NULL
       for calloc(0, 0), which would be treated as an error. Some platforms
       would return a pointer with no memory behind it, which would break
       pymalloc.  To solve these problems, allocate an extra byte. */
    if (nelem == 0 || elsize == 0) {
        nelem = 1;
        elsize = 1;
    }
    return calloc(nelem, elsize);
}

static void *
_PyMem_RawRealloc(void *ctx, void *ptr, size_t size)
{
    if (size == 0)
        size = 1;
    return realloc(ptr, size);
}

static void
_PyMem_RawFree(void *ctx, void *ptr)
{
    free(ptr);
}


#ifdef MS_WINDOWS
static void *
_PyObject_ArenaVirtualAlloc(void *ctx, size_t size)
{
    return VirtualAlloc(NULL, size,
                        MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
}

static void
_PyObject_ArenaVirtualFree(void *ctx, void *ptr, size_t size)
{
    VirtualFree(ptr, 0, MEM_RELEASE);
}

#elif defined(ARENAS_USE_MMAP)
static void *
_PyObject_ArenaMmap(void *ctx, size_t size)
{
    void *ptr;
    ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
               MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
    if (ptr == MAP_FAILED)
        return NULL;
    assert(ptr != NULL);
    return ptr;
}

static void
_PyObject_ArenaMunmap(void *ctx, void *ptr, size_t size)
{
    munmap(ptr, size);
}

#else
static void *
_PyObject_ArenaMalloc(void *ctx, size_t size)
{
    return malloc(size);
}

static void
_PyObject_ArenaFree(void *ctx, void *ptr, size_t size)
{
    free(ptr);
}
#endif


#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawCalloc, _PyMem_RawRealloc, _PyMem_RawFree
#ifdef WITH_PYMALLOC
#  define PYOBJ_FUNCS _PyObject_Malloc, _PyObject_Calloc, _PyObject_Realloc, _PyObject_Free
#else
#  define PYOBJ_FUNCS PYRAW_FUNCS
#endif
#define PYMEM_FUNCS PYOBJ_FUNCS

typedef struct {
    /* We tag each block with an API ID in order to tag API violations */
    char api_id;
    PyMemAllocatorEx alloc;
} debug_alloc_api_t;
static struct {
    debug_alloc_api_t raw;
    debug_alloc_api_t mem;
    debug_alloc_api_t obj;
} _PyMem_Debug = {
    {'r', {NULL, PYRAW_FUNCS}},
    {'m', {NULL, PYMEM_FUNCS}},
    {'o', {NULL, PYOBJ_FUNCS}}
    };

#define PYRAWDBG_FUNCS \
    _PyMem_DebugRawMalloc, _PyMem_DebugRawCalloc, _PyMem_DebugRawRealloc, _PyMem_DebugRawFree
#define PYDBG_FUNCS \
    _PyMem_DebugMalloc, _PyMem_DebugCalloc, _PyMem_DebugRealloc, _PyMem_DebugFree

static PyMemAllocatorEx _PyMem_Raw = {
#ifdef Py_DEBUG
    &_PyMem_Debug.raw, PYRAWDBG_FUNCS
#else
    NULL, PYRAW_FUNCS
#endif
    };

static PyMemAllocatorEx _PyMem = {
#ifdef Py_DEBUG
    &_PyMem_Debug.mem, PYDBG_FUNCS
#else
    NULL, PYMEM_FUNCS
#endif
    };

static PyMemAllocatorEx _PyObject = {
#ifdef Py_DEBUG
    &_PyMem_Debug.obj, PYDBG_FUNCS
#else
    NULL, PYOBJ_FUNCS
#endif
    };

int
_PyMem_SetupAllocators(const char *opt)
{
    if (opt == NULL || *opt == '\0') {
        /* PYTHONMALLOC is empty or is not set or ignored (-E/-I command line
           options): use default allocators */
#ifdef Py_DEBUG
#  ifdef WITH_PYMALLOC
        opt = "pymalloc_debug";
#  else
        opt = "malloc_debug";
#  endif
#else
   /* !Py_DEBUG */
#  ifdef WITH_PYMALLOC
        opt = "pymalloc";
#  else
        opt = "malloc";
#  endif
#endif
    }

    if (strcmp(opt, "debug") == 0) {
        PyMem_SetupDebugHooks();
    }
    else if (strcmp(opt, "malloc") == 0 || strcmp(opt, "malloc_debug") == 0)
    {
        PyMemAllocatorEx alloc = {NULL, PYRAW_FUNCS};

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);

        if (strcmp(opt, "malloc_debug") == 0)
            PyMem_SetupDebugHooks();
    }
#ifdef WITH_PYMALLOC
    else if (strcmp(opt, "pymalloc") == 0
             || strcmp(opt, "pymalloc_debug") == 0)
    {
        PyMemAllocatorEx raw_alloc = {NULL, PYRAW_FUNCS};
        PyMemAllocatorEx mem_alloc = {NULL, PYMEM_FUNCS};
        PyMemAllocatorEx obj_alloc = {NULL, PYOBJ_FUNCS};

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &raw_alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &mem_alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &obj_alloc);

        if (strcmp(opt, "pymalloc_debug") == 0)
            PyMem_SetupDebugHooks();
    }
#endif
    else {
        /* unknown allocator */
        return -1;
    }
    return 0;
}

#undef PYRAW_FUNCS
#undef PYMEM_FUNCS
#undef PYOBJ_FUNCS
#undef PYRAWDBG_FUNCS
#undef PYDBG_FUNCS

static PyObjectArenaAllocator _PyObject_Arena = {NULL,
#ifdef MS_WINDOWS
    _PyObject_ArenaVirtualAlloc, _PyObject_ArenaVirtualFree
#elif defined(ARENAS_USE_MMAP)
    _PyObject_ArenaMmap, _PyObject_ArenaMunmap
#else
    _PyObject_ArenaMalloc, _PyObject_ArenaFree
#endif
    };

#ifdef WITH_PYMALLOC
static int
_PyMem_DebugEnabled(void)
{
    return (_PyObject.malloc == _PyMem_DebugMalloc);
}

int
_PyMem_PymallocEnabled(void)
{
    if (_PyMem_DebugEnabled()) {
        return (_PyMem_Debug.obj.alloc.malloc == _PyObject_Malloc);
    }
    else {
        return (_PyObject.malloc == _PyObject_Malloc);
    }
}
#endif

void
PyMem_SetupDebugHooks(void)
{
    PyMemAllocatorEx alloc;

    alloc.malloc = _PyMem_DebugRawMalloc;
    alloc.calloc = _PyMem_DebugRawCalloc;
    alloc.realloc = _PyMem_DebugRawRealloc;
    alloc.free = _PyMem_DebugRawFree;

    if (_PyMem_Raw.malloc != _PyMem_DebugRawMalloc) {
        alloc.ctx = &_PyMem_Debug.raw;
        PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &_PyMem_Debug.raw.alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
    }

    alloc.malloc = _PyMem_DebugMalloc;
    alloc.calloc = _PyMem_DebugCalloc;
    alloc.realloc = _PyMem_DebugRealloc;
    alloc.free = _PyMem_DebugFree;

    if (_PyMem.malloc != _PyMem_DebugMalloc) {
        alloc.ctx = &_PyMem_Debug.mem;
        PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &_PyMem_Debug.mem.alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
    }

    if (_PyObject.malloc != _PyMem_DebugMalloc) {
        alloc.ctx = &_PyMem_Debug.obj;
        PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &_PyMem_Debug.obj.alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);
    }
}

void
PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
{
    switch(domain)
    {
    case PYMEM_DOMAIN_RAW: *allocator = _PyMem_Raw; break;
    case PYMEM_DOMAIN_MEM: *allocator = _PyMem; break;
    case PYMEM_DOMAIN_OBJ: *allocator = _PyObject; break;
    default:
        /* unknown domain: set all attributes to NULL */
        allocator->ctx = NULL;
        allocator->malloc = NULL;
        allocator->calloc = NULL;
        allocator->realloc = NULL;
        allocator->free = NULL;
    }
}

void
PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
{
    switch(domain)
    {
    case PYMEM_DOMAIN_RAW: _PyMem_Raw = *allocator; break;
    case PYMEM_DOMAIN_MEM: _PyMem = *allocator; break;
    case PYMEM_DOMAIN_OBJ: _PyObject = *allocator; break;
    /* ignore unknown domain */
    }
}

void
PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
{
    *allocator = _PyObject_Arena;
}

void
PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
{
    _PyObject_Arena = *allocator;
}

void *
PyMem_RawMalloc(size_t size)
{
    /*
     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
     * Most python internals blindly use a signed Py_ssize_t to track
     * things without checking for overflows or negatives.
     * As size_t is unsigned, checking for size < 0 is not required.
     */
    if (size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
}

void *
PyMem_RawCalloc(size_t nelem, size_t elsize)
{
    /* see PyMem_RawMalloc() */
    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
        return NULL;
    return _PyMem_Raw.calloc(_PyMem_Raw.ctx, nelem, elsize);
}

void*
PyMem_RawRealloc(void *ptr, size_t new_size)
{
    /* see PyMem_RawMalloc() */
    if (new_size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem_Raw.realloc(_PyMem_Raw.ctx, ptr, new_size);
}

void PyMem_RawFree(void *ptr)
{
    _PyMem_Raw.free(_PyMem_Raw.ctx, ptr);
}

void *
PyMem_Malloc(size_t size)
{
    /* see PyMem_RawMalloc() */
    if (size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem.malloc(_PyMem.ctx, size);
}

void *
PyMem_Calloc(size_t nelem, size_t elsize)
{
    /* see PyMem_RawMalloc() */
    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
        return NULL;
    return _PyMem.calloc(_PyMem.ctx, nelem, elsize);
}

void *
PyMem_Realloc(void *ptr, size_t new_size)
{
    /* see PyMem_RawMalloc() */
    if (new_size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem.realloc(_PyMem.ctx, ptr, new_size);
}

void
PyMem_Free(void *ptr)
{
    _PyMem.free(_PyMem.ctx, ptr);
}

char *
_PyMem_RawStrdup(const char *str)
{
    size_t size;
    char *copy;

    size = strlen(str) + 1;
    copy = PyMem_RawMalloc(size);
    if (copy == NULL)
        return NULL;
    memcpy(copy, str, size);
    return copy;
}

char *
_PyMem_Strdup(const char *str)
{
    size_t size;
    char *copy;

    size = strlen(str) + 1;
    copy = PyMem_Malloc(size);
    if (copy == NULL)
        return NULL;
    memcpy(copy, str, size);
    return copy;
}

void *
PyObject_Malloc(size_t size)
{
    /* see PyMem_RawMalloc() */
    if (size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyObject.malloc(_PyObject.ctx, size);
}

void *
PyObject_Calloc(size_t nelem, size_t elsize)
{
    /* see PyMem_RawMalloc() */
    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
        return NULL;
    return _PyObject.calloc(_PyObject.ctx, nelem, elsize);
}

void *
PyObject_Realloc(void *ptr, size_t new_size)
{
    /* see PyMem_RawMalloc() */
    if (new_size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyObject.realloc(_PyObject.ctx, ptr, new_size);
}

void
PyObject_Free(void *ptr)
{
    _PyObject.free(_PyObject.ctx, ptr);
}


#ifdef WITH_PYMALLOC

#ifdef WITH_VALGRIND
#include <valgrind/valgrind.h>

/* If we're using GCC, use __builtin_expect() to reduce overhead of
   the valgrind checks */
#if defined(__GNUC__) && (__GNUC__ > 2) && defined(__OPTIMIZE__)
#  define UNLIKELY(value) __builtin_expect((value), 0)
#else
#  define UNLIKELY(value) (value)
#endif

/* -1 indicates that we haven't checked that we're running on valgrind yet. */
static int running_on_valgrind = -1;
#endif

/* An object allocator for Python.

   Here is an introduction to the layers of the Python memory architecture,
   showing where the object allocator is actually used (layer +2), It is
   called for every object allocation and deallocation (PyObject_New/Del),
   unless the object-specific allocators implement a proprietary allocation
   scheme (ex.: ints use a simple free list). This is also the place where
   the cyclic garbage collector operates selectively on container objects.


    Object-specific allocators
    _____   ______   ______       ________
   [ int ] [ dict ] [ list ] ... [ string ]       Python core         |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
    _______________________________       |                           |
   [   Python's object allocator   ]      |                           |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
    ______________________________________________________________    |
   [          Python's raw memory allocator (PyMem_ API)          ]   |
+1 | <----- Python memory (under PyMem manager's control) ------> |   |
    __________________________________________________________________
   [    Underlying general-purpose allocator (ex: C library malloc)   ]
 0 | <------ Virtual memory allocated for the python process -------> |

   =========================================================================
    _______________________________________________________________________
   [                OS-specific Virtual Memory Manager (VMM)               ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
    __________________________________   __________________________________
   [                                  ] [                                  ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |

*/
/*==========================================================================*/

/* A fast, special-purpose memory allocator for small blocks, to be used
   on top of a general-purpose malloc -- heavily based on previous art. */

/* Vladimir Marangozov -- August 2000 */

/*
 * "Memory management is where the rubber meets the road -- if we do the wrong
 * thing at any level, the results will not be good. And if we don't make the
 * levels work well together, we are in serious trouble." (1)
 *
 * (1) Paul R. Wilson, Mark S. Johnstone, Michael Neely, and David Boles,
 *    "Dynamic Storage Allocation: A Survey and Critical Review",
 *    in Proc. 1995 Int'l. Workshop on Memory Management, September 1995.
 */

/* #undef WITH_MEMORY_LIMITS */         /* disable mem limit checks  */

/*==========================================================================*/

/*
 * Allocation strategy abstract:
 *
 * For small requests, the allocator sub-allocates <Big> blocks of memory.
 * Requests greater than SMALL_REQUEST_THRESHOLD bytes are routed to the
 * system's allocator.
 *
 * Small requests are grouped in size classes spaced 8 bytes apart, due
 * to the required valid alignment of the returned address. Requests of
 * a particular size are serviced from memory pools of 4K (one VMM page).
 * Pools are fragmented on demand and contain free lists of blocks of one
 * particular size class. In other words, there is a fixed-size allocator
 * for each size class. Free pools are shared by the different allocators
 * thus minimizing the space reserved for a particular size class.
 *
 * This allocation strategy is a variant of what is known as "simple
 * segregated storage based on array of free lists". The main drawback of
 * simple segregated storage is that we might end up with lot of reserved
 * memory for the different free lists, which degenerate in time. To avoid
 * this, we partition each free list in pools and we share dynamically the
 * reserved space between all free lists. This technique is quite efficient
 * for memory intensive programs which allocate mainly small-sized blocks.
 *
 * For small requests we have the following table:
 *
 * Request in bytes     Size of allocated block      Size class idx
 * ----------------------------------------------------------------
 *        1-8                     8                       0
 *        9-16                   16                       1
 *       17-24                   24                       2
 *       25-32                   32                       3
 *       33-40                   40                       4
 *       41-48                   48                       5
 *       49-56                   56                       6
 *       57-64                   64                       7
 *       65-72                   72                       8
 *        ...                   ...                     ...
 *      497-504                 504                      62
 *      505-512                 512                      63
 *
 *      0, SMALL_REQUEST_THRESHOLD + 1 and up: routed to the underlying
 *      allocator.
 */

/*==========================================================================*/

/*
 * -- Main tunable settings section --
 */

/*
 * Alignment of addresses returned to the user. 8-bytes alignment works
 * on most current architectures (with 32-bit or 64-bit address busses).
 * The alignment value is also used for grouping small requests in size
 * classes spaced ALIGNMENT bytes apart.
 *
 * You shouldn't change this unless you know what you are doing.
 */
#define ALIGNMENT               8               /* must be 2^N */
#define ALIGNMENT_SHIFT         3

/* Return the number of bytes in size class I, as a uint. */
#define INDEX2SIZE(I) (((uint)(I) + 1) << ALIGNMENT_SHIFT)

/*
 * Max size threshold below which malloc requests are considered to be
 * small enough in order to use preallocated memory pools. You can tune
 * this value according to your application behaviour and memory needs.
 *
 * Note: a size threshold of 512 guarantees that newly created dictionaries
 * will be allocated from preallocated memory pools on 64-bit.
 *
 * The following invariants must hold:
 *      1) ALIGNMENT <= SMALL_REQUEST_THRESHOLD <= 512
 *      2) SMALL_REQUEST_THRESHOLD is evenly divisible by ALIGNMENT
 *
 * Although not required, for better performance and space efficiency,
 * it is recommended that SMALL_REQUEST_THRESHOLD is set to a power of 2.
 */
#define SMALL_REQUEST_THRESHOLD 512
#define NB_SMALL_SIZE_CLASSES   (SMALL_REQUEST_THRESHOLD / ALIGNMENT)

/*
 * The system's VMM page size can be obtained on most unices with a
 * getpagesize() call or deduced from various header files. To make
 * things simpler, we assume that it is 4K, which is OK for most systems.
 * It is probably better if this is the native page size, but it doesn't
 * have to be.  In theory, if SYSTEM_PAGE_SIZE is larger than the native page
 * size, then `POOL_ADDR(p)->arenaindex' could rarely cause a segmentation
 * violation fault.  4K is apparently OK for all the platforms that python
 * currently targets.
 */
#define SYSTEM_PAGE_SIZE        (4 * 1024)
#define SYSTEM_PAGE_SIZE_MASK   (SYSTEM_PAGE_SIZE - 1)

/*
 * Maximum amount of memory managed by the allocator for small requests.
 */
#ifdef WITH_MEMORY_LIMITS
#ifndef SMALL_MEMORY_LIMIT
#define SMALL_MEMORY_LIMIT      (64 * 1024 * 1024)      /* 64 MB -- more? */
#endif
#endif

/*
 * The allocator sub-allocates <Big> blocks of memory (called arenas) aligned
 * on a page boundary. This is a reserved virtual address space for the
 * current process (obtained through a malloc()/mmap() call). In no way this
 * means that the memory arenas will be used entirely. A malloc(<Big>) is
 * usually an address range reservation for <Big> bytes, unless all pages within
 * this space are referenced subsequently. So malloc'ing big blocks and not
 * using them does not mean "wasting memory". It's an addressable range
 * wastage...
 *
 * Arenas are allocated with mmap() on systems supporting anonymous memory
 * mappings to reduce heap fragmentation.
 */
#define ARENA_SIZE              (256 << 10)     /* 256KB */

#ifdef WITH_MEMORY_LIMITS
#define MAX_ARENAS              (SMALL_MEMORY_LIMIT / ARENA_SIZE)
#endif

/*
 * Size of the pools used for small blocks. Should be a power of 2,
 * between 1K and SYSTEM_PAGE_SIZE, that is: 1k, 2k, 4k.
 */
#define POOL_SIZE               SYSTEM_PAGE_SIZE        /* must be 2^N */
#define POOL_SIZE_MASK          SYSTEM_PAGE_SIZE_MASK

/*
 * -- End of tunable settings section --
 */

/*==========================================================================*/

/*
 * Locking
 *
 * To reduce lock contention, it would probably be better to refine the
 * crude function locking with per size class locking. I'm not positive
 * however, whether it's worth switching to such locking policy because
 * of the performance penalty it might introduce.
 *
 * The following macros describe the simplest (should also be the fastest)
 * lock object on a particular platform and the init/fini/lock/unlock
 * operations on it. The locks defined here are not expected to be recursive
 * because it is assumed that they will always be called in the order:
 * INIT, [LOCK, UNLOCK]*, FINI.
 */

/*
 * Python's threads are serialized, so object malloc locking is disabled.
 */
#define SIMPLELOCK_DECL(lock)   /* simple lock declaration              */
#define SIMPLELOCK_INIT(lock)   /* allocate (if needed) and initialize  */
#define SIMPLELOCK_FINI(lock)   /* free/destroy an existing lock        */
#define SIMPLELOCK_LOCK(lock)   /* acquire released lock */
#define SIMPLELOCK_UNLOCK(lock) /* release acquired lock */

/* When you say memory, my mind reasons in terms of (pointers to) blocks */
typedef uint8_t block;

/* Pool for small blocks. */
struct pool_header {
    union { block *_padding;
            uint count; } ref;          /* number of allocated blocks    */
    block *freeblock;                   /* pool's free list head         */
    struct pool_header *nextpool;       /* next pool of this size class  */
    struct pool_header *prevpool;       /* previous pool       ""        */
    uint arenaindex;                    /* index into arenas of base adr */
    uint szidx;                         /* block size class index        */
    uint nextoffset;                    /* bytes to virgin block         */
    uint maxnextoffset;                 /* largest valid nextoffset      */
};

typedef struct pool_header *poolp;

/* Record keeping for arenas. */
struct arena_object {
    /* The address of the arena, as returned by malloc.  Note that 0
     * will never be returned by a successful malloc, and is used
     * here to mark an arena_object that doesn't correspond to an
     * allocated arena.
     */
    uintptr_t address;

    /* Pool-aligned pointer to the next pool to be carved off. */
    block* pool_address;

    /* The number of available pools in the arena:  free pools + never-
     * allocated pools.
     */
    uint nfreepools;

    /* The total number of pools in the arena, whether or not available. */
    uint ntotalpools;

    /* Singly-linked list of available pools. */
    struct pool_header* freepools;

    /* Whenever this arena_object is not associated with an allocated
     * arena, the nextarena member is used to link all unassociated
     * arena_objects in the singly-linked `unused_arena_objects` list.
     * The prevarena member is unused in this case.
     *
     * When this arena_object is associated with an allocated arena
     * with at least one available pool, both members are used in the
     * doubly-linked `usable_arenas` list, which is maintained in
     * increasing order of `nfreepools` values.
     *
     * Else this arena_object is associated with an allocated arena
     * all of whose pools are in use.  `nextarena` and `prevarena`
     * are both meaningless in this case.
     */
    struct arena_object* nextarena;
    struct arena_object* prevarena;
};

#define POOL_OVERHEAD   _Py_SIZE_ROUND_UP(sizeof(struct pool_header), ALIGNMENT)

#define DUMMY_SIZE_IDX          0xffff  /* size class of newly cached pools */

/* Round pointer P down to the closest pool-aligned address <= P, as a poolp */
#define POOL_ADDR(P) ((poolp)_Py_ALIGN_DOWN((P), POOL_SIZE))

/* Return total number of blocks in pool of size index I, as a uint. */
#define NUMBLOCKS(I) ((uint)(POOL_SIZE - POOL_OVERHEAD) / INDEX2SIZE(I))

/*==========================================================================*/

/*
 * This malloc lock
 */
SIMPLELOCK_DECL(_malloc_lock)
#define LOCK()          SIMPLELOCK_LOCK(_malloc_lock)
#define UNLOCK()        SIMPLELOCK_UNLOCK(_malloc_lock)
#define LOCK_INIT()     SIMPLELOCK_INIT(_malloc_lock)
#define LOCK_FINI()     SIMPLELOCK_FINI(_malloc_lock)

/*
 * Pool table -- headed, circular, doubly-linked lists of partially used pools.

This is involved.  For an index i, usedpools[i+i] is the header for a list of
all partially used pools holding small blocks with "size class idx" i. So
usedpools[0] corresponds to blocks of size 8, usedpools[2] to blocks of size
16, and so on:  index 2*i <-> blocks of size (i+1)<<ALIGNMENT_SHIFT.

Pools are carved off an arena's highwater mark (an arena_object's pool_address
member) as needed.  Once carved off, a pool is in one of three states forever
after:

used == partially used, neither empty nor full
    At least one block in the pool is currently allocated, and at least one
    block in the pool is not currently allocated (note this implies a pool
    has room for at least two blocks).
    This is a pool's initial state, as a pool is created only when malloc
    needs space.
    The pool holds blocks of a fixed size, and is in the circular list headed
    at usedpools[i] (see above).  It's linked to the other used pools of the
    same size class via the pool_header's nextpool and prevpool members.
    If all but one block is currently allocated, a malloc can cause a
    transition to the full state.  If all but one block is not currently
    allocated, a free can cause a transition to the empty state.

full == all the pool's blocks are currently allocated
    On transition to full, a pool is unlinked from its usedpools[] list.
    It's not linked to from anything then anymore, and its nextpool and
    prevpool members are meaningless until it transitions back to used.
    A free of a block in a full pool puts the pool back in the used state.
    Then it's linked in at the front of the appropriate usedpools[] list, so
    that the next allocation for its size class will reuse the freed block.

empty == all the pool's blocks are currently available for allocation
    On transition to empty, a pool is unlinked from its usedpools[] list,
    and linked to the front of its arena_object's singly-linked freepools list,
    via its nextpool member.  The prevpool member has no meaning in this case.
    Empty pools have no inherent size class:  the next time a malloc finds
    an empty list in usedpools[], it takes the first pool off of freepools.
    If the size class needed happens to be the same as the size class the pool
    last had, some pool initialization can be skipped.


Block Management

Blocks within pools are again carved out as needed.  pool->freeblock points to
the start of a singly-linked list of free blocks within the pool.  When a
block is freed, it's inserted at the front of its pool's freeblock list.  Note
that the available blocks in a pool are *not* linked all together when a pool
is initialized.  Instead only "the first two" (lowest addresses) blocks are
set up, returning the first such block, and setting pool->freeblock to a
one-block list holding the second such block.  This is consistent with that
pymalloc strives at all levels (arena, pool, and block) never to touch a piece
of memory until it's actually needed.

So long as a pool is in the used state, we're certain there *is* a block
available for allocating, and pool->freeblock is not NULL.  If pool->freeblock
points to the end of the free list before we've carved the entire pool into
blocks, that means we simply haven't yet gotten to one of the higher-address
blocks.  The offset from the pool_header to the start of "the next" virgin
block is stored in the pool_header nextoffset member, and the largest value
of nextoffset that makes sense is stored in the maxnextoffset member when a
pool is initialized.  All the blocks in a pool have been passed out at least
once when and only when nextoffset > maxnextoffset.


Major obscurity:  While the usedpools vector is declared to have poolp
entries, it doesn't really.  It really contains two pointers per (conceptual)
poolp entry, the nextpool and prevpool members of a pool_header.  The
excruciating initialization code below fools C so that

    usedpool[i+i]

"acts like" a genuine poolp, but only so long as you only reference its
nextpool and prevpool members.  The "- 2*sizeof(block *)" gibberish is
compensating for that a pool_header's nextpool and prevpool members
immediately follow a pool_header's first two members:

    union { block *_padding;
            uint count; } ref;
    block *freeblock;

each of which consume sizeof(block *) bytes.  So what usedpools[i+i] really
contains is a fudged-up pointer p such that *if* C believes it's a poolp
pointer, then p->nextpool and p->prevpool are both p (meaning that the headed
circular list is empty).

It's unclear why the usedpools setup is so convoluted.  It could be to
minimize the amount of cache required to hold this heavily-referenced table
(which only *needs* the two interpool pointer members of a pool_header). OTOH,
referencing code has to remember to "double the index" and doing so isn't
free, usedpools[0] isn't a strictly legal pointer, and we're crucially relying
on that C doesn't insert any padding anywhere in a pool_header at or before
the prevpool member.
**************************************************************************** */

#define PTA(x)  ((poolp )((uint8_t *)&(usedpools[2*(x)]) - 2*sizeof(block *)))
#define PT(x)   PTA(x), PTA(x)

static poolp usedpools[2 * ((NB_SMALL_SIZE_CLASSES + 7) / 8) * 8] = {
    PT(0), PT(1), PT(2), PT(3), PT(4), PT(5), PT(6), PT(7)
#if NB_SMALL_SIZE_CLASSES > 8
    , PT(8), PT(9), PT(10), PT(11), PT(12), PT(13), PT(14), PT(15)
#if NB_SMALL_SIZE_CLASSES > 16
    , PT(16), PT(17), PT(18), PT(19), PT(20), PT(21), PT(22), PT(23)
#if NB_SMALL_SIZE_CLASSES > 24
    , PT(24), PT(25), PT(26), PT(27), PT(28), PT(29), PT(30), PT(31)
#if NB_SMALL_SIZE_CLASSES > 32
    , PT(32), PT(33), PT(34), PT(35), PT(36), PT(37), PT(38), PT(39)
#if NB_SMALL_SIZE_CLASSES > 40
    , PT(40), PT(41), PT(42), PT(43), PT(44), PT(45), PT(46), PT(47)
#if NB_SMALL_SIZE_CLASSES > 48
    , PT(48), PT(49), PT(50), PT(51), PT(52), PT(53), PT(54), PT(55)
#if NB_SMALL_SIZE_CLASSES > 56
    , PT(56), PT(57), PT(58), PT(59), PT(60), PT(61), PT(62), PT(63)
#if NB_SMALL_SIZE_CLASSES > 64
#error "NB_SMALL_SIZE_CLASSES should be less than 64"
#endif /* NB_SMALL_SIZE_CLASSES > 64 */
#endif /* NB_SMALL_SIZE_CLASSES > 56 */
#endif /* NB_SMALL_SIZE_CLASSES > 48 */
#endif /* NB_SMALL_SIZE_CLASSES > 40 */
#endif /* NB_SMALL_SIZE_CLASSES > 32 */
#endif /* NB_SMALL_SIZE_CLASSES > 24 */
#endif /* NB_SMALL_SIZE_CLASSES > 16 */
#endif /* NB_SMALL_SIZE_CLASSES >  8 */
};

/*==========================================================================
Arena management.

`arenas` is a vector of arena_objects.  It contains maxarenas entries, some of
which may not be currently used (== they're arena_objects that aren't
currently associated with an allocated arena).  Note that arenas proper are
separately malloc'ed.

Prior to Python 2.5, arenas were never free()'ed.  Starting with Python 2.5,
we do try to free() arenas, and use some mild heuristic strategies to increase
the likelihood that arenas eventually can be freed.

unused_arena_objects

    This is a singly-linked list of the arena_objects that are currently not
    being used (no arena is associated with them).  Objects are taken off the
    head of the list in new_arena(), and are pushed on the head of the list in
    PyObject_Free() when the arena is empty.  Key invariant:  an arena_object
    is on this list if and only if its .address member is 0.

usable_arenas

    This is a doubly-linked list of the arena_objects associated with arenas
    that have pools available.  These pools are either waiting to be reused,
    or have not been used before.  The list is sorted to have the most-
    allocated arenas first (ascending order based on the nfreepools member).
    This means that the next allocation will come from a heavily used arena,
    which gives the nearly empty arenas a chance to be returned to the system.
    In my unscientific tests this dramatically improved the number of arenas
    that could be freed.

Note that an arena_object associated with an arena all of whose pools are
currently in use isn't on either list.
*/

/* Array of objects used to track chunks of memory (arenas). */
static struct arena_object* arenas = NULL;
/* Number of slots currently allocated in the `arenas` vector. */
static uint maxarenas = 0;

/* The head of the singly-linked, NULL-terminated list of available
 * arena_objects.
 */
static struct arena_object* unused_arena_objects = NULL;

/* The head of the doubly-linked, NULL-terminated at each end, list of
 * arena_objects associated with arenas that have pools available.
 */
static struct arena_object* usable_arenas = NULL;

/* How many arena_objects do we initially allocate?
 * 16 = can allocate 16 arenas = 16 * ARENA_SIZE = 4MB before growing the
 * `arenas` vector.
 */
#define INITIAL_ARENA_OBJECTS 16

/* Number of arenas allocated that haven't been free()'d. */
static size_t narenas_currently_allocated = 0;

/* Total number of times malloc() called to allocate an arena. */
static size_t ntimes_arena_allocated = 0;
/* High water mark (max value ever seen) for narenas_currently_allocated. */
static size_t narenas_highwater = 0;

static Py_ssize_t _Py_AllocatedBlocks = 0;

Py_ssize_t
_Py_GetAllocatedBlocks(void)
{
    return _Py_AllocatedBlocks;
}


/* Allocate a new arena.  If we run out of memory, return NULL.  Else
 * allocate a new arena, and return the address of an arena_object
 * describing the new arena.  It's expected that the caller will set
 * `usable_arenas` to the return value.
 */
static struct arena_object*
new_arena(void)
{
    struct arena_object* arenaobj;
    uint excess;        /* number of bytes above pool alignment */
    void *address;
    static int debug_stats = -1;

    if (debug_stats == -1) {
        char *opt = Py_GETENV("PYTHONMALLOCSTATS");
        debug_stats = (opt != NULL && *opt != '\0');
    }
    if (debug_stats)
        _PyObject_DebugMallocStats(stderr);

    if (unused_arena_objects == NULL) {
        uint i;
        uint numarenas;
        size_t nbytes;

        /* Double the number of arena objects on each allocation.
         * Note that it's possible for `numarenas` to overflow.
         */
        numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS;
        if (numarenas <= maxarenas)
            return NULL;                /* overflow */
#if SIZEOF_SIZE_T <= SIZEOF_INT
        if (numarenas > SIZE_MAX / sizeof(*arenas))
            return NULL;                /* overflow */
#endif
        nbytes = numarenas * sizeof(*arenas);
        arenaobj = (struct arena_object *)PyMem_RawRealloc(arenas, nbytes);
        if (arenaobj == NULL)
            return NULL;
        arenas = arenaobj;

        /* We might need to fix pointers that were copied.  However,
         * new_arena only gets called when all the pages in the
         * previous arenas are full.  Thus, there are *no* pointers
         * into the old array. Thus, we don't have to worry about
         * invalid pointers.  Just to be sure, some asserts:
         */
        assert(usable_arenas == NULL);
        assert(unused_arena_objects == NULL);

        /* Put the new arenas on the unused_arena_objects list. */
        for (i = maxarenas; i < numarenas; ++i) {
            arenas[i].address = 0;              /* mark as unassociated */
            arenas[i].nextarena = i < numarenas - 1 ?
                                   &arenas[i+1] : NULL;
        }

        /* Update globals. */
        unused_arena_objects = &arenas[maxarenas];
        maxarenas = numarenas;
    }

    /* Take the next available arena object off the head of the list. */
    assert(unused_arena_objects != NULL);
    arenaobj = unused_arena_objects;
    unused_arena_objects = arenaobj->nextarena;
    assert(arenaobj->address == 0);
    address = _PyObject_Arena.alloc(_PyObject_Arena.ctx, ARENA_SIZE);
    if (address == NULL) {
        /* The allocation failed: return NULL after putting the
         * arenaobj back.
         */
        arenaobj->nextarena = unused_arena_objects;
        unused_arena_objects = arenaobj;
        return NULL;
    }
    arenaobj->address = (uintptr_t)address;

    ++narenas_currently_allocated;
    ++ntimes_arena_allocated;
    if (narenas_currently_allocated > narenas_highwater)
        narenas_highwater = narenas_currently_allocated;
    arenaobj->freepools = NULL;
    /* pool_address <- first pool-aligned address in the arena
       nfreepools <- number of whole pools that fit after alignment */
    arenaobj->pool_address = (block*)arenaobj->address;
    arenaobj->nfreepools = ARENA_SIZE / POOL_SIZE;
    assert(POOL_SIZE * arenaobj->nfreepools == ARENA_SIZE);
    excess = (uint)(arenaobj->address & POOL_SIZE_MASK);
    if (excess != 0) {
        --arenaobj->nfreepools;
        arenaobj->pool_address += POOL_SIZE - excess;
    }
    arenaobj->ntotalpools = arenaobj->nfreepools;

    return arenaobj;
}

/*
address_in_range(P, POOL)

Return true if and only if P is an address that was allocated by pymalloc.
POOL must be the pool address associated with P, i.e., POOL = POOL_ADDR(P)
(the caller is asked to compute this because the macro expands POOL more than
once, and for efficiency it's best for the caller to assign POOL_ADDR(P) to a
variable and pass the latter to the macro; because address_in_range is
called on every alloc/realloc/free, micro-efficiency is important here).

Tricky:  Let B be the arena base address associated with the pool, B =
arenas[(POOL)->arenaindex].address.  Then P belongs to the arena if and only if

    B <= P < B + ARENA_SIZE

Subtracting B throughout, this is true iff

    0 <= P-B < ARENA_SIZE

By using unsigned arithmetic, the "0 <=" half of the test can be skipped.

Obscure:  A PyMem "free memory" function can call the pymalloc free or realloc
before the first arena has been allocated.  `arenas` is still NULL in that
case.  We're relying on that maxarenas is also 0 in that case, so that
(POOL)->arenaindex < maxarenas  must be false, saving us from trying to index
into a NULL arenas.

Details:  given P and POOL, the arena_object corresponding to P is AO =
arenas[(POOL)->arenaindex].  Suppose obmalloc controls P.  Then (barring wild
stores, etc), POOL is the correct address of P's pool, AO.address is the
correct base address of the pool's arena, and P must be within ARENA_SIZE of
AO.address.  In addition, AO.address is not 0 (no arena can start at address 0
(NULL)).  Therefore address_in_range correctly reports that obmalloc
controls P.

Now suppose obmalloc does not control P (e.g., P was obtained via a direct
call to the system malloc() or realloc()).  (POOL)->arenaindex may be anything
in this case -- it may even be uninitialized trash.  If the trash arenaindex
is >= maxarenas, the macro correctly concludes at once that obmalloc doesn't
control P.

Else arenaindex is < maxarena, and AO is read up.  If AO corresponds to an
allocated arena, obmalloc controls all the memory in slice AO.address :
AO.address+ARENA_SIZE.  By case assumption, P is not controlled by obmalloc,
so P doesn't lie in that slice, so the macro correctly reports that P is not
controlled by obmalloc.

Finally, if P is not controlled by obmalloc and AO corresponds to an unused
arena_object (one not currently associated with an allocated arena),
AO.address is 0, and the second test in the macro reduces to:

    P < ARENA_SIZE

If P >= ARENA_SIZE (extremely likely), the macro again correctly concludes
that P is not controlled by obmalloc.  However, if P < ARENA_SIZE, this part
of the test still passes, and the third clause (AO.address != 0) is necessary
to get the correct result:  AO.address is 0 in this case, so the macro
correctly reports that P is not controlled by obmalloc (despite that P lies in
slice AO.address : AO.address + ARENA_SIZE).

Note:  The third (AO.address != 0) clause was added in Python 2.5.  Before
2.5, arenas were never free()'ed, and an arenaindex < maxarena always
corresponded to a currently-allocated arena, so the "P is not controlled by
obmalloc, AO corresponds to an unused arena_object, and P < ARENA_SIZE" case
was impossible.

Note that the logic is excruciating, and reading up possibly uninitialized
memory when P is not controlled by obmalloc (to get at (POOL)->arenaindex)
creates problems for some memory debuggers.  The overwhelming advantage is
that this test determines whether an arbitrary address is controlled by
obmalloc in a small constant time, independent of the number of arenas
obmalloc controls.  Since this test is needed at every entry point, it's
extremely desirable that it be this fast.
*/

static bool ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS
address_in_range(void *p, poolp pool)
{
    // Since address_in_range may be reading from memory which was not allocated
    // by Python, it is important that pool->arenaindex is read only once, as
    // another thread may be concurrently modifying the value without holding
    // the GIL. The following dance forces the compiler to read pool->arenaindex
    // only once.
    uint arenaindex = *((volatile uint *)&pool->arenaindex);
    return arenaindex < maxarenas &&
        (uintptr_t)p - arenas[arenaindex].address < ARENA_SIZE &&
        arenas[arenaindex].address != 0;
}

/*==========================================================================*/

/* malloc.  Note that nbytes==0 tries to return a non-NULL pointer, distinct
 * from all other currently live pointers.  This may not be possible.
 */

/*
 * The basic blocks are ordered by decreasing execution frequency,
 * which minimizes the number of jumps in the most common cases,
 * improves branching prediction and instruction scheduling (small
 * block allocations typically result in a couple of instructions).
 * Unless the optimizer reorders everything, being too smart...
 */

static void *
_PyObject_Alloc(int use_calloc, void *ctx, size_t nelem, size_t elsize)
{
    size_t nbytes;
    block *bp;
    poolp pool;
    poolp next;
    uint size;

    _Py_AllocatedBlocks++;

    assert(nelem <= PY_SSIZE_T_MAX / elsize);
    nbytes = nelem * elsize;

#ifdef WITH_VALGRIND
    if (UNLIKELY(running_on_valgrind == -1))
        running_on_valgrind = RUNNING_ON_VALGRIND;
    if (UNLIKELY(running_on_valgrind))
        goto redirect;
#endif

    if (nelem == 0 || elsize == 0)
        goto redirect;

    if ((nbytes - 1) < SMALL_REQUEST_THRESHOLD) {
        LOCK();
        /*
         * Most frequent paths first
         */
        size = (uint)(nbytes - 1) >> ALIGNMENT_SHIFT;
        pool = usedpools[size + size];
        if (pool != pool->nextpool) {
            /*
             * There is a used pool for this size class.
             * Pick up the head block of its free list.
             */
            ++pool->ref.count;
            bp = pool->freeblock;
            assert(bp != NULL);
            if ((pool->freeblock = *(block **)bp) != NULL) {
                UNLOCK();
                if (use_calloc)
                    memset(bp, 0, nbytes);
                return (void *)bp;
            }
            /*
             * Reached the end of the free list, try to extend it.
             */
            if (pool->nextoffset <= pool->maxnextoffset) {
                /* There is room for another block. */
                pool->freeblock = (block*)pool +
                                  pool->nextoffset;
                pool->nextoffset += INDEX2SIZE(size);
                *(block **)(pool->freeblock) = NULL;
                UNLOCK();
                if (use_calloc)
                    memset(bp, 0, nbytes);
                return (void *)bp;
            }
            /* Pool is full, unlink from used pools. */
            next = pool->nextpool;
            pool = pool->prevpool;
            next->prevpool = pool;
            pool->nextpool = next;
            UNLOCK();
            if (use_calloc)
                memset(bp, 0, nbytes);
            return (void *)bp;
        }

        /* There isn't a pool of the right size class immediately
         * available:  use a free pool.
         */
        if (usable_arenas == NULL) {
            /* No arena has a free pool:  allocate a new arena. */
#ifdef WITH_MEMORY_LIMITS
            if (narenas_currently_allocated >= MAX_ARENAS) {
                UNLOCK();
                goto redirect;
            }
#endif
            usable_arenas = new_arena();
            if (usable_arenas == NULL) {
                UNLOCK();
                goto redirect;
            }
            usable_arenas->nextarena =
                usable_arenas->prevarena = NULL;
        }
        assert(usable_arenas->address != 0);

        /* Try to get a cached free pool. */
        pool = usable_arenas->freepools;
        if (pool != NULL) {
            /* Unlink from cached pools. */
            usable_arenas->freepools = pool->nextpool;

            /* This arena already had the smallest nfreepools
             * value, so decreasing nfreepools doesn't change
             * that, and we don't need to rearrange the
             * usable_arenas list.  However, if the arena has
             * become wholly allocated, we need to remove its
             * arena_object from usable_arenas.
             */
            --usable_arenas->nfreepools;
            if (usable_arenas->nfreepools == 0) {
                /* Wholly allocated:  remove. */
                assert(usable_arenas->freepools == NULL);
                assert(usable_arenas->nextarena == NULL ||
                       usable_arenas->nextarena->prevarena ==
                       usable_arenas);

                usable_arenas = usable_arenas->nextarena;
                if (usable_arenas != NULL) {
                    usable_arenas->prevarena = NULL;
                    assert(usable_arenas->address != 0);
                }
            }
            else {
                /* nfreepools > 0:  it must be that freepools
                 * isn't NULL, or that we haven't yet carved
                 * off all the arena's pools for the first
                 * time.
                 */
                assert(usable_arenas->freepools != NULL ||
                       usable_arenas->pool_address <=
                       (block*)usable_arenas->address +
                           ARENA_SIZE - POOL_SIZE);
            }
        init_pool:
            /* Frontlink to used pools. */
            next = usedpools[size + size]; /* == prev */
            pool->nextpool = next;
            pool->prevpool = next;
            next->nextpool = pool;
            next->prevpool = pool;
            pool->ref.count = 1;
            if (pool->szidx == size) {
                /* Luckily, this pool last contained blocks
                 * of the same size class, so its header
                 * and free list are already initialized.
                 */
                bp = pool->freeblock;
                assert(bp != NULL);
                pool->freeblock = *(block **)bp;
                UNLOCK();
                if (use_calloc)
                    memset(bp, 0, nbytes);
                return (void *)bp;
            }
            /*
             * Initialize the pool header, set up the free list to
             * contain just the second block, and return the first
             * block.
             */
            pool->szidx = size;
            size = INDEX2SIZE(size);
            bp = (block *)pool + POOL_OVERHEAD;
            pool->nextoffset = POOL_OVERHEAD + (size << 1);
            pool->maxnextoffset = POOL_SIZE - size;
            pool->freeblock = bp + size;
            *(block **)(pool->freeblock) = NULL;
            UNLOCK();
            if (use_calloc)
                memset(bp, 0, nbytes);
            return (void *)bp;
        }

        /* Carve off a new pool. */
        assert(usable_arenas->nfreepools > 0);
        assert(usable_arenas->freepools == NULL);
        pool = (poolp)usable_arenas->pool_address;
        assert((block*)pool <= (block*)usable_arenas->address +
                               ARENA_SIZE - POOL_SIZE);
        pool->arenaindex = (uint)(usable_arenas - arenas);
        assert(&arenas[pool->arenaindex] == usable_arenas);
        pool->szidx = DUMMY_SIZE_IDX;
        usable_arenas->pool_address += POOL_SIZE;
        --usable_arenas->nfreepools;

        if (usable_arenas->nfreepools == 0) {
            assert(usable_arenas->nextarena == NULL ||
                   usable_arenas->nextarena->prevarena ==
                   usable_arenas);
            /* Unlink the arena:  it is completely allocated. */
            usable_arenas = usable_arenas->nextarena;
            if (usable_arenas != NULL) {
                usable_arenas->prevarena = NULL;
                assert(usable_arenas->address != 0);
            }
        }

        goto init_pool;
    }

    /* The small block allocator ends here. */

redirect:
    /* Redirect the original request to the underlying (libc) allocator.
     * We jump here on bigger requests, on error in the code above (as a
     * last chance to serve the request) or when the max memory limit
     * has been reached.
     */
    {
        void *result;
        if (use_calloc)
            result = PyMem_RawCalloc(nelem, elsize);
        else
            result = PyMem_RawMalloc(nbytes);
        if (!result)
            _Py_AllocatedBlocks--;
        return result;
    }
}

static void *
_PyObject_Malloc(void *ctx, size_t nbytes)
{
    return _PyObject_Alloc(0, ctx, 1, nbytes);
}

static void *
_PyObject_Calloc(void *ctx, size_t nelem, size_t elsize)
{
    return _PyObject_Alloc(1, ctx, nelem, elsize);
}

/* free */

static void
_PyObject_Free(void *ctx, void *p)
{
    poolp pool;
    block *lastfree;
    poolp next, prev;
    uint size;

    if (p == NULL)      /* free(NULL) has no effect */
        return;

    _Py_AllocatedBlocks--;

#ifdef WITH_VALGRIND
    if (UNLIKELY(running_on_valgrind > 0))
        goto redirect;
#endif

    pool = POOL_ADDR(p);
    if (address_in_range(p, pool)) {
        /* We allocated this address. */
        LOCK();
        /* Link p to the start of the pool's freeblock list.  Since
         * the pool had at least the p block outstanding, the pool
         * wasn't empty (so it's already in a usedpools[] list, or
         * was full and is in no list -- it's not in the freeblocks
         * list in any case).
         */
        assert(pool->ref.count > 0);            /* else it was empty */
        *(block **)p = lastfree = pool->freeblock;
        pool->freeblock = (block *)p;
        if (lastfree) {
            struct arena_object* ao;
            uint nf;  /* ao->nfreepools */

            /* freeblock wasn't NULL, so the pool wasn't full,
             * and the pool is in a usedpools[] list.
             */
            if (--pool->ref.count != 0) {
                /* pool isn't empty:  leave it in usedpools */
                UNLOCK();
                return;
            }
            /* Pool is now empty:  unlink from usedpools, and
             * link to the front of freepools.  This ensures that
             * previously freed pools will be allocated later
             * (being not referenced, they are perhaps paged out).
             */
            next = pool->nextpool;
            prev = pool->prevpool;
            next->prevpool = prev;
            prev->nextpool = next;

            /* Link the pool to freepools.  This is a singly-linked
             * list, and pool->prevpool isn't used there.
             */
            ao = &arenas[pool->arenaindex];
            pool->nextpool = ao->freepools;
            ao->freepools = pool;
            nf = ++ao->nfreepools;

            /* All the rest is arena management.  We just freed
             * a pool, and there are 4 cases for arena mgmt:
             * 1. If all the pools are free, return the arena to
             *    the system free().
             * 2. If this is the only free pool in the arena,
             *    add the arena back to the `usable_arenas` list.
             * 3. If the "next" arena has a smaller count of free
             *    pools, we have to "slide this arena right" to
             *    restore that usable_arenas is sorted in order of
             *    nfreepools.
             * 4. Else there's nothing more to do.
             */
            if (nf == ao->ntotalpools) {
                /* Case 1.  First unlink ao from usable_arenas.
                 */
                assert(ao->prevarena == NULL ||
                       ao->prevarena->address != 0);
                assert(ao ->nextarena == NULL ||
                       ao->nextarena->address != 0);

                /* Fix the pointer in the prevarena, or the
                 * usable_arenas pointer.
                 */
                if (ao->prevarena == NULL) {
                    usable_arenas = ao->nextarena;
                    assert(usable_arenas == NULL ||
                           usable_arenas->address != 0);
                }
                else {
                    assert(ao->prevarena->nextarena == ao);
                    ao->prevarena->nextarena =
                        ao->nextarena;
                }
                /* Fix the pointer in the nextarena. */
                if (ao->nextarena != NULL) {
                    assert(ao->nextarena->prevarena == ao);
                    ao->nextarena->prevarena =
                        ao->prevarena;
                }
                /* Record that this arena_object slot is
                 * available to be reused.
                 */
                ao->nextarena = unused_arena_objects;
                unused_arena_objects = ao;

                /* Free the entire arena. */
                _PyObject_Arena.free(_PyObject_Arena.ctx,
                                     (void *)ao->address, ARENA_SIZE);
                ao->address = 0;                        /* mark unassociated */
                --narenas_currently_allocated;

                UNLOCK();
                return;
            }
            if (nf == 1) {
                /* Case 2.  Put ao at the head of
                 * usable_arenas.  Note that because
                 * ao->nfreepools was 0 before, ao isn't
                 * currently on the usable_arenas list.
                 */
                ao->nextarena = usable_arenas;
                ao->prevarena = NULL;
                if (usable_arenas)
                    usable_arenas->prevarena = ao;
                usable_arenas = ao;
                assert(usable_arenas->address != 0);

                UNLOCK();
                return;
            }
            /* If this arena is now out of order, we need to keep
             * the list sorted.  The list is kept sorted so that
             * the "most full" arenas are used first, which allows
             * the nearly empty arenas to be completely freed.  In
             * a few un-scientific tests, it seems like this
             * approach allowed a lot more memory to be freed.
             */
            if (ao->nextarena == NULL ||
                         nf <= ao->nextarena->nfreepools) {
                /* Case 4.  Nothing to do. */
                UNLOCK();
                return;
            }
            /* Case 3:  We have to move the arena towards the end
             * of the list, because it has more free pools than
             * the arena to its right.
             * First unlink ao from usable_arenas.
             */
            if (ao->prevarena != NULL) {
                /* ao isn't at the head of the list */
                assert(ao->prevarena->nextarena == ao);
                ao->prevarena->nextarena = ao->nextarena;
            }
            else {
                /* ao is at the head of the list */
                assert(usable_arenas == ao);
                usable_arenas = ao->nextarena;
            }
            ao->nextarena->prevarena = ao->prevarena;

            /* Locate the new insertion point by iterating over
             * the list, using our nextarena pointer.
             */
            while (ao->nextarena != NULL &&
                            nf > ao->nextarena->nfreepools) {
                ao->prevarena = ao->nextarena;
                ao->nextarena = ao->nextarena->nextarena;
            }

            /* Insert ao at this point. */
            assert(ao->nextarena == NULL ||
                ao->prevarena == ao->nextarena->prevarena);
            assert(ao->prevarena->nextarena == ao->nextarena);

            ao->prevarena->nextarena = ao;
            if (ao->nextarena != NULL)
                ao->nextarena->prevarena = ao;

            /* Verify that the swaps worked. */
            assert(ao->nextarena == NULL ||
                      nf <= ao->nextarena->nfreepools);
            assert(ao->prevarena == NULL ||
                      nf > ao->prevarena->nfreepools);
            assert(ao->nextarena == NULL ||
                ao->nextarena->prevarena == ao);
            assert((usable_arenas == ao &&
                ao->prevarena == NULL) ||
                ao->prevarena->nextarena == ao);

            UNLOCK();
            return;
        }
        /* Pool was full, so doesn't currently live in any list:
         * link it to the front of the appropriate usedpools[] list.
         * This mimics LRU pool usage for new allocations and
         * targets optimal filling when several pools contain
         * blocks of the same size class.
         */
        --pool->ref.count;
        assert(pool->ref.count > 0);            /* else the pool is empty */
        size = pool->szidx;
        next = usedpools[size + size];
        prev = next->prevpool;
        /* insert pool before next:   prev <-> pool <-> next */
        pool->nextpool = next;
        pool->prevpool = prev;
        next->prevpool = pool;
        prev->nextpool = pool;
        UNLOCK();
        return;
    }

#ifdef WITH_VALGRIND
redirect:
#endif
    /* We didn't allocate this address. */
    PyMem_RawFree(p);
}

/* realloc.  If p is NULL, this acts like malloc(nbytes).  Else if nbytes==0,
 * then as the Python docs promise, we do not treat this like free(p), and
 * return a non-NULL result.
 */

static void *
_PyObject_Realloc(void *ctx, void *p, size_t nbytes)
{
    void *bp;
    poolp pool;
    size_t size;

    if (p == NULL)
        return _PyObject_Alloc(0, ctx, 1, nbytes);

#ifdef WITH_VALGRIND
    /* Treat running_on_valgrind == -1 the same as 0 */
    if (UNLIKELY(running_on_valgrind > 0))
        goto redirect;
#endif

    pool = POOL_ADDR(p);
    if (address_in_range(p, pool)) {
        /* We're in charge of this block */
        size = INDEX2SIZE(pool->szidx);
        if (nbytes <= size) {
            /* The block is staying the same or shrinking.  If
             * it's shrinking, there's a tradeoff:  it costs
             * cycles to copy the block to a smaller size class,
             * but it wastes memory not to copy it.  The
             * compromise here is to copy on shrink only if at
             * least 25% of size can be shaved off.
             */
            if (4 * nbytes > 3 * size) {
                /* It's the same,
                 * or shrinking and new/old > 3/4.
                 */
                return p;
            }
            size = nbytes;
        }
        bp = _PyObject_Alloc(0, ctx, 1, nbytes);
        if (bp != NULL) {
            memcpy(bp, p, size);
            _PyObject_Free(ctx, p);
        }
        return bp;
    }
#ifdef WITH_VALGRIND
 redirect:
#endif
    /* We're not managing this block.  If nbytes <=
     * SMALL_REQUEST_THRESHOLD, it's tempting to try to take over this
     * block.  However, if we do, we need to copy the valid data from
     * the C-managed block to one of our blocks, and there's no portable
     * way to know how much of the memory space starting at p is valid.
     * As bug 1185883 pointed out the hard way, it's possible that the
     * C-managed block is "at the end" of allocated VM space, so that
     * a memory fault can occur if we try to copy nbytes bytes starting
     * at p.  Instead we punt:  let C continue to manage this block.
     */
    if (nbytes)
        return PyMem_RawRealloc(p, nbytes);
    /* C doesn't define the result of realloc(p, 0) (it may or may not
     * return NULL then), but Python's docs promise that nbytes==0 never
     * returns NULL.  We don't pass 0 to realloc(), to avoid that endcase
     * to begin with.  Even then, we can't be sure that realloc() won't
     * return NULL.
     */
    bp = PyMem_RawRealloc(p, 1);
    return bp ? bp : p;
}

#else   /* ! WITH_PYMALLOC */

/*==========================================================================*/
/* pymalloc not enabled:  Redirect the entry points to malloc.  These will
 * only be used by extensions that are compiled with pymalloc enabled. */

Py_ssize_t
_Py_GetAllocatedBlocks(void)
{
    return 0;
}

#endif /* WITH_PYMALLOC */


/*==========================================================================*/
/* A x-platform debugging allocator.  This doesn't manage memory directly,
 * it wraps a real allocator, adding extra debugging info to the memory blocks.
 */

/* Special bytes broadcast into debug memory blocks at appropriate times.
 * Strings of these are unlikely to be valid addresses, floats, ints or
 * 7-bit ASCII.
 */
#undef CLEANBYTE
#undef DEADBYTE
#undef FORBIDDENBYTE
#define CLEANBYTE      0xCB    /* clean (newly allocated) memory */
#define DEADBYTE       0xDB    /* dead (newly freed) memory */
#define FORBIDDENBYTE  0xFB    /* untouchable bytes at each end of a block */

static size_t serialno = 0;     /* incremented on each debug {m,re}alloc */

/* serialno is always incremented via calling this routine.  The point is
 * to supply a single place to set a breakpoint.
 */
static void
bumpserialno(void)
{
    ++serialno;
}

#define SST SIZEOF_SIZE_T

/* Read sizeof(size_t) bytes at p as a big-endian size_t. */
static size_t
read_size_t(const void *p)
{
    const uint8_t *q = (const uint8_t *)p;
    size_t result = *q++;
    int i;

    for (i = SST; --i > 0; ++q)
        result = (result << 8) | *q;
    return result;
}

/* Write n as a big-endian size_t, MSB at address p, LSB at
 * p + sizeof(size_t) - 1.
 */
static void
write_size_t(void *p, size_t n)
{
    uint8_t *q = (uint8_t *)p + SST - 1;
    int i;

    for (i = SST; --i >= 0; --q) {
        *q = (uint8_t)(n & 0xff);
        n >>= 8;
    }
}

/* Let S = sizeof(size_t).  The debug malloc asks for 4*S extra bytes and
   fills them with useful stuff, here calling the underlying malloc's result p:

p[0: S]
    Number of bytes originally asked for.  This is a size_t, big-endian (easier
    to read in a memory dump).
p[S]
    API ID.  See PEP 445.  This is a character, but seems undocumented.
p[S+1: 2*S]
    Copies of FORBIDDENBYTE.  Used to catch under- writes and reads.
p[2*S: 2*S+n]
    The requested memory, filled with copies of CLEANBYTE.
    Used to catch reference to uninitialized memory.
    &p[2*S] is returned.  Note that this is 8-byte aligned if pymalloc
    handled the request itself.
p[2*S+n: 2*S+n+S]
    Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.
p[2*S+n+S: 2*S+n+2*S]
    A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
    and _PyMem_DebugRealloc.
    This is a big-endian size_t.
    If "bad memory" is detected later, the serial number gives an
    excellent way to set a breakpoint on the next run, to capture the
    instant at which this block was passed out.
*/

static void *
_PyMem_DebugRawAlloc(int use_calloc, void *ctx, size_t nbytes)
{
    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
    uint8_t *p;           /* base address of malloc'ed block */
    uint8_t *tail;        /* p + 2*SST + nbytes == pointer to tail pad bytes */
    size_t total;       /* nbytes + 4*SST */

    bumpserialno();
    total = nbytes + 4*SST;
    if (nbytes > PY_SSIZE_T_MAX - 4*SST)
        /* overflow:  can't represent total as a Py_ssize_t */
        return NULL;

    if (use_calloc)
        p = (uint8_t *)api->alloc.calloc(api->alloc.ctx, 1, total);
    else
        p = (uint8_t *)api->alloc.malloc(api->alloc.ctx, total);
    if (p == NULL)
        return NULL;

    /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
    write_size_t(p, nbytes);
    p[SST] = (uint8_t)api->api_id;
    memset(p + SST + 1, FORBIDDENBYTE, SST-1);

    if (nbytes > 0 && !use_calloc)
        memset(p + 2*SST, CLEANBYTE, nbytes);

    /* at tail, write pad (SST bytes) and serialno (SST bytes) */
    tail = p + 2*SST + nbytes;
    memset(tail, FORBIDDENBYTE, SST);
    write_size_t(tail + SST, serialno);

    return p + 2*SST;
}

static void *
_PyMem_DebugRawMalloc(void *ctx, size_t nbytes)
{
    return _PyMem_DebugRawAlloc(0, ctx, nbytes);
}

static void *
_PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize)
{
    size_t nbytes;
    assert(elsize == 0 || nelem <= PY_SSIZE_T_MAX / elsize);
    nbytes = nelem * elsize;
    return _PyMem_DebugRawAlloc(1, ctx, nbytes);
}

/* The debug free first checks the 2*SST bytes on each end for sanity (in
   particular, that the FORBIDDENBYTEs with the api ID are still intact).
   Then fills the original bytes with DEADBYTE.
   Then calls the underlying free.
*/
static void
_PyMem_DebugRawFree(void *ctx, void *p)
{
    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
    uint8_t *q = (uint8_t *)p - 2*SST;  /* address returned from malloc */
    size_t nbytes;

    if (p == NULL)
        return;
    _PyMem_DebugCheckAddress(api->api_id, p);
    nbytes = read_size_t(q);
    nbytes += 4*SST;
    if (nbytes > 0)
        memset(q, DEADBYTE, nbytes);
    api->alloc.free(api->alloc.ctx, q);
}

static void *
_PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
{
    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
    uint8_t *q = (uint8_t *)p, *oldq;
    uint8_t *tail;
    size_t total;       /* nbytes + 4*SST */
    size_t original_nbytes;
    int i;

    if (p == NULL)
        return _PyMem_DebugRawAlloc(0, ctx, nbytes);

    _PyMem_DebugCheckAddress(api->api_id, p);
    bumpserialno();
    original_nbytes = read_size_t(q - 2*SST);
    total = nbytes + 4*SST;
    if (nbytes > PY_SSIZE_T_MAX - 4*SST)
        /* overflow:  can't represent total as a Py_ssize_t */
        return NULL;

    /* Resize and add decorations. We may get a new pointer here, in which
     * case we didn't get the chance to mark the old memory with DEADBYTE,
     * but we live with that.
     */
    oldq = q;
    q = (uint8_t *)api->alloc.realloc(api->alloc.ctx, q - 2*SST, total);
    if (q == NULL)
        return NULL;

    if (q == oldq && nbytes < original_nbytes) {
        /* shrinking:  mark old extra memory dead */
        memset(q + nbytes, DEADBYTE, original_nbytes - nbytes);
    }

    write_size_t(q, nbytes);
    assert(q[SST] == (uint8_t)api->api_id);
    for (i = 1; i < SST; ++i)
        assert(q[SST + i] == FORBIDDENBYTE);
    q += 2*SST;

    tail = q + nbytes;
    memset(tail, FORBIDDENBYTE, SST);
    write_size_t(tail + SST, serialno);

    if (nbytes > original_nbytes) {
        /* growing:  mark new extra memory clean */
        memset(q + original_nbytes, CLEANBYTE,
               nbytes - original_nbytes);
    }

    return q;
}

static void
_PyMem_DebugCheckGIL(void)
{
#ifdef WITH_THREAD
    if (!PyGILState_Check())
        Py_FatalError("Python memory allocator called "
                      "without holding the GIL");
#endif
}

static void *
_PyMem_DebugMalloc(void *ctx, size_t nbytes)
{
    _PyMem_DebugCheckGIL();
    return _PyMem_DebugRawMalloc(ctx, nbytes);
}

static void *
_PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize)
{
    _PyMem_DebugCheckGIL();
    return _PyMem_DebugRawCalloc(ctx, nelem, elsize);
}

static void
_PyMem_DebugFree(void *ctx, void *ptr)
{
    _PyMem_DebugCheckGIL();
    _PyMem_DebugRawFree(ctx, ptr);
}

static void *
_PyMem_DebugRealloc(void *ctx, void *ptr, size_t nbytes)
{
    _PyMem_DebugCheckGIL();
    return _PyMem_DebugRawRealloc(ctx, ptr, nbytes);
}

/* Check the forbidden bytes on both ends of the memory allocated for p.
 * If anything is wrong, print info to stderr via _PyObject_DebugDumpAddress,
 * and call Py_FatalError to kill the program.
 * The API id, is also checked.
 */
static void
_PyMem_DebugCheckAddress(char api, const void *p)
{
    const uint8_t *q = (const uint8_t *)p;
    char msgbuf[64];
    char *msg;
    size_t nbytes;
    const uint8_t *tail;
    int i;
    char id;

    if (p == NULL) {
        msg = "didn't expect a NULL pointer";
        goto error;
    }

    /* Check the API id */
    id = (char)q[-SST];
    if (id != api) {
        msg = msgbuf;
        snprintf(msg, sizeof(msgbuf), "bad ID: Allocated using API '%c', verified using API '%c'", id, api);
        msgbuf[sizeof(msgbuf)-1] = 0;
        goto error;
    }

    /* Check the stuff at the start of p first:  if there's underwrite
     * corruption, the number-of-bytes field may be nuts, and checking
     * the tail could lead to a segfault then.
     */
    for (i = SST-1; i >= 1; --i) {
        if (*(q-i) != FORBIDDENBYTE) {
            msg = "bad leading pad byte";
            goto error;
        }
    }

    nbytes = read_size_t(q - 2*SST);
    tail = q + nbytes;
    for (i = 0; i < SST; ++i) {
        if (tail[i] != FORBIDDENBYTE) {
            msg = "bad trailing pad byte";
            goto error;
        }
    }

    return;

error:
    _PyObject_DebugDumpAddress(p);
    Py_FatalError(msg);
}

/* Display info to stderr about the memory block at p. */
static void
_PyObject_DebugDumpAddress(const void *p)
{
    const uint8_t *q = (const uint8_t *)p;
    const uint8_t *tail;
    size_t nbytes, serial;
    int i;
    int ok;
    char id;

    fprintf(stderr, "Debug memory block at address p=%p:", p);
    if (p == NULL) {
        fprintf(stderr, "\n");
        return;
    }
    id = (char)q[-SST];
    fprintf(stderr, " API '%c'\n", id);

    nbytes = read_size_t(q - 2*SST);
    fprintf(stderr, "    %" PY_FORMAT_SIZE_T "u bytes originally "
                    "requested\n", nbytes);

    /* In case this is nuts, check the leading pad bytes first. */
    fprintf(stderr, "    The %d pad bytes at p-%d are ", SST-1, SST-1);
    ok = 1;
    for (i = 1; i <= SST-1; ++i) {
        if (*(q-i) != FORBIDDENBYTE) {
            ok = 0;
            break;
        }
    }
    if (ok)
        fputs("FORBIDDENBYTE, as expected.\n", stderr);
    else {
        fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
            FORBIDDENBYTE);
        for (i = SST-1; i >= 1; --i) {
            const uint8_t byte = *(q-i);
            fprintf(stderr, "        at p-%d: 0x%02x", i, byte);
            if (byte != FORBIDDENBYTE)
                fputs(" *** OUCH", stderr);
            fputc('\n', stderr);
        }

        fputs("    Because memory is corrupted at the start, the "
              "count of bytes requested\n"
              "       may be bogus, and checking the trailing pad "
              "bytes may segfault.\n", stderr);
    }

    tail = q + nbytes;
    fprintf(stderr, "    The %d pad bytes at tail=%p are ", SST, tail);
    ok = 1;
    for (i = 0; i < SST; ++i) {
        if (tail[i] != FORBIDDENBYTE) {
            ok = 0;
            break;
        }
    }
    if (ok)
        fputs("FORBIDDENBYTE, as expected.\n", stderr);
    else {
        fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
                FORBIDDENBYTE);
        for (i = 0; i < SST; ++i) {
            const uint8_t byte = tail[i];
            fprintf(stderr, "        at tail+%d: 0x%02x",
                    i, byte);
            if (byte != FORBIDDENBYTE)
                fputs(" *** OUCH", stderr);
            fputc('\n', stderr);
        }
    }

    serial = read_size_t(tail + SST);
    fprintf(stderr, "    The block was made by call #%" PY_FORMAT_SIZE_T
                    "u to debug malloc/realloc.\n", serial);

    if (nbytes > 0) {
        i = 0;
        fputs("    Data at p:", stderr);
        /* print up to 8 bytes at the start */
        while (q < tail && i < 8) {
            fprintf(stderr, " %02x", *q);
            ++i;
            ++q;
        }
        /* and up to 8 at the end */
        if (q < tail) {
            if (tail - q > 8) {
                fputs(" ...", stderr);
                q = tail - 8;
            }
            while (q < tail) {
                fprintf(stderr, " %02x", *q);
                ++q;
            }
        }
        fputc('\n', stderr);
    }
    fputc('\n', stderr);

    fflush(stderr);
    _PyMem_DumpTraceback(fileno(stderr), p);
}


static size_t
printone(FILE *out, const char* msg, size_t value)
{
    int i, k;
    char buf[100];
    size_t origvalue = value;

    fputs(msg, out);
    for (i = (int)strlen(msg); i < 35; ++i)
        fputc(' ', out);
    fputc('=', out);

    /* Write the value with commas. */
    i = 22;
    buf[i--] = '\0';
    buf[i--] = '\n';
    k = 3;
    do {
        size_t nextvalue = value / 10;
        unsigned int digit = (unsigned int)(value - nextvalue * 10);
        value = nextvalue;
        buf[i--] = (char)(digit + '0');
        --k;
        if (k == 0 && value && i >= 0) {
            k = 3;
            buf[i--] = ',';
        }
    } while (value && i >= 0);

    while (i >= 0)
        buf[i--] = ' ';
    fputs(buf, out);

    return origvalue;
}

void
_PyDebugAllocatorStats(FILE *out,
                       const char *block_name, int num_blocks, size_t sizeof_block)
{
    char buf1[128];
    char buf2[128];
    PyOS_snprintf(buf1, sizeof(buf1),
                  "%d %ss * %" PY_FORMAT_SIZE_T "d bytes each",
                  num_blocks, block_name, sizeof_block);
    PyOS_snprintf(buf2, sizeof(buf2),
                  "%48s ", buf1);
    (void)printone(out, buf2, num_blocks * sizeof_block);
}


#ifdef WITH_PYMALLOC

#ifdef Py_DEBUG
/* Is target in the list?  The list is traversed via the nextpool pointers.
 * The list may be NULL-terminated, or circular.  Return 1 if target is in
 * list, else 0.
 */
static int
pool_is_in_list(const poolp target, poolp list)
{
    poolp origlist = list;
    assert(target != NULL);
    if (list == NULL)
        return 0;
    do {
        if (target == list)
            return 1;
        list = list->nextpool;
    } while (list != NULL && list != origlist);
    return 0;
}
#endif

/* Print summary info to "out" about the state of pymalloc's structures.
 * In Py_DEBUG mode, also perform some expensive internal consistency
 * checks.
 */
void
_PyObject_DebugMallocStats(FILE *out)
{
    uint i;
    const uint numclasses = SMALL_REQUEST_THRESHOLD >> ALIGNMENT_SHIFT;
    /* # of pools, allocated blocks, and free blocks per class index */
    size_t numpools[SMALL_REQUEST_THRESHOLD >> ALIGNMENT_SHIFT];
    size_t numblocks[SMALL_REQUEST_THRESHOLD >> ALIGNMENT_SHIFT];
    size_t numfreeblocks[SMALL_REQUEST_THRESHOLD >> ALIGNMENT_SHIFT];
    /* total # of allocated bytes in used and full pools */
    size_t allocated_bytes = 0;
    /* total # of available bytes in used pools */
    size_t available_bytes = 0;
    /* # of free pools + pools not yet carved out of current arena */
    uint numfreepools = 0;
    /* # of bytes for arena alignment padding */
    size_t arena_alignment = 0;
    /* # of bytes in used and full pools used for pool_headers */
    size_t pool_header_bytes = 0;
    /* # of bytes in used and full pools wasted due to quantization,
     * i.e. the necessarily leftover space at the ends of used and
     * full pools.
     */
    size_t quantization = 0;
    /* # of arenas actually allocated. */
    size_t narenas = 0;
    /* running total -- should equal narenas * ARENA_SIZE */
    size_t total;
    char buf[128];

    fprintf(out, "Small block threshold = %d, in %u size classes.\n",
            SMALL_REQUEST_THRESHOLD, numclasses);

    for (i = 0; i < numclasses; ++i)
        numpools[i] = numblocks[i] = numfreeblocks[i] = 0;

    /* Because full pools aren't linked to from anything, it's easiest
     * to march over all the arenas.  If we're lucky, most of the memory
     * will be living in full pools -- would be a shame to miss them.
     */
    for (i = 0; i < maxarenas; ++i) {
        uint j;
        uintptr_t base = arenas[i].address;

        /* Skip arenas which are not allocated. */
        if (arenas[i].address == (uintptr_t)NULL)
            continue;
        narenas += 1;

        numfreepools += arenas[i].nfreepools;

        /* round up to pool alignment */
        if (base & (uintptr_t)POOL_SIZE_MASK) {
            arena_alignment += POOL_SIZE;
            base &= ~(uintptr_t)POOL_SIZE_MASK;
            base += POOL_SIZE;
        }

        /* visit every pool in the arena */
        assert(base <= (uintptr_t) arenas[i].pool_address);
        for (j = 0; base < (uintptr_t) arenas[i].pool_address;
             ++j, base += POOL_SIZE) {
            poolp p = (poolp)base;
            const uint sz = p->szidx;
            uint freeblocks;

            if (p->ref.count == 0) {
                /* currently unused */
#ifdef Py_DEBUG
                assert(pool_is_in_list(p, arenas[i].freepools));
#endif
                continue;
            }
            ++numpools[sz];
            numblocks[sz] += p->ref.count;
            freeblocks = NUMBLOCKS(sz) - p->ref.count;
            numfreeblocks[sz] += freeblocks;
#ifdef Py_DEBUG
            if (freeblocks > 0)
                assert(pool_is_in_list(p, usedpools[sz + sz]));
#endif
        }
    }
    assert(narenas == narenas_currently_allocated);

    fputc('\n', out);
    fputs("class   size   num pools   blocks in use  avail blocks\n"
          "-----   ----   ---------   -------------  ------------\n",
          out);

    for (i = 0; i < numclasses; ++i) {
        size_t p = numpools[i];
        size_t b = numblocks[i];
        size_t f = numfreeblocks[i];
        uint size = INDEX2SIZE(i);
        if (p == 0) {
            assert(b == 0 && f == 0);
            continue;
        }
        fprintf(out, "%5u %6u "
                        "%11" PY_FORMAT_SIZE_T "u "
                        "%15" PY_FORMAT_SIZE_T "u "
                        "%13" PY_FORMAT_SIZE_T "u\n",
                i, size, p, b, f);
        allocated_bytes += b * size;
        available_bytes += f * size;
        pool_header_bytes += p * POOL_OVERHEAD;
        quantization += p * ((POOL_SIZE - POOL_OVERHEAD) % size);
    }
    fputc('\n', out);
    if (_PyMem_DebugEnabled())
        (void)printone(out, "# times object malloc called", serialno);
    (void)printone(out, "# arenas allocated total", ntimes_arena_allocated);
    (void)printone(out, "# arenas reclaimed", ntimes_arena_allocated - narenas);
    (void)printone(out, "# arenas highwater mark", narenas_highwater);
    (void)printone(out, "# arenas allocated current", narenas);

    PyOS_snprintf(buf, sizeof(buf),
        "%" PY_FORMAT_SIZE_T "u arenas * %d bytes/arena",
        narenas, ARENA_SIZE);
    (void)printone(out, buf, narenas * ARENA_SIZE);

    fputc('\n', out);

    total = printone(out, "# bytes in allocated blocks", allocated_bytes);
    total += printone(out, "# bytes in available blocks", available_bytes);

    PyOS_snprintf(buf, sizeof(buf),
        "%u unused pools * %d bytes", numfreepools, POOL_SIZE);
    total += printone(out, buf, (size_t)numfreepools * POOL_SIZE);

    total += printone(out, "# bytes lost to pool headers", pool_header_bytes);
    total += printone(out, "# bytes lost to quantization", quantization);
    total += printone(out, "# bytes lost to arena alignment", arena_alignment);
    (void)printone(out, "Total", total);
}

#endif /* #ifdef WITH_PYMALLOC */
#define PGEN
#include "parsetok.c"
/* Parser generator */

/* For a description, see the comments at end of this file */

#include "Python.h"
#include "pgenheaders.h"
#include "token.h"
#include "node.h"
#include "grammar.h"
#include "metagrammar.h"
#include "pgen.h"

extern int Py_DebugFlag;
extern int Py_IgnoreEnvironmentFlag; /* needed by Py_GETENV */


/* PART ONE -- CONSTRUCT NFA -- Cf. Algorithm 3.2 from [Aho&Ullman 77] */

typedef struct _nfaarc {
    int         ar_label;
    int         ar_arrow;
} nfaarc;

typedef struct _nfastate {
    int         st_narcs;
    nfaarc      *st_arc;
} nfastate;

typedef struct _nfa {
    int                 nf_type;
    char                *nf_name;
    int                 nf_nstates;
    nfastate            *nf_state;
    int                 nf_start, nf_finish;
} nfa;

/* Forward */
static void compile_rhs(labellist *ll,
                        nfa *nf, node *n, int *pa, int *pb);
static void compile_alt(labellist *ll,
                        nfa *nf, node *n, int *pa, int *pb);
static void compile_item(labellist *ll,
                         nfa *nf, node *n, int *pa, int *pb);
static void compile_atom(labellist *ll,
                         nfa *nf, node *n, int *pa, int *pb);

static int
addnfastate(nfa *nf)
{
    nfastate *st;

    nf->nf_state = (nfastate *)PyObject_REALLOC(nf->nf_state,
                                sizeof(nfastate) * (nf->nf_nstates + 1));
    if (nf->nf_state == NULL)
        Py_FatalError("out of mem");
    st = &nf->nf_state[nf->nf_nstates++];
    st->st_narcs = 0;
    st->st_arc = NULL;
    return st - nf->nf_state;
}

static void
addnfaarc(nfa *nf, int from, int to, int lbl)
{
    nfastate *st;
    nfaarc *ar;

    st = &nf->nf_state[from];
    st->st_arc = (nfaarc *)PyObject_REALLOC(st->st_arc,
                                  sizeof(nfaarc) * (st->st_narcs + 1));
    if (st->st_arc == NULL)
        Py_FatalError("out of mem");
    ar = &st->st_arc[st->st_narcs++];
    ar->ar_label = lbl;
    ar->ar_arrow = to;
}

static nfa *
newnfa(char *name)
{
    nfa *nf;
    static int type = NT_OFFSET; /* All types will be disjunct */

    nf = (nfa *)PyObject_MALLOC(sizeof(nfa));
    if (nf == NULL)
        Py_FatalError("no mem for new nfa");
    nf->nf_type = type++;
    nf->nf_name = name; /* XXX strdup(name) ??? */
    nf->nf_nstates = 0;
    nf->nf_state = NULL;
    nf->nf_start = nf->nf_finish = -1;
    return nf;
}

typedef struct _nfagrammar {
    int                 gr_nnfas;
    nfa                 **gr_nfa;
    labellist           gr_ll;
} nfagrammar;

/* Forward */
static void compile_rule(nfagrammar *gr, node *n);

static nfagrammar *
newnfagrammar(void)
{
    nfagrammar *gr;

    gr = (nfagrammar *)PyObject_MALLOC(sizeof(nfagrammar));
    if (gr == NULL)
        Py_FatalError("no mem for new nfa grammar");
    gr->gr_nnfas = 0;
    gr->gr_nfa = NULL;
    gr->gr_ll.ll_nlabels = 0;
    gr->gr_ll.ll_label = NULL;
    addlabel(&gr->gr_ll, ENDMARKER, "EMPTY");
    return gr;
}

static void
freenfagrammar(nfagrammar *gr)
{
    for (int i = 0; i < gr->gr_nnfas; i++) {
        PyObject_FREE(gr->gr_nfa[i]->nf_state);
    }
    PyObject_FREE(gr->gr_nfa);
    PyObject_FREE(gr);
}

static nfa *
addnfa(nfagrammar *gr, char *name)
{
    nfa *nf;

    nf = newnfa(name);
    gr->gr_nfa = (nfa **)PyObject_REALLOC(gr->gr_nfa,
                                  sizeof(nfa*) * (gr->gr_nnfas + 1));
    if (gr->gr_nfa == NULL)
        Py_FatalError("out of mem");
    gr->gr_nfa[gr->gr_nnfas++] = nf;
    addlabel(&gr->gr_ll, NAME, nf->nf_name);
    return nf;
}

#ifdef Py_DEBUG

static const char REQNFMT[] = "metacompile: less than %d children\n";

#define REQN(i, count) do { \
    if (i < count) { \
        fprintf(stderr, REQNFMT, count); \
        Py_FatalError("REQN"); \
    } \
} while (0)

#else
#define REQN(i, count)  /* empty */
#endif

static nfagrammar *
metacompile(node *n)
{
    nfagrammar *gr;
    int i;

    if (Py_DebugFlag)
        printf("Compiling (meta-) parse tree into NFA grammar\n");
    gr = newnfagrammar();
    REQ(n, MSTART);
    i = n->n_nchildren - 1; /* Last child is ENDMARKER */
    n = n->n_child;
    for (; --i >= 0; n++) {
        if (n->n_type != NEWLINE)
            compile_rule(gr, n);
    }
    return gr;
}

static void
compile_rule(nfagrammar *gr, node *n)
{
    nfa *nf;

    REQ(n, RULE);
    REQN(n->n_nchildren, 4);
    n = n->n_child;
    REQ(n, NAME);
    nf = addnfa(gr, n->n_str);
    n++;
    REQ(n, COLON);
    n++;
    REQ(n, RHS);
    compile_rhs(&gr->gr_ll, nf, n, &nf->nf_start, &nf->nf_finish);
    n++;
    REQ(n, NEWLINE);
}

static void
compile_rhs(labellist *ll, nfa *nf, node *n, int *pa, int *pb)
{
    int i;
    int a, b;

    REQ(n, RHS);
    i = n->n_nchildren;
    REQN(i, 1);
    n = n->n_child;
    REQ(n, ALT);
    compile_alt(ll, nf, n, pa, pb);
    if (--i <= 0)
        return;
    n++;
    a = *pa;
    b = *pb;
    *pa = addnfastate(nf);
    *pb = addnfastate(nf);
    addnfaarc(nf, *pa, a, EMPTY);
    addnfaarc(nf, b, *pb, EMPTY);
    for (; --i >= 0; n++) {
        REQ(n, VBAR);
        REQN(i, 1);
        --i;
        n++;
        REQ(n, ALT);
        compile_alt(ll, nf, n, &a, &b);
        addnfaarc(nf, *pa, a, EMPTY);
        addnfaarc(nf, b, *pb, EMPTY);
    }
}

static void
compile_alt(labellist *ll, nfa *nf, node *n, int *pa, int *pb)
{
    int i;
    int a, b;

    REQ(n, ALT);
    i = n->n_nchildren;
    REQN(i, 1);
    n = n->n_child;
    REQ(n, ITEM);
    compile_item(ll, nf, n, pa, pb);
    --i;
    n++;
    for (; --i >= 0; n++) {
        REQ(n, ITEM);
        compile_item(ll, nf, n, &a, &b);
        addnfaarc(nf, *pb, a, EMPTY);
        *pb = b;
    }
}

static void
compile_item(labellist *ll, nfa *nf, node *n, int *pa, int *pb)
{
    int i;
    int a, b;

    REQ(n, ITEM);
    i = n->n_nchildren;
    REQN(i, 1);
    n = n->n_child;
    if (n->n_type == LSQB) {
        REQN(i, 3);
        n++;
        REQ(n, RHS);
        *pa = addnfastate(nf);
        *pb = addnfastate(nf);
        addnfaarc(nf, *pa, *pb, EMPTY);
        compile_rhs(ll, nf, n, &a, &b);
        addnfaarc(nf, *pa, a, EMPTY);
        addnfaarc(nf, b, *pb, EMPTY);
        REQN(i, 1);
        n++;
        REQ(n, RSQB);
    }
    else {
        compile_atom(ll, nf, n, pa, pb);
        if (--i <= 0)
            return;
        n++;
        addnfaarc(nf, *pb, *pa, EMPTY);
        if (n->n_type == STAR)
            *pb = *pa;
        else
            REQ(n, PLUS);
    }
}

static void
compile_atom(labellist *ll, nfa *nf, node *n, int *pa, int *pb)
{
    int i;

    REQ(n, ATOM);
    i = n->n_nchildren;
    (void)i; /* Don't warn about set but unused */
    REQN(i, 1);
    n = n->n_child;
    if (n->n_type == LPAR) {
        REQN(i, 3);
        n++;
        REQ(n, RHS);
        compile_rhs(ll, nf, n, pa, pb);
        n++;
        REQ(n, RPAR);
    }
    else if (n->n_type == NAME || n->n_type == STRING) {
        *pa = addnfastate(nf);
        *pb = addnfastate(nf);
        addnfaarc(nf, *pa, *pb, addlabel(ll, n->n_type, n->n_str));
    }
    else
        REQ(n, NAME);
}

static void
dumpstate(labellist *ll, nfa *nf, int istate)
{
    nfastate *st;
    int i;
    nfaarc *ar;

    printf("%c%2d%c",
        istate == nf->nf_start ? '*' : ' ',
        istate,
        istate == nf->nf_finish ? '.' : ' ');
    st = &nf->nf_state[istate];
    ar = st->st_arc;
    for (i = 0; i < st->st_narcs; i++) {
        if (i > 0)
            printf("\n    ");
        printf("-> %2d  %s", ar->ar_arrow,
            Ta3Grammar_LabelRepr(&ll->ll_label[ar->ar_label]));
        ar++;
    }
    printf("\n");
}

static void
dumpnfa(labellist *ll, nfa *nf)
{
    int i;

    printf("NFA '%s' has %d states; start %d, finish %d\n",
        nf->nf_name, nf->nf_nstates, nf->nf_start, nf->nf_finish);
    for (i = 0; i < nf->nf_nstates; i++)
        dumpstate(ll, nf, i);
}


/* PART TWO -- CONSTRUCT DFA -- Algorithm 3.1 from [Aho&Ullman 77] */

static void
addclosure(bitset ss, nfa *nf, int istate)
{
    if (addbit(ss, istate)) {
        nfastate *st = &nf->nf_state[istate];
        nfaarc *ar = st->st_arc;
        int i;

        for (i = st->st_narcs; --i >= 0; ) {
            if (ar->ar_label == EMPTY)
                addclosure(ss, nf, ar->ar_arrow);
            ar++;
        }
    }
}

typedef struct _ss_arc {
    bitset      sa_bitset;
    int         sa_arrow;
    int         sa_label;
} ss_arc;

typedef struct _ss_state {
    bitset      ss_ss;
    int         ss_narcs;
    struct _ss_arc      *ss_arc;
    int         ss_deleted;
    int         ss_finish;
    int         ss_rename;
} ss_state;

typedef struct _ss_dfa {
    int         sd_nstates;
    ss_state *sd_state;
} ss_dfa;

/* Forward */
static void printssdfa(int xx_nstates, ss_state *xx_state, int nbits,
                       labellist *ll, const char *msg);
static void simplify(int xx_nstates, ss_state *xx_state);
static void convert(dfa *d, int xx_nstates, ss_state *xx_state);

static void
makedfa(nfagrammar *gr, nfa *nf, dfa *d)
{
    int nbits = nf->nf_nstates;
    bitset ss;
    int xx_nstates;
    ss_state *xx_state, *yy;
    ss_arc *zz;
    int istate, jstate, iarc, jarc, ibit;
    nfastate *st;
    nfaarc *ar;

    ss = newbitset(nbits);
    addclosure(ss, nf, nf->nf_start);
    xx_state = (ss_state *)PyObject_MALLOC(sizeof(ss_state));
    if (xx_state == NULL)
        Py_FatalError("no mem for xx_state in makedfa");
    xx_nstates = 1;
    yy = &xx_state[0];
    yy->ss_ss = ss;
    yy->ss_narcs = 0;
    yy->ss_arc = NULL;
    yy->ss_deleted = 0;
    yy->ss_finish = testbit(ss, nf->nf_finish);
    if (yy->ss_finish)
        printf("Error: nonterminal '%s' may produce empty.\n",
            nf->nf_name);

    /* This algorithm is from a book written before
       the invention of structured programming... */

    /* For each unmarked state... */
    for (istate = 0; istate < xx_nstates; ++istate) {
        size_t size;
        yy = &xx_state[istate];
        ss = yy->ss_ss;
        /* For all its states... */
        for (ibit = 0; ibit < nf->nf_nstates; ++ibit) {
            if (!testbit(ss, ibit))
                continue;
            st = &nf->nf_state[ibit];
            /* For all non-empty arcs from this state... */
            for (iarc = 0; iarc < st->st_narcs; iarc++) {
                ar = &st->st_arc[iarc];
                if (ar->ar_label == EMPTY)
                    continue;
                /* Look up in list of arcs from this state */
                for (jarc = 0; jarc < yy->ss_narcs; ++jarc) {
                    zz = &yy->ss_arc[jarc];
                    if (ar->ar_label == zz->sa_label)
                        goto found;
                }
                /* Add new arc for this state */
                size = sizeof(ss_arc) * (yy->ss_narcs + 1);
                yy->ss_arc = (ss_arc *)PyObject_REALLOC(
                                            yy->ss_arc, size);
                if (yy->ss_arc == NULL)
                    Py_FatalError("out of mem");
                zz = &yy->ss_arc[yy->ss_narcs++];
                zz->sa_label = ar->ar_label;
                zz->sa_bitset = newbitset(nbits);
                zz->sa_arrow = -1;
             found:             ;
                /* Add destination */
                addclosure(zz->sa_bitset, nf, ar->ar_arrow);
            }
        }
        /* Now look up all the arrow states */
        for (jarc = 0; jarc < xx_state[istate].ss_narcs; jarc++) {
            zz = &xx_state[istate].ss_arc[jarc];
            for (jstate = 0; jstate < xx_nstates; jstate++) {
                if (samebitset(zz->sa_bitset,
                    xx_state[jstate].ss_ss, nbits)) {
                    zz->sa_arrow = jstate;
                    goto done;
                }
            }
            size = sizeof(ss_state) * (xx_nstates + 1);
            xx_state = (ss_state *)PyObject_REALLOC(xx_state,
                                                        size);
            if (xx_state == NULL)
                Py_FatalError("out of mem");
            zz->sa_arrow = xx_nstates;
            yy = &xx_state[xx_nstates++];
            yy->ss_ss = zz->sa_bitset;
            yy->ss_narcs = 0;
            yy->ss_arc = NULL;
            yy->ss_deleted = 0;
            yy->ss_finish = testbit(yy->ss_ss, nf->nf_finish);
         done:          ;
        }
    }

    if (Py_DebugFlag)
        printssdfa(xx_nstates, xx_state, nbits, &gr->gr_ll,
                                        "before minimizing");

    simplify(xx_nstates, xx_state);

    if (Py_DebugFlag)
        printssdfa(xx_nstates, xx_state, nbits, &gr->gr_ll,
                                        "after minimizing");

    convert(d, xx_nstates, xx_state);

    for (int i = 0; i < xx_nstates; i++) {
        for (int j = 0; j < xx_state[i].ss_narcs; j++)
            delbitset(xx_state[i].ss_arc[j].sa_bitset);
        PyObject_FREE(xx_state[i].ss_arc);
    }
    PyObject_FREE(xx_state);
}

static void
printssdfa(int xx_nstates, ss_state *xx_state, int nbits,
           labellist *ll, const char *msg)
{
    int i, ibit, iarc;
    ss_state *yy;
    ss_arc *zz;

    printf("Subset DFA %s\n", msg);
    for (i = 0; i < xx_nstates; i++) {
        yy = &xx_state[i];
        if (yy->ss_deleted)
            continue;
        printf(" Subset %d", i);
        if (yy->ss_finish)
            printf(" (finish)");
        printf(" { ");
        for (ibit = 0; ibit < nbits; ibit++) {
            if (testbit(yy->ss_ss, ibit))
                printf("%d ", ibit);
        }
        printf("}\n");
        for (iarc = 0; iarc < yy->ss_narcs; iarc++) {
            zz = &yy->ss_arc[iarc];
            printf("  Arc to state %d, label %s\n",
                zz->sa_arrow,
                Ta3Grammar_LabelRepr(
                    &ll->ll_label[zz->sa_label]));
        }
    }
}


/* PART THREE -- SIMPLIFY DFA */

/* Simplify the DFA by repeatedly eliminating states that are
   equivalent to another oner.  This is NOT Algorithm 3.3 from
   [Aho&Ullman 77].  It does not always finds the minimal DFA,
   but it does usually make a much smaller one...  (For an example
   of sub-optimal behavior, try S: x a b+ | y a b+.)
*/

static int
samestate(ss_state *s1, ss_state *s2)
{
    int i;

    if (s1->ss_narcs != s2->ss_narcs || s1->ss_finish != s2->ss_finish)
        return 0;
    for (i = 0; i < s1->ss_narcs; i++) {
        if (s1->ss_arc[i].sa_arrow != s2->ss_arc[i].sa_arrow ||
            s1->ss_arc[i].sa_label != s2->ss_arc[i].sa_label)
            return 0;
    }
    return 1;
}

static void
renamestates(int xx_nstates, ss_state *xx_state, int from, int to)
{
    int i, j;

    if (Py_DebugFlag)
        printf("Rename state %d to %d.\n", from, to);
    for (i = 0; i < xx_nstates; i++) {
        if (xx_state[i].ss_deleted)
            continue;
        for (j = 0; j < xx_state[i].ss_narcs; j++) {
            if (xx_state[i].ss_arc[j].sa_arrow == from)
                xx_state[i].ss_arc[j].sa_arrow = to;
        }
    }
}

static void
simplify(int xx_nstates, ss_state *xx_state)
{
    int changes;
    int i, j;

    do {
        changes = 0;
        for (i = 1; i < xx_nstates; i++) {
            if (xx_state[i].ss_deleted)
                continue;
            for (j = 0; j < i; j++) {
                if (xx_state[j].ss_deleted)
                    continue;
                if (samestate(&xx_state[i], &xx_state[j])) {
                    xx_state[i].ss_deleted++;
                    renamestates(xx_nstates, xx_state,
                                 i, j);
                    changes++;
                    break;
                }
            }
        }
    } while (changes);
}


/* PART FOUR -- GENERATE PARSING TABLES */

/* Convert the DFA into a grammar that can be used by our parser */

static void
convert(dfa *d, int xx_nstates, ss_state *xx_state)
{
    int i, j;
    ss_state *yy;
    ss_arc *zz;

    for (i = 0; i < xx_nstates; i++) {
        yy = &xx_state[i];
        if (yy->ss_deleted)
            continue;
        yy->ss_rename = addstate(d);
    }

    for (i = 0; i < xx_nstates; i++) {
        yy = &xx_state[i];
        if (yy->ss_deleted)
            continue;
        for (j = 0; j < yy->ss_narcs; j++) {
            zz = &yy->ss_arc[j];
            addarc(d, yy->ss_rename,
                xx_state[zz->sa_arrow].ss_rename,
                zz->sa_label);
        }
        if (yy->ss_finish)
            addarc(d, yy->ss_rename, yy->ss_rename, 0);
    }

    d->d_initial = 0;
}


/* PART FIVE -- GLUE IT ALL TOGETHER */

static grammar *
maketables(nfagrammar *gr)
{
    int i;
    nfa *nf;
    dfa *d;
    grammar *g;

    if (gr->gr_nnfas == 0)
        return NULL;
    g = newgrammar(gr->gr_nfa[0]->nf_type);
                    /* XXX first rule must be start rule */
    g->g_ll = gr->gr_ll;

    for (i = 0; i < gr->gr_nnfas; i++) {
        nf = gr->gr_nfa[i];
        if (Py_DebugFlag) {
            printf("Dump of NFA for '%s' ...\n", nf->nf_name);
            dumpnfa(&gr->gr_ll, nf);
            printf("Making DFA for '%s' ...\n", nf->nf_name);
        }
        d = adddfa(g, nf->nf_type, nf->nf_name);
        makedfa(gr, gr->gr_nfa[i], d);
    }

    return g;
}

grammar *
pgen(node *n)
{
    nfagrammar *gr;
    grammar *g;

    gr = metacompile(n);
    g = maketables(gr);
    translatelabels(g);
    addfirstsets(g);
    freenfagrammar(gr);
    return g;
}

grammar *
Py_pgen(node *n)
{
  return pgen(n);
}

/*

Description

Input is a grammar in extended BNF (using * for repetition, + for
at-least-once repetition, [] for optional parts, | for alternatives and
() for grouping).  This has already been parsed and turned into a parse
tree.

Each rule is considered as a regular expression in its own right.
It is turned into a Non-deterministic Finite Automaton (NFA), which
is then turned into a Deterministic Finite Automaton (DFA), which is then
optimized to reduce the number of states.  See [Aho&Ullman 77] chapter 3,
or similar compiler books (this technique is more often used for lexical
analyzers).

The DFA's are used by the parser as parsing tables in a special way
that's probably unique.  Before they are usable, the FIRST sets of all
non-terminals are computed.

Reference

[Aho&Ullman 77]
    Aho&Ullman, Principles of Compiler Design, Addison-Wesley 1977
    (first edition)

*/

/* Parser generator main program */

/* This expects a filename containing the grammar as argv[1] (UNIX)
   or asks the console for such a file name (THINK C).
   It writes its output on two files in the current directory:
   - "graminit.c" gets the grammar as a bunch of initialized data
   - "graminit.h" gets the grammar's non-terminals as #defines.
   Error messages and status info during the generation process are
   written to stdout, or sometimes to stderr. */

/* XXX TO DO:
   - check for duplicate definitions of names (instead of fatal err)
*/

#define PGEN

#include "Python.h"
#include "pgenheaders.h"
#include "grammar.h"
#include "node.h"
#include "parsetok.h"
#include "pgen.h"

int Py_DebugFlag;
int Py_VerboseFlag;
int Py_IgnoreEnvironmentFlag;

/* Forward */
grammar *getgrammar(const char *filename);

void Py_Exit(int) _Py_NO_RETURN;

void
Py_Exit(int sts)
{
    exit(sts);
}

#ifdef WITH_THREAD
/* Needed by obmalloc.c */
int PyGILState_Check(void)
{ return 1; }
#endif

void _PyMem_DumpTraceback(int fd, const void *ptr)
{}

int
main(int argc, char **argv)
{
    grammar *g;
    FILE *fp;
    char *filename, *graminit_h, *graminit_c;

    if (argc != 4) {
        fprintf(stderr,
            "usage: %s grammar graminit.h graminit.c\n", argv[0]);
        Py_Exit(2);
    }
    filename = argv[1];
    graminit_h = argv[2];
    graminit_c = argv[3];
    g = getgrammar(filename);
    fp = fopen(graminit_c, "w");
    if (fp == NULL) {
        perror(graminit_c);
        Py_Exit(1);
    }
    if (Py_DebugFlag)
        printf("Writing %s ...\n", graminit_c);
    printgrammar(g, fp);
    fclose(fp);
    fp = fopen(graminit_h, "w");
    if (fp == NULL) {
        perror(graminit_h);
        Py_Exit(1);
    }
    if (Py_DebugFlag)
        printf("Writing %s ...\n", graminit_h);
    printnonterminals(g, fp);
    fclose(fp);
    freegrammar(g);
    Py_Exit(0);
    return 0; /* Make gcc -Wall happy */
}

grammar *
getgrammar(const char *filename)
{
    FILE *fp;
    node *n;
    grammar *g0, *g;
    perrdetail err;

    fp = fopen(filename, "r");
    if (fp == NULL) {
        perror(filename);
        Py_Exit(1);
    }
    g0 = meta_grammar();
    n = Ta3Parser_ParseFile(fp, filename, g0, g0->g_start,
                  (char *)NULL, (char *)NULL, &err);
    fclose(fp);
    if (n == NULL) {
        fprintf(stderr, "Parsing error %d, line %d.\n",
            err.error, err.lineno);
        if (err.text != NULL) {
            size_t len;
            int i;
            fprintf(stderr, "%s", err.text);
            len = strlen(err.text);
            if (len == 0 || err.text[len-1] != '\n')
                fprintf(stderr, "\n");
            for (i = 0; i < err.offset; i++) {
                if (err.text[i] == '\t')
                    putc('\t', stderr);
                else
                    putc(' ', stderr);
            }
            fprintf(stderr, "^\n");
            PyObject_FREE(err.text);
        }
        Py_Exit(1);
    }
    g = pgen(n);
    Ta3Node_Free(n);
    if (g == NULL) {
        printf("Bad grammar.\n");
        Py_Exit(1);
    }
    return g;
}

/* Can't happen in pgen */
PyObject*
PyErr_Occurred()
{
    return 0;
}

void
Py_FatalError(const char *msg)
{
    fprintf(stderr, "pgen: FATAL ERROR: %s\n", msg);
    Py_Exit(1);
}

/* No-nonsense my_readline() for tokenizer.c */

char *
PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt)
{
    size_t n = 1000;
    char *p = (char *)PyMem_MALLOC(n);
    char *q;
    if (p == NULL)
        return NULL;
    fprintf(stderr, "%s", prompt);
    q = fgets(p, n, sys_stdin);
    if (q == NULL) {
        *p = '\0';
        return p;
    }
    n = strlen(p);
    if (n > 0 && p[n-1] != '\n')
        p[n-1] = '\n';
    return (char *)PyMem_REALLOC(p, n+1);
}

/* No-nonsense fgets */
char *
Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj)
{
    return fgets(buf, n, stream);
}


#include <stdarg.h>

void
PySys_WriteStderr(const char *format, ...)
{
    va_list va;

    va_start(va, format);
    vfprintf(stderr, format, va);
    va_end(va);
}

/* Print a bunch of C initializers that represent a grammar */

#define PGEN

#include "pgenheaders.h"
#include "grammar.h"

/* Forward */
static void printarcs(int, dfa *, FILE *);
static void printstates(grammar *, FILE *);
static void printdfas(grammar *, FILE *);
static void printlabels(grammar *, FILE *);

void
printgrammar(grammar *g, FILE *fp)
{
    fprintf(fp, "/* Generated by Parser/pgen */\n\n");
    fprintf(fp, "#include \"pgenheaders.h\"\n");
    fprintf(fp, "#include \"grammar.h\"\n");
    fprintf(fp, "extern grammar _Ta3Parser_Grammar;\n");
    printdfas(g, fp);
    printlabels(g, fp);
    fprintf(fp, "grammar _Ta3Parser_Grammar = {\n");
    fprintf(fp, "    %d,\n", g->g_ndfas);
    fprintf(fp, "    dfas,\n");
    fprintf(fp, "    {%d, labels},\n", g->g_ll.ll_nlabels);
    fprintf(fp, "    %d\n", g->g_start);
    fprintf(fp, "};\n");
}

void
printnonterminals(grammar *g, FILE *fp)
{
    dfa *d;
    int i;

    fprintf(fp, "/* Generated by Parser/pgen */\n\n");

    d = g->g_dfa;
    for (i = g->g_ndfas; --i >= 0; d++)
        fprintf(fp, "#define %s %d\n", d->d_name, d->d_type);
}

static void
printarcs(int i, dfa *d, FILE *fp)
{
    arc *a;
    state *s;
    int j, k;

    s = d->d_state;
    for (j = 0; j < d->d_nstates; j++, s++) {
        fprintf(fp, "static arc arcs_%d_%d[%d] = {\n",
            i, j, s->s_narcs);
        a = s->s_arc;
        for (k = 0; k < s->s_narcs; k++, a++)
            fprintf(fp, "    {%d, %d},\n", a->a_lbl, a->a_arrow);
        fprintf(fp, "};\n");
    }
}

static void
printstates(grammar *g, FILE *fp)
{
    state *s;
    dfa *d;
    int i, j;

    d = g->g_dfa;
    for (i = 0; i < g->g_ndfas; i++, d++) {
        printarcs(i, d, fp);
        fprintf(fp, "static state states_%d[%d] = {\n",
            i, d->d_nstates);
        s = d->d_state;
        for (j = 0; j < d->d_nstates; j++, s++)
            fprintf(fp, "    {%d, arcs_%d_%d},\n",
                s->s_narcs, i, j);
        fprintf(fp, "};\n");
    }
}

static void
printdfas(grammar *g, FILE *fp)
{
    dfa *d;
    int i, j, n;

    printstates(g, fp);
    fprintf(fp, "static dfa dfas[%d] = {\n", g->g_ndfas);
    d = g->g_dfa;
    for (i = 0; i < g->g_ndfas; i++, d++) {
        fprintf(fp, "    {%d, \"%s\", %d, %d, states_%d,\n",
            d->d_type, d->d_name, d->d_initial, d->d_nstates, i);
        fprintf(fp, "     \"");
        n = NBYTES(g->g_ll.ll_nlabels);
        for (j = 0; j < n; j++)
            fprintf(fp, "\\%03o", d->d_first[j] & 0xff);
        fprintf(fp, "\"},\n");
    }
    fprintf(fp, "};\n");
}

static void
printlabels(grammar *g, FILE *fp)
{
    label *l;
    int i;

    fprintf(fp, "static label labels[%d] = {\n", g->g_ll.ll_nlabels);
    l = g->g_ll.ll_label;
    for (i = g->g_ll.ll_nlabels; --i >= 0; l++) {
        if (l->lb_str == NULL)
            fprintf(fp, "    {%d, 0},\n", l->lb_type);
        else
            fprintf(fp, "    {%d, \"%s\"},\n",
                l->lb_type, l->lb_str);
    }
    fprintf(fp, "};\n");
}
#include "Python.h"

/* Our own locale-independent ctype.h-like macros */

const unsigned int _Py_ctype_table[256] = {
    0, /* 0x0 '\x00' */
    0, /* 0x1 '\x01' */
    0, /* 0x2 '\x02' */
    0, /* 0x3 '\x03' */
    0, /* 0x4 '\x04' */
    0, /* 0x5 '\x05' */
    0, /* 0x6 '\x06' */
    0, /* 0x7 '\x07' */
    0, /* 0x8 '\x08' */
    PY_CTF_SPACE, /* 0x9 '\t' */
    PY_CTF_SPACE, /* 0xa '\n' */
    PY_CTF_SPACE, /* 0xb '\v' */
    PY_CTF_SPACE, /* 0xc '\f' */
    PY_CTF_SPACE, /* 0xd '\r' */
    0, /* 0xe '\x0e' */
    0, /* 0xf '\x0f' */
    0, /* 0x10 '\x10' */
    0, /* 0x11 '\x11' */
    0, /* 0x12 '\x12' */
    0, /* 0x13 '\x13' */
    0, /* 0x14 '\x14' */
    0, /* 0x15 '\x15' */
    0, /* 0x16 '\x16' */
    0, /* 0x17 '\x17' */
    0, /* 0x18 '\x18' */
    0, /* 0x19 '\x19' */
    0, /* 0x1a '\x1a' */
    0, /* 0x1b '\x1b' */
    0, /* 0x1c '\x1c' */
    0, /* 0x1d '\x1d' */
    0, /* 0x1e '\x1e' */
    0, /* 0x1f '\x1f' */
    PY_CTF_SPACE, /* 0x20 ' ' */
    0, /* 0x21 '!' */
    0, /* 0x22 '"' */
    0, /* 0x23 '#' */
    0, /* 0x24 '$' */
    0, /* 0x25 '%' */
    0, /* 0x26 '&' */
    0, /* 0x27 "'" */
    0, /* 0x28 '(' */
    0, /* 0x29 ')' */
    0, /* 0x2a '*' */
    0, /* 0x2b '+' */
    0, /* 0x2c ',' */
    0, /* 0x2d '-' */
    0, /* 0x2e '.' */
    0, /* 0x2f '/' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x30 '0' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x31 '1' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x32 '2' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x33 '3' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x34 '4' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x35 '5' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x36 '6' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x37 '7' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x38 '8' */
    PY_CTF_DIGIT|PY_CTF_XDIGIT, /* 0x39 '9' */
    0, /* 0x3a ':' */
    0, /* 0x3b ';' */
    0, /* 0x3c '<' */
    0, /* 0x3d '=' */
    0, /* 0x3e '>' */
    0, /* 0x3f '?' */
    0, /* 0x40 '@' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x41 'A' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x42 'B' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x43 'C' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x44 'D' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x45 'E' */
    PY_CTF_UPPER|PY_CTF_XDIGIT, /* 0x46 'F' */
    PY_CTF_UPPER, /* 0x47 'G' */
    PY_CTF_UPPER, /* 0x48 'H' */
    PY_CTF_UPPER, /* 0x49 'I' */
    PY_CTF_UPPER, /* 0x4a 'J' */
    PY_CTF_UPPER, /* 0x4b 'K' */
    PY_CTF_UPPER, /* 0x4c 'L' */
    PY_CTF_UPPER, /* 0x4d 'M' */
    PY_CTF_UPPER, /* 0x4e 'N' */
    PY_CTF_UPPER, /* 0x4f 'O' */
    PY_CTF_UPPER, /* 0x50 'P' */
    PY_CTF_UPPER, /* 0x51 'Q' */
    PY_CTF_UPPER, /* 0x52 'R' */
    PY_CTF_UPPER, /* 0x53 'S' */
    PY_CTF_UPPER, /* 0x54 'T' */
    PY_CTF_UPPER, /* 0x55 'U' */
    PY_CTF_UPPER, /* 0x56 'V' */
    PY_CTF_UPPER, /* 0x57 'W' */
    PY_CTF_UPPER, /* 0x58 'X' */
    PY_CTF_UPPER, /* 0x59 'Y' */
    PY_CTF_UPPER, /* 0x5a 'Z' */
    0, /* 0x5b '[' */
    0, /* 0x5c '\\' */
    0, /* 0x5d ']' */
    0, /* 0x5e '^' */
    0, /* 0x5f '_' */
    0, /* 0x60 '`' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x61 'a' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x62 'b' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x63 'c' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x64 'd' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x65 'e' */
    PY_CTF_LOWER|PY_CTF_XDIGIT, /* 0x66 'f' */
    PY_CTF_LOWER, /* 0x67 'g' */
    PY_CTF_LOWER, /* 0x68 'h' */
    PY_CTF_LOWER, /* 0x69 'i' */
    PY_CTF_LOWER, /* 0x6a 'j' */
    PY_CTF_LOWER, /* 0x6b 'k' */
    PY_CTF_LOWER, /* 0x6c 'l' */
    PY_CTF_LOWER, /* 0x6d 'm' */
    PY_CTF_LOWER, /* 0x6e 'n' */
    PY_CTF_LOWER, /* 0x6f 'o' */
    PY_CTF_LOWER, /* 0x70 'p' */
    PY_CTF_LOWER, /* 0x71 'q' */
    PY_CTF_LOWER, /* 0x72 'r' */
    PY_CTF_LOWER, /* 0x73 's' */
    PY_CTF_LOWER, /* 0x74 't' */
    PY_CTF_LOWER, /* 0x75 'u' */
    PY_CTF_LOWER, /* 0x76 'v' */
    PY_CTF_LOWER, /* 0x77 'w' */
    PY_CTF_LOWER, /* 0x78 'x' */
    PY_CTF_LOWER, /* 0x79 'y' */
    PY_CTF_LOWER, /* 0x7a 'z' */
    0, /* 0x7b '{' */
    0, /* 0x7c '|' */
    0, /* 0x7d '}' */
    0, /* 0x7e '~' */
    0, /* 0x7f '\x7f' */
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
};


const unsigned char _Py_ctype_tolower[256] = {
    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
    0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
    0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27,
    0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,
    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
    0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f,
    0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67,
    0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f,
    0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77,
    0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f,
    0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67,
    0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f,
    0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77,
    0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f,
    0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
    0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f,
    0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97,
    0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f,
    0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7,
    0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf,
    0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7,
    0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf,
    0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
    0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf,
    0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7,
    0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf,
    0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7,
    0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef,
    0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
    0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff,
};

const unsigned char _Py_ctype_toupper[256] = {
    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,
    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
    0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17,
    0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27,
    0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,
    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
    0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f,
    0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47,
    0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f,
    0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57,
    0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f,
    0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47,
    0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f,
    0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57,
    0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f,
    0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
    0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f,
    0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97,
    0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f,
    0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7,
    0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf,
    0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7,
    0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf,
    0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
    0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf,
    0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7,
    0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf,
    0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7,
    0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef,
    0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7,
    0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff,
};

#define PGEN
#include "tokenizer.c"
static void
    _Py_IDENTIFIER(_fields);
    fields = _PyObject_GetAttrId((PyObject*)Py_TYPE(self), &PyId__fields);
    if (!fields)
        PyErr_Clear();
    if (PyTuple_GET_SIZE(args) > 0) {
        if (numfields != PyTuple_GET_SIZE(args)) {
            PyErr_Format(PyExc_TypeError, "%.400s constructor takes %s"
                         "%zd positional argument%s",
                         Py_TYPE(self)->tp_name,
                         numfields == 0 ? "" : "either 0 or ",
                         numfields, numfields == 1 ? "" : "s");
        for (i = 0; i < PyTuple_GET_SIZE(args); i++) {
            /* cannot be reached when fields is NULL */
            PyObject *name = PySequence_GetItem(fields, i);
            if (!name) {
                res = -1;
                goto cleanup;
            }
            res = PyObject_SetAttr(self, name, PyTuple_GET_ITEM(args, i));
            Py_DECREF(name);
            if (res < 0)
                goto cleanup;
        }
    PyObject *res;
    PyObject *dict = _PyObject_GetAttrId(self, &PyId___dict__);
    if (dict == NULL) {
        if (PyErr_ExceptionMatches(PyExc_AttributeError))
            PyErr_Clear();
        else
            return NULL;
        res = Py_BuildValue("O()O", Py_TYPE(self), dict);
        Py_DECREF(dict);
        return res;
    result = PyObject_CallFunction((PyObject*)&PyType_Type, "s(O){sOss}",
                    type, base, "_fields", fnames, "__module__", "_ast3");
    _Py_IDENTIFIER(_attributes);
        PyDict_SetItemString(d, "_fields", empty_tuple) < 0 ||
        PyDict_SetItemString(d, "_attributes", empty_tuple) < 0) {
static int exists_not_none(PyObject *obj, _Py_Identifier *id)
{
    int isnone;
    PyObject *attr = _PyObject_GetAttrId(obj, id);
    if (!attr) {
        PyErr_Clear();
        return 0;
    }
    isnone = attr == Py_None;
    Py_DECREF(attr);
    return !isnone;
}

        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        Py_INCREF(Py_None);
        return Py_None;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Module");
        if (_PyObject_HasAttrId(obj, &PyId_type_ignores)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_ignores);
            if (tmp == NULL) goto failed;
                type_ignore_ty value;
                res = obj2ast_type_ignore(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(type_ignores, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"type_ignores\" missing from Module");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Interactive");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Expression");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_argtypes)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_argtypes);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(argtypes, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"argtypes\" missing from FunctionType");
        if (_PyObject_HasAttrId(obj, &PyId_returns)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_returns);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"returns\" missing from FunctionType");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Suite");
            return 1;
    if (_PyObject_HasAttrId(obj, &PyId_lineno)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from stmt");
    if (_PyObject_HasAttrId(obj, &PyId_col_offset)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from stmt");
        return 1;
        if (_PyObject_HasAttrId(obj, &PyId_name)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_name);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from FunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_args)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_args);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from FunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from FunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_decorator_list)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(decorator_list, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from FunctionDef");
        if (exists_not_none(obj, &PyId_returns)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_returns);
            if (tmp == NULL) goto failed;
        } else {
            returns = NULL;
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_name)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_name);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from AsyncFunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_args)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_args);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from AsyncFunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFunctionDef");
        if (_PyObject_HasAttrId(obj, &PyId_decorator_list)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(decorator_list, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from AsyncFunctionDef");
        if (exists_not_none(obj, &PyId_returns)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_returns);
            if (tmp == NULL) goto failed;
        } else {
            returns = NULL;
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_name)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_name);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from ClassDef");
        if (_PyObject_HasAttrId(obj, &PyId_bases)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_bases);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(bases, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"bases\" missing from ClassDef");
        if (_PyObject_HasAttrId(obj, &PyId_keywords)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_keywords);
            if (tmp == NULL) goto failed;
                keyword_ty value;
                res = obj2ast_keyword(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(keywords, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from ClassDef");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ClassDef");
        if (_PyObject_HasAttrId(obj, &PyId_decorator_list)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_decorator_list);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(decorator_list, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from ClassDef");
            return 1;
        if (exists_not_none(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            value = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_targets)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_targets);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(targets, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Delete");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_targets)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_targets);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(targets, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Assign");
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Assign");
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_target)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_target);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AugAssign");
        if (_PyObject_HasAttrId(obj, &PyId_op)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_op);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from AugAssign");
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from AugAssign");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_target)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_target);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AnnAssign");
        if (_PyObject_HasAttrId(obj, &PyId_annotation)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_annotation);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"annotation\" missing from AnnAssign");
        if (exists_not_none(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            value = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_simple)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_simple);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"simple\" missing from AnnAssign");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_target)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_target);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from For");
        if (_PyObject_HasAttrId(obj, &PyId_iter)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_iter);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from For");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from For");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(orelse, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from For");
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_target)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_target);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AsyncFor");
        if (_PyObject_HasAttrId(obj, &PyId_iter)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_iter);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from AsyncFor");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFor");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(orelse, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from AsyncFor");
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_test)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_test);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from While");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from While");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(orelse, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from While");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_test)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_test);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from If");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from If");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(orelse, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from If");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_items)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_items);
            if (tmp == NULL) goto failed;
                withitem_ty value;
                res = obj2ast_withitem(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(items, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from With");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from With");
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_items)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_items);
            if (tmp == NULL) goto failed;
                withitem_ty value;
                res = obj2ast_withitem(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(items, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from AsyncWith");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncWith");
        if (exists_not_none(obj, &PyId_type_comment)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
            if (tmp == NULL) goto failed;
        } else {
            type_comment = NULL;
        if (exists_not_none(obj, &PyId_exc)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_exc);
            if (tmp == NULL) goto failed;
        } else {
            exc = NULL;
        if (exists_not_none(obj, &PyId_cause)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_cause);
            if (tmp == NULL) goto failed;
        } else {
            cause = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Try");
        if (_PyObject_HasAttrId(obj, &PyId_handlers)) {
            int res;
            Py_ssize_t len;
            tmp = _PyObject_GetAttrId(obj, &PyId_handlers);
            if (tmp == NULL) goto failed;
                excepthandler_ty value;
                res = obj2ast_excepthandler(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(handlers, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"handlers\" missing from Try");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(orelse, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from Try");
        if (_PyObject_HasAttrId(obj, &PyId_finalbody)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_finalbody);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(finalbody, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"finalbody\" missing from Try");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_test)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_test);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from Assert");
        if (exists_not_none(obj, &PyId_msg)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_msg);
            if (tmp == NULL) goto failed;
        } else {
            msg = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_names)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_names);
            if (tmp == NULL) goto failed;
                alias_ty value;
                res = obj2ast_alias(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(names, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Import");
            return 1;
        if (exists_not_none(obj, &PyId_module)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_module);
            if (tmp == NULL) goto failed;
        } else {
            module = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_names)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_names);
            if (tmp == NULL) goto failed;
                alias_ty value;
                res = obj2ast_alias(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(names, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from ImportFrom");
        if (exists_not_none(obj, &PyId_level)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_level);
            if (tmp == NULL) goto failed;
        } else {
            level = 0;
        if (_PyObject_HasAttrId(obj, &PyId_names)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_names);
            if (tmp == NULL) goto failed;
                identifier value;
                res = obj2ast_identifier(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(names, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Global");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_names)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_names);
            if (tmp == NULL) goto failed;
                identifier value;
                res = obj2ast_identifier(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(names, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Nonlocal");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Expr");
            return 1;
    if (_PyObject_HasAttrId(obj, &PyId_lineno)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from expr");
    if (_PyObject_HasAttrId(obj, &PyId_col_offset)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from expr");
        return 1;
        if (_PyObject_HasAttrId(obj, &PyId_op)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_op);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BoolOp");
        if (_PyObject_HasAttrId(obj, &PyId_values)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_values);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(values, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from BoolOp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_left)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_left);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from BinOp");
        if (_PyObject_HasAttrId(obj, &PyId_op)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_op);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BinOp");
        if (_PyObject_HasAttrId(obj, &PyId_right)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_right);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"right\" missing from BinOp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_op)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_op);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from UnaryOp");
        if (_PyObject_HasAttrId(obj, &PyId_operand)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_operand);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"operand\" missing from UnaryOp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_args)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_args);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Lambda");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Lambda");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_test)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_test);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from IfExp");
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from IfExp");
        if (_PyObject_HasAttrId(obj, &PyId_orelse)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_orelse);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from IfExp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_keys)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_keys);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(keys, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"keys\" missing from Dict");
        if (_PyObject_HasAttrId(obj, &PyId_values)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_values);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(values, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from Dict");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elts)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elts);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(elts, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Set");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elt)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elt);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from ListComp");
        if (_PyObject_HasAttrId(obj, &PyId_generators)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_generators);
            if (tmp == NULL) goto failed;
                comprehension_ty value;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(generators, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from ListComp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elt)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elt);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from SetComp");
        if (_PyObject_HasAttrId(obj, &PyId_generators)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_generators);
            if (tmp == NULL) goto failed;
                comprehension_ty value;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(generators, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from SetComp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_key)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_key);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"key\" missing from DictComp");
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from DictComp");
        if (_PyObject_HasAttrId(obj, &PyId_generators)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_generators);
            if (tmp == NULL) goto failed;
                comprehension_ty value;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(generators, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from DictComp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elt)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elt);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from GeneratorExp");
        if (_PyObject_HasAttrId(obj, &PyId_generators)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_generators);
            if (tmp == NULL) goto failed;
                comprehension_ty value;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(generators, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from GeneratorExp");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Await");
            return 1;
        if (exists_not_none(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            value = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from YieldFrom");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_left)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_left);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from Compare");
        if (_PyObject_HasAttrId(obj, &PyId_ops)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ops);
            if (tmp == NULL) goto failed;
                cmpop_ty value;
                res = obj2ast_cmpop(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(ops, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ops\" missing from Compare");
        if (_PyObject_HasAttrId(obj, &PyId_comparators)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_comparators);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(comparators, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"comparators\" missing from Compare");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_func)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_func);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"func\" missing from Call");
        if (_PyObject_HasAttrId(obj, &PyId_args)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_args);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(args, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Call");
        if (_PyObject_HasAttrId(obj, &PyId_keywords)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_keywords);
            if (tmp == NULL) goto failed;
                keyword_ty value;
                res = obj2ast_keyword(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(keywords, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from Call");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_n)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_n);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"n\" missing from Num");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_s)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_s);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Str");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from FormattedValue");
        if (exists_not_none(obj, &PyId_conversion)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_conversion);
            if (tmp == NULL) goto failed;
        } else {
            conversion = 0;
        if (exists_not_none(obj, &PyId_format_spec)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_format_spec);
            if (tmp == NULL) goto failed;
        } else {
            format_spec = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_values)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_values);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(values, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from JoinedStr");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_s)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_s);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Bytes");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from NameConstant");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Constant");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Attribute");
        if (_PyObject_HasAttrId(obj, &PyId_attr)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_attr);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"attr\" missing from Attribute");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Attribute");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Subscript");
        if (_PyObject_HasAttrId(obj, &PyId_slice)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_slice);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"slice\" missing from Subscript");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Subscript");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Starred");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Starred");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_id)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_id);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"id\" missing from Name");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Name");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elts)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elts);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(elts, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from List");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from List");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_elts)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_elts);
            if (tmp == NULL) goto failed;
                expr_ty value;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(elts, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Tuple");
        if (_PyObject_HasAttrId(obj, &PyId_ctx)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_ctx);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Tuple");
            return 1;
        if (exists_not_none(obj, &PyId_lower)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_lower);
            if (tmp == NULL) goto failed;
        } else {
            lower = NULL;
        if (exists_not_none(obj, &PyId_upper)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_upper);
            if (tmp == NULL) goto failed;
        } else {
            upper = NULL;
        if (exists_not_none(obj, &PyId_step)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_step);
            if (tmp == NULL) goto failed;
        } else {
            step = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_dims)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_dims);
            if (tmp == NULL) goto failed;
                slice_ty value;
                res = obj2ast_slice(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(dims, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"dims\" missing from ExtSlice");
            return 1;
        if (_PyObject_HasAttrId(obj, &PyId_value)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_value);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Index");
            return 1;
    if (_PyObject_HasAttrId(obj, &PyId_target)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_target);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from comprehension");
    if (_PyObject_HasAttrId(obj, &PyId_iter)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_iter);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from comprehension");
    if (_PyObject_HasAttrId(obj, &PyId_ifs)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_ifs);
        if (tmp == NULL) goto failed;
            expr_ty value;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
            asdl_seq_SET(ifs, i, value);
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"ifs\" missing from comprehension");
    if (_PyObject_HasAttrId(obj, &PyId_is_async)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_is_async);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"is_async\" missing from comprehension");
        return 1;
    if (_PyObject_HasAttrId(obj, &PyId_lineno)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from excepthandler");
    if (_PyObject_HasAttrId(obj, &PyId_col_offset)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from excepthandler");
        return 1;
        if (exists_not_none(obj, &PyId_type)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_type);
            if (tmp == NULL) goto failed;
        } else {
            type = NULL;
        if (exists_not_none(obj, &PyId_name)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_name);
            if (tmp == NULL) goto failed;
        } else {
            name = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_body)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_body);
            if (tmp == NULL) goto failed;
                stmt_ty value;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &value, arena);
                asdl_seq_SET(body, i, value);
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ExceptHandler");
            return 1;
    if (_PyObject_HasAttrId(obj, &PyId_args)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_args);
        if (tmp == NULL) goto failed;
            arg_ty value;
            res = obj2ast_arg(PyList_GET_ITEM(tmp, i), &value, arena);
            asdl_seq_SET(args, i, value);
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from arguments");
    if (exists_not_none(obj, &PyId_vararg)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_vararg);
        if (tmp == NULL) goto failed;
    } else {
        vararg = NULL;
    if (_PyObject_HasAttrId(obj, &PyId_kwonlyargs)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_kwonlyargs);
        if (tmp == NULL) goto failed;
            arg_ty value;
            res = obj2ast_arg(PyList_GET_ITEM(tmp, i), &value, arena);
            asdl_seq_SET(kwonlyargs, i, value);
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"kwonlyargs\" missing from arguments");
    if (_PyObject_HasAttrId(obj, &PyId_kw_defaults)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_kw_defaults);
        if (tmp == NULL) goto failed;
            expr_ty value;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
            asdl_seq_SET(kw_defaults, i, value);
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"kw_defaults\" missing from arguments");
    if (exists_not_none(obj, &PyId_kwarg)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_kwarg);
        if (tmp == NULL) goto failed;
    } else {
        kwarg = NULL;
    if (_PyObject_HasAttrId(obj, &PyId_defaults)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_defaults);
        if (tmp == NULL) goto failed;
            expr_ty value;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &value, arena);
            asdl_seq_SET(defaults, i, value);
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"defaults\" missing from arguments");
        return 1;
    if (_PyObject_HasAttrId(obj, &PyId_arg)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_arg);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"arg\" missing from arg");
    if (exists_not_none(obj, &PyId_annotation)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_annotation);
        if (tmp == NULL) goto failed;
    } else {
        annotation = NULL;
    if (exists_not_none(obj, &PyId_type_comment)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_type_comment);
        if (tmp == NULL) goto failed;
    } else {
        type_comment = NULL;
    if (_PyObject_HasAttrId(obj, &PyId_lineno)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from arg");
    if (_PyObject_HasAttrId(obj, &PyId_col_offset)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_col_offset);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from arg");
        return 1;
    if (exists_not_none(obj, &PyId_arg)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_arg);
        if (tmp == NULL) goto failed;
    } else {
        arg = NULL;
    if (_PyObject_HasAttrId(obj, &PyId_value)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_value);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from keyword");
        return 1;
    if (_PyObject_HasAttrId(obj, &PyId_name)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_name);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from alias");
    if (exists_not_none(obj, &PyId_asname)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_asname);
        if (tmp == NULL) goto failed;
    } else {
        asname = NULL;
    if (_PyObject_HasAttrId(obj, &PyId_context_expr)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_context_expr);
        if (tmp == NULL) goto failed;
    } else {
        PyErr_SetString(PyExc_TypeError, "required field \"context_expr\" missing from withitem");
    if (exists_not_none(obj, &PyId_optional_vars)) {
        tmp = _PyObject_GetAttrId(obj, &PyId_optional_vars);
        if (tmp == NULL) goto failed;
    } else {
        optional_vars = NULL;
        if (_PyObject_HasAttrId(obj, &PyId_lineno)) {
            tmp = _PyObject_GetAttrId(obj, &PyId_lineno);
            if (tmp == NULL) goto failed;
        } else {
            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from TypeIgnore");
            return 1;
static struct PyModuleDef _astmodule3 = {
  PyModuleDef_HEAD_INIT, "_ast3", NULL, 0, ast3_methods
    m = PyModule_Create(&_astmodule3);
#if PY_MINOR_VERSION < 4
#define PyErr_ProgramTextObject PyErr_ProgramText
#define PyMem_RawMalloc PyMem_Malloc
#define PyMem_RawRealloc PyMem_Realloc
#define PyMem_RawFree PyMem_Free
        assert(0);
        return "(unknown)";
    PyObject *c_normalize_args; /* Normalization argument tuple. */
static asdl_seq *ast_for_suite(struct compiling *, const node *);
static stmt_ty ast_for_with_stmt(struct compiling *, const node *, int);
static stmt_ty ast_for_for_stmt(struct compiling *, const node *, int);
static expr_ty ast_for_call(struct compiling *, const node *, expr_ty);
    c->c_normalize_args = Py_BuildValue("(sN)", "NFKC", Py_None);
    if (!c->c_normalize_args) {
        Py_CLEAR(c->c_normalize);
        return 0;
    }
    PyTuple_SET_ITEM(c->c_normalize_args, 1, NULL);
        PyTuple_SET_ITEM(c->c_normalize_args, 1, id);
        id2 = PyObject_Call(c->c_normalize, c->c_normalize_args, NULL);
    assert(0);
    return 0;
                     PyObject *filename, int feature_version,
                     PyArena *arena)
    c.c_normalize_args = NULL;
                    goto out;
                asdl_seq_SET(type_ignores, i, ti);
        PyTuple_SET_ITEM(c.c_normalize_args, 1, NULL);
        Py_DECREF(c.c_normalize_args);
               int feature_version, PyArena *arena)
    if (PyUnicode_CompareWithASCIIString(name, "__debug__") == 0) {
            if (PyUnicode_CompareWithASCIIString(name, *p) == 0) {
                if (i < NCH(n) && TYPE(CHILD(n, i)) == COMMA)
    if (nposargs + nkwonlyargs > 255) {
        ast_error(c, n, "more than 255 arguments");
        return NULL;
    }

                if (i < NCH(n) && TYPE(CHILD(n, i)) == COMMA)
                    if (i < NCH(n) && TYPE(CHILD(n, i)) == TYPE_COMMENT) {
                    i += 2; /* the star and the name */
                    if (i < NCH(n) && TYPE(CHILD(n, i)) == COMMA)
                        i += 1; /* the comma, if present */
                    if (i < NCH(n) && TYPE(CHILD(n, i)) == TYPE_COMMENT) {
                if (i < NCH(n) && TYPE(CHILD(n, i)) == COMMA)
        d = ast_for_call(c, CHILD(n, 3), name_expr);
ast_for_funcdef_impl(struct compiling *c, const node *n,
                     asdl_seq *decorator_seq, int is_async)
                "Async functions are only supported in Python 3.5 and greater");
                                type_comment, LINENO(n),
                                n->n_col_offset, c->c_arena);
                           type_comment, LINENO(n),
                           n->n_col_offset, c->c_arena);
    /* async_funcdef: ASYNC funcdef */
    REQ(CHILD(n, 0), ASYNC);
    return ast_for_funcdef_impl(c, CHILD(n, 1), decorator_seq,
                                1 /* is_async */);
                                0 /* is_async */);
    /* async_stmt: ASYNC (funcdef | with_stmt | for_stmt) */
    REQ(CHILD(n, 0), ASYNC);
            return ast_for_funcdef_impl(c, CHILD(n, 1), NULL,
                                        1 /* is_async */);
            return ast_for_with_stmt(c, CHILD(n, 1),
                                     1 /* is_async */);
            return ast_for_for_stmt(c, CHILD(n, 1),
                                    1 /* is_async */);
    int is_async;
    is_async = 0;
    if (TYPE(CHILD(n, 0)) == ASYNC) {
        is_async = 1;
    if (NCH(n) == (5 + is_async)) {
        n = CHILD(n, 4 + is_async);
        if (TYPE(CHILD(n, 0)) == ASYNC) {
        for_ch = CHILD(n, 1 + is_async);
        expression = ast_for_expr(c, CHILD(n, 3 + is_async));
        if (NCH(n) == (5 + is_async)) {
            n = CHILD(n, 4 + is_async);
        pynum = parsenumber(c, s);
            return ast_for_call(c, CHILD(n, 1), left_expr);
        /* there was an AWAIT */
       atom_expr: [AWAIT] atom trailer*
ast_for_call(struct compiling *c, const node *n, expr_ty func)
    int i, nargs, nkeywords, ngens;
    ngens = 0;
            else if (TYPE(CHILD(ch, 1)) == comp_for)
                ngens++;
    if (ngens > 1 || (ngens && (nargs || nkeywords))) {
        ast_error(c, n, "Generator expression must be parenthesized "
                  "if not sole argument");
        return NULL;
    }
    if (nargs + nkeywords + ngens > 255) {
        ast_error(c, n, "more than 255 arguments");
        return NULL;
    }

    args = _Ta3_asdl_seq_new(nargs + ngens, c->c_arena);
ast_for_for_stmt(struct compiling *c, const node *n, int is_async)
        return AsyncFor(target, expression, suite_seq, seq,
                        type_comment, LINENO(n), n->n_col_offset,
        return For(target, expression, suite_seq, seq,
                   type_comment, LINENO(n), n->n_col_offset,
ast_for_with_stmt(struct compiling *c, const node *n, int is_async)
        return AsyncWith(items, body, type_comment, LINENO(n), n->n_col_offset, c->c_arena);
        return ClassDef(classname, NULL, NULL, s, decorator_seq, LINENO(n),
                        n->n_col_offset, c->c_arena);
        s = ast_for_suite(c, CHILD(n,5));
        return ClassDef(classname, NULL, NULL, s, decorator_seq, LINENO(n),
                        n->n_col_offset, c->c_arena);
        call = ast_for_call(c, CHILD(n, 3), dummy);
    PyObject *u;
            if (*s & 0x80) {
            assert(p - buf <= Py_SIZE(u));
    return PyUnicode_DecodeUnicodeEscape(s, len, NULL);
    return PyBytes_DecodeEscape(s, len, NULL, 0, NULL);
    int all_whitespace = 1;
    int kind;
    void *data;
    PyObject *o, *fstring_name;
    Py_ssize_t i;
    /* We know there are no escapes here, because backslashes are not allowed,
       and we know it's utf-8 encoded (per PEP 263).  But, in order to check
       that each char is not whitespace, we need to decode it to unicode.
       Which is unfortunate, but such is life. */

    /* If the substring is all whitespace, it's an error.  We need to catch
       this here, and not when we call PyParser_ASTFromString, because turning
       the expression '' in to '()' would go from being invalid to valid. */
    /* Note that this code says an empty string is all whitespace.  That's
       important.  There's a test for it: f'{}'. */
    o = PyUnicode_DecodeUTF8(expr_start, expr_end-expr_start, NULL);
    if (o == NULL)
        return NULL;
    len = PyUnicode_GET_LENGTH(o);
    kind = PyUnicode_KIND(o);
    data = PyUnicode_DATA(o);
    for (i = 0; i < len; i++) {
        if (!Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, i))) {
            all_whitespace = 0;
    Py_DECREF(o);
    if (all_whitespace) {
    /* Reuse len to be the length of the utf-8 input string. */
    if (str == NULL)
    const char *literal_start = *str;
    const char *literal_end;
    int in_named_escape = 0;
    for (; *str < end; (*str)++) {
        char ch = **str;
        if (!in_named_escape && ch == '{' && (*str)-literal_start >= 2 &&
            *(*str-2) == '\\' && *(*str-1) == 'N') {
            in_named_escape = 1;
        } else if (in_named_escape && ch == '}') {
            in_named_escape = 0;
        } else if (ch == '{' || ch == '}') {
                if (*str+1 < end && *(*str+1) == ch) {
                    literal_end = *str+1;
                    *str += 2;
    literal_end = *str;
    assert(*str <= end);
    assert(*str == end || **str == '{' || **str == '}');
    if (literal_start != literal_end) {
                                                    literal_end-literal_start,
                                                   literal_end-literal_start);
        assert(!state->last_str ||
               PyUnicode_GET_LENGTH(state->last_str) != 0);
    if(state->expr_list.size == 0) {
    /* If there's only one expression, return it. Otherwise, we need
       to join them together. */
    if (seq->size == 1)
        return seq->elements[0];

static arc arcs_81_0[2] = {
    {21, 1},
    {102, 2},
    {102, 2},
    {67, 3},
    {103, 4},
static arc arcs_81_4[1] = {
    {113, 5},
static arc arcs_81_5[2] = {
    {172, 6},
static arc arcs_81_6[1] = {
    {0, 6},
};
static state states_81[7] = {
    {2, arcs_81_0},
    {1, arcs_81_4},
    {2, arcs_81_5},
    {1, arcs_81_6},
static arc arcs_82_0[1] = {
    {98, 1},
    {115, 2},
static arc arcs_82_2[2] = {
    {172, 3},
static arc arcs_82_3[1] = {
    {0, 3},
};
static state states_82[4] = {
    {1, arcs_82_0},
    {2, arcs_82_2},
    {1, arcs_82_3},
    {23, 1},
    {0, 1},
static state states_83[2] = {
    {175, 1},
static arc arcs_84_1[2] = {
    {176, 2},
static arc arcs_84_2[1] = {
    {0, 2},
};
static state states_84[3] = {
    {2, arcs_84_1},
    {1, arcs_84_2},
static arc arcs_85_0[2] = {
    {78, 1},
    {9, 2},
static arc arcs_85_1[1] = {
    {26, 2},
    {2, arcs_85_0},
    {1, arcs_85_1},
static arc arcs_86_0[1] = {
    {178, 1},
static arc arcs_86_1[2] = {
    {2, 1},
    {7, 2},
    {1, arcs_86_0},
    {2, arcs_86_1},
    {13, 1},
    {179, 2},
    {15, 3},
static arc arcs_87_3[1] = {
static arc arcs_87_4[1] = {
static arc arcs_87_5[1] = {
static state states_87[6] = {
    {1, arcs_87_0},
    {2, arcs_87_1},
    {1, arcs_87_2},
    {1, arcs_87_3},
    {1, arcs_87_4},
    {1, arcs_87_5},
static arc arcs_88_0[3] = {
static arc arcs_88_1[2] = {
static arc arcs_88_2[3] = {
static arc arcs_88_3[1] = {
static arc arcs_88_4[4] = {
static arc arcs_88_5[2] = {
static arc arcs_88_6[2] = {
static arc arcs_88_7[1] = {
static arc arcs_88_8[3] = {
static arc arcs_88_9[2] = {
static arc arcs_88_10[2] = {
static state states_88[11] = {
    {3, arcs_88_0},
    {2, arcs_88_1},
    {3, arcs_88_2},
    {1, arcs_88_3},
    {4, arcs_88_4},
    {2, arcs_88_5},
    {2, arcs_88_6},
    {1, arcs_88_7},
    {3, arcs_88_8},
    {2, arcs_88_9},
    {2, arcs_88_10},
};
static dfa dfas[89] = {
     "\004\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\204\000"},
     "\204\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\204\000"},
     "\000\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\204\000"},
     "\000\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\200\000"},
     "\000\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\200\000"},
     "\000\000\000\000\000\000\000\000\000\074\000\000\000\000\000\000\000\000\000\000\000\200\000"},
     "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000"},
     "\004\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\200\000"},
    {337, "comp_for", 0, 7, states_81,
    {338, "comp_if", 0, 4, states_82,
    {339, "encoding_decl", 0, 2, states_83,
    {340, "yield_expr", 0, 3, states_84,
     "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000"},
    {341, "yield_arg", 0, 3, states_85,
    {342, "func_type_input", 0, 3, states_86,
    {343, "func_type", 0, 6, states_87,
    {344, "typelist", 0, 11, states_88,
static label labels[180] = {
    {340, 0},
    {337, 0},
    {338, 0},
    {341, 0},
    89,
    {180, labels},
gcc -I Parser -I Include $(python3-config --includes) \
    -o pgen.out \
    Parser/acceler.c \
    Parser/grammar1.c \
    Parser/node.c \
    Parser/parser.c \
    Parser/bitset.c \
    Parser/grammar.c \
    Pgen/listnode.c \
    Pgen/metagrammar.c \
    Pgen/firstsets.c \
    Pgen/pgen.c \
    Pgen/obmalloc.c \
    Pgen/dynamic_annotations.c \
    Pgen/mysnprintf.c \
    Pgen/pyctype.c \
    Pgen/tokenizer_pgen.c \
    Pgen/printgrammar.c \
    Pgen/parsetok_pgen.c \
    Pgen/pgenmain.c


MAX_VER = 6
        'ast3/Include/compile.h',
gobjdump -t $PROJ_DIR/build/lib*/_ast${1}.*.so | grep ' g ' | grep -v UND > "exported_symbols${1}.txt"
echo "Symbols written to exported_symbols${1}.txt.  You should edit this file to "
echo "remove any symbols you still want to export (like PyInit functions) "
echo "and to make each line contain only a function name you want updated "
echo "(and none of the other output) before running update_exported_symbols."
python3 ast3/Parser/asdl_c.py -h ast3/Include/ ast3/Parser/Python.asdl
python3 ast3/Parser/asdl_c.py -c ast3/Python/ ast3/Parser/Python.asdl
echo 'Compiling pgen'
gcc -I ast3/Parser -I ast3/Include $(python3-config --includes) \
    -o tools/pgen3 \
    ast3/Parser/acceler.c \
    ast3/Parser/grammar1.c \
    ast3/Parser/node.c \
    ast3/Parser/parser.c \
    ast3/Parser/bitset.c \
    ast3/Parser/grammar.c \
    ast3/Pgen/listnode.c \
    ast3/Pgen/metagrammar.c \
    ast3/Pgen/firstsets.c \
    ast3/Pgen/pgen.c \
    ast3/Pgen/obmalloc.c \
    ast3/Pgen/dynamic_annotations.c \
    ast3/Pgen/mysnprintf.c \
    ast3/Pgen/pyctype.c \
    ast3/Pgen/tokenizer_pgen.c \
    ast3/Pgen/printgrammar.c \
    ast3/Pgen/parsetok_pgen.c \
    ast3/Pgen/pgenmain.c

__version__ = "1.2.1-dev"
LATEST_MINOR_VERSION = 6
  1. Compile the module with `python3 setup.py build`.
  2. Generate the list of exported symbols with `./tools/find_exported_symbols 3`.
     The script may require updating to work on your platform, but should serve
     as a useful guide at minimum.
  3. The exported symbols will be written to `exported_symbols3.txt`.  Make
     sure this file looks sane, then remove `_PyInit__ast3` (which we want to
     export) and delete the excess output at the beginning of each line (including
     the leading `_` of each symbol) to end up with a list of function names to
     change.
  4. Run `./tools/update_exported_symbols 3`, which updates the exported
     symbols in all the `ast3` files with sed.  It may take a few seconds to run.
     If you're on Linux, the script will need some very slight modification to
     work properly (due to cross-platform sed argument differences).  Verify the
     changes look sane.
     ([d1ec7d0](https://github.com/python/typed_ast/commit/d1ec7d07cb6a7fe016d9446a196dfa3b86c5acf6))
  5. Update `Parser/asdl_c.py`.  Use the changes from git history to guide you.
     ([29dbec4](https://github.com/python/typed_ast/commit/29dbec47aa145d84e5faaa431ce3b3afca233b3d))
6. Add `Custom/typed_ast.c` back to setup.py.  Temporarily remove references to
   `TYPE_COMMENT` and `Py_func_type_input` from `Custom/typed_ast.c` to allow it
   to compile.
   guide you here.  Diffing the previous `ast3` against it's external symbol
11. Make the changes necessary so `ast3` can compile on the previous Python
    version.  This is new territory every time.  Spelunking in the rest of the
    details.  The changes in the previous `ast3` should likely suffice here.

Fixed code:
                    const char *s = "";
                const char *tmp;
#include "compile-ast3.h"
    node *n;
    if (feature_version >= 7)
        iflags |= PyPARSE_ASYNC_ALWAYS;
    n = Ta3Parser_ParseStringObject(s, filename,
                                    &_Ta3Parser_Grammar, start, &err,
                                    &iflags);
# https://devguide.python.org/grammar/
sync_comp_for: 'for' exprlist 'in' or_test [comp_iter]
comp_for: [ASYNC] sync_comp_for





























































expr_ty _Ta3_Str(string s, string kind, int lineno, int col_offset, PyArena *arena);
#ifndef Py_LIMITED_API

/* _PyAST_ExprAsUnicode is defined in ast_unparse.c */
extern PyObject * _PyAST_ExprAsUnicode(expr_ty);

#endif /* !Py_LIMITED_API */

#define BYTE            char
#define BITSPERBYTE     (8*sizeof(BYTE))
#define NBYTES(nbits)   (((nbits) + BITSPERBYTE - 1) / BITSPERBYTE)
#define BIT2BYTE(ibit)  ((ibit) / BITSPERBYTE)
#define BIT2SHIFT(ibit) ((ibit) % BITSPERBYTE)
#define BIT2MASK(ibit)  (1 << BIT2SHIFT(ibit))
#define BYTE2BIT(ibyte) ((ibyte) * BITSPERBYTE)
#define Py_func_type_input 343
#define E_OK            10      /* No error */
#define E_EOF           11      /* End Of File */
#define E_INTR          12      /* Interrupted */
#define E_TOKEN         13      /* Bad token */
#define E_SYNTAX        14      /* Syntax error */
#define E_NOMEM         15      /* Ran out of memory */
#define E_DONE          16      /* Parsing complete */
#define E_ERROR         17      /* Execution error */
#define E_TABSPACE      18      /* Inconsistent mixing of tabs and spaces */
#define E_OVERFLOW      19      /* Node had too many children */
#define E_TOODEEP       20      /* Too many indentation levels */
#define E_DEDENT        21      /* No matching outer block for dedent */
#define E_DECODE        22      /* Error in decoding into Unicode */
#define E_EOFS          23      /* EOF in triple-quoted string */
#define E_EOLS          24      /* EOL in single-quoted string */
#define E_LINECONT      25      /* Unexpected characters after a line continuation */
#define E_BADSINGLE     27      /* Ill-formed single statement input */
#define sync_comp_for 337
#define comp_for 338
#define comp_if 339
#define encoding_decl 340
#define yield_expr 341
#define yield_arg 342
#define func_type_input 343
#define func_type 344
#define typelist 345
    int          lb_type;
    char        *lb_str;
#define EMPTY 0         /* Label number 0 is by definition the empty label */
    int          ll_nlabels;
    label       *ll_label;
    short       a_lbl;          /* Label of this arc */
    short       a_arrow;        /* State where this arc goes to */
    int          s_narcs;
    arc         *s_arc;         /* Array of arcs */
    int          s_lower;       /* Lowest label index */
    int          s_upper;       /* Highest label index */
    int         *s_accel;       /* Accelerator */
    int          s_accept;      /* Nonzero for accepting state */
    int          d_type;        /* Non-terminal this represents */
    char        *d_name;        /* For printing */
    int          d_initial;     /* Initial state */
    int          d_nstates;
    state       *d_state;       /* Array of states */
    bitset       d_first;
    int          g_ndfas;
    dfa         *g_dfa;         /* Array of DFAs */
    labellist    g_ll;
    int          g_start;       /* Start symbol of the grammar */
    int          g_accel;       /* Set if accelerators present */
    short               n_type;
    char                *n_str;
    int                 n_lineno;
    int                 n_col_offset;
    int                 n_nchildren;
    struct _node        *n_child;
extern node * Ta3Node_New(int type);
#define NCH(n)          ((n)->n_nchildren)
#define CHILD(n, i)     (&(n)->n_child[i])
#define RCHILD(n, i)    (CHILD(n, NCH(n) + i))
#define TYPE(n)         ((n)->n_type)
#define STR(n)          ((n)->n_str)
#define PyPARSE_YIELD_IS_KEYWORD        0x0001
#define PyPARSE_DONT_IMPLY_DEDENT       0x0002
#define PyPARSE_WITH_IS_KEYWORD         0x0003
#define PyPARSE_ASYNC_ALWAYS  0x8000
extern void PyParser_SetError(perrdetail *);
extern void PyParser_ClearError(perrdetail *);
#define ENDMARKER       0
#define NAME            1
#define NUMBER          2
#define STRING          3
#define NEWLINE         4
#define INDENT          5
#define DEDENT          6
#define LPAR            7
#define RPAR            8
#define LSQB            9
#define RSQB            10
#define COLON           11
#define COMMA           12
#define SEMI            13
#define PLUS            14
#define MINUS           15
#define STAR            16
#define SLASH           17
#define VBAR            18
#define AMPER           19
#define LESS            20
#define GREATER         21
#define EQUAL           22
#define DOT             23
#define PERCENT         24
#define LBRACE          25
#define RBRACE          26
#define EQEQUAL         27
#define NOTEQUAL        28
#define LESSEQUAL       29
#define GREATEREQUAL    30
#define TILDE           31
#define CIRCUMFLEX      32
#define LEFTSHIFT       33
#define RIGHTSHIFT      34
#define DOUBLESTAR      35
#define PLUSEQUAL       36
#define MINEQUAL        37
#define STAREQUAL       38
#define SLASHEQUAL      39
#define PERCENTEQUAL    40
#define AMPEREQUAL      41
#define VBAREQUAL       42
#define CIRCUMFLEXEQUAL 43
#define LEFTSHIFTEQUAL  44
#define RIGHTSHIFTEQUAL 45
#define DOUBLESTAREQUAL 46
#define DOUBLESLASH     47
#define ATEQUAL         50
#define OP              53
#define AWAIT           54
#define ASYNC           55
#define ERRORTOKEN      58
/* These aren't used by the C tokenizer but are needed for tokenize.py */
#define COMMENT         59
#define NL              60
#define ENCODING                61
#define N_TOKENS        62
#define NT_OFFSET               256
#define ISTERMINAL(x)           ((x) < NT_OFFSET)
#define ISNONTERMINAL(x)        ((x) >= NT_OFFSET)
#define ISEOF(x)                ((x) == ENDMARKER)
extern const char * _Ta3Parser_TokenNames[]; /* Token names */
         | Str(string s, string kind) -- need to specify raw, unicode, etc?
            if line:
                line = (" " * TABSIZE * depth) + line
            self.file.write(line + "\n")
        self.emit("if (lookup_attr_id(obj, &PyId_%s, &tmp) < 0) {" % field.name, depth)
        self.emit("return 1;", depth+1)
        self.emit("}", depth)
        if not field.opt:
            self.emit("if (tmp == NULL) {", depth)
            message = "required field \\\"%s\\\" missing from %s" % (field.name, name)
            format = "PyErr_SetString(PyExc_TypeError, \"%s\");"
            self.emit(format % message, depth+1, reflow=False)
            self.emit("return 1;", depth+1)
            self.emit("if (tmp == NULL || tmp == Py_None) {", depth)
            self.emit("Py_CLEAR(tmp);", depth+1)
            if self.isNumeric(field):
                self.emit("%s = 0;" % field.name, depth+1)
            elif not self.isSimpleType(field):
                self.emit("%s = NULL;" % field.name, depth+1)
            else:
                raise TypeError("could not determine the default value for %s" % field.name)
        self.emit("}", depth)
        self.emit("else {", depth)

            self.emit("%s val;" % ctype, depth+2)
            self.emit("res = obj2ast_%s(PyList_GET_ITEM(tmp, i), &val, arena);" %
            self.emit("asdl_seq_SET(%s, i, val);" % field.name, depth+2)
_Py_IDENTIFIER(_fields);
_Py_IDENTIFIER(_attributes);

    /* bpo-31095: UnTrack is needed before calling any callbacks */
    PyObject_GC_UnTrack(self);
static int
    return 0;
}

static int lookup_attr_id(PyObject *v, _Py_Identifier *name, PyObject **result)
{
    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
    if (!oname) {
        *result = NULL;
        return -1;
    }
    *result = PyObject_GetAttr(v, oname);
    if (*result == NULL) {
        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
            return -1;
        }
        PyErr_Clear();
    }
    return 0;
    if (lookup_attr_id((PyObject*)Py_TYPE(self), &PyId__fields, &fields) < 0) {
        goto cleanup;
    }

    if (numfields < PyTuple_GET_SIZE(args)) {
        PyErr_Format(PyExc_TypeError, "%.400s constructor takes at most "
                     "%zd positional argument%s",
                     Py_TYPE(self)->tp_name,
                     numfields, numfields == 1 ? "" : "s");
        res = -1;
        goto cleanup;
    }
    for (i = 0; i < PyTuple_GET_SIZE(args); i++) {
        /* cannot be reached when fields is NULL */
        PyObject *name = PySequence_GetItem(fields, i);
        if (!name) {
        res = PyObject_SetAttr(self, name, PyTuple_GET_ITEM(args, i));
        Py_DECREF(name);
        if (res < 0)
            goto cleanup;
    PyObject *dict;
    if (lookup_attr_id(self, &PyId___dict__, &dict) < 0) {
        return NULL;
        return Py_BuildValue("O()N", Py_TYPE(self), dict);
    _Py_IDENTIFIER(__module__);
    _Py_IDENTIFIER(_ast3);
    result = PyObject_CallFunction((PyObject*)&PyType_Type, "s(O){OOOO}",
                    type, base,
                    _PyUnicode_FromId(&PyId__fields), fnames,
                    _PyUnicode_FromId(&PyId___module__),
                    _PyUnicode_FromId(&PyId__ast3));
        _PyDict_SetItemId(d, &PyId__fields, empty_tuple) < 0 ||
        _PyDict_SetItemId(d, &PyId__attributes, empty_tuple) < 0) {
        self.emit("PyObject *ast3_parse(PyObject *self, PyObject *args);", 0)
        self.emit("static PyMethodDef ast3_methods[] = {", 0)
        self.emit('    {"_parse",  ast3_parse, METH_VARARGS, "Parse string into typed AST."},', 0)
        self.emit("    {NULL, NULL, 0, NULL}", 0)
        self.emit("};", 0)
        self.emit("static struct PyModuleDef _astmodule = {", 0)
        self.emit('    PyModuleDef_HEAD_INIT, "_ast3", NULL, 0, ast3_methods', 0)
        self.emit('m = PyModule_Create(&_astmodule);', 1)
        self.emit("Py_RETURN_NONE;", 2)
    if H_FILE:
        with open(H_FILE, "w") as f:
            f.write(auto_gen_msg)
            f.write('#include "asdl.h"\n\n')
            c = ChainOfVisitors(TypeDefVisitor(f),
                                StructVisitor(f),
                                PrototypeVisitor(f),
                                )
            c.visit(mod)
            f.write("PyObject* Ta3AST_mod2obj(mod_ty t);\n")
            f.write("mod_ty Ta3AST_obj2mod(PyObject* ast, PyArena* arena, int mode);\n")
            f.write("int Ta3AST_Check(PyObject* obj);\n")

    if C_FILE:
        with open(C_FILE, "w") as f:
            f.write(auto_gen_msg)
            f.write('#include <stddef.h>\n')
            f.write('\n')
            f.write('#include "Python.h"\n')
            f.write('#include "%s-ast.h"\n' % mod.name)
            f.write('\n')
            f.write("static PyTypeObject AST_type;\n")
            v = ChainOfVisitors(
                PyTypesDeclareVisitor(f),
                PyTypesVisitor(f),
                Obj2ModPrototypeVisitor(f),
                FunctionVisitor(f),
                ObjVisitor(f),
                Obj2ModVisitor(f),
                ASTModuleVisitor(f),
                PartingShots(f),
                )
            v.visit(mod)
    H_FILE = ''
    C_FILE = ''
            H_FILE = v
            C_FILE = v
    if H_FILE and C_FILE:
    abort();
    int              s_state;       /* State in current DFA */
    dfa             *s_dfa;         /* Current DFA */
    struct _node    *s_parent;      /* Where to add next node */
    stackentry      *s_top;         /* Top entry */
    stackentry       s_base[MAXSTACK];/* Array of stack entries */
                                    /* NB The stack grows down */
    stack           p_stack;        /* Stack of parser states */
    grammar         *p_grammar;     /* Grammar to use */
    node            *p_tree;        /* Top of parse tree */
    unsigned long   p_flags;        /* see co_flags in Include/code.h */
    if (*flags & PyPARSE_ASYNC_ALWAYS)
        tok->async_always = 1;
        len = (a != NULL && b != NULL) ? b - a : 0;
                err_ret->expected = NOTEQUAL;
        if (a != NULL && a >= tok->line_start) {
        }
        else {
        }
#ifndef _PyObject_CallNoArg
#define _PyObject_CallNoArg(func) PyObject_CallObject(func, NULL)
#endif

/* Alternate tab spacing */
#define ALTTABSIZE 1

    "COMMENT",
    "NL",
    "ENCODING",
    tok->async_always = 0;
        bufobj = _PyObject_CallNoArg(tok->decoding_readline);
        PyObject *bufobj = _PyObject_CallNoArg(readline);
            buf = _PyObject_CallNoArg(tok->decoding_readline);
                if (newtok == NULL) {
                    Py_DECREF(u);
                    tok->done = E_NOMEM;
                    return EOF;
                }
    tok->done = E_TABSPACE;
    tok->cur = tok->inp;
    return ERRORTOKEN;
                col = (col / tok->tabsize + 1) * tok->tabsize;
                altcol = (altcol / ALTTABSIZE + 1) * ALTTABSIZE;
                    return indenterror(tok);
                    return indenterror(tok);
                    return indenterror(tok);
        /* Process the various legal combinations of b"", r"", u"", and f"". */
            if (tok->async_always || tok->async_def) {
    int async_always;     /* =1 if async/await are always keywords */
_Py_IDENTIFIER(_fields);
_Py_IDENTIFIER(_attributes);

    /* bpo-31095: UnTrack is needed before calling any callbacks */
    PyObject_GC_UnTrack(self);
static int
    return 0;
}

static int lookup_attr_id(PyObject *v, _Py_Identifier *name, PyObject **result)
{
    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
    if (!oname) {
        *result = NULL;
        return -1;
    }
    *result = PyObject_GetAttr(v, oname);
    if (*result == NULL) {
        if (!PyErr_ExceptionMatches(PyExc_AttributeError)) {
            return -1;
        }
        PyErr_Clear();
    }
    return 0;
    if (lookup_attr_id((PyObject*)Py_TYPE(self), &PyId__fields, &fields) < 0) {
        goto cleanup;
    }

    if (numfields < PyTuple_GET_SIZE(args)) {
        PyErr_Format(PyExc_TypeError, "%.400s constructor takes at most "
                     "%zd positional argument%s",
                     Py_TYPE(self)->tp_name,
                     numfields, numfields == 1 ? "" : "s");
        res = -1;
        goto cleanup;
    }
    for (i = 0; i < PyTuple_GET_SIZE(args); i++) {
        /* cannot be reached when fields is NULL */
        PyObject *name = PySequence_GetItem(fields, i);
        if (!name) {
        res = PyObject_SetAttr(self, name, PyTuple_GET_ITEM(args, i));
        Py_DECREF(name);
        if (res < 0)
            goto cleanup;
    PyObject *dict;
    if (lookup_attr_id(self, &PyId___dict__, &dict) < 0) {
        return NULL;
        return Py_BuildValue("O()N", Py_TYPE(self), dict);
    _Py_IDENTIFIER(__module__);
    _Py_IDENTIFIER(_ast3);
    result = PyObject_CallFunction((PyObject*)&PyType_Type, "s(O){OOOO}",
                    type, base,
                    _PyUnicode_FromId(&PyId__fields), fnames,
                    _PyUnicode_FromId(&PyId___module__),
                    _PyUnicode_FromId(&PyId__ast3));
        _PyDict_SetItemId(d, &PyId__fields, empty_tuple) < 0 ||
        _PyDict_SetItemId(d, &PyId__attributes, empty_tuple) < 0) {
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        Py_RETURN_NONE;
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Module");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_type_ignores, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"type_ignores\" missing from Module");
        else {
                type_ignore_ty val;
                res = obj2ast_type_ignore(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(type_ignores, i, val);
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Interactive");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Expression");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_argtypes, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"argtypes\" missing from FunctionType");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(argtypes, i, val);
        }
        if (lookup_attr_id(obj, &PyId_returns, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"returns\" missing from FunctionType");
        else {
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Suite");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
    if (lookup_attr_id(obj, &PyId_lineno, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from stmt");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_col_offset, &tmp) < 0) {
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from stmt");
        return 1;
    }
    else {
        if (lookup_attr_id(obj, &PyId_name, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from FunctionDef");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_args, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from FunctionDef");
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from FunctionDef");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_decorator_list, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from FunctionDef");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(decorator_list, i, val);
        }
        if (lookup_attr_id(obj, &PyId_returns, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            returns = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_name, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from AsyncFunctionDef");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_args, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from AsyncFunctionDef");
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFunctionDef");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_decorator_list, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from AsyncFunctionDef");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(decorator_list, i, val);
        }
        if (lookup_attr_id(obj, &PyId_returns, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            returns = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_name, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from ClassDef");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_bases, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"bases\" missing from ClassDef");
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(bases, i, val);
        }
        if (lookup_attr_id(obj, &PyId_keywords, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from ClassDef");
        else {
                keyword_ty val;
                res = obj2ast_keyword(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(keywords, i, val);
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ClassDef");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_decorator_list, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"decorator_list\" missing from ClassDef");
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(decorator_list, i, val);
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            value = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_targets, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Delete");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(targets, i, val);
        if (lookup_attr_id(obj, &PyId_targets, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"targets\" missing from Assign");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(targets, i, val);
        }
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Assign");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_target, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AugAssign");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_op, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from AugAssign");
        else {
        }
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from AugAssign");
        else {
        if (lookup_attr_id(obj, &PyId_target, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AnnAssign");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_annotation, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"annotation\" missing from AnnAssign");
        else {
        }
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            value = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_simple, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"simple\" missing from AnnAssign");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_target, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from For");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_iter, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from For");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from For");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from For");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(orelse, i, val);
        }
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_target, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from AsyncFor");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_iter, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from AsyncFor");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncFor");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from AsyncFor");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(orelse, i, val);
        }
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_test, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from While");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from While");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from While");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(orelse, i, val);
        if (lookup_attr_id(obj, &PyId_test, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from If");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from If");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from If");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(orelse, i, val);
        if (lookup_attr_id(obj, &PyId_items, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from With");
            return 1;
        }
        else {
                withitem_ty val;
                res = obj2ast_withitem(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(items, i, val);
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from With");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_items, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"items\" missing from AsyncWith");
            return 1;
        }
        else {
                withitem_ty val;
                res = obj2ast_withitem(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(items, i, val);
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from AsyncWith");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type_comment = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_exc, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            exc = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_cause, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            cause = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Try");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
        }
        if (lookup_attr_id(obj, &PyId_handlers, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"handlers\" missing from Try");
            return 1;
        }
        else {
            int res;
            Py_ssize_t len;
                excepthandler_ty val;
                res = obj2ast_excepthandler(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(handlers, i, val);
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from Try");
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(orelse, i, val);
        }
        if (lookup_attr_id(obj, &PyId_finalbody, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"finalbody\" missing from Try");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(finalbody, i, val);
        if (lookup_attr_id(obj, &PyId_test, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from Assert");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_msg, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            msg = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_names, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Import");
            return 1;
        }
        else {
                alias_ty val;
                res = obj2ast_alias(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(names, i, val);
        if (lookup_attr_id(obj, &PyId_module, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            module = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_names, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from ImportFrom");
            return 1;
        }
        else {
                alias_ty val;
                res = obj2ast_alias(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(names, i, val);
        }
        if (lookup_attr_id(obj, &PyId_level, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            level = 0;
        }
        else {
        if (lookup_attr_id(obj, &PyId_names, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Global");
            return 1;
        }
        else {
                identifier val;
                res = obj2ast_identifier(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(names, i, val);
        if (lookup_attr_id(obj, &PyId_names, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"names\" missing from Nonlocal");
            return 1;
        }
        else {
                identifier val;
                res = obj2ast_identifier(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(names, i, val);
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Expr");
            return 1;
        }
        else {
    if (lookup_attr_id(obj, &PyId_lineno, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from expr");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_col_offset, &tmp) < 0) {
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from expr");
        return 1;
    }
    else {
        if (lookup_attr_id(obj, &PyId_op, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BoolOp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_values, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from BoolOp");
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(values, i, val);
        if (lookup_attr_id(obj, &PyId_left, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from BinOp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_op, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from BinOp");
        else {
        }
        if (lookup_attr_id(obj, &PyId_right, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"right\" missing from BinOp");
        else {
        if (lookup_attr_id(obj, &PyId_op, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"op\" missing from UnaryOp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_operand, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"operand\" missing from UnaryOp");
        else {
        if (lookup_attr_id(obj, &PyId_args, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Lambda");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from Lambda");
        else {
        if (lookup_attr_id(obj, &PyId_test, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"test\" missing from IfExp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from IfExp");
        else {
        }
        if (lookup_attr_id(obj, &PyId_orelse, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"orelse\" missing from IfExp");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_keys, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"keys\" missing from Dict");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(keys, i, val);
        }
        if (lookup_attr_id(obj, &PyId_values, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from Dict");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(values, i, val);
        if (lookup_attr_id(obj, &PyId_elts, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Set");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(elts, i, val);
        if (lookup_attr_id(obj, &PyId_elt, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from ListComp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_generators, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from ListComp");
        else {
                comprehension_ty val;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(generators, i, val);
        if (lookup_attr_id(obj, &PyId_elt, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from SetComp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_generators, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from SetComp");
        else {
                comprehension_ty val;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(generators, i, val);
        if (lookup_attr_id(obj, &PyId_key, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"key\" missing from DictComp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from DictComp");
        else {
        }
        if (lookup_attr_id(obj, &PyId_generators, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from DictComp");
        else {
                comprehension_ty val;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(generators, i, val);
        if (lookup_attr_id(obj, &PyId_elt, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elt\" missing from GeneratorExp");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_generators, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"generators\" missing from GeneratorExp");
        else {
                comprehension_ty val;
                res = obj2ast_comprehension(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(generators, i, val);
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Await");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            value = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from YieldFrom");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_left, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"left\" missing from Compare");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_ops, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ops\" missing from Compare");
        else {
                cmpop_ty val;
                res = obj2ast_cmpop(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(ops, i, val);
        }
        if (lookup_attr_id(obj, &PyId_comparators, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"comparators\" missing from Compare");
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(comparators, i, val);
        if (lookup_attr_id(obj, &PyId_func, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"func\" missing from Call");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_args, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from Call");
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(args, i, val);
        }
        if (lookup_attr_id(obj, &PyId_keywords, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"keywords\" missing from Call");
        else {
                keyword_ty val;
                res = obj2ast_keyword(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(keywords, i, val);
        if (lookup_attr_id(obj, &PyId_n, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"n\" missing from Num");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_s, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Str");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from FormattedValue");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_conversion, &tmp) < 0) {
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            conversion = 0;
        }
        else {
        if (lookup_attr_id(obj, &PyId_format_spec, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            format_spec = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_values, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"values\" missing from JoinedStr");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(values, i, val);
        if (lookup_attr_id(obj, &PyId_s, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"s\" missing from Bytes");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from NameConstant");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Constant");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Attribute");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_attr, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"attr\" missing from Attribute");
        else {
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Attribute");
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Subscript");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_slice, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"slice\" missing from Subscript");
        else {
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Subscript");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Starred");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Starred");
            return 1;
        }
        else {
        if (lookup_attr_id(obj, &PyId_id, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"id\" missing from Name");
            return 1;
        }
        else {
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Name");
        else {
        if (lookup_attr_id(obj, &PyId_elts, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from List");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(elts, i, val);
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from List");
        else {
        if (lookup_attr_id(obj, &PyId_elts, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"elts\" missing from Tuple");
            return 1;
        }
        else {
                expr_ty val;
                res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(elts, i, val);
        }
        if (lookup_attr_id(obj, &PyId_ctx, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"ctx\" missing from Tuple");
        else {
        if (lookup_attr_id(obj, &PyId_lower, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            lower = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_upper, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            upper = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_step, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            step = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_dims, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"dims\" missing from ExtSlice");
            return 1;
        }
        else {
                slice_ty val;
                res = obj2ast_slice(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(dims, i, val);
        if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Index");
            return 1;
        }
        else {
    if (lookup_attr_id(obj, &PyId_target, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"target\" missing from comprehension");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_iter, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"iter\" missing from comprehension");
    else {
    }
    if (lookup_attr_id(obj, &PyId_ifs, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"ifs\" missing from comprehension");
    else {
            expr_ty val;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
            asdl_seq_SET(ifs, i, val);
    }
    if (lookup_attr_id(obj, &PyId_is_async, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"is_async\" missing from comprehension");
    else {
    if (lookup_attr_id(obj, &PyId_lineno, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from excepthandler");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_col_offset, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from excepthandler");
    else {
        if (lookup_attr_id(obj, &PyId_type, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            type = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_name, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL || tmp == Py_None) {
            Py_CLEAR(tmp);
            name = NULL;
        }
        else {
        if (lookup_attr_id(obj, &PyId_body, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"body\" missing from ExceptHandler");
            return 1;
        }
        else {
                stmt_ty val;
                res = obj2ast_stmt(PyList_GET_ITEM(tmp, i), &val, arena);
                asdl_seq_SET(body, i, val);
    if (lookup_attr_id(obj, &PyId_args, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"args\" missing from arguments");
        return 1;
    }
    else {
            arg_ty val;
            res = obj2ast_arg(PyList_GET_ITEM(tmp, i), &val, arena);
            asdl_seq_SET(args, i, val);
    }
    if (lookup_attr_id(obj, &PyId_vararg, &tmp) < 0) {
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        vararg = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_kwonlyargs, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"kwonlyargs\" missing from arguments");
        return 1;
    }
    else {
            arg_ty val;
            res = obj2ast_arg(PyList_GET_ITEM(tmp, i), &val, arena);
            asdl_seq_SET(kwonlyargs, i, val);
    }
    if (lookup_attr_id(obj, &PyId_kw_defaults, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"kw_defaults\" missing from arguments");
    else {
            expr_ty val;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
            asdl_seq_SET(kw_defaults, i, val);
    }
    if (lookup_attr_id(obj, &PyId_kwarg, &tmp) < 0) {
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        kwarg = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_defaults, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"defaults\" missing from arguments");
        return 1;
    }
    else {
            expr_ty val;
            res = obj2ast_expr(PyList_GET_ITEM(tmp, i), &val, arena);
            asdl_seq_SET(defaults, i, val);
    if (lookup_attr_id(obj, &PyId_arg, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"arg\" missing from arg");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_annotation, &tmp) < 0) {
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        annotation = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_type_comment, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        type_comment = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_lineno, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from arg");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_col_offset, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"col_offset\" missing from arg");
    else {
    if (lookup_attr_id(obj, &PyId_arg, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        arg = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_value, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from keyword");
        return 1;
    }
    else {
    if (lookup_attr_id(obj, &PyId_name, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"name\" missing from alias");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_asname, &tmp) < 0) {
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        asname = NULL;
    }
    else {
    if (lookup_attr_id(obj, &PyId_context_expr, &tmp) < 0) {
        return 1;
    }
    if (tmp == NULL) {
        PyErr_SetString(PyExc_TypeError, "required field \"context_expr\" missing from withitem");
        return 1;
    }
    else {
    }
    if (lookup_attr_id(obj, &PyId_optional_vars, &tmp) < 0) {
    if (tmp == NULL || tmp == Py_None) {
        Py_CLEAR(tmp);
        optional_vars = NULL;
    }
    else {
        if (lookup_attr_id(obj, &PyId_lineno, &tmp) < 0) {
            return 1;
        }
        if (tmp == NULL) {
            PyErr_SetString(PyExc_TypeError, "required field \"lineno\" missing from TypeIgnore");
            return 1;
        }
        else {
static struct PyModuleDef _astmodule = {
    PyModuleDef_HEAD_INIT, "_ast3", NULL, 0, ast3_methods
    m = PyModule_Create(&_astmodule);
#include "pythonrun.h"
// VS 2010 doesn't have <stdbool.h>...
typedef int bool;
#define false 0
#define true 1

#ifndef _PyObject_FastCall
static PyObject *
_PyObject_FastCall(PyObject *func, PyObject *const *args, int nargs)
{
    PyObject *t, *res;
    int i;

    t = PyTuple_New(nargs);
    if (t == NULL) {
        return NULL;
    }
    for (i = 0; i < nargs; i++) {
        if (PyTuple_SetItem(t, i, args[i]) < 0) {
            Py_DECREF(t);
            return NULL;
        }
    }
    res = PyObject_CallObject(func, t);
    Py_DECREF(t);
    return res;
}
#endif

#if PY_MINOR_VERSION < 6
#define _PyUnicode_EqualToASCIIString(a, b) (PyUnicode_CompareWithASCIIString((a), (b)) == 0)

static PyObject *
_PyBytes_DecodeEscape(const char *s,
                      Py_ssize_t len,
                      const char *errors,
                      Py_ssize_t unicode,
                      const char *recode_encoding,
                      const char **first_invalid_escape)
{
    *first_invalid_escape = NULL;
    return PyBytes_DecodeEscape(s, len, errors, unicode, recode_encoding);
}

PyObject *
_PyUnicode_DecodeUnicodeEscape(const char *s,
                               Py_ssize_t size,
                               const char *errors,
                               const char **first_invalid_escape)
{
    *first_invalid_escape = NULL;
    return PyUnicode_DecodeUnicodeEscape(s, size, errors);
}
        abort();
static asdl_seq *ast_for_suite(struct compiling *c, const node *n);
static stmt_ty ast_for_with_stmt(struct compiling *, const node *, bool);
static stmt_ty ast_for_for_stmt(struct compiling *, const node *, bool);
static expr_ty ast_for_call(struct compiling *, const node *, expr_ty, bool);
        PyObject *form;
        PyObject *args[2];
        _Py_IDENTIFIER(NFKC);
        form = _PyUnicode_FromId(&PyId_NFKC);
        if (form == NULL) {
            Py_DECREF(id);
            return NULL;
        }
        args[0] = form;
        args[1] = id;
        id2 = _PyObject_FastCall(c->c_normalize, args, 2);
        if (!PyUnicode_Check(id2)) {
            PyErr_Format(PyExc_TypeError,
                         "unicodedata.normalize() must return a string, not "
                         "%.200s",
                         Py_TYPE(id2)->tp_name);
            Py_DECREF(id2);
            return NULL;
        }
    abort();
                      PyObject *filename, int feature_version,
                      PyArena *arena)
                   goto out;
               asdl_seq_SET(type_ignores, i, ti);
                int feature_version, PyArena *arena)
    if (_PyUnicode_EqualToASCIIString(name, "__debug__")) {
            if (_PyUnicode_EqualToASCIIString(name, *p)) {
                /* fall through */
                /* fall through */
                if (TYPE(CHILD(n, i)) == COMMA)
                if (TYPE(CHILD(n, i)) == COMMA)
                    if (TYPE(CHILD(n, i)) == TYPE_COMMENT) {
                i += 2; /* the star and the name */
                if (TYPE(CHILD(n, i)) == COMMA)
                    i += 1; /* the comma, if present */
                    if (TYPE(CHILD(n, i)) == TYPE_COMMENT) {
                if (TYPE(CHILD(n, i)) == COMMA)
        d = ast_for_call(c, CHILD(n, 3), name_expr, true);
ast_for_funcdef_impl(struct compiling *c, const node *n0,
                     asdl_seq *decorator_seq, bool is_async)
    const node * const n = is_async ? CHILD(n0, 1) : n0;
                  "Async functions are only supported in Python 3.5 and greater");
                                type_comment, LINENO(n0), n0->n_col_offset, c->c_arena);
                           type_comment, LINENO(n), n->n_col_offset, c->c_arena);
    /* async_funcdef: 'async' funcdef */
    REQ(CHILD(n, 0), NAME);
    assert(strcmp(STR(CHILD(n, 0)), "async") == 0);
    return ast_for_funcdef_impl(c, n, decorator_seq,
                                true /* is_async */);
                                false /* is_async */);
    /* async_stmt: 'async' (funcdef | with_stmt | for_stmt) */
    REQ(CHILD(n, 0), NAME);
    assert(strcmp(STR(CHILD(n, 0)), "async") == 0);
            return ast_for_funcdef_impl(c, n, NULL,
                                        true /* is_async */);
            return ast_for_with_stmt(c, n,
                                     true /* is_async */);
            return ast_for_for_stmt(c, n,
                                    true /* is_async */);
    if (NCH(n) == 2) {
        REQ(CHILD(n, 0), NAME);
        assert(strcmp(STR(CHILD(n, 0)), "async") == 0);
        n = CHILD(n, 1);
    else if (NCH(n) == 1) {
        n = CHILD(n, 0);
    }
    else {
        goto error;
    }
    if (NCH(n) == (5)) {
        n = CHILD(n, 4);
  error:
        node *sync_n;
        if (NCH(n) == 2) {
            REQ(CHILD(n, 0), NAME);
            assert(strcmp(STR(CHILD(n, 0)), "async") == 0);
            sync_n = CHILD(n, 1);
        else {
            sync_n = CHILD(n, 0);
        }
        REQ(sync_n, sync_comp_for);
        for_ch = CHILD(sync_n, 1);
        expression = ast_for_expr(c, CHILD(sync_n, 3));
        if (NCH(sync_n) == 5) {
            n = CHILD(sync_n, 4);
        pynum = parsenumber(c, STR(ch));
            return ast_for_call(c, CHILD(n, 1), left_expr, true);
        /* there was an 'await' */
       atom_expr: ['await'] atom trailer*
ast_for_call(struct compiling *c, const node *n, expr_ty func, bool allowgen)
    int i, nargs, nkeywords;
            else if (TYPE(CHILD(ch, 1)) == comp_for) {
                nargs++;
                if (!allowgen) {
                    ast_error(c, ch, "invalid syntax");
                    return NULL;
                }
                if (NCH(n) > 1) {
                    ast_error(c, ch, "Generator expression must be parenthesized");
                    return NULL;
                }
            }
    args = _Ta3_asdl_seq_new(nargs, c->c_arena);

            /* fall through */
            if (!str)
                return NULL;
ast_for_for_stmt(struct compiling *c, const node *n0, bool is_async)
    const node * const n = is_async ? CHILD(n0, 1) : n0;
        return AsyncFor(target, expression, suite_seq, seq, type_comment,
                        LINENO(n0), n0->n_col_offset,
        return For(target, expression, suite_seq, seq, type_comment,
                   LINENO(n), n->n_col_offset,
ast_for_with_stmt(struct compiling *c, const node *n0, bool is_async)
    const node * const n = is_async ? CHILD(n0, 1) : n0;
        return AsyncWith(items, body, type_comment, LINENO(n0), n0->n_col_offset, c->c_arena);
        return ClassDef(classname, NULL, NULL, s, decorator_seq,
                        LINENO(n), n->n_col_offset, c->c_arena);
        s = ast_for_suite(c, CHILD(n, 5));
        return ClassDef(classname, NULL, NULL, s, decorator_seq,
                        LINENO(n), n->n_col_offset, c->c_arena);
        call = ast_for_call(c, CHILD(n, 3), dummy, false);
    if (dup == NULL) {
        return PyErr_NoMemory();
    }
static int
warn_invalid_escape_sequence(struct compiling *c, const node *n,
                             unsigned char first_invalid_escape_char)
{
    PyObject *msg = PyUnicode_FromFormat("invalid escape sequence \\%c",
                                         first_invalid_escape_char);
    if (msg == NULL) {
        return -1;
    }
    if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, msg,
                                   c->c_filename, LINENO(n),
                                   NULL, NULL) < 0)
    {
        if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) {
            const char *s;

            /* Replace the DeprecationWarning exception with a SyntaxError
               to get a more accurate error report */
            PyErr_Clear();

            s = PyUnicode_AsUTF8(msg);
            if (s != NULL) {
                ast_error(c, n, s);
            }
        }
        Py_DECREF(msg);
        return -1;
    }
    Py_DECREF(msg);
    return 0;
}

    PyObject *v, *u;
    const char *first_invalid_escape;
            if (s >= end || *s & 0x80) {
                if (s >= end)
                    break;
            assert(p - buf <= PyBytes_GET_SIZE(u));
    v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape);

    if (v != NULL && first_invalid_escape != NULL) {
        if (warn_invalid_escape_sequence(c, n, *first_invalid_escape) < 0) {
            /* We have not decref u before because first_invalid_escape points
               inside u. */
            Py_XDECREF(u);
            Py_DECREF(v);
            return NULL;
        }
    }
    Py_XDECREF(u);
    return v;
    const char *first_invalid_escape;
    PyObject *result = _PyBytes_DecodeEscape(s, len, NULL, 0, NULL,
                                             &first_invalid_escape);
    if (result == NULL)
        return NULL;

    if (first_invalid_escape != NULL) {
        if (warn_invalid_escape_sequence(c, n, *first_invalid_escape) < 0) {
            Py_DECREF(result);
            return NULL;
        }
    }
    return result;
}

/* Shift locations for the given node and all its children by adding `lineno`
   and `col_offset` to existing locations. */
static void fstring_shift_node_locations(node *n, int lineno, int col_offset)
{
    int i;
    n->n_col_offset = n->n_col_offset + col_offset;
    for (i = 0; i < NCH(n); ++i) {
        if (n->n_lineno && n->n_lineno < CHILD(n, i)->n_lineno) {
            /* Shifting column offsets unnecessary if there's been newlines. */
            col_offset = 0;
        }
        fstring_shift_node_locations(CHILD(n, i), lineno, col_offset);
    }
    n->n_lineno = n->n_lineno + lineno;
}

/* Fix locations for the given node and its children.

   `parent` is the enclosing node.
   `n` is the node which locations are going to be fixed relative to parent.
   `expr_str` is the child node's string representation, including braces.
*/
static void
fstring_fix_node_location(const node *parent, node *n, char *expr_str)
{
    char *substr = NULL;
    char *start;
    int lines = LINENO(parent) - 1;
    int cols = parent->n_col_offset;
    /* Find the full fstring to fix location information in `n`. */
    while (parent && parent->n_type != STRING)
        parent = parent->n_child;
    if (parent && parent->n_str) {
        substr = strstr(parent->n_str, expr_str);
        if (substr) {
            start = substr;
            while (start > parent->n_str) {
                if (start[0] == '\n')
                    break;
                start--;
            }
            cols += substr - start;
            /* Fix lineno in mulitline strings. */
            while ((substr = strchr(substr + 1, '\n')))
                lines--;
        }
    }
    fstring_shift_node_locations(n, lines, cols);
    node *mod_n;
    const char *s;
    PyObject *fstring_name;
    /* If the substring is all whitespace, it's an error.  We need to catch this
       here, and not when we call PyParser_SimpleParseStringFlagsFilename,
       because turning the expression '' in to '()' would go from being invalid
       to valid. */
    for (s = expr_start; s != expr_end; s++) {
        char c = *s;
        /* The Python parser ignores only the following whitespace
           characters (\r already is converted to \n). */
        if (!(c == ' ' || c == '\t' || c == '\n' || c == '\f')) {
    if (s == expr_end) {
    if (str == NULL) {
        PyErr_NoMemory();
    }
    mod_n = PyParser_SimpleParseStringFlagsFilename(str, "<fstring>",
                                                    Py_eval_input, 0);
    if (!mod_n) {
        PyMem_RawFree(str);
        return NULL;
    }
    /* Reuse str to find the correct column offset. */
    str[0] = '{';
    str[len+1] = '}';
    fstring_fix_node_location(n, mod_n, str);
    Ta3Node_Free(mod_n);
    const char *s = *str;
    const char *literal_start = s;
    while (s < end) {
        char ch = *s++;
        if (!raw && ch == '\\' && s < end) {
            ch = *s++;
            if (ch == 'N') {
                if (s < end && *s++ == '{') {
                    while (s < end && *s++ != '}') {
                    }
                    continue;
                }
                break;
            }
            if (ch == '{' && warn_invalid_escape_sequence(c, n, ch) < 0) {
                return -1;
            }
        }
        if (ch == '{' || ch == '}') {
                if (s < end && *s == ch) {
                    *str = s + 1;
                    *str = s - 1;
            s--;
    *str = s;
    assert(s <= end);
    assert(s == end || *s == '{' || *s == '}');
    if (literal_start != s) {
                                                    s - literal_start,
                                                   s - literal_start);
    int fmode;
    state->fmode = 0;
    state->fmode = 1;
            /*  Note that the literal can be zero length, if the
                input string is "\\\n" or "\\\r", among others. */
    if (!state->fmode) {
        assert(!state->expr_list.size);
static arc arcs_81_0[1] = {
    {102, 1},
    {67, 2},
    {103, 3},
    {113, 4},
static arc arcs_81_4[2] = {
    {172, 5},
    {0, 4},
static arc arcs_81_5[1] = {
static state states_81[6] = {
    {1, arcs_81_0},
    {2, arcs_81_4},
    {1, arcs_81_5},
static arc arcs_82_0[2] = {
    {21, 1},
    {174, 2},
    {174, 2},
static arc arcs_82_2[1] = {
static state states_82[3] = {
    {2, arcs_82_0},
    {1, arcs_82_2},
    {98, 1},
    {115, 2},
static arc arcs_83_2[2] = {
    {172, 3},
    {0, 2},
};
static arc arcs_83_3[1] = {
    {0, 3},
};
static state states_83[4] = {
    {2, arcs_83_2},
    {1, arcs_83_3},
    {23, 1},
static arc arcs_84_1[1] = {
static state states_84[2] = {
    {1, arcs_84_1},
static arc arcs_85_0[1] = {
    {176, 1},
static arc arcs_85_1[2] = {
    {177, 2},
    {0, 1},
    {1, arcs_85_0},
    {2, arcs_85_1},
static arc arcs_86_0[2] = {
    {78, 1},
    {9, 2},
static arc arcs_86_1[1] = {
    {26, 2},
    {2, arcs_86_0},
    {1, arcs_86_1},
    {179, 1},
    {2, 1},
    {7, 2},
    {0, 2},
};
static state states_87[3] = {
    {1, arcs_87_0},
    {2, arcs_87_1},
    {1, arcs_87_2},
};
static arc arcs_88_0[1] = {
    {13, 1},
};
static arc arcs_88_1[2] = {
    {180, 2},
static arc arcs_88_2[1] = {
    {15, 3},
};
static arc arcs_88_3[1] = {
static arc arcs_88_4[1] = {
static arc arcs_88_5[1] = {
static state states_88[6] = {
    {1, arcs_88_0},
    {2, arcs_88_1},
    {1, arcs_88_2},
    {1, arcs_88_3},
    {1, arcs_88_4},
    {1, arcs_88_5},
static arc arcs_89_0[3] = {
static arc arcs_89_1[2] = {
static arc arcs_89_2[3] = {
static arc arcs_89_3[1] = {
static arc arcs_89_4[4] = {
static arc arcs_89_5[2] = {
static arc arcs_89_6[2] = {
static arc arcs_89_7[1] = {
static arc arcs_89_8[3] = {
static arc arcs_89_9[2] = {
static arc arcs_89_10[2] = {
static state states_89[11] = {
    {3, arcs_89_0},
    {2, arcs_89_1},
    {3, arcs_89_2},
    {1, arcs_89_3},
    {4, arcs_89_4},
    {2, arcs_89_5},
    {2, arcs_89_6},
    {1, arcs_89_7},
    {3, arcs_89_8},
    {2, arcs_89_9},
    {2, arcs_89_10},
};
static dfa dfas[90] = {
     "\004\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\004\001"},
     "\204\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\004\001"},
     "\000\050\340\000\004\000\000\000\024\174\022\016\144\011\040\004\000\200\041\121\076\004\001"},
     "\000\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\000\001"},
     "\000\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\000\001"},
     "\000\000\000\000\000\000\000\000\000\074\000\000\000\000\000\000\000\000\000\000\000\000\001"},
     "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001"},
     "\004\040\200\000\004\000\000\000\024\174\022\016\000\000\040\004\000\200\041\121\076\000\001"},
    {337, "sync_comp_for", 0, 6, states_81,
     "\000\000\000\000\000\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\000\000"},
    {338, "comp_for", 0, 3, states_82,
    {339, "comp_if", 0, 4, states_83,
    {340, "encoding_decl", 0, 2, states_84,
    {341, "yield_expr", 0, 3, states_85,
     "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001"},
    {342, "yield_arg", 0, 3, states_86,
    {343, "func_type_input", 0, 3, states_87,
    {344, "func_type", 0, 6, states_88,
    {345, "typelist", 0, 11, states_89,
static label labels[181] = {
    {341, 0},
    {338, 0},
    {337, 0},
    {340, 0},
    {345, 0},
    90,
    {181, labels},
MAX_VER = 7
        'ast3/Include/compile-ast3.h',
diff --git a/ast3/Grammar/Grammar b/ast3/Grammar/Grammar
index b139e9f..dfd730f 100644
--- a/ast3/Grammar/Grammar
@@ -14,7 +14,10 @@
 #       single_input is a single interactive statement;
 #       file_input is a module or sequence of commands read from an input file;
 #       eval_input is the input for the eval() functions.
+#       func_type_input is a PEP 484 Python 2 function type comment
 # NB: compound_stmt in single_input is followed by extra NEWLINE!
+# NB: due to the way TYPE_COMMENT is tokenized it will always be followed by a
+#      NEWLINE
 single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
 file_input: (NEWLINE | stmt)* ENDMARKER
 eval_input: testlist NEWLINE* ENDMARKER
@@ -24,14 +27,14 @@ decorators: decorator+
 decorated: decorators (classdef | funcdef | async_funcdef)
 
 async_funcdef: ASYNC funcdef
-funcdef: 'def' NAME parameters ['->' test] ':' suite
+funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] suite
 
 parameters: '(' [typedargslist] ')'
-typedargslist: (tfpdef ['=' test] (',' tfpdef ['=' test])* [',' [
-        '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
-      | '**' tfpdef [',']]]
-  | '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
-  | '**' tfpdef [','])
+typedargslist: (tfpdef ['=' test] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] [
+        '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]])
+      | '**' tfpdef [','] [TYPE_COMMENT]]])
+  | '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]])
+  | '**' tfpdef [','] [TYPE_COMMENT])
 tfpdef: NAME [':' test]
 varargslist: (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [
         '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
@@ -46,7 +49,7 @@ simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
 small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
              import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
 expr_stmt: testlist_star_expr (annassign | augassign (yield_expr|testlist) |
-                     ('=' (yield_expr|testlist_star_expr))*)
+                     ('=' (yield_expr|testlist_star_expr))* [TYPE_COMMENT])
 annassign: ':' test ['=' test]
 testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
 augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' |
@@ -78,17 +81,18 @@ compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef
 async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
 if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
 while_stmt: 'while' test ':' suite ['else' ':' suite]
-for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
+for_stmt: 'for' exprlist 'in' testlist ':' [TYPE_COMMENT] suite ['else' ':' suite]
 try_stmt: ('try' ':' suite
            ((except_clause ':' suite)+
             ['else' ':' suite]
             ['finally' ':' suite] |
            'finally' ':' suite))
-with_stmt: 'with' with_item (',' with_item)*  ':' suite
+with_stmt: 'with' with_item (',' with_item)*  ':' [TYPE_COMMENT] suite
 with_item: test ['as' expr]
 # NB compile.c makes sure that the default except clause is last
 except_clause: 'except' [test ['as' NAME]]
-suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
+# the TYPE_COMMENT in suites is only parsed for funcdefs, but can't go elsewhere due to ambiguity
+suite: simple_stmt | NEWLINE [TYPE_COMMENT NEWLINE] INDENT stmt+ DEDENT
 
 test: or_test ['if' or_test 'else' test] | lambdef
 test_nocond: or_test | lambdef_nocond
@@ -154,3 +158,10 @@ encoding_decl: NAME
 
 yield_expr: 'yield' [yield_arg]
 yield_arg: 'from' test | testlist
+
+func_type_input: func_type NEWLINE* ENDMARKER
+func_type: '(' [typelist] ')' '->' test
+# typelist is a modified typedargslist (see above)
+typelist: (test (',' test)* [','
+       ['*' [test] (',' test)* [',' '**' test] | '**' test]]
+     |  '*' [test] (',' test)* [',' '**' test] | '**' test)
diff --git a/ast3/Parser/Python.asdl b/ast3/Parser/Python.asdl
index f470ad1..7bde99c 100644
--- a/ast3/Parser/Python.asdl
@@ -6,17 +6,18 @@
 
 module Python
 {
-    mod = Module(stmt* body)
+    mod = Module(stmt* body, type_ignore *type_ignores)
         | Interactive(stmt* body)
         | Expression(expr body)
+        | FunctionType(expr* argtypes, expr returns)
 
         -- not really an actual node but useful in Jython's typesystem.
         | Suite(stmt* body)
 
     stmt = FunctionDef(identifier name, arguments args,
-                       stmt* body, expr* decorator_list, expr? returns)
+                       stmt* body, expr* decorator_list, expr? returns, string? type_comment)
           | AsyncFunctionDef(identifier name, arguments args,
-                             stmt* body, expr* decorator_list, expr? returns)
+                             stmt* body, expr* decorator_list, expr? returns, string? type_comment)
 
           | ClassDef(identifier name,
              expr* bases,
@@ -26,18 +27,18 @@ module Python
           | Return(expr? value)
 
           | Delete(expr* targets)
-          | Assign(expr* targets, expr value)
+          | Assign(expr* targets, expr value, string? type_comment)
           | AugAssign(expr target, operator op, expr value)
           -- 'simple' indicates that we annotate simple name without parens
           | AnnAssign(expr target, expr annotation, expr? value, int simple)
 
           -- use 'orelse' because else is a keyword in target languages
-          | For(expr target, expr iter, stmt* body, stmt* orelse)
-          | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse)
+          | For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
+          | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
           | While(expr test, stmt* body, stmt* orelse)
           | If(expr test, stmt* body, stmt* orelse)
-          | With(withitem* items, stmt* body)
-          | AsyncWith(withitem* items, stmt* body)
+          | With(withitem* items, stmt* body, string? type_comment)
+          | AsyncWith(withitem* items, stmt* body, string? type_comment)
 
           | Raise(expr? exc, expr? cause)
           | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
@@ -118,7 +119,7 @@ module Python
     arguments = (arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults,
                  arg? kwarg, expr* defaults)
 
-    arg = (identifier arg, expr? annotation)
+    arg = (identifier arg, expr? annotation, string? type_comment)
            attributes (int lineno, int col_offset)
 
     -- keyword arguments supplied to call (NULL identifier for **kwargs)
@@ -128,5 +129,7 @@ module Python
     alias = (identifier name, identifier? asname)
 
     withitem = (expr context_expr, expr? optional_vars)
+
+    type_ignore = TypeIgnore(int lineno)
 }
 
--- /Users/guido/src/cpython37/Parser/asdl_c.py	2018-09-10 08:18:23.000000000 -0700
@@ -270,9 +270,9 @@
         margs = "a0"
         for i in range(1, len(args)+1):
             margs += ", a%d" % i
-        self.emit("#define %s(%s) _Py_%s(%s)" % (name, margs, name, margs), 0,
+        self.emit("#define %s(%s) _Ta3_%s(%s)" % (name, margs, name, margs), 0,
                 reflow=False)
-        self.emit("%s _Py_%s(%s);" % (ctype, name, argstr), False)
+        self.emit("%s _Ta3_%s(%s);" % (ctype, name, argstr), False)
 
     def visitProduct(self, prod, name):
         self.emit_function(name, get_c_type(name),
@@ -531,9 +531,9 @@
             self.emit("}", depth+1)
             self.emit("len = PyList_GET_SIZE(tmp);", depth+1)
             if self.isSimpleType(field):
-                self.emit("%s = _Py_asdl_int_seq_new(len, arena);" % field.name, depth+1)
+                self.emit("%s = _Ta3_asdl_int_seq_new(len, arena);" % field.name, depth+1)
             else:
-                self.emit("%s = _Py_asdl_seq_new(len, arena);" % field.name, depth+1)
+                self.emit("%s = _Ta3_asdl_seq_new(len, arena);" % field.name, depth+1)
             self.emit("if (%s == NULL) goto failed;" % field.name, depth+1)
             self.emit("for (i = 0; i < len; i++) {", depth+1)
             self.emit("%s val;" % ctype, depth+2)
@@ -729,8 +729,8 @@
 };
 
 static PyTypeObject AST_type = {
-    PyVarObject_HEAD_INIT(&PyType_Type, 0)
-    "_ast.AST",
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_ast3.AST",
     sizeof(AST_object),
     0,
     (destructor)ast_dealloc, /* tp_dealloc */
@@ -774,7 +774,7 @@
 static PyTypeObject* make_type(char *type, PyTypeObject* base, char**fields, int num_fields)
 {
     _Py_IDENTIFIER(__module__);
-    _Py_IDENTIFIER(_ast);
+    _Py_IDENTIFIER(_ast3);
     PyObject *fnames, *result;
     int i;
     fnames = PyTuple_New(num_fields);
@@ -791,7 +791,7 @@
                     type, base,
                     _PyUnicode_FromId(&PyId__fields), fnames,
                     _PyUnicode_FromId(&PyId___module__),
-                    _PyUnicode_FromId(&PyId__ast));
+                    _PyUnicode_FromId(&PyId__ast3));
     Py_DECREF(fnames);
     return (PyTypeObject*)result;
 }
@@ -1010,11 +1010,16 @@
 class ASTModuleVisitor(PickleVisitor):
 
     def visitModule(self, mod):
+        self.emit("PyObject *ast3_parse(PyObject *self, PyObject *args);", 0)
+        self.emit("static PyMethodDef ast3_methods[] = {", 0)
+        self.emit('    {"_parse",  ast3_parse, METH_VARARGS, "Parse string into typed AST."},', 0)
+        self.emit("    {NULL, NULL, 0, NULL}", 0)
+        self.emit("};", 0)
         self.emit("static struct PyModuleDef _astmodule = {", 0)
-        self.emit('  PyModuleDef_HEAD_INIT, "_ast"', 0)
+        self.emit('    PyModuleDef_HEAD_INIT, "_ast3", NULL, 0, ast3_methods', 0)
         self.emit("};", 0)
         self.emit("PyMODINIT_FUNC", 0)
-        self.emit("PyInit__ast(void)", 0)
+        self.emit("PyInit__ast3(void)", 0)
         self.emit("{", 0)
         self.emit("PyObject *m, *d;", 1)
         self.emit("if (!init_types()) return NULL;", 1)
@@ -1199,7 +1204,7 @@
 class PartingShots(StaticVisitor):
 
     CODE = """
-PyObject* PyAST_mod2obj(mod_ty t)
+PyObject* Ta3AST_mod2obj(mod_ty t)
 {
     if (!init_types())
         return NULL;
@@ -1207,7 +1212,7 @@
 }
 
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
-mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
+mod_ty Ta3AST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
     mod_ty res;
     PyObject *req_type[3];
@@ -1237,7 +1242,7 @@
         return res;
 }
 
-int PyAST_Check(PyObject* obj)
+int Ta3AST_Check(PyObject* obj)
 {
     if (!init_types())
         return -1;
@@ -1276,9 +1281,9 @@
                                 PrototypeVisitor(f),
                                 )
             c.visit(mod)
-            f.write("PyObject* PyAST_mod2obj(mod_ty t);\n")
-            f.write("mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode);\n")
-            f.write("int PyAST_Check(PyObject* obj);\n")
+            f.write("PyObject* Ta3AST_mod2obj(mod_ty t);\n")
+            f.write("mod_ty Ta3AST_obj2mod(PyObject* ast, PyArena* arena, int mode);\n")
+            f.write("int Ta3AST_Check(PyObject* obj);\n")
 
     if C_FILE:
         with open(C_FILE, "w") as f:
diff --git a/ast3/Python/ast.c b/ast3/Python/ast.c
index e12f8e6..1fa762d 100644
--- a/ast3/Python/ast.c
@@ -665,6 +665,13 @@ new_identifier(const char *n, struct compiling *c)
 
 #define NEW_IDENTIFIER(n) new_identifier(STR(n), c)
 
+static string
+new_type_comment(const char *s, struct compiling *c)
+{
+  return PyUnicode_DecodeUTF8(s, strlen(s), NULL);
+}
+#define NEW_TYPE_COMMENT(n) new_type_comment(STR(n), c)
+
 static int
 ast_error(struct compiling *c, const node *n, const char *errmsg)
 {
@@ -734,11 +741,15 @@ num_stmts(const node *n)
         case simple_stmt:
             return NCH(n) / 2; /* Divide by 2 to remove count of semi-colons */
         case suite:
+            /* suite: simple_stmt | NEWLINE [TYPE_COMMENT NEWLINE] INDENT stmt+ DEDENT */
             if (NCH(n) == 1)
                 return num_stmts(CHILD(n, 0));
             else {
+                i = 2;
                 l = 0;
-                for (i = 2; i < (NCH(n) - 1); i++)
+                if (TYPE(CHILD(n, 1)) == TYPE_COMMENT)
+                    i += 2;
+                for (; i < (NCH(n) - 1); i++)
                     l += num_stmts(CHILD(n, i));
                 return l;
             }
@@ -763,10 +774,13 @@ Ta3AST_FromNodeObject(const node *n, PyCompilerFlags *flags,
 {
     int i, j, k, num;
     asdl_seq *stmts = NULL;
+    asdl_seq *type_ignores = NULL;
     stmt_ty s;
     node *ch;
     struct compiling c;
     mod_ty res = NULL;
+    asdl_seq *argtypes = NULL;
+    expr_ty ret, arg;
 
     c.c_arena = arena;
     /* borrowed reference */
@@ -806,7 +820,23 @@ Ta3AST_FromNodeObject(const node *n, PyCompilerFlags *flags,
                     }
                 }
             }
-            res = Module(stmts, arena);
+
+            /* Type ignores are stored under the ENDMARKER in file_input. */
+            ch = CHILD(n, NCH(n) - 1);
+            REQ(ch, ENDMARKER);
+            num = NCH(ch);
+            type_ignores = _Ta3_asdl_seq_new(num, arena);
+            if (!type_ignores)
+                goto out;
+
+            for (i = 0; i < num; i++) {
+                type_ignore_ty ti = TypeIgnore(LINENO(CHILD(ch, i)), arena);
+                if (!ti)
+                    goto out;
+                asdl_seq_SET(type_ignores, i, ti);
+            }
+
+            res = Module(stmts, type_ignores, arena);
             break;
         case eval_input: {
             expr_ty testlist_ast;
@@ -857,6 +887,40 @@ Ta3AST_FromNodeObject(const node *n, PyCompilerFlags *flags,
                 res = Interactive(stmts, arena);
             }
             break;
+        case func_type_input:
+            n = CHILD(n, 0);
+            REQ(n, func_type);
+
+            if (TYPE(CHILD(n, 1)) == typelist) {
+                ch = CHILD(n, 1);
+                /* this is overly permissive -- we don't pay any attention to
+                 * stars on the args -- just parse them into an ordered list */
+                num = 0;
+                for (i = 0; i < NCH(ch); i++) {
+                    if (TYPE(CHILD(ch, i)) == test)
+                        num++;
+                }
+
+                argtypes = _Ta3_asdl_seq_new(num, arena);
+
+                j = 0;
+                for (i = 0; i < NCH(ch); i++) {
+                    if (TYPE(CHILD(ch, i)) == test) {
+                        arg = ast_for_expr(&c, CHILD(ch, i));
+                        if (!arg)
+                            goto out;
+                        asdl_seq_SET(argtypes, j++, arg);
+                    }
+                }
+            }
+            else
+                argtypes = _Ta3_asdl_seq_new(0, arena);
+
+            ret = ast_for_expr(&c, CHILD(n, NCH(n) - 1));
+            if (!ret)
+                goto out;
+            res = FunctionType(argtypes, ret, arena);
+            break;
         default:
             PyErr_Format(PyExc_SystemError,
                          "invalid node %d for Ta3AST_FromNode", TYPE(n));
@@ -1250,7 +1314,7 @@ ast_for_arg(struct compiling *c, const node *n)
             return NULL;
     }
 
-    ret = arg(name, annotation, LINENO(n), n->n_col_offset, c->c_arena);
+    ret = arg(name, annotation, NULL, LINENO(n), n->n_col_offset, c->c_arena);
     if (!ret)
         return NULL;
     return ret;
@@ -1308,12 +1372,19 @@ handle_keywordonly_args(struct compiling *c, const node *n, int start,
                     goto error;
                 if (forbidden_name(c, argname, ch, 0))
                     goto error;
-                arg = arg(argname, annotation, LINENO(ch), ch->n_col_offset,
+                arg = arg(argname, annotation, NULL, LINENO(ch), ch->n_col_offset,
                           c->c_arena);
                 if (!arg)
                     goto error;
                 asdl_seq_SET(kwonlyargs, j++, arg);
-                i += 2; /* the name and the comma */
+                i += 1; /* the name */
+                if (TYPE(CHILD(n, i)) == COMMA)
+                    i += 1; /* the comma, if present */
+                break;
+            case TYPE_COMMENT:
+                /* arg will be equal to the last argument processed */
+                arg->type_comment = NEW_TYPE_COMMENT(ch);
+                i += 1;
                 break;
             case DOUBLESTAR:
                 return i;
@@ -1448,11 +1519,14 @@ ast_for_arguments(struct compiling *c, const node *n)
                 if (!arg)
                     return NULL;
                 asdl_seq_SET(posargs, k++, arg);
-                i += 2; /* the name and the comma */
+                i += 1; /* the name */
+                if (TYPE(CHILD(n, i)) == COMMA)
+                    i += 1; /* the comma, if present */
                 break;
             case STAR:
                 if (i+1 >= NCH(n) ||
-                    (i+2 == NCH(n) && TYPE(CHILD(n, i+1)) == COMMA)) {
+                    (i+2 == NCH(n) && (TYPE(CHILD(n, i+1)) == COMMA
+                                       || TYPE(CHILD(n, i+1)) == TYPE_COMMENT))) {
                     ast_error(c, CHILD(n, i),
                         "named arguments must follow bare *");
                     return NULL;
@@ -1461,6 +1535,13 @@ ast_for_arguments(struct compiling *c, const node *n)
                 if (TYPE(ch) == COMMA) {
                     int res = 0;
                     i += 2; /* now follows keyword only arguments */
+
+                    if (TYPE(CHILD(n, i)) == TYPE_COMMENT) {
+                        ast_error(c, CHILD(n, i),
+                                "bare * has associated type comment");
+                        return NULL;
+                    }
+
                     res = handle_keywordonly_args(c, n, i,
                                                   kwonlyargs, kwdefaults);
                     if (res == -1) return NULL;
@@ -1471,7 +1552,15 @@ ast_for_arguments(struct compiling *c, const node *n)
                     if (!vararg)
                         return NULL;
 
-                    i += 3;
+                i += 2; /* the star and the name */
+                if (TYPE(CHILD(n, i)) == COMMA)
+                    i += 1; /* the comma, if present */
+
+                    if (TYPE(CHILD(n, i)) == TYPE_COMMENT) {
+                        vararg->type_comment = NEW_TYPE_COMMENT(CHILD(n, i));
+                        i += 1;
+                    }
+
                     if (i < NCH(n) && (TYPE(CHILD(n, i)) == tfpdef
                                     || TYPE(CHILD(n, i)) == vfpdef)) {
                         int res = 0;
@@ -1488,7 +1577,19 @@ ast_for_arguments(struct compiling *c, const node *n)
                 kwarg = ast_for_arg(c, ch);
                 if (!kwarg)
                     return NULL;
-                i += 3;
+                i += 2; /* the double star and the name */
+                if (TYPE(CHILD(n, i)) == COMMA)
+                    i += 1; /* the comma, if present */
+                break;
+            case TYPE_COMMENT:
+                assert(i);
+
+                if (kwarg)
+                    arg = kwarg;
+
+                /* arg will be equal to the last argument processed */
+                arg->type_comment = NEW_TYPE_COMMENT(ch);
+                i += 1;
                 break;
             default:
                 PyErr_Format(PyExc_SystemError,
@@ -1593,12 +1694,14 @@ static stmt_ty
 ast_for_funcdef_impl(struct compiling *c, const node *n,
                      asdl_seq *decorator_seq, int is_async)
 {
-    /* funcdef: 'def' NAME parameters ['->' test] ':' suite */
+    /* funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] suite */
     identifier name;
     arguments_ty args;
     asdl_seq *body;
     expr_ty returns = NULL;
     int name_i = 1;
+    node *tc;
+    string type_comment = NULL;
 
     REQ(n, funcdef);
 
@@ -1616,17 +1719,30 @@ ast_for_funcdef_impl(struct compiling *c, const node *n,
             return NULL;
         name_i += 2;
     }
+    if (TYPE(CHILD(n, name_i + 3)) == TYPE_COMMENT) {
+        type_comment = NEW_TYPE_COMMENT(CHILD(n, name_i + 3));
+        name_i += 1;
+    }
     body = ast_for_suite(c, CHILD(n, name_i + 3));
     if (!body)
         return NULL;
 
+    if (!type_comment && NCH(CHILD(n, name_i + 3)) > 1) {
+        /* If the function doesn't have a type comment on the same line, check
+         * if the suite has a type comment in it. */
+        tc = CHILD(CHILD(n, name_i + 3), 1);
+
+        if (TYPE(tc) == TYPE_COMMENT)
+            type_comment = NEW_TYPE_COMMENT(tc);
+    }
+
     if (is_async)
         return AsyncFunctionDef(name, args, body, decorator_seq, returns,
-                                LINENO(n),
+                                type_comment, LINENO(n),
                                 n->n_col_offset, c->c_arena);
     else
         return FunctionDef(name, args, body, decorator_seq, returns,
-                           LINENO(n),
+                           type_comment, LINENO(n),
                            n->n_col_offset, c->c_arena);
 }
 
@@ -2896,15 +3012,16 @@ ast_for_expr_stmt(struct compiling *c, const node *n)
 {
     REQ(n, expr_stmt);
     /* expr_stmt: testlist_star_expr (annassign | augassign (yield_expr|testlist) |
-                            ('=' (yield_expr|testlist_star_expr))*)
+                           ('=' (yield_expr|testlist_star_expr))* [TYPE_COMMENT])
        annassign: ':' test ['=' test]
        testlist_star_expr: (test|star_expr) (',' test|star_expr)* [',']
        augassign: '+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^='
                 | '<<=' | '>>=' | '**=' | '//='
        test: ... here starts the operator precedence dance
      */
+    int num = NCH(n);
 
-    if (NCH(n) == 1) {
+    if (num == 1 || (num == 2 && TYPE(CHILD(n, 1)) == TYPE_COMMENT)) {
         expr_ty e = ast_for_testlist(c, CHILD(n, 0));
         if (!e)
             return NULL;
@@ -3020,17 +3137,22 @@ ast_for_expr_stmt(struct compiling *c, const node *n)
         }
     }
     else {
-        int i;
+        int i, nch_minus_type, has_type_comment;
         asdl_seq *targets;
         node *value;
         expr_ty expression;
+        string type_comment;
 
         /* a normal assignment */
         REQ(CHILD(n, 1), EQUAL);
-        targets = _Ta3_asdl_seq_new(NCH(n) / 2, c->c_arena);
+
+        has_type_comment = TYPE(CHILD(n, num - 1)) == TYPE_COMMENT;
+        nch_minus_type = num - has_type_comment;
+
+        targets = _Ta3_asdl_seq_new(nch_minus_type / 2, c->c_arena);
         if (!targets)
             return NULL;
-        for (i = 0; i < NCH(n) - 2; i += 2) {
+        for (i = 0; i < nch_minus_type - 2; i += 2) {
             expr_ty e;
             node *ch = CHILD(n, i);
             if (TYPE(ch) == yield_expr) {
@@ -3047,14 +3169,18 @@ ast_for_expr_stmt(struct compiling *c, const node *n)
 
             asdl_seq_SET(targets, i / 2, e);
         }
-        value = CHILD(n, NCH(n) - 1);
+        value = CHILD(n, nch_minus_type - 1);
         if (TYPE(value) == testlist_star_expr)
             expression = ast_for_testlist(c, value);
         else
             expression = ast_for_expr(c, value);
         if (!expression)
             return NULL;
-        return Assign(targets, expression, LINENO(n), n->n_col_offset, c->c_arena);
+        if (has_type_comment)
+            type_comment = NEW_TYPE_COMMENT(CHILD(n, nch_minus_type));
+        else
+            type_comment = NULL;
+        return Assign(targets, expression, type_comment, LINENO(n), n->n_col_offset, c->c_arena);
     }
 }
 
@@ -3461,7 +3587,7 @@ ast_for_assert_stmt(struct compiling *c, const node *n)
 static asdl_seq *
 ast_for_suite(struct compiling *c, const node *n)
 {
-    /* suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT */
+    /* suite: simple_stmt | NEWLINE [TYPE_COMMENT NEWLINE] INDENT stmt+ DEDENT */
     asdl_seq *seq;
     stmt_ty s;
     int i, total, num, end, pos = 0;
@@ -3491,7 +3617,11 @@ ast_for_suite(struct compiling *c, const node *n)
         }
     }
     else {
-        for (i = 2; i < (NCH(n) - 1); i++) {
+        i = 2;
+        if (TYPE(CHILD(n, 1)) == TYPE_COMMENT)
+            i += 2;
+
+        for (; i < (NCH(n) - 1); i++) {
             ch = CHILD(n, i);
             REQ(ch, stmt);
             num = num_stmts(ch);
@@ -3692,11 +3822,15 @@ ast_for_for_stmt(struct compiling *c, const node *n, int is_async)
     expr_ty expression;
     expr_ty target, first;
     const node *node_target;
-    /* for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite] */
+    int has_type_comment;
+    string type_comment;
+    /* for_stmt: 'for' exprlist 'in' testlist ':' [TYPE_COMMENT] suite ['else' ':' suite] */
     REQ(n, for_stmt);
 
-    if (NCH(n) == 9) {
-        seq = ast_for_suite(c, CHILD(n, 8));
+    has_type_comment = TYPE(CHILD(n, 5)) == TYPE_COMMENT;
+
+    if (NCH(n) == 9 + has_type_comment) {
+        seq = ast_for_suite(c, CHILD(n, 8 + has_type_comment));
         if (!seq)
             return NULL;
     }
@@ -3716,17 +3850,22 @@ ast_for_for_stmt(struct compiling *c, const node *n, int is_async)
     expression = ast_for_testlist(c, CHILD(n, 3));
     if (!expression)
         return NULL;
-    suite_seq = ast_for_suite(c, CHILD(n, 5));
+    suite_seq = ast_for_suite(c, CHILD(n, 5 + has_type_comment));
     if (!suite_seq)
         return NULL;
 
+    if (has_type_comment)
+        type_comment = NEW_TYPE_COMMENT(CHILD(n, 5));
+    else
+        type_comment = NULL;
+
     if (is_async)
         return AsyncFor(target, expression, suite_seq, seq,
-                        LINENO(n), n->n_col_offset,
+                        type_comment, LINENO(n), n->n_col_offset,
                         c->c_arena);
     else
         return For(target, expression, suite_seq, seq,
-                   LINENO(n), n->n_col_offset,
+                   type_comment, LINENO(n), n->n_col_offset,
                    c->c_arena);
 }
 
@@ -3872,20 +4011,24 @@ ast_for_with_item(struct compiling *c, const node *n)
     return withitem(context_expr, optional_vars, c->c_arena);
 }
 
-/* with_stmt: 'with' with_item (',' with_item)* ':' suite */
+/* with_stmt: 'with' with_item (',' with_item)*  ':' [TYPE_COMMENT] suite */
 static stmt_ty
 ast_for_with_stmt(struct compiling *c, const node *n, int is_async)
 {
-    int i, n_items;
+    int i, n_items, nch_minus_type, has_type_comment;
     asdl_seq *items, *body;
+    string type_comment;
 
     REQ(n, with_stmt);
 
-    n_items = (NCH(n) - 2) / 2;
+    has_type_comment = TYPE(CHILD(n, NCH(n) - 2)) == TYPE_COMMENT;
+    nch_minus_type = NCH(n) - has_type_comment;
+
+    n_items = (nch_minus_type - 2) / 2;
     items = _Ta3_asdl_seq_new(n_items, c->c_arena);
     if (!items)
         return NULL;
-    for (i = 1; i < NCH(n) - 2; i += 2) {
+    for (i = 1; i < nch_minus_type - 2; i += 2) {
         withitem_ty item = ast_for_with_item(c, CHILD(n, i));
         if (!item)
             return NULL;
@@ -3896,10 +4039,15 @@ ast_for_with_stmt(struct compiling *c, const node *n, int is_async)
     if (!body)
         return NULL;
 
+    if (has_type_comment)
+        type_comment = NEW_TYPE_COMMENT(CHILD(n, NCH(n) - 2));
+    else
+        type_comment = NULL;
+
     if (is_async)
-        return AsyncWith(items, body, LINENO(n), n->n_col_offset, c->c_arena);
+        return AsyncWith(items, body, type_comment, LINENO(n), n->n_col_offset, c->c_arena);
     else
-        return With(items, body, LINENO(n), n->n_col_offset, c->c_arena);
+        return With(items, body, type_comment, LINENO(n), n->n_col_offset, c->c_arena);
 }
 
 static stmt_ty
# This requires GNU binutils (e.g. brew install binutils).

/usr/local/opt/binutils/bin/gobjdump -t $PROJ_DIR/build/lib*/_ast${1}.*.so \
    | grep ' g ' \
    | grep -v UND \
    | sed 's/.* _//' \
    | grep -v PyInit__ast \
    | grep 'Py' \
    > "exported_symbols${1}.txt"
diff --git a/ast3/Parser/parsetok.c b/ast3/Parser/parsetok.c
index 9f01a0d..5529feb 100644
--- a/ast3/Parser/parsetok.c
@@ -177,6 +177,38 @@ warn(const char *msg, const char *filename, int lineno)
 #endif
 #endif
 
+typedef struct {
+    int *items;
+    size_t size;
+    size_t num_items;
+} growable_int_array;
+
+int growable_int_array_init(growable_int_array *arr, size_t initial_size) {
+    assert(initial_size > 0);
+    arr->items = malloc(initial_size * sizeof(*arr->items));
+    arr->size = initial_size;
+    arr->num_items = 0;
+
+    return arr->items != NULL;
+}
+
+int growable_int_array_add(growable_int_array *arr, int item) {
+    if (arr->num_items >= arr->size) {
+        arr->size *= 2;
+        arr->items = realloc(arr->items, arr->size * sizeof(*arr->items));
+        if (!arr->items)
+            return 0;
+    }
+
+    arr->items[arr->num_items] = item;
+    arr->num_items++;
+    return 1;
+}
+
+void growable_int_array_deallocate(growable_int_array *arr) {
+    free(arr->items);
+}
+
 /* Parse input coming from the given tokenizer structure.
    Return error code. */
 
@@ -188,6 +220,13 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret,
     node *n;
     int started = 0;
 
+    growable_int_array type_ignores;
+    if (!growable_int_array_init(&type_ignores, 10)) {
+        err_ret->error = E_NOMEM;
+        Ta3Tokenizer_Free(tok);
+        return NULL;
+    }
+
     if ((ps = Ta3Parser_New(g, start)) == NULL) {
         err_ret->error = E_NOMEM;
         Ta3Tokenizer_Free(tok);
@@ -259,6 +298,14 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret,
         else
             col_offset = -1;
 
+        if (type == TYPE_IGNORE) {
+            if (!growable_int_array_add(&type_ignores, tok->lineno)) {
+                err_ret->error = E_NOMEM;
+                break;
+            }
+            continue;
+        }
+
         if ((err_ret->error =
              Ta3Parser_AddToken(ps, (int)type, str,
                                tok->lineno, col_offset,
@@ -275,6 +322,22 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret,
         n = ps->p_tree;
         ps->p_tree = NULL;
 
+        if (n->n_type == file_input) {
+            /* Put type_ignore nodes in the ENDMARKER of file_input. */
+            int num;
+            node *ch;
+            size_t i;
+
+            num = NCH(n);
+            ch = CHILD(n, num - 1);
+            REQ(ch, ENDMARKER);
+
+            for (i = 0; i < type_ignores.num_items; i++) {
+                Ta3Node_AddChild(ch, TYPE_IGNORE, NULL, type_ignores.items[i], 0);
+            }
+        }
+        growable_int_array_deallocate(&type_ignores);
+
 #ifndef PGEN
         /* Check that the source for a single input statement really
            is a single statement by looking at what is left in the
#!/bin/bash -ex

# Automate steps 1-4 of update_process.md (Mac).

HERE=$(dirname ${BASH_SOURCE[0]})
cd $HERE/..
pwd

CPYTHON=~/src/cpython37

DIRS="Grammar Include Parser Python"
C_FILES="Parser/acceler.c Parser/bitset.c Parser/grammar.c Parser/grammar1.c Parser/node.c Parser/parser.c Parser/parsetok.c Parser/tokenizer.c Python/asdl.c Python/ast.c Python/graminit.c Python/Python-ast.c"
H_FILES="Include/asdl.h Include/ast.h Include/bitset.h Include/errcode.h Include/graminit.h Include/grammar.h Include/node.h Include/parsetok.h Include/Python-ast.h Include/token.h Parser/parser.h Parser/tokenizer.h"
OTHER_FILES="Grammar/Grammar Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py"

for dir in $DIRS
do
    rm -rf ast3/$dir
    mkdir -p ast3/$dir
done

for file in $C_FILES $H_FILES $OTHER_FILES
do
    cp $CPYTHON/$file ast3/$file
done

./tools/update_header_guards 3

rm -rf build
grep -v ast3/Custom setup.py | python3.7 - build

./tools/find_exported_symbols 3
./tools/update_exported_symbols 3

patch ast3/Parser/asdl_c.py <tools/asdl_c.patch

python3.7 ast3/Parser/asdl_c.py -h ast3/Include/Python-ast.h ast3/Parser/Python.asdl
python3.7 ast3/Parser/asdl_c.py -c ast3/Python/Python-ast.c ast3/Parser/Python.asdl

python3.7 setup.py build

# Lots of manual changes go here...

##PYTHONPATH=build/lib.macosx-10.9-x86_64-3.7/ python3.7 -m pytest -s ast3/tests
diff --git a/ast3/Include/token.h b/ast3/Include/token.h
index a657fdd..d0b2b94 100644
--- a/ast3/Include/token.h
@@ -68,8 +68,10 @@ extern "C" {
 /* These aren't used by the C tokenizer but are needed for tokenize.py */
 #define COMMENT         55
 #define NL              56
-#define ENCODING                57
-#define N_TOKENS        58
+#define ENCODING        57
+#define TYPE_IGNORE     58
+#define TYPE_COMMENT    59
+#define N_TOKENS        60
 
 /* Special definitions for cooperation with parser */
 
diff --git a/ast3/Parser/tokenizer.c b/ast3/Parser/tokenizer.c
index 617a744..667fb4a 100644
--- a/ast3/Parser/tokenizer.c
@@ -105,10 +105,16 @@ const char *_Ta3Parser_TokenNames[] = {
     "OP",
     "AWAIT",
     "ASYNC",
+    "TYPE_IGNORE",
+    "TYPE_COMMENT",
     "<ERRORTOKEN>",
     "<N_TOKENS>"
 };
 
+/* Spaces in this constant are treated as "zero or more spaces or tabs" when
+   tokenizing. */
+static const char* type_comment_prefix = "# type: ";
+
 
 /* Create and initialize a new tok_state structure */
 
@@ -1493,10 +1499,56 @@ tok_get(struct tok_state *tok, char **p_start, char **p_end)
     /* Set start of current token */
     tok->start = tok->cur - 1;
 
-    /* Skip comment */
+    /* Skip comment, unless it's a type comment */
     if (c == '#') {
-        while (c != EOF && c != '\n') {
+        const char *prefix, *p, *type_start;
+
+        while (c != EOF && c != '\n')
             c = tok_nextc(tok);
+
+        p = tok->start;
+        prefix = type_comment_prefix;
+        while (*prefix && p < tok->cur) {
+            if (*prefix == ' ') {
+                while (*p == ' ' || *p == '\t')
+                    p++;
+            } else if (*prefix == *p) {
+                p++;
+            } else {
+                break;
+            }
+
+            prefix++;
+        }
+
+        /* This is a type comment if we matched all of type_comment_prefix. */
+        if (!*prefix) {
+            int is_type_ignore = 1;
+            tok_backup(tok, c);  /* don't eat the newline or EOF */
+
+            type_start = p;
+
+            is_type_ignore = tok->cur >= p + 6 && memcmp(p, "ignore", 6) == 0;
+            p += 6;
+            while (is_type_ignore && p < tok->cur) {
+              if (*p == '#')
+                  break;
+              is_type_ignore = is_type_ignore && (*p == ' ' || *p == '\t');
+              p++;
+            }
+
+            if (is_type_ignore) {
+                /* If this type ignore is the only thing on the line, consume the newline also. */
+                if (blankline) {
+                    tok_nextc(tok);
+                    tok->atbol = 1;
+                }
+                return TYPE_IGNORE;
+            } else {
+                *p_start = (char *) type_start;  /* after type_comment_prefix */
+                *p_end = tok->cur;
+                return TYPE_COMMENT;
+            }
         }
     }
 
python3 ast3/Parser/asdl_c.py -h ast3/Include/Python-ast.h ast3/Parser/Python.asdl
python3 ast3/Parser/asdl_c.py -c ast3/Python/Python-ast.c ast3/Parser/Python.asdl
echo 'Copying pgen'
cp ~/src/cpython37/Parser/pgen tools/pgen3
__version__ = "1.3.0.dev0"
LATEST_MINOR_VERSION = 7

    When feature_version>=7, 'async' and 'await' are always keywords.
(At a high level, steps 1-4 alter the code so it uses `Ta3` instead of
`Py` as a prefix for globals; the next few steps add support for type
comments to the lexer, grammar and "asdl" machinery; then we add
support for `feature_version`; finally we work on making the code
compatible with older Python versions and other platforms.)

Note that steps 1-4 can be automated using tools/script and various
other files in tools/.  You need to install GNU binutils in order
to be able to use gobjdump in tools/find_exported_symbols
(e.g. `brew install binutils`).  The script assumes you're on a Mac
and your CPython source tree is at `~/src/cpython37`.


   1. Starting with an empty `build` directory, compile the module with `python3 setup.py build`.
   2. Generate the list of exported symbols with `./tools/find_exported_symbols 3`.
      The script may require updating to work on your platform, but should serve
      as a useful guide at minimum.
   3. The exported symbols will be written to `exported_symbols3.txt`.  Make
      sure this file looks sane, then remove `_PyInit__ast3` (which we want to
      export) to end up with a list of function names to change.
   4. Run `./tools/update_exported_symbols 3`, which updates the exported
      symbols in all the `ast3` files with sed.  It may take a few seconds to run.
      If you're on Linux, the script will need some very slight modification to
      work properly (due to cross-platform sed argument differences).  Verify the
      changes look sane.
      ([d1ec7d0](https://github.com/python/typed_ast/commit/d1ec7d07cb6a7fe016d9446a196dfa3b86c5acf6))
   5. Update `Parser/asdl_c.py`.  Use the changes from git history to guide you.
      (Don't be distracted by the generated files in that commit; look at asdl_c.py only.
      Much of this renames _ast to _ast3 and substitutes certain _Py_ prefixes with _Ta3_.)
      Update the generated files with `tools/update_ast3_asdl`.
      ([29dbec4](https://github.com/python/typed_ast/commit/29dbec47aa145d84e5faaa431ce3b3afca233b3d))

6. Add `Custom/typed_ast.c` back to setup.py.  To allow it to compile,
   temporarily comment out references to `TYPE_COMMENT` and `Py_func_type_input` and
   also the `feature_version` argument in the call to `Ta3AST_FromNodeObject()`.
   guide you here.  Diffing the previous `ast3` against its external symbol
   A bit more detail:

   - Add the `TYPE_IGNORE` and `TYPE_COMMENT` symbols to `Include/token.h`, and updating `N_TOKENS`;
     also add theze to the list of strings in `Parser/tokenizer.c` (in the same order!)
   - NOTE: As of Python 3.7, the `ASYNC` and `AWAIT` symbols also need to be added to both places
   - Update `Parser/Python.asdl` to add `type_comments` and `type_ignores` to various definitions
     and run `tools/update_ast3_asdl`; this updates `Include/Python-ast.h` and `Python/Python-ast.c`
   - Reapply other patches to `Parser/parsetok.c` and `Parser/tokenizer.c` (these implement
     recognition of type comments)
   - Add `[TYPE_COMMENT]` to various places in `Grammar/Grammar`, and then run
     `tools/update_ast3_grammar`; this updates `Python/graminit.c` and `Include/graminit.h`
   - NOTE: As of Python 3.7, this is problematic because the upstream developers like to add
     dependencies on CPython internals to pgen.  I ended up copying some files into CPython,
     running pgen there (`make regen-grammar`), and copying the results back
   - Copy the definition of `Py_func_type_input` from `Python/graminit.h` to `Include/compile.h`
   - NOTE: As of Python 3.7, compile.h depends on CPython internals; I ended up creating a small
     file compile-ast3.h with just the four symbols we need (maybe we even only need the one)
   - Attempt compilation and fix errors, e.g. add an extra argument to `Module(stmts, arena)`
     to pass `type_ignores`

11. Make the changes necessary so `ast3` can compile on previous Python
    versions.  This is new territory every time.  Spelunking in the rest of the
    details.