Repository: python/cpython
Pull request: gh-116022: Improve `repr()` of AST nodes
closes #116022

Based on the original issue the new `repr()` has:

- limited depth: nodes which are to deep are represented as `Node(...)`
- compressed list/tuple fields (e.g. `body`): more than 2 items are shown as `[repr(first), ..., repr(last)]`

Examples (output is formatted to be more readable):

```python
>>> import ast
>>> ast.parse("x = 3")
Module(
    body=[Assign(targets=[Name(...)], value=Constant(...), type_comment=None)],
    type_ignores=[]
)


>>> ast.parse('x=7; y=4; c,z=3,4')
Module(
    body=[
        Assign(targets=[Name(...)], value=Constant(...), type_comment=None),
        ...,
        Assign(targets=[Tuple(...)], value=Tuple(...), type_comment=None)
    ],
    type_ignores=[]
)


>>> from pathlib import Path
>>> import typing
>>> ast.parse(Path(typing.__file__).read_text())
Module(
    body=[
        Expr(value=Constant(...)),
        ...,
        FunctionDef(
            name="__getattr__",
            args=arguments(...),
            body=[Expr(...), ..., Return(...)],
            decorator_list=[],
            returns=None,
            type_comment=None,
            type_params=[]
        )
    ],
    type_ignores=[]
)

```
Happy to take feedback :)

<!-- gh-issue-number: gh-116022 -->
* Issue: gh-116022
<!-- /gh-issue-number -->


Problem statement:
Improve `repr()` of AST nodes
# Feature or enhancement

### Proposal:

I often use `ast.parse` in the terminal to explore what the AST looks like:

```
>>> ast.parse("x = 3")
<ast.Module object at 0x105450b50>
```

But I have to remember to use `ast.dump()` to get useful output:
```
>>> ast.dump(ast.parse("x = 3"))
"Module(body=[Assign(targets=[Name(id='x', ctx=Store())], value=Constant(value=3))], type_ignores=[])"
```

It would be nice if the default repr() of AST nodes was more like the output of `ast.dump()`, so it's easier to see at a glance how it works.

One concern would be around the size of the output:

```
>>> from pathlib import Path
>>> import typing
>>> typing_py = Path(typing.__file__).read_text()
>>> len(ast.dump(ast.parse(typing_py)))
304244
```

As a middle ground, we could limit the depth of the AST provided in the repr(), e.g. to 2 levels, and also the number of list elements provided.

The repr() of a module's AST might then look something like:

```
Module(body=[Expr(value=Constant(...)), ..., Assign(targets=[Name(...)], value=Constant(...))], type_ignores=[])
```


### Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

### Links to previous discussion of this feature:

_No response_

<!-- gh-linked-prs -->
### Linked PRs
* gh-117046
<!-- /gh-linked-prs -->

Hints from issue discussion:
This is something that would be useful for me as well since I also sometimes poke around the AST :smile: 

Were you planning to implement this yourself or do you think I could give it a try? Looking at the code, it'll probably have to be implemented in C since that's where the base AST type is defined.
I will do it eventually if no one else gets to it but haven't started yet. You're welcome to start working on it! This might need changes to `asdl_c.py` which is the Python code that generates the C code for the AST.

Commit message: Add news entry

Patch to review:
diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-03-19-22-21-22.gh-issue-116022.iyHENN.rst b/Misc/NEWS.d/next/Core and Builtins/2024-03-19-22-21-22.gh-issue-116022.iyHENN.rst
new file mode 100644
index 00000000000000..f696a4e30fb89b
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2024-03-19-22-21-22.gh-issue-116022.iyHENN.rst	
@@ -0,0 +1 @@
+Improve ``repr()`` of AST nodes
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index 59cc391881ab86..e9621a70b8dbca 100755
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -1129,8 +1129,171 @@ def visitModule(self, mod):
     {NULL}
 };
 
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth);
+
+static PyObject *
+ast_repr_list(PyObject *list, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    Py_ssize_t length = PySequence_Size(list);
+    if (length == 0) {
+        return PyObject_Repr(list);
+    }
+
+    _PyUnicodeWriter writer;
+    bool is_list = PyList_Check(list);
+    PyObject *items[2];
+
+    items[0] = PySequence_GetItem(list, 0);
+    if (length > 1) {
+        items[1] = PySequence_GetItem(list, length - 1);
+    }
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? '[' : '(') < 0) {
+        goto error;
+    }
+
+    for (Py_ssize_t i = 0; i < Py_MIN(length, 2); i++) {
+        PyObject *item = items[i];
+        PyObject *item_repr;
+
+        if (PyType_IsSubtype(Py_TYPE(item), (PyTypeObject *)state->AST_type)) {
+            item_repr = ast_repr_max_depth((AST_object*)item, depth - 1);
+        } else {
+            item_repr = PyObject_Repr(item);
+        }
+        if (!item_repr) {
+            goto error;
+        }
+        if (i > 0) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ", 2) < 0) {
+                goto error;
+            }
+        }
+        if (_PyUnicodeWriter_WriteStr(&writer, item_repr) < 0) {
+            Py_DECREF(item_repr);
+            goto error;
+        }
+        if (i == 0 && length > 2) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ...", 5) < 0) {
+                Py_DECREF(item_repr);
+                goto error;
+            }
+        }
+        Py_DECREF(item_repr);
+    }
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? ']' : ')') < 0) {
+        goto error;
+    }
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    if (depth == 0) {
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    int status = Py_ReprEnter((PyObject*)self);
+    if (status != 0) {
+        if (status < 0)
+            return NULL;
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *fields;
+    PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields);
+    Py_ssize_t numfields = PySequence_Size(fields);
+
+    if (numfields == 0) {
+        Py_ReprLeave((PyObject*)self);
+        Py_DECREF(fields);
+        return PyUnicode_FromFormat("%s()", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *repr = NULL;
+    for (Py_ssize_t i = 0; i < numfields; i++) {
+        PyObject *name = PySequence_GetItem(fields, i);
+        if (!name) {
+            goto error;
+        }
+
+        PyObject *value = PyObject_GetAttr((PyObject*)self, name);
+        if (!value) {
+            Py_DECREF(name);
+            goto error;
+        }
+
+        PyObject *value_repr;
+        if (PyList_Check(value) || PyTuple_Check(value)) {
+            value_repr = ast_repr_list(value, depth);
+        } else if(PyType_IsSubtype(Py_TYPE(value), (PyTypeObject *)state->AST_type)) {
+            value_repr = ast_repr_max_depth((AST_object*)value, depth - 1);
+        } else {
+            value_repr = PyObject_Repr(value);
+        }
+
+        if (!value_repr) {
+            Py_DECREF(name);
+            Py_DECREF(value);
+            goto error;
+        }
+
+        if (i == 0) {
+            repr = PyUnicode_FromFormat("%U=%U", name, value_repr);
+        } else {
+            PyObject *prev_repr = repr;
+            PyObject *tmp = PyUnicode_FromFormat(", %U=%U", name, value_repr);
+            repr = PyUnicode_Concat(prev_repr, tmp);
+            Py_DECREF(prev_repr);
+            Py_DECREF(tmp);
+        }
+        Py_DECREF(name);
+        Py_DECREF(value);
+        Py_DECREF(value_repr);
+    }
+
+    Py_ReprLeave((PyObject*)self);
+    Py_DECREF(fields);
+    PyObject *tmp = repr;
+    repr = PyUnicode_FromFormat("%s(%U)", Py_TYPE(self)->tp_name, repr);
+    Py_DECREF(tmp);
+    return repr;
+
+error:
+    Py_ReprLeave((PyObject*)self);
+    Py_DECREF(fields);
+    return NULL;
+}
+
+static PyObject *
+ast_repr(AST_object *self)
+{
+    return ast_repr_max_depth(self, 2);
+}
+
 static PyType_Slot AST_type_slots[] = {
     {Py_tp_dealloc, ast_dealloc},
+    {Py_tp_repr, ast_repr},
     {Py_tp_getattro, PyObject_GenericGetAttr},
     {Py_tp_setattro, PyObject_GenericSetAttr},
     {Py_tp_traverse, ast_traverse},
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index 7b591ddaa29869..9047422089f9d1 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -5275,8 +5275,171 @@ static PyGetSetDef ast_type_getsets[] = {
     {NULL}
 };
 
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth);
+
+static PyObject *
+ast_repr_list(PyObject *list, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    Py_ssize_t length = PySequence_Size(list);
+    if (length == 0) {
+        return PyObject_Repr(list);
+    }
+
+    _PyUnicodeWriter writer;
+    bool is_list = PyList_Check(list);
+    PyObject *items[2];
+
+    items[0] = PySequence_GetItem(list, 0);
+    if (length > 1) {
+        items[1] = PySequence_GetItem(list, length - 1);
+    }
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? '[' : '(') < 0) {
+        goto error;
+    }
+
+    for (Py_ssize_t i = 0; i < Py_MIN(length, 2); i++) {
+        PyObject *item = items[i];
+        PyObject *item_repr;
+
+        if (PyType_IsSubtype(Py_TYPE(item), (PyTypeObject *)state->AST_type)) {
+            item_repr = ast_repr_max_depth((AST_object*)item, depth - 1);
+        } else {
+            item_repr = PyObject_Repr(item);
+        }
+        if (!item_repr) {
+            goto error;
+        }
+        if (i > 0) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ", 2) < 0) {
+                goto error;
+            }
+        }
+        if (_PyUnicodeWriter_WriteStr(&writer, item_repr) < 0) {
+            Py_DECREF(item_repr);
+            goto error;
+        }
+        if (i == 0 && length > 2) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ...", 5) < 0) {
+                Py_DECREF(item_repr);
+                goto error;
+            }
+        }
+        Py_DECREF(item_repr);
+    }
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? ']' : ')') < 0) {
+        goto error;
+    }
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    if (depth == 0) {
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    int status = Py_ReprEnter((PyObject*)self);
+    if (status != 0) {
+        if (status < 0)
+            return NULL;
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *fields;
+    PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields);
+    Py_ssize_t numfields = PySequence_Size(fields);
+
+    if (numfields == 0) {
+        Py_ReprLeave((PyObject*)self);
+        Py_DECREF(fields);
+        return PyUnicode_FromFormat("%s()", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *repr = NULL;
+    for (Py_ssize_t i = 0; i < numfields; i++) {
+        PyObject *name = PySequence_GetItem(fields, i);
+        if (!name) {
+            goto error;
+        }
+
+        PyObject *value = PyObject_GetAttr((PyObject*)self, name);
+        if (!value) {
+            Py_DECREF(name);
+            goto error;
+        }
+
+        PyObject *value_repr;
+        if (PyList_Check(value) || PyTuple_Check(value)) {
+            value_repr = ast_repr_list(value, depth);
+        } else if(PyType_IsSubtype(Py_TYPE(value), (PyTypeObject *)state->AST_type)) {
+            value_repr = ast_repr_max_depth((AST_object*)value, depth - 1);
+        } else {
+            value_repr = PyObject_Repr(value);
+        }
+
+        if (!value_repr) {
+            Py_DECREF(name);
+            Py_DECREF(value);
+            goto error;
+        }
+
+        if (i == 0) {
+            repr = PyUnicode_FromFormat("%U=%U", name, value_repr);
+        } else {
+            PyObject *prev_repr = repr;
+            PyObject *tmp = PyUnicode_FromFormat(", %U=%U", name, value_repr);
+            repr = PyUnicode_Concat(prev_repr, tmp);
+            Py_DECREF(prev_repr);
+            Py_DECREF(tmp);
+        }
+        Py_DECREF(name);
+        Py_DECREF(value);
+        Py_DECREF(value_repr);
+    }
+
+    Py_ReprLeave((PyObject*)self);
+    Py_DECREF(fields);
+    PyObject *tmp = repr;
+    repr = PyUnicode_FromFormat("%s(%U)", Py_TYPE(self)->tp_name, repr);
+    Py_DECREF(tmp);
+    return repr;
+
+error:
+    Py_ReprLeave((PyObject*)self);
+    Py_DECREF(fields);
+    return NULL;
+}
+
+static PyObject *
+ast_repr(AST_object *self)
+{
+    return ast_repr_max_depth(self, 2);
+}
+
 static PyType_Slot AST_type_slots[] = {
     {Py_tp_dealloc, ast_dealloc},
+    {Py_tp_repr, ast_repr},
     {Py_tp_getattro, PyObject_GenericGetAttr},
     {Py_tp_setattro, PyObject_GenericSetAttr},
     {Py_tp_traverse, ast_traverse},


Code review comments:
diff_hunk: @@ -5275,8 +5275,171 @@ static PyGetSetDef ast_type_getsets[] = {
     {NULL}
 };
 
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth);
+
+static PyObject *
+ast_repr_list(PyObject *list, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    Py_ssize_t length = PySequence_Size(list);
+    if (length == 0) {
+        return PyObject_Repr(list);
+    }
+
+    _PyUnicodeWriter writer;
+    bool is_list = PyList_Check(list);
+    PyObject *items[2];
+
+    items[0] = PySequence_GetItem(list, 0);
+    if (length > 1) {
+        items[1] = PySequence_GetItem(list, length - 1);
+    }
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? '[' : '(') < 0) {
+        goto error;
+    }
+
+    for (Py_ssize_t i = 0; i < Py_MIN(length, 2); i++) {
+        PyObject *item = items[i];
+        PyObject *item_repr;
+
+        if (PyType_IsSubtype(Py_TYPE(item), (PyTypeObject *)state->AST_type)) {
+            item_repr = ast_repr_max_depth((AST_object*)item, depth - 1);
+        } else {
+            item_repr = PyObject_Repr(item);
+        }
+        if (!item_repr) {
+            goto error;
+        }
+        if (i > 0) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ", 2) < 0) {
+                goto error;
+            }
+        }
+        if (_PyUnicodeWriter_WriteStr(&writer, item_repr) < 0) {
+            Py_DECREF(item_repr);
+            goto error;
+        }
+        if (i == 0 && length > 2) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ...", 5) < 0) {
+                Py_DECREF(item_repr);
+                goto error;
+            }
+        }
+        Py_DECREF(item_repr);
+    }
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? ']' : ')') < 0) {
+        goto error;
+    }
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    if (depth == 0) {
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    int status = Py_ReprEnter((PyObject*)self);
+    if (status != 0) {
+        if (status < 0)
+            return NULL;
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *fields;
+    PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields);
+    Py_ssize_t numfields = PySequence_Size(fields);
+
+    if (numfields == 0) {
+        Py_ReprLeave((PyObject*)self);
+        Py_DECREF(fields);
+        return PyUnicode_FromFormat("%s()", Py_TYPE(self)->tp_name);
+    }
+
+    PyObject *repr = NULL;
+    for (Py_ssize_t i = 0; i < numfields; i++) {
+        PyObject *name = PySequence_GetItem(fields, i);
+        if (!name) {
+            goto error;
+        }
+
+        PyObject *value = PyObject_GetAttr((PyObject*)self, name);
+        if (!value) {
+            Py_DECREF(name);
+            goto error;
+        }
+
+        PyObject *value_repr;
+        if (PyList_Check(value) || PyTuple_Check(value)) {
+            value_repr = ast_repr_list(value, depth);
+        } else if(PyType_IsSubtype(Py_TYPE(value), (PyTypeObject *)state->AST_type)) {
+            value_repr = ast_repr_max_depth((AST_object*)value, depth - 1);
+        } else {
+            value_repr = PyObject_Repr(value);
+        }
+
+        if (!value_repr) {
+            Py_DECREF(name);
+            Py_DECREF(value);
+            goto error;
+        }
+
+        if (i == 0) {
+            repr = PyUnicode_FromFormat("%U=%U", name, value_repr);
+        } else {
+            PyObject *prev_repr = repr;
+            PyObject *tmp = PyUnicode_FromFormat(", %U=%U", name, value_repr);
+            repr = PyUnicode_Concat(prev_repr, tmp);
+            Py_DECREF(prev_repr);
+            Py_DECREF(tmp);
path: Python/Python-ast.c
text: @author:
This part is a bit iffy, I'll try to refactor it using `_PyUnicodeWriter`

diff_hunk: @@ -1129,8 +1129,171 @@ def visitModule(self, mod):
     {NULL}
 };
 
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth);
+
+static PyObject *
+ast_repr_list(PyObject *list, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    Py_ssize_t length = PySequence_Size(list);
+    if (length == 0) {
+        return PyObject_Repr(list);
+    }
+
+    _PyUnicodeWriter writer;
+    bool is_list = PyList_Check(list);
+    PyObject *items[2];
+
+    items[0] = PySequence_GetItem(list, 0);
+    if (length > 1) {
+        items[1] = PySequence_GetItem(list, length - 1);
+    }
+
+    _PyUnicodeWriter_Init(&writer);
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? '[' : '(') < 0) {
+        goto error;
+    }
+
+    for (Py_ssize_t i = 0; i < Py_MIN(length, 2); i++) {
+        PyObject *item = items[i];
+        PyObject *item_repr;
+
+        if (PyType_IsSubtype(Py_TYPE(item), (PyTypeObject *)state->AST_type)) {
+            item_repr = ast_repr_max_depth((AST_object*)item, depth - 1);
+        } else {
+            item_repr = PyObject_Repr(item);
+        }
+        if (!item_repr) {
+            goto error;
+        }
+        if (i > 0) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ", 2) < 0) {
+                goto error;
+            }
+        }
+        if (_PyUnicodeWriter_WriteStr(&writer, item_repr) < 0) {
+            Py_DECREF(item_repr);
+            goto error;
+        }
+        if (i == 0 && length > 2) {
+            if (_PyUnicodeWriter_WriteASCIIString(&writer, ", ...", 5) < 0) {
+                Py_DECREF(item_repr);
+                goto error;
+            }
+        }
+        Py_DECREF(item_repr);
+    }
+
+    if (_PyUnicodeWriter_WriteChar(&writer, is_list ? ']' : ')') < 0) {
+        goto error;
+    }
+
+    return _PyUnicodeWriter_Finish(&writer);
+
+error:
+    _PyUnicodeWriter_Dealloc(&writer);
+    return NULL;
+}
+
+static PyObject *
+ast_repr_max_depth(AST_object *self, int depth)
+{
+    struct ast_state *state = get_ast_state();
+    if (state == NULL) {
+        return NULL;
+    }
+
+    if (depth == 0) {
+        return PyUnicode_FromFormat("%s(...)", Py_TYPE(self)->tp_name);
+    }
+
+    int status = Py_ReprEnter((PyObject*)self);
+    if (status != 0) {
+        if (status < 0)
+            return NULL;
path: Parser/asdl_c.py
text: @user1:
Nitpick: According to PEP7, braces are always required.

```suggestion
        if (status < 0) {
            return NULL;
        }
```