Repository: ipython/ipython

Issue:
Proposal: highlight exact operation in tracebacks
Hi! I recently created [executing](https://github.com/alexmojaki/executing), a library which can identify the AST node being executed by a frame in many cases. I'd like to use this to improve Python tracebacks in many places, starting with IPython. It's always disappointed me that Python only points to a line in each frame in tracebacks, especially when the expression/statement at fault spans many lines. Sometimes it can be really hard to interpret.

[This toy library](https://github.com/alexmojaki/heartrate) offers a nice example of what tracebacks using `executing` can look like:

![heartrate](https://camo.githubusercontent.com/fa5b5cff6ad8059896612028a1c1cac68dc7f21b/68747470733a2f2f6d656469612e67697068792e636f6d2f6d656469612f56495171593879796a596b684e66774632392f67697068792e676966)

Basically, highlighting the text background could make it clear what's being executed without interfering with other coloring and formatting such as syntax highlighting.

I tried to jump to coding this straight away but quickly saw that it's quite complicated. The introspection magic stuff is already done, it's just a matter of updating the formatting in `ultratb`, but that still looks quite hard, especially as I'm new to this code. So I'm creating this issue first to check if the idea is acceptable before wasting my time.

I'm expecting two main concerns: reliability and dependencies.

[`executing` is extremely reliable](https://github.com/alexmojaki/executing#is-it-reliable). By that I mean that I'm confident it will never identify the wrong node. If the linked explanation doesn't convince you, I encourage you to try finding a case where it goes wrong. If it helps, I'm also using `executing` in two other libraries of mine: [`sorcery`](https://github.com/alexmojaki/sorcery) and [`snoop`](https://github.com/alexmojaki/snoop). However `executing` often can't identify the node at all, so not all frames will have something highlighted. Over time this situation will improve as I implement more operations.

Two new dependencies would need to be added: `executing` and [`asttokens`](https://github.com/gristlabs/asttokens) (to convert AST nodes to source code positions, I don't know any alternative). The only transitive dependency from these is `six`. In my experience `asttokens` is an excellent library with active and responsive contributors.

Does this sound good to you?


Discussion:
Ultratb is a real mess and really old. Any refactor to it is welcomed; I've been willing from some time to separate the traceback information extracting from the formatting, to potentially make a nice HTML repr... 

I don't have any objections to add dependencies; we might want to make sure they are packaged in conda/debian and some of our downstream redistributor and if not asses the difficulty this would add for them. (asttokens already available, and six as well)

@scopatz and @gforsyth  might be interested in this for xonsh. If we want/can extract these into common packages I'm all for it. 

There is also https://github.com/qix-/better-exceptions that looks nice; though showing variable values can be a security issue s can't be enabled by default.

If that gets in, It will probably be for an 8.0 release. 
Totally in favor of extracting these components into a common package
Would be happy to collaborate :)
@alexmojaki I would love to see subclasses of the standard library `traceback` module classes that include this extra information. Does something like this already exist?
@asmeurer not right now, no.

If you're coming from python-ideas, I should clarify that there I was only talking about including the appropriate *lines*. That's easy to do and it's fairly obvious what it would look like.

This issue is about doing more than that - highlighting the exact call, subscript, etc. *within* the line(s). That means either adding colour or some additional symbols to the output.
I'm not coming from python-ideas (I'm coming from Twitter), so I don't know what you're talking about there.

I think it would be awesome if Python included this information in frames natively. Is this what the python-ideas discussion was? I'm assuming this has been discussed before, but I'm not sure why it isn't done. It would really help things like debugging as well, e.g., if you could step into an inner function call directly.

To clarify, I personally use the `traceback` module to generate tracebacks, and I would love to extend them with this column info. I don't use whatever IPython uses to generate tracebacks. The formatting would be a separate problem. I just want the information there. So for instance, a custom [`TracebackException`](https://docs.python.org/3/library/traceback.html#traceback.TracebackException) subclass that adds `col_start` and `col_end` attributes. 
Huh, maybe it's just coincidental timing. What conversation on Twitter led you here?

> The formatting would be a separate problem. I just want the information there.

Well that's great, because that's what [`stack_data`](https://github.com/alexmojaki/stack_data) is for! Here's an example:

```python
import sys

import stack_data


def print_stack():
    options = stack_data.Options(before=0, after=0)
    for frame_info in stack_data.FrameInfo.stack_data(sys.exc_info()[2], options):
        if isinstance(frame_info, stack_data.FrameInfo):
            print(f'  File "{frame_info.filename}", '
                  f'line {frame_info.lineno}, '
                  f'in {frame_info.executing.code_qualname()}')
            for line in frame_info.lines:
                if line is stack_data.LINE_GAP:
                    print("       (...)")
                else:
                    ranges = line.executing_node_ranges
                    # print(ranges)  # The raw column data is in here
                    
                    def convert_range(_):
                        return '{{', '}}'

                    markers = stack_data.markers_from_ranges(ranges, convert_range)
                    print(f"{'-->' if line.is_current else '   '} {line.lineno:4} | {line.render(markers)}")
        else:
            print(f"... {frame_info.description} ...\n")


def main():
    lst = [[[0]]]
    try:
        print(
            lst[0][1][2]
        )
    except IndexError:
        print_stack()


main()
```
> Huh, maybe it's just coincidental timing. What conversation on Twitter led you here?

https://twitter.com/Mbussonn/status/1227386489987723264

I don't want to hijack this conversation any more. I'll open an issue on the stack_data repo. 

Resolving patch:
diff --git a/IPython/core/ultratb.py b/IPython/core/ultratb.py
--- a/IPython/core/ultratb.py
+++ b/IPython/core/ultratb.py
@@ -98,6 +98,7 @@
 
 import stack_data
 from pygments.formatters.terminal256 import Terminal256Formatter
+from pygments.styles import get_style_by_name
 
 # IPython's own modules
 from IPython import get_ipython
@@ -714,7 +715,9 @@ def get_records(self, etb, number_of_lines_of_context, tb_offset):
         after = context // 2
         before = context - after
         if self.has_colors:
-            formatter = Terminal256Formatter()
+            style = get_style_by_name('default')
+            style = stack_data.style_with_executing_node(style, 'bg:#00005f')
+            formatter = Terminal256Formatter(style=style)
         else:
             formatter = None
         options = stack_data.Options(


Test patch: