Repository: gristlabs/asttokens

Issue:
Consider a clearer optional-functionality API
Currently `ASTTokens` serves a number of different roles depending on what arguments its passed. With https://github.com/gristlabs/asttokens/pull/93 this is growing further and the workarounds to type checking being added in that PR perhaps suggest that the limits of the current approach are being reached. https://github.com/gristlabs/asttokens/pull/95 will add another construction, though not another role.
I understand the goals of these different roles are around avoiding redundant parsing/tree-building work where not needed, which is a reasonable goal. I'm not suggesting a removal of functionality here, rather that the functionality be rearranged for clarity and to aid type checking.

There are at least the following modes of use:
``` python
atok = ASTTokens(source, parse=True)
assert atok.tree, "Placate type check even though we know this is always valid"

atok = ASTTokens(source, tree=existing_tree)
assert atok.tree, "Placate type check even though we know this is always valid"
```

There is also currently `ASTTokens(source, tree=None, parse=False)`, though it's not clear what use that has (https://github.com/gristlabs/asttokens/issues/94#issuecomment-1252862533).

With #93 there's also this case:
``` python
atok = ASTTokens(source, init_tokens=False)
atok.get_token_from_offset(42)  # Fail at runtime, but no type-check warnings
```

These cases are somewhat "obvious", however if the construction of the `ASTTokens` instance is far from the point of use they could easily be very non-obvious.

We can observe that `ASTTokens` contains broadly three classes of functionality:
- `tokens`-only
- `tree`-only
- both together

With the goal of enabling efficient use of the related functionalities I can see a couple of paths forwards:
- lazy evaluation of the relevant structured data from the source
- failures when using functionality with missing data

For the former I would advocate for moving all the public-member types to being non-optional and using something like https://pypi.org/project/cached-property/ (or [`functools.cached_property`](https://docs.python.org/3/library/functools.html#functools.cached_property) in Python 3.8+) to hide the evaluations. This would still support users passing in pre-computed values if they want but means that all usages will "just work":
``` python
atok = ASTTokens(source)
tree: ast.AST = atok.tree  # no error now that `.tree` is non-optional

atok = ASTTokens(source, init_tokens=False)  # request lazy tokens initialisation
atok.get_token_from_offset(42)  # works due to lazy parsing
```

I can see that this approach may be undesirable in some cases, perhaps where there are reasons to prefer not doing the parsing _at all_.

This leads to the other approach (errors for missing functionality), for which I propose using separate classes:

``` python
# ASTSource is a thin wrapper around LineNumbers. (Very open to better names.)
# Does not take a tree or tokens at all, but works with un-marked AST nodes
asrc = ASTSource(source)
asrc.tree  # type check error: no such member
asrc.get_text(node)  # type check error: no such member
asrc.get_token_from_offset(42)  # type check error: no such member
asrc.get_text_unmarked(node)  # works fine

# TokensSource is a new type which contains the tokens/source utils currently in ASTTokens
# It wraps LineNumbers similarly to ASTSource, but is otherwise unrelated
# It does not have a tree
tsrc = TokensSource(source)  # parses tokens if not passed
tsrc.tree  # type check error: no such member
tsrc.get_text(node)  # type check error: no such member
tsrc.get_token_from_offset(42)  # works fine
tsrc.get_text_unmarked(node)  # type check error: no such member

# ASTTokens now always has a tree and tokens, creating them both if not required
# It probably inherits from TokensSource and drops the `get_text*_unmarked` methods as they no longer make sense here
atok = ASTTokens(source)
tree: ast.AST = atok.tree  # no error now that `.tree` is non-optional
atok.get_text(node)  # works fine
atok.get_token_from_offset(42)  # also works fine
atok.get_text_unmarked(node)  # type check error: no such member
```

Aside from the question of whether `ASTTokens` keeps its `get_text*_unmarked` methods, the usages here are essentially compatible with existing code. The constructor signature might change, though keeping the existing arguments for compatibility is probably possible; if so then `ASTTokens(source, tree=None, parse=False)` would now be at least a runtime error (with `@overload`s it could be a type error too).

Note also that this approach is compatible with another class, perhaps `LazyASTTokens` which looks like `ASTTokens` but has lazy evaluation. This could easily be added later (or not at all) depending on whether it ends up being useful.

The main gain here is that each of these specialised classes addresses a specific use-case and data-requirements, which remove the need for conditional presence checks (or asserts) throughout both the library code and client code. This will make the code clearer, easier to maintain and (due to the type checker support) more robust as well as offering the performance gains desired.

Discussion:

Patch:
diff --git a/asttokens/__init__.py b/asttokens/__init__.py
index e8686d4..eeda0ed 100644
--- a/asttokens/__init__.py
+++ b/asttokens/__init__.py
@@ -19,6 +19,6 @@ transformations.
 """
 
 from .line_numbers import LineNumbers
-from .asttokens import ASTTokens
+from .asttokens import ASTText, ASTTokens, supports_tokenless
 
-__all__ = ['ASTTokens', 'LineNumbers']
+__all__ = ['ASTText', 'ASTTokens', 'LineNumbers', 'supports_tokenless']
diff --git a/asttokens/asttokens.py b/asttokens/asttokens.py
index ed1b336..d3c9d01 100644
--- a/asttokens/asttokens.py
+++ b/asttokens/asttokens.py
@@ -12,24 +12,81 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import abc
 import ast
 import bisect
 import sys
 import token
 from ast import Module
-from typing import Iterable, Iterator, List, Optional, Tuple, Any, cast, TYPE_CHECKING
+from typing import Iterable, Iterator, List, Optional, Tuple, Any, cast, TYPE_CHECKING, Type
 
 import six
 from six.moves import xrange  # pylint: disable=redefined-builtin
 
 from .line_numbers import LineNumbers
-from .util import Token, match_token, is_non_coding_token, patched_generate_tokens, last_stmt, generate_tokens
+from .util import Token, match_token, is_non_coding_token, patched_generate_tokens, last_stmt, annotate_fstring_nodes, generate_tokens
 
 if TYPE_CHECKING:  # pragma: no cover
   from .util import AstNode, TokenInfo
 
 
-class ASTTokens(object):
+class ASTTextBase(six.with_metaclass(abc.ABCMeta, object)):
+  def __init__(self, source_text, filename):
+    # type: (Any, str) -> None
+    # FIXME: Strictly, the type of source_text is one of the six string types, but hard to specify with mypy given
+    # https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+
+    self._filename = filename
+
+    # Decode source after parsing to let Python 2 handle coding declarations.
+    # (If the encoding was not utf-8 compatible, then even if it parses correctly,
+    # we'll fail with a unicode error here.)
+    source_text = six.ensure_text(source_text)
+
+    self._text = source_text
+    self._line_numbers = LineNumbers(source_text)
+
+  @abc.abstractmethod
+  def get_text_positions(self, node, padded):
+    # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
+    """
+    Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node.
+    If the positions can't be determined, or the nodes don't correspond to any particular text,
+    returns ``(1, 0)`` for both.
+
+    ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``.
+    This means that if ``padded`` is True, the start position will be adjusted to include
+    leading whitespace if ``node`` is a multiline statement.
+    """
+    raise NotImplementedError
+
+  def get_text_range(self, node, padded=True):
+    # type: (AstNode, bool) -> Tuple[int, int]
+    """
+    Returns the (startpos, endpos) positions in source text corresponding to the given node.
+    Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text.
+
+    See ``get_text_positions()`` for details on the ``padded`` argument.
+    """
+    start, end = self.get_text_positions(node, padded)
+    return (
+      self._line_numbers.line_to_offset(*start),
+      self._line_numbers.line_to_offset(*end),
+    )
+
+  def get_text(self, node, padded=True):
+    # type: (AstNode, bool) -> str
+    """
+    Returns the text corresponding to the given node.
+    Returns '' for nodes (like `Load`) that don't correspond to any particular text.
+
+    See ``get_text_positions()`` for details on the ``padded`` argument.
+    """
+    start, end = self.get_text_range(node, padded)
+    return self._text[start: end]
+
+
+class ASTTokens(ASTTextBase, object):
   """
   ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and
   as tokens, and is used to mark and access token and position information.
@@ -48,25 +105,19 @@ class ASTTokens(object):
   If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST
   tree created separately.
   """
+
   def __init__(self, source_text, parse=False, tree=None, filename='<unknown>', tokens=None):
     # type: (Any, bool, Optional[Module], str, Iterable[TokenInfo]) -> None
-    # FIXME: Strictly, the type of source_type is one of the six string types, but hard to specify with mypy given
+    # FIXME: Strictly, the type of source_text is one of the six string types, but hard to specify with mypy given
     # https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
 
-    self._filename = filename
-    self._tree = ast.parse(source_text, filename) if parse else tree
+    super(ASTTokens, self).__init__(source_text, filename)
 
-    # Decode source after parsing to let Python 2 handle coding declarations.
-    # (If the encoding was not utf-8 compatible, then even if it parses correctly,
-    # we'll fail with a unicode error here.)
-    source_text = six.ensure_text(source_text)
-
-    self._text = source_text
-    self._line_numbers = LineNumbers(source_text)
+    self._tree = ast.parse(source_text, filename) if parse else tree
 
     # Tokenize the code.
     if tokens is None:
-      tokens = generate_tokens(source_text)
+      tokens = generate_tokens(self._text)
     self._tokens = list(self._translate_tokens(tokens))
 
     # Extract the start positions of all tokens, so that we can quickly map positions to tokens.
@@ -75,7 +126,6 @@ class ASTTokens(object):
     if self._tree:
       self.mark_tokens(self._tree)
 
-
   def mark_tokens(self, root_node):
     # type: (Module) -> None
     """
@@ -88,7 +138,6 @@ class ASTTokens(object):
     from .mark_tokens import MarkTokens # to avoid import loops
     MarkTokens(self).visit_tree(root_node)
 
-
   def _translate_tokens(self, original_tokens):
     # type: (Iterable[TokenInfo]) -> Iterator[Token]
     """
@@ -209,86 +258,188 @@ class ASTTokens(object):
     """
     return self.token_range(node.first_token, node.last_token, include_extra=include_extra)
 
-  def get_text_range(self, node):
-    # type: (AstNode) -> Tuple[int, int]
+  def get_text_positions(self, node, padded):
+    # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
     """
-    After mark_tokens() has been called, returns the (startpos, endpos) positions in source text
-    corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond
-    to any particular text.
+    Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node.
+    If the positions can't be determined, or the nodes don't correspond to any particular text,
+    returns ``(1, 0)`` for both.
+
+    ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``.
+    This means that if ``padded`` is True, the start position will be adjusted to include
+    leading whitespace if ``node`` is a multiline statement.
     """
     if not hasattr(node, 'first_token'):
-      return (0, 0)
+      return (1, 0), (1, 0)
 
-    start = node.first_token.startpos
-    if any(match_token(t, token.NEWLINE) for t in self.get_tokens(node)):
-      # Multi-line nodes would be invalid unless we keep the indentation of the first node.
-      start = self._text.rfind('\n', 0, start) + 1
+    start = node.first_token.start
+    end = node.last_token.end
+    if padded and any(match_token(t, token.NEWLINE) for t in self.get_tokens(node)):
+      # Set col_offset to 0 to include leading indentation for multiline statements.
+      start = (start[0], 0)
 
-    return (start, node.last_token.endpos)
+    return start, end
 
-  def get_text(self, node):
-    # type: (AstNode) -> str
-    """
-    After mark_tokens() has been called, returns the text corresponding to the given node. Returns
-    '' for nodes (like `Load`) that don't correspond to any particular text.
-    """
-    start, end = self.get_text_range(node)
-    return self._text[start : end]
 
-  def get_text_range_unmarked(self, node):
-    # type: (ast.AST) -> Tuple[int, int]
+class ASTText(ASTTextBase, object):
+  """
+  Supports the same ``get_text*`` methods as ``ASTTokens``,
+  but uses the AST to determine the text positions instead of tokens.
+  This is faster than ``ASTTokens`` as it requires less setup work.
+
+  It also (sometimes) supports nodes inside f-strings, which ``ASTTokens`` doesn't.
+
+  Astroid trees are not supported at all and will raise an error.
+
+  Some node types and/or Python versions are not supported.
+  In these cases the ``get_text*`` methods will fall back to using ``ASTTokens``
+  which incurs the usual setup cost the first time.
+  If you want to avoid this, check ``supports_tokenless(node)`` before calling ``get_text*`` methods.
+  """
+  def __init__(self, source_text, tree=None, filename='<unknown>'):
+    # type: (Any, Optional[Module], str) -> None
+    # FIXME: Strictly, the type of source_text is one of the six string types, but hard to specify with mypy given
+    # https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+
+    if not isinstance(tree, (ast.AST, type(None))):
+      raise NotImplementedError('ASTText only supports AST trees')
+
+    super(ASTText, self).__init__(source_text, filename)
+
+    self._tree = tree
+    if self._tree is not None:
+      annotate_fstring_nodes(self._tree)
+
+    self._asttokens = None  # type: Optional[ASTTokens]
+
+  @property
+  def tree(self):
+    # type: () -> Module
+    if self._tree is None:
+      self._tree = ast.parse(self._text, self._filename)
+      annotate_fstring_nodes(self._tree)
+    return self._tree
+
+  @property
+  def asttokens(self):
+    # type: () -> ASTTokens
+    if self._asttokens is None:
+      self._asttokens = ASTTokens(
+          self._text,
+          tree=self.tree,
+          filename=self._filename,
+      )
+    return self._asttokens
+
+  def _get_text_positions_tokenless(self, node, padded):
+    # type: (ast.AST, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
     """
-    Like get_text_range(), but works without requiring mark_tokens() to have been called.
-    Requires Python 3.8+. Doesn't support astroid.
+    Version of ``get_text_positions()`` that doesn't use tokens.
     """
-    if sys.version_info < (3, 8):
-      raise NotImplementedError('Requires Python 3.8+')
-
-    if not isinstance(node, ast.AST):
-      raise NotImplementedError('Not supported for astroid')
+    if sys.version_info[:2] < (3, 8):
+      raise AssertionError("This method should only be called internally after checking supports_tokenless()")
 
     if isinstance(node, ast.Module):
       # Modules don't have position info, so just return the range of the whole text.
-      # get_text does something different, but its behavior seems weird and inconsistent.
+      # The token-using method does something different, but its behavior seems weird and inconsistent.
       # For example, in a file with only comments, it only returns the first line.
       # It's hard to imagine a case when this matters.
-      return 0, len(self._text)
-    elif not hasattr(node, 'lineno'):
-      return 0, 0
+      return (1, 0), self._line_numbers.offset_to_line(len(self._text))
+
+    if not hasattr(node, 'lineno'):
+      return (1, 0), (1, 0)
+
+    assert node  # tell mypy that node is not None, which we allowed up to here for compatibility
+
+    decorators = getattr(node, 'decorator_list', [])
+    if decorators:
+      # Function/Class definition nodes are marked by AST as starting at def/class,
+      # not the first decorator. This doesn't match the token-using behavior,
+      # or inspect.getsource(), and just seems weird.
+      start_node = decorators[0]
     else:
-      decorators = getattr(node, 'decorator_list', [])
-      if decorators:
-        # Function/Class definition nodes are marked by AST as starting at def/class,
-        # not the first decorator. This doesn't match the original get_text[_range] behavior,
-        # or inspect.getsource(), and just seems weird.
-        start_node = decorators[0]
-      else:
-        start_node = node
-      start = self._line_numbers.utf8_to_offset(
-        start_node.lineno,
-        start_node.col_offset,
-      )
-      # Like get_text_range(), keep the indentation of the first line
-      # of a multi-line, multi-statement node.
-      if last_stmt(node).lineno != node.lineno:
-        start = self._text.rfind('\n', 0, start) + 1
+      start_node = node
 
-    # To match the behavior of get_text_range, we exclude trailing semicolons and comments.
+    if padded and last_stmt(node).lineno != node.lineno:
+      # Include leading indentation for multiline statements.
+      start_col_offset = 0
+    else:
+      start_col_offset = self._line_numbers.from_utf8_col(start_node.lineno, start_node.col_offset)
+
+    start = (start_node.lineno, start_col_offset)
+
+    # To match the token-using behaviour, we exclude trailing semicolons and comments.
     # This means that for blocks containing multiple statements, we have to use the last one
     # instead of the actual node for end_lineno and end_col_offset.
     end_node = last_stmt(node)
-    end = self._line_numbers.utf8_to_offset(
-      cast(int, end_node.end_lineno),
-      cast(int, end_node.end_col_offset),
-    )
+    end_lineno = cast(int, end_node.end_lineno)
+    end_col_offset = cast(int, end_node.end_col_offset)
+    end_col_offset = self._line_numbers.from_utf8_col(end_lineno, end_col_offset)
+    end = (end_lineno, end_col_offset)
 
     return start, end
 
-  def get_text_unmarked(self, node):
-    # type: (ast.AST) -> str
+  def get_text_positions(self, node, padded):
+    # type: (AstNode, bool) -> Tuple[Tuple[int, int], Tuple[int, int]]
     """
-    Like get_text(), but works without requiring mark_tokens() to have been called.
-    Requires Python 3.8+. Doesn't support astroid.
+    Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node.
+    If the positions can't be determined, or the nodes don't correspond to any particular text,
+    returns ``(1, 0)`` for both.
+
+    ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``.
+    This means that if ``padded`` is True, the start position will be adjusted to include
+    leading whitespace if ``node`` is a multiline statement.
     """
-    start, end = self.get_text_range_unmarked(node)
-    return self._text[start:end]
+    if getattr(node, "_broken_positions", None):
+      # This node was marked in util.annotate_fstring_nodes as having untrustworthy lineno/col_offset.
+      return (1, 0), (1, 0)
+
+    if supports_tokenless(node):
+      return self._get_text_positions_tokenless(node, padded)
+
+    return self.asttokens.get_text_positions(node, padded)
+
+
+# Node types that _get_text_positions_tokenless doesn't support. Only relevant for Python 3.8+.
+_unsupported_tokenless_types = ()  # type: Tuple[Type[ast.AST], ...]
+if sys.version_info[:2] >= (3, 8):
+  _unsupported_tokenless_types += (
+    # no lineno
+    ast.arguments, ast.withitem,
+  )
+  if sys.version_info[:2] == (3, 8):
+    _unsupported_tokenless_types += (
+      # _get_text_positions_tokenless works incorrectly for these types due to bugs in Python 3.8.
+      ast.arg, ast.Starred,
+      # no lineno in 3.8
+      ast.Slice, ast.ExtSlice, ast.Index, ast.keyword,
+    )
+
+
+def supports_tokenless(node=None):
+  # type: (Any) -> bool
+  """
+  Returns True if the Python version and the node (if given) are supported by
+  the ``get_text*`` methods of ``ASTText`` without falling back to ``ASTTokens``.
+  See ``ASTText`` for why this matters.
+
+  The following cases are not supported:
+
+    - Python 3.7 and earlier
+    - PyPy
+    - Astroid nodes (``get_text*`` methods of ``ASTText`` will raise an error)
+    - ``ast.arguments`` and ``ast.withitem``
+    - The following nodes in Python 3.8 only:
+      - ``ast.arg``
+      - ``ast.Starred``
+      - ``ast.Slice``
+      - ``ast.ExtSlice``
+      - ``ast.Index``
+      - ``ast.keyword``
+  """
+  return (
+      isinstance(node, (ast.AST, type(None)))
+      and not isinstance(node, _unsupported_tokenless_types)
+      and sys.version_info[:2] >= (3, 8)
+      and 'pypy' not in sys.version.lower()
+  )
diff --git a/asttokens/line_numbers.py b/asttokens/line_numbers.py
index e302ac1..aaf76ce 100644
--- a/asttokens/line_numbers.py
+++ b/asttokens/line_numbers.py
@@ -63,13 +63,6 @@ class LineNumbers(object):
     else:
       return min(self._line_offsets[line] + max(0, column), self._text_len)
 
-  def utf8_to_offset(self, line, utf8_column):
-    # type: (int, int) -> int
-    """
-    Converts 1-based line number and 0-based utf8 column to 0-based character offset into text.
-    """
-    return self.line_to_offset(line, self.from_utf8_col(line, utf8_column))
-
   def offset_to_line(self, offset):
     # type: (int) -> Tuple[int, int]
     """
diff --git a/asttokens/util.py b/asttokens/util.py
index 02b85e8..96fa931 100644
--- a/asttokens/util.py
+++ b/asttokens/util.py
@@ -123,7 +123,7 @@ def generate_tokens(text):
 
 
 def iter_children_func(node):
-  # type: (Module) -> Callable
+  # type: (AST) -> Callable
   """
   Returns a function which yields all direct children of a AST node,
   skipping children that are singleton nodes.
@@ -266,7 +266,7 @@ def visit_tree(node, previsit, postvisit):
 
 
 def walk(node):
-  # type: (Module) -> Iterator[Union[Module, AstNode]]
+  # type: (AST) -> Iterator[Union[Module, AstNode]]
   """
   Recursively yield all descendant nodes in the tree starting at ``node`` (including ``node``
   itself), using depth-first pre-order traversal (yieling parents before their children).
@@ -393,3 +393,68 @@ def last_stmt(node):
   if child_stmts:
     return last_stmt(child_stmts[-1])
   return node
+
+
+if sys.version_info[:2] >= (3, 8):
+  from functools import lru_cache
+
+  @lru_cache(maxsize=None)
+  def fstring_positions_work():
+    # type: () -> bool
+    """
+    The positions attached to nodes inside f-string FormattedValues have some bugs
+    that were fixed in Python 3.9.7 in https://github.com/python/cpython/pull/27729.
+    This checks for those bugs more concretely without relying on the Python version.
+    Specifically this checks:
+     - Values with a format spec or conversion
+     - Repeated (i.e. identical-looking) expressions
+     - Multiline f-strings implicitly concatenated.
+    """
+    source = """(
+      f"a {b}{b} c {d!r} e {f:g} h {i:{j}} k {l:{m:n}}"
+      f"a {b}{b} c {d!r} e {f:g} h {i:{j}} k {l:{m:n}}"
+      f"{x + y + z} {x} {y} {z} {z} {z!a} {z:z}"
+    )"""
+    tree = ast.parse(source)
+    name_nodes = [node for node in ast.walk(tree) if isinstance(node, ast.Name)]
+    name_positions = [(node.lineno, node.col_offset) for node in name_nodes]
+    positions_are_unique = len(set(name_positions)) == len(name_positions)
+    correct_source_segments = all(
+      ast.get_source_segment(source, node) == node.id
+      for node in name_nodes
+    )
+    return positions_are_unique and correct_source_segments
+
+  def annotate_fstring_nodes(tree):
+    # type: (ast.AST) -> None
+    """
+    Add a special attribute `_broken_positions` to nodes inside f-strings
+    if the lineno/col_offset cannot be trusted.
+    """
+    for joinedstr in walk(tree):
+      if not isinstance(joinedstr, ast.JoinedStr):
+        continue
+      for part in joinedstr.values:
+        # The ast positions of the FormattedValues/Constant nodes span the full f-string, which is weird.
+        setattr(part, '_broken_positions', True)  # use setattr for mypy
+
+        if isinstance(part, ast.FormattedValue):
+          if not fstring_positions_work():
+            for child in walk(part.value):
+              setattr(child, '_broken_positions', True)
+
+          if part.format_spec:  # this is another JoinedStr
+            # Again, the standard positions span the full f-string.
+            setattr(part.format_spec, '_broken_positions', True)
+            # Recursively handle this inner JoinedStr in the same way.
+            # While this is usually automatic for other nodes,
+            # the children of f-strings are explicitly excluded in iter_children_ast.
+            annotate_fstring_nodes(part.format_spec)
+else:
+  def fstring_positions_work():
+    # type: () -> bool
+    return False
+
+  def annotate_fstring_nodes(_tree):
+    # type: (ast.AST) -> None
+    pass
diff --git a/tox.ini b/tox.ini
index d6e05cf..dd35acb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,7 +7,7 @@
 envlist = py{27,35,36,37,38,39,310,py,py3}
 
 [testenv]
-commands = pytest
+commands = pytest {posargs}
 deps =
     .[test]
 passenv =


Test patch:
diff --git a/tests/test_mark_tokens.py b/tests/test_mark_tokens.py
index 5a429e6..cebb226 100644
--- a/tests/test_mark_tokens.py
+++ b/tests/test_mark_tokens.py
@@ -31,7 +31,7 @@ class TestMarkTokens(unittest.TestCase):
 
   def create_mark_checker(self, source, verify=True):
     atok = self.create_asttokens(source)
-    checker = tools.MarkChecker(atok)
+    checker = tools.MarkChecker(atok, self.is_astroid_test)
 
     # The last token should always be an ENDMARKER
     # None of the nodes should contain that token
@@ -442,12 +442,12 @@ bar = ('x y z'   # comment2
     self.assertEqual(m.view_nodes_at(2, 4), {'Name:x', 'Subscript:x[4]'})
 
   if not six.PY2:
-    def test_bad_unmarked_types(self):
-      # Cases where get_text_unmarked is incorrect in 3.8.
+    def test_bad_tokenless_types(self):
+      # Cases where _get_text_positions_tokenless is incorrect in 3.8.
       source = textwrap.dedent("""
         def foo(*, name: str):  # keyword-only argument with type annotation
           pass
-        
+
         f(*(x))  # ast.Starred with parentheses
       """)
       self.create_mark_checker(source)
diff --git a/tests/test_tokenless.py b/tests/test_tokenless.py
new file mode 100644
index 0000000..04bd757
--- /dev/null
+++ b/tests/test_tokenless.py
@@ -0,0 +1,129 @@
+import ast
+import sys
+import unittest
+
+import astroid
+
+from asttokens import ASTText, supports_tokenless
+from asttokens.util import fstring_positions_work
+
+source = """
+x = 1
+if x > 0:
+  for i in range(10):
+    print(i)
+else:
+  print('negative')
+
+def foo(bar):
+  pass
+
+print(f"{xx + 22} is negative {1.23:.2f} {'a'!r} {yy =} {aa:{bb}}")
+
+import a
+import b as c, d.e as f
+from foo.bar import baz as spam
+"""
+
+fstring_node_dumps = [
+  ast.dump(ast.parse(s).body[0].value)  # type: ignore
+  for s in ["xx", "yy", "aa", "bb", "xx + 22", "22", "1.23", "'a'"]
+]
+
+
+def is_fstring_internal_node(node):
+  """
+  Returns True if the given node is an internal node in an f-string.
+  Only applies for nodes parsed from the source above.
+  """
+  return ast.dump(node) in fstring_node_dumps
+
+
+def is_fstring_format_spec(node):
+  """
+  Returns True if the given node is a format specifier in an f-string.
+  Only applies for nodes parsed from the source above.
+  """
+  return (
+      isinstance(node, ast.JoinedStr)
+      and len(node.values) == 1
+      and (
+          (
+              isinstance(node.values[0], ast.Str)
+              and node.values[0].value in ['.2f']
+          ) or (
+              isinstance(node.values[0], ast.FormattedValue)
+              and isinstance(node.values[0].value, ast.Name)
+              and node.values[0].value.id == 'bb'
+          )
+      )
+  )
+
+
+@unittest.skipUnless(supports_tokenless(), "Python version does not support not using tokens")
+class TestTokenless(unittest.TestCase):
+  def test_get_text_tokenless(self):
+    atok = ASTText(source)
+
+    for node in ast.walk(atok.tree):
+      if not isinstance(node, (ast.arguments, ast.arg)):
+        self.check_node(atok, node)
+        self.assertTrue(supports_tokenless(node), node)
+
+    # Check that we didn't need to fall back to using tokens
+    self.assertIsNone(atok._asttokens)
+
+    has_tokens = False
+    for node in ast.walk(atok.tree):
+      self.check_node(atok, node)
+
+      if isinstance(node, ast.arguments):
+        has_tokens = True
+
+      self.assertEqual(atok._asttokens is not None, has_tokens)
+
+    # Now we have started using tokens as fallback
+    self.assertIsNotNone(atok._asttokens)
+    self.assertTrue(has_tokens)
+
+  def check_node(self, atok, node):
+    if not hasattr(node, 'lineno'):
+      self.assertEqual(ast.get_source_segment(source, node), None)
+      atok_text = atok.get_text(node)
+      if not isinstance(node, (ast.arg, ast.arguments)):
+        self.assertEqual(atok_text, source if isinstance(node, ast.Module) else '', node)
+      return
+
+    for padded in [True, False]:
+      ast_text = ast.get_source_segment(source, node, padded=padded)
+      atok_text = atok.get_text(node, padded=padded)
+      if ast_text:
+        if (
+          ast_text.startswith("f") and isinstance(node, (ast.Str, ast.FormattedValue))
+          or is_fstring_format_spec(node)
+          or (not fstring_positions_work() and is_fstring_internal_node(node))
+        ):
+          self.assertEqual(atok_text, "", node)
+        else:
+          self.assertEqual(atok_text, ast_text, node)
+          self.assertEqual(
+            atok.get_text_positions(node, padded=False),
+            (
+              (node.lineno, node.col_offset),
+              (node.end_lineno, node.end_col_offset),
+            ),
+          )
+
+  def test_lazy_asttext_astroid_errors(self):
+    builder = astroid.builder.AstroidBuilder()
+    tree = builder.string_build(source)
+    with self.assertRaises(NotImplementedError):
+      ASTText(source, tree)
+
+
+class TestFstringPositionsWork(unittest.TestCase):
+  def test_fstring_positions_work(self):
+    self.assertEqual(
+      fstring_positions_work() and supports_tokenless(),
+      sys.version_info >= (3, 9, 7),
+    )
diff --git a/tests/tools.py b/tests/tools.py
index fa38606..e5f57ff 100644
--- a/tests/tools.py
+++ b/tests/tools.py
@@ -5,9 +5,8 @@ import io
 import os
 import re
 import sys
-from typing import Tuple, Type
 
-from asttokens import util
+from asttokens import util, supports_tokenless, ASTText
 
 
 def get_fixture_path(*path_parts):
@@ -37,9 +36,11 @@ class MarkChecker(object):
   """
   Helper tool to parse and mark an AST tree, with useful methods for verifying it.
   """
-  def __init__(self, atok):
+  def __init__(self, atok, is_astroid_test):
     self.atok = atok
     self.all_nodes = collect_nodes_preorder(self.atok.tree)
+    if not is_astroid_test:
+      self.atext = ASTText(atok.text, atok.tree, atok.filename)
 
   def get_nodes_at(self, line, col):
     """Returns all nodes that start with the token at the given position."""
@@ -71,10 +72,16 @@ class MarkChecker(object):
     number of nodes that were tested this way.
     """
     test_case.longMessage = True
+
+    if supports_tokenless() and not test_case.is_astroid_test:
+      num_supported = sum(supports_tokenless(n) for n in self.all_nodes)
+      num_nodes = len(self.all_nodes)
+      test_case.assertGreater(num_supported / num_nodes, 0.5, (num_supported, num_nodes))
+
     tested_nodes = 0
     for node in self.all_nodes:
       text = self.atok.get_text(node)
-      self.check_get_text_unmarked(node, test_case, text)
+      self.check_get_text_tokenless(node, test_case, text)
 
       if not (
           util.is_stmt(node) or
@@ -114,62 +121,42 @@ class MarkChecker(object):
 
     return tested_nodes
 
-  def check_get_text_unmarked(self, node, test_case, text):
+  def check_get_text_tokenless(self, node, test_case, text):
     """
     Check that `text` (returned from get_text()) usually returns the same text
-    as get_text_unmarked.
+    whether from `ASTTokens` or `ASTText`.
     """
 
-    if (
-        test_case.is_astroid_test
-        or sys.version_info < (3, 8)
-        or 'pypy' in sys.version.lower()
-    ):
-      # These cases are not supported by get_text_unmarked
+    if test_case.is_astroid_test or not supports_tokenless():
       return
 
-    text_unmarked = self.atok.get_text_unmarked(node)
+    text_tokenless = self.atext.get_text(node)
     if isinstance(node, ast.alias):
-      self._check_alias_unmarked(node, test_case, text_unmarked)
-    elif not isinstance(node, self.bad_unmarked_types):
+      self._check_alias_tokenless(node, test_case, text_tokenless)
+    elif isinstance(node, ast.Module):
+      test_case.assertEqual(text_tokenless, self.atext._text)
+    elif supports_tokenless(node):
       has_lineno = hasattr(node, 'lineno')
-      test_case.assertEqual(has_lineno, text_unmarked != '')
+      test_case.assertEqual(has_lineno, text_tokenless != '')
       if has_lineno:
-        test_case.assertEqual(text, text_unmarked, ast.dump(node))
+        test_case.assertEqual(text, text_tokenless, ast.dump(node))
       else:
-        # get_text_unmarked can't work with nodes without lineno.
+        # _get_text_positions_tokenless can't work with nodes without lineno.
         # Double-check that such nodes are unusual.
         test_case.assertFalse(util.is_stmt(node) or util.is_expr(node))
         with test_case.assertRaises(SyntaxError, msg=(text, ast.dump(node))):
           test_case.parse_snippet(text, node)
 
-  # Node types that check_get_text_unmarked should ignore. Only relevant for Python 3.8+.
-  bad_unmarked_types = ()  # type: Tuple[Type[ast.AST], ...]
-  if sys.version_info[:2] >= (3, 8):
-    bad_unmarked_types = (
-      # get_text_unmarked does something sensible for modules, but it differs from get_text.
-      ast.Module,
-      # no lineno
-      ast.arguments, ast.withitem,
-    )
-    if sys.version_info[:2] == (3, 8):
-      bad_unmarked_types += (
-        # get_text_unmarked works incorrectly for these types due to bugs in Python 3.8.
-        ast.arg, ast.Starred,
-        # no lineno in 3.8
-        ast.Slice, ast.ExtSlice, ast.Index, ast.keyword,
-      )
-
-  def _check_alias_unmarked(self, node, test_case, text_unmarked):
+  def _check_alias_tokenless(self, node, test_case, text):
     if sys.version_info < (3, 10):
       # Before 3.10, aliases don't have position information
-      test_case.assertEqual(text_unmarked, '')
-    # For 3.10+, the original get_text often returns the wrong value for aliases.
-    # So to verify get_text_unmarked, we instead check the general form.
+      test_case.assertEqual(text, '')
+    # For 3.10+, ASTTokens.get_text often returns the wrong value for aliases.
+    # So to verify ASTText.get_text, we instead check the general form.
     elif node.asname:
-      test_case.assertEqual(text_unmarked.split(), [node.name, 'as', node.asname])
+      test_case.assertEqual(text.split(), [node.name, 'as', node.asname])
     else:
-      test_case.assertEqual(text_unmarked, node.name)
+      test_case.assertEqual(text, node.name)
 
 
 def repr_tree(node):