Repository: felix-hilden/sphinx-codeautolink

Issue:
Support highlight language fallbacks in blocks
Some documentation, like Python's [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) use [Sphinx's Literal Blocks](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#literal-blocks) (i.e. just `::`) instead of the `.. code-block::` directive.

It would be nice to support these docs too. The extension would need to support the [`highlight`](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-highlight) directive and the [`highlight_language`](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-highlight_language) configuration option (the latter being used in the Python docs).

Discussion:
@cmarqu: Oh, I see that the extension already supports this.

I thought it's probably just that the example I was using (the docs for Python itself) sets `highlight_language` to `python3` which only becomes supported with #163, but this alone is not sufficient... I'll investigate more.@cmarqu: I think the extension is missing fallback handling to [highlight](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-highlight) and [highlight_language](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-highlight_language),
like in https://github.com/sphinx-doc/sphinx/blob/c49d925b4d861d28cc11356846f687c64069887d/sphinx/transforms/post_transforms/code.py#L30

To support [highlight_language](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-highlight_language) which is enough to support most of the Python docs, I crudely hacked this incomplete solution:
```diff
diff --git a/src/sphinx_codeautolink/extension/__init__.py b/src/sphinx_codeautolink/extension/__init__.py
index 1ccf33f..29d1620 100644
--- a/src/sphinx_codeautolink/extension/__init__.py
+++ b/src/sphinx_codeautolink/extension/__init__.py
@@ -138,6 +138,7 @@ def parse_blocks(self, app, doctree) -> None:
             global_preface=self.global_preface,
             custom_blocks=self.custom_blocks,
             concat_default=self.concat_default,
+            highlight_language=self.highlight_language,
         )
         doctree.walkabout(visitor)
         self.cache.transforms[visitor.current_document] = visitor.source_transforms
diff --git a/src/sphinx_codeautolink/extension/block.py b/src/sphinx_codeautolink/extension/block.py
index 3e0fe49..454c805 100644
--- a/src/sphinx_codeautolink/extension/block.py
+++ b/src/sphinx_codeautolink/extension/block.py
@@ -113,6 +113,7 @@ def __init__(
         global_preface: list[str],
         custom_blocks: dict[str, Callable[[str], str]],
         concat_default: bool,
+        highlight_language: str,
         **kwargs,
     ) -> None:
         super().__init__(*args, **kwargs)
@@ -129,6 +130,7 @@ def __init__(
         self.concat_global = concat_default
         self.concat_section = False
         self.concat_sources = []
+        self.highlight_language = highlight_language
         self.skip = None
 
     def unknown_visit(self, node) -> None:
@@ -185,7 +187,7 @@ def visit_doctest_block(self, node):
 
     def visit_literal_block(self, node: nodes.literal_block):
         """Visit a generic literal block."""
-        return self.parse_source(node, node.get("language", None))
+        return self.parse_source(node, node.get("language", self.highlight_language))
 
     def parse_source(
         self, node: nodes.literal_block | nodes.doctest_block, language: str | None
```@felix-hilden: Thanks for the suggestion and info! Thanks to that I was able to figure out the whole chain pretty easily 👌 After the linked PR is merged, we'll support:

- the default lexer (sphinx assumes them to be python, we ignored them previously)
- literal blocks
- the highlight directive
- highlight_language configuration

I'll still have to make sure we don't *fail* on an invalid default block (which is also Sphinx's assumption), but other than that I would appreciate a quick review on it to make sure it matches what you had in mind, just in case!@cmarqu: Awesome! I'll review in a bit.
Related, I am testing the waters over here: https://discuss.python.org/t/add-the-sphinx-codeautolink-extension-to-doc-build-process/80814, I hope that's okay with you.@cmarqu: > Related, I am testing the waters over here: https://discuss.python.org/t/add-the-sphinx-codeautolink-extension-to-doc-build-process/80814, I hope that's okay with you.

(Obviously, the honors of opening such a PR would belong to you first.)@cmarqu: It seems that Sphinx is automatically handling `pycon` syntax for Literal Blocks, even if `highlight_language` is set to `python3` and no other selection is made - see e.g. the code examples in the https://docs.python.org/3/howto/unicode.html#the-string-type section.

EDIT: it's likely this function: https://github.com/sphinx-doc/sphinx/blob/2364f169ade55916164dbeecbee656afb8fd549f/sphinx/transforms/post_transforms/code.py#L115-L132
used in
https://github.com/sphinx-doc/sphinx/blob/2364f169ade55916164dbeecbee656afb8fd549f/sphinx/transforms/post_transforms/code.py#L99-L100

`sphinx-codeautolink` instead is requiring explicit language selection for its parser to be happy.@felix-hilden: Oh, that's awesome! Thank you for proposing it 🙏

Also thanks for the info on the pycon support 👌 I'll see what makes sense for us to do!@felix-hilden: I created another issue to track that, it wasn't obvious to me on the first look but I'll explore more later. I want to get this in independently 👍 

Solution patch:
diff --git a/docs/src/about.rst b/docs/src/about.rst
index fb79c08..26ab561 100644
--- a/docs/src/about.rst
+++ b/docs/src/about.rst
@@ -18,7 +18,7 @@ Caveats
 -------
 - **Only works with HTML documentation**, disabled otherwise. If the extension
   is off, it silently removes directives that would produce output.
-- **Only processes literal blocks, not inline code**. Sphinx has great tools
+- **Only processes blocks, not inline code**. Sphinx has great tools
   for linking definitions inline, and longer code should be in a block anyway.
 - **Doesn't run example code**. Therefore all possible resolvable types are not
   found, and the runtime correctness of code cannot be validated.
diff --git a/docs/src/examples.rst b/docs/src/examples.rst
index ac25e5a..345af29 100644
--- a/docs/src/examples.rst
+++ b/docs/src/examples.rst
@@ -21,15 +21,16 @@ be analysed and linked to known reference documentation entries.
 
 Different import styles are supported, along with all Python syntax.
 Star imports might be particularly handy in code examples.
-`Doctest <https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html
-#doctest-blocks>`_ and console blocks using :code:`.. code:: pycon` work too.
-Including code via :rst:dir:`literalinclude` requires using one of the following parameters:
-:code:`:language: python`, :code:`:language: python3`,
-:code:`:language: py`, :code:`:language: py3`, :code:`:language: pyi`,
-:code:`:language: sage`, :code:`:language: bazel`, :code:`:language: starlark`
-(i.e., the list of
-`Pygments Lexers <https://pygments.org/docs/lexers/#pygments.lexers.python.PythonLexer>`_).
 
+A number of other types of blocks are also supported
+along with standard Sphinx configuration, like:
+
+- Literal blocks, the ``.. highlight::`` directive
+  and ``highlight_language`` configuration
+- `Doctest blocks <https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#doctest-blocks>`_
+- Console blocks using :code:`.. code:: pycon`
+- Including code via :rst:dir:`literalinclude`, but requires using one of the
+  `Python Lexers <https://pygments.org/docs/lexers/#pygments.lexers.python.PythonLexer>`_
 
 .. code:: pycon
 
diff --git a/docs/src/index.rst b/docs/src/index.rst
index 8c51d5b..61a2340 100644
--- a/docs/src/index.rst
+++ b/docs/src/index.rst
@@ -70,7 +70,7 @@ Caveats
 For a more thorough explanation, see :ref:`about`.
 
 - Only works with HTML documentation
-- Only processes literal blocks, not inline code
+- Only processes blocks, not inline code
 - Doesn't run example code
 - Parsing and type hint resolving is incomplete
 
diff --git a/docs/src/reference.rst b/docs/src/reference.rst
index 507a545..86d4a2f 100644
--- a/docs/src/reference.rst
+++ b/docs/src/reference.rst
@@ -57,21 +57,27 @@ Configuration
 
 .. confval:: codeautolink_warn_on_missing_inventory
 
-   Type: ``bool``. Issue warning when an object cannot be found in
+   Type: ``bool``. Warn when an object cannot be found in
    the inventory (autodoc or intersphinx). Defaults to :code:`False`.
 
 .. confval:: codeautolink_warn_on_failed_resolve
 
-   Type: ``bool``. Issue warning when failing to resolve the canonical location
+   Type: ``bool``. Warn when failing to resolve the canonical location
    of an object that a code element references. Defaults to :code:`False`.
 
 .. confval:: codeautolink_warn_on_no_backreference
 
-   Type: ``bool``. Issue warning when no backreference could be found
+   Type: ``bool``. Warn when no backreference could be found
    from reference documentation using the :rst:dir:`autolink-examples` table.
    This highlights objects for which no tutorial, example or how-to exists.
    Defaults to :code:`False`.
 
+.. confval:: codeautolink_warn_on_default_parse_fail
+
+   Type: ``bool``. Warn when a code block using the ``default`` lexer
+   cannot be parsed as Python. By default these cases are ignored by the
+   syntax highlighter, so we match the behavior. Defaults to :code:`False`.
+
 Directives
 ----------
 .. rst:directive:: .. autolink-examples:: object
@@ -96,7 +102,7 @@ Directives
 
 .. rst:directive:: .. autolink-concat:: [mode]
 
-   Toggle literal block concatenation.
+   Toggle block concatenation.
    Concatenated code blocks are treated as a continuous source,
    so that imports and statements in previous blocks affect later blocks.
    Concatenation is begun at the directive, not applied retroactively.
diff --git a/docs/src/release_notes.rst b/docs/src/release_notes.rst
index c4cae34..52fc360 100644
--- a/docs/src/release_notes.rst
+++ b/docs/src/release_notes.rst
@@ -15,6 +15,10 @@ sphinx-codeautolink adheres to
   a corresponding tutorial or how-to (:issue:`161`)
 - Add more Pygments lexer aliases in code blocks (:issue:`160`)
 - Fix undocumented class attribute leading to a crash (:issue:`165`)
+- Support the ``default`` lexer, literal blocks, ``.. highlight::`` directive
+  and ``highlight_language`` configuration (:issue:`166`)
+- Add :confval:`codeautolink_warn_on_default_parse_fail` to warn about
+  failing to link code blocks without a language parameter (:issue:`166`)
 
 0.16.2 (2025-01-16)
 -------------------
diff --git a/src/sphinx_codeautolink/__init__.py b/src/sphinx_codeautolink/__init__.py
index 0f34af7..07092d3 100644
--- a/src/sphinx_codeautolink/__init__.py
+++ b/src/sphinx_codeautolink/__init__.py
@@ -49,6 +49,12 @@ def setup(app: Sphinx):
         rebuild="html",
         types=[bool],
     )
+    app.add_config_value(
+        "codeautolink_warn_on_default_parse_fail",
+        default=False,
+        rebuild="html",
+        types=[bool],
+    )
 
     app.add_directive("autolink-concat", directive.Concat)
     app.add_directive("autolink-examples", directive.Examples)
diff --git a/src/sphinx_codeautolink/extension/__init__.py b/src/sphinx_codeautolink/extension/__init__.py
index 1ccf33f..a003c0d 100644
--- a/src/sphinx_codeautolink/extension/__init__.py
+++ b/src/sphinx_codeautolink/extension/__init__.py
@@ -75,6 +75,8 @@ class SphinxCodeAutoLink:
         self.warn_missing_inventory = None
         self.warn_failed_resolve = None
         self.warn_no_backreference = None
+        self.warn_default_parse_fail = None
+        self.highlight_lang = None
 
         # Populated once
         self.outdated_docs: set[str] = set()
@@ -105,6 +107,10 @@ class SphinxCodeAutoLink:
         self.warn_missing_inventory = app.config.codeautolink_warn_on_missing_inventory
         self.warn_failed_resolve = app.config.codeautolink_warn_on_failed_resolve
         self.warn_no_backreference = app.config.codeautolink_warn_on_no_backreference
+        self.warn_default_parse_fail = (
+            app.config.codeautolink_warn_on_default_parse_fail
+        )
+        self.highlight_lang = app.config.highlight_language
 
         # Append static resources path so references in setup() are valid
         app.config.html_static_path.append(
@@ -138,6 +144,8 @@ class SphinxCodeAutoLink:
             global_preface=self.global_preface,
             custom_blocks=self.custom_blocks,
             concat_default=self.concat_default,
+            default_highlight_lang=self.highlight_lang,
+            warn_default_parse_fail=self.warn_default_parse_fail,
         )
         doctree.walkabout(visitor)
         self.cache.transforms[visitor.current_document] = visitor.source_transforms
diff --git a/src/sphinx_codeautolink/extension/block.py b/src/sphinx_codeautolink/extension/block.py
index 3e0fe49..503ea23 100644
--- a/src/sphinx_codeautolink/extension/block.py
+++ b/src/sphinx_codeautolink/extension/block.py
@@ -19,6 +19,7 @@ from .directive import ConcatMarker, PrefaceMarker, SkipMarker
 
 # list from https://pygments.org/docs/lexers/#pygments.lexers.python.PythonLexer
 BUILTIN_BLOCKS = {
+    "default": None,
     "python": None,
     "python3": None,
     "py": None,
@@ -104,7 +105,7 @@ else:
 
 
 class CodeBlockAnalyser(nodes.SparseNodeVisitor):
-    """Transform literal blocks of Python with links to reference documentation."""
+    """Transform blocks of Python with links to reference documentation."""
 
     def __init__(
         self,
@@ -113,6 +114,8 @@ class CodeBlockAnalyser(nodes.SparseNodeVisitor):
         global_preface: list[str],
         custom_blocks: dict[str, Callable[[str], str]],
         concat_default: bool,
+        default_highlight_lang: str | None,
+        warn_default_parse_fail: bool,
         **kwargs,
     ) -> None:
         super().__init__(*args, **kwargs)
@@ -130,6 +133,8 @@ class CodeBlockAnalyser(nodes.SparseNodeVisitor):
         self.concat_section = False
         self.concat_sources = []
         self.skip = None
+        self.highlight_lang = default_highlight_lang
+        self.warn_default_parse_fail = warn_default_parse_fail
 
     def unknown_visit(self, node) -> None:
         """Handle and delete custom directives, ignore others."""
@@ -162,6 +167,10 @@ class CodeBlockAnalyser(nodes.SparseNodeVisitor):
     def unknown_departure(self, node) -> None:
         """Ignore unknown nodes."""
 
+    def visit_highlightlang(self, node) -> None:
+        """Set expected highlight language."""
+        self.highlight_lang = node["lang"]
+
     def visit_title(self, node) -> None:
         """Track section names and break concatenation and skipping."""
         self.title_stack.append(node.astext())
@@ -185,7 +194,7 @@ class CodeBlockAnalyser(nodes.SparseNodeVisitor):
 
     def visit_literal_block(self, node: nodes.literal_block):
         """Visit a generic literal block."""
-        return self.parse_source(node, node.get("language", None))
+        return self.parse_source(node, node.get("language", self.highlight_lang))
 
     def parse_source(
         self, node: nodes.literal_block | nodes.doctest_block, language: str | None
@@ -231,6 +240,9 @@ class CodeBlockAnalyser(nodes.SparseNodeVisitor):
         try:
             names = parse_names(modified_source, node)
         except SyntaxError as e:
+            if language == "default" and not self.warn_default_parse_fail:
+                return
+
             show_source = self._format_source_for_error(
                 self.global_preface, self.concat_sources, prefaces, source
             )
diff --git a/src/sphinx_codeautolink/extension/directive.py b/src/sphinx_codeautolink/extension/directive.py
index de39f77..57774ec 100644
--- a/src/sphinx_codeautolink/extension/directive.py
+++ b/src/sphinx_codeautolink/extension/directive.py
@@ -62,7 +62,7 @@ class ConcatMarker(nodes.Element):
 
 
 class Concat(Directive):
-    """Toggle and cut literal block concatenation in a document."""
+    """Toggle and cut block concatenation in a document."""
 
     has_content = False
     required_arguments = 0


Test patch:
diff --git a/tests/extension/ref/pygments_lexer_code-block_bazel.txt b/tests/extension/ref/lexer_bazel.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_bazel.txt
rename to tests/extension/ref/lexer_bazel.txt
diff --git a/tests/extension/ref/lexer_default.txt b/tests/extension/ref/lexer_default.txt
new file mode 100644
index 0000000..40eecb8
--- /dev/null
+++ b/tests/extension/ref/lexer_default.txt
@@ -0,0 +1,14 @@
+test_project
+test_project.bar
+# split
+# split
+Test project
+============
+
+.. code::
+
+   import test_project
+
+   test_project.bar()
+
+.. automodule:: test_project
diff --git a/tests/extension/ref/lexer_default_fail.txt b/tests/extension/ref/lexer_default_fail.txt
new file mode 100644
index 0000000..202c75c
--- /dev/null
+++ b/tests/extension/ref/lexer_default_fail.txt
@@ -0,0 +1,10 @@
+# split
+# split
+Test project
+============
+
+.. code::
+
+   invalid python in a default block should not crash
+
+.. automodule:: test_project
diff --git a/tests/extension/ref/ref_doctest.txt b/tests/extension/ref/lexer_doctest.txt
similarity index 100%
rename from tests/extension/ref/ref_doctest.txt
rename to tests/extension/ref/lexer_doctest.txt
diff --git a/tests/extension/ref/ref_extdoctest_doctest.txt b/tests/extension/ref/lexer_extdoctest_doctest.txt
similarity index 100%
rename from tests/extension/ref/ref_extdoctest_doctest.txt
rename to tests/extension/ref/lexer_extdoctest_doctest.txt
diff --git a/tests/extension/ref/ref_extdoctest_testcode.txt b/tests/extension/ref/lexer_extdoctest_testcode.txt
similarity index 100%
rename from tests/extension/ref/ref_extdoctest_testcode.txt
rename to tests/extension/ref/lexer_extdoctest_testcode.txt
diff --git a/tests/extension/ref/lexer_highlight_language.txt b/tests/extension/ref/lexer_highlight_language.txt
new file mode 100644
index 0000000..aa8ab0b
--- /dev/null
+++ b/tests/extension/ref/lexer_highlight_language.txt
@@ -0,0 +1,12 @@
+# split
+highlight_language = "none"
+# split
+Test project
+============
+
+Here's an example::
+
+   import test_project
+   test_project.bar()
+
+.. automodule:: test_project
diff --git a/tests/extension/ref/lexer_highlight_none.txt b/tests/extension/ref/lexer_highlight_none.txt
new file mode 100644
index 0000000..60aa1a3
--- /dev/null
+++ b/tests/extension/ref/lexer_highlight_none.txt
@@ -0,0 +1,13 @@
+# split
+# split
+Test project
+============
+
+.. highlight:: none
+
+Here's an example::
+
+   import test_project
+   test_project.bar()
+
+.. automodule:: test_project
diff --git a/tests/extension/ref/ref_ipython.txt b/tests/extension/ref/lexer_ipython.txt
similarity index 100%
rename from tests/extension/ref/ref_ipython.txt
rename to tests/extension/ref/lexer_ipython.txt
diff --git a/tests/extension/ref/ref_ipython3.txt b/tests/extension/ref/lexer_ipython3.txt
similarity index 100%
rename from tests/extension/ref/ref_ipython3.txt
rename to tests/extension/ref/lexer_ipython3.txt
diff --git a/tests/extension/ref/ref_ipython_directive.txt b/tests/extension/ref/lexer_ipython_directive.txt
similarity index 100%
rename from tests/extension/ref/ref_ipython_directive.txt
rename to tests/extension/ref/lexer_ipython_directive.txt
diff --git a/tests/extension/ref/ref_ipython_directive_comment.txt b/tests/extension/ref/lexer_ipython_directive_comment.txt
similarity index 100%
rename from tests/extension/ref/ref_ipython_directive_comment.txt
rename to tests/extension/ref/lexer_ipython_directive_comment.txt
diff --git a/tests/extension/ref/ref_ipython_syntax.txt b/tests/extension/ref/lexer_ipython_syntax.txt
similarity index 100%
rename from tests/extension/ref/ref_ipython_syntax.txt
rename to tests/extension/ref/lexer_ipython_syntax.txt
diff --git a/tests/extension/ref/lexer_literal.txt b/tests/extension/ref/lexer_literal.txt
new file mode 100644
index 0000000..47103ca
--- /dev/null
+++ b/tests/extension/ref/lexer_literal.txt
@@ -0,0 +1,13 @@
+test_project
+test_project.bar
+# split
+# split
+Test project
+============
+
+Here's an example::
+
+   import test_project
+   test_project.bar()
+
+.. automodule:: test_project
diff --git a/tests/extension/ref/non_python_block.txt b/tests/extension/ref/lexer_non_python_block.txt
similarity index 90%
rename from tests/extension/ref/non_python_block.txt
rename to tests/extension/ref/lexer_non_python_block.txt
index efdb828..eacddf1 100644
--- a/tests/extension/ref/non_python_block.txt
+++ b/tests/extension/ref/lexer_non_python_block.txt
@@ -3,7 +3,7 @@
 Test project
 ============
 
-.. code::
+.. code:: c
 
    import test_project
 
diff --git a/tests/extension/ref/ref_py.txt b/tests/extension/ref/lexer_py.txt
similarity index 100%
rename from tests/extension/ref/ref_py.txt
rename to tests/extension/ref/lexer_py.txt
diff --git a/tests/extension/ref/pygments_lexer_code-block_py3.txt b/tests/extension/ref/lexer_py3.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_py3.txt
rename to tests/extension/ref/lexer_py3.txt
diff --git a/tests/extension/ref/ref_pycon.txt b/tests/extension/ref/lexer_pycon.txt
similarity index 100%
rename from tests/extension/ref/ref_pycon.txt
rename to tests/extension/ref/lexer_pycon.txt
diff --git a/tests/extension/ref/pygments_lexer_code-block_pyi.txt b/tests/extension/ref/lexer_pyi.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_pyi.txt
rename to tests/extension/ref/lexer_pyi.txt
diff --git a/tests/extension/ref/pygments_lexer_code-block_python3.txt b/tests/extension/ref/lexer_python3.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_python3.txt
rename to tests/extension/ref/lexer_python3.txt
diff --git a/tests/extension/ref/pygments_lexer_code-block_sage.txt b/tests/extension/ref/lexer_sage.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_sage.txt
rename to tests/extension/ref/lexer_sage.txt
diff --git a/tests/extension/ref/pygments_lexer_code-block_starlark.txt b/tests/extension/ref/lexer_starlark.txt
similarity index 100%
rename from tests/extension/ref/pygments_lexer_code-block_starlark.txt
rename to tests/extension/ref/lexer_starlark.txt
diff --git a/tests/extension/ref/preface_consumed_non_python.txt b/tests/extension/ref/preface_consumed_non_python.txt
index c907693..6249121 100644
--- a/tests/extension/ref/preface_consumed_non_python.txt
+++ b/tests/extension/ref/preface_consumed_non_python.txt
@@ -4,7 +4,7 @@ Test project
 ============
 
 .. autolink-preface:: import test_project
-.. code::
+.. code:: c
 
    test_project.bar()