Repository: sphinx-gallery/sphinx-gallery

Issue:
custom within-subsection sorting doesn't work as documented
The [example in the docs](https://sphinx-gallery.github.io/stable/configuration.html#custom-sort-keys) of using `FunctionSortKey`:

```py
"minigallery_sort_order": FunctionSortKey(lambda filename: filename[:10])
```

is guaranteed not to work, because non-string values are tested with `inspect.isclass()` wihtin the `_get_class` helper --- but `isclass()` returns `False` for class *instances* (and the recommended usage always generates a class instance).

If instead I follow [the other recommendation in the docs](https://sphinx-gallery.github.io/dev/configuration.html#importing-callables) to use a FQN of an importable function, I still get an error:

```
In [1]: from sphinx_gallery.gen_rst import _get_class
In [2]: sphinx_gallery_conf = {
   ...:     "doc_module": "mne_bids",
   ...:     "reference_url": {
   ...:         "mne_bids": None,
   ...:     },
   ...:     "backreferences_dir": "generated",
   ...:     "examples_dirs": "../examples",
   ...:     "within_subsection_order": "mne_bids.utils.example_sorter",  # <---- HERE
   ...:     "gallery_dirs": "auto_examples",
   ...:     "filename_pattern": "^((?!sgskip).)*$",
   ...: }
   ...: 
In [3]: _get_class(sphinx_gallery_conf, "within_subsection_order")
---------------------------------------------------------------------------
ConfigError                               Traceback (most recent call last)
Cell In[3], line 1
----> 1 _get_class(sphinx_gallery_conf, "within_subsection_order")

File /opt/sphinx-gallery/sphinx_gallery/gen_rst.py:1645, in _get_class(gallery_conf, key)
   1640         raise ConfigError(
   1641             f"{what} must be a fully qualified name string, could not import "
   1642             f"{attr} from {mod}"
   1643         )
   1644 if not inspect.isclass(val):
-> 1645     raise ConfigError(
   1646         f"{what} must be 1) a fully qualified name (string) that resolves "
   1647         f"to a class or 2) or a class itself, got {val.__class__.__name__} "
   1648         f"({repr(val)})"
   1649     )
   1650 return val

ConfigError: sphinx_gallery_conf['within_subsection_order'] must be 1) a fully qualified name (string) that resolves to a class or 2) or a class itself, got function (<function example_sorter at 0x7d99b40f2160>)
```

I suspect that this error was introduced in #1289 (but haven't dug deeply, maybe it's older)

I'll submit a PR shortly

Discussion:
larsoner: @lucyleeow I did not follow all discussion so I'll let you review, feel free to merge if you're happy. I can look deeper if it would help to have another set of eyes or you're busy though!
drammock: one other important note: now that `FunctionSortKey` returns a type (rather than an instance) it will fail sphinx's `is_serializable` checks. So might be worth highlighting that in the docs / *strongly* encouraging the use of FQN strings as the preferred way to do custom sorting.
lucyleeow: Half way through review I realised that I already figured out that `subsection_order` and `minigallery_sort_order` worked differently to `within_subsection_order`, when reviewing #1226. Thus I specifically removed `within_subsection_order` from the docs (see: https://sphinx-gallery.github.io/stable/configuration.html#custom-sort-keys and my last note: https://github.com/sphinx-gallery/sphinx-gallery/pull/1226#issuecomment-1918365134

(also which part of the docs were you reading? They can definitely be improved, I was not sure where to put the section on custom sort keys as you can use them for several 'unrelated' configs :weary: )

I do like that this now supports `within_subsection_order` BUT I think it would no longer work for `subsection_order` and `minigallery_sort_order` as these both use `_get_callables`, which can't be a class.

https://github.com/sphinx-gallery/sphinx-gallery/blob/9947c26a7457f7ef35b41cee7d6fa89e1f36c67d/sphinx_gallery/gen_rst.py#L1698-L1701

(obviously we need to improve our testing)

Ideally, I think I would love to get rid off `_get_class`, but this would involve:

* internally make the convenience classes (e.g., `ExampleTitleSortKey`) functions instead of classes, but keep the names as camelcase (since we ask the user to pass as string anyway). I think we only had classes for the stable `__repr__` but as we are passing a string now, this is not necessary. I think this would also aid the `within_subsection_order` and `FunctionSortKey` problem, as we would no longer be doing:

https://github.com/sphinx-gallery/sphinx-gallery/blob/9947c26a7457f7ef35b41cee7d6fa89e1f36c67d/sphinx_gallery/gen_rst.py#L569

* maybe getting rid of `FunctionSortKey` altogether (a quick [search on github](https://github.com/search?q=FunctionSortKey&type=code&p=1) seems to show that it is not really used), it was introduced not long before the serialization stuff. 

Sorry @larsoner, have to bring you in here, you'll be able to come up with a better solution than my tired brain.
Also apologies if I have made a mistake somewhere.


drammock: > Thus I specifically removed `within_subsection_order` from the docs (see: https://sphinx-gallery.github.io/stable/configuration.html#custom-sort-keys and my last note: [#1226 (comment)](https://github.com/sphinx-gallery/sphinx-gallery/pull/1226#issuecomment-1918365134)

Ah yes, I somehow missed that `within_subsection_order` wasn't mentioned in the "Custom Sort Keys" section of the config docs!  So my earlier statement of "the docs say you can X but it doesn't work" is not strictly correct.

> maybe getting rid of FunctionSortKey altogether (a quick [search on github](https://github.com/search?q=FunctionSortKey&type=code&p=1) seems to show that it is not really used)

-0.5 from me; this PR came about because I want to use it (on subsection orders) and couldn't get it to work. I'd be content though if passing an FQN string of a sorting function worked on subsection orders.  Note that at present (in this PR) `FunctionSortKey` is getting used internally to convert the imported FQN into a callable class (to parallel the other SortKey classes), so even if removed from the public API, it might still be in the codebase... unless:

> I would love to get rid off `_get_class`

...then maybe `FunctionSortKey` could go away entirely. I can look into that.

> I do like that this now supports `within_subsection_order` BUT I think it would no longer work for `subsection_order` and `minigallery_sort_order` as these both use `_get_callables`, which can't be a class.

I'll add a test of whether FunctionSortKey still works on `subsection_order` and `minigallery_sort_order`, and report back.
lucyleeow: I completely agree that ideally I would like all 3 configs to be able to take a custom sort key and for them to work similarly.

I think a function and string FQN should work on all 3 but we've just got to get the class thing working.

I thought of getting rid of `_get_class` because maybe we could get `within_subsection_order` to work with `_get_callables`, by changing the convenience classes (e.g., `ExampleTitleSortKey`) to be functions as they are now given as strong. It would also be nice to reduce some complexity in the code. Hopefully I haven't missed something important though.

We can keep `FunctionSortKey`, I think we can make it work. But I think we'll  want to recommend using functions.

Thanks for working on this!
lucyleeow: Thinking about it more, maybe the convenience classes could be a 'callable' (i.e. class instance) instead of a class. This would allow us to use `_get_callables` and keep these CamelCased classes as classes, and not have to change them to functions? Instead of init-ing with the `src_dir` we can just pass it the full path?

The only other thing we would want to make sure works is any existing user's own classes used for `within_section_order` (as we know the `FunctionSortKey` does not work). They could have done this by subclassing `_SortKey` (a [search on github](https://github.com/search?q=%22_SortKey%22&type=code) for this class was not helpful as I don't think you can do a literal string search over all of github) or making their own class that works the same way. I don't know how common this is but I don't think we could get rid of `_get_class` without breaking existing users own `within_subsection_order` classes.

I guess the other option is to loosen, so a class instance or a class are both accepted? Not sure what the best solution is here. I would like to go for getting rid of `_get_class` but I don't want to break existing users' code/force them to change their code.

Apologies for patchiness, I am on holiday but I will try to keep on top of this.
larsoner: > and IIUC we don't expect users to pass in an actual class right? just instances?

It looks like at least mpl does:

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/conf.py#L305

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/conf.py#L187

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/sphinxext/gallery_order.py#L125

Does that change things?
drammock: > > and IIUC we don't expect users to pass in an actual class right? just instances?
> 
> It looks like at least mpl does [...] Does that change things?

ah, yes it would. thanks for the tip, I'll rethink this.
larsoner: > then MPL's custom sorter class should work too, and the test suite still passes locally.

Do you see an easy enough way to modify `tinybuild` to have mpl's use case? That way we can test the "class" case there but hopefully also test the "class instance" case in more targeted tests like I think you currently have in `test_gen_gallery.py`

Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 4632910..0b34163 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.9
+    rev: v0.7.2
     hooks:
       - id: ruff-format
         exclude: plot_syntaxerror
@@ -35,11 +35,11 @@ repos:
       - id: sphinx-lint
 
   - repo: https://github.com/tox-dev/pyproject-fmt
-    rev: 2.3.1
+    rev: v2.5.0
     hooks:
       - id: pyproject-fmt
         additional_dependencies: [tox]
   - repo: https://github.com/abravalheri/validate-pyproject
-    rev: v0.20.2
+    rev: v0.22
     hooks:
       - id: validate-pyproject
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 01989e5..7ee3769 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -47,6 +47,13 @@ file, inside a ``sphinx_gallery_conf`` dictionary.
 - ``reset_modules`` and ``reset_modules_order`` (:ref:`reset_modules`)
 - ``parallel`` (:ref:`parallel`)
 
+.. admonition:: Diverging from Jupyter
+    :class: danger
+
+    Sphinx-gallery attempts to render examples to HTML in a manner largely consistent with what a user will experience when they download the corresponding ``.ipynb`` notebook file and run it locally.
+    Some options, such as ``'capture_repr': (),``, will make these behaviors less consistent.
+    Consider using these options sparingly as it could lead to confusion or sub-optimal experiences for users!
+
 **Cross-referencing**
 
 - ``reference_url``, ``prefer_full_module`` (:ref:`link_to_documentation`)
@@ -107,6 +114,7 @@ Some options can also be set or overridden on a file-by-file basis:
 
 Some options can be set on a per-code-block basis in a file:
 
+- ``# sphinx_gallery_capture_repr_block`` (:ref:`capture_repr`)
 - ``# sphinx_gallery_defer_figures`` (:ref:`defer_figures`)
 - ``# sphinx_gallery_multi_image_block`` (:ref:`multi_image`)
 
@@ -2411,11 +2419,15 @@ are:
   present in some objects, for example, pandas dataframes.
 
 Output capture can be controlled globally by the ``capture_repr`` configuration
-setting or file-by-file by adding a comment to the example file, which overrides
+setting, file-by-file by adding a comment to the example file, which overrides
 any global setting::
 
     # sphinx_gallery_capture_repr = ()
 
+, or block-by-block by adding a comment to the code block, which overrides any
+global or file setting::
+
+    # sphinx_gallery_capture_repr_block = ()
 
 The default setting is::
 
@@ -2445,8 +2457,7 @@ From another perspective, take for example the following code block::
 ``'Hello world'`` would be captured for every ``capture_repr`` setting as this
 is directed to standard output. Further,
 
-* if ``capture_repr`` is an empty tuple, nothing else would be
-  captured.
+* if ``capture_repr`` is an empty tuple, nothing else would be captured.
 * if ``capture_repr`` is ``('__repr__')``, ``2`` would also be captured.
 * if ``capture_repr`` is ``('_repr_html_', '__repr__')`` (the default)
   Sphinx-Gallery would first attempt to capture ``_repr_html_``. Since this
@@ -2477,8 +2488,27 @@ method which would thus be captured. You can prevent this by:
     plt.plot([1, 2, 3, 4], [1, 4, 9, 16])
     plt.show()
 
-The unwanted string output will not occur if ``'capture_repr'`` is an empty
-tuple or does not contain ``__repr__`` or ``__str__``.
+* add a ``# sphinx_gallery_capture_repr_block = ()`` comment to the code block.
+  The unwanted string output will not occur if ``'capture_repr'`` is an empty
+  tuple or, at least, does not contain ``__repr__`` or ``__str__``.
+  For example::
+
+    # sphinx_gallery_capture_repr_block = ()
+    import matplotlib.pyplot as plt
+
+    plt.plot([1, 2, 3, 4], [1, 4, 9, 16])
+
+.. admonition:: Note for Plotly users
+   :class: note
+
+   The suggestions above also apply to Plotly users. Plotly figures have
+   several `update methods
+   <https://plotly.com/python/creating-and-updating-figures/>`_ that
+   implicitly return the updated figure object. You can set a block-level
+   ``# sphinx_gallery_capture_repr_block = ()`` comment to prevent these
+   from being captured, or assign the return values to a variable (*e.g.,*
+   ``fig = fig.update_layout(...)``).
+
 
 .. _regular expressions: https://docs.python.org/3/library/re.html
 
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index 9568447..1d14a8b 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -33,7 +33,6 @@ from .gen_rst import (
     SPHX_GLR_SIG,
     _get_call_memory_and_base,
     _get_callables,
-    _get_class,
     _get_gallery_header,
     generate_dir_rst,
 )
@@ -497,8 +496,10 @@ def _fill_gallery_conf_defaults(sphinx_gallery_conf, app=None, check_keys=True):
             f'got {gallery_conf["show_api_usage"]}'
         )
 
-    # classes (not pickleable so need to resolve using fully qualified name)
-    _get_class(gallery_conf, "within_subsection_order")  # make sure it works
+    # check `within_subsection_order`
+    _get_callables(
+        gallery_conf, "within_subsection_order", src_dir=""
+    )  # make sure it works
 
     _update_gallery_conf_exclude_implicit_doc(gallery_conf)
 
@@ -527,9 +528,10 @@ def get_subsections(srcdir, examples_dir, gallery_conf, check_for_header=True):
     if gallery_conf["subsection_order"] is None:
         sortkey = None
     else:
+        subsec_order = gallery_conf.get("subsection_order")
+        if isinstance(subsec_order, list):
+            gallery_conf["subsection_order"] = ExplicitOrder(subsec_order)
         (sortkey,) = _get_callables(gallery_conf, "subsection_order")
-        if isinstance(sortkey, list):
-            sortkey = ExplicitOrder(sortkey)
     subfolders = [subfolder for subfolder in os.listdir(examples_dir)]
     if check_for_header:
         subfolders = [
@@ -664,7 +666,7 @@ def _build_recommender(gallery_conf, gallery_dir_abs_path, subsecs):
                 # NOTE we don't take account of `ignore_pattern` and ignore
                 # ext in `example_extensions`
                 [fname for fname in Path(src_dir).iterdir() if fname.suffix == ".py"],
-                key=_get_class(gallery_conf, "within_subsection_order")(src_dir),
+                key=_get_callables(gallery_conf, "within_subsection_order", src_dir)[0],
             )
             gallery_py_files.append(
                 [os.path.join(src_dir, fname) for fname in py_files]
@@ -1138,7 +1140,7 @@ def _make_graph(fname, entries, gallery_conf):
                 dg.node(node_to, **node_kwargs)
                 dg.edge(node_from, node_to, color=API_COLORS["edge"])
         # add modules with all API entries
-        for module in gallery_conf["_sg_api_entries"]["module"]:
+        for module in gallery_conf["_sg_api_entries"].get("module", []):
             struct = module.split(".")
             for i in range(len(struct) - 1):
                 if struct[i + 1] not in lut:
diff --git a/sphinx_gallery/gen_rst.py b/sphinx_gallery/gen_rst.py
index 2faa858..504e39d 100644
--- a/sphinx_gallery/gen_rst.py
+++ b/sphinx_gallery/gen_rst.py
@@ -566,7 +566,7 @@ def generate_dir_rst(
     listdir = _collect_gallery_files([src_dir], gallery_conf)
     # sort them
     sorted_listdir = sorted(
-        listdir, key=_get_class(gallery_conf, "within_subsection_order")(src_dir)
+        listdir, key=_get_callables(gallery_conf, "within_subsection_order", src_dir)[0]
     )
 
     # Add div containing all thumbnails;
@@ -888,7 +888,7 @@ def _get_last_repr(capture_repr, ___):
 
 
 def _get_code_output(
-    is_last_expr, example_globals, gallery_conf, logging_tee, images_rst, file_conf
+    is_last_expr, example_globals, gallery_conf, logging_tee, images_rst, capture_repr
 ):
     """Obtain standard output and html output in reST.
 
@@ -921,7 +921,6 @@ def _get_code_output(
         ignore_repr = False
         if gallery_conf["ignore_repr_types"]:
             ignore_repr = re.search(gallery_conf["ignore_repr_types"], str(type(___)))
-        capture_repr = file_conf.get("capture_repr", gallery_conf["capture_repr"])
         if capture_repr != () and not ignore_repr:
             last_repr, repr_meth = _get_last_repr(capture_repr, ___)
 
@@ -1007,11 +1006,20 @@ def execute_code_block(
     )
     need_save_figures = defer_figs_match is None
 
+    block_conf = py_source_parser.extract_file_config(block.content)
+
     # Add `sphinx_gallery_multi_image_block` setting to block variables
     # (extract config rather than just regex search since the option's value is needed)
-    script_vars["multi_image"] = py_source_parser.extract_file_config(
-        block.content
-    ).get("multi_image_block")
+    script_vars["multi_image"] = block_conf.get("multi_image_block")
+
+    # Determine the block's `capture_repr` value by prioritising the block-level
+    # `sphinx_gallery_capture_repr_block` setting, then the file-level
+    # `sphinx_gallery_capture_repr` setting, and finally the
+    # global `capture_repr` gallery setting.
+    capture_repr = block_conf.get(
+        "capture_repr_block",
+        file_conf.get("capture_repr", gallery_conf["capture_repr"]),
+    )
 
     # Add file_conf to script_vars to be read by image scrapers
     script_vars["file_conf"] = file_conf
@@ -1059,7 +1067,7 @@ def execute_code_block(
             gallery_conf,
             logging_tee,
             images_rst,
-            file_conf,
+            capture_repr,
         )
     finally:
         _reset_cwd_syspath(cwd, sys_path)
@@ -1615,44 +1623,20 @@ def save_rst_example(
     _replace_md5(write_file_new, mode="t")
 
 
-def _get_class(gallery_conf, key):
-    """Get a class for the given conf key."""
-    what = f"sphinx_gallery_conf[{repr(key)}]"
-    val = gallery_conf[key]
-    if key == "within_subsection_order":
-        alias = (  # convenience aliases
-            "ExampleTitleSortKey",
-            "FileNameSortKey",
-            "FileSizeSortKey",
-            "NumberOfCodeLinesSortKey",
-        )
-        if val in alias:
-            val = f"sphinx_gallery.sorting.{val}"
-    if isinstance(val, str):
-        if "." not in val:
-            raise ConfigError(
-                f"{what} must be a fully qualified name string, got {val}"
-            )
-        mod, attr = val.rsplit(".", 1)
-        try:
-            val = getattr(importlib.import_module(mod), attr)
-        except Exception:
-            raise ConfigError(
-                f"{what} must be a fully qualified name string, could not import "
-                f"{attr} from {mod}"
-            )
-    if not inspect.isclass(val):
-        raise ConfigError(
-            f"{what} must be 1) a fully qualified name (string) that resolves "
-            f"to a class or 2) or a class itself, got {val.__class__.__name__} "
-            f"({repr(val)})"
-        )
-    return val
-
-
-def _get_callables(gallery_conf, key):
+def _get_callables(gallery_conf, key, src_dir=None):
     """Get callables for the given conf key, returning tuple of callable(s)."""
-    singletons = ("reset_argv", "minigallery_sort_order", "subsection_order")
+    builtin_aliases = (
+        "ExampleTitleSortKey",
+        "FileNameSortKey",
+        "FileSizeSortKey",
+        "NumberOfCodeLinesSortKey",
+    )
+    singletons = (
+        "reset_argv",
+        "minigallery_sort_order",
+        "subsection_order",
+        "within_subsection_order",
+    )
     # the following should be the case (internal use only):
     assert key in ("image_scrapers", "reset_modules", "jupyterlite") + singletons, key
     which = gallery_conf[key]
@@ -1671,6 +1655,10 @@ def _get_callables(gallery_conf, key):
         else:
             readable = f"{key}[{wi}]={repr(what)}"
         if isinstance(what, str):
+            # use fully qualified name to resolve builtin callable classes
+            # (otherwise not serializable)
+            if what in builtin_aliases:
+                what = f"sphinx_gallery.sorting.{what}"
             if "." in what:
                 mod, attr = what.rsplit(".", 1)
                 try:
@@ -1695,6 +1683,10 @@ def _get_callables(gallery_conf, key):
                     raise ConfigError(f"Unknown string option for {readable}: {what}")
                 what = _reset_dict[what]
             which[wi] = what
+        # make sure classes get instantiated (so they become callable)
+        if key == "within_subsection_order" and inspect.isclass(what):
+            what = what(src_dir)
+            which[wi] = what
         if inspect.isclass(what):
             raise ConfigError(
                 f"Got class rather than callable instance for {readable}: {what}"
diff --git a/sphinx_gallery/scrapers.py b/sphinx_gallery/scrapers.py
index bcb12ff..efff376 100644
--- a/sphinx_gallery/scrapers.py
+++ b/sphinx_gallery/scrapers.py
@@ -273,7 +273,9 @@ def _anim_rst(anim, image_path, gallery_conf):
     dpi = rcParams["savefig.dpi"]
     if dpi == "figure":
         dpi = fig.dpi
-    video_uri = video.relative_to(gallery_conf["src_dir"]).as_posix()
+    # relative_to doesn't work on windows
+    # video_uri = video.relative_to(gallery_conf["src_dir"]).as_posix()
+    video_uri = PurePosixPath(os.path.relpath(video, gallery_conf["src_dir"]))
     html = _ANIMATION_VIDEO_RST.format(
         video=f"/{video_uri}",
         width=int(fig_size[0] * dpi),
diff --git a/sphinx_gallery/utils.py b/sphinx_gallery/utils.py
index 0c7e37a..b68db1b 100644
--- a/sphinx_gallery/utils.py
+++ b/sphinx_gallery/utils.py
@@ -316,3 +316,26 @@ def _format_toctree(items, includehidden=False):
     st += "\n"
 
     return st
+
+
+_CUSTOM_EXAMPLE_ORDER = [
+    "plot_1.py",
+    "plot_3.py",
+    "plot_2.py",
+    "plot_5.py",
+    "plot_6.py",
+    "plot_4.py",
+    "plot_8.py",
+    "plot_7.py",
+    "plot_9.py",
+]
+
+
+def _custom_example_sorter(filename):
+    """Importable custom sorter func, used in our test suite."""
+    return _CUSTOM_EXAMPLE_ORDER.index(filename)
+
+
+def _custom_subsection_sorter(foldername):
+    """Importable custom sorter func for subsection folders, used in our test suite."""
+    return foldername[::-1]


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d6f888e..3dfd0fd 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -15,7 +15,6 @@ jobs:
   pytest:
     name: '${{ matrix.os }} / ${{ matrix.python }} / ${{ matrix.distrib }} / ${{ matrix.sphinx_version }}'
     timeout-minutes: 30
-    continue-on-error: true
     runs-on: ${{ matrix.os }}
     defaults:
       run:
@@ -25,6 +24,7 @@ jobs:
       SPHINX_VERSION: '${{ matrix.sphinx_version }}'
       DISTRIB: '${{ matrix.distrib }}'
     strategy:
+      fail-fast: false
       matrix:
         include:
           - os: ubuntu-latest  # newest possible
diff --git a/sphinx_gallery/tests/test_gen_gallery.py b/sphinx_gallery/tests/test_gen_gallery.py
index a73f0bf..e580116 100644
--- a/sphinx_gallery/tests/test_gen_gallery.py
+++ b/sphinx_gallery/tests/test_gen_gallery.py
@@ -50,10 +50,10 @@ def test_bad_config():
     ):
         _fill_gallery_conf_defaults(sphinx_gallery_conf)
     sphinx_gallery_conf = dict(within_subsection_order="sphinx_gallery.a.b.Key")
-    with pytest.raises(ConfigError, match="must be a fully qualified"):
+    with pytest.raises(ConfigError, match=r"Unknown string option.*when importing"):
         _fill_gallery_conf_defaults(sphinx_gallery_conf)
     sphinx_gallery_conf = dict(within_subsection_order=1.0)
-    with pytest.raises(ConfigError, match="a fully qualified.*got float"):
+    with pytest.raises(ConfigError, match="must be callable"):
         _fill_gallery_conf_defaults(sphinx_gallery_conf)
     sphinx_gallery_conf = dict(minigallery_sort_order=int)
     with pytest.raises(ConfigError, match="Got class rather than callable instance"):
@@ -268,75 +268,187 @@ def test_spaces_in_files_warn(sphinx_app_wrapper):
     assert msg.format(m) in build_warn
 
 
-def _check_order(sphinx_app, key):
+def _check_order(sphinx_app, key, expected_order=None):
     """Iterates through sphx-glr-thumbcontainer divs and reads key from the tooltip.
 
-    Used to test that these keys (in index.rst) appear in a specific order.
+    Used to test that these keys (in index.rst) appear in a specific order. The `.py`
+    files include a source-of-truth line indicating the position in the sort order that
+    the thumbnail for that file should occur, for the various built-in sorters.
+
+    The regex below extracts info from that source-of-truth line to check that the
+    thumbnail ordering was correct --- unless `expected_order` is provided, in which
+    case an alternative regex extracts the digit-part of the filename from the crossref,
+    and compares the order of crossrefs to `expected_order`.
     """
     index_fname = Path(sphinx_app.outdir, "..", "ex", "index.rst")
     order = list()
-    regex = f".*:{key}=(.):.*"
+    if key is None:
+        regex = r".*:ref:`sphx_glr_ex_(?:(?:first|second)-subsection_)?plot_(\d)\.py`"
+        locator = ":ref:"
+    else:
+        regex = rf".*:{key}=(\d):.*"
+        locator = "sphx-glr-thumbcontainer"
     with open(index_fname, "r", encoding="utf-8") as fid:
         for line in fid:
-            if "sphx-glr-thumbcontainer" in line:
-                order.append(int(re.match(regex, line).group(1)))
-    assert len(order) == 3
-    assert order == [1, 2, 3]
-
-
-@pytest.mark.add_conf(
-    content="""
-sphinx_gallery_conf = {
-    'examples_dirs': 'src',
-    'gallery_dirs': 'ex',
-}"""
+            if locator in line:
+                order.append(re.match(regex, line).group(1))
+    expected_order = expected_order or list("123456789")
+    assert order == expected_order
+
+
+# ======================================================================================
+# SUBSECTION AND WITHIN-SUBSECTION SORTING TESTS
+# ======================================================================================
+# pre-construct the pytest params for the builtin sorters, as they're all very similar
+_template_conf_for_builtin_sorters = """
+sphinx_gallery_conf = {{
+    "examples_dirs": "src",
+    "gallery_dirs": "ex",
+    "within_subsection_order": "{}",
+}}"""
+_builtin_sorters = dict(
+    lines="NumberOfCodeLinesSortKey",  # the default
+    filesize="FileSizeSortKey",
+    filename="FileNameSortKey",
+    title="ExampleTitleSortKey",
+)
+_params_for_testing_builtin_sorters = (
+    pytest.param(
+        kind,  # `sort_key` param
+        None,  # `expected_order` (order is extracted from comments in test site pages)
+        id=f"within_subsection_sort_by_{kind}",
+        marks=pytest.mark.add_conf(
+            content=_template_conf_for_builtin_sorters.format(sorter)
+        ),
+    )
+    for kind, sorter in _builtin_sorters.items()
+)
+# now a more flexible config template, for changing subsection and/or within-subsection
+# order (possibly at the same time):
+_template_conf = """
+{imports}
+sphinx_gallery_conf = {{
+    "examples_dirs": "src",
+    "gallery_dirs": "ex",
+    "subsection_order": {subsection_order},
+    "within_subsection_order": {within_subsection_order},
+}}"""
+# fill in the template config with different test cases and wrap in an `add_conf` mark
+_subsection_explicit_order = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports="from sphinx_gallery.sorting import ExplicitOrder",
+        subsection_order='ExplicitOrder(["src/second-subsection", "src/first-subsection"])',
+        within_subsection_order='"NumberOfCodeLinesSortKey"',  # this is the default
+    )
 )
-def test_example_sorting_default(sphinx_app_wrapper):
-    """Test sorting of examples by default key (number of code lines)."""
-    sphinx_app = sphinx_app_wrapper.create_sphinx_app()
-    _check_order(sphinx_app, "lines")
-
-
-@pytest.mark.add_conf(
-    content="""
-sphinx_gallery_conf = {
-    'examples_dirs': 'src',
-    'gallery_dirs': 'ex',
-    'within_subsection_order': "FileSizeSortKey",
-}"""
+_subsection_explicit_order_list = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports="",
+        subsection_order='["src/second-subsection", "src/first-subsection"]',
+        within_subsection_order='"NumberOfCodeLinesSortKey"',  # this is the default
+    )
 )
-def test_example_sorting_filesize(sphinx_app_wrapper):
-    """Test sorting of examples by filesize."""
-    sphinx_app = sphinx_app_wrapper.create_sphinx_app()
-    _check_order(sphinx_app, "filesize")
-
-
-@pytest.mark.add_conf(
-    content="""
-sphinx_gallery_conf = {
-    'examples_dirs': 'src',
-    'gallery_dirs': 'ex',
-    'within_subsection_order': "FileNameSortKey",
-}"""
+_both_custom_fqn = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports="",
+        subsection_order='"sphinx_gallery.utils._custom_subsection_sorter"',
+        within_subsection_order='"sphinx_gallery.utils._custom_example_sorter"',
+    )
+)
+_within_subsection_custom_fqn = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports="",
+        subsection_order="None",  # the default, AKA, sort by foldername
+        within_subsection_order='"sphinx_gallery.utils._custom_example_sorter"',
+    )
+)
+_custom_func = """
+from sphinx_gallery.sorting import FunctionSortKey
+
+def custom_sorter(filename):
+    ORDER = [
+        "plot_3.py",
+        "plot_2.py",
+        "plot_1.py",
+        "plot_6.py",
+        "plot_5.py",
+        "plot_4.py",
+        "plot_9.py",
+        "plot_7.py",
+        "plot_8.py",
+    ]
+    return ORDER.index(filename)
+"""
+_within_subsection_custom_func = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports=_custom_func,
+        subsection_order="None",  # the default, AKA, sort by foldername
+        within_subsection_order="FunctionSortKey(custom_sorter)",
+    )
+)
+_subsection_fqn_within_subsection_custom_func = pytest.mark.add_conf(
+    content=_template_conf.format(
+        imports=_custom_func,
+        subsection_order='"sphinx_gallery.utils._custom_subsection_sorter"',
+        within_subsection_order="FunctionSortKey(custom_sorter)",
+    )
 )
-def test_example_sorting_filename(sphinx_app_wrapper):
-    """Test sorting of examples by filename."""
-    sphinx_app = sphinx_app_wrapper.create_sphinx_app()
-    _check_order(sphinx_app, "filename")
 
 
-@pytest.mark.add_conf(
-    content="""
-sphinx_gallery_conf = {
-    'examples_dirs': 'src',
-    'gallery_dirs': 'ex',
-    'within_subsection_order': "ExampleTitleSortKey",
-}"""
+@pytest.mark.parametrize(
+    "sort_key,expected_order",
+    (
+        pytest.param(
+            "lines",
+            # ↓ natural order except subsection 2 (789) precedes subsection 1 (456)
+            list("123789456"),
+            id="subsection_sort_by_ExplicitOrder",
+            marks=_subsection_explicit_order,
+        ),
+        pytest.param(
+            "lines",
+            # ↓ natural order except subsection 2 (789) precedes subsection 1 (456)
+            list("123789456"),
+            id="subsection_sort_by_list",
+            marks=_subsection_explicit_order_list,
+        ),
+        pytest.param(
+            None,
+            # ↓ order determined by `utils._CUSTOM_EXAMPLE_ORDER`
+            list("132564879"),
+            id="within_subsection_sort_by_custom_FQN",
+            marks=_within_subsection_custom_fqn,
+        ),
+        pytest.param(
+            None,
+            # ↓ order set by `utils._CUSTOM_EXAMPLE_ORDER`, with middle (564) and
+            # ↓ last (879) thirds swapped due to subsection reordering
+            list("132879564"),
+            id="subsection_and_within_subsection_both_sort_by_custom_FQN",
+            marks=_both_custom_fqn,
+        ),
+        pytest.param(
+            None,
+            # ↓ order set by local `_custom_func` variable within this test file
+            list("321654978"),
+            id="within_subsection_sort_by_custom_func",
+            marks=_within_subsection_custom_func,
+        ),
+        pytest.param(
+            None,
+            # ↓ order set by `_custom_func` in this test file, with middle (654) and
+            # ↓ last (978) thirds swapped due to subsection reordering
+            list("321978654"),
+            id="subsection_sort_by_FQN_and_within_subsection_sort_by_custom_func",
+            marks=_subsection_fqn_within_subsection_custom_func,
+        ),
+        *_params_for_testing_builtin_sorters,
+    ),
 )
-def test_example_sorting_title(sphinx_app_wrapper):
-    """Test sorting of examples by title."""
+def test_example_sorting(sphinx_app_wrapper, sort_key, expected_order):
+    """Test sorting of examples with fully qualified name of a custom sort function."""
     sphinx_app = sphinx_app_wrapper.create_sphinx_app()
-    _check_order(sphinx_app, "title")
+    _check_order(sphinx_app, sort_key, expected_order=expected_order)
 
 
 def test_collect_gallery_files(tmpdir, gallery_conf):
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index 260dc2a..b8d26ac 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -1043,24 +1043,33 @@ def test_capture_repr(
 
 
 @pytest.mark.parametrize(
-    "caprepr_gallery, caprepr_file, expected_out",
+    "caprepr_gallery, caprepr_file, caprepr_block, expected_out",
     [
-        pytest.param(tuple(), ("__repr__",), "2", id="() --> repr"),
-        pytest.param(("__repr__",), "()", "", id="repr --> ()"),
+        pytest.param(tuple(), "()", '("__repr__",)', "2", id="() --> () --> repr"),
+        pytest.param(tuple(), ("__repr__",), None, "2", id="() --> repr --> None"),
+        pytest.param(("__repr__",), "()", None, "", id="repr --> () --> None"),
+        pytest.param(tuple(), ("__repr__",), "()", "", id="() --> repr --> ()"),
+        pytest.param(("__repr__",), "()", "()", "", id="repr --> () --> ()"),
     ],
 )
-def test_per_file_capture_repr(
+def test_capture_repr_per_file_and_per_block(
     gallery_conf,
     caprepr_gallery,
     caprepr_file,
+    caprepr_block,
     expected_out,
     req_mpl,
     req_pil,
     script_vars,
 ):
-    """Tests that per file capture_repr overrides gallery_conf."""
+    """Tests that per file/block capture_repr overrides gallery_conf."""
     compiler = codeop.Compile()
-    code_block = Block("code", "a=2\n2", 1)
+    caprepr_block = (
+        f"# sphinx_gallery_capture_repr_block={caprepr_block}\n"
+        if caprepr_block
+        else ""
+    )
+    code_block = Block("code", f"{caprepr_block}a=2\n2", 1)
     gallery_conf["capture_repr"] = caprepr_gallery
     file_conf = {"capture_repr": caprepr_file}
     output = sg.execute_code_block(
@@ -1070,7 +1079,7 @@ def test_per_file_capture_repr(
 
 
 def test_ignore_repr_types(gallery_conf, req_mpl, req_pil, script_vars):
-    """Tests output capturing with various capture_repr settings."""
+    """Tests that `ignore_repr_types` correctly ignores the output of a specific type."""
     compiler = codeop.Compile()
     code_block = Block("code", "a=2\na", 1)
     gallery_conf["ignore_repr_types"] = r"int"
diff --git a/sphinx_gallery/tests/testconfs/src/first-subsection/GALLERY_HEADER.rst b/sphinx_gallery/tests/testconfs/src/first-subsection/GALLERY_HEADER.rst
new file mode 100644
index 0000000..7b9b928
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/first-subsection/GALLERY_HEADER.rst
@@ -0,0 +1,2 @@
+The First Subsection Test Gallery
+=================================
diff --git a/sphinx_gallery/tests/testconfs/src/first-subsection/plot_4.py b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_4.py
new file mode 100644
index 0000000..bbc9f32
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_4.py
@@ -0,0 +1,14 @@
+"""
+======
+E test
+======
+
+:filename=4:title=5:lines=4:filesize=5:
+"""
+
+print("foofoofoofoofoofoofoofoo")
+print("barbarbarbarbarbarbarbar")
+
+# %%
+#
+# .. minigallery:: src/plot_2.py
diff --git a/sphinx_gallery/tests/testconfs/src/first-subsection/plot_5.py b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_5.py
new file mode 100644
index 0000000..7487e40
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_5.py
@@ -0,0 +1,29 @@
+"""
+======
+F test
+======
+
+:filename=5:title=6:lines=6:filesize=4:
+"""
+
+# let's
+
+# add
+
+# some
+
+# lines
+
+# to
+
+# test
+
+# sorting
+
+# by
+
+# number
+
+# of
+
+# lines
diff --git a/sphinx_gallery/tests/testconfs/src/first-subsection/plot_6.py b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_6.py
new file mode 100644
index 0000000..5fa1b57
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/first-subsection/plot_6.py
@@ -0,0 +1,22 @@
+#!/usr/bin/env python2
+"""
+.. _extra_ref:
+
+=======================================================================================
+D is a file with a title even longer than `plot_1.py`, to make the filesize even larger
+=======================================================================================
+
+:filename=6:title=4:lines=5:filesize=6:
+"""
+
+# let's
+# add
+# some
+# lines
+# to
+# test
+# sorting
+# by
+# number
+# of
+# lines
diff --git a/sphinx_gallery/tests/testconfs/src/second-subsection/GALLERY_HEADER.rst b/sphinx_gallery/tests/testconfs/src/second-subsection/GALLERY_HEADER.rst
new file mode 100644
index 0000000..6aa361e
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/second-subsection/GALLERY_HEADER.rst
@@ -0,0 +1,2 @@
+The Second Subsection Test Gallery
+==================================
diff --git a/sphinx_gallery/tests/testconfs/src/second-subsection/plot_7.py b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_7.py
new file mode 100644
index 0000000..d5d5ce5
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_7.py
@@ -0,0 +1,14 @@
+"""
+======
+E test
+======
+
+:filename=7:title=8:lines=7:filesize=8:
+"""
+
+print("foofoofoofoofoofoofoofoo")
+print("barbarbarbarbarbarbarbar")
+
+# %%
+#
+# .. minigallery:: src/plot_9.py
diff --git a/sphinx_gallery/tests/testconfs/src/second-subsection/plot_8.py b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_8.py
new file mode 100644
index 0000000..9753d8a
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_8.py
@@ -0,0 +1,29 @@
+"""
+======
+F test
+======
+
+:filename=8:title=9:lines=9:filesize=7:
+"""
+
+# let's
+
+# add
+
+# some
+
+# lines
+
+# to
+
+# test
+
+# sorting
+
+# by
+
+# number
+
+# of
+
+# lines
diff --git a/sphinx_gallery/tests/testconfs/src/second-subsection/plot_9.py b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_9.py
new file mode 100644
index 0000000..c8d2831
--- /dev/null
+++ b/sphinx_gallery/tests/testconfs/src/second-subsection/plot_9.py
@@ -0,0 +1,22 @@
+#!/usr/bin/env python2
+"""
+.. _extra_ref:
+
+=======================================================================================
+D is a file with a title even longer than `plot_1.py`, to make the filesize even larger
+=======================================================================================
+
+:filename=9:title=7:lines=8:filesize=9:
+"""
+
+# let's
+# add
+# some
+# lines
+# to
+# test
+# sorting
+# by
+# number
+# of
+# lines
diff --git a/sphinx_gallery/tests/tinybuild/doc/conf.py b/sphinx_gallery/tests/tinybuild/doc/conf.py
index 9aae106..eaf83bf 100644
--- a/sphinx_gallery/tests/tinybuild/doc/conf.py
+++ b/sphinx_gallery/tests/tinybuild/doc/conf.py
@@ -4,6 +4,18 @@ import os.path as op
 
 import sphinx
 
+
+class CustomSortKey:
+    """Basically a clone of FilenameSortKey, for testing custom class sorters."""
+
+    def __init__(self, src_dir):
+        self.src_dir = src_dir
+
+    def __call__(self, item):
+        """Provide the custom sort key."""
+        return item
+
+
 # Where our helpers live
 util_root = "sphinx_gallery.tests.tinybuild.utils"
 
@@ -73,7 +85,7 @@ sphinx_gallery_conf = {
     ],
     "backreferences_dir": "gen_modules/backreferences",
     "subsection_order": f"{util_root}.noop_key",
-    "within_subsection_order": "FileNameSortKey",
+    "within_subsection_order": CustomSortKey,
     "minigallery_sort_order": f"{util_root}.mock_sort",
     "image_scrapers": (f"{util_root}.matplotlib_format_scraper",),
     "expected_failing_examples": [