Repository: sphinx-gallery/sphinx-gallery

Issue:
Variable 'indexst' is undefined when using custom index.rst
When using a custom `index.rst` for the gallery, the following exception is thrown:

```
Traceback (most recent call last):
  File "/home/runner/work/ansys-tools-local-product-launcher/ansys-tools-local-product-launcher/.tox/doc/lib/python3.12/site-packages/sphinx/cmd/build.py", line 332, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/ansys-tools-local-product-launcher/ansys-tools-local-product-launcher/.tox/doc/lib/python3.12/site-packages/sphinx/application.py", line 292, in __init__
    self._init_builder()
  File "/home/runner/work/ansys-tools-local-product-launcher/ansys-tools-local-product-launcher/.tox/doc/lib/python3.12/site-packages/sphinx/application.py", line 366, in _init_builder
    self.events.emit('builder-inited')
  File "/home/runner/work/ansys-tools-local-product-launcher/ansys-tools-local-product-launcher/.tox/doc/lib/python3.12/site-packages/sphinx/events.py", line 109, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function generate_gallery_rst at 0x7f98e601d440> for event 'builder-inited' threw an exception (exception: cannot access local variable 'indexst' where it is not associated with a value)
```

From [this CI run](https://github.com/ansys/ansys-tools-local-product-launcher/actions/runs/11121042825/job/30899262996?pr=186), associated code: https://github.com/ansys/ansys-tools-local-product-launcher/tree/maint/remove_upper_limits [1]

I believe this was introduced in #1332, by removing this code block https://github.com/sphinx-gallery/sphinx-gallery/pull/1332/files#diff-016e2fd897b8720412d190b7936fb295e31dc953c73fcb912ca1362ec8ca2191L660-L663

```python
        else:
            # we are not going to use the index.rst.new that gets made here,
            # but go through the motions to run through all the subsections...
            indexst = "Never used!"
```

However, I'm not sure if the correct fix is to simply re-introduce the code block, or if there is a better solution.

[1] this branch updates `sphinx-gallery` from `0.15.0` to `0.17.1`

Discussion:
lucyleeow: A `__pycache__` gets added to `sphinx_gallery/tests/testconfs/src` during the new `test_own_index_first` test.

When the folder has it's own `index.rst` file, we do not check that these folders have a `index.rst` or gallery header file when we collect subgallery folders. We do not need a index.rst/header file as we do not write the root gallery index or any nested sub gallery index files when the user provides their own root gallery `index.rst`.

@larsoner I've amended the code such that we do not collect any folders that called `__pycache__`. I think we could also avoid collecting any 'dunder' folder nested galleries as well. WDYT?

Patch:
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 8d3762d..1241374 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -2537,13 +2537,16 @@ Manually passing ``index.rst``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 You can bypass Sphinx-Gallery automatically creating an  ``index.rst`` from a
-``GALLERY_HEADER.rst`` in a gallery directory or subdirectory. If your
+``GALLERY_HEADER.rst`` in a gallery directory or nested sub-gallery directory. If your
 ``copyfile_regex`` includes ``index.rst``, and you have an ``index.rst`` in the
-gallery-source instead of the 'GALLERY_HEADER' file, Sphinx-Gallery will use that instead of
-the index it automatically makes.  If you do this, you are responsible for
+gallery-source (i.e., a :ref:`examples_dirs <multiple_galleries_config>` directory),
+Sphinx-Gallery will use that instead and not make an index file for that gallery
+or any of its sub-galleries.
+If you pass your own ``index.rst`` file, you are responsible for
 adding your own Sphinx ``toctree`` in that index (or elsewhere in your Sphinx
 documentation) that includes any gallery items or other files in that
-directory.
+directory. You are also responsible for adding any necessary ``index.rst``
+files for that gallery's sub-galleries.
 
 .. _show_api_usage:
 
diff --git a/sphinx_gallery/gen_gallery.py b/sphinx_gallery/gen_gallery.py
index c3b29f4..38d9b53 100644
--- a/sphinx_gallery/gen_gallery.py
+++ b/sphinx_gallery/gen_gallery.py
@@ -504,7 +504,7 @@ def _fill_gallery_conf_defaults(sphinx_gallery_conf, app=None, check_keys=True):
     return gallery_conf
 
 
-def get_subsections(srcdir, examples_dir, gallery_conf, check_for_index=True):
+def get_subsections(srcdir, examples_dir, gallery_conf, check_for_header=True):
     """Return the list of subsections of a gallery.
 
     Parameters
@@ -515,8 +515,8 @@ def get_subsections(srcdir, examples_dir, gallery_conf, check_for_index=True):
         path to the examples directory relative to conf.py
     gallery_conf : Dict[str, Any]
         Sphinx-Gallery configuration dictionary.
-    check_for_index : bool
-        only return subfolders contain a GALLERY_HEADER file, default True
+    check_for_header : bool
+        only return subfolders that contain a GALLERY_HEADER file, default True
 
     Returns
     -------
@@ -530,22 +530,27 @@ def get_subsections(srcdir, examples_dir, gallery_conf, check_for_index=True):
         if isinstance(sortkey, list):
             sortkey = ExplicitOrder(sortkey)
     subfolders = [subfolder for subfolder in os.listdir(examples_dir)]
-    if check_for_index:
+    if check_for_header:
         subfolders = [
             subfolder
             for subfolder in subfolders
+            # Return is not `None` only when a gallery head file is found
             if _get_gallery_header(
                 os.path.join(examples_dir, subfolder), gallery_conf, raise_error=False
             )
             is not None
         ]
     else:
-        # just make sure its a directory
+        # just make sure its a directory, that is not `__pycache__`
         subfolders = [
             subfolder
             for subfolder in subfolders
-            if os.path.isdir(os.path.join(examples_dir, subfolder))
+            if (
+                subfolder != "__pycache__"
+                and os.path.isdir(os.path.join(examples_dir, subfolder))
+            )
         ]
+
     base_examples_dir_path = os.path.relpath(examples_dir, srcdir)
     subfolders_with_path = [
         os.path.join(base_examples_dir_path, item) for item in subfolders
@@ -748,11 +753,13 @@ def generate_gallery_rst(app):
             is_subsection=False,
         )
 
-        # `this_context` is None when user provides own index.rst
+        # `this_content` is None when user provides own index.rst
         sg_root_index = this_content is not None
         costs += this_costs
         write_computation_times(gallery_conf, gallery_dir_abs_path, this_costs)
 
+        # `indexst` variable must exist, as passed to `_finish_index_rst`
+        indexst = ""
         # Create root gallery index.rst
         if sg_root_index:
             # :orphan: to suppress "not included in TOCTREE" sphinx warnings
@@ -768,7 +775,7 @@ def generate_gallery_rst(app):
             app.builder.srcdir,
             examples_dir_abs_path,
             gallery_conf,
-            check_for_index=sg_root_index,
+            check_for_header=sg_root_index,
         )
         for subsection in subsecs:
             src_dir = os.path.join(examples_dir_abs_path, subsection)


Test patch:
diff --git a/sphinx_gallery/tests/conftest.py b/sphinx_gallery/tests/conftest.py
index 4087a41..4f904d9 100644
--- a/sphinx_gallery/tests/conftest.py
+++ b/sphinx_gallery/tests/conftest.py
@@ -16,6 +16,20 @@ from sphinx_gallery import docs_resolv, gen_gallery, gen_rst, py_source_parser
 from sphinx_gallery.scrapers import _import_matplotlib
 from sphinx_gallery.utils import _get_image
 
+INDEX_RST = """
+=============
+Own index.rst
+=============
+
+Own index.rst file.
+
+.. toctree::
+
+    plot_1
+    plot_2
+    plot_3
+"""
+
 NESTED_PY = """\"\"\"
 Header
 ======
@@ -218,17 +232,20 @@ def sphinx_app_wrapper(tmpdir, conf_file, rst_file, req_mpl, req_pil):
     # Copy files to 'examples/' as well because default `examples_dirs` is
     # '../examples' - for tests where we don't update config
     shutil.copytree((_fixturedir / "src"), (Path(tmpdir) / "examples"))
-    if rst_file:
-        with open((srcdir / "minigallery_test.rst"), "w") as rstfile:
-            rstfile.write(rst_file)
+    if rst_file == "own index.rst":
+        with open((srcdir / "src" / "index.rst"), "w") as file:
+            file.write(INDEX_RST)
+    elif rst_file:
+        with open((srcdir / "minigallery_test.rst"), "w") as file:
+            file.write(rst_file)
         # Add nested gallery
         if "sub_folder/sub_sub_folder" in rst_file:
             dir_path = srcdir / "src" / "sub_folder" / "sub_sub_folder"
             dir_path.mkdir(parents=True)
-            with open((dir_path / "plot_nested.py"), "w") as pyfile:
-                pyfile.write(NESTED_PY)
-            with open((dir_path / "GALLERY_HEADER.rst"), "w") as rstfile:
-                rstfile.write(GALLERY_HEADER)
+            with open((dir_path / "plot_nested.py"), "w") as file:
+                file.write(NESTED_PY)
+            with open((dir_path / "GALLERY_HEADER.rst"), "w") as file:
+                file.write(GALLERY_HEADER)
 
     base_config = f"""
 import os
diff --git a/sphinx_gallery/tests/test_gen_gallery.py b/sphinx_gallery/tests/test_gen_gallery.py
index 09eeb56..a73f0bf 100644
--- a/sphinx_gallery/tests/test_gen_gallery.py
+++ b/sphinx_gallery/tests/test_gen_gallery.py
@@ -407,6 +407,21 @@ def test_collect_gallery_files_ignore_pattern(tmpdir, gallery_conf):
     assert collected_files == expected_files
 
 
+@pytest.mark.add_conf(
+    content=r"""
+sphinx_gallery_conf = {
+    'examples_dirs': 'src',
+    'gallery_dirs': 'ex',
+    'copyfile_regex': r'.*\.rst',
+}"""
+)
+@pytest.mark.add_rst(file="own index.rst")
+def test_own_index_first(sphinx_app_wrapper):
+    """Test `generate_gallery_rst` works when own index gallery is first (and only)."""
+    # Issue #1382
+    sphinx_app_wrapper.build_sphinx_app()
+
+
 @pytest.mark.add_conf(
     content="""
 sphinx_gallery_conf = {
diff --git a/sphinx_gallery/tests/test_gen_rst.py b/sphinx_gallery/tests/test_gen_rst.py
index ea94196..260dc2a 100644
--- a/sphinx_gallery/tests/test_gen_rst.py
+++ b/sphinx_gallery/tests/test_gen_rst.py
@@ -651,6 +651,7 @@ def test_gen_dir_rst(gallery_conf, gallery_header):
     ],
 )
 def test_gen_dir_rst_invalid_header(gallery_conf, gallery_header):
+    """Check `_get_gallery_header` raises error for invalid header extension."""
     with open(os.path.join(gallery_conf["src_dir"], gallery_header), "wb") as fid:
         fid.write("Testing\n=======\n\nÓscar here.".encode())
     with pytest.raises(ExtensionError, match="does not have a GALLERY_HEADER"):