Repository: python/peps

Issue:
Infra: Fix non-PEP txt files being included and erroring out the build
Presently, as described in #2404 , Sphinx attempts to render any `.txt` files not explicitly excluded in `exclude_patterns` of `conf.py`, in the root or any subdirectories. This results in build failure if any are found, particularly in untracked files which we don't control (e.g. `vendor.txt` in venvs).

Per the [Sphinx docs](https://www.sphinx-doc.org/en/master/usage/configuration.html#id11) we can use full Unix shell-style glob syntax, so adding the glob `**/*.txt` to `exclude_patterns` neatly fixes most instance of this problem by ensuring that `.txt` files are not matched unless they are in the root directory (as all the valid PEPs are).

To handle any remaining cases, i.e. non-PEP `.txt` files that _are_ in the root directory and not explicitly excluded, adding `*[!0-9].txt`,`*[!0-9]?.txt`, `*[!0-9]??.txt`, `*[!0-9]???.txt`, `*[!p]?????.txt`, `*[!e]??????.txt`, and `*[!p]???????.txt` ensures that to be included, `.txt` files must match the equivalent  of the regex `^pep?[0-9]{4}\.txt$`, which is only used for PEPs.

This solution was tested on some sample files in the root and subdirectories, including the reported `venv-spam/vendor.txt`, and several with varying degrees of names somewhat similar to PEPs, and it excluded all of them, while not excluding any valid PEPs.

It also obviates the need for some of our other explicit excludes, but I opted to leave them for now

Fix #2404 
`make sphinx` Breaks when Certain Untracked Files are Present
* `make sphinx` fails if I have some file like "pep-0684-old.rst" (the filename regex in pep_sphinx_extensions/pep_zero_generator/pep_index_generator.py isn't precise enough)
* `make sphinx` fails if I have a venv dir named "venv-spam" sitting at the top of the repo (tries to parse a vendor.txt file inside the venv)

In both cases I'd expect it to work fine, ignoring the relevant files.


Discussion:
I am -1 on this; whilst a nice idea I think it overcomplicates this list massively for a edge-case.

If glob syntax is supported, why not just add `*env*` to the blacklist?

A
The cited example and the stopgap fix is one specific case, but the actual problem @ericsnowcurrently describes in #2404 is general: any `.txt` file added anywhere in this repo, and any `.txt` file added by any tool at a user- or repo-level `.gitignore`d directory, file or path will be interpreted as a reST PEP file and attempted to be built/rendered accordingly (which will likely hard fail the build), unless explicitly deny-listed.

Adding `*env*` is merely a stopgap fix to one very specific case (venvs/virtualenvs that happen to be named `*env*`) of a much more general problem, leaving any other instance of the above not explicitly special-cased into the `exclude_patterns` unsolved. Instead, this PR implements the "allow-list" approach for legacy `.txt` PEPs that @ericsnowcurrently suggests using the existing supported `exclude_pattern` syntax.

I'm not sure I see how adding a handful of globs to an exclude list, while eliminating many of the existing special cases in the process (and avoid having to add an unbounded number more in the future), "massively" increases complexity. Since it as confirmed per my testing, it effectively eliminates any chance of a false negative while producing no false positives, and no more `txt` PEPs will be added, I don't see a practical downside that would motivate us to not simply fix this the right way from the start.
The first case is easier to solve, the second is not possible as far as I'm aware -- Sphinx has no way to specify a whitelist of files -- the `exclude_files` list is the only mechanism and as such must exhaustivley list things. Sorry.

A
I'm not familiar with how Sphinx works.  It is surprising, though, that it can't handle the case.  IIUC, it is blindly walking the entire directory tree and doesn't support a wildcard whitelist.  What does it need to recurse in the PEP repo?
> IIUC, it is blindly walking the entire directory tree and doesn't support a wildcard whitelist. What does it need to recurse in the PEP repo?

You're right. And I'd ask the same question on default recursion. One thing that might be possible is adding `venv*` to `exclude_files` in `conf.py` & hoping Sphinx does string-matching on paths, though untested.

A
Sorry for the delay responding; I've been dealing with urgent family issues most of today.

Good catch. It seems like the issue is ultimately due to having to try to render files with the legacy `.txt` extension along with `.rst`, which results in irrelevant `.txt` files being picked up. (Sidenote: it seems like they were mass renamed before, but that was reverted, IIRC).

However, since full Unix-style glob syntax is supported in `exclude_patterns`, we can exclude any `.txt` files in subdirectories (like this example), and any in the root directory that don't match the exact pattern used for `.txt` PEPs (i.e. the equivalent of the regex `^pep?[0-9]{4}\.txt$`), which fully resolves the issue for the general case.

I implemented and tested this on some sample files in the root and subdirectories (including your example, and a few with varying degrees of similar names), and it excluded all of them, while not excluding any valid PEPs. Therefore, I opened PR #2405 which fixes this. 

Resolving patch:
diff --git a/conf.py b/conf.py
--- a/conf.py
+++ b/conf.py
@@ -8,6 +8,11 @@
 
 sys.path.append(str(Path("pep_sphinx_extensions").absolute()))
 
+# Add 'include_patterns' as a config variable
+from sphinx.config import Config
+Config.config_values['include_patterns'] = [], 'env', []
+del Config
+
 # -- Project information -----------------------------------------------------
 
 project = "PEPs"
@@ -25,24 +30,20 @@
 }
 
 # List of patterns (relative to source dir) to ignore when looking for source files.
+include_patterns = [
+    # Required for Sphinx
+    "contents.rst",
+    # PEP files
+    "pep-????.rst",
+    "pep-????.txt",
+    # PEP ancillary files
+    "pep-????/*.rst",
+    # Documentation
+    "docs/*.rst",
+]
 exclude_patterns = [
-    # Windows:
-    "Thumbs.db",
-    ".DS_Store",
-    # Python:
-    ".venv",
-    "venv",
-    "requirements.txt",
-    # Sphinx:
-    "build",
-    "output.txt",  # Link-check output
-    # PEPs:
-    "pep-0012",
-    "README.rst",
-    "CONTRIBUTING.rst",
-    "pep_sphinx_extensions/LICENCE.rst",
-    # Miscellaneous
-    ".codespell",
+    # PEP Template
+    "pep-0012/pep-NNNN.rst",
 ]
 
 # -- Options for HTML output -------------------------------------------------
diff --git a/pep_sphinx_extensions/__init__.py b/pep_sphinx_extensions/__init__.py
--- a/pep_sphinx_extensions/__init__.py
+++ b/pep_sphinx_extensions/__init__.py
@@ -6,6 +6,7 @@
 
 from docutils.writers.html5_polyglot import HTMLTranslator
 from sphinx import environment
+from sphinx import project
 
 from pep_sphinx_extensions.pep_processor.html import pep_html_builder
 from pep_sphinx_extensions.pep_processor.html import pep_html_translator
@@ -16,6 +17,37 @@
 
 if TYPE_CHECKING:
     from sphinx.application import Sphinx
+    from sphinx.config import Config
+
+
+def find_files(self: environment.BuildEnvironment, config: Config, _b) -> None:
+    """Find all pep source files."""
+    import fnmatch
+    from pathlib import Path
+
+    root = Path(self.project.srcdir).absolute()
+    self.project.docnames = set()
+    for pattern in config.include_patterns:
+        for path in root.glob(pattern):
+            filename = str(path.relative_to(root))
+            if any(fnmatch.fnmatch(filename, pattern) for pattern in config.exclude_patterns):
+                continue
+
+            doc_name = self.project.path2doc(filename)
+            if not doc_name:
+                continue
+
+            if doc_name not in self.project.docnames:
+                self.project.docnames.add(doc_name)
+                continue
+
+            other_files = [str(f.relative_to(root)) for f in root.glob(f"{doc_name}.*")]
+            project.logger.warning(
+                f'multiple files found for the document "{doc_name}": {other_files!r}\n'
+                f'Use {self.doc2path(doc_name)!r} for the build.', once=True)
+
+
+environment.BuildEnvironment.find_files = find_files
 
 
 def _depart_maths():


Test patch: