Repository: mkdocs/mkdocs

Issue:
Hidden files/directories
Revival of #1738.

Could you consider adding an option to serve hidden static files and templates? I cannot see any reason why one wouldn't want to serve these hidden files, and in particular the `.well-known` directory is used for a lot of things that one could want in their documentation, including:
- ACME challenges (for Let's Encrypt)
- [`security.txt`](https://securitytxt.org/)
- Keybase Site Verification
- Do-Not-Track policies
- HTTPS Opportunistic Encryption

see https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml

It would be very useful to have this as a built-in option. A plugin, which could work around this, seems completely unnecessary. Could you explain why this is not enabled, and reconsider the behaviour, or at least add an exception for the [well standardised](https://tools.ietf.org/html/rfc5785)  `.well-known` directory? Or is there a good reason I'm missing not to do this?

Discussion:
oprypin: I'm wondering, maybe for the sake of plugins it would be better to keep these files in the `files` collection but then only exclude them from being copied to the destination.
oprypin: Yet another idea I just had- for the sake of writing drafts, these pages would still be shown in `mkdocs serve` (perhaps with a forcibly inserted message about that on the page) but then excluded from `build`
pawamoy: That would be really useful indeed. I wonder if the corresponding nav entry for a given excluded file/page should also be removed when building, because when I write something for my blog I have to comment/uncomment the post in the nav until it's ready.

Patch:
diff --git a/.flake8 b/.flake8
index f074e308..8b95ac5a 100644
--- a/.flake8
+++ b/.flake8
@@ -1,3 +1,2 @@
 [flake8]
-max-line-length = 119
-extend-ignore = E203, TC004
+extend-ignore = E501, E203, TC004
diff --git a/docs/user-guide/configuration.md b/docs/user-guide/configuration.md
index f4497024..1fb55c94 100644
--- a/docs/user-guide/configuration.md
+++ b/docs/user-guide/configuration.md
@@ -291,6 +291,69 @@ server root and effectively points to `https://example.com/bugs/`. Of course, th
 list of all the Markdown files found within the `docs_dir` and its
 sub-directories. Index files will always be listed first within a sub-section.
 
+### exclude_docs
+
+NEW: **New in version 1.5.**
+
+This config defines patterns of files (under [`docs_dir`](#docs_dir)) to not be picked up into the built site.
+
+Example:
+
+```yaml
+exclude_docs: |
+    api-config.json    # A file with this name anywhere.
+    drafts/            # A "drafts" directory anywhere.
+    /requirements.txt  # Top-level "docs/requirements.txt".
+    *.py               # Any file with this extension anywhere.
+    !/foo/example.py   # But keep this particular file.
+```
+
+This follows the [.gitignore pattern format](https://git-scm.com/docs/gitignore#_pattern_format).
+
+Note that `mkdocs serve` does *not* follow this setting and instead displays excluded documents but with a "DRAFT" mark. To prevent this effect, you can run `mkdocs serve --clean`.
+
+The following defaults are always implicitly prepended - to exclude dot-files (and directories) as well as the top-level `templates` directory:
+
+```yaml
+exclude_docs: |
+    .*
+    /templates/
+```
+
+So, in order to really start this config fresh, you'd need to specify a negated version of these entries first.
+
+Otherwise you could for example opt only certain dot-files back into the site:
+
+```yaml
+exclude_docs: |
+    !.assets  # Don't exclude '.assets' although all other '.*' are excluded
+```
+
+### not_in_nav
+
+NEW: **New in version 1.5.**
+
+NOTE: This option does *not* actually exclude anything from the nav.
+
+If you want to include some docs into the site but intentionally exclude them from the nav, normally MkDocs warns about this.
+
+Adding such patterns of files (relative to [`docs_dir`](#docs_dir)) into the `not_in_nav` config will prevent such warnings.
+
+Example:
+
+```yaml
+nav:
+    - Foo: foo.md
+    - Bar: bar.md
+
+not_in_nav: |
+    /private.md
+```
+
+As the previous option, this follows the .gitignore pattern format.
+
+NOTE: Adding a given file to [`exclude_docs`](#exclude_docs) takes precedence over and implies `not_in_nav`.
+
 ## Build directories
 
 ### theme
diff --git a/docs/user-guide/writing-your-docs.md b/docs/user-guide/writing-your-docs.md
index 66a10067..91b63fe8 100644
--- a/docs/user-guide/writing-your-docs.md
+++ b/docs/user-guide/writing-your-docs.md
@@ -27,10 +27,7 @@ extensions may be used for your Markdown source files: `markdown`, `mdown`,
 directory will be rendered in the built site regardless of any settings.
 
 NOTE:
-Files and directories with names which begin with a dot (for example:
-`.foo.md` or `.bar/baz.md`) are ignored by MkDocs, which matches the
-behavior of most web servers. There is no option to override this
-behavior.
+Files and directories with names which begin with a dot (for example: `.foo.md` or `.bar/baz.md`) are ignored by MkDocs. This can be overridden with the [`exclude_docs` config](configuration.md#exclude_docs).
 
 You can also create multi-page documentation, by creating several Markdown
 files:
diff --git a/mkdocs/__main__.py b/mkdocs/__main__.py
index 677a2f78..e1a9b695 100644
--- a/mkdocs/__main__.py
+++ b/mkdocs/__main__.py
@@ -46,6 +46,10 @@ def _showwarning(message, category, filename, lineno, file=None, line=None):
 
 
 def _enable_warnings():
+    from mkdocs.commands import build
+
+    build.log.addFilter(utils.DuplicateFilter())
+
     warnings.simplefilter('module', DeprecationWarning)
     warnings.showwarning = _showwarning
 
@@ -111,8 +115,9 @@ site_dir_help = "The directory to output the result of the documentation build."
 use_directory_urls_help = "Use directory URLs when building pages (the default)."
 reload_help = "Enable the live reloading in the development server (this is the default)"
 no_reload_help = "Disable the live reloading in the development server."
-dirty_reload_help = (
-    "Enable the live reloading in the development server, but only re-build files that have changed"
+serve_dirty_help = "Only re-build files that have changed."
+serve_clean_help = (
+    "Build the site without any effects of `mkdocs serve` - pure `mkdocs build`, then serve."
 )
 commit_message_help = (
     "A commit message to use when committing to the "
@@ -220,21 +225,23 @@ def cli():
 
 @cli.command(name="serve")
 @click.option('-a', '--dev-addr', help=dev_addr_help, metavar='<IP:PORT>')
-@click.option('--livereload', 'livereload', flag_value='livereload', help=reload_help, default=True)
+@click.option('--livereload', 'livereload', flag_value='livereload', default=True, hidden=True)
 @click.option('--no-livereload', 'livereload', flag_value='no-livereload', help=no_reload_help)
-@click.option('--dirtyreload', 'livereload', flag_value='dirty', help=dirty_reload_help)
+@click.option('--dirtyreload', 'build_type', flag_value='dirty', hidden=True)
+@click.option('--dirty', 'build_type', flag_value='dirty', help=serve_dirty_help)
+@click.option('-c', '--clean', 'build_type', flag_value='clean', help=serve_clean_help)
 @click.option('--watch-theme', help=watch_theme_help, is_flag=True)
 @click.option(
     '-w', '--watch', help=watch_help, type=click.Path(exists=True), multiple=True, default=[]
 )
 @common_config_options
 @common_options
-def serve_command(dev_addr, livereload, watch, **kwargs):
+def serve_command(**kwargs):
     """Run the builtin development server"""
     from mkdocs.commands import serve
 
     _enable_warnings()
-    serve.serve(dev_addr=dev_addr, livereload=livereload, watch=watch, **kwargs)
+    serve.serve(**kwargs)
 
 
 @cli.command(name="build")
diff --git a/mkdocs/commands/build.py b/mkdocs/commands/build.py
index a491ecbc..1612e55c 100644
--- a/mkdocs/commands/build.py
+++ b/mkdocs/commands/build.py
@@ -5,7 +5,7 @@ import logging
 import os
 import time
 from typing import TYPE_CHECKING, Any, Sequence
-from urllib.parse import urlsplit
+from urllib.parse import urljoin, urlsplit
 
 import jinja2
 from jinja2.exceptions import TemplateNotFound
@@ -13,28 +13,18 @@ from jinja2.exceptions import TemplateNotFound
 import mkdocs
 from mkdocs import utils
 from mkdocs.exceptions import Abort, BuildError
-from mkdocs.structure.files import File, Files, get_files
+from mkdocs.structure.files import File, Files, InclusionLevel, _set_exclusions, get_files
 from mkdocs.structure.nav import Navigation, get_navigation
+from mkdocs.structure.pages import Page
+from mkdocs.utils import DuplicateFilter  # noqa - legacy re-export
 
 if TYPE_CHECKING:
     from mkdocs.config.defaults import MkDocsConfig
-    from mkdocs.structure.pages import Page
-
-
-class DuplicateFilter:
-    """Avoid logging duplicate messages."""
-
-    def __init__(self) -> None:
-        self.msgs: set[str] = set()
-
-    def __call__(self, record: logging.LogRecord) -> bool:
-        rv = record.msg not in self.msgs
-        self.msgs.add(record.msg)
-        return rv
 
+if TYPE_CHECKING:
+    from mkdocs.livereload import LiveReloadServer
 
 log = logging.getLogger(__name__)
-log.addFilter(DuplicateFilter())
 
 
 def get_context(
@@ -202,6 +192,7 @@ def _build_page(
     nav: Navigation,
     env: jinja2.Environment,
     dirty: bool = False,
+    excluded: bool = False,
 ) -> None:
     """Pass a Page to theme template and write output to site_dir."""
 
@@ -229,6 +220,13 @@ def _build_page(
             'page_context', context, page=page, config=config, nav=nav
         )
 
+        if excluded:
+            page.content = (
+                '<div class="mkdocs-draft-marker" title="This page will not be included into the built site.">'
+                'DRAFT'
+                '</div>' + (page.content or '')
+            )
+
         # Render the template.
         output = template.render(context)
 
@@ -254,7 +252,9 @@ def _build_page(
         raise
 
 
-def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False) -> None:
+def build(
+    config: MkDocsConfig, live_server: LiveReloadServer | None = None, dirty: bool = False
+) -> None:
     """Perform a full site build."""
 
     logger = logging.getLogger('mkdocs')
@@ -265,6 +265,8 @@ def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False)
     if config.strict:
         logging.getLogger('mkdocs').addHandler(warning_counter)
 
+    inclusion = InclusionLevel.all if live_server else InclusionLevel.is_included
+
     try:
         start = time.monotonic()
 
@@ -297,6 +299,8 @@ def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False)
 
         # Run `files` plugin events.
         files = config.plugins.run_event('files', files, config=config)
+        # If plugins have added files but haven't set their inclusion level, calculate it again.
+        _set_exclusions(files._files, config)
 
         nav = get_navigation(files, config)
 
@@ -304,10 +308,21 @@ def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False)
         nav = config.plugins.run_event('nav', nav, config=config, files=files)
 
         log.debug("Reading markdown pages.")
-        for file in files.documentation_pages():
+        excluded = []
+        for file in files.documentation_pages(inclusion=inclusion):
             log.debug(f"Reading: {file.src_uri}")
+            if file.page is None and file.inclusion.is_excluded():
+                if live_server:
+                    excluded.append(urljoin(live_server.url, file.url))
+                Page(None, file, config)
             assert file.page is not None
             _populate_page(file.page, config, files, dirty)
+        if excluded:
+            log.info(
+                "The following pages are being built only for the preview "
+                "but will be excluded from `mkdocs build` per `exclude_docs`:\n  - "
+                + "\n  - ".join(excluded)
+            )
 
         # Run `env` plugin events.
         env = config.plugins.run_event('env', env, config=config, files=files)
@@ -316,7 +331,7 @@ def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False)
         # with lower precedence get written first so that files with higher precedence can overwrite them.
 
         log.debug("Copying static assets.")
-        files.copy_static_files(dirty=dirty)
+        files.copy_static_files(dirty=dirty, inclusion=inclusion)
 
         for template in config.theme.static_templates:
             _build_theme_template(template, env, files, config, nav)
@@ -325,10 +340,12 @@ def build(config: MkDocsConfig, live_server: bool = False, dirty: bool = False)
             _build_extra_template(template, files, config, nav)
 
         log.debug("Building markdown pages.")
-        doc_files = files.documentation_pages()
+        doc_files = files.documentation_pages(inclusion=inclusion)
         for file in doc_files:
             assert file.page is not None
-            _build_page(file.page, config, doc_files, nav, env, dirty)
+            _build_page(
+                file.page, config, doc_files, nav, env, dirty, excluded=file.inclusion.is_excluded()
+            )
 
         # Run `post_build` plugin events.
         config.plugins.run_event('post_build', config=config)
diff --git a/mkdocs/commands/serve.py b/mkdocs/commands/serve.py
index ee197d62..3ad76b9b 100644
--- a/mkdocs/commands/serve.py
+++ b/mkdocs/commands/serve.py
@@ -28,6 +28,7 @@ def serve(
     theme=None,
     theme_dir=None,
     livereload='livereload',
+    build_type=None,
     watch_theme=False,
     watch=[],
     **kwargs,
@@ -58,8 +59,13 @@ def serve(
         **kwargs,
     )
 
-    live_server = livereload in ('dirty', 'livereload')
-    dirty = livereload == 'dirty'
+    is_clean = build_type == 'clean'
+    is_dirty = build_type == 'dirty'
+
+    config = get_config()
+    config['plugins'].run_event(
+        'startup', command=('build' if is_clean else 'serve'), dirty=is_dirty
+    )
 
     def builder(config: MkDocsConfig | None = None):
         log.info("Building documentation...")
@@ -75,31 +81,28 @@ def serve(
         # Override a few config settings after validation
         config.site_url = f'http://{config.dev_addr}{mount_path(config)}'
 
-        build(config, live_server=live_server, dirty=dirty)
+        build(config, live_server=None if is_clean else server, dirty=is_dirty)
 
-    config = get_config()
-    config['plugins'].run_event('startup', command='serve', dirty=dirty)
+    host, port = config.dev_addr
+    server = LiveReloadServer(
+        builder=builder, host=host, port=port, root=site_dir, mount_path=mount_path(config)
+    )
+
+    def error_handler(code) -> bytes | None:
+        if code in (404, 500):
+            error_page = join(site_dir, f'{code}.html')
+            if isfile(error_page):
+                with open(error_page, 'rb') as f:
+                    return f.read()
+        return None
+
+    server.error_handler = error_handler
 
     try:
         # Perform the initial build
         builder(config)
 
-        host, port = config.dev_addr
-        server = LiveReloadServer(
-            builder=builder, host=host, port=port, root=site_dir, mount_path=mount_path(config)
-        )
-
-        def error_handler(code) -> bytes | None:
-            if code in (404, 500):
-                error_page = join(site_dir, f'{code}.html')
-                if isfile(error_page):
-                    with open(error_page, 'rb') as f:
-                        return f.read()
-            return None
-
-        server.error_handler = error_handler
-
-        if live_server:
+        if livereload:
             # Watch the documentation files, the config file and the theme files.
             server.watch(config.docs_dir)
             server.watch(config.config_file_path)
diff --git a/mkdocs/config/config_options.py b/mkdocs/config/config_options.py
index 7ffc599c..fb8a912f 100644
--- a/mkdocs/config/config_options.py
+++ b/mkdocs/config/config_options.py
@@ -28,6 +28,8 @@ from urllib.parse import quote as urlquote
 from urllib.parse import urlsplit, urlunsplit
 
 import markdown
+import pathspec
+import pathspec.gitignore
 
 from mkdocs import plugins, theme, utils
 from mkdocs.config.base import (
@@ -1118,3 +1120,15 @@ class Hooks(BaseConfigOption[List[types.ModuleType]]):
         plugins = config[self.plugins_key]
         for name, hook in config[key_name].items():
             plugins[name] = hook
+
+
+class PathSpec(BaseConfigOption[pathspec.gitignore.GitIgnoreSpec]):
+    """A path pattern based on gitignore-like syntax."""
+
+    def run_validation(self, value: object) -> pathspec.gitignore.GitIgnoreSpec:
+        if not isinstance(value, str):
+            raise ValidationError(f'Expected a multiline string, but a {type(value)} was given.')
+        try:
+            return pathspec.gitignore.GitIgnoreSpec.from_lines(lines=value.splitlines())
+        except ValueError as e:
+            raise ValidationError(str(e))
diff --git a/mkdocs/config/defaults.py b/mkdocs/config/defaults.py
index ba6681d3..ade08ff8 100644
--- a/mkdocs/config/defaults.py
+++ b/mkdocs/config/defaults.py
@@ -26,6 +26,12 @@ class MkDocsConfig(base.Config):
     """Defines the structure of the navigation."""
     pages = c.Deprecated(removed=True, moved_to='nav')
 
+    exclude_docs = c.Optional(c.PathSpec())
+    """Gitignore-like patterns of files (relative to docs dir) to exclude from the site."""
+
+    not_in_nav = c.Optional(c.PathSpec())
+    """Gitignore-like patterns of files (relative to docs dir) that are not intended to be in the nav."""
+
     site_url = c.Optional(c.URL(is_dir=True))
     """The full URL to where the documentation will be hosted."""
 
diff --git a/mkdocs/livereload/__init__.py b/mkdocs/livereload/__init__.py
index 58066e95..1267979b 100644
--- a/mkdocs/livereload/__init__.py
+++ b/mkdocs/livereload/__init__.py
@@ -70,7 +70,6 @@ class LiveReloadServer(socketserver.ThreadingMixIn, wsgiref.simple_server.WSGISe
         mount_path: str = "/",
         polling_interval: float = 0.5,
         shutdown_delay: float = 0.25,
-        **kwargs,
     ) -> None:
         self.builder = builder
         self.server_name = host
@@ -88,7 +87,7 @@ class LiveReloadServer(socketserver.ThreadingMixIn, wsgiref.simple_server.WSGISe
         # To allow custom error pages.
         self.error_handler: Callable[[int], bytes | None] = lambda code: None
 
-        super().__init__((host, port), _Handler, **kwargs)
+        super().__init__((host, port), _Handler, bind_and_activate=False)
         self.set_app(self.serve_request)
 
         self._wanted_epoch = _timestamp()  # The version of the site that started building.
@@ -151,15 +150,21 @@ class LiveReloadServer(socketserver.ThreadingMixIn, wsgiref.simple_server.WSGISe
             self.observer.unschedule(self._watch_refs.pop(path))
 
     def serve(self):
-        self.observer.start()
+        try:
+            self.server_bind()
+            self.server_activate()
+
+            self.observer.start()
 
-        paths_str = ", ".join(f"'{_try_relativize_path(path)}'" for path in self._watched_paths)
-        log.info(f"Watching paths for changes: {paths_str}")
+            paths_str = ", ".join(f"'{_try_relativize_path(path)}'" for path in self._watched_paths)
+            log.info(f"Watching paths for changes: {paths_str}")
 
-        log.info(f"Serving on {self.url}")
-        self.serve_thread.start()
+            log.info(f"Serving on {self.url}")
+            self.serve_thread.start()
 
-        self._build_loop()
+            self._build_loop()
+        finally:
+            self.server_close()
 
     def _build_loop(self):
         while True:
diff --git a/mkdocs/plugins.py b/mkdocs/plugins.py
index af54f24c..2c8ecca0 100644
--- a/mkdocs/plugins.py
+++ b/mkdocs/plugins.py
@@ -111,7 +111,7 @@ class BasePlugin(Generic[SomeConfig]):
 
         Parameters:
             command: the command that MkDocs was invoked with, e.g. "serve" for `mkdocs serve`.
-            dirty: whether `--dirtyreload` or `--dirty` flags were passed.
+            dirty: whether `--dirty` flag was passed.
         """
 
     def on_shutdown(self) -> None:
diff --git a/mkdocs/structure/files.py b/mkdocs/structure/files.py
index 5d148c4c..49f7565a 100644
--- a/mkdocs/structure/files.py
+++ b/mkdocs/structure/files.py
@@ -1,15 +1,20 @@
 from __future__ import annotations
 
+import enum
 import fnmatch
 import logging
 import os
 import posixpath
 import shutil
+import warnings
 from pathlib import PurePath
-from typing import TYPE_CHECKING, Any, Iterable, Iterator, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, Callable, Iterable, Iterator, Mapping, Sequence
 from urllib.parse import quote as urlquote
 
 import jinja2.environment
+import pathspec
+import pathspec.gitignore
+import pathspec.util
 
 from mkdocs import utils
 
@@ -21,13 +26,38 @@ if TYPE_CHECKING:
 log = logging.getLogger(__name__)
 
 
+class InclusionLevel(enum.Enum):
+    Excluded = -2
+    """The file is excluded from the final site, but will still be populated during `mkdocs serve`."""
+    NotInNav = -1
+    """The file is part of the site, but doesn't produce nav warnings."""
+    Undefined = 0
+    """Still needs to be computed based on the config. If the config doesn't kick in, acts the same as `included`."""
+    Included = 1
+    """The file is part of the site. Documentation pages that are omitted from the nav will produce warnings."""
+
+    def all(self):
+        return True
+
+    def is_included(self):
+        return self.value > self.Excluded.value
+
+    def is_excluded(self):
+        return self.value <= self.Excluded.value
+
+    def is_in_nav(self):
+        return self.value > self.NotInNav.value
+
+    def is_not_in_nav(self):
+        return self.value <= self.NotInNav.value
+
+
 class Files:
     """A collection of [File][mkdocs.structure.files.File] objects."""
 
     def __init__(self, files: list[File]) -> None:
         self._files = files
         self._src_uris: dict[str, File] | None = None
-        self._documentation_pages: Sequence[File] | None = None
 
     def __iter__(self) -> Iterator[File]:
         """Iterate over the files within."""
@@ -60,25 +90,30 @@ class Files:
 
     def append(self, file: File) -> None:
         """Append file to Files collection."""
-        self._src_uris = self._documentation_pages = None
+        self._src_uris = None
         self._files.append(file)
 
     def remove(self, file: File) -> None:
         """Remove file from Files collection."""
-        self._src_uris = self._documentation_pages = None
+        self._src_uris = None
         self._files.remove(file)
 
-    def copy_static_files(self, dirty: bool = False) -> None:
+    def copy_static_files(
+        self,
+        dirty: bool = False,
+        *,
+        inclusion: Callable[[InclusionLevel], bool] = InclusionLevel.is_included,
+    ) -> None:
         """Copy static files from source to destination."""
         for file in self:
-            if not file.is_documentation_page():
+            if not file.is_documentation_page() and inclusion(file.inclusion):
                 file.copy_file(dirty)
 
-    def documentation_pages(self) -> Sequence[File]:
+    def documentation_pages(
+        self, *, inclusion: Callable[[InclusionLevel], bool] = InclusionLevel.is_included
+    ) -> Sequence[File]:
         """Return iterable of all Markdown page file objects."""
-        if self._documentation_pages is None:
-            self._documentation_pages = [file for file in self if file.is_documentation_page()]
-        return self._documentation_pages
+        return [file for file in self if file.is_documentation_page() and inclusion(file.inclusion)]
 
     def static_pages(self) -> Sequence[File]:
         """Return iterable of all static page file objects."""
@@ -155,6 +190,9 @@ class File:
     url: str
     """The URI of the destination file relative to the destination directory as a string."""
 
+    inclusion: InclusionLevel
+    """Whether the file will be excluded from the built site."""
+
     @property
     def src_path(self) -> str:
         """Same as `src_uri` (and synchronized with it) but will use backslashes on Windows. Discouraged."""
@@ -183,6 +221,7 @@ class File:
         use_directory_urls: bool,
         *,
         dest_uri: str | None = None,
+        inclusion: InclusionLevel = InclusionLevel.Undefined,
     ) -> None:
         self.page = None
         self.src_path = path
@@ -193,6 +232,7 @@ class File:
         self.url = self._get_url(use_directory_urls)
         self.abs_src_path = os.path.normpath(os.path.join(src_dir, self.src_uri))
         self.abs_dest_path = os.path.normpath(os.path.join(dest_dir, self.dest_uri))
+        self.inclusion = inclusion
 
     def __eq__(self, other) -> bool:
         return (
@@ -276,52 +316,77 @@ class File:
         return self.src_uri.endswith('.css')
 
 
+_default_exclude = pathspec.gitignore.GitIgnoreSpec.from_lines(['.*', '/templates/'])
+
+
+def _set_exclusions(files: Iterable[File], config: MkDocsConfig | Mapping[str, Any]) -> None:
+    """Re-calculate which files are excluded, based on the patterns in the config."""
+    exclude: pathspec.gitignore.GitIgnoreSpec | None = config.get('exclude_docs')
+    exclude = _default_exclude + exclude if exclude else _default_exclude
+    nav_exclude: pathspec.gitignore.GitIgnoreSpec | None = config.get('not_in_nav')
+
+    for file in files:
+        if file.inclusion == InclusionLevel.Undefined:
+            if exclude.match_file(file.src_uri):
+                file.inclusion = InclusionLevel.Excluded
+            elif nav_exclude and nav_exclude.match_file(file.src_uri):
+                file.inclusion = InclusionLevel.NotInNav
+            else:
+                file.inclusion = InclusionLevel.Included
+
+
 def get_files(config: MkDocsConfig | Mapping[str, Any]) -> Files:
     """Walk the `docs_dir` and return a Files collection."""
-    files = []
-    exclude = ['.*', '/templates']
-
+    files: list[File] = []
+    conflicting_files: list[tuple[File, File]] = []
     for source_dir, dirnames, filenames in os.walk(config['docs_dir'], followlinks=True):
         relative_dir = os.path.relpath(source_dir, config['docs_dir'])
-
-        for dirname in list(dirnames):
-            path = os.path.normpath(os.path.join(relative_dir, dirname))
-            # Skip any excluded directories
-            if _filter_paths(basename=dirname, path=path, is_dir=True, exclude=exclude):
-                dirnames.remove(dirname)
         dirnames.sort()
-
-        for filename in _sort_files(filenames):
-            path = os.path.normpath(os.path.join(relative_dir, filename))
-            # Skip any excluded files
-            if _filter_paths(basename=filename, path=path, is_dir=False, exclude=exclude):
-                continue
-            # Skip README.md if an index file also exists in dir
-            if filename == 'README.md' and 'index.md' in filenames:
+        filenames.sort(key=_file_sort_key)
+
+        files_by_dest: dict[str, File] = {}
+        for filename in filenames:
+            file = File(
+                os.path.join(relative_dir, filename),
+                config['docs_dir'],
+                config['site_dir'],
+                config['use_directory_urls'],
+            )
+            # Skip README.md if an index file also exists in dir (part 1)
+            prev_file = files_by_dest.setdefault(file.dest_uri, file)
+            if prev_file is not file:
+                conflicting_files.append((prev_file, file))
+            files.append(file)
+            prev_file = file
+
+    _set_exclusions(files, config)
+    # Skip README.md if an index file also exists in dir (part 2)
+    for a, b in conflicting_files:
+        if b.inclusion.is_included():
+            if a.inclusion.is_included():
                 log.warning(
-                    f"Both index.md and README.md found. Skipping README.md from {source_dir}"
+                    f"Excluding '{a.src_uri}' from the site because it conflicts with '{b.src_uri}'."
                 )
-                continue
-            files.append(
-                File(path, config['docs_dir'], config['site_dir'], config['use_directory_urls'])
-            )
+            files.remove(a)
+        else:
+            files.remove(b)
 
     return Files(files)
 
 
-def _sort_files(filenames: Iterable[str]) -> list[str]:
+def _file_sort_key(f: str):
     """Always sort `index` or `README` as first filename in list."""
+    return (os.path.splitext(f)[0] not in ('index', 'README'), f)
 
-    def key(f):
-        if os.path.splitext(f)[0] in ['index', 'README']:
-            return (0,)
-        return (1, f)
 
-    return sorted(filenames, key=key)
+def _sort_files(filenames: Iterable[str]) -> list[str]:
+    return sorted(filenames, key=_file_sort_key)
 
 
 def _filter_paths(basename: str, path: str, is_dir: bool, exclude: Iterable[str]) -> bool:
-    """.gitignore style file filtering."""
+    warnings.warn(
+        "_filter_paths is not used since MkDocs 1.5 and will be removed soon.", DeprecationWarning
+    )
     for item in exclude:
         # Items ending in '/' apply only to directories.
         if item.endswith('/') and not is_dir:
diff --git a/mkdocs/structure/nav.py b/mkdocs/structure/nav.py
index f770f200..4e7b7dc3 100644
--- a/mkdocs/structure/nav.py
+++ b/mkdocs/structure/nav.py
@@ -147,7 +147,8 @@ class Link:
 
 def get_navigation(files: Files, config: MkDocsConfig | Mapping[str, Any]) -> Navigation:
     """Build site navigation from config and files."""
-    nav_config = config['nav'] or nest_paths(f.src_uri for f in files.documentation_pages())
+    documentation_pages = files.documentation_pages()
+    nav_config = config['nav'] or nest_paths(f.src_uri for f in documentation_pages)
     items = _data_to_navigation(nav_config, files, config)
     if not isinstance(items, list):
         items = [items]
@@ -159,19 +160,20 @@ def get_navigation(files: Files, config: MkDocsConfig | Mapping[str, Any]) -> Na
     _add_previous_and_next_links(pages)
     _add_parent_links(items)
 
-    missing_from_config = [file for file in files.documentation_pages() if file.page is None]
+    missing_from_config = []
+    for file in documentation_pages:
+        if file.page is None:
+            # Any documentation files not found in the nav should still have an associated page, so we
+            # create them here. The Page object will automatically be assigned to `file.page` during
+            # its creation (and this is the only way in which these page objects are accessible).
+            Page(None, file, config)
+            if file.inclusion.is_in_nav():
+                missing_from_config.append(file.src_path)
     if missing_from_config:
         log.info(
             'The following pages exist in the docs directory, but are not '
-            'included in the "nav" configuration:\n  - {}'.format(
-                '\n  - '.join(file.src_path for file in missing_from_config)
-            )
+            'included in the "nav" configuration:\n  - ' + '\n  - '.join(missing_from_config)
         )
-        # Any documentation files not found in the nav should still have an associated page, so we
-        # create them here. The Page object will automatically be assigned to `file.page` during
-        # its creation (and this is the only way in which these page objects are accessible).
-        for file in missing_from_config:
-            Page(None, file, config)
 
     links = _get_by_type(items, Link)
     for link in links:
@@ -184,11 +186,10 @@ def get_navigation(files: Files, config: MkDocsConfig | Mapping[str, Any]) -> Na
                 "configuration, which presumably points to an external resource."
             )
         else:
-            msg = (
+            log.warning(
                 f"A relative path to '{link.url}' is included in the 'nav' "
-                "configuration, which is not found in the documentation files"
+                "configuration, which is not found in the documentation files."
             )
-            log.warning(msg)
     return Navigation(items, pages)
 
 
@@ -210,6 +211,11 @@ def _data_to_navigation(data, files: Files, config: MkDocsConfig | Mapping[str,
     title, path = data if isinstance(data, tuple) else (None, data)
     file = files.get_file_from_path(path)
     if file:
+        if file.inclusion.is_excluded():
+            log.info(
+                f"A relative path to '{file.src_path}' is included in the 'nav' "
+                "configuration, but this file is excluded from the built site."
+            )
         return Page(title, file, config)
     return Link(title, path)
 
diff --git a/mkdocs/structure/pages.py b/mkdocs/structure/pages.py
index b7c3dade..4f38f2ea 100644
--- a/mkdocs/structure/pages.py
+++ b/mkdocs/structure/pages.py
@@ -354,6 +354,11 @@ class _RelativePathTreeprocessor(markdown.treeprocessors.Treeprocessor):
                 f"'{target_uri}' which is not found in the documentation files."
             )
             return url
+        if target_file.inclusion.is_excluded():
+            log.info(
+                f"Documentation file '{self.file.src_uri}' contains a link to "
+                f"'{target_uri}' which is excluded from the built site."
+            )
         path = target_file.url_relative_to(self.file)
         components = (scheme, netloc, path, query, fragment)
         return urlunsplit(components)
diff --git a/mkdocs/utils/__init__.py b/mkdocs/utils/__init__.py
index e2319758..27202d08 100644
--- a/mkdocs/utils/__init__.py
+++ b/mkdocs/utils/__init__.py
@@ -436,6 +436,18 @@ def nest_paths(paths):
     return nested
 
 
+class DuplicateFilter:
+    """Avoid logging duplicate messages."""
+
+    def __init__(self) -> None:
+        self.msgs: set[str] = set()
+
+    def __call__(self, record: logging.LogRecord) -> bool:
+        rv = record.msg not in self.msgs
+        self.msgs.add(record.msg)
+        return rv
+
+
 class CountHandler(logging.NullHandler):
     """Counts all logged messages >= level."""
 
diff --git a/pyproject.toml b/pyproject.toml
index ffe843e8..5805a183 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -44,6 +44,7 @@ dependencies = [
     "typing-extensions >=3.10; python_version < '3.8'",
     "packaging >=20.5",
     "mergedeep >=1.3.4",
+    "pathspec >=0.11.1",
     "platformdirs >=2.2.0",
     "colorama >=0.4; platform_system == 'Windows'",
 ]
@@ -64,6 +65,7 @@ min-versions = [
     "typing-extensions ==3.10; python_version < '3.8'",
     "packaging ==20.5",
     "mergedeep ==1.3.4",
+    "pathspec ==0.11.1",
     "platformdirs ==2.2.0",
     "colorama ==0.4; platform_system == 'Windows'",
     "babel ==2.9.0",


Test patch:
diff --git a/mkdocs/tests/base.py b/mkdocs/tests/base.py
index 513e903d..28e3fd3c 100644
--- a/mkdocs/tests/base.py
+++ b/mkdocs/tests/base.py
@@ -34,6 +34,8 @@ def load_config(**cfg) -> MkDocsConfig:
     if 'docs_dir' not in cfg:
         # Point to an actual dir to avoid a 'does not exist' error on validation.
         cfg['docs_dir'] = os.path.join(path_base, 'docs')
+    if 'plugins' not in cfg:
+        cfg['plugins'] = []
     conf = MkDocsConfig(config_file_path=cfg['config_file_path'])
     conf.load_dict(cfg)
 
@@ -123,20 +125,8 @@ class PathAssertionMixin:
             msg = self._formatMessage(None, f"The path '{path}' is not a file that exists")
             raise self.failureException(msg)
 
-    def assertPathNotFile(self, *parts):
-        path = os.path.join(*parts)
-        if os.path.isfile(path):
-            msg = self._formatMessage(None, f"The path '{path}' is a file that exists")
-            raise self.failureException(msg)
-
     def assertPathIsDir(self, *parts):
         path = os.path.join(*parts)
         if not os.path.isdir(path):
             msg = self._formatMessage(None, f"The path '{path}' is not a directory that exists")
             raise self.failureException(msg)
-
-    def assertPathNotDir(self, *parts):
-        path = os.path.join(*parts)
-        if os.path.isfile(path):
-            msg = self._formatMessage(None, f"The path '{path}' is a directory that exists")
-            raise self.failureException(msg)
diff --git a/mkdocs/tests/build_tests.py b/mkdocs/tests/build_tests.py
index fa29bf6b..bc161fce 100644
--- a/mkdocs/tests/build_tests.py
+++ b/mkdocs/tests/build_tests.py
@@ -1,16 +1,25 @@
 #!/usr/bin/env python
+from __future__ import annotations
 
+import contextlib
+import textwrap
 import unittest
+from pathlib import Path
+from typing import TYPE_CHECKING
 from unittest import mock
 
 from mkdocs.commands import build
 from mkdocs.exceptions import PluginError
+from mkdocs.livereload import LiveReloadServer
 from mkdocs.structure.files import File, Files
 from mkdocs.structure.nav import get_navigation
 from mkdocs.structure.pages import Page
 from mkdocs.tests.base import PathAssertionMixin, load_config, tempdir
 from mkdocs.utils import meta
 
+if TYPE_CHECKING:
+    from mkdocs.config.defaults import MkDocsConfig
+
 
 def build_page(title, path, config, md_src=''):
     """Helper which returns a Page object."""
@@ -22,6 +31,13 @@ def build_page(title, path, config, md_src=''):
     return page, files
 
 
+def testing_server(root, builder=lambda: None, mount_path="/"):
+    with mock.patch("socket.socket"):
+        return LiveReloadServer(
+            builder, host="localhost", port=123, root=root, mount_path=mount_path
+        )
+
+
 class BuildTests(PathAssertionMixin, unittest.TestCase):
     def _get_env_with_null_translations(self, config):
         env = config.theme.get_env()
@@ -371,11 +387,13 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
         mock_open.assert_called_once()
 
     @tempdir(files={'index.md': 'page content'})
-    @mock.patch(
-        'mkdocs.plugins.PluginCollection.run_event', side_effect=PluginError('Error message.')
-    )
-    def test_populate_page_read_plugin_error(self, docs_dir, mock_open):
+    def test_populate_page_read_plugin_error(self, docs_dir):
+        def on_page_markdown(*args, **kwargs):
+            raise PluginError('Error message.')
+
         cfg = load_config(docs_dir=docs_dir)
+        cfg.plugins.events['page_markdown'].append(on_page_markdown)
+
         file = File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])
         page = Page('Foo', file, cfg)
         with self.assertLogs('mkdocs') as cm:
@@ -385,13 +403,12 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
             '\n'.join(cm.output),
             "ERROR:mkdocs.commands.build:Error reading page 'index.md':",
         )
-        mock_open.assert_called_once()
 
     # Test build._build_page
 
     @tempdir()
     def test_build_page(self, site_dir):
-        cfg = load_config(site_dir=site_dir, nav=['index.md'], plugins=[])
+        cfg = load_config(site_dir=site_dir, nav=['index.md'])
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -406,7 +423,7 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
     @tempdir()
     @mock.patch('jinja2.environment.Template.render', return_value='')
     def test_build_page_empty(self, site_dir, render_mock):
-        cfg = load_config(site_dir=site_dir, nav=['index.md'], plugins=[])
+        cfg = load_config(site_dir=site_dir, nav=['index.md'])
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -418,14 +435,14 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
             '\n'.join(cm.output),
             "INFO:mkdocs.commands.build:Page skipped: 'index.md'. Generated empty output.",
         )
-        self.assertPathNotFile(site_dir, 'index.html')
+        self.assertPathNotExists(site_dir, 'index.html')
         render_mock.assert_called_once()
 
     @tempdir(files={'index.md': 'page content'})
     @tempdir(files={'index.html': '<p>page content</p>'})
     @mock.patch('mkdocs.utils.write_file')
     def test_build_page_dirty_modified(self, site_dir, docs_dir, mock_write_file):
-        cfg = load_config(docs_dir=docs_dir, site_dir=site_dir, nav=['index.md'], plugins=[])
+        cfg = load_config(docs_dir=docs_dir, site_dir=site_dir, nav=['index.md'])
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -442,7 +459,7 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
     @tempdir(files={'testing.html': '<p>page content</p>'})
     @mock.patch('mkdocs.utils.write_file')
     def test_build_page_dirty_not_modified(self, site_dir, mock_write_file):
-        cfg = load_config(site_dir=site_dir, nav=['testing.md'], plugins=[])
+        cfg = load_config(site_dir=site_dir, nav=['testing.md'])
         fs = [File('testing.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -458,7 +475,7 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
 
     @tempdir()
     def test_build_page_custom_template(self, site_dir):
-        cfg = load_config(site_dir=site_dir, nav=['index.md'], plugins=[])
+        cfg = load_config(site_dir=site_dir, nav=['index.md'])
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -474,7 +491,7 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
     @tempdir()
     @mock.patch('mkdocs.utils.write_file', side_effect=OSError('Error message.'))
     def test_build_page_error(self, site_dir, mock_write_file):
-        cfg = load_config(site_dir=site_dir, nav=['index.md'], plugins=[])
+        cfg = load_config(site_dir=site_dir, nav=['index.md'])
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -493,11 +510,12 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
         mock_write_file.assert_called_once()
 
     @tempdir()
-    @mock.patch(
-        'mkdocs.plugins.PluginCollection.run_event', side_effect=PluginError('Error message.')
-    )
-    def test_build_page_plugin_error(self, site_dir, mock_write_file):
-        cfg = load_config(site_dir=site_dir, nav=['index.md'], plugins=[])
+    def test_build_page_plugin_error(self, site_dir):
+        def on_page_context(*args, **kwargs):
+            raise PluginError('Error message.')
+
+        cfg = load_config(site_dir=site_dir, nav=['index.md'])
+        cfg.plugins.events['page_context'].append(on_page_context)
         fs = [File('index.md', cfg['docs_dir'], cfg['site_dir'], cfg['use_directory_urls'])]
         files = Files(fs)
         nav = get_navigation(files, cfg)
@@ -513,7 +531,6 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
             '\n'.join(cm.output),
             "ERROR:mkdocs.commands.build:Error building page 'index.md':",
         )
-        mock_write_file.assert_called_once()
 
     # Test build.build
 
@@ -560,6 +577,172 @@ class BuildTests(PathAssertionMixin, unittest.TestCase):
         self.assertPathNotExists(site_dir, 'main.html')
         self.assertPathNotExists(site_dir, 'locales')
 
+    @contextlib.contextmanager
+    def _assert_build_logs(self, expected):
+        with self.assertLogs('mkdocs') as cm:
+            yield
+        msgs = [f'{r.levelname}:{r.message}' for r in cm.records]
+        if msgs and msgs[0].startswith('INFO:Cleaning site directory'):
+            del msgs[0]
+        if msgs and msgs[0].startswith('INFO:Building documentation to directory'):
+            del msgs[0]
+        if msgs and msgs[-1].startswith('INFO:Documentation built'):
+            del msgs[-1]
+        self.assertEqual('\n'.join(msgs), textwrap.dedent(expected).strip('\n'))
+
+    @tempdir(
+        files={
+            'test/foo.md': 'page1 content, [bar](bar.md)',
+            'test/bar.md': 'page2 content, [baz](baz.md)',
+            'test/baz.md': 'page3 content, [foo](foo.md)',
+            '.zoo.md': 'page4 content',
+        }
+    )
+    @tempdir()
+    def test_exclude_pages_with_invalid_links(self, site_dir, docs_dir):
+        cfg = load_config(
+            docs_dir=docs_dir,
+            site_dir=site_dir,
+            use_directory_urls=False,
+            exclude_docs='ba*.md',
+        )
+
+        with self.subTest(live_server=None):
+            expected_logs = '''
+                INFO:Documentation file 'test/foo.md' contains a link to 'test/bar.md' which is excluded from the built site.
+            '''
+            with self._assert_build_logs(expected_logs):
+                build.build(cfg)
+            self.assertPathIsFile(site_dir, 'test', 'foo.html')
+            self.assertPathNotExists(site_dir, 'test', 'baz.html')
+            self.assertPathNotExists(site_dir, '.zoo.html')
+
+        server = testing_server(site_dir, mount_path='/documentation/')
+        with self.subTest(live_server=server):
+            expected_logs = '''
+                INFO:Documentation file 'test/bar.md' contains a link to 'test/baz.md' which is excluded from the built site.
+                INFO:Documentation file 'test/foo.md' contains a link to 'test/bar.md' which is excluded from the built site.
+                INFO:The following pages are being built only for the preview but will be excluded from `mkdocs build` per `exclude_docs`:
+                  - http://localhost:123/documentation/.zoo.html
+                  - http://localhost:123/documentation/test/bar.html
+                  - http://localhost:123/documentation/test/baz.html
+            '''
+            with self._assert_build_logs(expected_logs):
+                build.build(cfg, live_server=server)
+
+            foo_path = Path(site_dir, 'test', 'foo.html')
+            self.assertTrue(foo_path.is_file())
+            self.assertNotIn('DRAFT', foo_path.read_text())
+
+            baz_path = Path(site_dir, 'test', 'baz.html')
+            self.assertPathIsFile(baz_path)
+            self.assertIn('DRAFT', baz_path.read_text())
+
+            self.assertPathIsFile(site_dir, '.zoo.html')
+
+    @tempdir(
+        files={
+            'foo/README.md': 'page1 content',
+            'foo/index.md': 'page2 content',
+        }
+    )
+    @tempdir()
+    def test_conflicting_readme_and_index(self, site_dir, docs_dir):
+        cfg = load_config(docs_dir=docs_dir, site_dir=site_dir, use_directory_urls=False)
+
+        for server in None, testing_server(site_dir):
+            with self.subTest(live_server=server):
+                expected_logs = '''
+                    WARNING:Excluding 'foo/README.md' from the site because it conflicts with 'foo/index.md'.
+                '''
+                with self._assert_build_logs(expected_logs):
+                    build.build(cfg, live_server=server)
+
+                index_path = Path(site_dir, 'foo', 'index.html')
+                self.assertPathIsFile(index_path)
+                self.assertRegex(index_path.read_text(), r'page2 content')
+
+    @tempdir(
+        files={
+            'foo/README.md': 'page1 content',
+            'foo/index.md': 'page2 content',
+        }
+    )
+    @tempdir()
+    def test_exclude_readme_and_index(self, site_dir, docs_dir):
+        cfg = load_config(
+            docs_dir=docs_dir, site_dir=site_dir, use_directory_urls=False, exclude_docs='index.md'
+        )
+
+        for server in None, testing_server(site_dir):
+            with self.subTest(live_server=server):
+                with self._assert_build_logs(''):
+                    build.build(cfg, live_server=server)
+
+                index_path = Path(site_dir, 'foo', 'index.html')
+                self.assertPathIsFile(index_path)
+                self.assertRegex(index_path.read_text(), r'page1 content')
+
+    @tempdir(
+        files={
+            'foo.md': 'page1 content',
+            'bar.md': 'page2 content',
+        }
+    )
+    @tempdir()
+    @tempdir()
+    def test_plugins_adding_files_and_interacting(self, tmp_dir, site_dir, docs_dir):
+        def on_files_1(files: Files, config: MkDocsConfig) -> Files:
+            # Plugin 1 generates a file.
+            Path(tmp_dir, 'SUMMARY.md').write_text('foo.md\nbar.md\n')
+            files.append(File('SUMMARY.md', tmp_dir, config.site_dir, config.use_directory_urls))
+            return files
+
+        def on_files_2(files: Files, config: MkDocsConfig) -> None:
+            # Plugin 2 reads that file and uses it to configure the nav.
+            f = files.get_file_from_path('SUMMARY.md')
+            assert f is not None
+            config.nav = Path(f.abs_src_path).read_text().splitlines()
+
+        for server in None, testing_server(site_dir):
+            for exclude in 'full', 'nav', None:
+                with self.subTest(live_server=server, exclude=exclude):
+                    cfg = load_config(
+                        docs_dir=docs_dir,
+                        site_dir=site_dir,
+                        use_directory_urls=False,
+                        exclude_docs='SUMMARY.md' if exclude == 'full' else '',
+                        not_in_nav='SUMMARY.md' if exclude == 'nav' else '',
+                    )
+                    cfg.plugins.events['files'] += [on_files_1, on_files_2]
+
+                    expected_logs = ''
+                    if exclude is None:
+                        expected_logs = '''
+                            INFO:The following pages exist in the docs directory, but are not included in the "nav" configuration:
+                              - SUMMARY.md
+                        '''
+                    if exclude == 'full' and server:
+                        expected_logs = '''
+                            INFO:The following pages are being built only for the preview but will be excluded from `mkdocs build` per `exclude_docs`:
+                              - http://localhost:123/SUMMARY.html
+                        '''
+                    with self._assert_build_logs(expected_logs):
+                        build.build(cfg, live_server=server)
+
+                    foo_path = Path(site_dir, 'foo.html')
+                    self.assertPathIsFile(foo_path)
+                    self.assertRegex(
+                        foo_path.read_text(),
+                        r'href="foo.html"[\s\S]+href="bar.html"',  # Nav order is respected
+                    )
+
+                    summary_path = Path(site_dir, 'SUMMARY.html')
+                    if exclude == 'full' and not server:
+                        self.assertPathNotExists(summary_path)
+                    else:
+                        self.assertPathExists(summary_path)
+
     # Test build.site_directory_contains_stale_files
 
     @tempdir(files=['index.html'])
diff --git a/mkdocs/tests/cli_tests.py b/mkdocs/tests/cli_tests.py
index 67c40d64..ae60991a 100644
--- a/mkdocs/tests/cli_tests.py
+++ b/mkdocs/tests/cli_tests.py
@@ -22,6 +22,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -53,6 +54,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr='0.0.0.0:80',
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -69,6 +71,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=True,
             theme=None,
@@ -87,6 +90,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme='readthedocs',
@@ -105,6 +109,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -123,6 +128,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -139,6 +145,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -155,6 +162,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='no-livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
@@ -165,12 +173,13 @@ class CLITests(unittest.TestCase):
 
     @mock.patch('mkdocs.commands.serve.serve', autospec=True)
     def test_serve_dirtyreload(self, mock_serve):
-        result = self.runner.invoke(cli.cli, ["serve", '--dirtyreload'], catch_exceptions=False)
+        result = self.runner.invoke(cli.cli, ["serve", '--dirty'], catch_exceptions=False)
 
         self.assertEqual(result.exit_code, 0)
         mock_serve.assert_called_once_with(
             dev_addr=None,
-            livereload='dirty',
+            livereload='livereload',
+            build_type='dirty',
             config_file=None,
             strict=None,
             theme=None,
@@ -187,6 +196,7 @@ class CLITests(unittest.TestCase):
         mock_serve.assert_called_once_with(
             dev_addr=None,
             livereload='livereload',
+            build_type=None,
             config_file=None,
             strict=None,
             theme=None,
diff --git a/mkdocs/tests/livereload_tests.py b/mkdocs/tests/livereload_tests.py
index 91586f6b..7959485a 100644
--- a/mkdocs/tests/livereload_tests.py
+++ b/mkdocs/tests/livereload_tests.py
@@ -38,7 +38,6 @@ def testing_server(root, builder=lambda: None, mount_path="/"):
             root=root,
             mount_path=mount_path,
             polling_interval=0.2,
-            bind_and_activate=False,
         )
         server.setup_environ()
     server.observer.start()
diff --git a/mkdocs/tests/plugin_tests.py b/mkdocs/tests/plugin_tests.py
index 64f84741..316bd5c7 100644
--- a/mkdocs/tests/plugin_tests.py
+++ b/mkdocs/tests/plugin_tests.py
@@ -265,27 +265,27 @@ class TestPluginCollection(unittest.TestCase):
                 build_errors.append(error)
 
         cfg = load_config(site_dir=site_dir)
-        cfg['plugins']['errorplugin'] = PluginRaisingError(error_on='pre_page')
+        cfg.plugins['errorplugin'] = PluginRaisingError(error_on='pre_page')
         with self.assertLogs('mkdocs', level='ERROR'):
             self.assertRaises(Abort, build.build, cfg)
 
         cfg = load_config(site_dir=site_dir)
-        cfg['plugins']['errorplugin'] = PluginRaisingError(error_on='page_markdown')
+        cfg.plugins['errorplugin'] = PluginRaisingError(error_on='page_markdown')
         with self.assertLogs('mkdocs', level='ERROR'):
             self.assertRaises(Abort, build.build, cfg)
 
         cfg = load_config(site_dir=site_dir)
-        cfg['plugins']['errorplugin'] = PluginRaisingError(error_on='page_content')
+        cfg.plugins['errorplugin'] = PluginRaisingError(error_on='page_content')
         with self.assertLogs('mkdocs', level='ERROR'):
             self.assertRaises(Abort, build.build, cfg)
 
         cfg = load_config(site_dir=site_dir)
-        cfg['plugins']['errorplugin'] = PluginRaisingError(error_on='post_page')
+        cfg.plugins['errorplugin'] = PluginRaisingError(error_on='post_page')
         with self.assertLogs('mkdocs', level='ERROR'):
             self.assertRaises(ValueError, build.build, cfg)
 
         cfg = load_config(site_dir=site_dir)
-        cfg['plugins']['errorplugin'] = PluginRaisingError(error_on='')
+        cfg.plugins['errorplugin'] = PluginRaisingError(error_on='')
         build.build(cfg)
 
         self.assertEqual(len(build_errors), 4)
diff --git a/mkdocs/tests/search_tests.py b/mkdocs/tests/search_tests.py
index ada861ca..dd8acb24 100644
--- a/mkdocs/tests/search_tests.py
+++ b/mkdocs/tests/search_tests.py
@@ -476,8 +476,8 @@ class SearchIndexTests(unittest.TestCase):
                 self.assertEqual(errors, [])
                 self.assertEqual(warnings, [])
 
-                base_cfg = load_config()
-                base_cfg['plugins']['search'].config.indexing = option
+                base_cfg = load_config(plugins=['search'])
+                base_cfg.plugins['search'].config.indexing = option
 
                 pages = [
                     test_page('Home', 'index.md', base_cfg),
diff --git a/mkdocs/tests/structure/file_tests.py b/mkdocs/tests/structure/file_tests.py
index 2316ab81..95366271 100644
--- a/mkdocs/tests/structure/file_tests.py
+++ b/mkdocs/tests/structure/file_tests.py
@@ -3,7 +3,7 @@ import sys
 import unittest
 from unittest import mock
 
-from mkdocs.structure.files import File, Files, _filter_paths, _sort_files, get_files
+from mkdocs.structure.files import File, Files, _sort_files, get_files
 from mkdocs.tests.base import PathAssertionMixin, load_config, tempdir
 
 
@@ -60,7 +60,7 @@ class TestFiles(PathAssertionMixin, unittest.TestCase):
 
         self.assertEqual(
             _sort_files(['a.md', 'index.md', 'b.md', 'index.html']),
-            ['index.md', 'index.html', 'a.md', 'b.md'],
+            ['index.html', 'index.md', 'a.md', 'b.md'],
         )
 
         self.assertEqual(
@@ -427,34 +427,6 @@ class TestFiles(PathAssertionMixin, unittest.TestCase):
             os.path.normpath(os.path.join(ddir, 'favicon.ico')),
         )
 
-    def test_filter_paths(self):
-        # Root level file
-        self.assertFalse(_filter_paths('foo.md', 'foo.md', False, ['bar.md']))
-        self.assertTrue(_filter_paths('foo.md', 'foo.md', False, ['foo.md']))
-
-        # Nested file
-        self.assertFalse(_filter_paths('foo.md', 'baz/foo.md', False, ['bar.md']))
-        self.assertTrue(_filter_paths('foo.md', 'baz/foo.md', False, ['foo.md']))
-
-        # Wildcard
-        self.assertFalse(_filter_paths('foo.md', 'foo.md', False, ['*.txt']))
-        self.assertTrue(_filter_paths('foo.md', 'foo.md', False, ['*.md']))
-
-        # Root level dir
-        self.assertFalse(_filter_paths('bar', 'bar', True, ['/baz']))
-        self.assertFalse(_filter_paths('bar', 'bar', True, ['/baz/']))
-        self.assertTrue(_filter_paths('bar', 'bar', True, ['/bar']))
-        self.assertTrue(_filter_paths('bar', 'bar', True, ['/bar/']))
-
-        # Nested dir
-        self.assertFalse(_filter_paths('bar', 'foo/bar', True, ['/bar']))
-        self.assertFalse(_filter_paths('bar', 'foo/bar', True, ['/bar/']))
-        self.assertTrue(_filter_paths('bar', 'foo/bar', True, ['bar/']))
-
-        # Files that look like dirs (no extension). Note that `is_dir` is `False`.
-        self.assertFalse(_filter_paths('bar', 'bar', False, ['bar/']))
-        self.assertFalse(_filter_paths('bar', 'foo/bar', False, ['bar/']))
-
     def test_get_relative_url_use_directory_urls(self):
         to_files = [
             'index.md',
@@ -659,10 +631,15 @@ class TestFiles(PathAssertionMixin, unittest.TestCase):
     def test_get_files(self, tdir):
         config = load_config(docs_dir=tdir, extra_css=['bar.css'], extra_javascript=['bar.js'])
         files = get_files(config)
-        expected = ['index.md', 'bar.css', 'bar.html', 'bar.jpg', 'bar.js', 'bar.md', 'readme.md']
         self.assertIsInstance(files, Files)
-        self.assertEqual(len(files), len(expected))
-        self.assertEqual([f.src_path for f in files], expected)
+        self.assertEqual(
+            [f.src_path for f in files if f.inclusion.is_included()],
+            ['index.md', 'bar.css', 'bar.html', 'bar.jpg', 'bar.js', 'bar.md', 'readme.md'],
+        )
+        self.assertEqual(
+            [f.src_path for f in files if f.inclusion.is_excluded()],
+            ['.dotfile', 'templates/foo.html'],
+        )
 
     @tempdir(
         files=[
@@ -673,10 +650,8 @@ class TestFiles(PathAssertionMixin, unittest.TestCase):
     def test_get_files_include_readme_without_index(self, tdir):
         config = load_config(docs_dir=tdir)
         files = get_files(config)
-        expected = ['README.md', 'foo.md']
         self.assertIsInstance(files, Files)
-        self.assertEqual(len(files), len(expected))
-        self.assertEqual([f.src_path for f in files], expected)
+        self.assertEqual([f.src_path for f in files], ['README.md', 'foo.md'])
 
     @tempdir(
         files=[
@@ -691,12 +666,11 @@ class TestFiles(PathAssertionMixin, unittest.TestCase):
             files = get_files(config)
         self.assertRegex(
             '\n'.join(cm.output),
-            r"^WARNING:mkdocs.structure.files:Both index.md and README.md found. Skipping README.md .+$",
+            r"^WARNING:mkdocs.structure.files:"
+            r"Excluding 'README.md' from the site because it conflicts with 'index.md'.$",
         )
-        expected = ['index.md', 'foo.md']
         self.assertIsInstance(files, Files)
-        self.assertEqual(len(files), len(expected))
-        self.assertEqual([f.src_path for f in files], expected)
+        self.assertEqual([f.src_path for f in files], ['index.md', 'foo.md'])
 
     @tempdir()
     @tempdir(files={'test.txt': 'source content'})
diff --git a/mkdocs/tests/structure/nav_tests.py b/mkdocs/tests/structure/nav_tests.py
index b576ba2a..5751429e 100644
--- a/mkdocs/tests/structure/nav_tests.py
+++ b/mkdocs/tests/structure/nav_tests.py
@@ -159,9 +159,9 @@ class SiteNavigationTests(unittest.TestCase):
             cm.output,
             [
                 "WARNING:mkdocs.structure.nav:A relative path to 'missing.html' is included "
-                "in the 'nav' configuration, which is not found in the documentation files",
+                "in the 'nav' configuration, which is not found in the documentation files.",
                 "WARNING:mkdocs.structure.nav:A relative path to 'example.com' is included "
-                "in the 'nav' configuration, which is not found in the documentation files",
+                "in the 'nav' configuration, which is not found in the documentation files.",
             ],
         )
         self.assertEqual(str(site_navigation).strip(), expected)