Repository: lukasgeiter/mkdocs-awesome-pages-plugin

Issue:
Grouping of Rest items is not working
I'm experiencing a `TypeError` error with the following configuration:

**.pages:**
```yaml
order: desc
nav:
  - 2020 Blogs:
    - ... | regex=2020[A-Z0-9a-z_-]+.md
  - 2019 Blogs:
    - ... | regex=2019[A-Z0-9a-z_-]+.md
```

However the following configuration is working:
```yaml
order: desc
nav:
  - ... | regex=2020[A-Z0-9a-z_-]+.md
```

**Error:**
```python
Traceback (most recent call last):
  File "/usr/local/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/mkdocs/__main__.py", line 133, in serve_command
    serve.serve(
  File "/usr/local/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 141, in serve
    config = builder()
  File "/usr/local/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 136, in builder
    build(config, live_server=live_server, dirty=dirty)
  File "/usr/local/lib/python3.8/site-packages/mkdocs/commands/build.py", line 266, in build
    nav = config['plugins'].run_event('nav', nav, config=config, files=files)
  File "/usr/local/lib/python3.8/site-packages/mkdocs/plugins.py", line 94, in run_event
    result = method(item, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/plugin.py", line 45, in on_nav
    return AwesomeNavigation(nav.items, Options(**self.config), config['docs_dir'], explicit_sections).to_mkdocs()
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/navigation.py", line 43, in __init__
    self.meta = NavigationMeta(items, options, docs_dir, explicit_sections)
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/navigation.py", line 183, in __init__
    self.root = Meta.try_load_from(join_paths(root_path, self.options.filename))
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/meta.py", line 131, in try_load_from
    return Meta.load_from(path)
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/meta.py", line 175, in load_from
    nav = [MetaNavItem.from_yaml(item, path) for item in nav]
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/meta.py", line 175, in <listcomp>
    nav = [MetaNavItem.from_yaml(item, path) for item in nav]
  File "/usr/local/lib/python3.8/site-packages/mkdocs_awesome_pages_plugin/meta.py", line 41, in from_yaml
    raise TypeError('Invalid nav item format {type} [{context}]'.format(type=item, context=context))
TypeError: Invalid nav item format {'2020 Blogs': ['... | regex=2020[A-Z0-9a-z_-]+.md']} [/docs/docs/.pages]
```

Also I'm building a custom Docker image for [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/), to include the **awesome-pages-plugin**: - https://squidfunk.github.io/mkdocs-material/getting-started/ - `How to add plugins to the Docker image?`

Discussion:
dimrozakis: Hi @lukasgeiter 

I created this PR because I wanted to reorganize the pages in a more nested hierarchy without changing their paths (and their links). This is working properly for me in the site I'm managing. All preexisting tests pass and I've added a couple of new tests for the new functionality.

Some notes:

1. I'm not very familiar with `mkdocs-awesome-pages-plugin`. This was a quick implementation which I'm sure could be improved upon. I'm not sure if you're happy with the use of nested functions. Or the term "virtual" section. Feel free to make suggestions or direct edits.
2. As previously said, this is working for me and all preexisting and new tests pass, but I haven't covered every possible use case in the tests. For example I don't test nested "virtual" sections. Or what happens if a virtual section has the same name as a real section. I can't 100% guarantee that this works properly in combination with every other feature of the plugin.
3. I haven't documented this feature in the README.
4. I'm not sure but I think that this can also satisfy the use case of https://github.com/lukasgeiter/mkdocs-awesome-pages-plugin#combine-custom-navigation--file-structure with a top level .pages instead of defining the nav section in mkdocs.yml.

Looking forward to your feedback. And thanks for creating this plugin.
lukasgeiter: Hey @dimrozakis 

Thanks for the PR! Right now I'm a bit short on time, but I'll take a closer look in the coming days.
dimrozakis: @lukasgeiter thanks for the review. I've addressed all your comments. Any thoughts on points 2,3,4 from https://github.com/lukasgeiter/mkdocs-awesome-pages-plugin/pull/59#issuecomment-1021559894?

Patch:
diff --git a/mkdocs_awesome_pages_plugin/meta.py b/mkdocs_awesome_pages_plugin/meta.py
index baa4acf..d04c62c 100644
--- a/mkdocs_awesome_pages_plugin/meta.py
+++ b/mkdocs_awesome_pages_plugin/meta.py
@@ -14,7 +14,7 @@ class DuplicateRestItemError(Exception):
 
 
 class MetaNavItem:
-    def __init__(self, value: str, title: Optional[str] = None):
+    def __init__(self, value: Union[str, List["MetaNavItem"]], title: Optional[str] = None):
         self.value = value
         self.title = title
 
@@ -34,8 +34,11 @@ class MetaNavItem:
 
         if isinstance(item, dict) and len(item) == 1:
             (title, value) = list(item.items())[0]
-            if isinstance(value, str) and isinstance(title, str):
-                return MetaNavItem(value, title)
+            if isinstance(title, str):
+                if isinstance(value, str):
+                    return MetaNavItem(value, title)
+                elif isinstance(value, list):
+                    return MetaNavItem([MetaNavItem.from_yaml(it, context) for it in value], title)
 
         raise TypeError("Invalid nav item format {type} [{context}]".format(type=item, context=context))
 
diff --git a/mkdocs_awesome_pages_plugin/navigation.py b/mkdocs_awesome_pages_plugin/navigation.py
index 77a530b..4fa9ea4 100644
--- a/mkdocs_awesome_pages_plugin/navigation.py
+++ b/mkdocs_awesome_pages_plugin/navigation.py
@@ -11,7 +11,7 @@ from mkdocs.structure.nav import (
 )
 from mkdocs.structure.pages import Page
 
-from .meta import Meta, MetaNavRestItem, RestItemList
+from .meta import Meta, MetaNavItem, MetaNavRestItem, RestItemList
 from .options import Options
 from .utils import dirname, basename, join_paths
 
@@ -37,6 +37,10 @@ class HideInRootHasNoEffect(Warning):
         )
 
 
+class VirtualSection(Section):
+    pass
+
+
 class AwesomeNavigation:
     def __init__(
         self,
@@ -65,7 +69,7 @@ class AwesomeNavigation:
         result = []
 
         for item in children:
-            if isinstance(item, Section):
+            if isinstance(item, Section) and not isinstance(item, VirtualSection):
                 item = self._process_section(item, collapse)
                 if item is None:
                     continue
@@ -86,19 +90,25 @@ class AwesomeNavigation:
 
         items_by_basename = {basename(self._get_item_path(item)): item for item in items}
 
-        result = []
+        used_items = []
         rest_items = RestItemList()
 
-        for index, meta_item in enumerate(meta.nav):
-            if isinstance(meta_item, MetaNavRestItem):
-                rest_items.append(meta_item)
-                result.append(meta_item)
-            else:
-                if meta_item.value in items_by_basename:
+        def _make_nav_rec(meta_nav: List[MetaNavItem]) -> List[Union[NavigationItem, MetaNavRestItem]]:
+            result = []
+            for meta_item in meta_nav:
+                if isinstance(meta_item, MetaNavRestItem):
+                    rest_items.append(meta_item)
+                    result.append(meta_item)
+
+                elif isinstance(meta_item.value, list):
+                    result.append(VirtualSection(meta_item.title, children=_make_nav_rec(meta_item.value)))
+
+                elif meta_item.value in items_by_basename:
                     item = items_by_basename[meta_item.value]
                     if meta_item.title is not None:
                         item.title = meta_item.title
                     result.append(item)
+                    used_items.append(item)
 
                 elif meta_item.title is not None:
                     result.append(Link(meta_item.title, meta_item.value))
@@ -109,21 +119,30 @@ class AwesomeNavigation:
                         raise warning
                     else:
                         warnings.warn(warning)
+            return result
+
+        result = _make_nav_rec(meta.nav)
 
         if rest_items:
             rest = {rest_item: [] for rest_item in rest_items}
 
             for item in items:
-                if item not in result:
+                if item not in used_items:
                     path = basename(self._get_item_path(item))
                     for rest_item in rest_items:
                         if rest_item.matches(path):
                             rest[rest_item].append(item)
                             break
 
-            for index, item in enumerate(result):
-                if isinstance(item, MetaNavRestItem):
-                    result[index : index + 1] = rest[item]
+            def _expand_rest_rec(result: List[Union[NavigationItem, MetaNavRestItem]]):
+                for index, item in enumerate(result):
+                    if isinstance(item, MetaNavRestItem):
+                        result[index : index + 1] = rest[item]
+                    elif isinstance(item, Section):
+                        if item.children:
+                            _expand_rest_rec(item.children)
+
+            _expand_rest_rec(result)
 
         return result
 


Test patch:
diff --git a/mkdocs_awesome_pages_plugin/tests/e2e/test_nav.py b/mkdocs_awesome_pages_plugin/tests/e2e/test_nav.py
index 8721d14..cd5f6ad 100644
--- a/mkdocs_awesome_pages_plugin/tests/e2e/test_nav.py
+++ b/mkdocs_awesome_pages_plugin/tests/e2e/test_nav.py
@@ -423,3 +423,49 @@ class TestNav(E2ETestCase):
     def test_not_found_not_strict(self):
         with self.assertWarns(NavEntryNotFound):
             self.mkdocs(self.createConfig(strict=False), [self.pagesFile(nav=["1.md", "..."])])
+
+    def test_virtual_section(self):
+        navigation = self.mkdocs(
+            self.config,
+            [
+                "a.md",
+                "b.md",
+                (
+                    "x",
+                    [
+                        "1a.md",
+                        "1b.md",
+                        "2a.md",
+                        "2b.md",
+                        "3.md",
+                        "4.md",
+                        self.pagesFile(
+                            nav=[
+                                "4.md",
+                                {"BB": ["... | *b.md"]},
+                                "...",
+                                {"AA": ["... | *a.md"]},
+                            ]
+                        ),
+                    ],
+                ),
+                self.pagesFile(nav=["...", "b.md"]),
+            ],
+        )
+
+        self.assertEqual(
+            navigation,
+            [
+                ("A", "/a"),
+                (
+                    "X",
+                    [
+                        ("4", "/x/4"),
+                        ("BB", [("1b", "/x/1b"), ("2b", "/x/2b")]),
+                        ("3", "/x/3"),
+                        ("AA", [("1a", "/x/1a"), ("2a", "/x/2a")]),
+                    ],
+                ),
+                ("B", "/b"),
+            ],
+        )
diff --git a/mkdocs_awesome_pages_plugin/tests/navigation/test_nav.py b/mkdocs_awesome_pages_plugin/tests/navigation/test_nav.py
index a8c244d..1179eea 100644
--- a/mkdocs_awesome_pages_plugin/tests/navigation/test_nav.py
+++ b/mkdocs_awesome_pages_plugin/tests/navigation/test_nav.py
@@ -388,3 +388,39 @@ class TestNav(NavigationTestCase):
                 ],
                 strict=False,
             )
+
+    def test_virtual_section(self):
+        navigation = self.createAwesomeNavigation(
+            [
+                self.page("1"),
+                self.page("2a"),
+                self.page("2b"),
+                self.page("2c"),
+                self.page("3a"),
+                self.page("3b"),
+                self.page("3c"),
+                self.page("4"),
+                Meta(
+                    nav=[
+                        MetaNavItem("4.md"),
+                        MetaNavItem([MetaNavItem("2a.md"), MetaNavItem("3a.md")], "AA"),
+                        MetaNavRestItem("..."),
+                        MetaNavItem([MetaNavItem("2b.md"), MetaNavItem("3b.md")], "BB"),
+                        MetaNavItem("1.md"),
+                    ]
+                ),
+            ]
+        )
+
+        self.assertNavigationEqual(
+            navigation.items,
+            [
+                self.page("4"),
+                self.section("AA", [self.page("2a"), self.page("3a")]),
+                self.page("2c"),
+                self.page("3c"),
+                self.section("BB", [self.page("2b"), self.page("3b")]),
+                self.page("1"),
+            ],
+        )
+        self.assertValidNavigation(navigation.to_mkdocs())