Repository: wagtail/wagtail

Issue:
PathOverflow: Path Overflow from: '**********ZZZZ'
### Issue Summary

Creating a new page from the administration area raises a PathOverflow error.

```
PathOverflow: Path Overflow from: '000100020003ZZZZ'
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "wagtail/admin/urls/__init__.py", line 102, in wrapper
    return view_func(request, *args, **kwargs)
  File "wagtail/admin/decorators.py", line 34, in decorated_view
    return view_func(request, *args, **kwargs)
  File "wagtail/admin/views/pages.py", line 224, in create
    parent_page.add_child(instance=page)
  File "treebeard/mp_tree.py", line 1013, in add_child
    return MP_AddChildHandler(self, **kwargs).process()
  File "treebeard/mp_tree.py", line 387, in process
    newobj.path = self.node.get_last_child()._inc_path()
  File "treebeard/mp_tree.py", line 1114, in _inc_path
    raise PathOverflow(_("Path Overflow from: '%s'" % (self.path, )))
```

The only information I could find potentially regarding the issue was on SO at the following link: https://stackoverflow.com/questions/54166821/wagtail-pathoverflow-on-adding-new-child-page

There are only a handful of pages, so I'm not sure that particular issue applies though.

### Steps to Reproduce

I haven't been able to reproduce this myself, however a client has been seeing this behavior, which at the time I've just resolved by modifying the ZZZZ at the end to be numeric. I'm in the process of gathering some steps they may have taken to replicate it from this end (where they tried to add the page from)

* I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: no


### Technical details

* Python version: 3.6.8
* Django version: 2.1.4
* Wagtail version: 2.4
* Browser version: N/A



Discussion:
Hi @simoncoulton - thanks for the report. As mentioned on the Stack Overflow question, this is something that can theoretically happen after a very large number of page creations / moves, but not something we'd expect to see in normal usage. Unfortunately without details of how to reproduce the issue, there's no way of knowing whether this is a Wagtail bug or something else (e.g. an import script going haywire) and no real way that we can follow this up. I'd be happy to re-open this if you're able to find steps to reproduce this, though.
Hi, @simoncoulton. I can confirm this is a bug. I also have exactly the same problem with my production website. This is a very small website with only about 10 pages. And I don't think we would easily overflow Material Path tree's subpage path (this is 36^4=1.68million pages) with one editor working on it manually, no matter how hard she works.

Being familiar with Material Path tree in treebeard, I ventured to look at wagtailcore_page, and noticed the culprit page with MP path, basically I have these 5 page paths
````
...
000100020006  <-- blog root
0001000200060001
0001000200060002
0001000200060003
000100020006ZZZZ <--- culprit
...
````

Of course I won't be able to add any new subpage to blog root page as the culprit page already has the largest subpage path sequence number possible. My guess is some bug in wagtail's code probably shoots the path sequence number up after some exception. When I search "treebeard django path overflow" on google, actually this page is the first one. Therefore it's unlikely the problem of treebeard package itself.

@gasman hope this is enough info for you to re-open the case and trace back the bug in wagtail code.

Band-Aid fix: For anybody seeking a temporary solution, the fix is simple, I just change 000100020006ZZZZ to 0001000200060004, with 0004 being the next sequence number in MP tree's dictionary after 0003.

Disclaimer: If anybody seeing this wants to try the fix,  please bear in mind directly modifying the database is always a very bad and very risky practice. You'd better know what you're doing, especially if you have more subpages below ****ZZZZ. You'll need to modify all subpages paths as well.
@dany-nonstop Unfortunately that doesn't really give us any new information - all we know is that there is some rare sequence of events resulting in the ZZZZ path. Without a consistent way of reproducing this, there's no way to determine whether that's caused by Treebeard, Wagtail, your own code, or a third-party package.
Hi, @gasman I've been trying to figure out where the zzzz comes from, and am able to 100% reproduce the error.

How to reproduce: 
- in code: create a page that subclasses `Page`, and make sure it has some content that must be validated, let's call it `Subpage` class
- in admin UI: create a page of `Subpage` class under a parent page who has at least a child page already, and make sure to fill in some content, but cannot be validated, then click preview (of course if says the page is not valid), then fill the content to make it valid, then click preview button again (again this time it's valid and the preview is displayed correctly)
- in database: you may verify now this page with ZZZZ path appears, and all of the time fields are still empty, still in draft mode.
- back to admin UI: now you cannot save the page due to validation error, the only thing you can do at this moment is to abandon the page -- but it would appear as in the list of the parent page's children page list, with "Updated" column as "none". 
obviously this page should not appear here, and should not have the Status as "Live" because I didn't even save it! And data is actually corrupted here. (I removed other pages here in the screenshot)
![image](https://user-images.githubusercontent.com/56896461/93150482-1da02b80-f6ae-11ea-86e1-c2fd8ffaf104.png)
- in database: you may see that the number of child pages do not match the number of children in `numchild` column any more. Although you can delete the child page, `numchild` would always be one less than the actual number.
![image](https://user-images.githubusercontent.com/56896461/93151590-3827d400-f6b1-11ea-93b6-f66adb662743.png)

and when this page cannot be validated and thereafter abandoned, the record would stay there and get the database corrupted. The only way to fix the problem is to manually delete this row.


OK, coming back to the root of the problem, I think it's this line in `PreviewOnCreate` class. In my understanding, it tries to create the Page object in the database for preview, and takes the last available path in the path range, which ends with ZZZZ normally:
https://github.com/wagtail/wagtail/blob/882f8f3cf8ddd79c30e611a48882b309e90dad0c/wagtail/admin/views/pages.py#L664

I'm under the impression that this practice of using the last available path is very fragile. Suppose two people working on the same website at the same time and wants to preview their respective pages, there is a chance that either one of them will fail, or one will overwrite the other at this zzzz path. Do you think it is probably cleaner to abandon this "preview without save" idea. Ask users to always save the preview as the draft before allowing them to preview. So even when they work in parallel, these pages will have different paths and won't collide. Hopefully this bug will disappear as well.

@dany-nonstop I can't reproduce this on a vanilla Wagtail 2.10.1 install. In your steps above, the second step (first failing to preview and then successfully doing so) correctly does not put anything in the database page table. Looking at [the code](https://github.com/wagtail/wagtail/blob/master/wagtail/admin/views/pages/preview.py) for `PreviewOnCreate`, although Wagtail does create a new page instance to render the preview, it never gets saved to the DB.

I also tested against 2.4, since that's the version you reported, and can't reproduce there either. If I save after previewing, the page gets a valid path.

Can you reproduce your steps above on a totally new instance of Wagtail? Do you possibly have any other hooks or custom code that might be running on preview and saving the page to the DB?

Thanks for investigating this further, @dany-nonstop and @chosak!

Even if this does turn out to be a result of custom code, I think setting the path to ZZZZ is probably enough of a "foot gun" that it's worth avoiding that setup in Wagtail's code. On further digging, it looks like the older implementation of preview (prior to #3383) actually had some code for setting the temporary path to current-maximum-plus-one instead:

https://github.com/wagtail/wagtail/blob/0d1c277e85c1ba3b73742811b9dd5d75a2493633/wagtail/wagtailadmin/views/pages.py#L609-L621

...however, I evidently failed to notice that this was being overwritten a few lines further down, and consequently this code was dropped as redundant in #3383. I think we should reinstate that version.
@chosak thanks for spending time investigating this issue. I'll try to make another clean install and have minimum custom code to try it again and report back to you.
@gasman I think this code you quoted is a much more reliable way in handling preview. It's highly appreciated that the good old implementation can be restored if not better. 
I would still insist that a more fundamental approach would be to eliminate preview w/o save function altogether. The comment in L610-L613 already stated the potential risk. The code is really duplicating some of treebeard's internal mechanism. Theoretically any uncaught exceptions in the future (be it in Treebeard, Wagtail, or 3rd party code) could potentially leave the database in some inconsistent state, hard to clean for inexperienced users / developers. And bugs of this type is hard to reproduce and debug, like we're experiencing right now.
Besides cleaner code (less code, and especially avoid calling any treebeard internal reserved function starting with underscore), actually I would say that no-preview-before-saving-draft approach would be more beneficial to the end users as well. It could help them avoid potential data loss, when previewed but forget to save.
> I would still insist that a more fundamental approach would be to eliminate preview w/o save function altogether

FWIW I would miss the ability to preview without saving. This is something I do quite often, e.g. when I'm demonstrating Wagtail, or testing new StreamField blocks. Without this feature, I'd have to remember to delete draft pages.
@tomdyson I kinda of agree with you that preview without saving feature is beautiful and convenient. But the potential data corruption is a little risky IMHO. Once it really happens nobody would think it's fun to dig into the database to have it cleaned up. For the long term stability of your production system, I feel this small feature loss is worth of it -- unless we find a way to not write anything to the database to preview.
I don't see any reason to believe that previewing unsaved pages is inherently more risky than any other feature of Wagtail. Yes, there was an error here (with an as-yet-undetermined root cause that may or may not be within Wagtail), but that could have happened anywhere.
I also can't reproduce. Same as @chosak . :/
- Add Page and leave a field (required char) empty.
- Hit preview, gives form validation error.
- Fill the char field, hit preview, preview opens successfully.
- There is no entry in db.

After publish the page is correctly saved, I checked the db.

Resolving patch:
diff --git a/wagtail/admin/views/pages/preview.py b/wagtail/admin/views/pages/preview.py
--- a/wagtail/admin/views/pages/preview.py
+++ b/wagtail/admin/views/pages/preview.py
@@ -113,9 +113,15 @@ def get_page(self):
         # hopefully enable tree traversal operations
         # to at least partially work.
         page.depth = parent_page.depth + 1
-        # Puts the page at the maximum possible path
+        # Puts the page at the next available path
         # for a child of `parent_page`.
-        page.path = Page._get_children_path_interval(parent_page.path)[1]
+        if parent_page.is_leaf():
+            # set the path as the first child of parent_page
+            page.path = page._get_path(parent_page.path, page.depth, 1)
+        else:
+            # add the new page after the last child of parent_page
+            page.path = parent_page.get_last_child()._inc_path()
+
         return page
 
     def get_form(self, page, query_dict):


Test patch: