Repository: django-cms/django-cms

Issue:
MPTT cache_tree_children error on filtered pages queryset
CMSChangeList querysets now get sorted by tree_id and lft. This solves issues like #835 (sorting with NULL parents was inconsistent across database engines), but the same MPTT error occurs if the queryset filters out pages from the higher levels, especially from the first trees.

For instance, this can happen when the user applies filters on templates or status, or if the first visible page according to her page permissions is on a higher level than any of the others.

This is because `mptt.templatetags.mptt_tags.cache_tree_children` assumes the first node in the queryset to be the root.

I fixed this by trapping the ValueError exception MPTT raises in `CMSChangeList.set_items` to give up caching:

```
    try:
        mptt_tags.cache_tree_children(pages)
    except ValueError:
        #restore initial list
        pages = all_pages[:]
```

I also had to change the subsequent call to page.get_children to

```
        children = list(page.get_children())
```

because negative indexing on children (a queryset, without caching) was not permitted.

I'm forking 2.2RC2 and will be submitting this patch soon.



Discussion:
I was able to reproduce this issue. kind of worrying...

Not entirely sure I agree with the proposed fix though, investigating...

I see three possible solutions (please contribute others if you can think of them):
1. Use your proposed fix (which is suboptimal for performance reasons)
2. Disable the ability to filter pages on changelist
3. Find a way to do 'cache_tree_children' regardless of tree state.

Not 100% sure what way to go yet.

1 is problematic due to perfomance reasons, 2 will probably get some negative reaction from people using that feature and 3 is the hardest to implement.

I agree: it's more a workaround than a fix, but I was in a hurry because I needed people to go on with the tests, and it looked like disabling the caching would only impact performance.
(By the way, I haven't submitted anything yet because I didn't have time to write proper test.)

I'm afraid using cache_tree_children with filtered querysets could also produce unexpected results without actually causing any errors.

Consider a simple structure with two trees with one subnode each, like this:

```
1
    1.1
2
    2.1
```

If node 2 is excluded from the queryset (permissions, changelist filters, whatever) node 2.1 ends up as a child of node 1. This also makes my workaround worthless.

So it looks like cache_tree_children is perfectly safe only if you're dealing with complete subtrees.

I might have missed something. I will try to look into this tomorrow, and hope to come up with something useful.

well i'd say it's not `cache_tree_chidren`s fault, but how the hell should we _display_ a partial tree?

That's what makes me kinda favor route 2...

I always assumed users with partial permissions would only see the hierarchy whenever possible. In fact, they already do: if a user is only allowed to edit the home page and a subtree starting from a subnode of a different tree, she will see both nodes as roots in the changelist.

Actually, I don't think there's anything wrong with flattening the structure whenever a changelist filter is applied, as the application already does: if I'm looking for pages awaiting approval, or using a specific template, I sort of expect a list of pages, not a tree. The only problem in this case is that mptt _does_ expect a proper tree. (As a side note, if the result is a simple list of pages, with no hierarchy, we can probably skip all the children processing)

The real issue, here, is tree disruption caused by user permissions, and IMO route 2 is not a solution, because it only covers changelist filters.

One possibility would be always showing full subtrees and only allow users to enter the pages they can actually edit, i.e. always retrieving and displaying the full ancestry of editable nodes, leaving unmodifiable parents inactive and hiding completely inaccessible subtrees. This would probably be a nightmare performance-wise, unless we find a way to leverage the cache globally.

Or we could leave it as it is (i.e. tree fragments and isolated pages on the same level as the root), but we need to fix the way parent-children relationships are cached to prevent a misrepresentation.


Resolving patch:
diff --git a/cms/admin/change_list.py b/cms/admin/change_list.py
--- a/cms/admin/change_list.py
+++ b/cms/admin/change_list.py
@@ -8,12 +8,42 @@
 from django.contrib.admin.views.main import ChangeList, ALL_VAR, IS_POPUP_VAR, \
     ORDER_TYPE_VAR, ORDER_VAR, SEARCH_VAR
 from django.contrib.sites.models import Site
-from mptt.templatetags import mptt_tags
-
 
 
 COPY_VAR = "copy"
 
+
+def cache_tree_children(queryset):
+    """
+    For all items in the queryset, set the '_cached_children' attribute to a
+    list. This attribute is in turn used by the 'get_children' method on the
+    item, which would otherwise (if '_cached_children' is not set) cause a 
+    database query.
+    
+    The queryset MUST BE ORDERED BY 'lft', 'tree_id'! Otherwise this function
+    will raise a ValueError.
+    """
+    parents_dict = {}
+    lastleft = -1 # integrity check
+    lasttree = -1 # integrity check
+    for obj in queryset:
+        parents_dict[obj.pk] = obj
+        if obj.tree_id == lasttree and obj.lft < lastleft: # integrity check
+                raise ValueError('Objects passed in the wrong order, must be ordered by the mptt left attribute and tree id')
+        lastleft = obj.lft # integrity check
+        lasttree = obj.tree_id # integrity check
+        # set the '_cached_children' attribute
+        obj._cached_children = []
+        # get the parent of this object (if available) via parent_id
+        parent = parents_dict.get(obj.parent_id, None)
+        if parent:
+            # if there is a parent, append the current object to the _cached_children
+            # list of the parent. Since the objects are ordered by lft, tree_id
+            # the _cached_children attribute will always have been set by this
+            # function already.
+            parent._cached_children.append(obj)
+
+
 class CMSChangeList(ChangeList):
     '''
     Renders a Changelist - In our case it looks like a tree - it's the list of
@@ -50,7 +80,6 @@ def get_query_set(self, request=None):
                 self.root_query_set = self.root_query_set.filter(pk__in=permissions)
             self.real_queryset = True
             qs = qs.filter(site=self._current_site)
-        qs = qs.order_by('tree_id',  'lft')
         return qs
     
     def is_filtered(self):
@@ -118,12 +147,10 @@ def set_items(self, request):
         except NoHomeFound:
             home_pk = 0
             
-        # Make the pages a tree (using MPTT awesomeness).
-        # This is normally a tag filter, but it's really nice in our case too:
-        # It caches children for every page in the list we pass it, so no
-        # further queries are needed.
-
-        mptt_tags.cache_tree_children(pages)
+        # Unfortunately we cannot use the MPTT builtin code for pre-caching
+        # the children here, because MPTT expects the tree to be 'complete'
+        # and otherwise complaints about 'invalid item order'
+        cache_tree_children(pages)
         
         for page in pages:
            
@@ -177,10 +204,19 @@ def set_items(self, request):
                     page.ancestors_ascending = []
                 page.home_pk_cache = home_pk
             
-            # childrens is the reverse accessor for the parent foreign key.
-            # We want to set it so the JSTree can display it nicely in admin.
-            # "childrens" is the fully cached version of children.
-            page.childrens = children
+            # Because 'children' is the reverse-FK accessor for the 'parent'
+            # FK from Page->Page, we have to use wrong English here and set
+            # an attribute called 'childrens'. We are aware that this is WRONG
+            # but what should we do?
+            
+            # If the queryset is filtered, do NOT set the 'childrens' attribute
+            # since *ALL* pages will be in the 'root_pages' list and therefore
+            # be displayed. (If the queryset is filtered, the result is not a
+            # tree but rather a flat list).
+            if self.is_filtered():
+                page.childrens = []
+            else:
+                page.childrens = children
         
         # TODO: OPTIMIZE!!
         titles = Title.objects.filter(page__in=ids)


Test patch: