Repository: django-cms/django-cms

Issue:
Fixture loading in Postgres
Get the following error when loading json fixtures with Postgres and django 1.3.1

IntegrityError: duplicate key value violates unique constraint "cms_placeholder_pkey"

Forked repository and created test case for this on https://github.com/mthornhill/django-cms
to recreate
1. clone directory
git clone https://mthornhill@github.com/mthornhill/django-cms.git
2. make a virtual environment
cd django-cms
virtualenv . --no-site-packages
3. run FixtureTestCase
./runtests.sh -d 13 --rebuild-env FixtureTestCase



Discussion:
I got the same issue.

+++ Vote up

It seems, that this is caused by the signal receiver `cms.signals.update_placeholder`

I guess placeholders are already created from the fixture but the call to `Page.rescan_placeholder()` tries that again.

I found a workaround. I made my own loaddata command that disconnects the signal first:

``` python
from django.core.management.commands import loaddata
from django.db.models import signals
from cms.signals import update_placeholders
from cms.models import Page

class Command(loaddata.Command):

    def handle(self, *fixture_labels, **options):
        signals.post_save.disconnect(update_placeholders, sender=Page)

        loaddata.Command.handle(self, *fixture_labels, **options)

        signals.post_save.connect(update_placeholders, sender=Page)
```

yes the problem are the signals, which I'd claim is an upstream (django) bug and therefore I'll close this

Hm, I wouldn't agree.

Firing `post_save` signals for models that are loaded through `loaddata` cannot be said to be always wrong.

I claim that **usually** you `loaddata` that you got via `dumpdata`, meaning the signals already got fired.

That is true, but a `post_save` doesn't necessarily access the database. It might create files for example or send emails. Both might be wanted also for `loaddata`.

still, what's the fix in our case? check sys.argv in all post_save signals? How about django telling us in post_save that it was created from fixtures loading?

Hm yes, or Django could fire a `pre_loaddata` signal.

But the only receiver that causes problems is the `update_placeholders()`.

That means only this would have to check for `sys.argv`.

I came across this issue today and found a Django issue about it: https://code.djangoproject.com/ticket/8399

From the discussion there it sounds reasonable to look for a "raw" argument in the signal handler (but I can't find a pointer in the Django docs ATM).

The raw argument is documented here:
https://docs.djangoproject.com/en/dev/ref/signals/#pre-save
It is available for pre and post save signals. It seems it could solve this issue.

@ojii Has there been any movement on this issue? is it still classed as a django bug or is it a cms bug?

No, there hasn't been any movement on this. For me this doesn't have any priority. If someone can come up with practical solutions, I'd be happy to listen though.

The trick looking for raw works for me - just decorate all the listener functions with it:

```
from functools import wraps
from django.dispatch import receiver

def disable_for_loaddata(signal_handler):
    """Decorator that turns off signal handlers when loading fixture data.
    """

    @wraps(signal_handler)
    def wrapper(*args, **kwargs):
        if kwargs['raw']:
            return
        signal_handler(*args, **kwargs)
    return wrapper


@receiver(post_save, sender=YOURMODEL)
@disable_for_loaddata
def yoursignalhandler(sender, instance, created, **kwargs):
    # do your thing 
```

Can I add my voice to those who'd like this fixed. I'm pretty sure this has been a thread raised on django trac and closed wontfix, so fixing django_cms is the only realistic option for those of us loading fixtures into postgres.

+1 That looks like a great solution.

How about a 3rd party Django app that does nothing but load fixtures without firing signals?

EDIT: Also, the "raw" kwarg does not necessarily indicate fixture loading.

@ojii Really? When else does the raw kwarg get passed in? I might need to think about that...

A third party app might work (I presume you mean just to provide a wrapper for loaddata) but it feels a bit clunky. Why must all postgres users maintain an additional dependency? If a load data wrapper is preferred why not provide it within django_cms itself? (Although I do think decorating update_placeholders() as described would be neater).

CMS should strive to do the right thing. regardless of whether raw=True is only presented by the fixture loader, It's meaning is clear.

> `raw` -- A boolean; True if the model is saved exactly as presented (i.e. when loading a fixture). One should not query/modify other records in the database as the database might not be in a consistent state yet.

In this case stopping the signal handler from firing if it touches the db. is exactly the right thing to be doing.

As far as a custom fixture loader/dumper, I'm working on a specific cms loader/dumper that does single pages or page trees. perhaps this will be helpful in the future, as long as raw=True is respected by cms.

@twoolie  I totally agree... interesting that passing raw=True to the receiver doesn't seem to be documented anywhere. Have you actually tested doing it this way?

@bigfudge you're right, raw=True arg to signal connect is not correct. I Forgot I was using a shim around @receiver that allows filtering on arbitrary fields. I've edited my previous comment to reflect that.

I'm having this issue as well.
It's a pain in the ass to be honest, the loaddata feature is really useful when working in team.

@ojii what about adding a documentation note about this so people know the cause and potential workarounds without having to dig to github ? 

Its honestly embarrassing that this is still open given the ease of the fix and the django docs that state that it's the right thing to do. I'll submit a pull and hopefully that will end postgres users pain.


Resolving patch:
diff --git a/cms/signals.py b/cms/signals.py
--- a/cms/signals.py
+++ b/cms/signals.py
@@ -196,7 +196,8 @@ def post_save_page(instance, **kwargs):
 
 
 def update_placeholders(instance, **kwargs):
-    instance.rescan_placeholders()
+    if not kwargs.get('raw'):
+        instance.rescan_placeholders()
 
 
 def invalidate_menu_cache(instance, **kwargs):


Test patch:
diff --git a/cms/tests/__init__.py b/cms/tests/__init__.py
--- a/cms/tests/__init__.py
+++ b/cms/tests/__init__.py
@@ -31,8 +31,9 @@
 from cms.tests.urlutils import *
 from cms.tests.views import *
 from cms.tests.management import *
+from cms.tests.fixture_loading import *
 from cms.tests.menu_page_viewperm import *
 from cms.tests.menu_page_viewperm_staff import *
 from cms.tests.nested_plugins import *
 from cms.tests.check import *
-from cms.tests.no_i18n import *
\ No newline at end of file
+from cms.tests.no_i18n import *
diff --git a/cms/tests/fixture_loading.py b/cms/tests/fixture_loading.py
new file mode 100644
--- /dev/null
+++ b/cms/tests/fixture_loading.py
@@ -0,0 +1,37 @@
+# -*- coding: utf-8 -*-
+import tempfile
+import codecs
+
+try:
+    from cStringIO import StringIO
+except:
+    from io import StringIO
+
+from django.core.management import call_command
+
+from cms.test_utils.fixtures.navextenders import NavextendersFixture
+from cms.test_utils.testcases import SettingsOverrideTestCase
+from cms.models import Page
+
+
+class FixtureTestCase(NavextendersFixture, SettingsOverrideTestCase):
+
+    def test_fixture_load(self):
+        """
+        This test dumps a live set of pages, cleanup the database and load it
+        again.
+        This makes fixtures unnecessary and it's easier to maintain.
+        """
+        output = StringIO()
+        dump = tempfile.mkstemp(".json")
+        call_command('dumpdata', 'cms', indent=3, stdout=output)
+        Page.objects.all().delete()
+        output.seek(0)
+        with codecs.open(dump[1], 'w', 'utf-8') as dumpfile:
+            dumpfile.write(output.read())
+
+        self.assertEqual(0, Page.objects.count())
+        # Transaction disable, otherwise the connection it the test would be
+        # isolated from the data loaded in the different command connection
+        call_command('loaddata', dump[1], commit=False, stdout=output)
+        self.assertEqual(10, Page.objects.count())