Repository: learningequality/kolibri

Issue:
When using postgres, unable to import content nodes with long field values
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->

When running Kolibri with a Postgres DB backend, trying to import content that contains a `ContentNode` with `description` or `license_description` longer than 400 chars throws an error (see below).

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->

When we run with a SQLite backend, it [happily ignores](https://chrisbartos.com/articles/sqlite-doesnt-care-about-your-max_length/) the violation of the `max_length` attribute on these fields. Postgres is more strict (as it should be).

Expectation would be one of the following:
- Don't allow published databases to have strings > 400 char in these fields to start with. The ship's probably already sailed on that one...
- Truncate these strings during import, to respect the 400 char limit. Simple, but kind of ugly.
- Increase this limit within the Kolibri schema. This is probably preferred...

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

Prevents use of Postgres backend for many content sources (e.g. KA). We now have users wanting/trying to use Postgres, and we want to do performance testing on the same.

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/kolibri.log
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->


```
INFO:kolibri.content.utils.channel_import:Importing ContentNode data
Traceback (most recent call last):
  File ".virtualenvs/kolibri/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri', 'console_scripts', 'kolibri')()
  File "kolibri/utils/cli.py", line 653, in main
    start(port, daemon=daemon)
  File "kolibri/utils/cli.py", line 265, in start
    update()
  File "kolibri/utils/cli.py", line 249, in update
    update_channel_metadata()
  File "kolibri/content/utils/annotation.py", line 38, in update_channel_metadata
    import_channel_from_local_db(channel_id)
  File "kolibri/content/utils/channel_import.py", line 383, in import_channel_from_local_db
    import_manager.import_channel_data()
  File "kolibri/content/utils/channel_import.py", line 238, in import_channel_data
    raise e
sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(400)
```


### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

1. Use a custom settings file (e.g. `pg_settings.py`) with something like this (making sure to create the database and user in postgres first):

```
from __future__ import absolute_import, print_function, unicode_literals

from kolibri.deployment.default.settings.base import *  # noqa

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'NAME': 'kolibri',
    },
}
```
2. Set the env var for using the postgres settings: `export DJANGO_SETTINGS_MODULE=pg_settings`
3. Run `kolibri manage migrate` to initialize the database.
4. Run `kolibri manage importchannel network 1ceff53605e55bef987d88e0908658c5` to attempt to import a channel with fields that are too long.

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->

Commit ddeed38c764d9f218ea067b4aeb3d5d514db8e96 on latest release-v0.9.x




Discussion:
Looks as if these fields are `TextField` (no length limit) on Studio:

https://github.com/learningequality/studio/blob/develop/contentcuration/contentcuration/models.py#L573
https://github.com/learningequality/studio/blob/develop/contentcuration/contentcuration/models.py#L614

Going to write some regression tests in Kolibri, and then try switching the fields there to `TextField` as well.

Resolving patch:
diff --git a/kolibri/content/migrations/0008_auto_20180429_1709.py b/kolibri/content/migrations/0008_auto_20180429_1709.py
new file mode 100644
--- /dev/null
+++ b/kolibri/content/migrations/0008_auto_20180429_1709.py
@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.10 on 2018-04-30 00:09
+from __future__ import unicode_literals
+
+from django.db import migrations
+from django.db import models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('content', '0007_auto_20180212_1155'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='contentnode',
+            name='description',
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AlterField(
+            model_name='contentnode',
+            name='license_description',
+            field=models.TextField(blank=True, null=True),
+        ),
+    ]
diff --git a/kolibri/content/models.py b/kolibri/content/models.py
--- a/kolibri/content/models.py
+++ b/kolibri/content/models.py
@@ -15,12 +15,15 @@
 from django.utils.encoding import python_2_unicode_compatible
 from django.utils.text import get_valid_filename
 from jsonfield import JSONField
-from kolibri.core.fields import DateTimeTzField
-from le_utils.constants import content_kinds, file_formats, format_presets
+from le_utils.constants import content_kinds
+from le_utils.constants import file_formats
+from le_utils.constants import format_presets
 from le_utils.constants.languages import LANGUAGE_DIRECTIONS
-from mptt.models import MPTTModel, TreeForeignKey
+from mptt.models import MPTTModel
+from mptt.models import TreeForeignKey
 
 from .utils import paths
+from kolibri.core.fields import DateTimeTzField
 
 PRESET_LOOKUP = dict(format_presets.choices)
 
@@ -91,7 +94,7 @@ class ContentNode(MPTTModel):
     id = UUIDField(primary_key=True)
     parent = TreeForeignKey('self', null=True, blank=True, related_name='children', db_index=True)
     license_name = models.CharField(max_length=50, null=True, blank=True)
-    license_description = models.CharField(max_length=400, null=True, blank=True)
+    license_description = models.TextField(null=True, blank=True)
     has_prerequisite = models.ManyToManyField('self', related_name='prerequisite_for', symmetrical=False, blank=True)
     related = models.ManyToManyField('self', symmetrical=True, blank=True)
     tags = models.ManyToManyField(ContentTag, symmetrical=False, related_name='tagged_content', blank=True)
@@ -106,7 +109,7 @@ class ContentNode(MPTTModel):
     content_id = UUIDField(db_index=True)
     channel_id = UUIDField(db_index=True)
 
-    description = models.CharField(max_length=400, blank=True, null=True)
+    description = models.TextField(blank=True, null=True)
     sort_order = models.FloatField(blank=True, null=True)
     license_owner = models.CharField(max_length=200, blank=True)
     author = models.CharField(max_length=200, blank=True)


Test patch:
diff --git a/kolibri/content/test/test_channel_import.py b/kolibri/content/test/test_channel_import.py
--- a/kolibri/content/test/test_channel_import.py
+++ b/kolibri/content/test/test_channel_import.py
@@ -15,6 +15,7 @@
 from .test_content_app import ContentNodeTestBase
 from kolibri.content import models as content
 from kolibri.content.models import ChannelMetadata
+from kolibri.content.models import CONTENT_SCHEMA_VERSION
 from kolibri.content.models import ContentNode
 from kolibri.content.models import NO_VERSION
 from kolibri.content.models import V020BETA1
@@ -319,37 +320,31 @@ def create_channel():
 
 DATA_PATH_TEMPLATE = os.path.join(os.path.dirname(__file__), '../fixtures/{name}_content_data.json')
 
-class NaiveImportTestCase(ContentNodeTestBase, TransactionTestCase):
-    """
-    Integration test for naive import - this is run using a TransactionTestCase,
+
+class ContentImportTestBase(TransactionTestCase):
+    """This is run using a TransactionTestCase,
     as by default, Django runs each test inside an atomic context in order to easily roll back
     any changes to the DB. However, as we are setting things in the Django DB using SQLAlchemy
     these changes are not caught by this atomic context, and data will persist across tests.
     In order to deal with this, we call an explicit db flush at the end of every test case,
     both this flush and the SQLAlchemy insertions can cause issues with the atomic context used
-    by Django.
-    """
-
-    # When incrementing content schema versions, this should be incremented to the new version
-    # A new TestCase for importing for this old version should then be subclassed from this TestCase
-    # See 'NoVersionImportTestCase' below for an example
-    #
-    # TODO: rtibbles Revert this change that only tests unversioned import
-    name = NO_VERSION
-
-    legacy_schema = None
+    by Django."""
 
     @property
     def schema_name(self):
         return self.legacy_schema or self.name
 
+    @property
+    def data_name(self):
+        return self.legacy_schema or self.name
+
     def setUp(self):
         try:
             self.set_content_fixture()
         except (IOError, EOFError):
             print('No content schema and/or data for {name}'.format(name=self.schema_name))
 
-        super(NaiveImportTestCase, self).setUp()
+        super(ContentImportTestBase, self).setUp()
 
     def set_content_fixture(self):
         self.content_engine = create_engine('sqlite:///:memory:', convert_unicode=True)
@@ -357,7 +352,7 @@ def set_content_fixture(self):
         with open(SCHEMA_PATH_TEMPLATE.format(name=self.schema_name), 'rb') as f:
             metadata = pickle.load(f)
 
-        with open(DATA_PATH_TEMPLATE.format(name=self.schema_name), 'r') as f:
+        with open(DATA_PATH_TEMPLATE.format(name=self.data_name), 'r') as f:
             data = json.load(f)
 
         metadata.bind = self.content_engine
@@ -388,12 +383,44 @@ def get_engine(self, connection_string):
 
     def tearDown(self):
         call_command('flush', interactive=False)
-        super(NaiveImportTestCase, self).tearDown()
+        super(ContentImportTestBase, self).tearDown()
 
     @classmethod
     def tearDownClass(cls):
         django_connection_engine().dispose()
-        super(NaiveImportTestCase, cls).tearDownClass()
+        super(ContentImportTestBase, cls).tearDownClass()
+
+
+class NaiveImportTestCase(ContentNodeTestBase, ContentImportTestBase):
+    """
+    Integration test for naive import
+    """
+
+    # When incrementing content schema versions, this should be incremented to the new version
+    # A new TestCase for importing for this old version should then be subclassed from this TestCase
+    # See 'NoVersionImportTestCase' below for an example
+    #
+    # TODO: rtibbles Revert this change that only tests unversioned import
+    name = NO_VERSION
+
+    legacy_schema = None
+
+
+class ImportLongDescriptionsTestCase(ContentImportTestBase, TransactionTestCase):
+    """
+    When using Postgres, char limits on fields are enforced strictly. This was causing errors importing as described in:
+    https://github.com/learningequality/kolibri/issues/3600
+    """
+
+    name = CONTENT_SCHEMA_VERSION
+    legacy_schema = None
+    data_name = "longdescriptions"
+
+    longdescription = "soverylong" * 45
+
+    def test_long_descriptions(self):
+        self.assertEqual(ContentNode.objects.get(id="32a941fb77c2576e8f6b294cde4c3b0c").license_description, self.longdescription)
+        self.assertEqual(ContentNode.objects.get(id="2e8bac07947855369fe2d77642dfc870").description, self.longdescription)
 
 
 # class NoVersionImportTestCase(NaiveImportTestCase):