Repository: uccser/cs-unplugged

Issue:
Fix migration handling on travis
Unit tests should run without migrations, with the the test database schema being determined entirely from the current state of the model files and the test settings.

Since merging #643 this has happened locally, but the tests running on Travis still build the schema using the migrations files. This causes issues when the current production database schema differs from that required for the unit tests (specifically, this is occurring for unit tests that test internationalisation features, when the production database only contains columns for English values). This actually should have prevented merging of #680, except ironically that branch was missing a migration file that removed old `de` and `fr` columns (created during feature development) from the database, and so they _were_ available to the unit tests. I won't be able to add that migration until this issue is resolved, as all i18n unit tests will fail until they are allowed to run without migrations.

Production database migrations should _still_ be tested - this basically requires booting the system and then running `./csu update` to check that everything loads correctly with no errors, essentially a dry run of the database update step in the deployment phase. 

Currently this actually happens at the start of _every_ travis job as part of the `./csu start` command. This seems overkill to me, and adds a massive overhead to each job needing to generate all thumbnails etc before actually running any tests. Instead, I'd suggest we refactor the csu script to separate out the 2 distinct phases that currently are both run as part of `./csu start`:
- Starting the docker containers (IMO, should be the only function of `./csu start`)
- Loading database content and other tasks such as static file handling, thumbnail generation etc. (`./csu update`, run from `./csu start`)

I propose the following changes:
- Remove the `./csu update` call from `./csu start`.
- Run all unit tests without migrations (both on travis and locally)
- Add a new travis job to run `./csu update` after `./csu start`, acting as a dry run of the database update step from deployment
- Add missing topics migration file to remove erroneous `de`/`fr` columns

Thoughts @JackMorganNZ @hayleyavw?




Discussion:

Resolving patch:
diff --git a/csunplugged/topics/migrations/0087_auto_20171122_0324.py b/csunplugged/topics/migrations/0087_auto_20171122_0324.py
new file mode 100644
--- /dev/null
+++ b/csunplugged/topics/migrations/0087_auto_20171122_0324.py
@@ -0,0 +1,318 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.5 on 2017-11-22 03:24
+from __future__ import unicode_literals
+
+import django.contrib.postgres.fields
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('topics', '0086_auto_20171108_0840'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='agegroup',
+            name='description_de',
+        ),
+        migrations.RemoveField(
+            model_name='agegroup',
+            name='description_fr',
+        ),
+        migrations.RemoveField(
+            model_name='classroomresource',
+            name='description_de',
+        ),
+        migrations.RemoveField(
+            model_name='classroomresource',
+            name='description_fr',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumarea',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumarea',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumintegration',
+            name='content_de',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumintegration',
+            name='content_fr',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumintegration',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='curriculumintegration',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='glossaryterm',
+            name='definition_de',
+        ),
+        migrations.RemoveField(
+            model_name='glossaryterm',
+            name='definition_fr',
+        ),
+        migrations.RemoveField(
+            model_name='glossaryterm',
+            name='term_de',
+        ),
+        migrations.RemoveField(
+            model_name='glossaryterm',
+            name='term_fr',
+        ),
+        migrations.RemoveField(
+            model_name='learningoutcome',
+            name='text_de',
+        ),
+        migrations.RemoveField(
+            model_name='learningoutcome',
+            name='text_fr',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='computational_thinking_links_de',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='computational_thinking_links_fr',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='content_de',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='content_fr',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='heading_tree_de',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='heading_tree_fr',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='programming_challenges_description_de',
+        ),
+        migrations.RemoveField(
+            model_name='lesson',
+            name='programming_challenges_description_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='content_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='content_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='extra_challenge_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='extra_challenge_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallenge',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengedifficulty',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengedifficulty',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='expected_result_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='expected_result_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='hints_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='hints_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='solution_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengeimplementation',
+            name='solution_fr',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengelanguage',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='programmingchallengelanguage',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='resourcedescription',
+            name='description_de',
+        ),
+        migrations.RemoveField(
+            model_name='resourcedescription',
+            name='description_fr',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='content_de',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='content_fr',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='name_fr',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='other_resources_de',
+        ),
+        migrations.RemoveField(
+            model_name='topic',
+            name='other_resources_fr',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='computational_thinking_links_de',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='computational_thinking_links_fr',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='content_de',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='content_fr',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='heading_tree_de',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='heading_tree_fr',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='name_de',
+        ),
+        migrations.RemoveField(
+            model_name='unitplan',
+            name='name_fr',
+        ),
+        migrations.AlterField(
+            model_name='agegroup',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='classroomresource',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='curriculumarea',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='curriculumintegration',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='glossaryterm',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='learningoutcome',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='lesson',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='programmingchallenge',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='programmingchallengedifficulty',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='programmingchallengeimplementation',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='programmingchallengelanguage',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='resourcedescription',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='topic',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+        migrations.AlterField(
+            model_name='unitplan',
+            name='languages',
+            field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=10), default=[], size=None),
+        ),
+    ]
diff --git a/csunplugged/utils/TranslatableModel.py b/csunplugged/utils/TranslatableModel.py
--- a/csunplugged/utils/TranslatableModel.py
+++ b/csunplugged/utils/TranslatableModel.py
@@ -29,7 +29,7 @@ def get_queryset(self):
 class TranslatableModel(models.Model):
     """Abstract base class for models needing to store list of available languages."""
 
-    languages = ArrayField(models.CharField(max_length=5), default=[])
+    languages = ArrayField(models.CharField(max_length=10), default=[])
 
     objects = models.Manager()
     translated_objects = TranslatedModelManager()


Test patch:
diff --git a/csunplugged/tests/resources/management/test_makeresourcethumbnails_command.py b/csunplugged/tests/resources/management/test_makeresourcethumbnails_command.py
--- a/csunplugged/tests/resources/management/test_makeresourcethumbnails_command.py
+++ b/csunplugged/tests/resources/management/test_makeresourcethumbnails_command.py
@@ -13,7 +13,7 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.test_data = ResourcesTestDataGenerator()
         self.language = "en"
-        self.THUMBNAIL_PATH = "staticfiles/img/resources/{}/thumbnails/{}"
+        self.THUMBNAIL_PATH = "build/img/resources/{}/thumbnails/{}"
 
     def test_makeresourcethumbnails_command_single_resource(self):
         self.test_data.create_resource(
@@ -22,9 +22,10 @@ def test_makeresourcethumbnails_command_single_resource(self):
             "resources/grid.html",
             "GridResourceGenerator",
         )
+        # TODO: Fix these tests, they shouldn't be writing files into the build directory
         management.call_command("makeresourcethumbnails")
-        open(self.THUMBNAIL_PATH.format("grid", "grid-paper_size-a4.png"))
-        open(self.THUMBNAIL_PATH.format("grid", "grid-paper_size-letter.png"))
+        open(self.THUMBNAIL_PATH.format("resource-grid", "resource-grid-paper_size-a4.png"))
+        open(self.THUMBNAIL_PATH.format("resource-grid", "resource-grid-paper_size-letter.png"))
 
     def test_makeresourcethumbnails_command_multiple_resources(self):
         self.test_data.create_resource(
@@ -39,8 +40,9 @@ def test_makeresourcethumbnails_command_multiple_resources(self):
             "resources/arrows.html",
             "ArrowsResourceGenerator",
         )
+        # TODO: Fix these tests, they shouldn't be writing files into the build directory
         management.call_command("makeresourcethumbnails")
-        open(self.THUMBNAIL_PATH.format("grid", "grid-paper_size-a4.png"))
-        open(self.THUMBNAIL_PATH.format("grid", "grid-paper_size-letter.png"))
-        open(self.THUMBNAIL_PATH.format("arrows", "arrows-paper_size-a4.png"))
-        open(self.THUMBNAIL_PATH.format("arrows", "arrows-paper_size-letter.png"))
+        open(self.THUMBNAIL_PATH.format("resource-grid", "resource-grid-paper_size-a4.png"))
+        open(self.THUMBNAIL_PATH.format("resource-grid", "resource-grid-paper_size-letter.png"))
+        open(self.THUMBNAIL_PATH.format("resource-arrows", "resource-arrows-paper_size-a4.png"))
+        open(self.THUMBNAIL_PATH.format("resource-arrows", "resource-arrows-paper_size-letter.png"))