Repository: carpentries/amy

Issue:
Improve development database (use [auto generated] fixtures of some sort)
The data that is loaded in the development database is pleasingly non identifying, but tries so hard to not break privacy that it breaks brains. 

I have downloaded and installed the test data as recommended. It's very hard to get an idea of what the software does because the names and emails are so appallingly random. Maybe try [fake factor](https://pypi.python.org/pypi/fake-factor) instead? Also, [a good wordlist can be found here](http://web.archive.org/web/20090918202746/http://tothink.com/mnemonic/wordlist.html) to make random words (like for "slug").



Discussion:
@datakid you're right, and I hope to get rid of the `db.sql` for version v1.1 or v1.2. Let me change this issue so that I don't forget. Thanks for the pointers, too!


Resolving patch:
diff --git a/scripts/anonymizer.py b/scripts/anonymizer.py
deleted file mode 100644
--- a/scripts/anonymizer.py
+++ /dev/null
@@ -1,262 +0,0 @@
-import sys
-from datetime import date, timedelta
-import random
-import shutil
-import sqlite3
-
-#------------------------------------------------------------
-
-ALPHA = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_'
-
-LOREM_IPSUM = [
-'''Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec a
-diam lectus. Sed sit amet ipsum mauris. Maecenas congue ligula ac quam
-viverra nec consectetur ante hendrerit. Donec et mollis
-dolor. Praesent et diam eget libero egestas mattis sit amet vitae
-augue. Nam tincidunt congue enim, ut porta lorem lacinia
-consectetur. Donec ut libero sed arcu vehicula ultricies a non
-tortor. Lorem ipsum dolor sit amet, consectetur adipiscing
-elit. Aenean ut gravida lorem. Ut turpis felis, pulvinar a semper sed,
-adipiscing id dolor. Pellentesque auctor nisi id magna consequat
-sagittis. Curabitur dapibus enim sit amet elit pharetra tincidunt
-feugiat nisl imperdiet. Ut convallis libero in urna ultrices
-accumsan. Donec sed odio eros. Donec viverra mi quis quam pulvinar at
-malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis
-parturient montes, nascetur ridiculus mus. In rutrum accumsan
-ultricies. Mauris vitae nisi at sem facilisis semper ac in est.'''
-,
-'''Vivamus fermentum semper porta. Nunc diam velit, adipiscing ut
-tristique vitae, sagittis vel odio. Maecenas convallis ullamcorper
-ultricies. Curabitur ornare, ligula semper consectetur sagittis, nisi
-diam iaculis velit, id fringilla sem nunc vel mi. Nam dictum, odio nec
-pretium volutpat, arcu ante placerat erat, non tristique elit urna et
-turpis. Quisque mi metus, ornare sit amet fermentum et, tincidunt et
-orci. Fusce eget orci a orci congue vestibulum. Ut dolor diam,
-elementum et vestibulum eu, porttitor vel elit. Curabitur venenatis
-pulvinar tellus gravida ornare. Sed et erat faucibus nunc euismod
-ultricies ut id justo. Nullam cursus suscipit nisi, et ultrices justo
-sodales nec. Fusce venenatis facilisis lectus ac semper. Aliquam at
-massa ipsum. Quisque bibendum purus convallis nulla ultrices
-ultricies. Nullam aliquam, mi eu aliquam tincidunt, purus velit
-laoreet tortor, viverra pretium nisi quam vitae mi. Fusce vel volutpat
-elit. Nam sagittis nisi dui.'''
-,
-'''Suspendisse lectus leo, consectetur in tempor sit amet, placerat quis
-neque. Etiam luctus porttitor lorem, sed suscipit est rutrum
-non. Curabitur lobortis nisl a enim congue semper. Aenean commodo
-ultrices imperdiet. Vestibulum ut justo vel sapien venenatis
-tincidunt. Phasellus eget dolor sit amet ipsum dapibus condimentum
-vitae quis lectus. Aliquam ut massa in turpis dapibus
-convallis. Praesent elit lacus, vestibulum at malesuada et, ornare et
-est. Ut augue nunc, sodales ut euismod non, adipiscing vitae
-orci. Mauris ut placerat justo. Mauris in ultricies enim. Quisque nec
-est eleifend nulla ultrices egestas quis ut quam. Donec sollicitudin
-lectus a mauris pulvinar id aliquam urna cursus. Cras quis ligula sem,
-vel elementum mi. Phasellus non ullamcorper urna.'''
-,
-'''Class aptent taciti sociosqu ad litora torquent per conubia nostra,
-per inceptos himenaeos. In euismod ultrices facilisis. Vestibulum
-porta sapien adipiscing augue congue id pretium lectus molestie. Proin
-quis dictum nisl. Morbi id quam sapien, sed vestibulum sem. Duis
-elementum rutrum mauris sed convallis. Proin vestibulum magna
-mi. Aenean tristique hendrerit magna, ac facilisis nulla hendrerit
-ut. Sed non tortor sodales quam auctor elementum. Donec hendrerit nunc
-eget elit pharetra pulvinar. Suspendisse id tempus tortor. Aenean
-luctus, elit commodo laoreet commodo, justo nisi consequat massa, sed
-vulputate quam urna quis eros. Donec vel.''']
-
-#------------------------------------------------------------
-
-def get_one(cursor, statement, *params):
-    cursor.execute(statement, params)
-    results = cursor.fetchall()
-    if len(results) == 0:
-        return None
-    return results[0][0]
-
-RANDWORD_SEEN = set()
-
-def randword(low, high):
-    while True:
-        r = ''.join([random.choice(ALPHA) for i in range(random.randrange(low, high))])
-        if r not in RANDWORD_SEEN:
-            RANDWORD_SEEN.add(r)
-            return r
-
-def change(cursor, table, field, func, *args):
-    lower = get_one(cursor, 'select min(id) from {0};'.format(table))
-    upper = get_one(cursor, 'select max(id) from {0};'.format(table))
-    assert (lower is not None) and (upper is not None), \
-           'No lower/upper bounds for {0}.{1}'.format(table, field)
-
-    if isinstance(field, str):
-        stmt = 'update {0} set {1}=? where id=?;'.format(table, field)
-    elif isinstance(field, tuple):
-        filler = ', '.join(['{0}=?'.format(f) for f in field])
-        stmt = 'update {0} set {1} where id=?;'.format(table, filler)
-    else:
-        assert False, 'Unknown field type "{0}" for "{1}"'.format(type(field), field)
-
-    for i in range(lower, upper+1):
-        vals = func(cursor, i, *args) + (i, )
-        try:
-            cursor.execute(stmt, vals)
-        except sqlite3.OperationalError as e:
-            print('FAILED (operational error):', stmt, vals, e)
-        except sqlite3.IntegrityError as e:
-            print('FAILED (integrity error):', stmt, vals, e)
-
-def tuplify(func):
-    def f(*args, **kwargs):
-        result = func(*args, **kwargs)
-        return (result,)
-    return f
-
-#------------------------------------------------------------
-
-def dates(cursor, i):
-    '''Generate start and end dates for workshop.'''
-    start = date(2012, 1, 1) + timedelta(random.randrange(4 * 365))
-    end = start + timedelta(random.randrange(4))
-    if end == start:
-        end = None
-    return (start, end)
-
-@tuplify
-def event_reg_key(cursor, i):
-    '''Generate random event registration key.'''
-    return str(1000000 + i)
-
-@tuplify
-def event_slug(cursor, i):
-    '''Generate event slugs once start/end dates and site names are set.'''
-    start = get_one(cursor, 'select start from workshops_event where id=?;', i)
-    if start is None:
-        return
-    year, month, day = start.split('-')
-    return '{0}-{1}-{2}-{3}'.format(year, month, day, randword(3, 8))
-
-@tuplify
-def url(cursor, i):
-    '''Generate something that looks like a URL.'''
-    _url = get_one(cursor, 'select url from workshops_event where id=?;', i)
-    if not _url:
-        return
-    return 'http://{0}.{1}/{2}-{3}'.format(*[randword(2, 10) for x in range(4)])
-
-@tuplify
-def lorem_ipsum(cursor, i):
-    '''Fill in a large text field.'''
-    if random.uniform(0.0, 1.0) < 0.05:
-        return ''
-    result = '\n'.join(LOREM_IPSUM[0:random.randrange(len(LOREM_IPSUM))])
-    return result
-
-@tuplify
-def monicker(cursor, i):
-    '''Generate a username-style field.'''
-    return randword(2, 10)
-
-@tuplify
-def multi_word(cursor, i, prob_multi, prob_null=0.0):
-    '''Fill in a multi-word field (e.g., site name or person's name).'''
-    if random.uniform(0.0, 1.0) < prob_null:
-        return None
-    elif random.uniform(0.0, 1.0) < prob_multi:
-        return '{0} {1}'.format(randword(2, 10), randword(2, 12))
-    else:
-        return randword(2, 10)
-
-@tuplify
-def domain(cursor, i):
-    '''Fill in site.domain.'''
-    fields = []
-    for x in range(2, random.randrange(4, 5)):
-        fields.append(randword(2, 10))
-    return '.'.join(fields)
-
-@tuplify
-def gender(cursor, i):
-    return random.choice('FMO')
-
-@tuplify
-def email(cursor, i):
-    if random.uniform(0.0, 1.0) < 0.05:
-        return None
-    return '{0}@{1}.{2}'.format(*[randword(2, 8) for x in range(3)])
-
-
-@tuplify
-def empty_string(cursor, i):
-    return ''
-
-
-@tuplify
-def rand_latitude(cursor, i):
-    return random.uniform(-90, 90)
-
-
-@tuplify
-def rand_longitude(cursor, i):
-    return random.uniform(0, 180)
-
-
-#------------------------------------------------------------
-
-def main():
-    assert len(sys.argv) == 4, 'Usage: {0} seed /path/to/source/db /path/to/destination/db'.format(sys.argv[0])
-    assert sys.argv[2] != sys.argv[3], 'Source and destination must be different database'
-
-    seed = int(sys.argv[1])
-    if seed == 0:
-        seed = None
-    db_src = sys.argv[2]
-    db_dst = sys.argv[3]
-
-    random.seed(seed)
-    shutil.copyfile(db_src, db_dst)
-    cnx = sqlite3.connect(db_dst)
-    cur = cnx.cursor()
-
-    change(cur, 'workshops_host', 'domain', domain)
-    change(cur, 'workshops_host', 'fullname', multi_word, 1.0)
-    change(cur, 'workshops_host', 'notes', lorem_ipsum)
-
-    change(cur, 'workshops_person', 'personal', multi_word, 0.1)
-    change(cur, 'workshops_person', 'middle', multi_word, 0.0, 0.9)
-    change(cur, 'workshops_person', 'family', multi_word, 0.1)
-    change(cur, 'workshops_person', 'gender', gender)
-    change(cur, 'workshops_person', 'email', email)
-    change(cur, 'workshops_person', 'github', monicker)
-    change(cur, 'workshops_person', 'twitter', monicker)
-    change(cur, 'workshops_person', 'url', url)
-    change(cur, 'workshops_person', 'username', monicker)
-    change(cur, 'workshops_person', 'password', empty_string)
-    change(cur, 'workshops_person', 'affiliation', empty_string)
-
-    change(cur, 'workshops_event', ('start', 'end'), dates)
-    change(cur, 'workshops_event', 'slug', event_slug)
-    change(cur, 'workshops_event', 'url', url)
-    change(cur, 'workshops_event', 'contact', empty_string)
-    change(cur, 'workshops_event', 'venue', empty_string)
-    change(cur, 'workshops_event', 'address', empty_string)
-    change(cur, 'workshops_event', 'latitude', rand_latitude)
-    change(cur, 'workshops_event', 'longitude', rand_longitude)
-    change(cur, 'workshops_event', 'reg_key', event_reg_key)
-    change(cur, 'workshops_event', 'notes', lorem_ipsum)
-
-    # we can't store historical changes!
-    cur.execute('delete from reversion_version;')
-    cur.execute('delete from reversion_revision;')
-
-    cnx.commit()
-    cur.close()
-    cnx.close()
-
-if __name__ == '__main__':
-    main()
-    # we need to populate reversion_* tables so that no-one needs to do that
-    # upon every `make database` call
-    print("REMEMBER! to run `./manage.py createinitialrevisions` on the new "
-          "database NOW.")
-    print("Next, run `sqlite3 SRC -cmd '.dump' > DEST.sql`")
diff --git a/workshops/management/commands/fake_database.py b/workshops/management/commands/fake_database.py
new file mode 100644
--- /dev/null
+++ b/workshops/management/commands/fake_database.py
@@ -0,0 +1,238 @@
+from datetime import timedelta
+import itertools
+import random
+
+from django.core.management.base import BaseCommand
+from django_countries import countries as Countries
+from faker import Faker
+
+from workshops.models import (
+    Airport,
+    Role,
+    Tag,
+    Badge,
+    Lesson,
+    Person,
+    Award,
+    Qualification,
+    Host,
+    Event,
+    Task,
+)
+
+
+class Command(BaseCommand):
+    help = 'Add fake data to the database.'
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--seed', action='store', default=None,
+            help='Provide an initial seed for randomization mechanism.',
+        )
+
+    def fake_airports(self, faker, count=5):
+        """Add some airports."""
+        # we're not doing anything here, since:
+        # 1. data migrations add some airports already
+        # 2. we'll have more airports as fixtures as of #626
+        pass
+
+    def fake_roles(self, faker):
+        """Provide fixed roles (before they end up in fixtures, see #626)."""
+        roles = ['helper', 'instructor', 'host', 'learner', 'organizer',
+                 'tutor', 'debriefed']
+        for role in roles:
+            Role.objects.create(name=role)
+
+    def fake_tags(self, faker):
+        """Provide fixed tags (before they end up in fixtures, see #626)."""
+        tags = [
+            ('SWC', 'Software Carpentry Workshop'),
+            ('DC', 'Data Carpentry Workshop'),
+            ('LC', 'Library Carpentry Workshop'),
+            ('WiSE', 'Women in Science and Engineering'),
+            ('TTT', 'Train the Trainers'),
+        ]
+        for tag, details in tags:
+            Tag.objects.create(name=tag, details=details)
+
+    def fake_instructors(self, faker, count=5, add_badge=True,
+                         add_qualifications=True):
+        """Add a few people with random instructor badge, random airport, and
+        random qualification."""
+        airports = list(Airport.objects.all())
+        badges = list(Badge.objects.all())
+        lessons = list(Lesson.objects.all())
+        for i in range(count):
+            user_name = faker.user_name()
+            emails = [faker.email(), faker.safe_email(), faker.free_email(),
+                      faker.company_email()]
+            gender = random.choice(Person.GENDER_CHOICES)[0]
+            if gender == 'F':
+                name = faker.first_name_female()
+                last_name = faker.last_name_female()
+            elif gender == 'M':
+                name = faker.first_name_male()
+                last_name = faker.last_name_male()
+            else:
+                name, last_name = faker.first_name(), faker.last_name()
+
+            person = Person.objects.create(
+                personal=name,
+                middle=None,
+                family=last_name,
+                email=random.choice(emails),
+                gender=gender,
+                may_contact=random.choice([True, False]),
+                airport=random.choice(airports),
+                github=user_name,
+                twitter=user_name,
+                url=faker.url(),
+                username=user_name,
+            )
+
+            if add_badge:
+                Award.objects.create(
+                    person=person, badge=random.choice(badges),
+                    awarded=faker.date_time_this_year(before_now=True,
+                                                      after_now=False).date(),
+                )
+            if add_qualifications:
+                for lesson in random.sample(lessons, 4):
+                    Qualification.objects.create(person=person, lesson=lesson)
+
+    def fake_noninstructors(self, faker, count=5):
+        """Add a few people who aren't instructors."""
+        return self.fake_instructors(
+            faker=faker, count=count, add_badge=False,
+            add_qualifications=False,
+        )
+
+    def fake_hosts(self, faker, count=5):
+        """Add some hosts for events."""
+        countries = list(Countries)
+        for i in range(count):
+            Host.objects.create(
+                domain=faker.domain_name(),
+                fullname=faker.company(),
+                country=random.choice(countries)[0],
+            )
+
+    def fake_current_events(self, faker, count=5):
+        """Ongoing and upcoming events."""
+        twodays = timedelta(days=2)
+        hosts = list(Host.objects.exclude(domain='self-organized'))
+        countries = list(Countries)
+
+        for i in range(count):
+            start = faker.date_time_this_year(before_now=False,
+                                              after_now=True).date()
+            Event.objects.create(
+                slug='{:%Y-%m-%d}-{}'.format(start, faker.slug()),
+                start=start,
+                end=start + twodays,
+                url=faker.url(),
+                host=random.choice(hosts),
+                # needed in order for event to be published
+                country=random.choice(countries)[0],
+                venue=faker.word().title(),
+                address=faker.sentence(nb_words=4, variable_nb_words=True),
+                latitude=random.uniform(-90, 90),
+                longitude=random.uniform(0, 180),
+            )
+
+    def fake_uninvoiced_events(self, faker, count=5):
+        """Preferably in the past, and with 'uninvoiced' status."""
+        twodays = timedelta(days=2)
+        countries = list(Countries)
+        hosts = list(Host.objects.exclude(domain='self-organized'))
+
+        for i in range(count):
+            start = faker.date_time_this_year(before_now=True,
+                                              after_now=False).date()
+            Event.objects.create(
+                slug='{:%Y-%m-%d}-{}'.format(start, faker.slug()),
+                start=start,
+                end=start + twodays,
+                url=faker.url(),
+                host=random.choice(hosts),
+                # needed in order for event to be published
+                country=random.choice(countries)[0],
+                venue=faker.word().title(),
+                address=faker.sentence(nb_words=4, variable_nb_words=True),
+                latitude=random.uniform(-90, 90),
+                longitude=random.uniform(0, 180),
+                # needed in order for event to be uninvoiced
+                invoice_status='not-invoiced',
+            )
+
+    def fake_unpublished_events(self, faker, count=5):
+        """Events with missing location data (which is required for publishing
+        them)."""
+        twodays = timedelta(days=2)
+        hosts = list(Host.objects.exclude(domain='self-organized'))
+
+        for i in range(count):
+            start = faker.date_time_this_year(before_now=True,
+                                              after_now=True).date()
+            Event.objects.create(
+                slug='{:%Y-%m-%d}-{}'.format(start, faker.slug()),
+                start=start,
+                end=start + twodays,
+                url=faker.url(),
+                host=random.choice(hosts),
+            )
+
+    def fake_self_organized_events(self, faker, count=5):
+        """Full-blown events with 'self-organized' host."""
+        twodays = timedelta(days=2)
+        self_organized = Host.objects.get(domain='self-organized')
+        countries = list(Countries)
+        invoice_statuses = Event.INVOICED_CHOICES
+
+        for i in range(count):
+            start = faker.date_time_this_year(before_now=True,
+                                              after_now=True).date()
+            Event.objects.create(
+                slug='{:%Y-%m-%d}-{}'.format(start, faker.slug()),
+                start=start,
+                end=start + twodays,
+                url=faker.url(),
+                host=self_organized,
+                # needed in order for event to be published
+                country=random.choice(countries)[0],
+                venue=faker.word().title(),
+                address=faker.sentence(nb_words=4, variable_nb_words=True),
+                latitude=random.uniform(-90, 90),
+                longitude=random.uniform(0, 180),
+                # needed in order for event to be uninvoiced
+                invoice_status=random.choice(invoice_statuses)[0],
+            )
+
+    def fake_tasks(self, faker, count=50):
+        events = Event.objects.all()
+        persons = Person.objects.all()
+        roles = Role.objects.all()
+        all_possible = itertools.product(events, persons, roles)
+
+        for event, person, role in random.sample(list(all_possible), count):
+            Task.objects.create(event=event, person=person, role=role)
+
+    def handle(self, *args, **options):
+        faker = Faker()
+
+        seed = options['seed']
+        if seed is not None:
+            faker.seed(seed)
+
+        self.fake_airports(faker)
+        self.fake_roles(faker)
+        self.fake_tags(faker)
+        self.fake_instructors(faker)
+        self.fake_noninstructors(faker)
+        self.fake_hosts(faker)
+        self.fake_current_events(faker)
+        self.fake_uninvoiced_events(faker)
+        self.fake_unpublished_events(faker)
+        self.fake_self_organized_events(faker)
+        self.fake_tasks(faker)


Test patch:
diff --git a/workshops/test/test_commands.py b/workshops/test/test_commands.py
--- a/workshops/test/test_commands.py
+++ b/workshops/test/test_commands.py
@@ -3,13 +3,26 @@
 These commands are run via `./manage.py command`."""
 
 from django.core.management import call_command
+from django.test import TestCase
 from django.utils.six import StringIO
 
 from workshops.management.commands.upgrade_instructor_profiles import (
-    Command,
+    Command as UpgradeInstructorProfileCommand,
     ALL_FIELDS
 )
-from workshops.models import Person
+from workshops.management.commands.fake_database import (
+    Command as FakeDatabaseCommand,
+    Faker
+)
+from workshops.models import (
+    Airport,
+    Role,
+    Tag,
+    Person,
+    Host,
+    Event,
+    Task,
+)
 
 from .base import TestBase
 
@@ -17,7 +30,7 @@
 class TestUpgradeInstructorProfile(TestBase):
     def setUp(self):
         super().setUp()
-        self.cmd = Command()
+        self.cmd = UpgradeInstructorProfileCommand()
 
     def test_domains_translation(self):
         "Make sure translating domains from string to list works as expected."
@@ -364,7 +377,7 @@ def test_update_subset_of_fields(self):
         assert self.hermione.family == 'Grangerdaughter'
 
     def test_process(self):
-        """Make sure the Command works well with CSVs (even ill-formatted)."""
+        """Make sure the command works well with CSVs (even ill-formatted)."""
         # we'll only test immunity to ill-formatted CSVs, not if they succeed
         # in updating instructors
         FN = [
@@ -391,7 +404,7 @@ def test_process(self):
         assert correct_list == [True, False, False], correct_list
 
     def test_output(self):
-        """Make sure the Command works well."""
+        """Make sure the command works well."""
 
         def stringify_streams(stream1, stream2):
             stream1.seek(0)
@@ -457,3 +470,60 @@ def stringify_streams(stream1, stream2):
         assert 'row 2' in stderr
         assert 'ERROR' in stderr
         assert 'ruby' in stderr
+
+
+class TestFakeDatabaseCommand(TestCase):
+    def setUp(self):
+        self.cmd = FakeDatabaseCommand()
+        self.seed = 12345
+        self.faker = Faker()
+        self.faker.seed(self.seed)
+
+    def test_no_airports_created(self):
+        """Make sure we don't create any airports.
+
+        We don't want to create them, because data migrations add some, and in
+        the future we want to add them via fixture (see #626)."""
+        airports_before = set(Airport.objects.all())
+        self.cmd.fake_airports(self.faker)
+        airports_after = set(Airport.objects.all())
+
+        self.assertEqual(airports_before, airports_after)
+
+    def test_new_roles_added(self):
+        """Make sure we add roles that are hard-coded. They'll end up in
+        fixtures in future (see #626)."""
+        roles = ['helper', 'instructor', 'host', 'learner', 'organizer',
+                 'tutor', 'debriefed']
+        self.assertFalse(Role.objects.filter(name__in=roles).exists())
+        self.cmd.fake_roles(self.faker)
+
+        self.assertEqual(set(roles),
+                         set(Role.objects.values_list('name', flat=True)))
+
+    def test_new_tags_added(self):
+        """Make sure we add tags that are hard-coded. They'll end up in
+        fixtures in future (see #626)."""
+        tags = ['SWC', 'DC', 'LC', 'WiSE', 'TTT', 'online', 'stalled',
+                'unresponsive']
+        self.assertNotEqual(set(tags),
+                            set(Tag.objects.values_list('name', flat=True)))
+
+        self.cmd.fake_tags(self.faker)
+        self.assertEqual(set(tags),
+                         set(Tag.objects.values_list('name', flat=True)))
+
+    def test_database_populated(self):
+        """Make sure the database is getting populated."""
+        self.assertFalse(Person.objects.exists())
+        self.assertFalse(Host.objects.exclude(domain='self-organized')
+                                     .exists())
+        self.assertFalse(Event.objects.exists())
+        self.assertFalse(Task.objects.exists())
+
+        call_command('fake_database', seed=self.seed)
+
+        self.assertTrue(Person.objects.exists())
+        self.assertTrue(Host.objects.exclude(domain='self-organized').exists())
+        self.assertTrue(Event.objects.exists())
+        self.assertTrue(Task.objects.exists())