Repository: learningequality/kolibri

Issue:
Errors when generateuserdata is used in Command Prompt under Windows 7 or 10.
### Observed behavior

Errors when the `generateuserdata` management command is used in the Command Prompt under Windows 7 or 10.
<img width="739" alt="Screen Shot 2020-06-18 at 3 40 25 PM" src="https://user-images.githubusercontent.com/175580/84998496-7d0b9f80-b182-11ea-83fd-f7d49ba7722f.png">

There are sample data generated BUT the errors are very misleading for the user.
<img width="739" alt="Screen Shot 2020-06-18 at 4 41 15 PM" src="https://user-images.githubusercontent.com/175580/84998564-944a8d00-b182-11ea-9313-db435f159f0e.png">


### Expected behavior
No errors must show when running the command.


### User-facing consequences
Works as intended but the error messages shown confuses the user.


### Errors and logs
[daemon.txt](https://github.com/learningequality/kolibri/files/4797367/daemon.txt)
[kolibri.txt](https://github.com/learningequality/kolibri/files/4797369/kolibri.txt)
[2020-06-18-thu--command prompt errors 1.txt](https://github.com/learningequality/kolibri/files/4797370/2020-06-18-thu--command.prompt.errors.1.txt)

Relevant lines in the Command Prompt:
```
...
--- Logging error ---
Traceback (most recent call last):
  File "c:\python36\lib\logging\__init__.py", line 996, in emit
    stream.write(msg)
  File "c:\python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 78-81: c
haracter maps to <undefined>
...
...
  File "c:\python36\lib\site-packages\kolibri\core\logger\management\commands\ge
nerateuserdata.py", line 212, in handle
    utils.logger_info("    Learner {learner}...".format(learner=user))
  File "c:\python36\lib\site-packages\kolibri\core\logger\utils\user_data.py", l
ine 49, in logger_info
    logger.info(message)
Message: '    Learner "????"@"Home Facility Admin"...'
Arguments: ()
←[37mINFO         Learner "????"@"Home Facility Admin"...←[0m
←[37mINFO         Learner "Natasja Bach"@"Home Facility Admin"...←[0m
←[37mINFO         Learner "Hansen M"@"Home Facility Admin"...←[0m
←[37mINFO         Learner "Leach J Martha"@"Home Facility Admin"...←[0m

C:\Users\IEUser>
```

### Steps to reproduce

1. Use the artifact at [release-v0.14.x-beta2](https://buildkite.com/learningequality/kolibri-python-package/builds/1137) to test in a Windows 7 or 10 VM.
1. Install Kolibri
1. Launch Command Prompt then run `kolibri manage generateuserdata`
1. Notice the errors shown.


### Note when Debugging
The logging function is already [wrapped in a catch-all exception handling function](https://github.com/learningequality/kolibri/blob/develop/kolibri/core/logger/utils/user_data.py#L44).  However, based on readings, this is NOT a Python issue but with the Command Prompt handling of Unicode character encoding under Windows 7 or 10.  

There are various work-arounds without the need to modify the source like: [Using UTF-8 Encoding (CHCP 65001) in Command Prompt / Windows Powershell (Windows 10)](https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window) - but haven't tried those.

So far the most reliable/technical explanation is provided here: https://wiki.python.org/moin/PrintFails - which states:

> Setting to "utf-8" is not recommended as this produces an inaccurate, garbled representation of the output to the console.  For best results, use your console's correct default codepage and a suitable error handler other than "strict".

### Possible Solutions

1. [Python, Unicode, and the Windows console](https://stackoverflow.com/a/32176732/845481) - references the [PrintFails](https://wiki.python.org/moin/PrintFails) topic above and provides the most simple solution as of Python v3.6:

    > `print(unicode_string)`


### Context
* Kolibri version: [release-v0.14.x-beta2](https://buildkite.com/learningequality/kolibri-python-package/builds/1137)
* Operating system: Windows 7 or 10 in a VM
* Browser: n/a
* Python: v3.6.8




Discussion:

Resolving patch:
diff --git a/kolibri/core/logger/management/commands/generateuserdata.py b/kolibri/core/logger/management/commands/generateuserdata.py
--- a/kolibri/core/logger/management/commands/generateuserdata.py
+++ b/kolibri/core/logger/management/commands/generateuserdata.py
@@ -126,6 +126,7 @@ def handle(self, *args, **options):
         num_exams = options["num_exams"]
         max_channels = options["max_channels"]
         device_name = options["device_name"]
+        verbosity = options.get("verbosity", 1)
         # TODO(cpauya):
         # num_groups = options["num_groups"]
         # channel_token = options["channel_token"]
@@ -145,13 +146,14 @@ def handle(self, *args, **options):
         now = timezone.now()
 
         facilities = utils.get_or_create_facilities(
-            n_facilities=n_facilities, device_name=device_name
+            n_facilities=n_facilities, device_name=device_name, verbosity=verbosity
         )
 
         # Device needs to be provisioned before adding superusers
         if no_onboarding:
             utils.logger_info(
-                "Provisioning device. Onboarding will be skipped after starting server."
+                "Provisioning device. Onboarding will be skipped after starting server.",
+                verbosity=verbosity,
             )
             provision_device()
 
@@ -160,12 +162,16 @@ def handle(self, *args, **options):
                 utils.logger_info(
                     'Creating superuser "superuser" with password "password" at facility {facility}.'.format(
                         facility=facility.name
-                    )
+                    ),
+                    verbosity=verbosity,
                 )
                 create_superuser(facility=facility)
 
             classrooms = utils.get_or_create_classrooms(
-                n_classes=n_classes, facility=facility, device_name=device_name
+                n_classes=n_classes,
+                facility=facility,
+                device_name=device_name,
+                verbosity=verbosity,
             )
 
             # TODO(cpauya):
@@ -181,7 +187,8 @@ def handle(self, *args, **options):
 
             if not channels:
                 utils.logger_info(
-                    "No channels found, cannot add channel activities for learners."
+                    "No channels found, cannot add channel activities for learners.",
+                    verbosity=verbosity,
                 )
 
             # Get all the user data at once so that it is distinct across classrooms
@@ -197,6 +204,7 @@ def handle(self, *args, **options):
                     user_data=classroom_user_data,
                     facility=facility,
                     device_name=device_name,
+                    verbosity=verbosity,
                 )
 
                 # Iterate through the slice of the facility_user_data specific to this classroom
@@ -209,20 +217,25 @@ def handle(self, *args, **options):
                         n_content_items = int(base_data["Age"])
 
                     # Loop over all local channels to generate data for each channel
-                    utils.logger_info("    Learner {learner}...".format(learner=user))
+                    utils.logger_info(
+                        "    Learner {learner}...".format(learner=user),
+                        verbosity=verbosity,
+                    )
                     for channel in channels:
                         # TODO(cpauya): check for issue as per Richard's report
                         # REF: https://github.com/learningequality/kolibri/pull/6983#issuecomment-638980072
                         utils.logger_info(
                             "      ==> Adding {channel} channel activity for learner {learner}...".format(
                                 channel=channel, learner=user
-                            )
+                            ),
+                            verbosity=verbosity,
                         )
                         utils.add_channel_activity_for_user(
                             n_content_items=n_content_items,
                             channel=channel,
                             user=user,
                             now=now,
+                            verbosity=verbosity,
                         )
 
                 # create lessons
@@ -232,6 +245,7 @@ def handle(self, *args, **options):
                     channels=ChannelMetadata.objects.all(),
                     lessons=num_lessons,
                     now=now,
+                    verbosity=verbosity,
                 )
 
                 # create exams
@@ -242,6 +256,7 @@ def handle(self, *args, **options):
                     exams=num_exams,
                     now=now,
                     device_name=device_name,
+                    verbosity=verbosity,
                 )
 
                 # # TODO(cpauya): create groups
diff --git a/kolibri/core/logger/utils/user_data.py b/kolibri/core/logger/utils/user_data.py
--- a/kolibri/core/logger/utils/user_data.py
+++ b/kolibri/core/logger/utils/user_data.py
@@ -41,24 +41,35 @@
 #####################
 
 
-def logger_info(message):
+def logger_info(message, verbosity=1):
     # Encapsulate logging in an exception handler to capture encoding errors:
-    # * UnicodeEncodeError
-    # TODO(cpauya): Don't just pass on everything, capture only encoding-related issues.
+    # Sadly, simply wrapping the printing code in an exception handler
+    # doesn't work on Windows, see: https://github.com/learningequality/kolibri/issues/7077
     try:
-        logger.info(message)
+        # MUST: Follow the verbosity mechanism of Django's management commands
+        # https://docs.djangoproject.com/en/1.11/ref/django-admin/#cmdoption-verbosity
+        # and only show when it's > 0.
+        # print("====> verbosity %s" % verbosity)
+        if verbosity > 0:
+            # REF: [Python, Unicode, and the Windows console](https://stackoverflow.com/a/32176732/845481)
+            print(message)
     except Exception:
+        # TODO(cpauya): Don't just pass on everything, capture only specific ones.
         pass
 
 
 def get_or_create_facilities(**options):
     n_facilities = options["n_facilities"]
     device_name = options.get("device_name", "")
+    verbosity = options.get("verbosity", 1)
 
     n_on_device = Facility.objects.all().count()
     n_to_create = n_facilities - n_on_device
     if n_to_create > 0:
-        print("Generating {n} facility object(s)".format(n=n_to_create))
+        logger_info(
+            "Generating {n} facility object(s)".format(n=n_to_create),
+            verbosity=verbosity,
+        )
         for i in range(0, n_to_create):
             facility_name = "Facility{i}".format(i=i + 1)
             if device_name:
@@ -68,7 +79,7 @@ def get_or_create_facilities(**options):
             facility.dataset.location = device_name
             facility.dataset.save()
             if created:
-                logger_info("==> CREATED FACILITY {f}".format(f=facility))
+                logger_info("==> CREATED FACILITY {f}".format(f=facility), verbosity)
 
     return Facility.objects.all()[0:n_facilities]
 
@@ -79,12 +90,14 @@ def get_or_create_classrooms(**options):
     n_on_device = Classroom.objects.filter(parent=facility).count()
     n_to_create = n_classes - n_on_device
     device_name = options.get("device_name", "")
+    verbosity = options.get("verbosity", 1)
 
     if n_to_create > 0:
         logger_info(
             "Generating {n} classroom object(s) for facility: {name}".format(
                 n=n_to_create, name=facility.name
-            )
+            ),
+            verbosity,
         )
         for i in range(0, n_to_create):
             class_name = "Class{i}{a}".format(i=i + 1, a=random.choice("ABCD"))
@@ -96,7 +109,7 @@ def get_or_create_classrooms(**options):
                 parent=facility, name=class_name,
             )
             if created:
-                logger_info("==> CREATED Class {c}".format(c=classroom))
+                logger_info("==> CREATED Class {c}".format(c=classroom), verbosity)
     return Classroom.objects.filter(parent=facility)[0:n_classes]
 
 
@@ -106,6 +119,7 @@ def get_or_create_classroom_users(**options):
     user_data = options["user_data"]
     facility = options["facility"]
     device_name = options.get("device_name", "")
+    verbosity = options.get("verbosity", 1)
 
     # The headers in the user_data.csv file that we use to generate user Full Names
     # Note, we randomly pick from these to give deliberately varied (and sometimes idiosyncratic)
@@ -125,7 +139,8 @@ def get_or_create_classroom_users(**options):
         logger_info(
             "Generating {n} user object(s) for class: {classroom} in facility: {facility}".format(
                 n=n_to_create, classroom=classroom, facility=facility
-            )
+            ),
+            verbosity=verbosity,
         )
         for i in range(0, n_to_create):
             # Get the first base data that does not have a matching user already
@@ -171,16 +186,18 @@ def add_channel_activity_for_user(**options):  # noqa: max-complexity=16
     channel = options["channel"]
     user = options["user"]
     now = options["now"]
+    verbosity = options.get("verbosity", 1)
 
     channel_id = channel.id
     default_channel_content = ContentNode.objects.exclude(
         kind=content_kinds.TOPIC
     ).filter(channel_id=channel_id)
 
-    logger.debug(
+    logger_info(
         "Generating {i} user interaction(s) for user: {user} for channel: {channel}".format(
             i=n_content_items, user=user, channel=channel.name
-        )
+        ),
+        verbosity=verbosity,
     )
     # Generate a content interaction history for this many content items
     for i in range(0, n_content_items):


Test patch:
diff --git a/docs/manual_testing/general_notes/index.rst b/docs/manual_testing/general_notes/index.rst
--- a/docs/manual_testing/general_notes/index.rst
+++ b/docs/manual_testing/general_notes/index.rst
@@ -133,24 +133,27 @@ Create 2 facilities, with 2 classes per facility, with 20 learners per class, 2
 Examples for a fresh Kolibri install (no imported channels)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For a fresh Kolibri installation, use this to automatically create superusers and skip on-boarding.  The super user username is ``superuser`` and password is ``password``.
+For a fresh Kolibri installation, use this to automatically create superusers and skip on-boarding (setup wizard).  The superuser username is ``superuser`` and password is ``password``.
 
 .. code-block:: bash
 
     kolibri manage generateuserdata --no-onboarding
 
-Create 2 facilities, with 2 classes per facility, with 20 learners per class, 2 interactions per learner.
+Create 2 facilities, with 2 classes per facility, with 20 learners per class.
 
 .. code-block:: bash
 
-    kolibri manage generateuserdata --facilities 2 --classes 2 --users 20 --num-content-items 2
+    kolibri manage generateuserdata --facilities 2 --classes 2 --users 20 --no-onboarding
+
 
 Notes
 ~~~~~
 
 1. If there are existing facilities, it will only create the remaining ones.  So if you already have one facility, specifying ``--facilities 2`` will create one more facility and its subsequent sample data.
-1. Use the `--max-channels` option to limit the number of channels for learners to interact with.  This saves a lot of time specially on large data samples.
-1. The ``--no-onboarding`` argument creates a super user for each facility with username ``superuser`` and password ``password``.
+
+2. Use the `--max-channels` option to limit the number of channels for learners to interact with.  This saves a lot of time specially on large data samples.
+
+3. The ``--no-onboarding`` argument creates a super user for each facility with username ``superuser`` and password ``password``.
 
 
 Collecting client and server errors using Sentry