Repository: learningequality/kolibri

Issue:
Content imported from CLI isn't visible within Kolibri
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
Channels imported from the CLI are invisible and unavailable in the Kolibri UI.

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
Channels imported from the CLI should be visible and available within the Kolibri UI.

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->
It's a regression, so users accustomed to importing content from the CLI will run into this blocker.

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->
Slack convo: https://learningequality.slack.com/archives/CB37UM23A/p1576257121027300
```
(kolibri) bjester@phizbo:~/Projects/learningequality/kolibri$ kolibri shell
WARNING:root:No C Extensions available for this platform.

INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from kolibri.core.content.models import ContentNode
>>> ContentNode.objects.filter(channel_id='effbdfe850dd56d4a595037a090e9225', available=True)
<ContentNodeQueryset []>
>>> ContentNode.objects.filter(channel_id='effbdfe850dd56d4a595037a090e9225', available=True).count()
0
>>> ContentNode.objects.filter(channel_id='effbdfe850dd56d4a595037a090e9225').count()
17
```

### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->
1) Set up a new Kolibri instance with a fresh home directory
2) Imported a channel and content from the command line
```
(kolibri) bjester@phizbo:~/Projects/learningequality/kolibri$ kolibri manage importchannel -- network --baseurl="https://develop.studio.learningequality.org" effbdfe850dd56d4a595037a090e9225
WARNING:root:No C Extensions available for this platform.

INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importchannel network --baseurl=https://develop.studio.learningequality.org effbdfe850dd56d4a595037a090e9225
INFO     Downloading data for channel id effbdfe850dd56d4a595037a090e9225
INFO     Importing ContentTag data
INFO     Importing ContentNode_has_prerequisite data
INFO     Importing ContentNode_related data
INFO     Importing ContentNode_tags data
INFO     Importing ContentNode data
INFO     Importing Language data
INFO     Importing File data
INFO     Importing LocalFile data
INFO     Importing AssessmentMetaData data
INFO     Importing ChannelMetadata data
INFO     Channel effbdfe850dd56d4a595037a090e9225 successfully imported into the database
(kolibri) bjester@phizbo:~/Projects/learningequality/kolibri$ kolibri manage importcontent -- network --baseurl="https://develop.studio.learningequality.org" effbdfe850dd56d4a595037a090e9225
WARNING:root:No C Extensions available for this platform.

INFO     Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Invoking command importcontent network --baseurl=https://develop.studio.learningequality.org effbdfe850dd56d4a595037a090e9225
  [##########################################################################################################################################################################################################]  100%INFO     Setting availability to True of 23 LocalFile objects based on passed in checksums
INFO     Setting availability of non-topic ContentNode objects based on LocalFile availability
INFO     Annotating ContentNode objects with children for 3 levels
INFO     Annotating ContentNode objects with children for level 3
INFO     Annotating ContentNode objects with children for level 2
INFO     Annotating ContentNode objects with children for level 1
```
4) Opened Kolibri > Learn
5) Observed no content available
6) Opened Kolibri > Device > Channels
7) Observed no channels shown as imported

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Branch: https://github.com/learningequality/kolibri/pull/6268
OS: Ubuntu 18.04
Browser: N/A


Discussion:

Resolving patch:
diff --git a/kolibri/core/content/management/commands/importcontent.py b/kolibri/core/content/management/commands/importcontent.py
--- a/kolibri/core/content/management/commands/importcontent.py
+++ b/kolibri/core/content/management/commands/importcontent.py
@@ -33,6 +33,20 @@
 FILE_NOT_TRANSFERRED = 0
 
 
+def lookup_channel_listing_status(channel_id, baseurl=None):
+    """
+    Look up the listing status of the channel from the remote, this is surfaced as a
+    `public` boolean field.
+    """
+    resp = requests.get(get_channel_lookup_url(identifier=channel_id, baseurl=baseurl))
+
+    if resp.status_code != 200:
+        return None
+
+    (channel_info,) = resp.json()
+    return channel_info.get("public", None)
+
+
 class Command(AsyncCommand):
     def add_arguments(self, parser):
         # let's save the parser in case we need to print a help statement
@@ -56,7 +70,7 @@ def add_arguments(self, parser):
             "-n",
             # Split the comma separated string we get, into a list of strings
             type=lambda x: x.split(","),
-            default=[],
+            default=None,
             required=False,
             dest="node_ids",
             help=node_ids_help_text,
@@ -73,7 +87,7 @@ def add_arguments(self, parser):
             "--exclude_node_ids",
             # Split the comma separated string we get, into a list of string
             type=lambda x: x.split(","),
-            default=[],
+            default=None,
             required=False,
             dest="exclude_node_ids",
             help=exclude_node_ids_help_text,
@@ -223,7 +237,7 @@ def _transfer(  # noqa: max-complexity=16
 
         # If we're downloading, check listing status
         if method == DOWNLOAD_METHOD:
-            public = self.lookup_channel_listing_status(
+            public = lookup_channel_listing_status(
                 channel_id=channel_id, baseurl=baseurl
             )
 
@@ -435,18 +449,3 @@ def handle_async(self, *args, **options):
                     options["command"]
                 )
             )
-
-    def lookup_channel_listing_status(self, channel_id, baseurl=None):
-        """
-        Look up the listing status of the channel from the remote, this is surfaced as a
-        `public` boolean field.
-        """
-        resp = requests.get(
-            get_channel_lookup_url(identifier=channel_id, baseurl=baseurl)
-        )
-
-        if resp.status_code == 404:
-            return None
-
-        (channel_info,) = resp.json()
-        return channel_info.get("public", None)


Test patch:
diff --git a/kolibri/core/content/test/test_import_export.py b/kolibri/core/content/test/test_import_export.py
--- a/kolibri/core/content/test/test_import_export.py
+++ b/kolibri/core/content/test/test_import_export.py
@@ -178,6 +178,10 @@ def test_remote_import_readtimeout(
         import_channel_mock.assert_not_called()
 
 
+@patch(
+    "kolibri.core.content.management.commands.importcontent.lookup_channel_listing_status",
+    return_value=False,
+)
 @patch(
     "kolibri.core.content.management.commands.importcontent.calculate_files_to_transfer"
 )
@@ -209,6 +213,7 @@ def test_remote_cancel_immediately(
         FileDownloadMock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         # Check behaviour if cancellation is called before any file download starts
         FileDownloadMock.return_value.__iter__.return_value = ["one", "two", "three"]
@@ -244,6 +249,7 @@ def test_remote_cancel_during_transfer(
         remote_path_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         # If transfer is cancelled during transfer of first file
         local_path = tempfile.mkstemp()[1]
@@ -294,6 +300,7 @@ def test_remote_cancel_after_file_copy_file_not_deleted(
         checksum_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         # If transfer is cancelled after transfer of first file
         local_path_1 = tempfile.mkstemp()[1]
@@ -328,6 +335,7 @@ def test_local_cancel_immediately(
         FileCopyMock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         # Local version of test above
         FileCopyMock.return_value.__iter__.return_value = ["one", "two", "three"]
@@ -357,6 +365,7 @@ def test_local_cancel_during_transfer(
         local_path_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         # Local version of test above
         local_dest_path = tempfile.mkstemp()[1]
@@ -390,6 +399,7 @@ def test_remote_cancel_during_connect_error(
         len_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         LocalFile.objects.filter(pk="6bdfea4a01830fdd4a585181c0b8068c").update(
             file_size=2201062
@@ -421,7 +431,12 @@ def test_remote_cancel_during_connect_error(
         "kolibri.core.content.management.commands.importcontent.paths.get_content_storage_file_path"
     )
     def test_remote_import_httperror_404(
-        self, path_mock, logger_mock, annotation_mock, files_to_transfer_mock
+        self,
+        path_mock,
+        logger_mock,
+        annotation_mock,
+        files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         local_dest_path_1 = tempfile.mkstemp()[1]
         local_dest_path_2 = tempfile.mkstemp()[1]
@@ -475,6 +490,7 @@ def test_remote_import_httperror_502(
         sleep_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         response = Response()
         response.status_code = 502
@@ -504,6 +520,7 @@ def test_remote_import_httperror_500(
         file_download_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         response = Response()
         response.status_code = 500
@@ -537,6 +554,7 @@ def test_remote_import_chunkedencodingerror(
         len_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         LocalFile.objects.filter(pk="6bdfea4a01830fdd4a585181c0b8068c").update(
             file_size=2201062
@@ -580,6 +598,7 @@ def test_local_import_oserror_dne(
         logger_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         dest_path = tempfile.mkstemp()[1]
         path_mock.side_effect = [dest_path, "/test/dne"]
@@ -605,6 +624,7 @@ def test_local_import_oserror_permission_denied(
         logger_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         dest_path = tempfile.mkstemp()[1]
         path_mock.side_effect = [dest_path, "/test/dne"]
@@ -637,6 +657,7 @@ def test_local_import_source_corrupted(
         remove_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         local_src_path = tempfile.mkstemp()[1]
         local_dest_path = tempfile.mkstemp()[1]
@@ -680,6 +701,7 @@ def test_local_import_source_corrupted_full_progress(
         isfile_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         """
         Ensure that when a file is imported that does not match the file size in the database
@@ -740,18 +762,14 @@ def test_local_import_source_corrupted_full_progress(
         "kolibri.core.content.management.commands.importcontent.AsyncCommand.is_cancelled",
         return_value=False,
     )
-    @patch(
-        "kolibri.core.content.management.commands.importcontent.Command.lookup_channel_listing_status",
-        return_value=False,
-    )
     def test_remote_import_source_corrupted(
         self,
-        lookup_channel_listing_status,
         is_cancelled_mock,
         path_mock,
         finalize_dest_mock,
         annotation_mock,
         files_to_transfer_mock,
+        channel_list_status_mock,
     ):
         dest_path_1 = tempfile.mkstemp()[1]
         dest_path_2 = tempfile.mkstemp()[1]
@@ -780,11 +798,55 @@ def test_remote_import_source_corrupted(
         annotation_mock.set_content_visibility.assert_called_with(
             self.the_channel_id,
             [],
-            exclude_node_ids=[],
+            exclude_node_ids=None,
             node_ids=["32a941fb77c2576e8f6b294cde4c3b0c"],
             public=False,
         )
 
+    @patch(
+        "kolibri.core.content.management.commands.importcontent.transfer.FileDownload.finalize"
+    )
+    @patch(
+        "kolibri.core.content.management.commands.importcontent.paths.get_content_storage_file_path"
+    )
+    @patch(
+        "kolibri.core.content.management.commands.importcontent.AsyncCommand.is_cancelled",
+        return_value=False,
+    )
+    def test_remote_import_full_import(
+        self,
+        is_cancelled_mock,
+        path_mock,
+        finalize_dest_mock,
+        annotation_mock,
+        files_to_transfer_mock,
+        channel_list_status_mock,
+    ):
+        dest_path_1 = tempfile.mkstemp()[1]
+        dest_path_2 = tempfile.mkstemp()[1]
+        path_mock.side_effect = [dest_path_1, dest_path_2]
+        LocalFile.objects.filter(pk="6bdfea4a01830fdd4a585181c0b8068c").update(
+            file_size=2201062
+        )
+        LocalFile.objects.filter(pk="211523265f53825b82f70ba19218a02e").update(
+            file_size=336974
+        )
+        files_to_transfer_mock.return_value = (
+            LocalFile.objects.filter(
+                pk__in=[
+                    "6bdfea4a01830fdd4a585181c0b8068c",
+                    "211523265f53825b82f70ba19218a02e",
+                ]
+            ),
+            10,
+        )
+        call_command(
+            "importcontent", "network", self.the_channel_id,
+        )
+        annotation_mock.set_content_visibility.assert_called_with(
+            self.the_channel_id, [], exclude_node_ids=None, node_ids=None, public=False,
+        )
+
 
 @override_option("Paths", "CONTENT_DIR", tempfile.mkdtemp())
 class ExportChannelTestCase(TestCase):