Repository: learningequality/kolibri

Issue:
Exercises missing when importing from any other peer apart from studio
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
Trying to import a channel via peer sync i.e. ```Local network or internet``` source, all other content but exercises are being imported. Importing from studio or a thumbdrive then all of the exercises copy fine.

Tried with a few other channels as well which have exercises (For e.g. ```CK-12 > Math > Elementary Math > Grade 1 > Addition and Subtraction facts to 10```). So it is not just a KA issue.

Rather it seems that the exercises created using Perseus framework are importing but not the rest.

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
All exercises part of the topics imported should have been imported but they weren't. 

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->

Confusing results for all.

### Errors and logs
<!--
Relevant logs from:

 * ~/.kolibri/kolibri.log


Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->
[snippet_kolibri.log](https://github.com/learningequality/kolibri/files/2795923/snippet_kolibri.log)

### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->

Select to import all contents from any of these channel topics via peer sync i.e. ```Local network or internet``` wherein the source has been seeded initially from the studio -
- ```Khan Academy (English) > Math >1st grade > Place value > Numbers 0 to 120```
- ```CK-12 > Math > Elementary Math > Grade 1 > Addition and Subtraction facts to 10```

After import match with source if all the exercises have been imported.

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Kolibri 0.11.1 - Source on Linux, Firefox 64.0.2
Kolibri 0.11.1 - Destination on Windows 10, Firefox 64.0.2




Discussion:
Thanks for the report, this seems very problematic! 
One thing I note in the log file is this line:
`ERROR 2019-01-25 16:52:06,943 importcontent An error occurred during content import: File http://192.168.4.34:8080/content/storage/7/3/735d4c10bac106a8a4196f8618207f6b.perseus is corrupted.`
Where an exercise file appears to be corrupted, wondering if this might be causing the issue (or perhaps we are wrongly flagging exercise files as corrupted).
@rtibbles I had also noticed this. It is possible that this message is in error as I found most of the Perseus based exercises imported correctly but there is no way I can be conclusive on this.
It's possible that the files are being transferred, but because of the error we might not be then making them as imported.

Resolving patch:
diff --git a/kolibri/core/content/management/commands/importcontent.py b/kolibri/core/content/management/commands/importcontent.py
--- a/kolibri/core/content/management/commands/importcontent.py
+++ b/kolibri/core/content/management/commands/importcontent.py
@@ -18,6 +18,10 @@
 
 logger = logging.getLogger(__name__)
 
+FILE_TRANSFERRED = 2
+FILE_SKIPPED = 1
+FILE_NOT_TRANSFERRED = 0
+
 
 class Command(AsyncCommand):
 
@@ -155,16 +159,16 @@ def _transfer(self, method, channel_id, path=None, node_ids=None, exclude_node_i
                 finished = False
                 try:
                     while not finished:
-                        finished, increment = self._start_file_transfer(
+                        finished, status = self._start_file_transfer(
                             f, filetransfer, overall_progress_update)
 
                         if self.is_cancelled():
                             break
 
-                        if increment == 2:
+                        if status == FILE_TRANSFERRED:
                             file_checksums_to_annotate.append(f.id)
-                        else:
-                            number_of_skipped_files += increment
+                        elif status == FILE_SKIPPED:
+                            number_of_skipped_files += 1
                 except Exception as e:
                     exception = e
                     break
@@ -186,10 +190,10 @@ def _start_file_transfer(self, f, filetransfer, overall_progress_update):
         """
         Start to transfer the file from network/disk to the destination.
         Return value:
-            * True, 2 - successfully transfer the file.
-            * True, 1 - the file does not exist so it is skipped.
-            * True, 0 - the transfer is cancelled.
-            * False, 0 - the transfer fails and needs to retry.
+            * True, FILE_TRANSFERRED - successfully transfer the file.
+            * True, FILE_SKIPPED - the file does not exist so it is skipped.
+            * True, FILE_NOT_TRANSFERRED - the transfer is cancelled.
+            * False, FILE_NOT_TRANSFERRED - the transfer fails and needs to retry.
         """
         try:
             # Save the current progress value
@@ -197,23 +201,34 @@ def _start_file_transfer(self, f, filetransfer, overall_progress_update):
             original_progress = self.progresstrackers[0].get_progress()
 
             with filetransfer, self.start_progress(total=filetransfer.total_size) as file_dl_progress_update:
-                # If size of the source file is smaller than the the size
-                # indicated in the database, it's very likely that the source
-                # file is corrupted. Skip this file.
-                if filetransfer.total_size < f.file_size:
-                    e = "File {} is corrupted.".format(filetransfer.source)
-                    logger.error("An error occurred during content import: {}".format(e))
-                    overall_progress_update(f.file_size)
-                    return True, 1
-
                 for chunk in filetransfer:
                     if self.is_cancelled():
                         filetransfer.cancel()
-                        return True, 0
+                        return True, FILE_NOT_TRANSFERRED
                     length = len(chunk)
                     overall_progress_update(length)
                     file_dl_progress_update(length)
-            return True, 2
+
+                # Ensure that if for some reason the total file size for the transfer
+                # is less than what we have marked in the database that we make up
+                # the difference so that the overall progress is never incorrect.
+                # This could happen, for example for a local transfer if a file
+                # has been replaced or corrupted (which we catch below) or for
+                # a remote transfer if the peer to peer transfer is is compressing
+                # files using gzip.
+                overall_progress_update(f.file_size - filetransfer.total_size)
+
+                # If size of the destination file is smaller than the size
+                # indicated in the database, it's very likely that the source
+                # file is corrupted. Skip importing this file.
+                dest_file_size_on_disk = os.path.getsize(filetransfer.dest)
+                if dest_file_size_on_disk < f.file_size:
+                    e = "File {} is corrupted.".format(filetransfer.source)
+                    logger.error("An error occurred during content import: {}".format(e))
+                    os.remove(filetransfer.dest)
+                    return True, FILE_SKIPPED
+
+            return True, FILE_TRANSFERRED
 
         except Exception as e:
             logger.error("An error occurred during content import: {}".format(e))
@@ -229,10 +244,10 @@ def _start_file_transfer(self, f, filetransfer, overall_progress_update):
                 logger.info('Waiting for 30 seconds before retrying import: {}\n'.format(
                     filetransfer.source))
                 sleep(30)
-                return False, 0
+                return False, FILE_NOT_TRANSFERRED
             else:
                 overall_progress_update(f.file_size)
-                return True, 1
+                return True, FILE_SKIPPED
 
     def handle_async(self, *args, **options):
         if options['command'] == 'network':


Test patch:
diff --git a/kolibri/core/content/test/test_import_export.py b/kolibri/core/content/test/test_import_export.py
--- a/kolibri/core/content/test/test_import_export.py
+++ b/kolibri/core/content/test/test_import_export.py
@@ -5,14 +5,16 @@
 from django.core.management import call_command
 from django.test import TestCase
 from mock import call
+from mock import MagicMock
 from mock import patch
 from requests import Session
+from requests.exceptions import ChunkedEncodingError
 from requests.exceptions import ConnectionError
 from requests.exceptions import HTTPError
 from requests.exceptions import ReadTimeout
 from requests.exceptions import SSLError
-from requests.exceptions import ChunkedEncodingError
 
+from kolibri.core.content.models import ContentNode
 from kolibri.core.content.models import LocalFile
 from kolibri.utils.tests.helpers import override_option
 
@@ -164,11 +166,14 @@ def test_remote_cancel_after_file_copy_file_not_deleted(self, is_cancelled_mock,
         # If transfer is cancelled after transfer of first file
         local_path_1 = tempfile.mkstemp()[1]
         local_path_2 = tempfile.mkstemp()[1]
+        with open(local_path_1, "w") as f:
+            f.write("a")
         local_path_mock.side_effect = [local_path_1, local_path_2]
         remote_path_mock.return_value = 'notest'
         # Mock this __iter__ so that the filetransfer can be looped over
         FileDownloadMock.return_value.__iter__.return_value = ['one', 'two', 'three']
         FileDownloadMock.return_value.total_size = 1
+        FileDownloadMock.return_value.dest = local_path_1
         LocalFile.objects.filter(files__contentnode__channel_id=self.the_channel_id).update(file_size=1)
         call_command("importcontent", "network", self.the_channel_id)
         # Check that the command itself was also cancelled.
@@ -292,18 +297,52 @@ def test_local_import_oserror_permission_denied(self, path_mock, getsize_mock, l
             self.assertTrue('Permission denied' in logger_mock.call_args_list[0][0][0])
             annotation_mock.annotate_content.assert_called()
 
-    @patch('kolibri.core.content.utils.transfer.os.path.getsize', return_value=0)
+    @patch('kolibri.core.content.management.commands.importcontent.os.remove')
     @patch('kolibri.core.content.management.commands.importcontent.os.path.isfile', return_value=False)
     @patch('kolibri.core.content.management.commands.importcontent.paths.get_content_storage_file_path')
     @patch('kolibri.core.content.management.commands.importcontent.AsyncCommand.cancel')
     @patch('kolibri.core.content.management.commands.importcontent.AsyncCommand.is_cancelled', side_effect=[False, False, True, True])
-    def test_local_import_source_corrupted(self, is_cancelled_mock, cancel_mock, path_mock, isfile_mock, getsize_mock, annotation_mock):
+    def test_local_import_source_corrupted(self, is_cancelled_mock, cancel_mock, path_mock, isfile_mock, remove_mock, annotation_mock):
         local_src_path = tempfile.mkstemp()[1]
         local_dest_path = tempfile.mkstemp()[1]
         LocalFile.objects.filter(files__contentnode="32a941fb77c2576e8f6b294cde4c3b0c").update(file_size=1)
         path_mock.side_effect = [local_dest_path, local_src_path]
         call_command('importcontent', 'disk', self.the_channel_id, 'destination', node_ids=['32a941fb77c2576e8f6b294cde4c3b0c'])
         cancel_mock.assert_called_with()
+        remove_mock.assert_called_with(local_dest_path)
+
+    @patch('kolibri.core.content.management.commands.importcontent.os.path.isfile', return_value=False)
+    @patch('kolibri.core.content.management.commands.importcontent.paths.get_content_storage_file_path')
+    @patch('kolibri.core.content.management.commands.importcontent.AsyncCommand.cancel')
+    @patch('kolibri.core.content.management.commands.importcontent.AsyncCommand.is_cancelled', return_value=False)
+    def test_local_import_source_corrupted_full_progress(self, is_cancelled_mock, cancel_mock, path_mock, isfile_mock, annotation_mock):
+        """
+        Ensure that when a file is imported that does not match the file size in the database
+        that the overall progress tracking for the content import process is properly updated
+        to reflect the size of the file in the database, not the file on disk.
+        This is important, as the total progress for the overall process is measured against
+        the total file size recorded in the database for all files, not for the the
+        transferred file size.
+        """
+        local_src_path = tempfile.mkstemp()[1]
+        with open(local_src_path, 'w') as f:
+            f.write('This is just a test')
+        src_file_size = os.path.getsize(local_src_path)
+        expected_file_size = 10000
+        local_dest_path = tempfile.mkstemp()[1]
+        os.remove(local_dest_path)
+        # Delete all but one file associated with ContentNode to reduce need for mocking
+        files = ContentNode.objects.get(id="32a941fb77c2576e8f6b294cde4c3b0c").files.all()
+        first_file = files.first()
+        files.exclude(id=first_file.id).delete()
+        LocalFile.objects.filter(files__contentnode="32a941fb77c2576e8f6b294cde4c3b0c").update(file_size=expected_file_size)
+        path_mock.side_effect = [local_dest_path, local_src_path]
+        mock_overall_progress = MagicMock()
+        mock_file_progress = MagicMock()
+        with patch('kolibri.core.tasks.management.commands.base.ProgressTracker') as progress_mock:
+            progress_mock.return_value.__enter__.side_effect = [mock_overall_progress, mock_file_progress]
+            call_command('importcontent', 'disk', self.the_channel_id, 'destination', node_ids=['32a941fb77c2576e8f6b294cde4c3b0c'])
+            mock_overall_progress.assert_called_with(expected_file_size - src_file_size)
 
 
 @override_option("Paths", "CONTENT_DIR", tempfile.mkdtemp())