Repository: learningequality/kolibri

Issue:
Content selection checkboxes do not show content in updated version
### Observed behavior

Using the IMPORT MORE function in Kolibri shows two videos in the testing folder:
![Screen Shot 2020-02-05 at 1 06 15 PM](https://user-images.githubusercontent.com/163966/73869511-5b83ca00-4818-11ea-8463-65af90721070.png)


### Expected behavior
There should be three videos in that folder, two alrady on device and one new 
![Screen Shot 2020-02-05 at 1 06 15 PM](https://user-images.githubusercontent.com/163966/73869704-ce8d4080-4818-11ea-8ed0-8b2dd654efb1.png)

see [on studio](https://studio.learningequality.org/channels/2d9ab082d771408eb7b48cd7d888e2e6/edit/4c98f9d).

The [db file](http://studio.learningequality.org/content/databases/2d9ab082d771408eb7b48cd7d888e2e6.sqlite3) served by Studio is v2 and it contains the new video.

Kolibri also seems to have read the correct DB file during UPDATE CHANNEL wizard, because the UI shows there are 7 content items in the channel and 6 already on device:
![Screen Shot 2020-02-05 at 1 11 48 PM](https://user-images.githubusercontent.com/163966/73869878-17dd9000-4819-11ea-8d50-4208575bcee8.png)
so problem seems to be just related to the selection component.




### User-facing consequences
Users will not be able to import new content added to their channel.

### Errors and logs

No errors in the logs, just standard import logging.

#### During initial import of v1

```
INFO 2020-02-05 17:46:27,191 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:46:27,892 channel_import Importing ContentTag data
INFO 2020-02-05 17:46:27,894 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:46:27,895 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:46:27,895 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:46:27,896 channel_import Importing ContentNode data
INFO 2020-02-05 17:46:27,898 channel_import Importing Language data
INFO 2020-02-05 17:46:27,898 channel_import Importing File data
INFO 2020-02-05 17:46:27,899 channel_import Importing LocalFile data
INFO 2020-02-05 17:46:27,900 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:46:27,901 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:46:27,919 channel_import Channel 2d9ab082d771408eb7b48cd7d888e2e6 successfully imported into the database
INFO 2020-02-05 17:46:43,788 annotation Setting availability to True of 10 LocalFile objects based on passed in checksums
INFO 2020-02-05 17:46:43,802 annotation Setting availability of non-topic ContentNode objects based on LocalFile availability
INFO 2020-02-05 17:46:43,862 annotation Annotating ContentNode objects with children for 2 levels
INFO 2020-02-05 17:46:43,868 annotation Annotating ContentNode objects with children for level 2
INFO 2020-02-05 17:46:43,870 annotation Annotating ContentNode objects with children for level 1
```

After clicking VIEW CHANGES to preview what has changed in v1 --> v2:
```
INFO 2020-02-05 17:48:06,690 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:48:07,320 channel_import Importing ContentTag data
INFO 2020-02-05 17:48:07,321 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:48:07,321 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:48:07,322 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:48:07,323 channel_import Importing ContentNode data
INFO 2020-02-05 17:48:07,323 channel_import Importing Language data
INFO 2020-02-05 17:48:07,324 channel_import Importing File data
INFO 2020-02-05 17:48:07,325 channel_import Importing LocalFile data
INFO 2020-02-05 17:48:07,326 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:48:07,326 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:48:07,348 annotation Setting availability of LocalFile objects based on disk availability
INFO 2020-02-05 17:48:07,422 annotation Setting availability to True of 10 LocalFile objects based on passed in checksums
```

During the UPDATE CHANNEL:
```
INFO 2020-02-05 17:48:21,811 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:48:21,867 channel_import Older version 1 of channel 2d9ab082d771408eb7b48cd7d888e2e6 already exists in database; removing old entries so we can upgrade to version 2
INFO 2020-02-05 17:48:21,887 channel_import Importing ContentTag data
INFO 2020-02-05 17:48:21,888 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:48:21,889 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:48:21,890 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:48:21,890 channel_import Importing ContentNode data
INFO 2020-02-05 17:48:21,892 channel_import Importing Language data
INFO 2020-02-05 17:48:21,892 channel_import Importing File data
INFO 2020-02-05 17:48:21,893 channel_import Importing LocalFile data
INFO 2020-02-05 17:48:21,894 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:48:21,894 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:48:21,909 channel_import Channel 2d9ab082d771408eb7b48cd7d888e2e6 successfully imported into the database
INFO 2020-02-05 17:48:21,915 annotation Setting availability of non-topic ContentNode objects based on LocalFile availability
INFO 2020-02-05 17:48:21,930 annotation Annotating ContentNode objects with children for 2 levels
INFO 2020-02-05 17:48:21,935 annotation Annotating ContentNode objects with children for level 2
INFO 2020-02-05 17:48:21,938 annotation Annotating ContentNode objects with children for level 1
```



…

### Steps to reproduce
 - Import channel v1 form Studio
 - add content to channel on Studio and publish v2
 - use UPDATE channel and IMPORT MORE functionality to try to get new content


…

### Context
 * Kolibri version: 0.13
 * Operating system: linux
 * Browser: chome


Content selection checkboxes do not show content in updated version
### Observed behavior

Using the IMPORT MORE function in Kolibri shows two videos in the testing folder:
![Screen Shot 2020-02-05 at 1 06 15 PM](https://user-images.githubusercontent.com/163966/73869511-5b83ca00-4818-11ea-8463-65af90721070.png)


### Expected behavior
There should be three videos in that folder, two alrady on device and one new 
![Screen Shot 2020-02-05 at 1 06 15 PM](https://user-images.githubusercontent.com/163966/73869704-ce8d4080-4818-11ea-8ed0-8b2dd654efb1.png)

see [on studio](https://studio.learningequality.org/channels/2d9ab082d771408eb7b48cd7d888e2e6/edit/4c98f9d).

The [db file](http://studio.learningequality.org/content/databases/2d9ab082d771408eb7b48cd7d888e2e6.sqlite3) served by Studio is v2 and it contains the new video.

Kolibri also seems to have read the correct DB file during UPDATE CHANNEL wizard, because the UI shows there are 7 content items in the channel and 6 already on device:
![Screen Shot 2020-02-05 at 1 11 48 PM](https://user-images.githubusercontent.com/163966/73869878-17dd9000-4819-11ea-8d50-4208575bcee8.png)
so problem seems to be just related to the selection component.




### User-facing consequences
Users will not be able to import new content added to their channel.

### Errors and logs

No errors in the logs, just standard import logging.

#### During initial import of v1

```
INFO 2020-02-05 17:46:27,191 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:46:27,892 channel_import Importing ContentTag data
INFO 2020-02-05 17:46:27,894 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:46:27,895 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:46:27,895 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:46:27,896 channel_import Importing ContentNode data
INFO 2020-02-05 17:46:27,898 channel_import Importing Language data
INFO 2020-02-05 17:46:27,898 channel_import Importing File data
INFO 2020-02-05 17:46:27,899 channel_import Importing LocalFile data
INFO 2020-02-05 17:46:27,900 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:46:27,901 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:46:27,919 channel_import Channel 2d9ab082d771408eb7b48cd7d888e2e6 successfully imported into the database
INFO 2020-02-05 17:46:43,788 annotation Setting availability to True of 10 LocalFile objects based on passed in checksums
INFO 2020-02-05 17:46:43,802 annotation Setting availability of non-topic ContentNode objects based on LocalFile availability
INFO 2020-02-05 17:46:43,862 annotation Annotating ContentNode objects with children for 2 levels
INFO 2020-02-05 17:46:43,868 annotation Annotating ContentNode objects with children for level 2
INFO 2020-02-05 17:46:43,870 annotation Annotating ContentNode objects with children for level 1
```

After clicking VIEW CHANGES to preview what has changed in v1 --> v2:
```
INFO 2020-02-05 17:48:06,690 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:48:07,320 channel_import Importing ContentTag data
INFO 2020-02-05 17:48:07,321 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:48:07,321 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:48:07,322 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:48:07,323 channel_import Importing ContentNode data
INFO 2020-02-05 17:48:07,323 channel_import Importing Language data
INFO 2020-02-05 17:48:07,324 channel_import Importing File data
INFO 2020-02-05 17:48:07,325 channel_import Importing LocalFile data
INFO 2020-02-05 17:48:07,326 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:48:07,326 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:48:07,348 annotation Setting availability of LocalFile objects based on disk availability
INFO 2020-02-05 17:48:07,422 annotation Setting availability to True of 10 LocalFile objects based on passed in checksums
```

During the UPDATE CHANNEL:
```
INFO 2020-02-05 17:48:21,811 importchannel Downloading data for channel id 2d9ab082d771408eb7b48cd7d888e2e6
INFO 2020-02-05 17:48:21,867 channel_import Older version 1 of channel 2d9ab082d771408eb7b48cd7d888e2e6 already exists in database; removing old entries so we can upgrade to version 2
INFO 2020-02-05 17:48:21,887 channel_import Importing ContentTag data
INFO 2020-02-05 17:48:21,888 channel_import Importing ContentNode_has_prerequisite data
INFO 2020-02-05 17:48:21,889 channel_import Importing ContentNode_related data
INFO 2020-02-05 17:48:21,890 channel_import Importing ContentNode_tags data
INFO 2020-02-05 17:48:21,890 channel_import Importing ContentNode data
INFO 2020-02-05 17:48:21,892 channel_import Importing Language data
INFO 2020-02-05 17:48:21,892 channel_import Importing File data
INFO 2020-02-05 17:48:21,893 channel_import Importing LocalFile data
INFO 2020-02-05 17:48:21,894 channel_import Importing AssessmentMetaData data
INFO 2020-02-05 17:48:21,894 channel_import Importing ChannelMetadata data
INFO 2020-02-05 17:48:21,909 channel_import Channel 2d9ab082d771408eb7b48cd7d888e2e6 successfully imported into the database
INFO 2020-02-05 17:48:21,915 annotation Setting availability of non-topic ContentNode objects based on LocalFile availability
INFO 2020-02-05 17:48:21,930 annotation Annotating ContentNode objects with children for 2 levels
INFO 2020-02-05 17:48:21,935 annotation Annotating ContentNode objects with children for level 2
INFO 2020-02-05 17:48:21,938 annotation Annotating ContentNode objects with children for level 1
```



…

### Steps to reproduce
 - Import channel v1 form Studio
 - add content to channel on Studio and publish v2
 - use UPDATE channel and IMPORT MORE functionality to try to get new content


…

### Context
 * Kolibri version: 0.13
 * Operating system: linux
 * Browser: chome




Discussion:
@jonboiser said: The ‘coral reefs’ node, if you look at the contentnode_granular network call has a ‘importable: false’ flag, which is why it’s not being shown on the UI.

@rtibbles said: Ah, this is probably an issue with the importability annotation not being refreshed on version change then. Because they are expensive, they are cached for an hour. 


@ivanistheone said: the simplest solution is to clear these caches after a channel is updated.
I shared a video with @lyw07 demonstrating what occurred when I did this with 0.13.1b1. 
@ivanistheone Is there a workaround you know for this? If this is indeed a cache issue, then usually restarting the server would fix it.
Note that in this case the cache used for the import availability is persisted to disk for it to be shared across processes, so restarting will not clear this particular cache. Clearing it on channel upgrade seems like the simplest. Will do a quick PR for this tomorrow.
As a least resistance fix, I am only going to clear the cache for Studio annotation, as that is the most likely to change in this scenario - and also the simplest to clear.
Peer and disk import will rely on the cache timeout instead - which seems reasonable, as they will tend to update on much longer timescales.
@jonboiser said: The ‘coral reefs’ node, if you look at the contentnode_granular network call has a ‘importable: false’ flag, which is why it’s not being shown on the UI.

@rtibbles said: Ah, this is probably an issue with the importability annotation not being refreshed on version change then. Because they are expensive, they are cached for an hour. 


@ivanistheone said: the simplest solution is to clear these caches after a channel is updated.
I shared a video with @lyw07 demonstrating what occurred when I did this with 0.13.1b1. 
@ivanistheone Is there a workaround you know for this? If this is indeed a cache issue, then usually restarting the server would fix it.
Note that in this case the cache used for the import availability is persisted to disk for it to be shared across processes, so restarting will not clear this particular cache. Clearing it on channel upgrade seems like the simplest. Will do a quick PR for this tomorrow.
As a least resistance fix, I am only going to clear the cache for Studio annotation, as that is the most likely to change in this scenario - and also the simplest to clear.
Peer and disk import will rely on the cache timeout instead - which seems reasonable, as they will tend to update on much longer timescales.

Resolving patch:
diff --git a/kolibri/core/content/management/commands/importchannel.py b/kolibri/core/content/management/commands/importchannel.py
--- a/kolibri/core/content/management/commands/importchannel.py
+++ b/kolibri/core/content/management/commands/importchannel.py
@@ -11,6 +11,7 @@
 from ...utils.annotation import update_content_metadata
 from ...utils.import_export_content import retry_import
 from kolibri.core.content.models import ContentNode
+from kolibri.core.content.utils.importability_annotation import clear_channel_stats
 from kolibri.core.errors import KolibriUpgradeError
 from kolibri.core.tasks.management.commands.base import AsyncCommand
 from kolibri.core.tasks.utils import db_task_write_lock
@@ -185,6 +186,9 @@ def _start_file_transfer(  # noqa: C901
                                     update_content_metadata(
                                         channel_id, node_ids=node_ids
                                     )
+                            if import_ran:
+                                # Clear any previously set channel availability stats for this channel
+                                clear_channel_stats(channel_id)
                         except channel_import.ImportCancelError:
                             # This will only occur if is_cancelled is True.
                             pass
diff --git a/kolibri/core/content/utils/importability_annotation.py b/kolibri/core/content/utils/importability_annotation.py
--- a/kolibri/core/content/utils/importability_annotation.py
+++ b/kolibri/core/content/utils/importability_annotation.py
@@ -251,6 +251,21 @@ def get_channel_annotation_stats(channel_id, checksums=None):
 cache = CrossProcessCache(3600)
 
 
+CHANNEL_STATS_CACHED_KEYS = "CHANNEL_STATS_CACHED_KEYS_{channel_id}"
+
+
+# Used for tracking which keys are cached for which channel
+# we can then clear these when necessary
+def register_key_as_cached(key, channel_id):
+    cached_keys = cache.get(
+        CHANNEL_STATS_CACHED_KEYS.format(channel_id=channel_id), set()
+    )
+    cached_keys.add(key)
+    cache.set(
+        CHANNEL_STATS_CACHED_KEYS.format(channel_id=channel_id), cached_keys, None
+    )
+
+
 def get_channel_stats_from_disk(channel_id, drive_id):
     CACHE_KEY = "DISK_CHANNEL_STATS_{drive_id}_{channel_id}".format(
         drive_id=drive_id, channel_id=channel_id
@@ -259,6 +274,7 @@ def get_channel_stats_from_disk(channel_id, drive_id):
         checksums = get_available_checksums_from_disk(channel_id, drive_id)
         channel_stats = get_channel_annotation_stats(channel_id, checksums)
         cache.set(CACHE_KEY, channel_stats, 3600)
+        register_key_as_cached(CACHE_KEY, channel_id)
     else:
         channel_stats = cache.get(CACHE_KEY)
     return channel_stats
@@ -272,6 +288,7 @@ def get_channel_stats_from_peer(channel_id, peer_id):
         checksums = get_available_checksums_from_remote(channel_id, peer_id)
         channel_stats = get_channel_annotation_stats(channel_id, checksums)
         cache.set(CACHE_KEY, channel_stats, 3600)
+        register_key_as_cached(CACHE_KEY, channel_id)
     else:
         channel_stats = cache.get(CACHE_KEY)
     return channel_stats
@@ -282,6 +299,16 @@ def get_channel_stats_from_studio(channel_id):
     if CACHE_KEY not in cache:
         channel_stats = get_channel_annotation_stats(channel_id)
         cache.set(CACHE_KEY, channel_stats, 3600)
+        register_key_as_cached(CACHE_KEY, channel_id)
     else:
         channel_stats = cache.get(CACHE_KEY)
     return channel_stats
+
+
+def clear_channel_stats(channel_id):
+    cached_keys = cache.get(
+        CHANNEL_STATS_CACHED_KEYS.format(channel_id=channel_id), set()
+    )
+    for key in cached_keys:
+        cache.delete(key)
+    cache.set(CHANNEL_STATS_CACHED_KEYS.format(channel_id=channel_id), set(), None)
diff --git a/kolibri/core/utils/cache.py b/kolibri/core/utils/cache.py
--- a/kolibri/core/utils/cache.py
+++ b/kolibri/core/utils/cache.py
@@ -36,3 +36,8 @@ def set(self, key, value, timeout=NOTHING, version=None):
         caches["default"].set(key, value, timeout=timeout, version=version)
         if cache_options["CACHE_BACKEND"] != "redis":
             caches["process_cache"].set(key, value, timeout=timeout, version=version)
+
+    def delete(self, key, version=None):
+        caches["default"].delete(key, version=version)
+        if cache_options["CACHE_BACKEND"] != "redis":
+            caches["process_cache"].delete(key, version=version)


Test patch:
diff --git a/kolibri/core/content/test/test_import_export.py b/kolibri/core/content/test/test_import_export.py
--- a/kolibri/core/content/test/test_import_export.py
+++ b/kolibri/core/content/test/test_import_export.py
@@ -177,6 +177,33 @@ def test_remote_import_readtimeout(
         cancel_mock.assert_called_with()
         import_channel_mock.assert_not_called()
 
+    @patch(
+        "kolibri.core.content.management.commands.importchannel.paths.get_content_database_file_url"
+    )
+    @patch(
+        "kolibri.core.content.management.commands.importchannel.paths.get_content_database_file_path"
+    )
+    @patch(
+        "kolibri.core.content.management.commands.importchannel.transfer.FileDownload"
+    )
+    @patch("kolibri.core.content.management.commands.importchannel.clear_channel_stats")
+    def test_remote_successful_import_clears_stats_cache(
+        self,
+        channel_stats_clear_mock,
+        FileDownloadMock,
+        local_path_mock,
+        remote_path_mock,
+        start_progress_mock,
+        import_channel_mock,
+    ):
+        local_path = tempfile.mkstemp()[1]
+        local_path_mock.return_value = local_path
+        remote_path_mock.return_value = "notest"
+        FileDownloadMock.return_value.__iter__.return_value = ["one", "two", "three"]
+        import_channel_mock.return_value = True
+        call_command("importchannel", "network", self.the_channel_id)
+        self.assertTrue(channel_stats_clear_mock.called)
+
 
 @patch(
     "kolibri.core.content.management.commands.importcontent.lookup_channel_listing_status",