Repository: learningequality/kolibri

Issue:
Clean up previously generated CSV files
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

## Observed behavior
When a new CSV file is generated for session and summary logs, the previously generated files that are stored in the user's database are no longer accessible for download in the UI. A record of the new CSV file request is saved to the logger model `GenerateCSVLogRequest` which contains the facility id, log type (session or summary), the selected start date, the selected end date, and the date that the request was made. There should only ever be one record for each facility and log type combination. The information in this modal is used to scan the database folder `log_exports` and retrieve the previously generated file for download.

Note: For session and summary log files, the file naming convention has been updated to include a date-range selection, with the dates formatted as `YYYY-MM-DD`.
Previous naming convention:
- `{facility_name}_{last 4 digits of facility ID}_{log export type}.csv`

Current naming convention:
- `{facility_name}_{last 4 digits of facility ID}_{log export type}_from_{start date}_to_{end date}.csv`

## Expected behavior
We should remove any files that do not have a record in the database. One way to achieve this is by scanning the folder `log_export`, which contains all exported log files, using the predictable file naming convention, and cross-referencing them with the file records stored in the database. Any session or summary files that do not have a corresponding record in the database should be deleted.

With a 1:1 correlation with the `GenerateCSVLogRequest` record, the database should only contain one exported file for each session and summary log type. A new async task should be created for performing this cleanup, and can be queued whenever a new log export is generated. If the task has a predictable id, it can be ensured that there is only one cleanup task going at once.

More details on the task system can be found [here](https://kolibri-dev.readthedocs.io/en/develop/backend_architecture/tasks.html).

## User-facing consequences
Ensures the database is free from redundant files and optimizes the user's database storage.

## Acceptance Criteria

- [ ] All previously generated session and summary log file exports that lack a corresponding record in the database have been deleted.
- [ ] The previously generated CSV file should be automatically removed from the database upon generating a new one.
- [ ] New python code should be covered by unit tests.



Discussion:
Hi @LianaHarris360, I would like to give this issue a try. Can you share some more details or screenshots about how to reproduce and test this issue?

Thanks.
Hi @Jaspreet-singh-1032 Thank you for your interest in working on this. At the moment this issue needs some more details to be clarified before it is ready to be tackled. Once it is fully prepared and ready for contributions, I will follow up with you.
Sure @LianaHarris360 , thanks.
@Jaspreet-singh-1032 More context has been added and this issue is ready for contributions. This [PR](https://github.com/learningequality/kolibri/pull/10106) contains more details for reproducing/testing this flow. Feel free to ask any clarifying questions that you may have.
Thanks @LianaHarris360, Will check this one.
Hi! @LianaHarris360 Could I work on this issue alongside with you and @Jaspreet-singh-1032...
Hi @GeekGawd, @Jaspreet-singh-1032 is actively working on this. If they are open to collaboration, they can respond here to let you know, but it may be easier to explore another issue that you could potentially contribute to.
Hi @GeekGawd, I will check if it's possible to divide this into any sub-task and then let you know.
thanks.

Resolving patch:
diff --git a/kolibri/core/logger/management/commands/exportlogs.py b/kolibri/core/logger/management/commands/exportlogs.py
--- a/kolibri/core/logger/management/commands/exportlogs.py
+++ b/kolibri/core/logger/management/commands/exportlogs.py
@@ -15,6 +15,7 @@
 from kolibri.core.logger.csv_export import classes_info
 from kolibri.core.logger.csv_export import csv_file_generator
 from kolibri.core.logger.models import GenerateCSVLogRequest
+from kolibri.core.logger.tasks import log_exports_cleanup
 from kolibri.core.tasks.management.commands.base import AsyncCommand
 from kolibri.core.tasks.utils import get_current_job
 from kolibri.utils.time_utils import local_now
@@ -169,3 +170,4 @@ def handle_async(self, *args, **options):
                 "date_requested": local_now(),
             },
         )
+        log_exports_cleanup.enqueue()
diff --git a/kolibri/core/logger/tasks.py b/kolibri/core/logger/tasks.py
--- a/kolibri/core/logger/tasks.py
+++ b/kolibri/core/logger/tasks.py
@@ -3,13 +3,19 @@
 from django.core.management import call_command
 from rest_framework import serializers
 
+from kolibri.core.auth.management.commands.bulkexportusers import (
+    CSV_EXPORT_FILENAMES as USER_CSV_EXPORT_FILENAMES,
+)
 from kolibri.core.auth.models import Facility
 from kolibri.core.logger.csv_export import CSV_EXPORT_FILENAMES
+from kolibri.core.logger.models import GenerateCSVLogRequest
 from kolibri.core.tasks.decorators import register_task
 from kolibri.core.tasks.permissions import IsAdminForJob
 from kolibri.core.tasks.validation import JobValidator
 from kolibri.utils import conf
 
+LOGS_CLEANUP_JOB_ID = "18"
+
 
 def get_filepath(log_type, facility_id, start_date, end_date):
     facility = Facility.objects.get(id=facility_id)
@@ -25,6 +31,53 @@ def get_filepath(log_type, facility_id, start_date, end_date):
     return filepath
 
 
+def get_valid_logs_csv_filenames():
+    """
+    Returns a set of valid filenames that should exist
+    based on the objects stored in GenerateCSVLogRequest.
+    Any other files except these filenames should be removed.
+    """
+    valid_filenames_set = set()
+    log_requests = GenerateCSVLogRequest.objects.all()
+    for log_request in log_requests:
+        full_path = get_filepath(
+            log_request.log_type,
+            log_request.facility_id,
+            log_request.selected_start_date.strftime("%Y-%m-%d"),
+            log_request.selected_end_date.strftime("%Y-%m-%d"),
+        )
+        valid_filenames_set.add(os.path.basename(full_path))
+    return valid_filenames_set
+
+
+def get_valid_users_csv_filenames():
+    """
+    Returns a set of valid filenames that should exist
+    based on the objects stored in Facility.
+    """
+    valid_filenames_set = set()
+    facilities = Facility.objects.values("id", "name").all()
+    for facility in facilities:
+        file_name = USER_CSV_EXPORT_FILENAMES["user"].format(
+            facility["name"], facility["id"][:4]
+        )
+        valid_filenames_set.add(file_name)
+    return valid_filenames_set
+
+
+def get_valid_filenames():
+    """
+    Returns a union set of valid filenames
+    for log exports and users csv exports.
+    These filenames are valid and will not be
+    cleaned from log_exports_cleanup.
+    """
+    valid_logs_filenames = get_valid_logs_csv_filenames()
+    valid_users_filenames = get_valid_users_csv_filenames()
+    valid_filenames_set = valid_logs_filenames.union(valid_users_filenames)
+    return valid_filenames_set
+
+
 class ExportLogCSVValidator(JobValidator):
     facility = serializers.PrimaryKeyRelatedField(
         queryset=Facility.objects.all(), required=False
@@ -116,3 +169,18 @@ def exportsummarylogcsv(facility_id, **kwargs):
         kwargs.get("end_date"),
         kwargs.get("locale"),
     )
+
+
+@register_task(job_id=LOGS_CLEANUP_JOB_ID)
+def log_exports_cleanup():
+    """
+    Cleanup log_exports csv files that does not have
+    related reocord in GenerateCSVLogRequest model
+    """
+    logs_dir = os.path.join(conf.KOLIBRI_HOME, "log_export")
+    if not os.path.isdir(logs_dir):
+        return
+    valid_filenames_set = get_valid_filenames()
+    for filename in os.listdir(logs_dir):
+        if filename not in valid_filenames_set:
+            os.remove(os.path.join(logs_dir, filename))


Test patch:
diff --git a/kolibri/core/logger/test/test_api.py b/kolibri/core/logger/test/test_api.py
--- a/kolibri/core/logger/test/test_api.py
+++ b/kolibri/core/logger/test/test_api.py
@@ -5,10 +5,12 @@
 """
 import csv
 import datetime
+import os
 import sys
 import tempfile
 import uuid
 
+import mock
 import pytz
 from django.core.management import call_command
 from django.urls import reverse
@@ -21,11 +23,17 @@
 from .factory_logger import ContentSummaryLogFactory
 from .factory_logger import FacilityUserFactory
 from .helpers import EvaluationMixin
+from kolibri.core.auth.management.commands.bulkexportusers import (
+    CSV_EXPORT_FILENAMES as USER_CSV_EXPORT_FILENAMES,
+)
 from kolibri.core.auth.test.helpers import provision_device
 from kolibri.core.auth.test.test_api import FacilityFactory
 from kolibri.core.content.models import ChannelMetadata
 from kolibri.core.content.models import ContentNode
 from kolibri.core.logger.csv_export import labels
+from kolibri.core.logger.tasks import get_filepath
+from kolibri.core.logger.tasks import log_exports_cleanup
+from kolibri.utils import conf
 from kolibri.utils.time_utils import local_now
 
 
@@ -142,6 +150,66 @@ def test_csv_download_unicode_username(self):
             ).exists()
         )
 
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_cleanup(self, mock_enqueue):
+        # generate summary csv
+        log_type = "summary"
+        start_date = "2023-03-05 00:00:00"
+        end_date = "2023-03-10 00:00:00"
+        filepath = get_filepath(log_type, self.facility.id, start_date, end_date)
+        call_command(
+            "exportlogs",
+            log_type=log_type,
+            output_file=filepath,
+            overwrite=True,
+            start_date=start_date,
+            end_date=end_date,
+        )
+
+        # generate another summary csv
+        start_date_2 = "2023-03-11 00:00:00"
+        end_date_2 = "2023-03-12 00:00:00"
+        filepath_2 = get_filepath(log_type, self.facility.id, start_date_2, end_date_2)
+        call_command(
+            "exportlogs",
+            log_type=log_type,
+            output_file=filepath_2,
+            overwrite=True,
+            start_date=start_date_2,
+            end_date=end_date_2,
+        )
+        # generate users csv
+        call_command(
+            "bulkexportusers",
+            facility=self.facility.id,
+            overwrite=True,
+        )
+        # execute cleanup
+        # latest should persist and the old one should be deleted
+        log_exports_cleanup()
+
+        logs_dir = os.path.join(conf.KOLIBRI_HOME, "log_export")
+        # currently there are two file. logs export and users csv export
+        assert len(os.listdir(logs_dir)) == 2
+        assert os.path.basename(filepath_2) in os.listdir(logs_dir)
+        assert os.path.basename(filepath) not in os.listdir(logs_dir)
+
+        # make sure the csv file for the record saved in the database exists
+        log_request = GenerateCSVLogRequest.objects.get(log_type=log_type)
+        date_format = "%Y-%m-%d"
+        expected_file_path = get_filepath(
+            log_request.log_type,
+            log_request.facility_id,
+            log_request.selected_start_date.strftime(date_format),
+            log_request.selected_end_date.strftime(date_format),
+        )
+        expected_users_csv_file_path = USER_CSV_EXPORT_FILENAMES["user"].format(
+            self.facility.name, self.facility.id[:4]
+        )
+        assert os.path.basename(expected_file_path) in os.listdir(logs_dir)
+        assert expected_users_csv_file_path in os.listdir(logs_dir)
+        assert mock_enqueue.has_calls(2)
+
 
 class ContentSessionLogCSVExportTestCase(APITestCase):
 
@@ -280,6 +348,66 @@ def test_csv_download_no_completion_timestamp(self):
             ).exists()
         )
 
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_cleanup(self, mock_enqueue):
+        # generate session csv
+        log_type = "session"
+        start_date = "2023-03-05 00:00:00"
+        end_date = "2023-03-10 00:00:00"
+        filepath = get_filepath(log_type, self.facility.id, start_date, end_date)
+        call_command(
+            "exportlogs",
+            log_type=log_type,
+            output_file=filepath,
+            overwrite=True,
+            start_date=start_date,
+            end_date=end_date,
+        )
+
+        # generate another session csv
+        start_date_2 = "2023-03-11 00:00:00"
+        end_date_2 = "2023-03-12 00:00:00"
+        filepath_2 = get_filepath(log_type, self.facility.id, start_date_2, end_date_2)
+        call_command(
+            "exportlogs",
+            log_type=log_type,
+            output_file=filepath_2,
+            overwrite=True,
+            start_date=start_date_2,
+            end_date=end_date_2,
+        )
+        # generate users csv
+        call_command(
+            "bulkexportusers",
+            facility=self.facility.id,
+            overwrite=True,
+        )
+        # execute cleanup
+        # latest csv should persist and the old one should be deleted
+        log_exports_cleanup()
+
+        logs_dir = os.path.join(conf.KOLIBRI_HOME, "log_export")
+        # currently there are two file. logs export and users csv export
+        assert len(os.listdir(logs_dir)) == 2
+        assert os.path.basename(filepath_2) in os.listdir(logs_dir)
+        assert os.path.basename(filepath) not in os.listdir(logs_dir)
+
+        # make sure the csv file for the record saved in the database exists
+        log_request = GenerateCSVLogRequest.objects.get(log_type=log_type)
+        date_format = "%Y-%m-%d"
+        expected_file_path = get_filepath(
+            log_request.log_type,
+            log_request.facility_id,
+            log_request.selected_start_date.strftime(date_format),
+            log_request.selected_end_date.strftime(date_format),
+        )
+        expected_users_csv_file_path = USER_CSV_EXPORT_FILENAMES["user"].format(
+            self.facility.name, self.facility.id[:4]
+        )
+        assert os.path.basename(expected_file_path) in os.listdir(logs_dir)
+        assert expected_users_csv_file_path in os.listdir(logs_dir)
+        assert mock_enqueue.has_calls(2)
+
 
 class MasteryLogViewSetTestCase(EvaluationMixin, APITestCase):
     def test_summary(self):
diff --git a/kolibri/plugins/facility/test/test_api.py b/kolibri/plugins/facility/test/test_api.py
--- a/kolibri/plugins/facility/test/test_api.py
+++ b/kolibri/plugins/facility/test/test_api.py
@@ -7,6 +7,7 @@
 import os
 import uuid
 
+import mock
 import pytz
 from django.core.management import call_command
 from django.urls import reverse
@@ -15,6 +16,7 @@
 from kolibri.core.auth.test.helpers import DUMMY_PASSWORD
 from kolibri.core.auth.test.helpers import provision_device
 from kolibri.core.auth.test.test_api import FacilityFactory
+from kolibri.core.logger.tasks import log_exports_cleanup
 from kolibri.core.logger.test.factory_logger import ContentSessionLogFactory
 from kolibri.core.logger.test.factory_logger import ContentSummaryLogFactory
 from kolibri.core.logger.test.factory_logger import FacilityUserFactory
@@ -67,7 +69,8 @@ def setUpTestData(cls):
         cls.start_date = datetime.datetime(2020, 10, 21, tzinfo=pytz.UTC).isoformat()
         cls.end_date = local_now().isoformat()
 
-    def test_csv_download_anonymous_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_anonymous_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="summary",
@@ -89,7 +92,8 @@ def test_csv_download_anonymous_permissions(self):
         )
         self.assertEqual(response.status_code, 403)
 
-    def test_csv_download_non_admin_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_non_admin_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="summary",
@@ -116,7 +120,8 @@ def test_csv_download_non_admin_permissions(self):
         )
         self.assertEqual(response.status_code, 403)
 
-    def test_csv_download_admin_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_admin_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="summary",
@@ -167,7 +172,8 @@ def setUpTestData(cls):
         cls.start_date = datetime.datetime(2020, 10, 21, tzinfo=pytz.UTC).isoformat()
         cls.end_date = local_now().isoformat()
 
-    def test_csv_download_anonymous_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_anonymous_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="session",
@@ -189,7 +195,8 @@ def test_csv_download_anonymous_permissions(self):
         )
         self.assertEqual(response.status_code, 403)
 
-    def test_csv_download_non_admin_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_non_admin_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="session",
@@ -216,7 +223,8 @@ def test_csv_download_non_admin_permissions(self):
         )
         self.assertEqual(response.status_code, 403)
 
-    def test_csv_download_admin_permissions(self):
+    @mock.patch.object(log_exports_cleanup, "enqueue", return_value=None)
+    def test_csv_download_admin_permissions(self, mock_enqueue):
         call_command(
             "exportlogs",
             log_type="session",