Repository: zulip/zulip

Issue:
The `delete_after_upload` flag in export_realm_wrapper should also delete the `output_dir`
https://github.com/zulip/zulip/blob/73a6f2a1a7fbfb939b8bdaded4148f9dcf8a6e81/zerver/lib/export.py#L1944-L1980

After a successful export and upload, the `delete_after_upload` flag will delete the `.tar.gz` file -- but will leave the full un-gzip'd `output_dir`.  This means that even successful exports will build up in `/tmp` over time.

We should make the `delete_after_upload` flag also delete the `output_dir`.


Discussion:
Hello @zulip/server-misc members, this issue was labeled with the "area: export/import" label, so you may want to check it out!

<!-- areaLabelAddition -->

Do we want to always delete the `output_dir` after completing the export?

I think the main reason to want to not do so is for development, and it seems like we could perhaps have a `--keep-unpacked-copy` option or something there.
@timabbott @alexmv we can also set different flags for zipped and unzipped copy, like `--delete-zipped-copy` and `--delete-unpacked-copy`. Adding `--keep-unpacked-copy` would require `--delete-after-upload` to be true.
I don't think we need different flags for the zipped vs unpacked version -- either I want a backup, or I don't, and what format I get doesn't really matter, since they're interchangeable.  And to Tim's point, I agree that defaulting to _not_ keeping a backup is probably the right default.
@alexmv Ok, I get it now. So we need need to add another flag `--keep-unpacked-copy`, which wouldn't delete `output_dir` if the flag is passed otherwise delete `output_dir` by default. I would like to work on this! 
@timabbott's preferences may differ, but here is what I would like:
 - Running `./manage.py export --upload` would default to leaving nothing left in `/tmp`
 - Running `./manage export` without `--upload` would leave a `.tar.gz` in `/tmp` and no unpacked directory
 - There is no `--delete-after-upload` flag, since that is the default.
 
 I don't think there's a need for a `--keep-unpacked-copy`, since making the directory from the successful tarball is easy, and an unsuccessful tarball will result in the unpacked directory still being on disk (since we won't have cleaned it up).
The `--keep-unpacked-copy` option (previous default behavior) is something that I used to use a lot when we were developing the export tool -- made the edit/refresh debugging cycle fast.  It may have outlived its purpose; I'd be fine with just removing the option and seeing if we ever find that we wish we had it.

So yeah, I think the above plan would be great.

Resolving patch:
diff --git a/zerver/lib/export.py b/zerver/lib/export.py
--- a/zerver/lib/export.py
+++ b/zerver/lib/export.py
@@ -1947,7 +1947,6 @@ def export_realm_wrapper(
     threads: int,
     upload: bool,
     public_only: bool,
-    delete_after_upload: bool,
     percent_callback: Optional[Callable[[Any], None]] = None,
     consent_message_id: Optional[int] = None,
 ) -> Optional[str]:
@@ -1958,9 +1957,8 @@ def export_realm_wrapper(
         public_only=public_only,
         consent_message_id=consent_message_id,
     )
-    print(f"Finished exporting to {output_dir}")
+    shutil.rmtree(output_dir)
     print(f"Tarball written to {tarball_path}")
-
     if not upload:
         return None
 
@@ -1971,12 +1969,10 @@ def export_realm_wrapper(
     public_url = zerver.lib.upload.upload_backend.upload_export_tarball(
         realm, tarball_path, percent_callback=percent_callback
     )
-    print()
-    print(f"Uploaded to {public_url}")
+    print(f"\nUploaded to {public_url}")
 
-    if delete_after_upload:
-        os.remove(tarball_path)
-        print(f"Successfully deleted the tarball at {tarball_path}")
+    os.remove(tarball_path)
+    print(f"Successfully deleted the tarball at {tarball_path}")
     return public_url
 
 
diff --git a/zerver/management/commands/export.py b/zerver/management/commands/export.py
--- a/zerver/management/commands/export.py
+++ b/zerver/management/commands/export.py
@@ -104,11 +104,6 @@ def add_arguments(self, parser: ArgumentParser) -> None:
             action="store_true",
             help="Whether to upload resulting tarball to s3 or LOCAL_UPLOADS_DIR",
         )
-        parser.add_argument(
-            "--delete-after-upload",
-            action="store_true",
-            help="Automatically delete the local tarball after a successful export",
-        )
         self.add_realm_args(parser, required=True)
 
     def handle(self, *args: Any, **options: Any) -> None:
@@ -215,7 +210,6 @@ def percent_callback(bytes_transferred: Any) -> None:
             threads=num_threads,
             upload=options["upload"],
             public_only=public_only,
-            delete_after_upload=options["delete_after_upload"],
             percent_callback=percent_callback,
             consent_message_id=consent_message_id,
         )
diff --git a/zerver/worker/queue_processors.py b/zerver/worker/queue_processors.py
--- a/zerver/worker/queue_processors.py
+++ b/zerver/worker/queue_processors.py
@@ -979,7 +979,6 @@ def failure_processor(event: Dict[str, Any]) -> None:
                     threads=6,
                     upload=True,
                     public_only=True,
-                    delete_after_upload=True,
                 )
             except Exception:
                 export_event.extra_data = orjson.dumps(


Test patch:
diff --git a/zerver/tests/test_management_commands.py b/zerver/tests/test_management_commands.py
--- a/zerver/tests/test_management_commands.py
+++ b/zerver/tests/test_management_commands.py
@@ -525,7 +525,6 @@ def test_command_with_consented_message_id(self) -> None:
                 realm=realm,
                 public_only=False,
                 consent_message_id=message.id,
-                delete_after_upload=False,
                 threads=mock.ANY,
                 output_dir=mock.ANY,
                 percent_callback=mock.ANY,