Repository: piskvorky/smart_open

Issue:
S3 SinglepartWriter writes on exception when garbage collected
#### Problem description

When there is an unhandled exception raised in context of [SinglepartWriter](https://github.com/piskvorky/smart_open/blob/b3d862106d1d22f891ec4bb1bacc9d092b42fcfe/smart_open/s3.py#L1108) side-effect occurs when writer is garbage-collected which results in unwanted write of partial file into S3. 

```
2024-04-20T06:02:19.817140Z [debug    ] Parsed JSON record             aws_request_id=00000000-0000-0000-0000-000000000000 instance_id=cf450c4a-21b7-452b-ad79-291cb87b11ab records_count=24 target_uri=s3://vf-localstack-nora-pii-data-retention/anonymized/vf-localstack-nora-pii-data-retention/data/vf-da-prod-nora-cdp-blueconic-dck-consumer-profiles-1-2023-03-25-03-46-20-0482e5d3-5733-4ed6-b836-a6bdd2401a2d.gz trace_id=00000000-0000-0000-0000-000000000000 uri=s3://vf-localstack-nora-pii-data-retention/data/vf-da-prod-nora-cdp-blueconic-dck-consumer-profiles-1-2023-03-25-03-46-20-0482e5d3-5733-4ed6-b836-a6bdd2401a2d.gz
Traceback (most recent call last):
  File "/Users/tsokolowski/Dev/code-werkz/vfc/vfn/gdf_pii_data_retention/src/lambda/anonymizer-s3/local.py", line 43, in <module>
    output = handler(payload, SimpleNamespace(aws_request_id=trace_id))
  File "/Users/tsokolowski/Dev/code-werkz/vfc/vfn/gdf_pii_data_retention/src/lambda/anonymizer-s3/src/anonymizer_s3/app.py", line 31, in handler
    anonymize(settings, di_container)(inbound_payload)
  File "/Users/tsokolowski/Dev/code-werkz/vfc/vfn/gdf_pii_data_retention/src/lib/piilib/src/piilib/s3/anonymizer/anonymize.py", line 127, in _
    _, lookups_hits = dispatch(task)
  File "/Users/tsokolowski/.pyenv/versions/3.9.18/lib/python3.9/functools.py", line 888, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/Users/tsokolowski/Dev/code-werkz/vfc/vfn/gdf_pii_data_retention/src/lib/piilib/src/piilib/s3/anonymizer/files/json_file.py", line 247, in _
    for raw_record_in, record_delimiter in json_parse(fin):
  File "/Users/tsokolowski/Dev/code-werkz/vfc/vfn/gdf_pii_data_retention/src/lib/piilib/src/piilib/s3/files/json_file.py", line 71, in json_parse
    buff = io.StringIO(old_buff.read())
KeyboardInterrupt
2024-04-20T06:02:20.791680Z [debug    ] smart_open.s3.SinglepartWriter('vf-localstack-nora-pii-data-retention', 'anonymized/vf-localstack-nora-pii-data-retention/data/vf-da-prod-nora-cdp-blueconic-dck-consumer-profiles-1-2023-03-25-03-46-20-0482e5d3-5733-4ed6-b836-a6bdd2401a2d.gz'): direct upload finished [smart_open.s3] target_uri=s3://vf-localstack-nora-pii-data-retention/anonymized/vf-localstack-nora-pii-data-retention/data/vf-da-prod-nora-cdp-blueconic-dck-consumer-profiles-1-2023-03-25-03-46-20-0482e5d3-5733-4ed6-b836-a6bdd2401a2d.gz uri=s3://vf-localstack-nora-pii-data-retention/data/vf-da-prod-nora-cdp-blueconic-dck-consumer-profiles-1-2023-03-25-03-46-20-0482e5d3-5733-4ed6-b836-a6bdd2401a2d.gz
```

Reason for this behaviour is `SinglepartWriter` inherits `io.BufferedIOBase` which invokes `close()` in `__del__()` descriptior.

#### Steps/code to reproduce the problem
- set `smart_open` logger level to DEBUG
- raise an unhandled exception in context of [SinglepartWriter](https://github.com/piskvorky/smart_open/blob/b3d862106d1d22f891ec4bb1bacc9d092b42fcfe/smart_open/s3.py#L1108)
- wait for the writer to be garbage collected
- observe `smart_open.s3.SinglepartWriter` log records indicating S3 write was performed

#### Versions
```
macOS-14.2.1-x86_64-i386-64bit
Python 3.9.18 (main, Nov 30 2023, 12:53:32)
[Clang 15.0.0 (clang-1500.0.40.1)]
smart_open 7.0.4
```


Discussion:
hi @donsokolone :wave:

how about setting `self._buf = None` in [`terminate`](https://github.com/piskvorky/smart_open/blob/v7.0.4/smart_open/s3.py#L1211)? then `close` is a no-op by the time the SinglepartWriter is garbage collected, [analogous to MultipartWriter](https://github.com/piskvorky/smart_open/blob/v7.0.4/smart_open/s3.py#L1038).

cc @mpenkov
@ddelange This is exactly what the fix should be, as I mentioned in #763. I will PR it in few moments.



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 85fc7d31..575f1f5c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -482,3 +482,4 @@ The old `smart_open.smart_open` function is deprecated, but continues to work as
 
   - support for multistream bzip files (PR #9, @pombredanne)
   - introduce this CHANGELOG
+
diff --git a/smart_open/s3.py b/smart_open/s3.py
index 09433ad7..60ae2a99 100644
--- a/smart_open/s3.py
+++ b/smart_open/s3.py
@@ -1209,8 +1209,8 @@ def write(self, b):
         return length
 
     def terminate(self):
-        """Nothing to cancel in single-part uploads."""
-        return
+        self._buf = None
+        logger.debug('%s: terminated singlepart upload', self)
 
     #
     # Internal methods.


Test patch:
diff --git a/smart_open/tests/test_s3.py b/smart_open/tests/test_s3.py
index ff44ad4f..61aec986 100644
--- a/smart_open/tests/test_s3.py
+++ b/smart_open/tests/test_s3.py
@@ -795,6 +795,26 @@ def test_str(self):
         with smart_open.s3.open(BUCKET_NAME, 'key', 'wb', multipart_upload=False) as fout:
             assert str(fout) == "smart_open.s3.SinglepartWriter('test-smartopen', 'key')"
 
+    def test_ensure_no_side_effects_on_exception(self):
+        class WriteError(Exception):
+            pass
+
+        s3_resource = _resource("s3")
+        obj = s3_resource.Object(BUCKET_NAME, KEY_NAME)
+
+        # wrap in closure to ease writer dereferencing
+        def _run():
+            with smart_open.s3.open(BUCKET_NAME, obj.key, "wb", multipart_upload=False) as fout:
+                fout.write(b"this should not be written")
+                raise WriteError
+
+        try:
+            _run()
+        except WriteError:
+            pass
+        finally:
+            self.assertRaises(s3_resource.meta.client.exceptions.NoSuchKey, obj.get)
+
 
 ARBITRARY_CLIENT_ERROR = botocore.client.ClientError(error_response={}, operation_name='bar')