Repository: aws-powertools/powertools-lambda-python

Issue:
Feature request: Clear log buffer before starting next one
### Use case

Original issue: https://github.com/aws-powertools/powertools-lambda-typescript/issues/3705

We are currently not clearing the buffer before the next invocation, which may mean that customer are experiencing excessive memory usage as it may store unflushed buffer from previous executions.

### Solution/User Experience

Clear the buffer if a new invocation arrives.

### Alternative solutions

```markdown

```

### Acknowledgment

- [x] This feature request meets [Powertools for AWS Lambda (Python) Tenets](https://docs.powertools.aws.dev/lambda/python/latest/#tenets)
- [x] Should this be considered in other Powertools for AWS Lambda languages? i.e. [Java](https://github.com/aws-powertools/powertools-lambda-java/), [TypeScript](https://github.com/aws-powertools/powertools-lambda-typescript/), and [.NET](https://github.com/aws-powertools/powertools-lambda-dotnet/)

Discussion:
@github-actions: ### ⚠️COMMENT VISIBILITY WARNING⚠️
This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a [team member](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/MAINTAINERS.md#current-maintainers) or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.@github-actions: This is now released under [3.9.0](https://github.com/aws-powertools/powertools-lambda-python/releases/tag/v3.9.0) version!

Solution patch:
diff --git a/aws_lambda_powertools/logging/logger.py b/aws_lambda_powertools/logging/logger.py
index 58b3c50ca..3e65c3fab 100644
--- a/aws_lambda_powertools/logging/logger.py
+++ b/aws_lambda_powertools/logging/logger.py
@@ -1127,6 +1127,12 @@ class Logger:
         tracer_id = get_tracer_id()
 
         if tracer_id and self._buffer_config:
+            if not self._buffer_cache.get(tracer_id):
+                # Detect new Lambda invocation context and reset buffer to maintain log isolation
+                # Ensures logs from previous invocations do not leak into current execution
+                # Prevent memory excessive usage
+                self._buffer_cache.clear()
+
             log_record: dict[str, Any] = _create_buffer_record(
                 level=level,
                 msg=msg,
diff --git a/docs/core/logger.md b/docs/core/logger.md
index 5d7c7941b..caf338185 100644
--- a/docs/core/logger.md
+++ b/docs/core/logger.md
@@ -681,13 +681,16 @@ sequenceDiagram
 
 4. **What happens if the log buffer reaches its maximum size?** Older logs are removed from the buffer to make room for new logs. This means that if the buffer is full, you may lose some logs if they are not flushed before the buffer reaches its maximum size. When this happens, we emit a warning when flushing the buffer to indicate that some logs have been dropped.
 
-5. **What timestamp is used when I flush the logs?** The timestamp preserves the original time when the log record was created. If you create a log record at 11:00:10 and flush it at 11:00:25, the log line will retain its original timestamp of 11:00:10.
+5. **How is the log size of a log line calculated?**
+The log size is calculated based on the size of the log line in bytes. This includes the size of the log message, any exception (if present), the log line location, additional keys, and the timestamp.
 
-6. **What happens if I try to add a log line that is bigger than max buffer size?** The log will be emitted directly to standard output and not buffered. When this happens, we emit a warning to indicate that the log line was too big to be buffered.
+6. **What timestamp is used when I flush the logs?** The timestamp preserves the original time when the log record was created. If you create a log record at 11:00:10 and flush it at 11:00:25, the log line will retain its original timestamp of 11:00:10.
 
-7. **What happens if Lambda times out without flushing the buffer?** Logs that are still in the buffer will be lost.
+7. **What happens if I try to add a log line that is bigger than max buffer size?** The log will be emitted directly to standard output and not buffered. When this happens, we emit a warning to indicate that the log line was too big to be buffered.
 
-8. **Do child loggers inherit the buffer?**  No, child loggers do not inherit the buffer from their parent logger but only the buffer configuration. This means that if you create a child logger, it will have its own buffer and will not share the buffer with the parent logger.
+8. **What happens if Lambda times out without flushing the buffer?** Logs that are still in the buffer will be lost.
+
+9. **Do child loggers inherit the buffer?**  No, child loggers do not inherit the buffer from their parent logger but only the buffer configuration. This means that if you create a child logger, it will have its own buffer and will not share the buffer with the parent logger.
 
 ### Built-in Correlation ID expressions
 


Test patch:
diff --git a/tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py b/tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py
index 1e5a104be..44e87d610 100644
--- a/tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py
+++ b/tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py
@@ -470,7 +470,12 @@ def test_buffer_configuration_propagation_across_child_logger_instances(stdout,
     assert primary_logger._buffer_cache != secondary_logger._buffer_cache
 
 
-def test_logger_buffer_is_cleared_between_lambda_invocations(stdout, service_name, monkeypatch, lambda_context):
+def test_logger_buffer_is_cleared_between_lambda_invocations_with_decorator(
+    stdout,
+    service_name,
+    monkeypatch,
+    lambda_context,
+):
     # Set initial trace ID for first Lambda invocation
     monkeypatch.setenv(constants.XRAY_TRACE_ID_ENV, "1-67c39786-5908a82a246fb67f3089263f")
 
@@ -491,3 +496,30 @@ def test_logger_buffer_is_cleared_between_lambda_invocations(stdout, service_nam
 
     # THEN Verify buffer for the original trace ID is cleared
     assert not logger._buffer_cache.get("1-67c39786-5908a82a246fb67f3089263f")
+
+
+def test_logger_buffer_is_cleared_between_lambda_invocations_without_decoration(
+    stdout,
+    service_name,
+    monkeypatch,
+    lambda_context,
+):
+    # Set initial trace ID for first Lambda invocation
+    monkeypatch.setenv(constants.XRAY_TRACE_ID_ENV, "1-67c39786-5908a82a246fb67f3089263f")
+
+    # GIVEN A logger configured with specific buffer parameters
+    logger_buffer_config = LoggerBufferConfig(max_bytes=10240)
+    logger = Logger(level="DEBUG", service=service_name, stream=stdout, buffer_config=logger_buffer_config)
+
+    def handler(event, context):
+        logger.debug("debug line")
+
+    # WHEN First Lambda invocation with initial trace ID
+    handler({}, lambda_context)
+
+    # WHEN New Lambda invocation arrives with different trace ID
+    monkeypatch.setenv(constants.XRAY_TRACE_ID_ENV, "2-ABC39786-5908a82a246fb67f3089263f")
+    handler({}, lambda_context)
+
+    # THEN Verify buffer for the original trace ID is cleared
+    assert not logger._buffer_cache.get("1-67c39786-5908a82a246fb67f3089263f")