Repository: python-babel/babel

Issue:
Memory Leak Detected
## Overview Description
There appears to be a memory leak generated by repeated calls to _dates.parse_pattern(dt, format, locale)_ if the function is called with a wide variety of different formats and locales.

This is because each time a new DateTimePattern is created for a new (format, locale), the object is cached to __pattern_cache_ (dict), which grows endlessly.

babel/dates.py:1598 (Babel 2.9.1)
```
def parse_pattern(pattern):
    """Parse date, time, and datetime format patterns."""
    ...

    # here is the problem
     _pattern_cache[pattern] = pat = DateTimePattern(pattern, u''.join(result))
```

Perhaps a better design could be to simply _lru_cache_ the _dates.parse_pattern()_ function ?
```
from functools import lru_cache

@lru_cache(maxsize=1000)
def parse_pattern(pattern):
    """Parse date, time, and datetime format patterns."""
    ...

```

## Steps to Reproduce

```
from datetime import datetime

from babel.localedata import locale_identifiers
from babel.dates import format_datetime

from pympler import tracker # track memory leaks(=> https://github.com/pympler/pympler)

# show initial memory usage
tr = tracker.SummaryTracker()
tr.print_diff()

# create some random datetime
d = datetime(2007, 4, 1, 13, 27, 53)

# create some datetime formats
custom_formats = [  r"M/d/yy, h:mm a" # short
                    ,r"MMM d, y, h:mm:ss a" # medium
                    ,r"MMMM d, y 'at' h:mm:ss a z" # long
                    ,r"EEEE, MMMM d, y 'at' h:mm:ss a zzzz" # full

                    ,r"EEEE, MMMM d, y 'at' hh:mm:ss zzz" # shorter timezone
                    ,r"EEEE, MMMM d, y 'at' hh:mm:ss zzzz" # full, 24hr
                        
                    ,r"EEEE, MMMM d, y 'at' hh:mm:ss"
                    ,r"EEEE, MMMM d, y 'at' h:mm:ss a"

                    ,r"EEEE, d MMM y hh:mm:ss"
                    ,r"EEEE, d MMM y h:mm:ss a"

                    ,r"d MMM y hh:mm:ss"
                    ,r"d MMM y h:mm:ss a"
                    ]

# call format_datetime for all locale/format combinations, about 9.4k combinations
for locale_name in locale_identifiers():
    for custom_format in custom_formats:
        s = format_datetime(d, locale=locale_name, format=custom_format)

# show difference in memory usage since start
tr.print_diff()




```

## Actual Results

**Initial Memory Snapshot**
                  types |   # objects |    total size

                   list |        3750 |     318.95 KB
                    str |        3747 |     260.45 KB
                    int |         817 |      22.34 KB

**Final Memory Snapshot**
                            types |   # objects |   total size

                             dict |      272282 |    113.17 MB
                              str |       21809 |      1.51 MB
                             list |       12416 |      1.12 MB
      babel.dates.DateTimePattern |        9668 |    453.19 KB
                            tuple |        6829 |    385.02 KB
      babel.numbers.NumberPattern |        7550 |    353.91 K

## Expected Results

## Reproducibility

## Additional Information


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) Report
> Merging [#967](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) (152a9e3) into [master](https://codecov.io/gh/python-babel/babel/commit/14e7c00eef2592823256c833a9180f374c77f7d4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) (14e7c00) will **decrease** coverage by `1.32%`.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##           master     #967      +/-   ##
==========================================
- Coverage   90.88%   89.56%   -1.32%     
==========================================
  Files          25       25              
  Lines        4334     4333       -1     
==========================================
- Hits         3939     3881      -58     
- Misses        395      452      +57     
```


| [Impacted Files](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel) | Coverage Δ | |
|---|---|---|
| [babel/dates.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvZGF0ZXMucHk=) | `86.37% <100.00%> (-0.87%)` | :arrow_down: |
| [babel/localedata.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvbG9jYWxlZGF0YS5weQ==) | `95.04% <100.00%> (-0.87%)` | :arrow_down: |
| [babel/localtime/\_win32.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvbG9jYWx0aW1lL193aW4zMi5weQ==) | `25.92% <0.00%> (-37.04%)` | :arrow_down: |
| [babel/localtime/\_helpers.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvbG9jYWx0aW1lL19oZWxwZXJzLnB5) | `42.30% <0.00%> (-19.24%)` | :arrow_down: |
| [babel/localtime/\_\_init\_\_.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvbG9jYWx0aW1lL19faW5pdF9fLnB5) | `75.00% <0.00%> (-8.34%)` | :arrow_down: |
| [babel/support.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvc3VwcG9ydC5weQ==) | `77.38% <0.00%> (-7.94%)` | :arrow_down: |
| [babel/messages/frontend.py](https://codecov.io/gh/python-babel/babel/pull/967?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel#diff-YmFiZWwvbWVzc2FnZXMvZnJvbnRlbmQucHk=) | `86.64% <0.00%> (-0.75%)` | :arrow_down: |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=python-babel)


Patch:
diff --git a/babel/dates.py b/babel/dates.py
index ce439ee..78c7fac 100644
--- a/babel/dates.py
+++ b/babel/dates.py
@@ -19,6 +19,7 @@ from __future__ import annotations
 
 import re
 import warnings
+from functools import lru_cache
 from typing import TYPE_CHECKING, SupportsInt
 
 try:
@@ -1667,10 +1668,8 @@ PATTERN_CHARS: dict[str, list[int] | None] = {
 #: in order of decreasing magnitude.
 PATTERN_CHAR_ORDER = "GyYuUQqMLlwWdDFgEecabBChHKkjJmsSAzZOvVXx"
 
-_pattern_cache = {}
 
-
-def parse_pattern(pattern: str) -> DateTimePattern:
+def parse_pattern(pattern: str | DateTimePattern) -> DateTimePattern:
     """Parse date, time, and datetime format patterns.
 
     >>> parse_pattern("MMMMd").format
@@ -1693,10 +1692,11 @@ def parse_pattern(pattern: str) -> DateTimePattern:
     """
     if isinstance(pattern, DateTimePattern):
         return pattern
+    return _cached_parse_pattern(pattern)
 
-    if pattern in _pattern_cache:
-        return _pattern_cache[pattern]
 
+@lru_cache(maxsize=1024)
+def _cached_parse_pattern(pattern: str) -> DateTimePattern:
     result = []
 
     for tok_type, tok_value in tokenize_pattern(pattern):
@@ -1710,9 +1710,7 @@ def parse_pattern(pattern: str) -> DateTimePattern:
             result.append('%%(%s)s' % (fieldchar * fieldnum))
         else:
             raise NotImplementedError(f"Unknown token type: {tok_type}")
-
-    _pattern_cache[pattern] = pat = DateTimePattern(pattern, ''.join(result))
-    return pat
+    return DateTimePattern(pattern, ''.join(result))
 
 
 def tokenize_pattern(pattern: str) -> list[tuple[str, str | tuple[str, int]]]:
diff --git a/babel/localedata.py b/babel/localedata.py
index f765a1e..a9c7c75 100644
--- a/babel/localedata.py
+++ b/babel/localedata.py
@@ -20,6 +20,7 @@ import sys
 import threading
 from collections import abc
 from collections.abc import Iterator, Mapping, MutableMapping
+from functools import lru_cache
 from itertools import chain
 from typing import Any
 
@@ -74,28 +75,24 @@ def exists(name: str) -> bool:
     return True if file_found else bool(normalize_locale(name))
 
 
+@lru_cache(maxsize=None)
 def locale_identifiers() -> list[str]:
     """Return a list of all locale identifiers for which locale data is
     available.
 
-    This data is cached after the first invocation in `locale_identifiers.cache`.
-
-    Removing the `locale_identifiers.cache` attribute or setting it to `None`
-    will cause this function to re-read the list from disk.
+    This data is cached after the first invocation.
+    You can clear the cache by calling `locale_identifiers.cache_clear()`.
 
     .. versionadded:: 0.8.1
 
     :return: a list of locale identifiers (strings)
     """
-    data = getattr(locale_identifiers, 'cache', None)
-    if data is None:
-        locale_identifiers.cache = data = [
-            stem
-            for stem, extension in
-            (os.path.splitext(filename) for filename in os.listdir(_dirname))
-            if extension == '.dat' and stem != 'root'
-        ]
-    return data
+    return [
+        stem
+        for stem, extension in
+        (os.path.splitext(filename) for filename in os.listdir(_dirname))
+        if extension == '.dat' and stem != 'root'
+    ]
 
 
 def load(name: os.PathLike[str] | str, merge_inherited: bool = True) -> dict[str, Any]:


Test patch:
diff --git a/tests/test_localedata.py b/tests/test_localedata.py
index 913922e..75c34b1 100644
--- a/tests/test_localedata.py
+++ b/tests/test_localedata.py
@@ -110,18 +110,15 @@ def test_locale_identifiers_cache(monkeypatch):
         rv = original_listdir(*args)
         listdir_calls.append((args, rv))
         return rv
-    monkeypatch.setattr(localedata.os, 'listdir', listdir_spy)
-
-    # In case we've already run some tests...
-    if hasattr(localedata.locale_identifiers, 'cache'):
-        del localedata.locale_identifiers.cache
 
+    monkeypatch.setattr(localedata.os, 'listdir', listdir_spy)
+    localedata.locale_identifiers.cache_clear()
     assert not listdir_calls
-    assert localedata.locale_identifiers()
+    l = localedata.locale_identifiers()
     assert len(listdir_calls) == 1
-    assert localedata.locale_identifiers() is localedata.locale_identifiers.cache
+    assert localedata.locale_identifiers() is l
     assert len(listdir_calls) == 1
-    localedata.locale_identifiers.cache = None
+    localedata.locale_identifiers.cache_clear()
     assert localedata.locale_identifiers()
     assert len(listdir_calls) == 2