Repository: elastic/apm-agent-python

Issue:
RuntimeError: dictionary changed size during iteration
**Describe the bug**

In one of our services, we're occasionally seeing a `RuntimeError: dictionary changed size during iteration` error within `elasticapm` code.

Here's the stack trace:

```
File "/usr/local/lib/python3.9/site-packages/cheroot/server.py", line 1300, in communicate
    req.respond()
  File "/usr/local/lib/python3.9/site-packages/cheroot/server.py", line 1090, in respond
    self.server.gateway(self).respond()
  File "/usr/local/lib/python3.9/site-packages/cheroot/wsgi.py", line 140, in respond
    response = self.req.server.wsgi_app(self.env, self.start_response)
  File "/usr/local/lib/python3.9/site-packages/cherrypy/_cptree.py", line 302, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/connexion/apps/abstract.py", line 254, in __call__
    return self.app(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2548, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1704, in handle_exception
    got_request_exception.send(self, exception=e)
  File "/usr/local/lib/python3.9/site-packages/blinker/base.py", line 263, in send
    return [(receiver, receiver(sender, **kwargs))
  File "/usr/local/lib/python3.9/site-packages/blinker/base.py", line 263, in <listcomp>
    return [(receiver, receiver(sender, **kwargs))
  File "/usr/local/lib/python3.9/site-packages/elasticapm/contrib/flask/__init__.py", line 103, in handle_exception
    self.client.capture_exception(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 279, in capture_exception
    return self.capture("Exception", exc_info=exc_info, handled=handled, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 248, in capture
    data = self._build_msg_for_logging(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/base.py", line 508, in _build_msg_for_logging
    result = handler.capture(self, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/events.py", line 168, in capture
    chained_cause = Exception.capture(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/events.py", line 168, in capture
    chained_cause = Exception.capture(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/events.py", line 96, in capture
    frames = get_stack_info(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/stacks.py", line 318, in get_stack_info
    result = get_frame_info(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/stacks.py", line 287, in get_frame_info
    f_locals = {varname: locals_processor_func(var) for varname, var in f_locals.items()}
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/stacks.py", line 287, in <dictcomp>
    f_locals = {varname: locals_processor_func(var) for varname, var in f_locals.items()}
  File "/usr/local/lib/python3.9/site-packages/elasticapm/events.py", line 103, in <lambda>
    locals_processor_func=lambda local_var: varmap(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/__init__.py", line 68, in varmap
    ret = func(name, var, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/events.py", line 104, in <lambda>
    lambda k, val: shorten(
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/encoding.py", line 202, in shorten
    var = transform(var)
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/encoding.py", line 133, in transform
    ret = dict((to_unicode(k), transform_rec(v)) for k, v in value.items())
  File "/usr/local/lib/python3.9/site-packages/elasticapm/utils/encoding.py", line 133, in <genexpr>
    ret = dict((to_unicode(k), transform_rec(v)) for k, v in value.items())
RuntimeError: dictionary changed size during iteration
```

**To Reproduce**

We're unsure of how to reproduce this but it isn't a one-off bug.

**Environment (please complete the following information)**
- OS: Linux
- Python version: 3.9.13
- Framework and version [e.g. Django 2.1]: Flask 2.2.2
- APM Server version: Unsure
- Agent version: 6.13.2


Discussion:

Resolving patch:
diff --git a/elasticapm/utils/encoding.py b/elasticapm/utils/encoding.py
--- a/elasticapm/utils/encoding.py
+++ b/elasticapm/utils/encoding.py
@@ -130,7 +130,8 @@ class value_type(list):
         except AttributeError:
             ret = None
     elif isinstance(value, dict):
-        ret = dict((to_unicode(k), transform_rec(v)) for k, v in value.items())
+        # iterate over a copy of the dictionary to avoid "dictionary changed size during iteration" issues
+        ret = dict((to_unicode(k), transform_rec(v)) for k, v in value.copy().items())
     elif isinstance(value, str):
         ret = to_unicode(value)
     elif isinstance(value, bytes):


Test patch: