internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: elastic/apm-agent-python

Issue:
Add support for custom transport json_serializer method
**Is your feature request related to a problem? Please describe.**
When I use 
```python
elasticapm.set_custom_context({'message from user': [{'key1': 'value1', 'key2': 'value2'}, {'key1': 'value3', 'key2': 'value4'}, ...]})
```
elastic apm web UI show in metadata my `message from user` something like:
|   |  |
| ------------- | ------------- |
| transaction.custom.message from user.key1  | value1  |
|   | value3 |
|   | ... |
| transaction.custom.message from user.key2  | value2  |
|   | value4 |
|   | ... |

but I expect:
|   |  |
| ------------- | ------------- |
| transaction.custom.message from user | [{'key1': 'value1', 'key2': 'value2'}, {'key1': 'value3, 'key2': 'value4'}, ...]}  |

this is not a problem when len('message from user') is not big, but when it more than 10 it will not good.

**Describe the solution you'd like**
It was great if when I create apm instance via `elasticapm.Client` I can pass parameter like `TRANSPORT_JSON_SERIALIZER` for example and associate it with my custom json serializer that will be used [here](https://github.com/elastic/apm-agent-python/blob/main/elasticapm/transport/base.py#L152).

**Describe alternatives you've considered**
Now it can be solved by cast list object to string representation like:
```python
elasticapm.set_custom_context({'message from user': orjson.dumps([{'key1': 'value1', 'key2': 'value2'}, {'key1': 'value3, 'key2': 'value4'}, ...])})
```
or any other json serialize library like json, simplejson, etc. But in this case I need to add orjson.dumps for every list values that I passed as value in set_custom_context method.


Discussion:

Resolving patch:
diff --git a/elasticapm/base.py b/elasticapm/base.py
--- a/elasticapm/base.py
+++ b/elasticapm/base.py
@@ -151,6 +151,9 @@ def __init__(self, config=None, **inline):
             "timeout": self.config.server_timeout,
             "processors": self.load_processors(),
         }
+        if config.transport_json_serializer:
+            transport_kwargs["json_serializer"] = config.transport_json_serializer
+
         self._api_endpoint_url = urllib.parse.urljoin(
             self.config.server_url if self.config.server_url.endswith("/") else self.config.server_url + "/",
             constants.EVENTS_API_PATH,
diff --git a/elasticapm/conf/__init__.py b/elasticapm/conf/__init__.py
--- a/elasticapm/conf/__init__.py
+++ b/elasticapm/conf/__init__.py
@@ -547,6 +547,7 @@ class Config(_ConfigBase):
     )
     service_node_name = _ConfigValue("SERVICE_NODE_NAME")
     environment = _ConfigValue("ENVIRONMENT")
+    transport_json_serializer = _ConfigValue("TRANSPORT_JSON_SERIALIZER")
     secret_token = _ConfigValue("SECRET_TOKEN")
     api_key = _ConfigValue("API_KEY")
     debug = _BoolConfigValue("DEBUG", default=False)


Test patch:
Repository: elastic/apm-agent-python

Issue:
[META 739] Add "s3/"-prefix to S3 destination.service.resource
See meta issue for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/739

[META 739] Add "s3/"-prefix to S3 destination.service.resource
See meta issue for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/739



Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/botocore.py b/elasticapm/instrumentation/packages/botocore.py
--- a/elasticapm/instrumentation/packages/botocore.py
+++ b/elasticapm/instrumentation/packages/botocore.py
@@ -123,7 +123,7 @@ def handle_s3(operation_name, service, instance, args, kwargs, context):
         bucket = ""
     signature = f"S3 {operation_name} {bucket}"
 
-    context["destination"]["service"] = {"name": span_subtype, "resource": bucket, "type": span_type}
+    context["destination"]["service"] = {"name": span_subtype, "resource": f"s3/{bucket}", "type": span_type}
 
     return HandlerInfo(signature, span_type, span_subtype, span_action, context)
 


Test patch:
diff --git a/tests/instrumentation/asyncio_tests/aiobotocore_tests.py b/tests/instrumentation/asyncio_tests/aiobotocore_tests.py
--- a/tests/instrumentation/asyncio_tests/aiobotocore_tests.py
+++ b/tests/instrumentation/asyncio_tests/aiobotocore_tests.py
@@ -119,7 +119,7 @@ async def test_s3(instrument, elasticapm_client, session):
         assert span["context"]["destination"]["port"] == LOCALSTACK_ENDPOINT_URL.port
         assert span["context"]["destination"]["cloud"]["region"] == "us-east-1"
         assert span["context"]["destination"]["service"]["name"] == "s3"
-        assert span["context"]["destination"]["service"]["resource"] == "xyz"
+        assert span["context"]["destination"]["service"]["resource"] == "s3/xyz"
         assert span["context"]["destination"]["service"]["type"] == "storage"
     assert spans[0]["name"] == "S3 CreateBucket xyz"
     assert spans[0]["action"] == "CreateBucket"
diff --git a/tests/instrumentation/botocore_tests.py b/tests/instrumentation/botocore_tests.py
--- a/tests/instrumentation/botocore_tests.py
+++ b/tests/instrumentation/botocore_tests.py
@@ -116,7 +116,7 @@ def test_s3(instrument, elasticapm_client):
         assert span["context"]["destination"]["port"] == LOCALSTACK_ENDPOINT_URL.port
         assert span["context"]["destination"]["cloud"]["region"] == "us-east-1"
         assert span["context"]["destination"]["service"]["name"] == "s3"
-        assert span["context"]["destination"]["service"]["resource"] == "xyz"
+        assert span["context"]["destination"]["service"]["resource"] == "s3/xyz"
         assert span["context"]["destination"]["service"]["type"] == "storage"
         assert span["context"]["http"]["request"]["id"]
     assert spans[0]["name"] == "S3 CreateBucket xyz"
Repository: elastic/apm-agent-python

Issue:
[META 759] Add S3 Bucket and Object Key to S3 instrumentation
See meta issue and spec for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/759
- Spec issue: https://github.com/elastic/apm/issues/760



Discussion:

Resolving patch:
diff --git a/elasticapm/instrumentation/packages/botocore.py b/elasticapm/instrumentation/packages/botocore.py
--- a/elasticapm/instrumentation/packages/botocore.py
+++ b/elasticapm/instrumentation/packages/botocore.py
@@ -116,11 +116,19 @@ def handle_s3(operation_name, service, instance, args, kwargs, context):
     span_type = "storage"
     span_subtype = "s3"
     span_action = operation_name
-    if len(args) > 1 and "Bucket" in args[1]:
-        bucket = args[1]["Bucket"]
+    if len(args) > 1:
+        bucket = args[1].get("Bucket", "")
+        key = args[1].get("Key", "")
     else:
         # TODO handle Access Points
         bucket = ""
+        key = ""
+    if bucket or key:
+        context["otel_attributes"] = {}
+        if bucket:
+            context["otel_attributes"]["aws.s3.bucket"] = bucket
+        if key:
+            context["otel_attributes"]["aws.s3.key"] = key
     signature = f"S3 {operation_name} {bucket}"
 
     context["destination"]["service"] = {"name": span_subtype, "resource": f"s3/{bucket}", "type": span_type}


Test patch:
diff --git a/tests/instrumentation/botocore_tests.py b/tests/instrumentation/botocore_tests.py
--- a/tests/instrumentation/botocore_tests.py
+++ b/tests/instrumentation/botocore_tests.py
@@ -123,6 +123,8 @@ def test_s3(instrument, elasticapm_client):
     assert spans[0]["action"] == "CreateBucket"
     assert spans[1]["name"] == "S3 PutObject xyz"
     assert spans[1]["action"] == "PutObject"
+    assert spans[1]["otel"]["attributes"]["aws.s3.bucket"] == "xyz"
+    assert spans[1]["otel"]["attributes"]["aws.s3.key"] == "abc"
     assert spans[2]["name"] == "S3 ListObjects xyz"
     assert spans[2]["action"] == "ListObjects"
Repository: elastic/apm-agent-python

Issue:
DroppedSpan has no attribute 'transaction' error in grpc/client_interceptor.py
We are experiencing the following error using automatic instrumentation of gRPC requests for transactions with more than 500 spans: `AttributeError: 'DroppedSpan' object has no attribute 'transaction'`. The error occurs at the following line
https://github.com/elastic/apm-agent-python/blob/479a69d61f728ec24de5fb7504c02dcbfe4f4c8a/elasticapm/contrib/grpc/client_interceptor.py#L197
**To Reproduce**

1. Start a transaction
2. Produce more spans than the transaction_max_spans
3. Call a gRPC endpoint

**Environment (please complete the following information)**
- agent version: 6.15.1



Discussion:
Thanks for the report! This should be pretty easy to fix.

Resolving patch:
diff --git a/elasticapm/contrib/grpc/client_interceptor.py b/elasticapm/contrib/grpc/client_interceptor.py
--- a/elasticapm/contrib/grpc/client_interceptor.py
+++ b/elasticapm/contrib/grpc/client_interceptor.py
@@ -35,7 +35,7 @@
 
 import elasticapm
 from elasticapm.conf import constants
-from elasticapm.traces import Span
+from elasticapm.traces import DroppedSpan, SpanType
 from elasticapm.utils import default_ports
 
 
@@ -193,8 +193,8 @@ def intercept_unary_unary(self, continuation, client_call_details, request):
     #       Future interface, though it may not.
     #     """
 
-    def attach_traceparent(self, client_call_details: _ClientCallDetails, span: Span):
-        if not span.transaction:
+    def attach_traceparent(self, client_call_details: _ClientCallDetails, span: SpanType) -> _ClientCallDetails:
+        if isinstance(span, DroppedSpan) or not span.transaction:
             return client_call_details
         meta = list(client_call_details.metadata) if client_call_details.metadata else []
         if constants.TRACEPARENT_HEADER_NAME not in meta:


Test patch:
diff --git a/tests/contrib/grpc/grpc_client_tests.py b/tests/contrib/grpc/grpc_client_tests.py
--- a/tests/contrib/grpc/grpc_client_tests.py
+++ b/tests/contrib/grpc/grpc_client_tests.py
@@ -219,6 +219,16 @@ def test_grpc_client_unsampled_transaction(instrument, sending_elasticapm_client
     assert len(payloads) == 1  # only the server_version request
 
 
+@pytest.mark.parametrize("sending_elasticapm_client", [{"transaction_max_spans": 1}], indirect=True)
+def test_grpc_client_max_spans(instrument, sending_elasticapm_client, grpc_client_and_server_url):
+    grpc_client, _ = grpc_client_and_server_url
+    transaction = sending_elasticapm_client.begin_transaction("request")
+    _ = grpc_client.GetServerResponse(Message(message="foo"))
+    _ = grpc_client.GetServerResponse(Message(message="bar"))
+    sending_elasticapm_client.end_transaction("grpc-test")
+    assert transaction.dropped_spans == 1
+
+
 def extract_events_from_payload(service_name, payloads):
     payloads = [
         payload for payload in payloads if payload and payload[0]["metadata"]["service"]["name"] == service_name
Repository: elastic/apm-agent-python

Issue:
[META 808] Make `include_process_args` an opt-in option
See meta issue and spec for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/808
- Spec issue: https://github.com/elastic/apm/issues/809

[META 808] Make `include_process_args` an opt-in option
See meta issue and spec for the description and details:
- Meta issue: https://github.com/elastic/apm/issues/808
- Spec issue: https://github.com/elastic/apm/issues/809



Discussion:
We don't currently have `include_process_args`. Checking on the [meta issue](https://github.com/elastic/apm/issues/808#issuecomment-1607984126) if this is relevant to us.

Edit: This issue includes adding the `include_process_args` config value.
We don't currently have `include_process_args`. Checking on the [meta issue](https://github.com/elastic/apm/issues/808#issuecomment-1607984126) if this is relevant to us.

Edit: This issue includes adding the `include_process_args` config value.

Resolving patch:
diff --git a/elasticapm/base.py b/elasticapm/base.py
--- a/elasticapm/base.py
+++ b/elasticapm/base.py
@@ -370,12 +370,14 @@ def get_service_info(self):
         return result
 
     def get_process_info(self):
-        return {
+        result = {
             "pid": os.getpid(),
             "ppid": os.getppid() if hasattr(os, "getppid") else None,
-            "argv": sys.argv,
             "title": None,  # Note: if we implement this, the value needs to be wrapped with keyword_field
         }
+        if self.config.include_process_args:
+            result["argv"] = sys.argv
+        return result
 
     def get_system_info(self):
         system_data = {
diff --git a/elasticapm/conf/__init__.py b/elasticapm/conf/__init__.py
--- a/elasticapm/conf/__init__.py
+++ b/elasticapm/conf/__init__.py
@@ -691,6 +691,7 @@ class Config(_ConfigBase):
         ],
         default=TRACE_CONTINUATION_STRATEGY.CONTINUE,
     )
+    include_process_args = _BoolConfigValue("INCLUDE_PROCESS_ARGS", default=False)
 
     @property
     def is_recording(self):


Test patch:
diff --git a/tests/client/client_tests.py b/tests/client/client_tests.py
--- a/tests/client/client_tests.py
+++ b/tests/client/client_tests.py
@@ -75,14 +75,17 @@ def test_service_info_node_name(elasticapm_client):
 
 
 def test_process_info(elasticapm_client):
-    with mock.patch.object(sys, "argv", ["a", "b", "c"]):
-        process_info = elasticapm_client.get_process_info()
+    process_info = elasticapm_client.get_process_info()
     assert process_info["pid"] == os.getpid()
     if hasattr(os, "getppid"):
         assert process_info["ppid"] == os.getppid()
     else:
         # Windows + Python 2.7
         assert process_info["ppid"] is None
+    assert "argv" not in process_info
+    elasticapm_client.config.update("1", include_process_args=True)
+    with mock.patch.object(sys, "argv", ["a", "b", "c"]):
+        process_info = elasticapm_client.get_process_info()
     assert process_info["argv"] == ["a", "b", "c"]
Repository: openvstorage/framework

Issue:
Wrong prefix for alba fragment cache
While investigating an issue I noticed this in the config of a proxy:

```
"fragment_cache": ["alba", {
"albamgr_cfg_url": "etcd://127.0.0.1:2379/ovs/vpools/e4d07c5c-5106-44e4-8fea-5d36e0f0be9f/proxies/95ee2c88-c861-4b7f-abaf-64169fd73c85/config/abm_aa", 
"cache_on_write": true, 
"manifest_cache_size": 17179869184, 
"cache_on_read": true, 
"bucket_strategy": ["1-to-1", {"prefix": "flash-roub", "preset": "default"
}]}]
```

As you can see there's a fragment cache of type alba configured (to be used for flash acceleration).
The prefix used for the cache however is the name of the flash backend, while I expected it to be the name of the backend (or some other identifier related to the backend) that is being accelerated.
This choice of prefix may lead to problems when the flash alba is also used to accelerate other backends. I strongly recommend changing the prefix used for caching.
I would also recommend adding a prefix/suffix to the prefix to identify this prefix is used as a cache.

That last remark would similarly also apply to when a backend is used as an osd in another backend.

There's also a difference in the prefixes for cache vs for alba-as-osd: one uses some name of the backend, and the other uses some guid. Maybe you want to streamline both into
{name}_{guid}_{fragment_cache/alba_osd} ?



Discussion:

Resolving patch:
diff --git a/ovs/lib/storagerouter.py b/ovs/lib/storagerouter.py
--- a/ovs/lib/storagerouter.py
+++ b/ovs/lib/storagerouter.py
@@ -799,7 +799,7 @@ def add_vpool(parameters):
                 fragment_cache_info = ['none']
             elif use_accelerated_alba is True:
                 fragment_cache_info = ['alba', {'albamgr_cfg_url': 'etcd://127.0.0.1:2379{0}'.format(config_tree.format('abm_aa')),
-                                                'bucket_strategy': ['1-to-1', {'prefix': vpool.metadata[storagerouter.guid]['name'],
+                                                'bucket_strategy': ['1-to-1', {'prefix': vpool.guid,
                                                                                'preset': vpool.metadata[storagerouter.guid]['preset']}],
                                                 'manifest_cache_size': 16 * 1024 * 1024 * 1024,
                                                 'cache_on_read': fragment_cache_on_read,
@@ -818,7 +818,7 @@ def add_vpool(parameters):
                 'fragment_cache': fragment_cache_info,
                 'transport': 'rdma' if has_rdma else 'tcp',
                 'albamgr_cfg_url': 'etcd://127.0.0.1:2379{0}'.format(config_tree.format('abm'))
-            }), raw=True)
+            }, indent=4), raw=True)
 
         storagedriver_config = StorageDriverConfiguration('storagedriver', vpool.guid, storagedriver.storagedriver_id)
         storagedriver_config.load()


Test patch:
Repository: openvstorage/framework

Issue:
Disable SHM by default
When running multiple vPools, SHM is currently having problems:

```
2016-06-23 14:32:53 176166 +0200 - perf-roub-04 - 125592/0x00007f382d7fa700 - volumedriverfs/FuseInterface - 000000000000c719 - info - run_: fuse loop exited
2016-06-23 14:32:53 176349 +0200 - perf-roub-04 - 125592/0x00007f3948a94940 - volumedriverfs/FuseInterface - 000000000000c71a - info - operator(): stopping network (xio) server
2016-06-23 14:32:53 177217 +0200 - perf-roub-04 - 125592/0x00007f3948a94940 - volumedriverfs/FuseInterface - 000000000000c71b - info - operator(): waiting for network (xio) thread
 to finish
2016-06-23 14:32:53 177307 +0200 - perf-roub-04 - 125592/0x00007f3948a94940 - volumedriverfs/FuseInterface - 000000000000c71c - info - operator(): stopping SHM server
2016-06-23 14:32:54 173291 +0200 - perf-roub-04 - 125592/0x00007f384dffb700 - volumedriverfs/MTServer - 000000000000c71d - info - work: You have 2 connections running 
```

Here we see that the volumedriver does not want to shutdown because it is waiting for SHM (from another vPool).

Best case scenario, you don't know how many vPools you will create so switch it off by default.
`"fs_enable_shm_interface": "1",` --> `"fs_enable_shm_interface": "0",`



Discussion:
I believe @redlicha has a patch?

The required changesets are still on a branch - https://github.com/openvstorage/volumedriver/tree/feature/shm-support-for-multiple-instances . They do require modification of the users of the shm interface as now the vpool + voldrv ID need to be specified, IOW blktap / qmu / tgt / ... need to be adapted to this in a user-visible way as these IDs need to be passed in by them.

@khenderick , @kvanhijf ,
I don't think disabling this for a while should be a big deal, right ?

Nope, shouldn't be too hard.

If we change the default please create a ticket to reverse.


Resolving patch:
diff --git a/ovs/lib/storagerouter.py b/ovs/lib/storagerouter.py
--- a/ovs/lib/storagerouter.py
+++ b/ovs/lib/storagerouter.py
@@ -843,7 +843,7 @@ def add_vpool(parameters):
             tlog_multiplier = current_vpool_configuration['tlog_multiplier']
             sco_factor = float(current_vpool_configuration['write_buffer']) / tlog_multiplier / sco_size
 
-        filesystem_config = {'fs_enable_shm_interface': 1,
+        filesystem_config = {'fs_enable_shm_interface': 0,
                              'fs_metadata_backend_arakoon_cluster_nodes': [],
                              'fs_metadata_backend_mds_nodes': [],
                              'fs_metadata_backend_type': 'MDS',


Test patch:
Repository: openvstorage/framework

Issue:
Storage Router Detail page: add vDisk tab
Currently it is hard to find out which vDisks are attached to which Storage Router. With the fact that we allows vDisks to migrate between datacenters it only become more complicated. Hence adding the vDisk tab to the Storage Router Detail page would make things easier to find out.

Use same layout as on the vpool detail page. Icon can be the vDisks icon as used on the vPool detail page. Use http://fontawesome.io/icon/server/ for the physical disk tab.



Discussion:

Resolving patch:
diff --git a/webapps/api/backend/views/vdisks.py b/webapps/api/backend/views/vdisks.py
--- a/webapps/api/backend/views/vdisks.py
+++ b/webapps/api/backend/views/vdisks.py
@@ -44,15 +44,20 @@ class VDiskViewSet(viewsets.ViewSet):
     @required_roles(['read'])
     @return_list(VDisk)
     @load()
-    def list(self, vpoolguid=None, query=None):
+    def list(self, vpoolguid=None, storagerouterguid=None, query=None):
         """
         Overview of all vDisks
         :param vpoolguid: Guid of the vPool to retrieve its disks
+        :param storagerouterguid: Guid of the StorageRouter to retrieve its disks
         :param query: A query to be executed if required
         """
         if vpoolguid is not None:
             vpool = VPool(vpoolguid)
             return vpool.vdisks
+        if storagerouterguid is not None:
+            storagerouter = StorageRouter(storagerouterguid)
+            return DataList(VDisk, {'type': DataList.where_operator.AND,
+                                    'items': [('guid', DataList.operator.IN, storagerouter.vdisks_guids)]})
         if query is not None:
             return DataList(VDisk, query)
         return VDiskList.get_vdisks()


Test patch:
Repository: openvstorage/framework

Issue:
vPool GUI work
- Add vPool wizard - Step 1
  - [x]  Remove backend type selection
    -Add vPool wizard - Step 2
  - [x] "Fragment Cache" should become "Cache"
  - [x] "Place Fragment Cache on disk" should be "Use local disk"
  - [x] "Use another ALBA Backend as Fragment Cache" should be "Use another Backend Cache"
- Add vPool wizard - Step 3
  - [x] Replace "StorageDriver Caching Information" by "Storage Driver Write Buffer"
  - [x] Remove Strategy
  - [x] Remove Deduplication
- Add vPool wizard - Step 4
  - [x]  https://github.com/openvstorage/framework/issues/783
- vPool overview page
  - [x] Remove Type, Connection and Login and replace by OVS Backend & preset
- vPool detail page
  - [x] Add backend info https://github.com/openvstorage/framework/issues/735
  - [x] Under configuration remove "Cache strategy", "Dedupe Mode"
  - [x] Under Mgmt actions add a link to the accelerated ALBA for each storage router (link to the backend)
  - [x] Under Mgmt actions make the Storage routers clickable


Discussion:
Some clarification around vPool overview page. The result should be following columns:

 status|name|backend (preset)|stored data|IOPS

https://github.com/openvstorage/framework/pull/1130 --> openvstorage-2.7.5-rev.4273.d9bb7fb
https://github.com/openvstorage/framework-alba-plugin/pull/267 --> openvstorage-backend-1.7.5-rev.811.c1b83e6


Resolving patch:
diff --git a/ovs/dal/migration/ovsmigrator.py b/ovs/dal/migration/ovsmigrator.py
--- a/ovs/dal/migration/ovsmigrator.py
+++ b/ovs/dal/migration/ovsmigrator.py
@@ -57,7 +57,6 @@ def migrate(previous_version):
             from ovs.dal.hybrids.client import Client
             from ovs.dal.hybrids.j_rolegroup import RoleGroup
             from ovs.dal.hybrids.j_roleclient import RoleClient
-            from ovs.dal.hybrids.backendtype import BackendType
             from ovs.dal.hybrids.servicetype import ServiceType
             from ovs.dal.hybrids.branding import Branding
             from ovs.dal.lists.backendtypelist import BackendTypeList
@@ -266,4 +265,11 @@ def migrate(previous_version):
                 vpool.metadata = new_metadata
                 vpool.save()
 
+            # Removal of READ role
+            from ovs.dal.lists.diskpartitionlist import DiskPartitionList
+            for partition in DiskPartitionList.get_partitions():
+                if 'READ' in partition.roles:
+                    partition.roles.remove('READ')
+                    partition.save()
+
         return OVSMigrator.THIS_VERSION


Test patch: