Repository: elastic/apm-agent-python

Issue:
Verify that `service.name` is required for data to be sent
We got reports of data from the Python agent making it to the APM Server without `service.name` being set. `service_name` is a required configuration, and we set `disable_send` to `True` if it is missing, so there might be a bug somewhere there.

https://github.com/elastic/apm-agent-python/blob/1d8f1835e6c0ea207b8321ec63ddd0a34784b296/elasticapm/conf/__init__.py#L255

https://github.com/elastic/apm-agent-python/blob/1d8f1835e6c0ea207b8321ec63ddd0a34784b296/elasticapm/base.py#L106-L110

cc @simitt 


Discussion:
What constitutes lack of `service.name`? Was it completely absent from the payload?

As far as I can tell, if `service.name` isn't set, Beni is right, we don't send anything.

That said, I wonder if we could be seeing an interaction with an odd value here?

For example, [our RegexValidator](https://github.com/elastic/apm-agent-python/blob/1d8f1835e6c0ea207b8321ec63ddd0a34784b296/elasticapm/conf/__init__.py#L255) uses the regex `"^[a-zA-Z0-9 _-]+$"`, which technically allows for a service name of `" "`, which could easily be stripped to an empty string (either by us or by the APM Server). Perhaps that fails validation on the APM server side or is dropped somewhere in the line?

Additionally, if a user set that config value to `False` or `None`, those would actually pass muster, because we convert to string in the validator. Perhaps these string values could be accidentally converted by either the APM server or by Elasticsearch and end up being `null`?

Those are the only two potential edge cases I can see, neither of which would be particularly easy to do accidentally. I haven't done any end to end testing yet, I've mostly just been following code paths.

@simitt Do either of those seem plausible or do I need to dig deeper? And who reported the initial issue? Can we get more information from that user?
@basepi service names consisting of only spaces are allowed and processed by APM Server, and the service would even show up in Kibana (with the name being blank). I don't think this would explain the issue. Strings such as `"None"` or `"False"` are processed as regular strings, also showing up in Kibana. 

Resolving patch:
diff --git a/elasticapm/conf/__init__.py b/elasticapm/conf/__init__.py
--- a/elasticapm/conf/__init__.py
+++ b/elasticapm/conf/__init__.py
@@ -69,9 +69,9 @@ def __get__(self, instance, owner):
         else:
             return self.default
 
-    def __set__(self, instance, value):
-        value = self._validate(instance, value)
-        instance._values[self.dict_key] = value
+    def __set__(self, config_instance, value):
+        value = self._validate(config_instance, value)
+        config_instance._values[self.dict_key] = value
 
     def _validate(self, instance, value):
         if value is None and self.required:
@@ -237,6 +237,11 @@ def update(self, config_dict=None, env_dict=None, inline_dict=None):
                     setattr(self, field, new_value)
                 except ConfigurationError as e:
                     self._errors[e.field_name] = str(e)
+            # if a field has not been provided by any config source, we have to check separately if it is required
+            if config_value.required and getattr(self, field) is None:
+                self._errors[config_value.dict_key] = "Configuration error: value for {} is required.".format(
+                    config_value.dict_key
+                )
 
     @property
     def values(self):
@@ -250,6 +255,12 @@ def values(self, values):
     def errors(self):
         return self._errors
 
+    def copy(self):
+        c = self.__class__()
+        c._errors = {}
+        c.values = self.values.copy()
+        return c
+
 
 class Config(_ConfigBase):
     service_name = _ConfigValue("SERVICE_NAME", validators=[RegexValidator("^[a-zA-Z0-9 _-]+$")], required=True)
@@ -393,8 +404,7 @@ def update(self, version, **config):
         :param config: a key/value map of new configuration
         :return: configuration errors, if any
         """
-        new_config = Config()
-        new_config.values = self._config.values.copy()
+        new_config = self._config.copy()
 
         # pass an empty env dict to ensure the environment doesn't get precedence
         new_config.update(inline_dict=config, env_dict={})


Test patch:
diff --git a/tests/config/tests.py b/tests/config/tests.py
--- a/tests/config/tests.py
+++ b/tests/config/tests.py
@@ -301,3 +301,17 @@ def test_capture_body_mapping(val, expected):
 def test_is_recording(enabled, recording, is_recording):
     c = Config(inline_dict={"enabled": enabled, "recording": recording, "service_name": "foo"})
     assert c.is_recording is is_recording
+
+
+def test_required_is_checked_if_field_not_provided():
+    class MyConfig(_ConfigBase):
+        this_one_is_required = _ConfigValue("this_one_is_required", type=int, required=True)
+        this_one_isnt = _ConfigValue("this_one_isnt", type=int, required=False)
+
+    assert MyConfig({"this_one_is_required": None}).errors
+    assert MyConfig({}).errors
+    assert MyConfig({"this_one_isnt": 1}).errors
+
+    c = MyConfig({"this_one_is_required": 1})
+    c.update({"this_one_isnt": 0})
+    assert not c.errors
diff --git a/tests/contrib/django/django_tests.py b/tests/contrib/django/django_tests.py
--- a/tests/contrib/django/django_tests.py
+++ b/tests/contrib/django/django_tests.py
@@ -1313,18 +1313,18 @@ def test_test_exception(urlopen_mock):
     assert "Success! We tracked the error successfully!" in output
 
 
-@pytest.mark.parametrize(
-    "django_elasticapm_client", [{"transport_class": "elasticapm.transport.http.Transport"}], indirect=True
-)
-def test_test_exception_fails(django_elasticapm_client):
+@mock.patch("elasticapm.transport.http.Transport.send")
+def test_test_exception_fails(mock_send):
     stdout = compat.StringIO()
+    mock_send.side_effect = Exception("boom")
     with override_settings(
-        ELASTIC_APM={"TRANSPORT_CLASS": "elasticapm.transport.http.Transport"},
+        ELASTIC_APM={"TRANSPORT_CLASS": "elasticapm.transport.http.Transport", "SERVICE_NAME": "testapp"},
         **middleware_setting(django.VERSION, ["foo", "elasticapm.contrib.django.middleware.TracingMiddleware"])
     ):
         call_command("elasticapm", "test", stdout=stdout, stderr=stdout)
     output = stdout.getvalue()
     assert "Oops" in output
+    assert "boom" in output
 
 
 def test_tracing_middleware_uses_test_client(client, django_elasticapm_client):