Repository: DataDog/integrations-extras

Issue:
Add support for filebeat version 7.x
**Docker image:** custom docker image based on `datadog/agent:7.28.0` with  this added to the `Dockerfile` `RUN agent integration install -r -t datadog-filebeat==1.0.0`

Since filebeat version `7.x`, the registry file is moved to [directory format][0]. Therefore, the current config will not work as the registry is not in a single file format anymore.

[Snippet][1] from the config example for filebeat:
```
instances:

    ## @param registry_file_path - string - required
    ## The absolute path to the registry file used by Filebeat.
    ##
    ## See https://www.elastic.co/guide/en/beats/filebeat/current/migration-registry-file.html
    #
  - registry_file_path: /var/lib/filebeat/registry
```

The error message from `docker logs datadog-agent`
```
2021-06-17 03:20:33 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | filebeat:71abd2e9bec66d7b | (filebeat.py:251) | Cannot read the registry log file at /usr/share/filebeat/data/registry: [Errno 21] Is a directory: '/usr/share/filebeat/data/registry'
```

So datadog-agent  and filebeat integration will not work with filebeat version 7.x for now.


[1]: https://github.com/DataDog/integrations-extras/blob/master/filebeat/datadog_checks/filebeat/data/conf.yaml.example#L46-L53
[0]: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-general-options.html#_registry_migrate_file



Discussion:
Hi, thank you for opening this issue
Filebeat is a community maintained integration, we welcome PRs and help from external contributors on this kind of integrations. Filebeat is owned by @wk8, i'm pinging them so they can have a look at it and decide if they want to add support for v7.x
Looks like a workaround is merged and released allowing registry file check to be ignored. Closing this issue for now.

Resolving patch:
diff --git a/filebeat/datadog_checks/filebeat/filebeat.py b/filebeat/datadog_checks/filebeat/filebeat.py
--- a/filebeat/datadog_checks/filebeat/filebeat.py
+++ b/filebeat/datadog_checks/filebeat/filebeat.py
@@ -261,6 +261,16 @@ def _parse_registry_file(self, registry_file_path):
         try:
             with open(registry_file_path) as registry_file:
                 return json.load(registry_file)
+
+        except ValueError:
+            contents = []
+            with open(registry_file_path) as registry_file:
+                for line in registry_file:
+                    content = json.loads(line)
+                    if 'op' not in content.keys():
+                        contents.append(content['v'])
+            return contents
+
         except IOError as ex:
             self.log.error("Cannot read the registry log file at %s: %s", registry_file_path, ex)
 


Test patch:
diff --git a/filebeat/tests/fixtures/malformed_json_registry.json b/filebeat/tests/fixtures/malformed_json_registry.json
--- a/filebeat/tests/fixtures/malformed_json_registry.json
+++ b/filebeat/tests/fixtures/malformed_json_registry.json
@@ -1,2 +1 @@
-{"op":"set","id":1}
-{"k":"filebeat::logs::native::396268-66306","v":{"id":"native::396268-66306","timestamp":[612296462,1629212377],"ttl":-1,"FileStateOS":{"inode":396268,"device":66306},"identifier_name":"native","prev_id":"","source":"/test_dd_agent/var/log/nginx/access.log","offset":0,"type":"log"}}
\ No newline at end of file
+malformed_json
diff --git a/filebeat/tests/fixtures/new_style_json_registry.json b/filebeat/tests/fixtures/new_style_json_registry.json
new file mode 100644
--- /dev/null
+++ b/filebeat/tests/fixtures/new_style_json_registry.json
@@ -0,0 +1,4 @@
+{"op":"set","id":1}
+{"k":"filebeat::logs::native::396268-66306","v":{"id":"native::396268-66306","timestamp":[612296462,1629212377],"ttl":-1,"FileStateOS":{"inode":277025,"device":51713},"identifier_name":"native","prev_id":"","source":"/test_dd_agent/var/log/nginx/access.log","offset":391747,"type":"log"}}
+{"op":"set","id":2}
+{"k":"filebeat::logs::native::396268-66306","v":{"id":"native::396268-66306","timestamp":[612296461,1629212378],"ttl":-1,"FileStateOS":{"inode":152172,"device":51713},"identifier_name":"native","prev_id":"","source":"/test_dd_agent/var/log/syslog","offset":1024917,"type":"log"}}
diff --git a/filebeat/tests/test_filebeat.py b/filebeat/tests/test_filebeat.py
--- a/filebeat/tests/test_filebeat.py
+++ b/filebeat/tests/test_filebeat.py
@@ -91,6 +91,27 @@ def test_registry_happy_path_with_legacy_format(aggregator):
     )
 
 
+# tests that we still support the format from filebeat > 7
+@pytest.mark.unit
+def test_registry_happy_path_with_new_style_format(aggregator):
+    config = _build_instance("new_style_json")
+    check = FilebeatCheck("filebeat", {}, [config])
+    with mocked_os_stat(
+        {
+            "/test_dd_agent/var/log/nginx/access.log": mocked_file_stats(394154, 277025, 51713),
+            "/test_dd_agent/var/log/syslog": mocked_file_stats(1024917, 152172, 51713),
+        }
+    ):
+        check.check(config)
+
+    aggregator.assert_metric(
+        "filebeat.registry.unprocessed_bytes", value=2407, tags=["source:/test_dd_agent/var/log/nginx/access.log"]
+    )
+    aggregator.assert_metric(
+        "filebeat.registry.unprocessed_bytes", value=0, tags=["source:/test_dd_agent/var/log/syslog"]
+    )
+
+
 def test_bad_config():
     check = FilebeatCheck("filebeat", {}, {})
     with pytest.raises(Exception) as excinfo: