Repository: zalando-incubator/kubernetes-log-watcher

Issue:
Make Journald monitor write rate limit configurable
Current settings:

```
"monitor_log_write_rate": 10000,
"monitor_log_max_write_burst": 200000 
```

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=h1) Report
> Merging [#42](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=desc) into [master](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/commit/914da75de2439781884273b859bf663fc5e1072c?src=pr&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42/graphs/tree.svg?src=pr&token=2NLvIPFi70&width=650&height=150)](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
+ Coverage   84.57%   84.65%   +0.07%     
==========================================
  Files           9        9              
  Lines         415      417       +2     
==========================================
+ Hits          351      353       +2     
  Misses         64       64
```


| [Impacted Files](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [kube\_log\_watcher/\_\_init\_\_.py](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree#diff-a3ViZV9sb2dfd2F0Y2hlci9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [kube\_log\_watcher/agents/scalyr.py](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree#diff-a3ViZV9sb2dfd2F0Y2hlci9hZ2VudHMvc2NhbHlyLnB5) | `91.73% <100%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=footer). Last update [914da75...fdaff26](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

elauria: 👍 
lmineiro: 👍 

Patch:
diff --git a/CHANGES.txt b/CHANGES.txt
index 94c60e3..1fe61c2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -2,6 +2,12 @@
 Kubernetes-log-watcher changelog
 ================================
 
+0.15 (2017-05-30)
+-----------------
+
+- Make Journald monitor write rate limit configurable. [#41]
+
+
 0.14 (2017-05-15)
 -----------------
 
diff --git a/README.rst b/README.rst
index 4654078..4ff49d2 100644
--- a/README.rst
+++ b/README.rst
@@ -251,30 +251,30 @@ Log watcher
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_CONTAINERS_PATH``
+WATCHER_CONTAINERS_PATH
   Containers directory path mounted from the host (Default: ``/var/lib/docker/containers``)
 
-``WATCHER_STRICT_LABELS``
+WATCHER_STRICT_LABELS
   If set then only containers running in pods with ``application`` and ``version`` metadata labels will be considered for log watching. (Default is ``False``)
 
   If not set then kubernetes-log-watcher will set ``application_id`` from *pod name*; in order to provide consistent attributes to log configuration agents.
 
-``WATCHER_AGENTS``
+WATCHER_AGENTS
    Comma separated string of required log configuration agents. (Required. Example: "scalyr,appdynamics")
 
-``WATCHER_CLUSTER_ID``
+WATCHER_CLUSTER_ID
    Kubernetes Cluster ID.
 
-``WATCHER_KUBE_URL``
+WATCHER_KUBE_URL
    URL to API proxy service. Service is expected to handle authentication to the Kubernetes cluster. If set, then log-watcher will not use serviceaccount config.
 
-``WATCHER_KUBERNETES_UPDATE_CERTIFICATES``
+WATCHER_KUBERNETES_UPDATE_CERTIFICATES
    [Deprecated] Call update-ca-certificates for Kubernetes service account ca.crt.
 
-``WATCHER_INTERVAL``
+WATCHER_INTERVAL
    Polling interval (secs) for the watcher to detect containers changes. (Default: 60 sec)
 
-``WATCHER_DEBUG``
+WATCHER_DEBUG
    Verbose output. (Default: False)
 
 Scalyr configuration agent
@@ -282,27 +282,32 @@ Scalyr configuration agent
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_SCALYR_API_KEY``
+WATCHER_SCALYR_API_KEY
   Scalyr API key. (Required).
 
-``WATCHER_SCALYR_DEST_PATH``
+WATCHER_SCALYR_DEST_PATH
   Scalyr configuration agent will symlink containers logs in this location. This is to provide more friendly name for log files. Typical log file name for a container will be in the form ``<application>-<version>.log``. (Required).
 
-``WATCHER_SCALYR_CONFIG_PATH``
+WATCHER_SCALYR_CONFIG_PATH
   Scalyr configuration file path. (Default: ``/etc/scalyr-agent-2/agent.json``)
 
-``WATCHER_SCALYR_JOURNALD``
+WATCHER_SCALYR_JOURNALD
   Scalyr should follow Journald logs. This is for node system processes log shipping (e.g. docker, kube) (Default: ``False``)
 
-``WATCHER_SCALYR_JOURNALD_ATTRIBUTES``
+WATCHER_SCALYR_JOURNALD_ATTRIBUTES
   Add attributes to Journald logs. By default ``cluster`` and ``node`` will be added by the configuration agent.
 
-``WATCHER_SCALYR_JOURNALD_EXTRA_FIELDS``
+WATCHER_SCALYR_JOURNALD_EXTRA_FIELDS
   Add extra Systemd Journald fields. Should be a JSON string. Example: '{"_COMM": "command"}'
 
-``WATCHER_SCALYR_JOURNALD_PATH``
+WATCHER_SCALYR_JOURNALD_PATH
   Journald logs path mounted from the host. (Default: ``/var/log/journald``)
 
+WATCHER_SCALYR_JOURNALD_WRITE_RATE
+  Journald monitor write rate. (Default: 10000)
+
+WATCHER_SCALYR_JOURNALD_WRITE_BURST
+  Journald monitor write burst. (Default: 200000)
 
 Scalyr custom parser
 ....................
@@ -321,8 +326,7 @@ AppDynamics configuration agent
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_APPDYNAMICS_DEST_PATH``
-
+WATCHER_APPDYNAMICS_DEST_PATH
   AppDynamics job files path. (Required).
 
 AppDynamics configuration agent could also add ``app_name`` and ``tier_name`` if ``appdynamics_app`` and ``appdynamics_tier`` were set in pod metadata labels.
diff --git a/kube_log_watcher/__init__.py b/kube_log_watcher/__init__.py
index 52a06cc..a3e6290 100644
--- a/kube_log_watcher/__init__.py
+++ b/kube_log_watcher/__init__.py
@@ -1,1 +1,1 @@
-__version__ = '0.14'
+__version__ = '0.15'
diff --git a/kube_log_watcher/agents/scalyr.py b/kube_log_watcher/agents/scalyr.py
index 4cfbdc1..25b913d 100644
--- a/kube_log_watcher/agents/scalyr.py
+++ b/kube_log_watcher/agents/scalyr.py
@@ -16,6 +16,8 @@ SCALYR_CONFIG_PATH = '/etc/scalyr-agent-2/agent.json'
 # If exists! we expect serialized json str: '[{"container": "my-container", "parser": "my-custom-parser"}]'
 SCALYR_ANNOTATION_PARSER = 'kubernetes-log-watcher/scalyr-parser'
 SCALYR_DEFAULT_PARSER = 'json'
+SCALYR_DEFAULT_WRITE_RATE = 10000
+SCALYR_DEFAULT_WRITE_BURST = 200000
 
 logger = logging.getLogger('kube_log_watcher')
 
@@ -54,6 +56,8 @@ class ScalyrAgent(BaseWatcher):
                 'journal_path': os.environ.get('WATCHER_SCALYR_JOURNALD_PATH'),
                 'attributes': json.loads(attributes_str),
                 'extra_fields': json.loads(extra_fields_str),
+                'write_rate': int(os.environ.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE', SCALYR_DEFAULT_WRITE_RATE)),
+                'write_burst': int(os.environ.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST', SCALYR_DEFAULT_WRITE_BURST)),
             }
             self.journald['attributes']['cluster'] = cluster_id
 
diff --git a/kube_log_watcher/templates/scalyr.json.jinja2 b/kube_log_watcher/templates/scalyr.json.jinja2
index 6442841..f8c7fb3 100644
--- a/kube_log_watcher/templates/scalyr.json.jinja2
+++ b/kube_log_watcher/templates/scalyr.json.jinja2
@@ -47,8 +47,8 @@
             {% endif %}
 
             "module": "scalyr_agent.builtin_monitors.journald_monitor",
-            "monitor_log_write_rate": 10000,
-            "monitor_log_max_write_burst": 200000
+            "monitor_log_write_rate": {{ monitor_journald.write_rate }},
+            "monitor_log_max_write_burst": {{ monitor_journald.write_burst }}
         }
         {% endif %}
     ]


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 2707ed5..6d5ad6a 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -10,7 +10,10 @@ APPDYNAMICS_DEST_PATH = '/var/log/watcher/'
 
 SCALYR_DEST_PATH = '/var/log/watcher/'
 SCALYR_KEY = 'scalyr-key-123'
-SCALYR_JOURNALD_DEFAULTS = {'journal_path': None, 'attributes': {'cluster': CLUSTER_ID}, 'extra_fields': {}}
+SCALYR_JOURNALD_DEFAULTS = {
+    'journal_path': None, 'attributes': {'cluster': CLUSTER_ID}, 'extra_fields': {}, 'write_rate': 10000,
+    'write_burst': 200000
+}
 SCALYR_ANNOTATION_PARSER = 'kubernetes-log-watcher/scalyr-parser'
 
 TARGET = {
diff --git a/tests/test_scalyr.py b/tests/test_scalyr.py
index 59155ab..1f14c56 100644
--- a/tests/test_scalyr.py
+++ b/tests/test_scalyr.py
@@ -24,6 +24,11 @@ ENVS = (
         'WATCHER_SCALYR_API_KEY': SCALYR_KEY, 'WATCHER_SCALYR_DEST_PATH': SCALYR_DEST_PATH,
         'WATCHER_SCALYR_JOURNALD': 'true',
     },
+    {
+        'WATCHER_SCALYR_API_KEY': SCALYR_KEY, 'WATCHER_SCALYR_DEST_PATH': SCALYR_DEST_PATH,
+        'WATCHER_SCALYR_JOURNALD': 'true', 'WATCHER_SCALYR_JOURNALD_WRITE_RATE': 1,
+        'WATCHER_SCALYR_JOURNALD_WRITE_BURST': 2,
+    },
 )
 
 KWARGS_KEYS = ('scalyr_key', 'cluster_id', 'logs', 'monitor_journald')
@@ -45,7 +50,12 @@ def assert_agent(agent, env):
     assert agent.config_path == env.get('WATCHER_SCALYR_CONFIG_PATH', SCALYR_CONFIG_PATH)
 
     journald = env.get('WATCHER_SCALYR_JOURNALD')
-    assert agent.journald == (SCALYR_JOURNALD_DEFAULTS if journald else None)
+    journald_defaults = copy.deepcopy(SCALYR_JOURNALD_DEFAULTS)
+    if env.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE'):
+        journald_defaults['write_rate'] = env.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE')
+    if env.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST'):
+        journald_defaults['write_burst'] = env.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST')
+    assert agent.journald == (journald_defaults if journald else None)
 
     assert agent.cluster_id == CLUSTER_ID
 
@@ -369,7 +379,10 @@ def test_remove_log_target(monkeypatch, env, exc):
         (
             {
                 'scalyr_key': SCALYR_KEY, 'cluster_id': CLUSTER_ID, 'logs': [],
-                'monitor_journald': {'journal_path': None, 'attributes': {}, 'extra_fields': {}},
+                'monitor_journald': {
+                    'journal_path': None, 'attributes': {}, 'extra_fields': {}, 'write_rate': 10000,
+                    'write_burst': 200000
+                },
             },
             {
                 'api_key': 'scalyr-key-123',
@@ -393,7 +406,9 @@ def test_remove_log_target(monkeypatch, env, exc):
                 'monitor_journald': {
                     'journal_path': '/var/log/journal',
                     'attributes': {'cluster': CLUSTER_ID, 'node': NODE},
-                    'extra_fields': {'_COMM': 'command'}
+                    'extra_fields': {'_COMM': 'command'},
+                    'write_rate': 10000,
+                    'write_burst': 200000,
                 },
             },
             {