Repository: Kinto/kinto

Issue:
It doesn't seems possible to configure Redis event listener from environment variables.
I tried to [get-stated](http://kinto.readthedocs.org/en/latest/get-started.html#using-docker-compose) and It's fine. kinto is great. 

Next I want to use the redis listener in the [Notifications section](http://kinto.readthedocs.org/en/latest/configuration/settings.html#notifications).
The some lines were added to docker-compose.yml.

```
redis:
    image: redis
web:
    ....
    links:
        - db
        - redis
    ....
    environment:
        ....
        KINTO_EVENT_LISTENERS: redis
        KINTO_EVENT_LISTENERS_REDIS_USE: cliquet.events.redis
        KINTO_EVENT_LISTENERS_REDIS_URL: redis://redis:6379/0
        KINTO_EVENT_LISTENERS_REDIS_POOL_SIZE: 5
        KINTO_EVENT_LISTENERS_REDIS_LISTNAME: queue
```

When `docker-compose up`, the error occurred.

```
web_1   | Traceback (most recent call last):
web_1   |   File "/usr/local/lib/python3.5/dist-packages/cliquet/initialization.py", line 399, in setup_listeners
web_1   |     listener = listener_mod.load_from_config(config, prefix)
web_1   | AttributeError: module 'redis' has no attribute 'load_from_config'
```

When the value of `KINTO_EVENT_LISTENERS` was changed to `cliquet.listeners.redis`, kinto was launched. (`redis` or `cliquet.events.redis` in the document are wrong?).

When I update the value, the another error occurred.

```
web_1   | 2016-03-25 02:01:13,648 ERROR [cliquet.initialization][waitress] "POST  /v1/buckets/.../records" 201 (11 ms) Unable to send the payload to Redis agent=curl/7.43.0 authn_type=basicauth collection_id=record collection_timestamp=1458871273636 errno=None exception=Traceback (most recent call last):
...
web_1   |   File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 538, in send_packed_command
web_1   |     self.connect()
web_1   |   File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 442, in connect
web_1   |     raise ConnectionError(self._error_message(e))
web_1   | redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
```

If the value of `KINTO_EVENT_LISTENERS_REDIS_URL` was changed to any value, the same error occurred. Why this environment value was not only able to set?

My environment is:
- Docker version 1.10.3, build 20f81dd
- docker-compose version 1.6.2, build 4d72027
- OS X EI Captain 10.11.3



Discussion:
> Is `cliquet.events.redis` in the document wrong?

Yes probably, we need to change that, do you mind filing a pull-request to fix this?

> redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
> If the value of KINTO_EVENT_LISTENERS_REDIS_URL was changed to any value, the same error occurred. 

That's a good point.

According to the code, it should be: `KINTO_EVENT_LISTENERS_REDIS_URL` so you are right.
I will investigate why it doesn't work.

So using:

```
kinto.event_listeners = redis
kinto.event_listeners.redis.use = cliquet.listeners.redis
kinto.event_listeners.redis.url = redis://redis:6379/0
kinto.event_listeners.redis.pool_size = 5
kinto.event_listeners.redis.listname = queue
```

It seems to work.

Using `KINTO_EVENT_LISTENERS_REDIS_URL=redis://redis:6739/0` instead it doesn't.

Thank you for your investigation!

Is this fixed (as by #516) ?

No :)

Note: This also applies to authentication settings (e.g `multiauth.policy.ipauth1.use`)

@leplatrem according to the document, I could use the custom config file in `docker-compose.yml`. But it seems `kinto.event_listeners` parameter was not affected. (no messages in logger and the another redis client cannot subscribe.)

@leplatrem It's worked! I misunderstood a event-listener behavior which is not `PUBLISH` but `LPUSH`.
And by the redis' default, keyspace events notifications are disabled. http://redis.io/topics/notifications Thanks.

:+1: 

Regarding notifications do you think it would be worth adding a mention in the documentation ?

No problem. I missed the usage of [redis listener](http://kinto.readthedocs.org/en/latest/tutorials/notifications-custom.html#run-asynchronous-code) in the documentation.


Resolving patch:
diff --git a/kinto/core/initialization.py b/kinto/core/initialization.py
--- a/kinto/core/initialization.py
+++ b/kinto/core/initialization.py
@@ -410,6 +410,7 @@ def setup_listeners(config):
 
     write_actions = (ACTIONS.CREATE, ACTIONS.UPDATE, ACTIONS.DELETE)
     settings = config.get_settings()
+    project_name = settings.get('project_name', '')
     listeners = aslist(settings['event_listeners'])
 
     for name in listeners:
@@ -421,7 +422,11 @@ def setup_listeners(config):
             prefix = 'event_listeners.%s.' % name.split('.')[-1]
             listener = listener_mod.load_from_config(config, prefix)
         except (ImportError, AttributeError):
-            listener_mod = config.maybe_dotted(settings[prefix + 'use'])
+            module_setting = prefix + "use"
+            # Read from ENV or settings.
+            module_value = utils.read_env(project_name + "." + module_setting,
+                                          settings.get(module_setting))
+            listener_mod = config.maybe_dotted(module_value)
             listener = listener_mod.load_from_config(config, prefix)
 
         # If StatsD is enabled, monitor execution time of listeners.
@@ -430,13 +435,25 @@ def setup_listeners(config):
             key = 'listeners.%s' % name
             listener = statsd_client.timer(key)(listener.__call__)
 
-        actions = aslist(settings.get(prefix + 'actions', ''))
+        # Optional filter by event action.
+        actions_setting = prefix + "actions"
+        # Read from ENV or settings.
+        actions_value = utils.read_env(project_name + "." + actions_setting,
+                                       settings.get(actions_setting, ""))
+        actions = aslist(actions_value)
         if len(actions) > 0:
             actions = ACTIONS.from_string_list(actions)
         else:
             actions = write_actions
 
-        resource_names = aslist(settings.get(prefix + 'resources', ''))
+        # Optional filter by event resource name.
+        resource_setting = prefix + "resources"
+        # Read from ENV or settings.
+        resource_value = utils.read_env(project_name + "." + resource_setting,
+                                        settings.get(resource_setting, ""))
+        resource_names = aslist(resource_value)
+
+        # Pyramid event predicates.
         options = dict(for_actions=actions, for_resources=resource_names)
 
         if ACTIONS.READ in actions:


Test patch:
diff --git a/kinto/tests/core/test_listeners.py b/kinto/tests/core/test_listeners.py
--- a/kinto/tests/core/test_listeners.py
+++ b/kinto/tests/core/test_listeners.py
@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 import json
+import os
 import uuid
 from contextlib import contextmanager
 from datetime import datetime
@@ -102,6 +103,50 @@ def test_same_callback_is_called_for_read_and_write_specified(self):
 
         self.assertEqual(self.redis_mocked.return_value.call_count, 2)
 
+    def test_loading_can_read_configuration_from_environment(self):
+        environ = {
+            "KINTO_EVENT_LISTENERS": "kvstore",
+            "KINTO_EVENT_LISTENERS_KVSTORE_USE": "kinto.core.listeners.redis",
+            "KINTO_EVENT_LISTENERS_KVSTORE_URL": "redis://redis:6379/0",
+            "KINTO_EVENT_LISTENERS_KVSTORE_POOL_SIZE": "5",
+            "KINTO_EVENT_LISTENERS_KVSTORE_LISTNAME": "queue",
+            "KINTO_EVENT_LISTENERS_KVSTORE_ACTIONS": "delete",
+            "KINTO_EVENT_LISTENERS_KVSTORE_RESOURCES": "toad",
+        }
+        os.environ.update(**environ)
+
+        config = self.make_app({
+            # With real/full initialization, these should not be necessary:
+            'project_name': 'kinto',
+            'event_listeners': 'kvstore'
+        })
+
+        # Listener is instantiated.
+        self.assertTrue(self.redis_mocked.called)
+
+        # Action filtering is read from ENV.
+        event = ResourceChanged(ACTIONS.DELETE, 123456, [], Request())
+        event.payload['resource_name'] = 'toad'
+        config.registry.notify(event)
+        self.assertTrue(self.redis_mocked.return_value.called)
+
+        self.redis_mocked.reset_mock()
+
+        # Action filtering is read from ENV.
+        event = ResourceChanged(ACTIONS.CREATE, 123456, [], Request())
+        config.registry.notify(event)
+        self.assertFalse(self.redis_mocked.return_value.called)
+
+        # Resource filtering is read from ENV.
+        event = ResourceChanged(ACTIONS.CREATE, 123456, [], Request())
+        event.payload['resource_name'] = 'mushroom'
+        config.registry.notify(event)
+        self.assertFalse(self.redis_mocked.return_value.called)
+
+        # Clean-up.
+        for k in environ.keys():
+            os.environ.pop(k)
+
 
 @contextmanager
 def broken_redis():