Repository: Kinto/kinto

Issue:
Backoff a percentage of clients
Add a _percentage_ setting for backoff. If specified, then send backoff header only if `random number < percentage`.



Discussion:
Hi @leplatrem, I would like to work on it but this is my first timer PR, do you think I'm qualified? Thanks a lot!
Hi @noisedispatch! 
I'm very glad you chose Kinto for your first PR! 

The amount of code to write for the feature itself is very small, so you'll probably have to be prepared for other kinds of obstacles like handling Git and writing a test that controls the expected behaviour.

But don't worry, we can help you!

Enjoy! ...and thanks again for proposing your help!
Hi @leplatrem ! Thanks for your approval and encouragement! A quick question, when I followed the tutorial to install kinto from source, I found an error here:

```
noisedispatch:~/workspace (master) $ make serve
.venv/bin/kinto --ini config/kinto.ini init
Traceback (most recent call last):
  File ".venv/bin/kinto", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3017, in <module>
    @_call_aside
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside
import platform
    f(*args, **kwargs)
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3030, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 661, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 674, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/ubuntu/workspace/.venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 853, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'transaction<2' distribution was not found and is required by kinto
```

I checked the release note of [transaction](https://pypi.python.org/pypi/transaction) and make a change: https://github.com/noisedispatch/kinto/blob/master/setup.py#L29. After which I can run kinto locally.

I guess your fork is outdated: https://github.com/Kinto/kinto/blob/master/setup.py#L29
Yeah Python dependencies and cache can be tricky when incompatible version ranges are modified. Let us know if you could get back on tracks!
Happy to help with this. For clarity, would it be a setting to only send backoff header under heavy load if, say, random.random()*100 < backoff.percentage?
The setting would be set by an administrator. Then in the Python code, we generate a random number, if it is below the percentage, we set the header, otherwise we don't. If we consider that the random number distribution is linear, then we know we'll have a certain percentage of hits to receive the header.
Hi @leplatrem, is this issue still available? This is my first attempt on contributing to a project. Just to make sure I understand what needs to be done, in case backoff is not none in (https://github.com/Kinto/kinto/blob/cad62f33cd668857a53820b2818c22ddd8602ce8/kinto/core/initialization.py#L134-L135) a random number is generated and the backoff header is sent only if random() < backoff percentage, right?
Also, we'd need a new field on the configuration file (e.g. kinto.backoff_percentage), right? 
Hi @jvce92 !

Thanks for your interested in the project! welcome on board!

Indeed, you'll have to introduce a new settings like `kinto.backoff_percentage` and if it is defined, then put the header only if random is below the specified value (which is could also be understood as «backoff probability» ;))
Do you have any idea how this could be tested?
Sorry for the delay, your message got lost in my pile of notifications :(

You would use a library called `mock` that could force the value of `random` :)

Something like that`(untested code):

```py
# initialization.py
import random

...
if random.random() > setting:
    response.headers["Backoff"] = ...
```

```py
# tests.py
import mock

with mock.patch('kinto.core.initialization.random.random', return_value=0.6):
    resp = self.app.get("/")
    self.assertIn("Backoff", resp.headers())
```

Resolving patch:
diff --git a/kinto/core/__init__.py b/kinto/core/__init__.py
--- a/kinto/core/__init__.py
+++ b/kinto/core/__init__.py
@@ -26,6 +26,7 @@
 
 DEFAULT_SETTINGS = {
     'backoff': None,
+    'backoff_percentage': None,
     'batch_max_requests': 25,
     'cache_backend': '',
     'cache_hosts': '',
diff --git a/kinto/core/initialization.py b/kinto/core/initialization.py
--- a/kinto/core/initialization.py
+++ b/kinto/core/initialization.py
@@ -3,6 +3,7 @@
 import warnings
 from datetime import datetime
 from dateutil import parser as dateparser
+import random
 
 from pyramid.events import NewRequest, NewResponse
 from pyramid.exceptions import ConfigurationError
@@ -132,7 +133,12 @@ def on_new_response(event):
         # Add backoff in response headers.
         backoff = config.registry.settings['backoff']
         if backoff is not None:
-            event.response.headers['Backoff'] = str(backoff)
+            backoff_percentage = config.registry.settings['backoff_percentage']
+            if backoff_percentage is not None:
+                if random.random() < (float(backoff_percentage) / 100.0):
+                    event.response.headers['Backoff'] = str(backoff)
+            else:
+                event.response.headers['Backoff'] = str(backoff)
 
     config.add_subscriber(on_new_response, NewResponse)
 


Test patch:
diff --git a/tests/core/test_views_errors.py b/tests/core/test_views_errors.py
--- a/tests/core/test_views_errors.py
+++ b/tests/core/test_views_errors.py
@@ -24,6 +24,21 @@ def test_backoff_headers_is_present_if_configured(self):
         self.assertIn('Backoff', response.headers)
         self.assertEquals(response.headers['Backoff'], '10')
 
+    def test_backoff_headers_is_present_if_less_than_percentage(self):
+        with mock.patch.dict(self.app.app.registry.settings, [('backoff', 10),
+                             ('backoff_percentage', 50)]):
+            with mock.patch('kinto.core.initialization.random.random', return_value=0.4):
+                response = self.app.get(self.sample_url, headers=self.headers, status=200)
+        self.assertIn('Backoff', response.headers)
+        self.assertEquals(response.headers['Backoff'], '10')
+
+    def test_backoff_headers_is_not_present_if_greater_than_percentage(self):
+        with mock.patch.dict(self.app.app.registry.settings, [('backoff', 10),
+                             ('backoff_percentage', 50)]):
+            with mock.patch('kinto.core.initialization.random.random', return_value=0.6):
+                response = self.app.get(self.sample_url, headers=self.headers, status=200)
+        self.assertNotIn('Backoff', response.headers)
+
     def test_backoff_headers_is_present_on_304(self):
         first = self.app.get(self.sample_url, headers=self.headers)
         etag = first.headers['ETag']
diff --git a/tests/test_configuration/test.ini b/tests/test_configuration/test.ini
--- a/tests/test_configuration/test.ini
+++ b/tests/test_configuration/test.ini
@@ -137,6 +137,7 @@ multiauth.policies = basicauth
 # https://kinto.readthedocs.io/en/latest/api/1.x/backoff.html#id1
 #
 # kinto.backoff =
+# kinto.backoff_percentage =
 # kinto.retry_after_seconds = 3
 # kinto.eos =
 # kinto.eos_message =