Repository: internetarchive/openlibrary

Issue:
Graphite statsd volume too large for memcached
`stats.ol.memcache.{hits,misses}` looks like it might be trigger too frequently. Maybe related to lazy loading related carousels (hitting memcache more freq)?

```
samuel-archive  [11:55 AM]
Hi, I have noticed that ol is  sending about 400/sec these statsd events: ol.memcache.hits:1|c to graphite.
Could you guys turn on sampling at 1/40 or so for this event?  thanks!
samuel-archive  [12:10 PM]
(my target is 10 / second for most of these stats...)
```

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@samuel-archive 

Graphite statsd volume too large for memcached
`stats.ol.memcache.{hits,misses}` looks like it might be trigger too frequently. Maybe related to lazy loading related carousels (hitting memcache more freq)?

```
samuel-archive  [11:55 AM]
Hi, I have noticed that ol is  sending about 400/sec these statsd events: ol.memcache.hits:1|c to graphite.
Could you guys turn on sampling at 1/40 or so for this event?  thanks!
samuel-archive  [12:10 PM]
(my target is 10 / second for most of these stats...)
```

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@samuel-archive 



Discussion:
notes:
https://statsd.readthedocs.io/en/v3.2.1/reference.html#incr
`StatsClient().incr(stat, count=1, rate=1)`
@tabshaikh this is likely from our RelatedWorkCarousels -- any updates / testing on this one?
I believe this should now be solved as we no longer are caching RelatedWorksCarousel.
Re-opening, looks like the presumed fix didn't wokr
I see `stats_hook` in `openlibrary/plugins/openlibrary/stats.py` which increments stats to memcache for hits and misses and `create_stats_client` which sets up the graphite connection using `pystatsd.Client`
It looks like we can sample per event which is what we seem to want to do:
https://statsd.readthedocs.io/en/v3.2.1/types.html

From the docs:
`rate` is a float between 0 and 1:

```
# Increment this counter 10% of the time.
statsd.incr('some.third.event', rate=0.1)
```

Except we're somehow using `pystatsd.Client` `increment` and not `incr`?

It looks like we're conditionally using `pystatsd` or if it's unavailable `statsd` and both have similar but slightly divergent APIs. I believe the following change should give us what we want: 

```
diff --git a/openlibrary/core/stats.py b/openlibrary/core/stats.py
index 7fab80c..047ac87 100644
--- a/openlibrary/core/stats.py
+++ b/openlibrary/core/stats.py
@@ -46,16 +46,16 @@ def put(key, value):
         l.debug("Putting %s as %s"%(value, key))
         client.timing(key, value)

-def increment(key, n=1):
+def increment(key, n=1, rate=1.0):
     "Increments the value of ``key`` by ``n``"
     global client
     if client:
-        l.debug("Incrementing %s"% key)
+        l.debug("Incrementing %s" % key)
         for i in range(n):
             try:
-                client.increment(key)
+                client.increment(key, sample_rate=rate)
             except AttributeError:
-                client.incr(key)
+
```
notes:
https://statsd.readthedocs.io/en/v3.2.1/reference.html#incr
`StatsClient().incr(stat, count=1, rate=1)`
@tabshaikh this is likely from our RelatedWorkCarousels -- any updates / testing on this one?
I believe this should now be solved as we no longer are caching RelatedWorksCarousel.
Re-opening, looks like the presumed fix didn't wokr
I see `stats_hook` in `openlibrary/plugins/openlibrary/stats.py` which increments stats to memcache for hits and misses and `create_stats_client` which sets up the graphite connection using `pystatsd.Client`
It looks like we can sample per event which is what we seem to want to do:
https://statsd.readthedocs.io/en/v3.2.1/types.html

From the docs:
`rate` is a float between 0 and 1:

```
# Increment this counter 10% of the time.
statsd.incr('some.third.event', rate=0.1)
```

Except we're somehow using `pystatsd.Client` `increment` and not `incr`?

It looks like we're conditionally using `pystatsd` or if it's unavailable `statsd` and both have similar but slightly divergent APIs. I believe the following change should give us what we want: 

```
diff --git a/openlibrary/core/stats.py b/openlibrary/core/stats.py
index 7fab80c..047ac87 100644
--- a/openlibrary/core/stats.py
+++ b/openlibrary/core/stats.py
@@ -46,16 +46,16 @@ def put(key, value):
         l.debug("Putting %s as %s"%(value, key))
         client.timing(key, value)

-def increment(key, n=1):
+def increment(key, n=1, rate=1.0):
     "Increments the value of ``key`` by ``n``"
     global client
     if client:
-        l.debug("Incrementing %s"% key)
+        l.debug("Incrementing %s" % key)
         for i in range(n):
             try:
-                client.increment(key)
+                client.increment(key, sample_rate=rate)
             except AttributeError:
-                client.incr(key)
+
```

Resolving patch:
diff --git a/openlibrary/core/stats.py b/openlibrary/core/stats.py
--- a/openlibrary/core/stats.py
+++ b/openlibrary/core/stats.py
@@ -41,19 +41,20 @@ def put(key, value):
     "Records this ``value`` with the given ``key``. It is stored as a millisecond count"
     global client
     if client:
-        l.debug("Putting %s as %s"%(value, key))
+        l.debug("Putting %s as %s" % (value, key))
         client.timing(key, value)
 
-def increment(key, n=1):
+
+def increment(key, n=1, rate=1.0):
     "Increments the value of ``key`` by ``n``"
     global client
     if client:
-        l.debug("Incrementing %s"% key)
+        l.debug("Incrementing %s" % key)
         for i in range(n):
             try:
-                client.increment(key)
+                client.increment(key, sample_rate=rate)
             except AttributeError:
-                client.incr(key)
+                client.incr(key, rate=rate)
 
 
 client = create_stats_client()
diff --git a/openlibrary/plugins/openlibrary/stats.py b/openlibrary/plugins/openlibrary/stats.py
--- a/openlibrary/plugins/openlibrary/stats.py
+++ b/openlibrary/plugins/openlibrary/stats.py
@@ -88,9 +88,9 @@ def stats_hook():
                 memcache_misses += 1
 
     if memcache_hits:
-        graphite_stats.increment('ol.memcache.hits', memcache_hits)
+        graphite_stats.increment('ol.memcache.hits', memcache_hits, rate=0.025)
     if memcache_misses:
-        graphite_stats.increment('ol.memcache.misses', memcache_misses)
+        graphite_stats.increment('ol.memcache.misses', memcache_misses, rate=0.025)
 
     for name, value in stats_summary.items():
         name = name.replace(".", "_")


Test patch: