Repository: DataDog/dd-agent

Issue:
[mysql] Inaccurate averaging of query run time 
The averaging of MySQL query run time is prone to biasing, because it's an unweighted average of per statement averages. Imagine a common statement runs 100 times, with an average latency (avg_timer_wait) of 10 milliseconds, and a rare statement runs only once, but it takes a full second to execute, so its avg_timer_wait is 1000 milliseconds. Then the AVG(avg_timer_wait) as calculated currently would be 505 milliseconds, which is artificially inflated by the one slow query. See https://github.com/DataDog/dd-agent/blob/master/checks.d/mysql.py#L1186

```
sql_avg_query_run_time = """
    SELECT schema_name, 
    SUM(count_star) cnt,
    ROUND(AVG(avg_timer_wait)/1000000) AS avg_us
FROM performance_schema.events_statements_summary_by_digest
WHERE schema_name IS NOT NULL
GROUP BY schema_name"""
```



Discussion:
CC @ovaistariq for his thoughts.

Thanks for the great explanation @johnaxel. 

Let's fix it with the 5.8.0 release.

The fix I would suggest for this particular query is summing all the elapsed time and dividing it by the number of total events:

```
ROUND((SUM(sum_timer_wait) / SUM(count_star)) / 1000000) AS avg_us
```

But I don't understand the 95th percentile runtime query well enough to translate it to that query.

@johnaxel would you mind writing the "the great explanation" that @yannmh is referring to here in the comments?

For some reason the description has been deleted. I've added it back. Thanks @ovaistariq 

It's an issue Github is having today—issue and PR bodies are getting overwritten. https://status.github.com/messages

I'm seeing the original message now but formatting is lost. For posterity, here it is again:

> The averaging of MySQL query run time is prone to biasing, because it's an unweighted average of per statement averages. Imagine a common statement runs 100 times, with an average latency (avg_timer_wait) of 10 milliseconds, and a rare statement runs only once, but it takes a full second to execute, so its avg_timer_wait is 1000 milliseconds. Then the AVG(avg_timer_wait) as calculated currently would be 505 milliseconds, which is artificially inflated by the one slow query. See https://github.com/DataDog/dd-agent/blob/master/checks.d/mysql.py#L1186

```
sql_avg_query_run_time = """
SELECT schema_name, 
SUM(count_star) cnt,
ROUND(AVG(avg_timer_wait)/1000000) AS avg_us
FROM performance_schema.events_statements_summary_by_digest
WHERE schema_name IS NOT NULL
GROUP BY schema_name"""
```

@johnaxel @irabinovitch thanks for adding back the description.

It is incorrect to say that the averaging of query run time is inaccurate. The number produced by the query is exposed as an average and not as a weighted average. However I agree that doing it as follows will make it accurate:

```
ROUND((SUM(sum_timer_wait) / SUM(count_star)) / 1000000) AS avg_us
```

The change needs to be made to the query in the function `_query_exec_time_per_schema`. 

The 95th percentile query in the function `_get_query_exec_time_95th_us` will not change because that is calculating the 95th percentile not the average.

Thanks @ovaistariq for giving it a look! And apologies if I was heavy-handed with the word "inaccurate". 

@johnaxel thanks for finding the inaccuracy ;)

@truthbk can you take a pass at it please ?


Resolving patch:
diff --git a/checks.d/mysql.py b/checks.d/mysql.py
--- a/checks.d/mysql.py
+++ b/checks.d/mysql.py
@@ -1188,7 +1188,7 @@ def _query_exec_time_per_schema(self, db):
         # Fetches the avg query execution time per schema and returns the
         # value in microseconds
 
-        sql_avg_query_run_time = """SELECT schema_name, SUM(count_star) cnt, ROUND(AVG(avg_timer_wait)/1000000) AS avg_us
+        sql_avg_query_run_time = """SELECT schema_name, ROUND((SUM(sum_timer_wait) / SUM(count_star)) / 1000000) AS avg_us
             FROM performance_schema.events_statements_summary_by_digest
             WHERE schema_name IS NOT NULL
             GROUP BY schema_name"""
@@ -1204,7 +1204,7 @@ def _query_exec_time_per_schema(self, db):
                 schema_query_avg_run_time = {}
                 for row in cursor.fetchall():
                     schema_name = str(row[0])
-                    avg_us = long(row[2])
+                    avg_us = long(row[1])
 
                     # set the tag as the dictionary key
                     schema_query_avg_run_time["schema:{0}".format(schema_name)] = avg_us


Test patch: