Repository: mozilla/kitsune

Issue:
top_contributors_questions DB queries are too expensive to include in questions_list view
We have had repeated performance problems where the DB CPU gets stuck near 100% due to queries similar to the following:
```sql
SELECT COUNT(*) FROM 
(SELECT `auth_user`.`id` AS Col1, COUNT(`questions_answer`.`id`) AS `query_count` 
 FROM `auth_user` 
 INNER JOIN `questions_answer` ON 
 (`auth_user`.`id` = `questions_answer`.`creator_id` ) 
 WHERE (`questions_answer`.`id`) IN 
 (SELECT U0.`id` FROM `questions_answer` U0 INNER JOIN `questions_question` U1 
  ON ( U0.`question_id` = U1.`id` ) 
  INNER JOIN `products_product` U4 ON ( U1.`product_id` = U4.`id` ) 
  WHERE (NOT (U0.`is_spam` = %s) AND NOT (U1.`is_spam` = %s) AND NOT (U0.`creator_id` = (U1.`creator_id`)) AND U0.`created` >= %s AND U1.`locale` = %s AND U4.`slug` = %s)) 
 GROUP BY `auth_user`.`id` ORDER BY NULL)
```

While the CPU is spiking, I could see many, many examples of this query in a MySQL `show processlist`, and you can also see an example of this query (and the ~20 sec execution time) [in NR](https://rpm.newrelic.com/accounts/1299394/applications/45097089/datastores#/overview/MySQL/trace/?id=9c10a0d9-d1b3-11e8-9f71-0242ac110007_0_2025&metric=Datastore%252Fstatement%252FMySQL%252Fauth_user%252Fselect)

I believe this query is from [top_contributors_questions](https://github.com/mozilla/kitsune/blob/master/kitsune/community/utils.py#L16) utility function [called by the question_list view](https://github.com/mozilla/kitsune/blob/master/kitsune/community/utils.py#L16). The results of this query are cached for 15 minutes, but when there is a cache miss the repeated attempts to update the cache (especially when particularly aggressive bots are scanning the site) cause the DB to max out the CPU and have required manual intervention to bring the site back to acceptable performance levels. 



Discussion:
We had a great discussion about this and we think we have a good path forward. The deal is this:

The query above generates the "Top Contriburors" list on the questions list pages (there is one for every product, and one for all products). This feature is revealed by hovering over the "Top Contributors" button on the list page on the right side:

<img width="978" alt="screen shot 2018-10-17 at 11 55 38" src="https://user-images.githubusercontent.com/19176817/47099757-fad81480-d203-11e8-90c3-6432d79f03de.png">

Due to the fact that this feature is hidden-by-default and is causing downtime and some significant developer resources we propose the following:

1. Disable this feature for now. This is the quickest fix for the current issue and should not significantly impact the usability of the pages.
2. Discuss this feature and decide whether to bring it back.
3. If we would like to bring it back, discuss the best UI for it (perhaps it should be visible by default) and code it so that the result is always cached and pre-calculated using a daily cron job.

Resolving patch:
diff --git a/kitsune/community/utils.py b/kitsune/community/utils.py
--- a/kitsune/community/utils.py
+++ b/kitsune/community/utils.py
@@ -51,7 +51,7 @@ def top_contributors_questions(start=None, end=None, locale=None, product=None,
     counts = _get_creator_counts(users, count, page)
 
     if use_cache:
-        cache.set(cache_key, counts, 60*15)  # 15 minutes
+        cache.set(cache_key, counts, 60*180)  # 3 hours
     return counts
 
 
@@ -99,7 +99,7 @@ def top_contributors_l10n(start=None, end=None, locale=None, product=None,
     counts = _get_creator_counts(users, count, page)
 
     if use_cache:
-        cache.set(cache_key, counts, 60*15)  # 15 minutes
+        cache.set(cache_key, counts, 60*180)  # 3 hours
     return counts
 
 
diff --git a/kitsune/questions/views.py b/kitsune/questions/views.py
--- a/kitsune/questions/views.py
+++ b/kitsune/questions/views.py
@@ -31,7 +31,6 @@
 from tidings.models import Watch
 
 from kitsune.access.decorators import permission_required, login_required
-from kitsune.community.utils import top_contributors_questions
 from kitsune.products.api import ProductSerializer, TopicSerializer
 from kitsune.products.models import Product, Topic
 from kitsune.questions import config
@@ -245,10 +244,8 @@ def question_list(request, template, product_slug):
 
     # Filter by locale for AAQ locales, and by locale + default for others.
     if request.LANGUAGE_CODE in QuestionLocale.objects.locales_list():
-        forum_locale = request.LANGUAGE_CODE
         locale_query = Q(locale=request.LANGUAGE_CODE)
     else:
-        forum_locale = settings.WIKI_DEFAULT_LANGUAGE
         locale_query = Q(locale=request.LANGUAGE_CODE)
         locale_query |= Q(locale=settings.WIKI_DEFAULT_LANGUAGE)
 
@@ -300,20 +297,6 @@ def question_list(request, template, product_slug):
     if request.user.is_authenticated():
         request.session['questions_owner'] = owner
 
-    # Get the top contributors for the locale and product.
-    # If we are in a product forum, limit the top contributors to that product.
-    if products and len(products) == 1:
-        product = products[0]
-    else:
-        product = None
-    try:
-        top_contributors, _ = top_contributors_questions(
-            locale=forum_locale, product=product)
-    except ES_EXCEPTIONS:
-        top_contributors = []
-        statsd.incr('questions.topcontributors.eserror')
-        log.exception('Support Forum Top contributors query failed.')
-
     data = {'questions': questions_page,
             'feeds': feed_urls,
             'filter': filter_,
@@ -334,7 +317,6 @@ def question_list(request, template, product_slug):
             'product_slug': product_slug,
             'multiple_products': multiple,
             'all_products': product_slug == 'all',
-            'top_contributors': top_contributors,
             'topic_list': topic_list,
             'topic': topic}
 


Test patch: