Repository: TabbycatDebate/tabbycat

Issue:
Add an 'important feedback' page to highlight feedback that is outside the expected range for a chair/panellist/trainee
So, at the risk of expanding the feedback system, there is some low hanging fruit in terms of more easily surfacing 'important' feedback. Currently we need to find things by source/target; which makes it hard to look across all pieces of feedback to highlight the most critical ones. A simple 'win' here would be to make a feedback page that highlights feedback that is outside the 'expected' range based on position; say:

- A chair that receives a 0 or 1 (normalised to a 0-5 range)
- A trainee that receives a 3-5 (normalised to a 0-5 range)
- A panellist that receives a 4-5 or a 0-1

The other way to play thing would be to show results that are more than 2 points higher or lower than an adj's current rating. While some configuration might be valuable here (and easy given the queries can be parametrised) a set of sensible defaults would still be a major win and relatively easy to create.


Discussion:
Should this be its own page, or should we incorporate this into the feedback overview table? I was thinking it could work something like this, except less ugly

<img width="1056" alt="screen shot 2017-11-11 at 4 31 28 pm" src="https://user-images.githubusercontent.com/10344453/32693749-f80df376-c6fd-11e7-865b-d9511c5c8022.png">


In my mind I was thinking it would be a seperate page similar to the current "Latest" page — i.e. a chronological list of the feedback 'cards' but one that is filtered to show scores that are significantly above or below the 'expected' range for that position (i.e. +/- >2 as a starting point).

While it would be great to add it to the feedback overview table also in some fashion there are a couple of difficulties here:
- The colors for the dots are already coded by position (chair, panellist trainee, etc) and those are currently the only easy way of getting a feel for how an adjudicator has tracked over the course of the rounds
- That said that color scheme (and the tooltips) aren't particularly discoverable by most adjudication cores and usually have to be pointed out to them. So I'd worry that any feature relying on adding extra features to that graph wouldn't be noticed
- In most cases when looking at this kind of 'unexpected' feedback its important to consider the context of the feedback in terms of who gave it, whether they won, what the actual justification for that ranking was, etc. The cards can provide this, but I'd worry if we were just indicating that the feedback occurred without an easy way to look up that feedback it might raise more questions than it answers.

@philipbelesky  I think I might give this one a shot - assuming you've not got any work in progress on it? Would that be okay with you?
@tfpk Go ahead! It looks like @BenMusch is busy with MIT-Tab so I don't think he had a chance to get far into it.
So I have a question regarding efficiency. I can write a property on AdjudicatorFeedback that gives us an idea of how different it is to past feedback, and python `filter` on that. Issue would be that that's loading a lot of `AdjudicatorFeedback`  from a tournament, and doing the filtering in python, which is generally not optimal. The other option is to use annotation, F expressions and django filtering to do it, but that limits us to filtering on fields in the DB. Do we care that it's inefficient, and if so what should I filter on? I'd personally suggest `feedback_importance=F('score') - F('adjudicator__test_score')`, but you guys would know better than me.
@czlee probably knows more about the efficiency mechanics of query construction, but given this is a static page there should be some leeway for a relatively expensive query as long as the total number of queries is relatively small.

I guess the other question this raises is in terms of how exactly we are determining feedback outside an expected range in terms of what the 'expected' score is that are being deviated from. My first thought would be to measure it against the current rating of each adjudicator, or perhaps their score at the time the feedback was submitted. The primary intent of the page *I think* would be to identify circumstances where either panellists or chairs are getting scores significantly below what would be expected for those positions (which often means something needs to be followed up) or to identify promising trainees or panellists that are getting feedback above where they are currently ranked. 

So in that scenario filtering relative to past feedback might not catch these, as if a trainees feedback is all relatively uniformly good, or a chair's uniformly bad it might not be surfaced as it appears consistent. In either the promote or demote case this typically means modifying the test score to push the ratings higher or lower. Although once we have a prototype we could play around with how that looks and perhaps try out some different options that filter the view, such as filtering by round or perhaps only looking for unexpectedly high vs unexpectedly low scores.
I definitely care if it's inefficient; we already have a lot of needlessly inefficient code in our code base and I'd rather not add to it. This also isn't a static page, in the sense that you have to crunch statistics every time someone loads the page. So, yes, you're right, I'd rather push the work into a complex query than post-process in Python. You can get pretty complex filters using just Django, so I'd be surprised if we couldn't get exactly what we wanted here, especially now that Django 2.0 supports filters on annotations. To give some idea of my position, I wrote the current code for aggregating the weighted score before I knew enough about Django annotations, I think if I had time I'd rewrite it now.

If you think about it, this feature doesn't make much sense except in a context where there is no automatic feedback integration into adj scores, and so adj scores are updated manually (via the test score) only. That being the case, you'd care mainly about differential between test score and average feedback score, right? But then, as the original issue description hints, what CA teams are sometimes really interested in isn't large general differentials in general, but specifically those that straddle the boundary between chair and panellist, and between panellist and trainee. The problem there is that Tabbycat doesn't have a well-defined boundary for chair/panellist—it just allocates as many chairs as it needs to. (There is a well-defined boundary for panellist/trainee.)

That all said, if we have to think that hard about how to design this, I posit that it's something we should wait on a concrete feature request for before implementing?
To expand upon the above and outline some off-github chats for @tfpk:

- For the first version/prototype we probably don't need to worry too much about the exact parameters or efficiency of how the 'important' feedback scores are calculated; this is something we can hone by running it through old tabs and in beta testing to fine tune what information it shows. Just getting the basics of the page up would still be a super useful PR.
- While as CZ points out, there are no exact bounds for chair/panelist, these can be inferred based on the relevant preferences and draw size (as the diagrams on Feedback Overview do). For the purposes of an initial PR I'd be fine with just using hardcoded numbers for the expected chair/panellist/trainee scores and I can sub those out for a more flexible calculation.
- The 'boundary' problem of having marginal chairs/panellists/trainees is probably a separate issue/feature; the feature I described earlier is a relatively concrete idea based on a couple of past tournaments I've tabbed. The intent isn't necessarily to even use the page to adjust the scores of the adjudicators based on this information, just that the page provides a filtered digest of feedback to highlight the most crucial bits (which from experience are those with scores much higher/lower than what would be typical of the position). 

Resolving patch:
diff --git a/tabbycat/adjfeedback/urls_admin.py b/tabbycat/adjfeedback/urls_admin.py
--- a/tabbycat/adjfeedback/urls_admin.py
+++ b/tabbycat/adjfeedback/urls_admin.py
@@ -28,6 +28,9 @@
     path('latest/',
         views.LatestFeedbackView.as_view(),
         name='adjfeedback-view-latest'),
+    path('important',
+        views.ImportantFeedbackView.as_view(),
+        name='adjfeedback-view-important'),
     path('source/list/',
         views.FeedbackBySourceView.as_view(),
         name='adjfeedback-view-by-source'),
diff --git a/tabbycat/adjfeedback/views.py b/tabbycat/adjfeedback/views.py
--- a/tabbycat/adjfeedback/views.py
+++ b/tabbycat/adjfeedback/views.py
@@ -4,7 +4,7 @@
 
 from django.contrib import messages
 from django.core.exceptions import ObjectDoesNotExist
-from django.db.models import Q
+from django.db.models import F, Q
 from django.http import JsonResponse
 from django.utils.translation import gettext as _
 from django.utils.translation import ngettext
@@ -252,6 +252,23 @@ def get_feedback_queryset(self):
         )
 
 
+class ImportantFeedbackView(FeedbackCardsView):
+    """View displaying the feedback in order of most 'important'."""
+
+    template_name = "feedback_important.html"
+
+    def get_feedback_queryset(self):
+        t = self.get_tournament()
+        return AdjudicatorFeedback.objects.annotate(
+            feedback_importance=F('score') - F('adjudicator__test_score')
+        ).filter(
+            Q(feedback_importance__gt=2) | Q(feedback_importance__lt=-2),
+            Q(adjudicator__tournament=t) |
+            Q(adjudicator__tournament__isnull=True)
+        ).order_by('-timestamp').select_related(
+            'adjudicator', 'source_adjudicator__adjudicator', 'source_team__team')
+
+
 class FeedbackFromSourceView(SingleObjectFromTournamentMixin, FeedbackCardsView):
     """Base class for views displaying feedback from a given team or adjudicator."""
 
diff --git a/tabbycat/utils/context_processors.py b/tabbycat/utils/context_processors.py
--- a/tabbycat/utils/context_processors.py
+++ b/tabbycat/utils/context_processors.py
@@ -28,7 +28,7 @@ def get_menu_highlight(request):
     menu = {}
     path = request.path.split('/')
     if "admin" in path:
-        if "options" in path or "participants" in path or "import" in path:
+        if "options" in path or "participants" in path or ("import" in path and "important" not in path):
             menu['options_nav'] = True
             if "options" in path:
                 menu['configuration_nav'] = True
@@ -56,6 +56,8 @@ def get_menu_highlight(request):
             menu['feedback_nav'] = True
             if "latest" in path:
                 menu['feedback_latest_nav'] = True
+            elif "important" in path:
+                menu['feedback_important_nav'] = True
             elif "source" in path:
                 menu['feedback_source_nav'] = True
             elif "target" in path:


Test patch: