Repository: Cloud-CV/EvalAI

Issue:
Log change in Challenge and Challenge Phase model when file fields are updated
__Problem__

We need a way to figure out if the `evaluation_script`field value is changed for Challenge Model or not. Same goes for `test_annotation` field in ChallengePhase Model.

__Please be more clear__ ?

When ever a model is saved, its saved function is called. So this task involves figuring out a way in save model to detect if the model is updated or added. If the model is updated, then further it should be figured out whether the `evaluation_script` is changed for Challenge model or not.

If `evaluation_script` of Challenge model is changed, then simply print a message stating the change. This print statement will later on act as a logging statement in case we need to debug something.

The above requested thing goes same for `test_annotation` field of ChallengePhase model.

__Why__ ?

Submission worker when starts loads all `evaluation_script` and `test_annotation` file in advance. In case if the Challenge Host updates any of the file, there is no way to communicate this to the worker

__Impact__ ?

It actually guarantees that whenever any submission is evaluated, it is evaluated against the write set of parameters.

__How__ ?

As of now, I think it can be done via following ways

* Either overriding the save method of each model and adding a hook over there to check if the instance is being updated or added. May be there is a flag or something which tells whether the instance is added or updated

* Either by [signals](https://docs.djangoproject.com/en/1.10/topics/signals/). Reason I am asking to use signal because we need to carry out the same set of action for more than one models(presently _Challenge_ and _ChallengePhase_). Doing it by signal will help us achieve more modularized code.

* Your suggestion ??

__TakeAways__

About how django model is saved, how signal works, and a brief intro about RabbitMQ and submission worker

Also I will prefer that __before coding__, its better to __discuss__ the approach over here only, so that we can get an idea and update if something is not right.


Discussion:
@TroJan Can I work on this?
@RishabhJain2018 : sure
@TroJan I researched about the above task and found that [signals](https://docs.djangoproject.com/en/1.10/topics/signals/) are a suitable way to implement it. Please see these links ([link1](http://stackoverflow.com/a/1197708) [link2](http://stackoverflow.com/questions/8119570/a-correct-case-to-use-django-signals)) link for reference.
Moreover, I also found a django-app [django-model-changes](https://github.com/kajic/django-model-changes) to track the state and changes of django model instances.
What are your views about it?
@RishabhJain2018 : yeah we will be using signals since we need this functionality for multiple models(presently _Challenge_ and _ChallengePhase_). I will prefer that instead of using `django-model-changes` we write our own code, because we do not need to track the history of change(as of now).
@TroJan so, I am proceeding with signals.
I have a doubt that where we have to print the message? with reference to your statement **If evaluation_script of Challenge model is changed, then simply print a message stating the change. This print statement will later on act as a logging statement in case we need to debug something.**

  
@RishabhJain2018 : the message will be printed in the catched signal. This message is kind of a logging statement telling what was the change. So it will be placed where the change is actually being detected. 
@RishabhJain2018 : any updates here?
@TroJan I apologise for the delay..I will finish this asap.

Resolving patch:
diff --git a/apps/base/models.py b/apps/base/models.py
--- a/apps/base/models.py
+++ b/apps/base/models.py
@@ -1,7 +1,12 @@
 from __future__ import unicode_literals
 
+import logging
+
 from django.db import models
 
+# Get an instance of a logger
+logger = logging.getLogger(__name__)
+
 
 class TimeStampedModel(models.Model):
     """
@@ -14,3 +19,24 @@ class TimeStampedModel(models.Model):
     class Meta:
         abstract = True
         app_label = 'base'
+
+
+def model_field_name(field_name, *args, **kwargs):
+    """
+    The decorator is used to pass model field names to create_post_model_field function for logging change.
+    """
+    def model_field_name_decorator(f, *args, **kwargs):
+        def model_field_name_wrapper(sender, instance, **kwargs):
+            f(sender, instance, field_name=field_name, **kwargs)
+        return model_field_name_wrapper
+    return model_field_name_decorator
+
+
+def create_post_model_field(sender, instance, field_name, **kwargs):
+    """
+    When any model field value changes, it is used to log the change.
+    """
+    if getattr(instance, '_original_{}'.format(field_name)) is False:
+        logger.info('{} for {} is added first time !'.format(field_name, instance.pk))
+    else:
+        logger.info('{} for {} changed !'.format(field_name, instance.pk))
diff --git a/apps/challenges/models.py b/apps/challenges/models.py
--- a/apps/challenges/models.py
+++ b/apps/challenges/models.py
@@ -4,8 +4,9 @@
 from django.utils import timezone
 from django.contrib.postgres.fields import JSONField
 from django.db import models
+from django.db.models import signals
 
-from base.models import (TimeStampedModel, )
+from base.models import (TimeStampedModel, model_field_name, create_post_model_field, )
 from base.utils import RandomFileName
 from participants.models import (ParticipantTeam, )
 
@@ -13,6 +14,11 @@
 class Challenge(TimeStampedModel):
 
     """Model representing a hosted Challenge"""
+
+    def __init__(self, *args, **kwargs):
+        super(Challenge, self).__init__(*args, **kwargs)
+        self._original_evaluation_script = self.evaluation_script
+
     title = models.CharField(max_length=100)
     short_description = models.TextField(null=True, blank=True)
     description = models.TextField(null=True, blank=True)
@@ -74,6 +80,10 @@ def is_active(self):
         return False
 
 
+signals.post_save.connect(model_field_name(field_name='evaluation_script')(create_post_model_field),
+                          sender=Challenge, weak=False)
+
+
 class DatasetSplit(TimeStampedModel):
     name = models.CharField(max_length=100)
     codename = models.CharField(max_length=100, unique=True)
@@ -89,6 +99,10 @@ class Meta:
 class ChallengePhase(TimeStampedModel):
 
     """Model representing a Challenge Phase"""
+    def __init__(self, *args, **kwargs):
+        super(ChallengePhase, self).__init__(*args, **kwargs)
+        self._original_test_annotation = self.test_annotation
+
     name = models.CharField(max_length=100)
     description = models.TextField()
     leaderboard_public = models.BooleanField(default=False)
@@ -130,6 +144,10 @@ def is_active(self):
         return False
 
 
+signals.post_save.connect(model_field_name(field_name='test_annotation')(create_post_model_field),
+                          sender=ChallengePhase, weak=False)
+
+
 class Leaderboard(TimeStampedModel):
 
     schema = JSONField()


Test patch: