Repository: Cloud-CV/EvalAI

Issue:
Limit number of concurrent submissions for a participant
## Current Scenario

Currently, a participant can make any number of submissions until he reaches the limit of doing max submissions per day. We would like to add another constraint where a participant can do at max 3  submissions and if he wants to do more then he has to wait until the status of the previously done submissions gets changed to `failed` or `canceled` or `finished`. This will help us to make sure that the participant cannot abuse the system by just making an unlimited number of submissions if the limit of doing max submissions per day is quite large. 

## Deliverables

- [ ] Add a new field `max_concurrent_submissions_allowed` integer field with default value set as 3.
- [ ] Add a constraint that 

```
if max_submissions_per_day < max_concurrent_submissions_allowed:
    set max_concurrent_submissions_allowed = max_submissions_per_day
```
Note: This would require to override the `save()` method I guess.
- [ ] Update the submission API to check:

```
if (number_of_submissions_with_running_status + number_of_submissions_with_submitted_status + number_of_submissions_with_submitting_status) == max_concurrent_submissions_allowed:
    # Don't allow the participant to do another submission
else:
    # Proceed and allow him to make submission
```

- [ ] When the participant is exceeding the limit then show a message like this:
`You have X submissions that are already being processed. Please wait for sometime to make another submission.`


Discussion:
@deshraj @RishabhJain2018 Can I start working on this?
@isht3 sure, go ahead. Assigning this to you. 

Resolving patch:
diff --git a/apps/challenges/migrations/0035_add_max_concurrent_submissions_allowed_field.py b/apps/challenges/migrations/0035_add_max_concurrent_submissions_allowed_field.py
new file mode 100644
--- /dev/null
+++ b/apps/challenges/migrations/0035_add_max_concurrent_submissions_allowed_field.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.10.2 on 2018-04-20 09:16
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('challenges', '0034_add_allow_block_domain_fields'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='challengephase',
+            name='max_concurrent_submissions_allowed',
+            field=models.PositiveIntegerField(default=3),
+        ),
+    ]
diff --git a/apps/challenges/models.py b/apps/challenges/models.py
--- a/apps/challenges/models.py
+++ b/apps/challenges/models.py
@@ -124,6 +124,7 @@ def __init__(self, *args, **kwargs):
     test_annotation = models.FileField(upload_to=RandomFileName("test_annotations"), default=False)
     max_submissions_per_day = models.PositiveIntegerField(default=100000, db_index=True)
     max_submissions = models.PositiveIntegerField(default=100000, db_index=True)
+    max_concurrent_submissions_allowed = models.PositiveIntegerField(default=3)
     codename = models.CharField(max_length=100, default="Phase Code Name")
     dataset_split = models.ManyToManyField(DatasetSplit, blank=True, through='ChallengePhaseSplit')
 
@@ -151,6 +152,15 @@ def is_active(self):
             return True
         return False
 
+    def save(self, *args, **kwargs):
+
+        # If the max_submissions_per_day is less than the max_concurrent_submissions_allowed.
+        if self.max_submissions_per_day < self.max_concurrent_submissions_allowed:
+            self.max_concurrent_submissions_allowed = self.max_submissions_per_day
+
+        challenge_phase_instance = super(ChallengePhase, self).save(*args, **kwargs)
+        return challenge_phase_instance
+
 
 signals.post_save.connect(model_field_name(field_name='test_annotation')(create_post_model_field),
                           sender=ChallengePhase, weak=False)
diff --git a/apps/jobs/views.py b/apps/jobs/views.py
--- a/apps/jobs/views.py
+++ b/apps/jobs/views.py
@@ -103,6 +103,19 @@ def challenge_submission(request, challenge_id, challenge_phase_id):
             response_data = {'error': 'You haven\'t participated in the challenge'}
             return Response(response_data, status=status.HTTP_403_FORBIDDEN)
 
+        # Fetch the number of submissions under progress.
+        submissions_in_progress_status = [Submission.SUBMITTED, Submission.SUBMITTING, Submission.RUNNING]
+        submissions_in_progress = Submission.objects.filter(
+                                                participant_team=participant_team_id,
+                                                challenge_phase=challenge_phase,
+                                                status__in=submissions_in_progress_status).count()
+
+        if submissions_in_progress >= challenge_phase.max_concurrent_submissions_allowed:
+            message = 'You have {} submissions that are being processed. \
+                       Please wait for them to finish and then try again.'
+            response_data = {'error': message.format(submissions_in_progress)}
+            return Response(response_data, status=status.HTTP_400_BAD_REQUEST)
+
         serializer = SubmissionSerializer(data=request.data,
                                           context={'participant_team': participant_team,
                                                    'challenge_phase': challenge_phase,


Test patch: