Repository: Cloud-CV/EvalAI

Issue:
Add Challenge preview mode for the challenge hosts
## Description
The challenge host must be able to view the challenge in preview mode before publishing the challenge. Also, they must be able to make edits and make submissions using their account.

## Proposed Approach

- When the challenge `is_active` flag is false, challenge host can only see the challenge and make submissions and see their entry on the leaderboard for testing purposes.

- When the challenge’s `is_active` flag is true, challenge hosts and others can see the challenge and make submissions but their submissions won’t go on leaderboard.

## Deliverables
- [ ] Update the backend API for showing the challenge to challenge host only.
- [ ] Update the leaderboard API that will exclude the submissions from the challenge host team.
Add Challenge preview mode for the challenge hosts
## Description
The challenge host must be able to view the challenge in preview mode before publishing the challenge. Also, they must be able to make edits and make submissions using their account.

## Proposed Approach

- When the challenge `is_active` flag is false, challenge host can only see the challenge and make submissions and see their entry on the leaderboard for testing purposes.

- When the challenge’s `is_active` flag is true, challenge hosts and others can see the challenge and make submissions but their submissions won’t go on leaderboard.

## Deliverables
- [ ] Update the backend API for showing the challenge to challenge host only.
- [ ] Update the leaderboard API that will exclude the submissions from the challenge host team.


Discussion:
@RishabhJain2018 @deshraj I would like to work on this issue! :smile: 
Sure @isht3 go ahead!
@isht3 I would recommend solving the existing issues that you are working on instead of working on new things. 
@deshraj Okay! 😄 
@deshraj Can I start?
@isht3 sure. :) 
@deshraj @RishabhJain2018 I'll have to define a separate `is_active` field in the back-end, yes? Also, what I thought was after the challenge was uploaded, the admin would have to verify, at the same time the challenge host could also preview it and make changes, but unless both the admin and challenge host approves it, it won't be published. It would be visible in the Challenges in the `My challenges section` where the host can disable, enable challenge and test it. Also, the host would be able to see whether the admin has approved it or not. Do drop in your thoughts! :smile: 
> @deshraj @RishabhJain2018 I'll have to define a separate is_active field in the back-end, yes? 

You can use the field `is_published` which is being controlled by the challenge host to make the challenge public or just to visible to himself. 

> lso, what I thought was after the challenge was uploaded, the admin would have to verify, at the same time the challenge host could also preview it and make changes, but unless both the admin and challenge host approves it, it won't be published. It would be visible in the Challenges in the My challenges section where the host can disable, enable challenge and test it. Also, the host would be able to see whether the admin has approved it or not. Do drop in your thoughts! 😄

Yes, you are right. So to make things more clear to you, here is the flow that I have in my mind:

```
1. You uploaded a challenge on EvalAI:
2. If you set `is_published` to False in zip config:
        - The challenge will be visible to you only. Challenge is visible to you only. You can make edits to the challenge. Once admin approves it, it will still remain private since you haven't set `is_published` to True. Once you set it to True, it will be publicly visible. 
3. Else if you set the `is_published` to True in zip config:
        - Challenge will be visible to you only for editing purposes. Once Admin approves it, will get public automatically.  
```
@deshraj So I should have a toggle button in the challenge for hosts to publish it, yes?
@isht3 yes, there should be a toggle button which will be visible to challenge host only.
@isht3 Any update with this? We have some of the challenge hosts who actually want this feature to be there really soon? Please let me know if you won't be able to work on it. 
@deshraj The challenge publish/private feature is done. The testing submissions for the challenge host will be done tonight. 
Is that timing okay?
Sounds good. Please let me know if you have questions. 
@RishabhJain2018 @deshraj I would like to work on this issue! :smile: 
Sure @isht3 go ahead!
@isht3 I would recommend solving the existing issues that you are working on instead of working on new things. 
@deshraj Okay! 😄 
@deshraj Can I start?
@isht3 sure. :) 
@deshraj @RishabhJain2018 I'll have to define a separate `is_active` field in the back-end, yes? Also, what I thought was after the challenge was uploaded, the admin would have to verify, at the same time the challenge host could also preview it and make changes, but unless both the admin and challenge host approves it, it won't be published. It would be visible in the Challenges in the `My challenges section` where the host can disable, enable challenge and test it. Also, the host would be able to see whether the admin has approved it or not. Do drop in your thoughts! :smile: 
> @deshraj @RishabhJain2018 I'll have to define a separate is_active field in the back-end, yes? 

You can use the field `is_published` which is being controlled by the challenge host to make the challenge public or just to visible to himself. 

> lso, what I thought was after the challenge was uploaded, the admin would have to verify, at the same time the challenge host could also preview it and make changes, but unless both the admin and challenge host approves it, it won't be published. It would be visible in the Challenges in the My challenges section where the host can disable, enable challenge and test it. Also, the host would be able to see whether the admin has approved it or not. Do drop in your thoughts! 😄

Yes, you are right. So to make things more clear to you, here is the flow that I have in my mind:

```
1. You uploaded a challenge on EvalAI:
2. If you set `is_published` to False in zip config:
        - The challenge will be visible to you only. Challenge is visible to you only. You can make edits to the challenge. Once admin approves it, it will still remain private since you haven't set `is_published` to True. Once you set it to True, it will be publicly visible. 
3. Else if you set the `is_published` to True in zip config:
        - Challenge will be visible to you only for editing purposes. Once Admin approves it, will get public automatically.  
```
@deshraj So I should have a toggle button in the challenge for hosts to publish it, yes?
@isht3 yes, there should be a toggle button which will be visible to challenge host only.
@isht3 Any update with this? We have some of the challenge hosts who actually want this feature to be there really soon? Please let me know if you won't be able to work on it. 
@deshraj The challenge publish/private feature is done. The testing submissions for the challenge host will be done tonight. 
Is that timing okay?
Sounds good. Please let me know if you have questions. 

Resolving patch:
diff --git a/apps/jobs/views.py b/apps/jobs/views.py
--- a/apps/jobs/views.py
+++ b/apps/jobs/views.py
@@ -214,11 +214,18 @@ def leaderboard(request, challenge_phase_split_id):
         response_data = {'error': 'Sorry, Default filtering key not found in leaderboard schema!'}
         return Response(response_data, status=status.HTTP_400_BAD_REQUEST)
 
+    # Exclude the submissions done by members of the host team
+    # while populating leaderboard
+    challenge_obj = challenge_phase_split.challenge_phase.challenge
+    challenge_hosts_emails = challenge_obj.creator.get_all_challenge_host_email()
+    leaderboard_data = LeaderboardData.objects.exclude(
+        submission__created_by__email__in=challenge_hosts_emails)
+
     # Get all the successful submissions related to the challenge phase split
-    leaderboard_data = LeaderboardData.objects.filter(
-        challenge_phase_split=challenge_phase_split,
-        submission__is_public=True,
-        submission__is_flagged=False).order_by('created_at')
+    leaderboard_data = leaderboard_data.filter(
+            challenge_phase_split=challenge_phase_split,
+            submission__is_public=True,
+            submission__is_flagged=False).order_by('created_at')
     leaderboard_data = leaderboard_data.annotate(
         filtering_score=RawSQL('result->>%s', (default_order_by, ), output_field=FloatField())).values(
             'id', 'submission__participant_team__team_name',


Test patch: