internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Cloud-CV/EvalAI

Issue:
Change the submission visibility to public by default
## Current Behaviour

Currently, when a submission is made then it is always private by default. A participant has to explicitly make it public to be visible on the leaderboard even if the leaderboard is public.

## Expected Behavior

The submission made should have the default visibility as public.

## Deliverables

- [ ] Change the `is_public` flag default value in Submissions model to `True`.
- [ ] Create the migrations for the same with name `Change_submission_visibility_default_to_public`



Discussion:
Can I take this issue?
Sure!

Resolving patch:
diff --git a/apps/jobs/migrations/0011_Change_submission_visibility_default_to_public.py b/apps/jobs/migrations/0011_Change_submission_visibility_default_to_public.py
new file mode 100644
--- /dev/null
+++ b/apps/jobs/migrations/0011_Change_submission_visibility_default_to_public.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.15 on 2019-01-21 19:20
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('jobs', '0010_Add_blank_in_submission_metadata'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='submission',
+            name='is_public',
+            field=models.BooleanField(default=True),
+        ),
+    ]
diff --git a/apps/jobs/models.py b/apps/jobs/models.py
--- a/apps/jobs/models.py
+++ b/apps/jobs/models.py
@@ -62,7 +62,7 @@ class Submission(TimeStampedModel):
         ChallengePhase, related_name='submissions')
     created_by = models.ForeignKey(User)
     status = models.CharField(max_length=30, choices=STATUS_OPTIONS, db_index=True)
-    is_public = models.BooleanField(default=False)
+    is_public = models.BooleanField(default=True)
     is_flagged = models.BooleanField(default=False)
     submission_number = models.PositiveIntegerField(default=0)
     download_count = models.IntegerField(default=0)


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Update seed script to add challenge slug
## Current Behaviour:
Currently, when the challenge is created using the seed script, the `slug` field is not populated.

## Expected Behaviour:
When the seed script is run, the `slug` field must be populated with the challenge title and it must include a year in the name to keep it unique like `abc-challenge-2019`


Discussion:
Can I work on this issue?
Sure!

Resolving patch:
diff --git a/scripts/seed.py b/scripts/seed.py
--- a/scripts/seed.py
+++ b/scripts/seed.py
@@ -1,4 +1,5 @@
 # Command to run : python manage.py shell  < scripts/seed.py
+import datetime
 import os
 import random
 import string
@@ -137,6 +138,9 @@ def create_challenge(title, start_date, end_date, host_team):
     evaluation_script = open(
         os.path.join(settings.BASE_DIR, 'examples', 'example1', 'sample_evaluation_script.zip'), 'rb')
     queue = ''.join(random.choice(string.ascii_letters) for _ in range(75))
+    year = datetime.date.today().year
+    slug = '{t}-{y}'.format(t=title, y=year)
+    slug = slug.lower().replace(" ", "-")
     Challenge.objects.create(
         title=title,
         short_description=fake.paragraph(),
@@ -150,6 +154,7 @@ def create_challenge(title, start_date, end_date, host_team):
         published=True,
         enable_forum=True,
         anonymous_leaderboard=False,
+        slug=slug,
         start_date=start_date,
         end_date=end_date,
         queue=queue,


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Add status in contact messages
## Current Behaviour

If there are multiple EvalAI admins then it is not clear in contact messages if one admin has resolved it or not.

## Expected Behaviour
Add a choice field in the contact model with status as `pending` and `resolved` having default as `pending`.

## Deliverables
- [ ] Add the fields in the model.
- [ ] Display it in Contact admin.


Discussion:
@RishabhJain2018 Can I work on this issue?
Sure!

Resolving patch:
diff --git a/apps/web/admin.py b/apps/web/admin.py
--- a/apps/web/admin.py
+++ b/apps/web/admin.py
@@ -7,7 +7,7 @@
 
 @admin.register(Contact)
 class ContactAdmin(admin.ModelAdmin):
-    list_display = ("name", "email", "message", "created_at", "modified_at",)
+    list_display = ("name", "email", "message", "status", "created_at", "modified_at",)
     search_fields = ("email",)
 
 
diff --git a/apps/web/migrations/0006_add_status_field_in_contact_model.py b/apps/web/migrations/0006_add_status_field_in_contact_model.py
new file mode 100644
--- /dev/null
+++ b/apps/web/migrations/0006_add_status_field_in_contact_model.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.15 on 2019-03-12 06:19
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('web', '0005_modified_default_attribute_of_visible_field_to_false'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='contact',
+            name='status',
+            field=models.CharField(choices=[('Pending', 'Pending'), ('Resolved', 'Resolved')], default='Pending', max_length=30),
+        ),
+    ]
diff --git a/apps/web/models.py b/apps/web/models.py
--- a/apps/web/models.py
+++ b/apps/web/models.py
@@ -7,9 +7,17 @@
 
 class Contact(TimeStampedModel):
     """Model representing details of User submitting queries."""
+    PENDING = 'Pending'
+    RESOLVED = 'Resolved'
+
+    STATUS_OPTIONS = (
+        (PENDING, PENDING),
+        (RESOLVED, RESOLVED),
+    )
     name = models.CharField(max_length=100,)
     email = models.EmailField(max_length=70,)
     message = models.CharField(max_length=500,)
+    status = models.CharField(max_length=30, choices=STATUS_OPTIONS, default=PENDING)
 
     def __str__(self):
         return '{0}: {1}: {2}'.format(self.name, self.email, self.message)


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Challenges not displayed on the challenge page if > 10
## Current Behaviour:
If there are more than 10 challenges on the challenge page then those challenges aren't displayed on the page.

## Expected Behaviour:
All the challenges should be displayed on the challenge page.

## Deliverables - 
- [ ] The API already returns the challenge data along with a URL for the next set of challenges.
- [ ] Make an API call to the next URL and append the data into the existing list in the controller.
- [ ] Terminate the appending once the `next` field is null in the API's response.


Discussion:
Can I work on this issue?
Sure!

Resolving patch:
diff --git a/settings/common.py b/settings/common.py
--- a/settings/common.py
+++ b/settings/common.py
@@ -152,7 +152,7 @@
 REST_FRAMEWORK = {
     'DEFAULT_PAGINATION_CLASS': (
         'rest_framework.pagination.LimitOffsetPagination'),
-    'PAGE_SIZE': 10,
+    'PAGE_SIZE': 100,
     'DEFAULT_PERMISSION_CLASSES': [
         'rest_framework.permissions.IsAuthenticatedOrReadOnly'
     ],


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Display error on UI while creating challenge
## Current Behaviour
When there are syntax errors in the `yaml` challenge configs then the challenge creation using zip [API](https://github.com/Cloud-CV/EvalAI/blob/c156f3d54c95861c4ac4eaef1d2e3419bab83a7a/apps/challenges/views.py#L575) fails with 500 Internal Server error which should not be the case.


## Expected Behaviour
The API should give a proper error with 406 error code. The error should be displayed on the UI.

## Resources
For creating a challenge zip, please use [EvalAI-Starters](https://github.com/Cloud-CV/EvalAI-Starters)



Discussion:
@RishabhJain2018 Can I work on this issue?
Sure!
@RishabhJain2018 Should I show all the syntax errors of the yaml file on the UI (over toast) or just show the message saying `There is a syntax error on the yaml file, Please correct them before creating a challenge!`
@RishabhJain2018 Can you please confirm my comment?
Hi @Sanji515, It'd nice to show all the specific errors.

Resolving patch:
diff --git a/apps/challenges/views.py b/apps/challenges/views.py
--- a/apps/challenges/views.py
+++ b/apps/challenges/views.py
@@ -573,8 +573,21 @@ def create_challenge_using_zip_file(request, challenge_host_team_pk):
         with open(join(BASE_LOCATION, unique_folder_name, yaml_file), "r") as stream:
             yaml_file_data = yaml.safe_load(stream)
     except (yaml.YAMLError, ScannerError) as exc:
+        # To get the problem description
+        if hasattr(exc, 'problem'):
+            error_description = exc.problem
+            # To capitalize the first alphabet of the problem description as default is in lowercase
+            error_description = error_description[0:].capitalize()
+        # To get the error line and column number
+        if hasattr(exc, 'problem_mark'):
+            mark = exc.problem_mark
+            line_number = mark.line + 1
+            column_number = mark.column + 1
+        message = (
+            '{} in line {}, column {}'.format(error_description, line_number, column_number)
+        )
         response_data = {
-            'error': exc
+            'error': message
         }
         return Response(response_data, status=status.HTTP_406_NOT_ACCEPTABLE)
 


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Outdated Python 2 code while creating challenge using zip file.
**Observed behavior:** When the serializer is not valid while creating a challenge, the error message never gets sent due to incompatible python 2 code which raises an error. Instead, the control is redirected to the except block, sending a generic message.

**Expected behavior:** The serializer error should be sent in the response when appropriate.

**Reason:** This happens because because of this line https://github.com/Cloud-CV/EvalAI/blob/1dde5467e6cfc4c7338f0107757025f5f80d1fed/apps/challenges/views.py#L940
In Python 3, dictionaries  return "view" objects from the `values()` which are iterable but don't support indexing. In python 2, they returned lists.
@RishabhJain2018 @deshraj 
Outdated Python 2 code while creating challenge using zip file.
**Observed behavior:** When the serializer is not valid while creating a challenge, the error message never gets sent due to incompatible python 2 code which raises an error. Instead, the control is redirected to the except block, sending a generic message.

**Expected behavior:** The serializer error should be sent in the response when appropriate.

**Reason:** This happens because because of this line https://github.com/Cloud-CV/EvalAI/blob/1dde5467e6cfc4c7338f0107757025f5f80d1fed/apps/challenges/views.py#L940
In Python 3, dictionaries  return "view" objects from the `values()` which are iterable but don't support indexing. In python 2, they returned lists.
@RishabhJain2018 @deshraj 


Discussion:
I'm working on this.
I'm working on this.

Resolving patch:
diff --git a/apps/challenges/views.py b/apps/challenges/views.py
--- a/apps/challenges/views.py
+++ b/apps/challenges/views.py
@@ -1112,7 +1112,7 @@ def create_challenge_using_zip_file(request, challenge_host_team_pk):
     except:  # noqa: E722
         try:
             if response_data:
-                response_data = {"error": response_data.values()[0]}
+                response_data = {"error": list(response_data.values())[0]}
                 return Response(
                     response_data, status=status.HTTP_406_NOT_ACCEPTABLE
                 )


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Display error message when Leaderboard schema is missing while creating challenge.
**Expected behavior**: A message like the following `"There is no leaderboard schema in the YAML configuration file. Please add it and then try again!"` with 406 status code should be returned when the schema is missing from the YAML file. This provides information to the user as to what needs to be done.

**Observed behavior:** A 500 internal server error.

**Deliverables:**
- [ ] Change the API to return appropriate response when the schema is not present.

@RishabhJain2018 @deshraj 


Display error message when Leaderboard schema is missing while creating challenge.
**Expected behavior**: A message like the following `"There is no leaderboard schema in the YAML configuration file. Please add it and then try again!"` with 406 status code should be returned when the schema is missing from the YAML file. This provides information to the user as to what needs to be done.

**Observed behavior:** A 500 internal server error.

**Deliverables:**
- [ ] Change the API to return appropriate response when the schema is not present.

@RishabhJain2018 @deshraj 




Discussion:
Thanks, @KhalidRmb for reporting it. Since you're working on another issue. This issue is open for all to work on.
@RishabhJain2018 can I take this up??
Sure @shiv6146 
Meanwhile @RishabhJain2018 please go through #2205 whenever you find time :+1: Thanks!
Thanks, @KhalidRmb for reporting it. Since you're working on another issue. This issue is open for all to work on.
@RishabhJain2018 can I take this up??
Sure @shiv6146 
Meanwhile @RishabhJain2018 please go through #2205 whenever you find time :+1: Thanks!

Resolving patch:
diff --git a/apps/challenges/views.py b/apps/challenges/views.py
--- a/apps/challenges/views.py
+++ b/apps/challenges/views.py
@@ -912,12 +912,19 @@ def create_challenge_using_zip_file(request, challenge_host_team_pk):
     ]
     """
     if leaderboard_schema:
-        if "default_order_by" not in leaderboard_schema[0].get("schema"):
-            message = (
-                "There is no 'default_order_by' key in leaderboard "
-                "schema. Please add it and then try again!"
-            )
-            response_data = {"error": message}
+        if 'schema' not in leaderboard_schema[0]:
+            message = ('There is no leaderboard schema in the YAML '
+                       'configuration file. Please add it and then try again!')
+            response_data = {
+                'error': message
+            }
+            return Response(response_data, status.HTTP_406_NOT_ACCEPTABLE)
+        if 'default_order_by' not in leaderboard_schema[0].get('schema'):
+            message = ('There is no \'default_order_by\' key in leaderboard '
+                       'schema. Please add it and then try again!')
+            response_data = {
+                'error': message
+            }
             return Response(response_data, status.HTTP_406_NOT_ACCEPTABLE)
         if "labels" not in leaderboard_schema[0].get("schema"):
             message = (


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
django.setup() not required for Remote submission worker.
Currently, the remote worker doesn't use the Django app directly as the submission worker does, and only references the models through the public API.

And also the error: `django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.` is shown while starting the worker.
**Deliverables:**
- [x] Remove the statement `django.setup()` from `remote_submission_worker.py`.

@RishabhJain2018 @deshraj 
django.setup() not required for Remote submission worker.
Currently, the remote worker doesn't use the Django app directly as the submission worker does, and only references the models through the public API.

And also the error: `django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.` is shown while starting the worker.
**Deliverables:**
- [x] Remove the statement `django.setup()` from `remote_submission_worker.py`.

@RishabhJain2018 @deshraj 


Discussion:

Resolving patch:
diff --git a/scripts/workers/remote_submission_worker.py b/scripts/workers/remote_submission_worker.py
--- a/scripts/workers/remote_submission_worker.py
+++ b/scripts/workers/remote_submission_worker.py
@@ -4,7 +4,6 @@
 from __future__ import unicode_literals
 
 import contextlib
-import django
 import importlib
 import json
 import logging
@@ -25,7 +24,6 @@
 COMPUTE_DIRECTORY_PATH = join(BASE_TEMP_DIR, "compute")
 
 logger = logging.getLogger(__name__)
-django.setup()
 
 AUTH_TOKEN = os.environ.get("AUTH_TOKEN")
 DJANGO_SERVER = os.environ.get("DJANGO_SERVER", "localhost")


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Outdated Python 2 code while creating challenge using zip file.
**Observed behavior:** When the serializer is not valid while creating a challenge, the error message never gets sent due to incompatible python 2 code which raises an error. Instead, the control is redirected to the except block, sending a generic message.

**Expected behavior:** The serializer error should be sent in the response when appropriate.

**Reason:** This happens because because of this line https://github.com/Cloud-CV/EvalAI/blob/1dde5467e6cfc4c7338f0107757025f5f80d1fed/apps/challenges/views.py#L940
In Python 3, dictionaries  return "view" objects from the `values()` which are iterable but don't support indexing. In python 2, they returned lists.
@RishabhJain2018 @deshraj 


Discussion:
I'm working on this.

Resolving patch:
diff --git a/apps/challenges/views.py b/apps/challenges/views.py
--- a/apps/challenges/views.py
+++ b/apps/challenges/views.py
@@ -1112,7 +1112,7 @@ def create_challenge_using_zip_file(request, challenge_host_team_pk):
     except:  # noqa: E722
         try:
             if response_data:
-                response_data = {"error": list(response_data.values())[0]}
+                response_data = {"error": response_data.values()[0]}
                 return Response(
                     response_data, status=status.HTTP_406_NOT_ACCEPTABLE
                 )


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
submission error 
- the expected behavior: 
when submitting a file to the challenge it should show the details of the submission 
- the observed behavior
when a file is submitted to the challenge all submission details are non as shown below 
![](https://drive.google.com/uc?export=view&id=1YtD6-ytled3_pFA3VIP1ynHmr_3z_M2E)
and the following error shows in terminal:
" worker_1  | [2019-07-06 10:06:45] ERROR Exception while receiving message from submission queue with error 'NoneType' object has no attribute 'get'
worker_1  | Traceback (most recent call last):
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 599, in process_submission_callback
worker_1  |     process_submission_message(body)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 578, in process_submission_message
worker_1  |     user_annotation_file_path,
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 325, in run_submission
worker_1  |     challenge_id
worker_1  | AttributeError: 'NoneType' object has no attribute 'get'" 
This was done using different challenges including the example from EvalAI_starters and the same problem happens every time 
- steps to reproduce the problem
submit a file to the challenge 
- which operating system and/or browser you're using, if relevant
Mac OSx, safari 

Thanks!


Discussion:
Try to restart your worker 
sudo docker-compose restart worker and submit again


Resolving patch:
diff --git a/scripts/workers/submission_worker.py b/scripts/workers/submission_worker.py
--- a/scripts/workers/submission_worker.py
+++ b/scripts/workers/submission_worker.py
@@ -245,6 +245,7 @@ def extract_challenge_data(challenge, phases):
 
     try:
         # import the challenge after everything is finished
+        importlib.invalidate_caches()
         challenge_module = importlib.import_module(
             CHALLENGE_IMPORT_STRING.format(challenge_id=challenge.id)
         )
@@ -672,7 +673,7 @@ def main():
         q_params["pk"] = challenge_pk
 
     if settings.DEBUG or settings.TEST:
-        if LIMIT_CONCURRENT_SUBMISSION_PROCESSING:
+        if eval(LIMIT_CONCURRENT_SUBMISSION_PROCESSING):
             if not challenge_pk:
                 logger.exception(
                     "Please add CHALLENGE_PK for the challenge to be loaded in the docker.env file."
@@ -697,7 +698,7 @@ def main():
     while True:
         for message in queue.receive_messages():
             if settings.DEBUG or settings.TEST:
-                if LIMIT_CONCURRENT_SUBMISSION_PROCESSING:
+                if eval(LIMIT_CONCURRENT_SUBMISSION_PROCESSING):
                     current_running_submissions_count = Submission.objects.filter(
                         challenge_phase__challenge=challenge.id,
                         status="running",


Test patch:
Repository: Cloud-CV/EvalAI

Issue:
Add models to disable/delete submissions
**Motivation**:
Some of the challenge hosts want to delete the 'canceled` submissions and the submission in which participants have uploaded non-standard submission files from their All Submissions view. Hence, we require this feature.

**Deliverables**:
- [ ] Add a boolean field `ignore_submission` with default as False in the Submissions model.
- [ ] Add migration file for the added models.
Add models to disable/delete submissions
**Motivation**:
Some of the challenge hosts want to delete the 'canceled` submissions and the submission in which participants have uploaded non-standard submission files from their All Submissions view. Hence, we require this feature.

**Deliverables**:
- [ ] Add a boolean field `ignore_submission` with default as False in the Submissions model.
- [ ] Add migration file for the added models.


Discussion:
@RishabhJain2018 can I take this up?
sure, @Ayukha .
@RishabhJain2018 can I take this up?
sure, @Ayukha .

Resolving patch:
diff --git a/apps/jobs/migrations/0016_add_ignore_submission_boolean_field_in_submission_model.py b/apps/jobs/migrations/0016_add_ignore_submission_boolean_field_in_submission_model.py
new file mode 100644
--- /dev/null
+++ b/apps/jobs/migrations/0016_add_ignore_submission_boolean_field_in_submission_model.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.23 on 2020-04-28 19:28
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("jobs", "0015_add_archived_as_a_submission_status"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="submission",
+            name="ignore_submission",
+            field=models.BooleanField(default=False),
+        ),
+    ]
diff --git a/apps/jobs/models.py b/apps/jobs/models.py
--- a/apps/jobs/models.py
+++ b/apps/jobs/models.py
@@ -116,6 +116,7 @@ class Submission(TimeStampedModel):
         blank=True,
         null=True,
     )
+    ignore_submission = models.BooleanField(default=False)
 
     def __str__(self):
         return "{}".format(self.id)


Test patch: