anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: webcomics/dosage
Issue: GoComics broken (again)

It appears that GoGomics is not working again. Here is the error message from I received today.

`GoComics/NonSequitur> ERROR: XPath //div[contains(concat(" ", @class, " "), " comic ")]//a[contains(concat(" ", @class, " "), " fa-caret-left ")] not found at URL http://www.gocomics.com/nonsequitur.`



Changed files:
dosagelib/plugins/gocomics.py

diff --git a/dosagelib/plugins/gocomics.py b/dosagelib/plugins/gocomics.py
index e13db54f3..c7e7f481d 100644
--- a/dosagelib/plugins/gocomics.py
+++ b/dosagelib/plugins/gocomics.py
@@ -6,7 +6,7 @@
 from __future__ import absolute_import, division, print_function
 
 from ..scraper import _ParserScraper
-from ..helpers import bounceStarter, xpath_class
+from ..helpers import indirectStarter, xpath_class
 
 
 class GoComics(_ParserScraper):
@@ -15,7 +15,8 @@ class GoComics(_ParserScraper):
     navSearch = '//div[{0}]//a[{1}]'
     prevSearch = navSearch.format(xpath_class('comic'), xpath_class('fa-caret-left'))
     nextSearch = navSearch.format(xpath_class('comic'), xpath_class('fa-caret-right'))
-    starter = bounceStarter
+    latestSearch = navSearch.format(xpath_class('gc-deck--cta-0'), xpath_class(''))
+    starter = indirectStarter
     help = 'Index format: yyyy/mm/dd'
 
     def __init__(self, name, path, lang=None):
Repository: red-hat-storage/ocs-ci
Issue: Address "pods "rook-ceph-detect-version-4dszn" not found" failure in pod/wait_for_storage_pods()

We are seeing multiple test failures which call `pod.wait_for_storage_pods()` due to the absence of `rook-ceph-detect-version` pod

```
E           ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc -n openshift-storage --kubeconfig /home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig describe Pod rook-ceph-detect-version-4dszn.
E           Error is Error from server (NotFound): pods "rook-ceph-detect-version-4dszn" not found
```

Could be that we need to ignore this pod while checking the storage pods status

Changed files:
ocs_ci/ocs/resources/pod.py, ocs_ci/ocs/constants.py

diff --git a/ocs_ci/ocs/constants.py b/ocs_ci/ocs/constants.py
index f6054cd3..7a889fa1 100644
--- a/ocs_ci/ocs/constants.py
+++ b/ocs_ci/ocs/constants.py
@@ -244,6 +244,7 @@ NOOBAA_CORE_POD_LABEL = "noobaa-core=noobaa"
 NOOBAA_OPERATOR_POD_LABEL = "noobaa-operator=deployment"
 NOOBAA_DB_LABEL = "noobaa-db=noobaa"
 NOOBAA_ENDPOINT_POD_LABEL = "noobaa-s3=noobaa"
+ROOK_CEPH_DETECT_VERSION_LABEL = "app=rook-ceph-detect-version"
 DEFAULT_DEVICESET_PVC_NAME = "ocs-deviceset"
 DEFAULT_MON_PVC_NAME = "rook-ceph-mon"
 OSD_PVC_GENERIC_LABEL = "ceph.rook.io/DeviceSet"
diff --git a/ocs_ci/ocs/resources/pod.py b/ocs_ci/ocs/resources/pod.py
index 1c2b46e3..6179912f 100644
--- a/ocs_ci/ocs/resources/pod.py
+++ b/ocs_ci/ocs/resources/pod.py
@@ -1306,6 +1306,11 @@ def wait_for_storage_pods(timeout=200):
     all_pod_obj = get_all_pods(
         namespace=defaults.ROOK_CLUSTER_NAMESPACE
     )
+    # Ignoring pods with "app=rook-ceph-detect-version" app label
+    all_pod_obj = [
+        pod for pod in all_pod_obj if constants.ROOK_CEPH_DETECT_VERSION_LABEL not in pod.get_labels()
+    ]
+
     for pod_obj in all_pod_obj:
         state = constants.STATUS_RUNNING
         if any(i in pod_obj.name for i in ['-1-deploy', 'ocs-deviceset']):
Repository: red-hat-storage/ocs-ci
Issue: AttributeError: 'AWSUPI' object has no attribute 'ocp_deployment'"

from chat

```
Itzhak Kave
,
8:38 AM
,
Hi. I am trying to destroy an aws upi cluster, but it failed. I found this error in the output: "  File "/home/jenkins/workspace/qe-destroy-ocs-cluster/ocs-ci/ocs_ci/deployment/aws.py", line 187, in destroy_volumes
    cluster_name = self.ocp_deployment.metadata.get('clusterName')
AttributeError: 'AWSUPI' object has no attribute 'ocp_deployment'"
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-destroy-ocs-cluster/4384/
```

Changed files:
ocs_ci/deployment/aws.py

diff --git a/ocs_ci/deployment/aws.py b/ocs_ci/deployment/aws.py
index 82b04654..ca4b5a9b 100644
--- a/ocs_ci/deployment/aws.py
+++ b/ocs_ci/deployment/aws.py
@@ -40,6 +40,10 @@ class AWSBase(Deployment):
         super(AWSBase, self).__init__()
         self.region = config.ENV_DATA['region']
         self.aws = AWSUtil(self.region)
+        if config.ENV_DATA.get('cluster_name'):
+            self.cluster_name = config.ENV_DATA['cluster_name']
+        else:
+            self.cluster_name = get_cluster_name(self.cluster_path)
 
     def create_ebs_volumes(self, worker_pattern, size=100):
         """
@@ -183,10 +187,9 @@ class AWSBase(Deployment):
 
     def destroy_volumes(self):
         try:
-            # Retrieve cluster name and AWS region from metadata
-            cluster_name = self.ocp_deployment.metadata.get('clusterName')
+            # Retrieve AWS region from metadata
             # Find and delete volumes
-            volume_pattern = f"{cluster_name}*"
+            volume_pattern = f"{self.cluster_name}*"
             logger.debug(f"Finding volumes with pattern: {volume_pattern}")
             volumes = self.aws.get_volumes_by_name_pattern(volume_pattern)
             logger.debug(f"Found volumes: \\n {volumes}")
Repository: red-hat-storage/ocs-ci
Issue: Misleading warning and error reported at the end of a test run

When I execute a test case via ocs-ci, at the and of the log I see the following lines:

```
16:30:45 - MainThread - ocs_ci.ocs.ocp - WARNING - Failed to get resource: namespace-test-47b4cf16197941bd9a004243ae8deea6 of kind: Project, selector: None, Error: Error during execution of command: oc -n namespace-test-47b4cf16197941bd9a004243ae8deea6 --kubeconfig /home/ocsqe/data/cluster-2020-01-21.1/auth/kubeconfig get Project namespace-test-47b4cf16197941bd9a004243ae8deea6 -n namespace-test-47b4cf16197941bd9a004243ae8deea6 -o yaml.
Error is Error from server (NotFound): namespaces "namespace-test-47b4cf16197941bd9a004243ae8deea6" not found

16:30:45 - MainThread - ocs_ci.ocs.ocp - ERROR - Number of attempts to get resource reached!
16:30:45 - MainThread - ocs_ci.ocs.ocp - INFO - Project namespace-test-47b4cf16197941bd9a004243ae8deea6 got deleted successfully
```

Reporting a warning and an error because the namespace which is being deleted during the teardown is misleading. The code should not report error for an expected outcome. This increases a cost of debugging when something actually goes wrong.

Changed files:
ocs_ci/ocs/ocp.py

diff --git a/ocs_ci/ocs/ocp.py b/ocs_ci/ocs/ocp.py
index b5067eb7..71ca23b8 100644
--- a/ocs_ci/ocs/ocp.py
+++ b/ocs_ci/ocs/ocp.py
@@ -209,7 +209,7 @@ class OCP(object):
                 )
                 retry -= 1
                 if not retry:
-                    log.error("Number of attempts to get resource reached!")
+                    log.warning("Number of attempts to get resource reached!")
                     raise
                 else:
                     log.info(
Repository: red-hat-storage/ocs-ci
Issue: Bug - CSI_ENABLE_SNAPSHOTTER value check is not fetching deployment by name

The code is looking at the order of deployments to get rook-ceph-operator deployment. The rook-ceph-operator deployment can be in any order in the list of deployments in csv. It is required to fetch the deployment with name instead of sticking to the structure of the output.

```
for pod_obj in provisioner_pods:
            pod_info = pod_obj.get()
            for container, image in get_images(data=pod_info).items():
                assert ('snapshot' not in container) and ('snapshot' not in image), (
                    f"Snapshot container is present in {pod_obj.name} pod. "
                    f"Container {container}. Image {image}"
                )
        assert {'name': 'CSI_ENABLE_SNAPSHOTTER', 'value': 'false'} in (
            ocs_csv.get()['spec']['install']['spec']['deployments'][0]['spec']['template']['spec']['containers'][0]['env']
>       ), "CSI_ENABLE_SNAPSHOTTER value is not set to 'false'."
E       AssertionError: CSI_ENABLE_SNAPSHOTTER value is not set to 'false'.```

Changed files:
ocs_ci/ocs/resources/storage_cluster.py

diff --git a/ocs_ci/ocs/resources/storage_cluster.py b/ocs_ci/ocs/resources/storage_cluster.py
index f85fbe5d..c132fa99 100644
--- a/ocs_ci/ocs/resources/storage_cluster.py
+++ b/ocs_ci/ocs/resources/storage_cluster.py
@@ -273,8 +273,12 @@ def ocs_install_verification(timeout=600, skip_osd_distribution_check=False):
                 f"Snapshot container is present in {pod_obj.name} pod. "
                 f"Container {container}. Image {image}"
             )
+    deployments = ocs_csv.get()['spec']['install']['spec']['deployments']
+    rook_ceph_operator_deployment = [
+        deployment_val for deployment_val in deployments if deployment_val['name'] == 'rook-ceph-operator'
+    ]
     assert {'name': 'CSI_ENABLE_SNAPSHOTTER', 'value': 'false'} in (
-        ocs_csv.get()['spec']['install']['spec']['deployments'][0]['spec']['template']['spec']['containers'][0]['env']
+        rook_ceph_operator_deployment[0]['spec']['template']['spec']['containers'][0]['env']
     ), "CSI_ENABLE_SNAPSHOTTER value is not set to 'false'."
     log.info("Verified: CSI snapshotter is not present.")
Repository: red-hat-storage/ocs-ci
Issue: AWS UPI deployment fails with cluster prefix

Ex1: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/1655/console
Ex2: https://ocs4-jenkins-csb-ocsqe.apps.ocp4.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/1659/console

The cluster prefix in these deployments is being passed to ocs-ci correctly but being altered by something in the UPI script process. The UPI script tries to create a VPC with a hyphen at the start (missing the prefix) and fails. We need to debug the UPI process and determine what is changing the cluster name string and removing the prefix.

Changed files:
ocs_ci/deployment/aws.py

diff --git a/ocs_ci/deployment/aws.py b/ocs_ci/deployment/aws.py
index 2d7c6ea1..df9fbc2c 100644
--- a/ocs_ci/deployment/aws.py
+++ b/ocs_ci/deployment/aws.py
@@ -256,6 +256,7 @@ class AWSUPI(AWSBase):
             # setup necessary env variables
             upi_env_vars = {
                 "INSTANCE_NAME_PREFIX": config.ENV_DATA["cluster_name"],
+                "CLUSTER_NAME": config.ENV_DATA["cluster_name"],
                 "AWS_REGION": config.ENV_DATA["region"],
                 "rhcos_ami": config.ENV_DATA.get("rhcos_ami"),
                 "route53_domain_name": config.ENV_DATA["base_domain"],
@@ -275,6 +276,8 @@ class AWSUPI(AWSBase):
                 logger.info("Setting ENV VAR to preserve bootstrap node")
                 upi_env_vars["remove_bootstrap"] = "No"
 
+            logger.info(f"UPI ENV VARS = {upi_env_vars}")
+
             for key, value in upi_env_vars.items():
                 if value:
                     os.environ[key] = value
Repository: carpentries/amy
Issue: Crash when viewing upcoming event

This is either a data import error or a bug in Amy; either way, Amy should give something better than a stack trace when something goes wrong.
1.  `make clean`
2.  `make database` to import faked data.
3.  `make serve` to run.
4.  go to http://127.0.0.1:8000/workshops/
5.  fifth event down under 'upcoming events' is labelled 2015-04-xx-ttt-nih - click on it.
6.  boom: "ObjectDoesNotExist at /workshops/event/2015-04-xx-ttt-nih"

Other upcoming events display properly, and occurs as of b2e0c6058.


Changed files:
workshops/models.py

diff --git a/workshops/models.py b/workshops/models.py
index dec147a7..605b1ab8 100644
--- a/workshops/models.py
+++ b/workshops/models.py
@@ -228,10 +228,13 @@ class Event(models.Model):
         The real indicator is the 'published' flag.
         '''
 
+        # match event ID
         if re.match(r'^\\d+$', ident):
             return Event.objects.get(id=ident)
 
-        if re.match(r'^\\d{4}-\\d{2}-\\d{2}-.+$', ident):
+        # match event slug in format YYYY-MM-**** (ie. this works for
+        # YYYY-MMMM-DD-**** too)
+        if re.match(r'^\\d{4}-\\d{2}-.+$', ident):
             return Event.objects.get(slug=ident)
 
         raise ObjectDoesNotExist(ident)
Repository: carpentries/amy
Issue: Cannot validate events, missing `check_file` function

I can't validate events, because `check_file` function is missing.

```
Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/workshops/event/2012-02-20-itcp/validate

Django Version: 1.7.1
Python Version: 2.7.9
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'workshops')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware')


Traceback:
File "/home/piotr/workspace/anaconda/envs/amy/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/piotr/workspace/amy/workshops/views.py" in validate_event
  287.             valid, error_messages = check_file(page_url, response.text)

Exception Type: NameError at /workshops/event/2012-02-20-itcp/validate
Exception Value: global name 'check_file' is not defined
```


Changed files:
workshops/views.py

diff --git a/workshops/views.py b/workshops/views.py
index d5ccd8fe..f0d3ecd1 100644
--- a/workshops/views.py
+++ b/workshops/views.py
@@ -15,6 +15,8 @@ from workshops.models import Site, Airport, Event, Person, Task, Cohort, Skill,
 from workshops.forms import SearchForm, InstructorsForm, PersonBulkAddForm
 from workshops.util import earth_distance
 
+from workshops.check import check_file
+
 from workshops.models import \\
     Airport, \\
     Award, \\
Repository: carpentries/amy
Issue: Remove the groups list.

The "new person" form shouldn't mention groups unless/until we start using Django's permission groups.


Changed files:
workshops/views.py

diff --git a/workshops/views.py b/workshops/views.py
index 548decbc..21c8c7d8 100644
--- a/workshops/views.py
+++ b/workshops/views.py
@@ -187,6 +187,13 @@ class AirportUpdate(LoginRequiredMixin, UpdateViewContext):
 #------------------------------------------------------------
 
 
+PERSON_FIELDS = [
+        field.name for field in Person._meta.fields
+    ] + [
+        'user_permissions',
+    ]
+
+
 @login_required
 def all_persons(request):
     '''List all persons.'''
@@ -380,12 +387,12 @@ def person_bulk_add_confirmation(request):
 
 class PersonCreate(LoginRequiredMixin, CreateViewContext):
     model = Person
-    fields = '__all__'
+    fields = PERSON_FIELDS
 
 
 class PersonUpdate(LoginRequiredMixin, UpdateViewContext):
     model = Person
-    fields = '__all__'
+    fields = PERSON_FIELDS
     pk_url_kwarg = 'person_id'
Repository: carpentries/amy
Issue: Event's start date must be earlier than it's end date

We should enforce that the start date is early that the end date.


Changed files:
workshops/test/test_event.py, workshops/forms.py

diff --git a/workshops/forms.py b/workshops/forms.py
index 179ad7fa..40dc4778 100644
--- a/workshops/forms.py
+++ b/workshops/forms.py
@@ -206,6 +206,15 @@ class EventForm(forms.ModelForm):
 
         return data
 
+    def clean_end(self):
+        """Ensure end >= start."""
+        start = self.cleaned_data['start']
+        end = self.cleaned_data['end']
+
+        if start and end and end < start:
+            raise forms.ValidationError('Must not be earlier than start date.')
+        return end
+
     class Meta:
         model = Event
         # reorder fields, don't display 'deleted' field
diff --git a/workshops/test/test_event.py b/workshops/test/test_event.py
index a401897e..c204c57f 100644
--- a/workshops/test/test_event.py
+++ b/workshops/test/test_event.py
@@ -324,6 +324,43 @@ class TestEventViews(TestBase):
         response = self.client.post(reverse('event_add'), data)
         assert response.status_code == 302
 
+    def test_start_date_gte_end_date(self):
+        """Ensure event's start date is earlier than it's end date.
+
+        This is a regression test for
+        https://github.com/swcarpentry/amy/issues/436"""
+        data = {
+            'site': self.test_site.id,
+            'tags': [self.test_tag.id],
+            'slug': 'test-event',
+            'start': date(2015, 7, 20),
+            'end': date(2015, 7, 19),
+        }
+        response = self.client.post(reverse('event_add'), data)
+        assert response.status_code == 200
+        content = response.content.decode('utf-8')
+        assert 'Must not be earlier than start date' in content
+
+        data = {
+            'site': self.test_site.id,
+            'tags': [self.test_tag.id],
+            'slug': 'test-event',
+            'start': date(2015, 7, 20),
+            'end': date(2015, 7, 20),
+        }
+        response = self.client.post(reverse('event_add'), data)
+        assert response.status_code == 302
+
+        data = {
+            'site': self.test_site.id,
+            'tags': [self.test_tag.id],
+            'slug': 'test-event2',
+            'start': date(2015, 7, 20),
+            'end': date(2015, 7, 21),
+        }
+        response = self.client.post(reverse('event_add'), data)
+        assert response.status_code == 302
+
 
 class TestEventNotes(TestBase):
     """Make sure notes once written are saved forever!"""
Repository: carpentries/amy
Issue: Search doesn't seem to include "site"

To replicate, search for "MSU". The event **2015-05-30-hastac-swc** isn't returned even though it was held at MSU and the site is "msu.edu".


Changed files:
workshops/views.py

diff --git a/workshops/views.py b/workshops/views.py
index 6fa79e5e..280ec037 100644
--- a/workshops/views.py
+++ b/workshops/views.py
@@ -820,7 +820,9 @@ def search(request):
             if form.cleaned_data['in_events']:
                 events = Event.objects.filter(
                     Q(slug__contains=term) |
-                    Q(notes__contains=term))
+                    Q(notes__contains=term) |
+                    Q(site__domain__contains=term) |
+                    Q(site__fullname__contains=term))
             if form.cleaned_data['in_persons']:
                 persons = Person.objects.filter(
                     Q(personal__contains=term) |
Repository: carpentries/amy
Issue: Add administrative page

The application needs an administrative page so that we can add admin accounts, etc.


Changed files:
workshops/admin.py, amy/settings.py, amy/urls.py

diff --git a/amy/settings.py b/amy/settings.py
index 2fa00c4f..3f2d33f7 100644
--- a/amy/settings.py
+++ b/amy/settings.py
@@ -79,7 +79,6 @@ INSTALLED_APPS = (
     'workshops',
     # this should be after 'workshops' because templates in
     # 'templates/registration/' clash
-    'django.contrib.admin',
     'crispy_forms',
     'selectable',
     'django_countries',
diff --git a/amy/urls.py b/amy/urls.py
index 714ffa79..1606fd14 100644
--- a/amy/urls.py
+++ b/amy/urls.py
@@ -3,7 +3,6 @@ from django.contrib import admin
 
 urlpatterns = patterns('',
     url(r'^workshops/', include('workshops.urls')),
-    url(r'^admin/', include(admin.site.urls)),
     # url(r'^account/', include('django.contrib.auth.urls')),
 
     # django views for authentication
diff --git a/workshops/admin.py b/workshops/admin.py
deleted file mode 100644
index 2db15657..00000000
--- a/workshops/admin.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from django.contrib import admin
-from workshops.models import Airport
-from workshops.models import Site
-
-admin.site.register(Airport)
-admin.site.register(Site)
Repository: carpentries/amy
Issue: Bulk-upload: New User added even in case of existing user

According to #225 , If a existing user with case insensitive email Id is uploaded with a new task, a new user is being added with that case insensitive email and no task is being added to the existing user.


Changed files:
workshops/util.py

diff --git a/workshops/util.py b/workshops/util.py
index 00a539d8..d9f698f5 100644
--- a/workshops/util.py
+++ b/workshops/util.py
@@ -184,7 +184,7 @@ def create_uploaded_persons_tasks(data):
                 if fields['email']:
                     # we should use existing Person or create one
                     p, created = Person.objects.get_or_create(
-                        email=fields['email'], defaults=fields
+                        email__iexact=fields['email'], defaults=fields
                     )
 
                     if created:
Repository: carpentries/amy
Issue: Server Error when editing community roles

Reported by @ErinBecker at `/workshops/person/<id>/edit/#communityroles` and `/communityroles/role/<another id>/edit/` in production.

Reproduced on test AMY:
```
TypeError at /communityroles/role/25/edit/
the JSON object must be str, bytes or bytearray, not list
Error during template rendering
In template /webapps/test-amy.carpentries.org/repo/amy/templates/bootstrap4/field.html, error at line 39

{% crispy_field field %}
```

Also encountered locally when creating a community role and immediately editing it:
```
Exception Value: | 'NoneType' object is not iterable
/home/eli/amy/amy/communityroles/fields.py, line 20, in get_context

```

Changed files:
amy/communityroles/tests/test_community_role_views.py, amy/communityroles/fields.py

diff --git a/amy/communityroles/fields.py b/amy/communityroles/fields.py
index 0a459e46..6ea1cf49 100644
--- a/amy/communityroles/fields.py
+++ b/amy/communityroles/fields.py
@@ -63,7 +63,7 @@ class CustomKeysWidget(forms.TextInput):
             values = data.getlist(name)
         except AttributeError:
             values = data.get(name, [])
-        return list(zip(self.labels, values))
+        return json.dumps(list(zip(self.labels, values)))
 
     def value_omitted_from_data(
         self, data: QueryDict, files: MultiValueDict, name: str
diff --git a/amy/communityroles/tests/test_community_role_views.py b/amy/communityroles/tests/test_community_role_views.py
index b74f46ee..ea7546e5 100644
--- a/amy/communityroles/tests/test_community_role_views.py
+++ b/amy/communityroles/tests/test_community_role_views.py
@@ -119,6 +119,26 @@ class TestCommunityRoleUpdateView(TestCommunityRoleMixin, TestBase):
         self.assertEqual(page.status_code, 302)  # should redirect to new object
         self.assertRedirects(page, redirect)
 
+    def test_invalid_form_data_returns_page_with_error(self):
+        """
+        Regression test for issue #2336.
+        """
+        # Arrange
+        url = reverse("communityrole_edit", args=[self.community_role.pk])
+        data = self.data.copy()
+        data.pop(
+            "communityrole-award"
+        )  # creates a validation error as link_to_award is True
+
+        # Act
+        page = self.client.post(url, data)
+
+        # Assert
+        self.assertIn(
+            "Award is required with community role Test Role",
+            page.content.decode("utf-8"),
+        )
+
 
 class TestCommunityRoleDeleteView(TestCommunityRoleMixin, TestBase):
     def setUp(self):
Repository: carpentries/amy
Issue: Bulk import workflow encounters IntegrityError when saving an organization

Currently, we allow organizations with the domain that contains the `www` subdomain. For eg: Google can exist as `www.google.com` as well as `google.com`, leading to `IntegrityError` while saving the first while the second exists.

Shouldn't we enforce one URL pattern and trim/add `www` to the `domain` field when saving an organization?

Testcase:

``` py
In [5]: Organization.objects.create(fullname='Google', domain='google.com')
Out[5]: <Organization: google.com>

In [6]: Organization.objects.create(fullname='Google', domain='www.google.com')
---------------------------------------------------------------------------
IntegrityError                            Traceback (most recent call last)
```


Changed files:
pydata/api.py

diff --git a/pydata/api.py b/pydata/api.py
index da287a2c..6402c07c 100644
--- a/pydata/api.py
+++ b/pydata/api.py
@@ -4,6 +4,7 @@ from urllib.parse import urljoin, urlparse
 
 import requests
 from django.conf import settings
+from django.db.models import Q
 
 from workshops.models import (
     Person,
@@ -118,14 +119,18 @@ class SponsorshipAPIClient(BaseAPIClient):
     model = Sponsorship
 
     def parse(self, sponsor):
+        domain = urlparse(sponsor['external_url']).netloc
+        organization = Organization.objects.filter(
+            Q(fullname=sponsor['name']) | Q(domain=domain)
+        ).first()
+        if not organization:
+            organization = Organization.objects.create(
+                fullname=sponsor['name'],
+                domain=domain,
+                notes=sponsor['annotation'],
+            )
         return Sponsorship(
-            organization=Organization.objects.get_or_create(
-                domain=urlparse(sponsor['external_url']).netloc,
-                defaults={
-                    'fullname': sponsor['name'],
-                    'notes': sponsor['annotation'],
-                },
-            )[0],
+            organization=organization,
             event=self.event,
             amount=sponsor['level']['cost'],
             contact=Person.objects.get_or_create(
Repository: carpentries/amy
Issue: Bulk-upload: empty entries in csv files

Today I got a strange output form a bulk upload:
"Successfully uploaded 37 persons and 24 tasks"
The entries were actully 24 and this number was recorded in the workshop page.

Piotr found out that there were empty cells in the csv files. 

Empty cells should be ignored.


Changed files:
workshops/util.py, workshops/test/test_util.py

diff --git a/workshops/test/test_util.py b/workshops/test/test_util.py
index b5a8b6bc..071bd3b0 100644
--- a/workshops/test/test_util.py
+++ b/workshops/test/test_util.py
@@ -56,6 +56,15 @@ john,m,doe,john@doe.com"""
         person_tasks, empty_fields = self.compute_from_string(bad_csv)
         self.assertTrue('email' in empty_fields)
 
+    def test_csv_with_empty_lines(self):
+        csv = """personal,middle,family,emailaddress
+john,m,doe,john@doe.com
+,,,"""
+        person_tasks, empty_fields = self.compute_from_string(csv)
+        self.assertEqual(len(person_tasks), 1)
+        person = person_tasks[0]
+        self.assertEqual(person['personal'], 'john')
+
     def test_empty_field(self):
         ''' Ensure we don't mis-order fields given blank data '''
         csv = """personal,middle,family,email
diff --git a/workshops/util.py b/workshops/util.py
index baacb65f..fbb7dcb8 100644
--- a/workshops/util.py
+++ b/workshops/util.py
@@ -71,6 +71,10 @@ def upload_person_task_csv(stream):
     empty_fields = set()
 
     for row in reader:
+        # skip empty lines in the CSV
+        if not any(row.values()):
+            continue
+
         entry = {}
         for col in Person.PERSON_UPLOAD_FIELDS:
             try:
Repository: carpentries/amy
Issue: Persons API bug - duplicate entries returned

Think I discovered a possible bug in persons API - possibly other parts of the API are affected. Certain number of duplicated entries are returned, e.g. when searching for UK instructors at page 2 of results there are two entries for "m.wallis@leeds.ac.uk", on page 3 there are 2 entries for "m.croucher@sheffield.ac.uk", and so on.

See for example page 2 of results:
https://amy.software-carpentry.org/api/v1/persons/?country=GB&email=&family=&github=&is_instructor=true&may_contact=&middle=&order_by=&page=2&personal=&publish_profile=&username= 

Changed files:
api/views.py

diff --git a/api/views.py b/api/views.py
index b2f43520..3cd1d66f 100644
--- a/api/views.py
+++ b/api/views.py
@@ -736,7 +736,8 @@ class PersonViewSet(viewsets.ReadOnlyModelViewSet):
     """List many people or retrieve only one person."""
     permission_classes = (IsAuthenticated, IsAdmin)
     queryset = Person.objects.all().select_related('airport') \\
-                     .prefetch_related('badges', 'domains', 'lessons')
+                     .prefetch_related('badges', 'domains', 'lessons') \\
+                     .distinct()
     serializer_class = PersonSerializer
     pagination_class = StandardResultsSetPagination
     filterset_class = PersonFilter
Repository: carpentries/amy
Issue: Membership edit fails due to form errors

The error is due to missing `comment` field in `MembershipForm`.

`MembershipForm.save()` should be placed in `MembershipCreateForm`, because that's where `comment` field is and that's what should trigger the signal.

Changed files:
amy/fiscal/forms.py, amy/fiscal/tests/test_membership.py

diff --git a/amy/fiscal/forms.py b/amy/fiscal/forms.py
index 448f3c7e..d672cb94 100644
--- a/amy/fiscal/forms.py
+++ b/amy/fiscal/forms.py
@@ -92,16 +92,6 @@ class MembershipForm(forms.ModelForm):
             'additional_instructor_training_seats',
         ]
 
-    def save(self, *args, **kwargs):
-        res = super().save(*args, **kwargs)
-
-        create_comment_signal.send(sender=self.__class__,
-                                   content_object=res,
-                                   comment=self.cleaned_data['comment'],
-                                   timestamp=None)
-
-        return res
-
 
 class MembershipCreateForm(MembershipForm):
     comment = MarkdownxFormField(
@@ -116,6 +106,16 @@ class MembershipCreateForm(MembershipForm):
         fields = MembershipForm.Meta.fields.copy()
         fields.append('comment')
 
+    def save(self, *args, **kwargs):
+        res = super().save(*args, **kwargs)
+
+        create_comment_signal.send(sender=self.__class__,
+                                   content_object=res,
+                                   comment=self.cleaned_data['comment'],
+                                   timestamp=None)
+
+        return res
+
 
 class SponsorshipForm(WidgetOverrideMixin, forms.ModelForm):
 
diff --git a/amy/fiscal/tests/test_membership.py b/amy/fiscal/tests/test_membership.py
index 61055293..a31be537 100644
--- a/amy/fiscal/tests/test_membership.py
+++ b/amy/fiscal/tests/test_membership.py
@@ -4,7 +4,7 @@ import itertools
 from django.urls import reverse
 from django_comments.models import Comment
 
-from fiscal.forms import MembershipCreateForm
+from fiscal.forms import MembershipCreateForm, MembershipForm
 from workshops.tests.base import TestBase
 from workshops.models import Membership, Organization, Event, Role, Tag, Task
 
@@ -210,3 +210,23 @@ class TestMembership(TestBase):
         comment = Comment.objects.first()
         self.assertEqual(comment.comment, 'This is a test comment.')
         self.assertIn(comment, Comment.objects.for_model(obj))
+
+    def test_workshop_edit_form_no_comment(self):
+        """Ensure workshop edit form works and doesn't provide `comment` field.
+
+        This is a regression test against #1437:
+        https://github.com/swcarpentry/amy/issues/1437
+        """
+        self.assertNotIn('comment', MembershipForm.Meta.fields)
+
+        self.assertEqual(Comment.objects.count(), 0)
+        data = {
+            'organization': self.org_alpha.pk,
+            'variant': 'partner',
+            'contribution_type': 'financial',
+            'additional_instructor_training_seats': 0,
+            'seats_instructor_training': 0,
+        }
+        form = MembershipForm(data, instance=self.current)
+        form.save()
+        self.assertEqual(Comment.objects.count(), 0)
Repository: carpentries/amy
Issue: "Forgotten" fields in Workshop Request -> Self Organised Submission migration

These fields aren't migrated:
* `created_at`
* `last_updated_at`
* `state`
* `country`
* `assigned_to`
* `event`

https://github.com/carpentries/amy/blob/develop/amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py

Changed files:
amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py

diff --git a/amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py b/amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py
index 44ba8135..738e667f 100644
--- a/amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py
+++ b/amy/extrequests/migrations/0006_migrate_workshop_requests_to_selforg_submissions.py
@@ -14,6 +14,12 @@ def migrate(apps, schema_editor):
 
     for wr in self_org_wr:
         sos = SelfOrganizedSubmission.objects.create(
+            created_at=wr.created_at,
+            last_updated_at=wr.last_updated_at,
+            state=wr.state,
+            assigned_to=wr.assigned_to,
+            event=wr.event,
+
             personal=wr.personal,
             family=wr.family,
             email=wr.email,
@@ -27,6 +33,7 @@ def migrate(apps, schema_editor):
             workshop_url=wr.self_organized_github,
             workshop_types_other='',
             workshop_types_other_explain='',
+            country=wr.country,
             language=wr.language,
 
             public_event=wr.public_event,
Repository: carpentries/amy
Issue: displaying duplicate entries

@chrismedrela writes: It looks like we need to replace:

`return self.filter(conditional).order_by('slug', 'id')`

with:

`return self.filter(conditional).order_by('slug', 'id').distinct()`

in `EventQuerySet.unpublished_events` [1].

The same change may be necessary in `EventQuerySet.published_events`

[1] https://github.com/swcarpentry/amy/blob/develop/workshops/models.py#L767


Changed files:
workshops/models.py, workshops/test/test_event.py

diff --git a/workshops/models.py b/workshops/models.py
index cacc7533..5eff9150 100644
--- a/workshops/models.py
+++ b/workshops/models.py
@@ -764,13 +764,13 @@ class EventQuerySet(models.query.QuerySet):
         """Return events considered as unpublished (see
         `unpublished_conditional` above)."""
         conditional = self.unpublished_conditional()
-        return self.filter(conditional).order_by('slug', 'id')
+        return self.filter(conditional).order_by('slug', 'id').distinct()
 
     def published_events(self):
         """Return events considered as published (see `unpublished_conditional`
         above)."""
         conditional = self.unpublished_conditional()
-        return self.exclude(conditional).order_by('-start', 'id')
+        return self.exclude(conditional).order_by('-start', 'id').distinct()
 
     def uninvoiced_events(self):
         '''Return a queryset for events that have not yet been invoiced.
@@ -787,7 +787,7 @@ class EventQuerySet(models.query.QuerySet):
 
     def ttt(self):
         """Return only TTT events."""
-        return self.filter(tags__name='TTT')
+        return self.filter(tags__name='TTT').distinct()
 
 
 @reversion.register
diff --git a/workshops/test/test_event.py b/workshops/test/test_event.py
index f3f14f12..f8e94a19 100644
--- a/workshops/test/test_event.py
+++ b/workshops/test/test_event.py
@@ -110,6 +110,25 @@ class TestEvent(TestBase):
         self.assertNotIn(event_considered_published,
                          Event.objects.unpublished_events())
 
+    def test_unpublished_events_displayed_once(self):
+        """Regression test: unpublished events can't be displayed more than
+        once on the dashboard.  Refer to #977."""
+        self._setUpTags()
+        unpublished_event = Event.objects.create(
+            slug='2016-10-20-unpublished',
+            start=date(2016, 10, 20),
+            end=date(2016, 10, 21),
+            host=Organization.objects.first(),
+            administrator=Organization.objects.first(),
+        )
+        unpublished_event.tags = Tag.objects.filter(name__in=['TTT', 'online'])
+
+        unpublished = Event.objects.unpublished_events().select_related('host')
+        self.assertIn(unpublished_event, unpublished)
+        self.assertEqual(
+            1, len(unpublished.filter(slug='2016-10-20-unpublished'))
+        )
+
     def test_delete_event(self):
         """Make sure deleted event and its tasks are no longer accessible."""
         event = Event.objects.get(slug="starts-today-ongoing")
Repository: carpentries/amy
Issue: Event form: improper validation of curricula and selected tags

`EventForm.clean_curricula` does try to validate selected curricula (corresponding tags must exist), but doesn't work with Mix & Match curriculum.

Changed files:
amy/workshops/tests/test_event.py, amy/workshops/forms.py

diff --git a/amy/workshops/forms.py b/amy/workshops/forms.py
index fecc1179..f389cf8e 100644
--- a/amy/workshops/forms.py
+++ b/amy/workshops/forms.py
@@ -557,10 +557,12 @@ class EventForm(forms.ModelForm):
         tags = self.cleaned_data['tags']
 
         try:
-            expected_tags = [
-                c.slug.split("-")[0].upper() for c in curricula
-                if c.active and not c.unknown
-            ]
+            expected_tags = []
+            for c in curricula:
+                if c.active and c.carpentry:
+                    expected_tags.append(c.carpentry)
+                elif c.active and c.mix_match:
+                    expected_tags.append('Circuits')
         except (ValueError, TypeError):
             expected_tags = []
 
diff --git a/amy/workshops/tests/test_event.py b/amy/workshops/tests/test_event.py
index 43a3a00d..d12d6d7f 100644
--- a/amy/workshops/tests/test_event.py
+++ b/amy/workshops/tests/test_event.py
@@ -737,6 +737,34 @@ class TestEventViews(TestBase):
         form = EventForm(data)
         self.assertNotIn('curricula', form.errors)
 
+    def test_curricula_circuits_tag(self):
+        """Ensure validation of `curricula` and `tags` fields."""
+        # missing tags
+        data = {
+            'slug': '2018-10-28-curriculum',
+            'host': self.org_alpha.pk,
+            # there has to be some tag
+            'tags': [Tag.objects.get(name='DC').pk],
+            'curricula': [
+                Curriculum.objects.get(slug='swc-python').pk,
+                Curriculum.objects.get(mix_match=True).pk,
+            ],
+        }
+        form = EventForm(data)
+        # we're missing SWC and Circuits
+        self.assertIn('curricula', form.errors)
+
+        # try adding SWC tag
+        data['tags'].append(Tag.objects.get(name='SWC').pk)
+        form = EventForm(data)
+        self.assertIn('curricula', form.errors)
+        # now we're missing only circuits
+
+        # try adding Circuits tag
+        data['tags'].append(Tag.objects.get(name='Circuits').pk)
+        form = EventForm(data)
+        self.assertNotIn('curricula', form.errors)
+
 
 class TestEventMerging(TestBase):
     def setUp(self):
Repository: carpentries/amy
Issue: Wrong permission name for `WorkshopInquiry*` views

* `WorkshopInquiryChange`
* `WorkshopInquirySetState`
* `workshopinquiry_accept_event`

... all use wrong permission name (`extrequests.change_workshopinquiry` instead of `extrequests.change_workshopinquiryrequest`).

Changed files:
amy/extrequests/views.py

diff --git a/amy/extrequests/views.py b/amy/extrequests/views.py
index 2070a5d5..8b104525 100644
--- a/amy/extrequests/views.py
+++ b/amy/extrequests/views.py
@@ -236,7 +236,7 @@ class WorkshopInquiryDetails(OnlyForAdminsMixin, AMYDetailView):
 
 class WorkshopInquiryChange(OnlyForAdminsMixin, PermissionRequiredMixin,
                             AMYUpdateView):
-    permission_required = 'extrequests.change_workshopinquiry'
+    permission_required = 'extrequests.change_workshopinquiryrequest'
     model = WorkshopInquiryRequest
     pk_url_kwarg = 'inquiry_id'
     form_class = WorkshopInquiryRequestAdminForm
@@ -244,7 +244,7 @@ class WorkshopInquiryChange(OnlyForAdminsMixin, PermissionRequiredMixin,
 
 
 class WorkshopInquirySetState(OnlyForAdminsMixin, ChangeRequestStateView):
-    permission_required = 'extrequests.change_workshopinquiry'
+    permission_required = 'extrequests.change_workshopinquiryrequest'
     model = WorkshopInquiryRequest
     pk_url_kwarg = 'inquiry_id'
     state_url_kwarg = 'state'
@@ -252,7 +252,7 @@ class WorkshopInquirySetState(OnlyForAdminsMixin, ChangeRequestStateView):
 
 
 @admin_required
-@permission_required(['extrequests.change_workshopinquiry',
+@permission_required(['extrequests.change_workshopinquiryrequest',
                       'workshops.add_event'],
                      raise_exception=True)
 def workshopinquiry_accept_event(request, inquiry_id):
Repository: carpentries/amy
Issue: 500 Server error when 'manual attendance' field is not set

Having 'Manual attendance' field unset causes a 500 Server Error. I tried leaving it unset for workshops where the attendance number is not known yet. It defaults to '0' - not sure if this is the desired behaviour (to have attendance at 0). If yes, disregard the issue.

Changed files:
amy/workshops/tests/test_event.py, amy/workshops/forms.py

diff --git a/amy/workshops/forms.py b/amy/workshops/forms.py
index a1da9863..03a4051e 100644
--- a/amy/workshops/forms.py
+++ b/amy/workshops/forms.py
@@ -575,6 +575,11 @@ class EventForm(forms.ModelForm):
 
         return curricula
 
+    def clean_manual_attendance(self):
+        """Regression: #1608 - fix 500 server error when field is cleared."""
+        manual_attendance = self.cleaned_data['manual_attendance'] or 0
+        return manual_attendance
+
     def save(self, *args, **kwargs):
         res = super().save(*args, **kwargs)
 
diff --git a/amy/workshops/tests/test_event.py b/amy/workshops/tests/test_event.py
index 41af7fdf..25fb4eb4 100644
--- a/amy/workshops/tests/test_event.py
+++ b/amy/workshops/tests/test_event.py
@@ -567,6 +567,24 @@ class TestEventViews(TestBase):
         f = EventForm(data)
         self.assertTrue(f.is_valid())
 
+    def test_empty_manual_attendance(self):
+        """Ensure we don't get 500 server error when field is left with empty
+        value.
+
+        This is a regression test for
+        https://github.com/swcarpentry/amy/issues/1608."""
+
+        data = {
+            'slug': '2016-06-30-test-event',
+            'host': self.test_host.id,
+            'tags': [self.test_tag.id],
+            'manual_attendance': '',
+        }
+        f = EventForm(data)
+        self.assertTrue(f.is_valid())
+        event = f.save()
+        self.assertEqual(event.manual_attendance, 0)
+
     def test_number_of_attendees_increasing(self):
         """Ensure event.attendance gets bigger after adding new learners."""
         event = Event.objects.get(slug='test_event_0')
Repository: carpentries/amy
Issue: Redirect index page

https://amy.software-carpentry.org/ should redirect to https://amy.software-carpentry.org/workshops rather than returning a 404.

Changed files:
workshops/test/test_security.py, amy/urls.py

diff --git a/amy/urls.py b/amy/urls.py
index 1d681aa9..5007764b 100644
--- a/amy/urls.py
+++ b/amy/urls.py
@@ -2,8 +2,10 @@ from django.conf import settings
 from django.conf.urls import include, url
 from django.contrib import admin
 from django.contrib.auth import views as auth_views
+from django.views.generic import RedirectView
 
 urlpatterns = [
+    url(r'^$', RedirectView.as_view(pattern_name='dispatch')),
     url(r'^workshops/admin/', include(admin.site.urls)),
 ]
 if settings.ENABLE_PYDATA:
diff --git a/workshops/test/test_security.py b/workshops/test/test_security.py
index 6006e909..809a3a83 100644
--- a/workshops/test/test_security.py
+++ b/workshops/test/test_security.py
@@ -2,7 +2,7 @@ from django.contrib.admin import ModelAdmin
 from django.contrib.admin.sites import AdminSite
 from django.contrib.auth.models import Group
 from django.core.urlresolvers import reverse
-from django.views.generic import View
+from django.views.generic import View, RedirectView
 from rest_framework.views import APIView
 
 from amy import urls
@@ -284,12 +284,15 @@ class TestViews(TestBase):
                     is_model_admin = isinstance(model_admin, ModelAdmin)
                     is_admin_site = isinstance(admin_site, AdminSite)
                     is_api_view = class_ is not None and issubclass(class_, APIView)
+                    is_redirect_view = class_ is not None and issubclass(class_, RedirectView)
                     is_view = class_ is not None and issubclass(class_, View)
 
                     if is_model_admin or is_admin_site:
                         pass  # ignore admin views
                     elif is_api_view:
                         pass  # ignore REST API views
+                    elif is_redirect_view:
+                        pass  # ignore pure redirect views
                     else:
                         assert is_view
Repository: carpentries/amy
Issue: Transform github repo url into website url

If a website's github repo is https://github.com/barack_obama/2016-01-01-whitehouse then we know the website is https://barack_obama.github.io/2016-01-01-whitehouse

When using the "Import" or "Update from URL" it requires the _website_ rather than the _repo_.  However, if given the repo, can't we have AMY transform the URL into a valid website URL?  

Often I get a repo url from an instructor (esp. for self organized) workshops and I manually do that transformation myself.


Changed files:
workshops/util.py, workshops/test/test_util.py

diff --git a/workshops/test/test_util.py b/workshops/test/test_util.py
index 35a4796f..eae27459 100644
--- a/workshops/test/test_util.py
+++ b/workshops/test/test_util.py
@@ -99,6 +99,8 @@ Other content.
             'http://swcarpentry.github.io/workshop-template',
             'https://swcarpentry.github.com/workshop-template',
             'http://swcarpentry.github.com/workshop-template/',
+            'http://github.com/swcarpentry/workshop-template',
+            'https://github.com/swcarpentry/workshop-template',
         ]
         expected_url = ('https://raw.githubusercontent.com/swcarpentry/'
                         'workshop-template/gh-pages/index.html')
diff --git a/workshops/util.py b/workshops/util.py
index 142cfb16..77b14ed6 100644
--- a/workshops/util.py
+++ b/workshops/util.py
@@ -473,8 +473,7 @@ class WrongWorkshopURL(ValueError):
     (see `generate_url_to_event_index` below)."""
 
     def __str__(self):
-        return ('Event\\'s URL doesn\\'t match Github website format '
-                '"http://user.github.io/2015-12-08-workshop".')
+        return 'Event\\'s URL doesn\\'t match Github website or repo format.'
 
 
 def generate_url_to_event_index(website_url):
@@ -482,11 +481,11 @@ def generate_url_to_event_index(website_url):
     file in GitHub repository."""
     template = ('https://raw.githubusercontent.com/{name}/{repo}'
                 '/gh-pages/index.html')
-    regex = Event.WEBSITE_REGEX
 
-    results = regex.match(website_url)
-    if results:
-        return template.format(**results.groupdict()), results.group('repo')
+    for regex in [Event.WEBSITE_REGEX, Event.REPO_REGEX]:
+        results = regex.match(website_url)
+        if results:
+            return template.format(**results.groupdict()), results.group('repo')
     raise WrongWorkshopURL()