internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: e-valuation/EvaP

Issue:
Django 2.1 upgrade
https://docs.djangoproject.com/en/2.1/releases/2.1/

There is a guide for upgrading: https://docs.djangoproject.com/en/2.1/howto/upgrade-version/

Basically
* Read the release notes
* update dependencies
* run tests with `python -Wa` and solve deprecation warnings
* put the new django into the requirements
* run tests, fix failures if any
* run tests with `python -Wa` and solve deprecation warnings again
* if there was any new feature in the release notes that might help us, use it

also, we need to check the installed python version on production, django 2.1 supports python 3.5 and newer.


Discussion:
production is currently running python 3.5, so that's fine.

Resolving patch:
diff --git a/evap/evaluation/migrations/0062_replace_textanswer_id_with_uuid.py b/evap/evaluation/migrations/0062_replace_textanswer_id_with_uuid.py
--- a/evap/evaluation/migrations/0062_replace_textanswer_id_with_uuid.py
+++ b/evap/evaluation/migrations/0062_replace_textanswer_id_with_uuid.py
@@ -33,13 +33,12 @@ class Migration(migrations.Migration):
             name='uuid',
             field=models.UUIDField(primary_key=False, default=uuid.uuid4, serialize=False, editable=False),
         ),
-        # rename the old id field before deleting it at the end of the
-        # migration for compatibility with the sqlite driver
-        migrations.RenameField(
-            model_name='textanswer',
-            old_name='id',
-            new_name='old_id'
-        ),
+        # this causes trouble with sqlite. We have two open bug reports with django for this, see
+        # https://code.djangoproject.com/ticket/29790 and https://code.djangoproject.com/ticket/28541
+        # We can not get this to work with sqlite and postgres right now and we want django2.1, we only
+        # support postgres here. For sqlite, you need to rename the field here and move the RemoveField to
+        # the end.
+        migrations.RemoveField(model_name='textanswer', name='id'),
         migrations.RenameField(
             model_name='textanswer',
             old_name='uuid',
@@ -54,5 +53,4 @@ class Migration(migrations.Migration):
             name='textanswer',
             options={'ordering': ['id'], 'verbose_name': 'text answer', 'verbose_name_plural': 'text answers'},
         ),
-        migrations.RemoveField(model_name='textanswer', name='old_id'),
     ]


Test patch:
Repository: e-valuation/EvaP

Issue:
Translation of import warning card titles missing
When importing course data and receiving warnings, they are grouped in cards. The card titles are not translated although they should be.


Discussion:

Resolving patch:
diff --git a/evap/staff/importers.py b/evap/staff/importers.py
--- a/evap/staff/importers.py
+++ b/evap/staff/importers.py
@@ -3,7 +3,7 @@
 
 from django.conf import settings
 from django.db import transaction
-from django.utils.translation import ugettext as _
+from django.utils.translation import ugettext_lazy, ugettext as _
 from django.utils.safestring import mark_safe
 from django.core.exceptions import ValidationError
 
@@ -590,10 +590,10 @@ def make_users_active(user_list):
 
 # Dictionary to translate internal keys to UI strings.
 WARNING_DESCRIPTIONS = {
-    ExcelImporter.W_NAME: _("Name mismatches"),
-    ExcelImporter.W_INACTIVE: _("Inactive users"),
-    ExcelImporter.W_EMAIL: _("Email mismatches"),
-    ExcelImporter.W_DUPL: _("Possible duplicates"),
-    ExcelImporter.W_GENERAL: _("General warnings"),
-    EnrollmentImporter.W_MANY: _("Unusually high number of enrollments")
+    ExcelImporter.W_NAME: ugettext_lazy("Name mismatches"),
+    ExcelImporter.W_INACTIVE: ugettext_lazy("Inactive users"),
+    ExcelImporter.W_EMAIL: ugettext_lazy("Email mismatches"),
+    ExcelImporter.W_DUPL: ugettext_lazy("Possible duplicates"),
+    ExcelImporter.W_GENERAL: ugettext_lazy("General warnings"),
+    EnrollmentImporter.W_MANY: ugettext_lazy("Unusually high number of enrollments")
 }


Test patch:
Repository: e-valuation/EvaP

Issue:
Evaluation preview button visibility
As a teaching assistant, I might be a contributor to a given course and therefore get my own feedback in the main evaluation. If that course also has an exam evaluation, I see that listed on my "own evaluations" page with the option to preview the questionnaire. However, as not being responsible, I miss the access rights to preview the linked page, resulting in an error.

I would like to either don't have the preview button (it already knows while rendering that page that I am not a contributor, shown through the corresponding icon next to the exam evaluation title) or to give me the rights to preview the questionnaire.


Discussion:
The preview page should be available for all responsibles and contributors. All of these users should see the button to the preview page. If a user is not responsible for the evaluation's course and is not a contributor in this evaluation there should be no button and the page should not be accessible.

Resolving patch:
diff --git a/evap/evaluation/templatetags/evaluation_filters.py b/evap/evaluation/templatetags/evaluation_filters.py
--- a/evap/evaluation/templatetags/evaluation_filters.py
+++ b/evap/evaluation/templatetags/evaluation_filters.py
@@ -99,6 +99,10 @@ def is_user_editor_or_delegate(evaluation, user):
     return evaluation.is_user_editor_or_delegate(user)
 
 
+@register.filter
+def is_user_responsible_or_contributor_or_delegate(evaluation, user):
+    return evaluation.is_user_responsible_or_contributor_or_delegate(user)
+
 @register.filter
 def message_class(level):
     return {


Test patch:
Repository: e-valuation/EvaP

Issue:
Remove AtLeastOne contributor constraint
We have a constraint that every evaluation must have at least one contributor. This was useful to define at least one person with access to the results.
Now, evaluations belong to courses which must have at least one responsible. This responsible can access the results also when not explicitly being added as a contributor. When creating exam evaluations we don't want to add additional questions and use contributions only for visibility permissions. If only the responsible is involved, we currently unnecessarily have to add the responsible as a contributor. So the constraint should be removed.


Discussion:

Resolving patch:
diff --git a/evap/staff/forms.py b/evap/staff/forms.py
--- a/evap/staff/forms.py
+++ b/evap/staff/forms.py
@@ -511,7 +511,7 @@ def clean(self):
             raise forms.ValidationError(_('You must have at least one of these.'))
 
 
-class ContributionFormSet(AtLeastOneFormSet):
+class ContributionFormSet(BaseInlineFormSet):
     def __init__(self, data=None, *args, **kwargs):
         data = self.handle_moved_contributors(data, **kwargs)
         super().__init__(data, *args, **kwargs)


Test patch:
diff --git a/evap/staff/tests/test_forms.py b/evap/staff/tests/test_forms.py
--- a/evap/staff/tests/test_forms.py
+++ b/evap/staff/tests/test_forms.py
@@ -341,6 +341,23 @@ def test_take_deleted_contributions_into_account(self):
         formset = contribution_formset(instance=evaluation, form_kwargs={'evaluation': evaluation}, data=data)
         self.assertTrue(formset.is_valid())
 
+    def test_there_can_be_no_contributions(self):
+        """
+            Tests that there can also be no contribution
+            Regression test for #1347
+        """
+        evaluation = mommy.make(Evaluation)
+        contribution_formset = inlineformset_factory(Evaluation, Contribution, formset=ContributionFormSet, form=ContributionForm, extra=0)
+
+        data = to_querydict({
+            'contributions-TOTAL_FORMS': 0,
+            'contributions-INITIAL_FORMS': 1,
+            'contributions-MAX_NUM_FORMS': 5,
+        })
+
+        formset = contribution_formset(instance=evaluation, form_kwargs={'evaluation': evaluation}, data=data)
+        self.assertTrue(formset.is_valid())
+
     def test_hidden_and_managers_only(self):
         """
             Asserts that hidden questionnaires are shown to managers only if they are already selected for a
Repository: e-valuation/EvaP

Issue:
Inline datatables localization files to speed up first paint
Right now, datatables gets the localization file in form of a URL (see [datatables.html](https://github.com/fsr-itse/EvaP/blob/master/evap/evaluation/templates/datatables.html)), that is, it starts an ajax request when it starts processing the tables, and waits with processing until the result has been received.

both locales should be included into the compressed javascript or inlined into the html template so they are loaded earlier.

we do something similar for the [bootstrap datetimepicker](https://github.com/fsr-itse/EvaP/blob/028b6301e3eed446d93ae8675030d82c68d46886/evap/evaluation/templates/bootstrap_datetimepicker.html). unfortunately, it's not that easy in this case, since the localization files are json files, not javascript files.

one approach would be to turn the json files to js files, and simply putting the datastructure inside into a variable with the name of the corresponding locale.


Discussion:
i'm currently working on completely removing the datatables from the results pages so this might become obsolete.
we are using datatables on other pages as well.
We can maybe replace those as well...

Resolving patch:
diff --git a/evap/evaluation/templatetags/evaluation_filters.py b/evap/evaluation/templatetags/evaluation_filters.py
--- a/evap/evaluation/templatetags/evaluation_filters.py
+++ b/evap/evaluation/templatetags/evaluation_filters.py
@@ -38,10 +38,10 @@
 # values for approval states shown to staff
 StateValues = namedtuple('StateValues', ('order', 'icon', 'filter', 'description'))
 APPROVAL_STATES = {
-    'new': StateValues(0, 'fas fa-circle icon-yellow', 'fa-circle icon-yellow', _('In preparation')),
-    'prepared': StateValues(2, 'far fa-square icon-gray', 'fa-square icon-gray', _('Awaiting editor review')),
-    'editor_approved': StateValues(1, 'far fa-check-square icon-yellow', 'fa-check-square icon-yellow', _('Approved by editor, awaiting manager review')),
-    'approved': StateValues(3, 'far fa-check-square icon-green', 'fa-check-square icon-green', _('Approved by manager')),
+    'new': StateValues(0, 'fas fa-circle icon-yellow', 'new', _('In preparation')),
+    'prepared': StateValues(2, 'far fa-square icon-gray', 'prepared', _('Awaiting editor review')),
+    'editor_approved': StateValues(1, 'far fa-check-square icon-yellow', 'editor_approved', _('Approved by editor, awaiting manager review')),
+    'approved': StateValues(3, 'far fa-check-square icon-green', 'approved', _('Approved by manager')),
 }
 
 


Test patch:
Repository: e-valuation/EvaP

Issue:
Add proxy users to selection lists for staff
#1304 specified that staff users still need to be able to select proxy users from selection lists. This wasn't implemented in #1336, so this should be changed.
Staff users should be able to select proxy users in all user selection fields.


Discussion:

Resolving patch:
diff --git a/evap/contributor/forms.py b/evap/contributor/forms.py
--- a/evap/contributor/forms.py
+++ b/evap/contributor/forms.py
@@ -70,6 +70,7 @@ def __init__(self, *args, **kwargs):
 
         self.fields['questionnaires'].queryset = Questionnaire.objects.contributor_questionnaires().filter(
             Q(visibility=Questionnaire.EDITORS) | Q(contributions__evaluation=self.evaluation)).distinct()
+        self.fields['contributor'].queryset = UserProfile.objects.exclude(is_active=False).exclude(is_proxy_user=True)
 
 
 class DelegatesForm(forms.ModelForm):
diff --git a/evap/staff/forms.py b/evap/staff/forms.py
--- a/evap/staff/forms.py
+++ b/evap/staff/forms.py
@@ -368,7 +368,7 @@ def save(self, *args, **kw):
 
 
 class ContributionForm(forms.ModelForm):
-    contributor = UserModelChoiceField(queryset=UserProfile.objects.exclude(is_active=False).exclude(is_proxy_user=True))
+    contributor = UserModelChoiceField(queryset=UserProfile.objects.exclude(is_active=False))
     responsibility = forms.ChoiceField(widget=forms.RadioSelect(), choices=Contribution.RESPONSIBILITY_CHOICES)
     evaluation = forms.ModelChoiceField(Evaluation.objects.all(), disabled=True, required=False, widget=forms.HiddenInput())
     questionnaires = forms.ModelMultipleChoiceField(


Test patch:
diff --git a/evap/staff/tests/test_forms.py b/evap/staff/tests/test_forms.py
--- a/evap/staff/tests/test_forms.py
+++ b/evap/staff/tests/test_forms.py
@@ -389,6 +389,12 @@ def test_hidden_and_managers_only(self):
         self.assertEqual(expected, set(formset.forms[0].fields['questionnaires'].queryset.all()))
         self.assertEqual(expected, set(formset.forms[1].fields['questionnaires'].queryset.all()))
 
+    def test_staff_can_select_proxy_user(self):
+        proxy_user = mommy.make(UserProfile, is_proxy_user=True)
+        course = mommy.make(Course, semester=mommy.make(Semester))
+        form = CourseForm(instance=course)
+        self.assertIn(proxy_user, form.fields['responsibles'].queryset)
+
 
 class ContributionFormset775RegressionTests(TestCase):
     """
@@ -519,6 +525,12 @@ def test_uniqueness_constraint_error_shown(self):
         self.assertIn('name_de', form.errors)
         self.assertEqual(form.errors['name_de'], ['Course with this Semester and Name (german) already exists.'])
 
+    def test_that_proxy_user_can_be_responsible(self):
+        course = mommy.make(Course, semester=mommy.make(Semester), degrees=[mommy.make(Degree)])
+        proxy = mommy.make(UserProfile, is_proxy_user=True, is_active=True)
+        form = CourseForm(instance=course)
+        self.assertIn(proxy, form.fields['responsibles'].queryset)
+
 
 class EvaluationFormTests(TestCase):
     def test_evaluation_form_same_name(self):
Repository: e-valuation/EvaP

Issue:
Badge for external responsible
Evaluations with (at least one) external responsible for their Course should get an additional `badge-warning` (in `evaluation_badges.html`), visible for managers and displaying `"External responsible"`.


Discussion:

Resolving patch:
diff --git a/evap/evaluation/models.py b/evap/evaluation/models.py
--- a/evap/evaluation/models.py
+++ b/evap/evaluation/models.py
@@ -323,6 +323,10 @@ def midterm_grade_documents(self):
     def responsibles_names(self):
         return ", ".join([responsible.full_name for responsible in self.responsibles.all().order_by("last_name")])
 
+    @property
+    def has_external_responsible(self):
+        return any(responsible.is_external for responsible in self.responsibles.all())
+
     @property
     def all_evaluations_finished(self):
         return not self.evaluations.exclude(state__in=['evaluated', 'reviewed', 'published']).exists()


Test patch:
diff --git a/evap/staff/tests/test_views.py b/evap/staff/tests/test_views.py
--- a/evap/staff/tests/test_views.py
+++ b/evap/staff/tests/test_views.py
@@ -376,6 +376,19 @@ def test_access_to_semester_with_archived_results(self):
         # managers can access the page
         self.app.get('/staff/semester/2', user='manager', status=200)
 
+    @override_settings(INSTITUTION_EMAIL_DOMAINS=["institution.com"])
+    def test_badge_for_external_responsibles(self):
+        responsible = mommy.make(UserProfile, email='a@institution.com')
+        course = mommy.make(Course, semester=self.semester, responsibles=[responsible])
+        mommy.make(Evaluation, course=course)
+        response = self.app.get(self.url, user='manager')
+        self.assertNotContains(response, 'External responsible')
+
+        responsible.email = 'r@external.com'
+        responsible.save()
+        response = self.app.get(self.url, user='manager')
+        self.assertContains(response, 'External responsible')
+
 
 class TestGetEvaluationsWithPrefetchedData(TestCase):
     def test_get_evaluations_with_prefetched_data(self):
Repository: e-valuation/EvaP

Issue:
Error message if course already exists on import
When importing enrollment data including at least one course with a name that already exists for this semester in the database, the importer fails silently. This shouldn't be the case.
The test import should display an error message if a course with the same (German or English) name already exists in the current semester.

There should also be a test for that.


Discussion:
In EnrollmentImporter.check_evaluation_data_correctness(), there is this line:
```python
already_exists = Evaluation.objects.filter(course__semester=semester, name_de=evaluation_data.name_de).exists()
```
It only checks for German names, maybe this has something to do with it.

Resolving patch:
diff --git a/evap/staff/importers.py b/evap/staff/importers.py
--- a/evap/staff/importers.py
+++ b/evap/staff/importers.py
@@ -278,9 +278,10 @@ def consolidate_enrollment_data(self):
 
     def check_evaluation_data_correctness(self, semester):
         for evaluation_data in self.evaluations.values():
-            already_exists = Evaluation.objects.filter(course__semester=semester, name_de=evaluation_data.name_de).exists()
-            if already_exists:
+            if Course.objects.filter(semester=semester, name_en=evaluation_data.name_en).exists():
                 self.errors.append(_("Course {} does already exist in this semester.").format(evaluation_data.name_en))
+            if Course.objects.filter(semester=semester, name_de=evaluation_data.name_de).exists():
+                self.errors.append(_("Course {} does already exist in this semester.").format(evaluation_data.name_de))
 
         degree_names = set()
         for evaluation_data in self.evaluations.values():


Test patch:
diff --git a/evap/staff/tests/test_importers.py b/evap/staff/tests/test_importers.py
--- a/evap/staff/tests/test_importers.py
+++ b/evap/staff/tests/test_importers.py
@@ -220,6 +220,19 @@ def test_invalid_file_error(self):
         self.assertIn('Errors occurred while parsing the input data. No data was imported.', errors_test)
         self.assertEqual(UserProfile.objects.count(), original_user_count)
 
+    def test_duplicate_course_error(self):
+        with open(self.filename_valid, "rb") as excel_file:
+            excel_content = excel_file.read()
+
+        semester = baker.make(Semester)
+        baker.make(Course, name_de="Stehlen", name_en="Stehlen", semester=semester)
+        baker.make(Course, name_de="Shine", name_en="Shine", semester=semester)
+
+        __, __, errors = EnrollmentImporter.process(excel_content, semester, None, None, test_run=False)
+
+        self.assertIn("Course Stehlen does already exist in this semester.", errors)
+        self.assertIn("Course Shine does already exist in this semester.", errors)
+
 
 class TestPersonImporter(TestCase):
     @classmethod
Repository: e-valuation/EvaP

Issue:
Crash when previewing results
Steps to reproduce:
1. Log in as evap
2. Select `Preview results` on "Subject Specific English 2" (breaks in German too)

Interestingly, I don't get any errors when viewing "Beautiful Data", which is also a preview, or "3d Computer Graphic - Software Architecture", which is public.

I am on `25401c018f8f304001d67dcd22f52861d01ddf3b` and reloaded the test data right before testing.

```
AttributeError at /results/semester/21/evaluation/347

'Course' object has no attribute 'evaluation_weight_sum'
```

Can anyone confirm this?


Discussion:
Can confirm, refreshing the results cache doesn't change anything (makes sense because the evaluation is not published yet). @pixunil can you please look into it?

We should also add a test which opens a result preview page.

Resolving patch:
diff --git a/evap/results/views.py b/evap/results/views.py
--- a/evap/results/views.py
+++ b/evap/results/views.py
@@ -216,7 +216,7 @@ def evaluation_detail(request, semester_id, evaluation_id):
     # if the evaluation is not published, the rendered results are not cached, so we need to attach distribution
     # information for rendering the distribution bar
     if evaluation.state != 'published':
-        evaluation = get_evaluations_with_prefetched_data([evaluation])[0]
+        evaluation = get_evaluations_with_course_result_attributes(get_evaluations_with_prefetched_data([evaluation]))[0]
 
     template_data = dict(
         evaluation=evaluation,


Test patch:
diff --git a/evap/results/tests/test_views.py b/evap/results/tests/test_views.py
--- a/evap/results/tests/test_views.py
+++ b/evap/results/tests/test_views.py
@@ -230,6 +230,24 @@ def test_wrong_state(self):
         url = '/results/semester/%s/evaluation/%s' % (self.semester.id, evaluation.id)
         self.app.get(url, user='student', status=403)
 
+    def test_preview_without_rating_answers(self):
+        evaluation = baker.make(Evaluation, state='evaluated', course=baker.make(Course, semester=self.semester))
+        url = f'/results/semester/{self.semester.id}/evaluation/{evaluation.id}'
+        self.app.get(url, user='manager')
+
+    def test_preview_with_rating_answers(self):
+        evaluation = baker.make(Evaluation, state='evaluated', course=baker.make(Course, semester=self.semester))
+        questionnaire = baker.make(Questionnaire, type=Questionnaire.Type.TOP)
+        likert_question = baker.make(Question, type=Question.LIKERT, questionnaire=questionnaire, order=1)
+        evaluation.general_contribution.questionnaires.set([questionnaire])
+        participants = baker.make(UserProfile, _quantity=20)
+        evaluation.participants.set(participants)
+        evaluation.voters.set(participants)
+        baker.make(RatingAnswerCounter, question=likert_question, contribution=evaluation.general_contribution, answer=1, count=20)
+
+        url = f'/results/semester/{self.semester.id}/evaluation/{evaluation.id}'
+        self.app.get(url, user='manager')
+
 
 class TestResultsSemesterEvaluationDetailViewFewVoters(WebTest):
     @classmethod
Repository: e-valuation/EvaP

Issue:
Questionnaire filtering on Evaluation creation broken
When creating a new Evaluation in the staff create evaluation form, in the general questionnaire list questionnaires which are set to be hidden but have never been used are listed. They should be hidden instead.


Discussion:

Resolving patch:
diff --git a/evap/staff/forms.py b/evap/staff/forms.py
--- a/evap/staff/forms.py
+++ b/evap/staff/forms.py
@@ -254,8 +254,10 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.fields['course'].queryset = Course.objects.filter(semester=semester)
 
-        self.fields['general_questionnaires'].queryset = Questionnaire.objects.general_questionnaires().filter(
-            Q(visibility=Questionnaire.Visibility.MANAGERS) | Q(visibility=Questionnaire.Visibility.EDITORS) | Q(contributions__evaluation=self.instance)).distinct()
+        visible_questionnaires = Q(visibility__in=(Questionnaire.Visibility.MANAGERS, Questionnaire.Visibility.EDITORS))
+        if self.instance.pk is not None:
+            visible_questionnaires |= Q(contributions__evaluation=self.instance)
+        self.fields['general_questionnaires'].queryset = Questionnaire.objects.general_questionnaires().filter(visible_questionnaires).distinct()
 
         self.fields['participants'].queryset = UserProfile.objects.exclude(is_active=False)
 


Test patch:
diff --git a/evap/staff/tests/test_forms.py b/evap/staff/tests/test_forms.py
--- a/evap/staff/tests/test_forms.py
+++ b/evap/staff/tests/test_forms.py
@@ -707,3 +707,15 @@ def test_locked_questionnaire(self):
         # Assert form is valid and locked questionnaire is removed
         form.save()
         self.assertEqual({questionnaire}, set(evaluation.general_contribution.questionnaires.all()))
+
+    def test_unused_questionnaire_visibility(self):
+        evaluation = baker.make(Evaluation)
+        questionnaire = baker.make(Questionnaire, visibility=Questionnaire.Visibility.HIDDEN, type=Questionnaire.Type.TOP)
+
+        form = EvaluationForm(instance=evaluation, semester=evaluation.course.semester)
+        self.assertNotIn(questionnaire, form.fields["general_questionnaires"].queryset)
+
+        evaluation.general_contribution.questionnaires.add(questionnaire)
+
+        form = EvaluationForm(instance=evaluation, semester=evaluation.course.semester)
+        self.assertIn(questionnaire, form.fields["general_questionnaires"].queryset)