Repository: e-valuation/EvaP

Issue:
Importer doesn't create general contributions
When importing enrollment data, no `general_contribution`s are created for the newly added evaluations. This prevents managers from, e.g., using the "assign questionnaires" feature which fails with an `AttributeError`:

```
File "staff/views.py", line 802, in semester_questionnaire_assign
    evaluation.general_contribution.questionnaires.set(form.cleaned_data[evaluation.course.type.name])

Exception Type: AttributeError
Exception Value: 'NoneType' object has no attribute 'questionnaires'
```

The importer should make sure that every new evaluation has a `general_contribution`.
Importer doesn't create general contributions
When importing enrollment data, no `general_contribution`s are created for the newly added evaluations. This prevents managers from, e.g., using the "assign questionnaires" feature which fails with an `AttributeError`:

```
File "staff/views.py", line 802, in semester_questionnaire_assign
    evaluation.general_contribution.questionnaires.set(form.cleaned_data[evaluation.course.type.name])

Exception Type: AttributeError
Exception Value: 'NoneType' object has no attribute 'questionnaires'
```

The importer should make sure that every new evaluation has a `general_contribution`.


Discussion:
Hm, shouldn't the questionnaire assign view just create it if users tried to use the general contribution? Or, if we dont want to allow that, it shouldnt display an option to assign to the general questionnaire, right?

Do we currently have some invariants about what questionnaires exist for an evaluation? If so, we should probably enforce them with database contraints (#1800 could get another point)
So the problem is that the importer [uses `bulk_create`](https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/staff/importers/enrollment.py#L739) instead of `save` to be faster, but we [use `save` to create additional objects, in this case, the general contribution](https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/models.py#L456-L462).

I wouldn't want to duplicate the "when a new evaluation is created, we also must create a general contribution for it" logic across the codebase.
I think the approach of overriding `save` and having `pre_save` / `post_save` signals while also having methods like `bulk_create` that completely circumvent them is generally broken in django.

We could defer creation of the general contribution to its first access using the getter for the `general_contribution` property. However, code can also try to access the database object via `Contribution.objects.filter(..., contributor=None)`. This would bypass any getters, causing errors. Searching for `contributor=None` gives me 17 places that look like they might break if the object doesn't exist in the database.

If we use `save` instead of `bulk_*` in the importer, the importer tests will get 4 seconds slower (11s instead of 7s), and the import of somewhat realistically sized file (500 students, 100 courses, 4 enrollments per student) will get ~8s slower (currently 40s vs 48s, but that's due to a performance bug in openpyxl. There is no good reason for the importer to take longer than a second, I think).

I would really like to keep the bulk operations for performance, but the only way to make them work with `save()` overrides and `pre_save`/`post_save` signals is with a custom, third method that performs all additional `on_save()` logic, and can take a list of objects, so it can work together with bulk operations. That would be a lot of extra infrastructure.

As a quick fix, we can revert to `save()` in `update_existing_and_create_new_courses`, `store_participations_in_db`, and `update_existing_and_create_new_user_profiles`.

For production right now, calling the `save()` on all instances without a general contribution should fix the problem, but might create weird log entries.
After #1841, it's ~7s (bulk_create) vs ~12s (save) for me locally for a 6000 rows import file. Probably acceptable for now, as it only affects the import run and not test runs.
Sooooo, should we take the 8s hit in a hotfix now-ish, and make it faster again next year? Or should I review the performance PR and we merge that in the next couple of days?
> Sooooo, should we take the 8s hit in a hotfix now-ish, and make it faster again next year? Or should I review the performance PR and we merge that in the next couple of days?

The performance PR should be unrelated, I just wanted to have it to have some numbers regarding performance impact of `save()` vs `bulk_create`. We can revert to using `save` in the importer for now.

---

I just stumbled across this test:
https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/tests/test_models.py#L212-L220

and I was a bit confused here. We create an evaluation without a `general_contribution` instance and expect the model methods to still work properly?

Also, the getter for the general contribution explicitly handles the "does not exist yet" state, and explicitly returns `None` instead of just creating the object in the database:
https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/models.py#L776-L784

Is the intention here to handle `Evaluation` instances that were not yet saved to the database? The test doesn't express that as it explicitly uses `bulk_create` with `prepare` instead of just `prepare`.

I guess this comes back to @niklasmohrin's point in https://github.com/e-valuation/EvaP/issues/1839#issuecomment-1366605212: What is our invariant? I see three possibilities
1. We do not allow `Evaluations` without a general contribution.
    * This does not work for newly created instances that are not yet saved to database.
2. We do not allow `Evaluations` stored in the database without a general contribution.
    * This matches our implementations of `save` and (at least some) methods that could be called with unsaved objects.
    * It requires all accesses to instances that are not known to be in the database to handle the case that the contribution does not yet exist -- I'd bet there's cases where we don't properly check this.
    * The test above would be wrong in this case.
3. `Evaluations` might not have a contribution.
    * Shouldn't be harder to handle than 2.? Code that requires existance could just use a get-or-create getter.
    * The importer would be correct in this case.
    
We considered allowing not to have a general contribution in #1246, it seems there was no good fundamental reason to enfore existence of the general contribution, it just happened to be what we first implemented. We didn't consider `bulk_` operations circumventing `save()` though.

Related (and funny) PR / issue combination: #1259 and #1260.
Hm, shouldn't the questionnaire assign view just create it if users tried to use the general contribution? Or, if we dont want to allow that, it shouldnt display an option to assign to the general questionnaire, right?

Do we currently have some invariants about what questionnaires exist for an evaluation? If so, we should probably enforce them with database contraints (#1800 could get another point)
So the problem is that the importer [uses `bulk_create`](https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/staff/importers/enrollment.py#L739) instead of `save` to be faster, but we [use `save` to create additional objects, in this case, the general contribution](https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/models.py#L456-L462).

I wouldn't want to duplicate the "when a new evaluation is created, we also must create a general contribution for it" logic across the codebase.
I think the approach of overriding `save` and having `pre_save` / `post_save` signals while also having methods like `bulk_create` that completely circumvent them is generally broken in django.

We could defer creation of the general contribution to its first access using the getter for the `general_contribution` property. However, code can also try to access the database object via `Contribution.objects.filter(..., contributor=None)`. This would bypass any getters, causing errors. Searching for `contributor=None` gives me 17 places that look like they might break if the object doesn't exist in the database.

If we use `save` instead of `bulk_*` in the importer, the importer tests will get 4 seconds slower (11s instead of 7s), and the import of somewhat realistically sized file (500 students, 100 courses, 4 enrollments per student) will get ~8s slower (currently 40s vs 48s, but that's due to a performance bug in openpyxl. There is no good reason for the importer to take longer than a second, I think).

I would really like to keep the bulk operations for performance, but the only way to make them work with `save()` overrides and `pre_save`/`post_save` signals is with a custom, third method that performs all additional `on_save()` logic, and can take a list of objects, so it can work together with bulk operations. That would be a lot of extra infrastructure.

As a quick fix, we can revert to `save()` in `update_existing_and_create_new_courses`, `store_participations_in_db`, and `update_existing_and_create_new_user_profiles`.

For production right now, calling the `save()` on all instances without a general contribution should fix the problem, but might create weird log entries.
After #1841, it's ~7s (bulk_create) vs ~12s (save) for me locally for a 6000 rows import file. Probably acceptable for now, as it only affects the import run and not test runs.
Sooooo, should we take the 8s hit in a hotfix now-ish, and make it faster again next year? Or should I review the performance PR and we merge that in the next couple of days?
> Sooooo, should we take the 8s hit in a hotfix now-ish, and make it faster again next year? Or should I review the performance PR and we merge that in the next couple of days?

The performance PR should be unrelated, I just wanted to have it to have some numbers regarding performance impact of `save()` vs `bulk_create`. We can revert to using `save` in the importer for now.

---

I just stumbled across this test:
https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/tests/test_models.py#L212-L220

and I was a bit confused here. We create an evaluation without a `general_contribution` instance and expect the model methods to still work properly?

Also, the getter for the general contribution explicitly handles the "does not exist yet" state, and explicitly returns `None` instead of just creating the object in the database:
https://github.com/e-valuation/EvaP/blob/e889dce5b2111f42f3cda777649c19d30de165be/evap/evaluation/models.py#L776-L784

Is the intention here to handle `Evaluation` instances that were not yet saved to the database? The test doesn't express that as it explicitly uses `bulk_create` with `prepare` instead of just `prepare`.

I guess this comes back to @niklasmohrin's point in https://github.com/e-valuation/EvaP/issues/1839#issuecomment-1366605212: What is our invariant? I see three possibilities
1. We do not allow `Evaluations` without a general contribution.
    * This does not work for newly created instances that are not yet saved to database.
2. We do not allow `Evaluations` stored in the database without a general contribution.
    * This matches our implementations of `save` and (at least some) methods that could be called with unsaved objects.
    * It requires all accesses to instances that are not known to be in the database to handle the case that the contribution does not yet exist -- I'd bet there's cases where we don't properly check this.
    * The test above would be wrong in this case.
3. `Evaluations` might not have a contribution.
    * Shouldn't be harder to handle than 2.? Code that requires existance could just use a get-or-create getter.
    * The importer would be correct in this case.
    
We considered allowing not to have a general contribution in #1246, it seems there was no good fundamental reason to enfore existence of the general contribution, it just happened to be what we first implemented. We didn't consider `bulk_` operations circumventing `save()` though.

Related (and funny) PR / issue combination: #1259 and #1260.

Resolving patch:
diff --git a/evap/staff/importers/enrollment.py b/evap/staff/importers/enrollment.py
--- a/evap/staff/importers/enrollment.py
+++ b/evap/staff/importers/enrollment.py
@@ -723,8 +723,9 @@ def update_existing_and_create_new_courses(
         for course_data in course_data_iterable
         if not course_data.merge_into_course
     ]
-    Course.objects.bulk_create(new_course_objects)
-    Course.update_log_after_bulk_create(new_course_objects)
+
+    for course in new_course_objects:
+        course.save()
 
     # Create one evaluation per newly created course
     evaluation_objects = [
@@ -736,24 +737,16 @@ def update_existing_and_create_new_courses(
         )
         for course in new_course_objects
     ]
-    Evaluation.objects.bulk_create(evaluation_objects)
-    Evaluation.update_log_after_bulk_create(evaluation_objects)
+
+    for evaluation in evaluation_objects:
+        evaluation.save()
 
     # Create M2M entries for the responsibles of the newly created courses
     responsible_emails = {course_data.responsible_email for course_data in course_data_iterable}
     responsible_objs_by_email = {obj.email: obj for obj in UserProfile.objects.filter(email__in=responsible_emails)}
 
-    responsibles_through_objects = [
-        Course.responsibles.through(
-            course=course,
-            userprofile=responsible_objs_by_email[course_data_by_name_en[course.name_en].responsible_email],
-        )
-        for course in new_course_objects
-    ]
-    Course.responsibles.through.objects.bulk_create(responsibles_through_objects)
-    Course.update_log_after_m2m_bulk_create(
-        new_course_objects, responsibles_through_objects, "course_id", "userprofile_id", "responsibles"
-    )
+    for course in new_course_objects:
+        course.responsibles.add(responsible_objs_by_email[course_data_by_name_en[course.name_en].responsible_email])
 
     # Create Contributions for the responsibles of the newly created courses
     evaluation_objects_by_course = {evaluation.course: evaluation for evaluation in evaluation_objects}
@@ -766,33 +759,21 @@ def update_existing_and_create_new_courses(
         )
         for course in new_course_objects
     ]
-    Contribution.objects.bulk_create(contribution_objects)
-    Contribution.update_log_after_bulk_create(contribution_objects)
+
+    for obj in contribution_objects:
+        obj.save()
 
     # Create M2M entries for the degrees of the newly created courses
-    degree_through_objects = [
-        Course.degrees.through(course_id=course.id, degree_id=degree_obj.id)
-        for course in new_course_objects
-        for degree_obj in course_data_by_name_en[course.name_en].degrees
-    ]
-    Course.degrees.through.objects.bulk_create(degree_through_objects)
-    Course.update_log_after_m2m_bulk_create(
-        new_course_objects, degree_through_objects, "course_id", "degree_id", "degrees"
-    )
+    for course in new_course_objects:
+        course.degrees.add(*course_data_by_name_en[course.name_en].degrees)
 
-    # Create M2M entries for the degrees of the courses that are updated
     courses_to_update = semester.courses.filter(
         name_en__in=[course_data.name_en for course_data in course_data_iterable if course_data.merge_into_course]
     )
-    degree_through_objects = [
-        Course.degrees.through(course_id=course.id, degree_id=degree_obj.id)
-        for course in courses_to_update
-        for degree_obj in course_data_by_name_en[course.name_en].degrees
-    ]
-    Course.degrees.through.objects.bulk_create(degree_through_objects, ignore_conflicts=True)
-    Course.update_log_after_m2m_bulk_create(
-        courses_to_update, degree_through_objects, "course_id", "degree_id", "degrees"
-    )
+
+    # Create M2M entries for the degrees of the courses that are updated
+    for course in courses_to_update:
+        course.degrees.add(*course_data_by_name_en[course.name_en].degrees)
 
 
 def store_participations_in_db(enrollment_rows: Iterable[EnrollmentParsedRow]):
@@ -807,30 +788,11 @@ def store_participations_in_db(enrollment_rows: Iterable[EnrollmentParsedRow]):
         for evaluation in Evaluation.objects.select_related("course").filter(course__name_en__in=course_names_en)
     }
 
-    course_id_participant_id_pairs_in_file = {
-        (evaluations_by_course_name_en[row.course_data.name_en].pk, users_by_email[row.student_data.email].pk)
-        for row in enrollment_rows
-    }
-
-    existing_course_id_participant_id_pairs = {
-        (participation.evaluation_id, participation.userprofile_id)
-        for participation in Evaluation.participants.through.objects.filter(
-            evaluation__in=evaluations_by_course_name_en.values()
+    participants_by_evaluation = defaultdict(list)
+    for row in enrollment_rows:
+        participants_by_evaluation[evaluations_by_course_name_en[row.course_data.name_en]].append(
+            users_by_email[row.student_data.email]
         )
-    }
-
-    course_id_participant_id_pairs = course_id_participant_id_pairs_in_file - existing_course_id_participant_id_pairs
 
-    participants_through_objects = [
-        Evaluation.participants.through(evaluation_id=evaluation_id, userprofile_id=userprofile_id)
-        for (evaluation_id, userprofile_id) in course_id_participant_id_pairs
-    ]
-
-    Evaluation.participants.through.objects.bulk_create(participants_through_objects)
-    Evaluation.update_log_after_m2m_bulk_create(
-        evaluations_by_course_name_en.values(),
-        participants_through_objects,
-        "evaluation_id",
-        "userprofile_id",
-        "participants",
-    )
+    for evaluation, participants in participants_by_evaluation.items():
+        evaluation.participants.add(*participants)
diff --git a/evap/staff/importers/user.py b/evap/staff/importers/user.py
--- a/evap/staff/importers/user.py
+++ b/evap/staff/importers/user.py
@@ -385,5 +385,8 @@ def update_existing_and_create_new_user_profiles(
     existing_user_profiles: Iterable[UserProfile],
     new_user_profiles: Iterable[UserProfile],
 ):
-    UserProfile.objects.bulk_update(existing_user_profiles, UserData.bulk_update_fields())
-    UserProfile.objects.bulk_create(new_user_profiles)
+    for user_profile in existing_user_profiles:
+        user_profile.save()
+
+    for user_profile in new_user_profiles:
+        user_profile.save()


Test patch:
diff --git a/evap/staff/tests/test_importers.py b/evap/staff/tests/test_importers.py
--- a/evap/staff/tests/test_importers.py
+++ b/evap/staff/tests/test_importers.py
@@ -223,7 +223,7 @@ def test_validation_error(self, mocked_validation):
         )
 
     @override_settings(DEBUG=False)
-    @patch("evap.evaluation.models.UserProfile.objects.bulk_create")
+    @patch("evap.evaluation.models.UserProfile.save")
     def test_unhandled_exception(self, mocked_db_access):
         mocked_db_access.side_effect = Exception("Contact your database admin right now!")
         result, importer_log = import_users(self.valid_excel_file_content, test_run=False)