Repository: fossasia/open-event-server

Issue:
Export CSV files of Attendees with random, duplicate and unsorted data
There are several issues with the exported CSV file of attendees and orders. To replicate the issue download a CSV from the attendee or order overview.

![Screenshot from 2023-06-26 13-50-37](https://github.com/fossasia/open-event-frontend/assets/1583873/c791b410-32a8-47c2-9150-cb0c8006708c)

Then check the CSVs of attendees and orders. They have random and duplicate data.

1. Firstly in a perfect way the columns would be ordered according to the order of questions on the web front-end.
2. Secondly, the columns with the data on Email consent, photo consent and other columns appear several times
3. Thirdly specifically the following columns are in the wrong place:
* Column AF "Email" should be behind column "O (Zipcode)
* Column P "áa" seems to have some random information
* Column Q to AE should be coming after column BG
4. The answer in the forms shows "True" or "False". Normal users do not know what this means. Just show the answer in the default language, e.g. "Yes".

[attendees-dc360898141111eea9e90242c0a87005.csv](https://github.com/fossasia/open-event-frontend/files/11868359/attendees-dc360898141111eea9e90242c0a87005.csv)



Discussion:
Issue is fixed with PR https://github.com/fossasia/open-event-server/pull/8952

## Detail

1. Ensuring exact column order from front-end matches order from exported file is not practical

The table in the front-end is a dynamic table with ability for drag-drop reordering 


![Image](https://github.com/fossasia/open-event-frontend/assets/49814026/17f99eea-355f-47ef-8e15-44850aa9cd87)

This means in order to sync the exact dynamic order form from front-end to back-end would require designing a sync feature just for form, I'm not sure focusing on this is practical. 

2. and 3.
Export functions have been rewritten to ensure  columns duplication

Example new csv generated: 

[attendees-9bf09f2a1a4011ee99640242ac130005.csv](https://github.com/fossasia/open-event-frontend/files/11946488/attendees-9bf09f2a1a4011ee99640242ac130005.csv)

4. Converted True/False in csv field to Yes/No



![Image](https://github.com/fossasia/open-event-frontend/assets/49814026/e3e52267-8427-44bf-a158-8f8145411e3a)



The issue is unfortunately not solved for existing events.
1. Questions are not sorted according to the order laid out in the form.
2. Questions appear twice. The reason why this is happening is probably because there are two ticket types with two different forms (several in-person tickets types and virtual ticket). If forms of different ticket types have the same question, there should only be one column in the CSV.

Compare
[attendees-0edb28e224e211ee9ed10242c0a81005.csv](https://github.com/fossasia/open-event-server/files/12073788/attendees-0edb28e224e211ee9ed10242c0a81005.csv)

Resolving patch:
diff --git a/app/api/helpers/csv_jobs_util.py b/app/api/helpers/csv_jobs_util.py
--- a/app/api/helpers/csv_jobs_util.py
+++ b/app/api/helpers/csv_jobs_util.py
@@ -72,25 +72,28 @@ def export_attendees_csv(attendees, custom_forms, attendee_form_dict):
             'Payment Type': str(attendee.order.paid_via)
             if attendee.order and attendee.order.paid_via
             else '',
-            'Payment Mode':  str(attendee.order.payment_mode)
+            'Payment Mode': str(attendee.order.payment_mode)
             if attendee.order and attendee.order.payment_mode
             else '',
-            'Ticket Name':  str(attendee.ticket.name)
-            if attendee.ticket and attendee.ticket.name else '',
+            'Ticket Name': str(attendee.ticket.name)
+            if attendee.ticket and attendee.ticket.name
+            else '',
             'Ticket Price': str(attendee.ticket.price)
             if attendee.ticket and attendee.ticket.price
             else '0',
             'Ticket Type': str(attendee.ticket.type)
-            if attendee.ticket and attendee.ticket.type else '',
+            if attendee.ticket and attendee.ticket.type
+            else '',
             'Tax ID': str(attendee.order.tax_business_info)
             if attendee.order.tax_business_info
             else '',
             'Address': str(attendee.order.address) if attendee.order.address else '',
-            'Company':  str(attendee.order.company) if attendee.order.company else '',
+            'Company': str(attendee.order.company) if attendee.order.company else '',
             'Country': str(attendee.order.country) if attendee.order.country else '',
-            'State':  str(attendee.order.state) if attendee.order.state else '',
+            'State': str(attendee.order.state) if attendee.order.state else '',
             'City': str(attendee.order.city) if attendee.order.city else '',
             'Zipcode': str(attendee.order.zipcode) if attendee.order.zipcode else '',
+            'Email': '',
         }
 
         for field in custom_forms:
@@ -106,12 +109,19 @@ def export_attendees_csv(attendees, custom_forms, attendee_form_dict):
             converted_header = attendee_form_dict.get(key)
             if field.is_complex:
                 fields_dict = attendee.complex_field_values
-                data[converted_header] = fields_dict.get(
-                    field.identifier, '') if fields_dict else ''
+                converted_header = field.identifier
+                data[converted_header] = (
+                    fields_dict.get(field.identifier, '') if fields_dict else ''
+                )
             else:
-                dict_value = (getattr(attendee, field.identifier, ''))
-                dict_value = "Yes" if str(dict_value) == "True" \
-                    else "No" if str(dict_value) == "False" else dict_value
+                dict_value = getattr(attendee, field.identifier, '')
+                dict_value = (
+                    "Yes"
+                    if str(dict_value) == "True"
+                    else "No"
+                    if str(dict_value) == "False"
+                    else dict_value
+                )
                 data[converted_header] = dict_value
         return_dict_list.append(data)
 
diff --git a/app/api/helpers/tasks.py b/app/api/helpers/tasks.py
--- a/app/api/helpers/tasks.py
+++ b/app/api/helpers/tasks.py
@@ -21,9 +21,11 @@
     From,
     Mail,
 )
+from sqlalchemy import asc, desc, func
 
 from app.api.chat.rocket_chat import rename_rocketchat_room
 from app.api.exports import event_export_task_base
+from app.api.helpers.csv_jobs_util import export_attendees_csv
 from app.api.helpers.db import safe_query, save_to_db
 from app.api.helpers.files import (
     create_save_image_sizes,
@@ -39,7 +41,7 @@
 from app.api.imports import import_event_task_base
 from app.instance import create_app
 from app.models import db
-from app.models.custom_form import CustomForms, ATTENDEE_CUSTOM_FORM
+from app.models.custom_form import ATTENDEE_CUSTOM_FORM, CustomForms
 from app.models.discount_code import DiscountCode
 from app.models.event import Event
 from app.models.exhibitor import Exhibitor
@@ -52,8 +54,8 @@
 from app.models.user import User
 from app.models.user_follow_group import UserFollowGroup
 from app.settings import get_settings
+
 from .import_helpers import update_import_job
-from app.api.helpers.csv_jobs_util import export_attendees_csv
 
 """
 Define all API v2 celery tasks here
@@ -537,9 +539,15 @@ def export_order_pdf_task(self, event_id):
 
 @celery.task(base=RequestContextTask, name='export.attendees.csv', bind=True)
 def export_attendees_csv_task(self, event_id):
-    attendees = db.session.query(TicketHolder).filter_by(event_id=event_id)
-    custom_forms = db.session.query(CustomForms).filter_by(
-        event_id=event_id, form=CustomForms.TYPE.ATTENDEE, is_included=True
+    attendees = (
+        db.session.query(TicketHolder)
+        .filter_by(event_id=event_id)
+        .order_by(desc(func.date(TicketHolder.created_at)))
+    )
+    custom_forms = (
+        db.session.query(CustomForms)
+        .filter_by(event_id=event_id, form=CustomForms.TYPE.ATTENDEE, is_included=True)
+        .order_by(asc("position"))
     )
 
     field_headers = list(ATTENDEE_CUSTOM_FORM.keys())
@@ -548,14 +556,14 @@ def custom_form_validation(cf_orm, field_headers):
         # set() is O(1) in membership testing
         field_headers_set = set(field_headers)
         forms_result = [None] * len(field_headers_set)
+        index_append = 0
 
         for row in cf_orm:
             if row.field_identifier in field_headers_set:
-
                 field_headers_set.discard(row.field_identifier)
-                index_append = field_headers.index(row.field_identifier)
                 # forms_result.append(row)
-                forms_result.insert(index_append, row)
+            forms_result.insert(index_append, row)
+            index_append += 1
 
         forms_result = [e for e in forms_result if e is not None]
         return forms_result
@@ -569,7 +577,11 @@ def custom_form_validation(cf_orm, field_headers):
         file_path = os.path.join(filedir, filename)
 
         dict_list = export_attendees_csv(attendees, custom_forms, ATTENDEE_CUSTOM_FORM)
-        csv_headers = list(dict_list[0].keys())
+        csv_headers = []
+        for row in dict_list:
+            for key in row.keys():
+                if key is not None and key not in csv_headers:
+                    csv_headers.append(key)
 
         with open(file_path, "w") as temp_file:
             writer = csv.DictWriter(temp_file, fieldnames=csv_headers)


Test patch: