Repository: ietf-tools/datatracker

Issue:
Proceedings assumptions about lumping group meetings together no longer hold
At https://datatracker.ietf.org/meeting/115/proceedings#gen:
![image](https://user-images.githubusercontent.com/10996692/209172077-7e96b4f6-b68a-47c9-80a6-3c6c3491eaaf.png)

This is an artifact of the proceedings (and meeting materials) code originally being built assuming groups behave like working groups, and lumping all the group activities into one row made sense.

The view now needs to pay attention to sessions with different names, and purposes - this should probably have been 5 separate rows, some of which would have slides. Only one of which would have had an agenda:

```
>>> Session.objects.filter(group__acronym='emodir',meeting__number=115).values_list('name',flat=True)
<SessionQuerySet ["New Participants' Quick Connections", "New Participants' Social Hour", "Tutorial: New Participants' Overview", "New Participants' Dinner", 'WG Chairs Forum']>
>>> Session.objects.filter(group__acronym='emodir',meeting__number=115).values_list('purpose',flat=True)
<SessionQuerySet ['presentation', 'social', 'tutorial', 'social', 'open_meeting']>
``` 


Discussion:
Digging a little to understand the problem space. There are 59 sessions with non-empty names
```
>>> m = Meeting.objects.get(number=115)
>>> m.session_set.filter(timeslotassignments__schedule=m.schedule).exclude(name='').distinct().count()
59
```
belonging to these groups/group types
```
>>> pp(list(m.session_set.filter(timeslotassignments__schedule=m.schedule).exclude(name='').values_list('group__acronym', 'group__type').distinct()))
[('iesg', 'ietf'),
 ('ietf', 'ietf'),
 ('secretariat', 'ietf'),
 ('hackathon', 'team'),
 ('tools', 'team'),
 ('iepg', 'team'),
 ('emodir', 'dir'),
 ('hotrfc', 'team'),
 ('systers', 'team')]
```
The emodir you called out (type = 'dir') and hackathon (type = 'team') seem to be rendered strangely in the proceedings. The hackathon shows up as "Hackathon Saturday", which I believe is the name of its first session.

For emodir, I think you're suggesting a separate line for each named session. For hackathon, I think we'd rather have everything gathered together, but named "Hackathon" instead of using the session name. Is there a simple rule or rules to follow, or do we need to add some kind of per-group configuration? It'd be nice if we could just use the group type to decide, but I'm not sure that'll be consistent.
We can work to change the way we model the sessions if we need to in order to make this something that works generically - just means more migration and training for the secretariat. What would need to be different in the data to make this more natural?

But for hackathon, I don't know that it should all be lumped together if the sessions are named differently - one bucket of stuff for the kickoff and another for the presentations would make more sense.
I guess I'd have thought the Hackathon is more like a set of working group sessions in that it's a single event broken over a couple days instead of a collection of mostly unrelated sessions. Splitting it into a couple buckets seems reasonable to me. However, if we naively break its sessions out for 115 we'll end up with five entries:
```
>>> pp(list(m.session_set.filter(timeslotassignments__schedule=m.schedule).filter(group__acronym='hackathon').values_list('group__acronym', 'group__type', 'name', 'type').order_by('group').distinct()))
[('hackathon', 'team', 'Hackathon Saturday', 'other'),
 ('hackathon', 'team', 'Hackathon Sunday', 'other'),
 ('hackathon', 'team', 'Kickoff', 'other'),
 ('hackathon', 'team', 'Project Results Presentations', 'other'),
 ('hackathon', 'team', 'Hackdemo Happy Hour', 'other')]
```
That's not how I would personally represent it on a proceedings, but it (almost) matches what appears on the agenda and is only slightly noisier. 

The session names don't quite agree with the agenda - the "Hackathon Saturday" and "...Sunday" entries are just "Hackathon". "Kickoff" is "Hackathon Kickoff" and "Project Results Presentations" is "Hackathon Results Presentations" - a more significant difference since we probably want to include "Hackathon" on the proceedings label. It may be that changing the session names is all that needs to be done. 

For the emodir sessions, should those all appear in GEN as they do now, or do they deserve their own section? Wondering if a 'dir' group deserves its own "area" level section.
Oh - and I'd much rather *not* add to the session/group modeling here if it can be avoided.
A quick and dirty-ish patch seems to behave pretty well.

This has two changes. First, I've replaced the `{% ifchanged session.group.acronym %}` that splits up rows to be `{% ifchanged session.group.acronym session.name %}`. Second, when `session.name` is not empty, instead of displaying agendas / slides / etc for all the group's sessions, I only show them for that specific session. The outcome is:
![Screenshot from 2023-01-11 17-21-02](https://user-images.githubusercontent.com/19472766/211920800-dcbbf9b7-f9e6-4fb5-bd57-e355de829bf3.png)
and
![Screenshot from 2023-01-11 17-21-25](https://user-images.githubusercontent.com/19472766/211920830-07b1c6d9-eb5e-48d1-87b3-01548ef69d7c.png)
The rest of the proceedings seem to be unaffected (but we'll want to check more carefully). 

This will need a bit more work to be real, though, because it should probably cope with the case where there is more than one session with the same non-empty name.

One other wrinkle - the sessions are still sorted in the table according to their group's acronym. Is that ok, or do we need to sort things more meaningfully? It will take some moderately involved refactoring to do this, unfortunately.
I think giving these sessions names rather than letting them be named by the group is just adding trouble.
@flynnliz - how has having them labelled helped?
```
 ('hackathon', 'team', 'Hackathon Saturday', 'other'),
 ('hackathon', 'team', 'Hackathon Sunday', 'other'),
```
>One other wrinkle - the sessions are still sorted in the table according to their group's acronym. Is that ok, or do we need to sort things more meaningfully? It will take some moderately involved refactoring to do this, unfortunately.

Sorted by acronym is the right thing to do. The tension you see may be resolved with the separate section you suggested at:

> For the emodir sessions, should those all appear in GEN as they do now, or do they deserve their own section? Wondering if a 'dir' group deserves its own "area" level section.

But dirs are a type of group inside an area, not a thing separate from an area.

What _might_ make sense is making labelled divisions when a section (like an area) has more than one type of group in it.
That will ultimately put a bunch of pressure on the current modeling of groups (people will disagree with dir vs review vs team), but that's good natural pressure to have.
> I think giving these sessions names rather than letting them be named by the group is just adding trouble. @flynnliz - how has having them labelled helped?
> 
> ```
>  ('hackathon', 'team', 'Hackathon Saturday', 'other'),
>  ('hackathon', 'team', 'Hackathon Sunday', 'other'),
> ```

For my use, I just need to know which session object to place into which timeslot, since without the Sat/Sun in the name there's no way to tell which is which. However, I'm also not sure that it matters, if they are both just the general Hackathon entry, which isn't associated with a Meetecho link or any specific materials. If drag-and-dropping either of the identical Hackathon sessions into either the Saturday or Sunday slot would be fine, then I don't need them to be named differently. 
It looks like for 115, the saturday and sunday each had an agenda attached (same file for each) and the kickoff slides wound up attached to the saturday session rather than the kickoff session. I assume the agendas were done that way so they'd show up on the agenda page properly and that the kickoff slides are an accident allowed by everything ending up in the same row.

Resolving patch:
diff --git a/ietf/meeting/views.py b/ietf/meeting/views.py
--- a/ietf/meeting/views.py
+++ b/ietf/meeting/views.py
@@ -3586,6 +3586,74 @@ def upcoming_json(request):
     response = HttpResponse(json.dumps(data, indent=2, sort_keys=False), content_type='application/json;charset=%s'%settings.DEFAULT_CHARSET)
     return response
 
+def organize_proceedings_sessions(sessions):
+    # Collect sessions by Group, then bin by session name (including sessions with blank names).
+    # If all of a group's sessions are 'notmeet', the processed data goes in not_meeting_sessions.
+    # Otherwise, the data goes in meeting_sessions.
+    meeting_groups = []
+    not_meeting_groups = []
+    for group_acronym, group_sessions in itertools.groupby(sessions, key=lambda s: s.group.acronym):
+        by_name = {}
+        is_meeting = False
+        all_canceled = True
+        group = None
+        for s in sorted(
+                group_sessions,
+                key=lambda gs: (
+                        gs.official_timeslotassignment().timeslot.time
+                        if gs.official_timeslotassignment() else datetime.datetime(datetime.MAXYEAR, 1, 1)
+                ),
+        ):
+            group = s.group
+            if s.current_status != 'notmeet':
+                is_meeting = True
+            if s.current_status != 'canceled':
+                all_canceled = False
+            by_name.setdefault(s.name, [])
+            if s.current_status != 'notmeet' or s.sessionpresentation_set.exists():
+                by_name[s.name].append(s)  # for notmeet, only include sessions with materials
+        for sess_name, ss in by_name.items():
+            def _format_materials(items):
+                """Format session/material for template
+
+                Input is a list of (session, materials) pairs. The materials value can be a single value or a list.
+                """
+                material_times = {}  # key is material, value is first timestamp it appeared
+                for s, mats in items:
+                    timestamp = s.official_timeslotassignment().timeslot.time
+                    if not isinstance(mats, list):
+                        mats = [mats]
+                    for mat in mats:
+                        if mat and mat not in material_times:
+                            material_times[mat] = timestamp
+                n_mats = len(material_times)
+                result = []
+                if n_mats == 1:
+                    result.append({'material': list(material_times)[0]})  # no 'time' when only a single material
+                elif n_mats > 1:
+                    for mat, timestamp in material_times.items():
+                        result.append({'material': mat, 'time': timestamp})
+                return result
+
+            entry = {
+                'group': group,
+                'name': sess_name,
+                'canceled': all_canceled,
+                # pass sessions instead of the materials here so session data (like time) is easily available
+                'agendas': _format_materials((s, s.agenda()) for s in ss),
+                'minutes': _format_materials((s, s.minutes()) for s in ss),
+                'bluesheets': _format_materials((s, s.bluesheets()) for s in ss),
+                'recordings': _format_materials((s, s.recordings()) for s in ss),
+                'slides': _format_materials((s, s.slides()) for s in ss),
+                'drafts': _format_materials((s, s.drafts()) for s in ss),
+            }
+            if is_meeting:
+                meeting_groups.append(entry)
+            else:
+                not_meeting_groups.append(entry)
+    return meeting_groups, not_meeting_groups
+
+
 def proceedings(request, num=None):
 
     meeting = get_meeting(num)
@@ -3606,36 +3674,48 @@ def proceedings(request, num=None):
     today_utc = date_today(datetime.timezone.utc)
 
     schedule = get_schedule(meeting, None)
-    sessions  = add_event_info_to_session_qs(
-        Session.objects.filter(meeting__number=meeting.number)
-    ).filter(
-        Q(timeslotassignments__schedule__in=[schedule, schedule.base if schedule else None]) | Q(current_status='notmeet')
-    ).select_related().order_by('-current_status')
-    plenaries = sessions.filter(name__icontains='plenary').exclude(current_status='notmeet')
-    ietf      = sessions.filter(group__parent__type__slug = 'area').exclude(group__acronym='edu')
-    irtf      = sessions.filter(group__parent__acronym = 'irtf')
-    training  = sessions.filter(group__acronym__in=['edu','iaoc'], type_id__in=['regular', 'other',]).exclude(current_status='notmeet')
-    iab       = sessions.filter(group__parent__acronym = 'iab').exclude(current_status='notmeet')
+    sessions  = (
+        meeting.session_set.with_current_status()
+        .filter(Q(timeslotassignments__schedule__in=[schedule, schedule.base if schedule else None])
+                | Q(current_status='notmeet'))
+        .select_related()
+        .order_by('-current_status')
+    )
 
-    cache_version = Document.objects.filter(session__meeting__number=meeting.number).aggregate(Max('time'))["time__max"]
+    plenaries, _ = organize_proceedings_sessions(
+        sessions.filter(name__icontains='plenary')
+        .exclude(current_status='notmeet')
+    )
+    irtf, _ = organize_proceedings_sessions(
+        sessions.filter(group__parent__acronym = 'irtf').order_by('group__acronym')
+    )
+    training, _ = organize_proceedings_sessions(
+        sessions.filter(group__acronym__in=['edu','iaoc'], type_id__in=['regular', 'other',])
+        .exclude(current_status='notmeet')
+    )
+    iab, _ = organize_proceedings_sessions(
+        sessions.filter(group__parent__acronym = 'iab')
+        .exclude(current_status='notmeet')
+    )
 
+    ietf = sessions.filter(group__parent__type__slug = 'area').exclude(group__acronym='edu').order_by('group__parent__acronym', 'group__acronym')
     ietf_areas = []
-    for area, sessions in itertools.groupby(sorted(ietf, key=lambda s: (s.group.parent.acronym, s.group.acronym)), key=lambda s: s.group.parent):
-        sessions = list(sessions)
-        meeting_groups = set(s.group_id for s in sessions if s.current_status != 'notmeet')
-        meeting_sessions = []
-        not_meeting_sessions = []
-        for s in sessions:
-            if s.current_status == 'notmeet' and s.group_id not in meeting_groups:
-                not_meeting_sessions.append(s)
-            else:
-                meeting_sessions.append(s)
-        ietf_areas.append((area, meeting_sessions, not_meeting_sessions))
+    for area, area_sessions in itertools.groupby(
+            ietf,
+            key=lambda s: s.group.parent
+    ):
+        meeting_groups, not_meeting_groups = organize_proceedings_sessions(area_sessions)
+        ietf_areas.append((area, meeting_groups, not_meeting_groups))
+
+    cache_version = Document.objects.filter(session__meeting__number=meeting.number).aggregate(Max('time'))["time__max"]
 
     with timezone.override(meeting.tz()):
         return render(request, "meeting/proceedings.html", {
             'meeting': meeting,
-            'plenaries': plenaries, 'ietf': ietf, 'training': training, 'irtf': irtf, 'iab': iab,
+            'plenaries': plenaries,
+            'training': training,
+            'irtf': irtf,
+            'iab': iab,
             'ietf_areas': ietf_areas,
             'cut_off_date': cut_off_date,
             'cor_cut_off_date': cor_cut_off_date,


Test patch:
diff --git a/ietf/meeting/tests_views.py b/ietf/meeting/tests_views.py
--- a/ietf/meeting/tests_views.py
+++ b/ietf/meeting/tests_views.py
@@ -7310,7 +7310,12 @@ def _assertProceedingsMaterialsDisplayed(self, response, meeting):
         )
 
     def test_proceedings(self):
-        """Proceedings should be displayed correctly"""
+        """Proceedings should be displayed correctly
+
+        Currently only tests that the view responds with a 200 response code and checks the ProceedingsMaterials
+        at the top of the proceedings. Ought to actually test the display of the individual group/session
+        materials as well.
+        """
         meeting = make_meeting_test_data(meeting=MeetingFactory(type_id='ietf', number='100'))
         session = Session.objects.filter(meeting=meeting, group__acronym="mars").first()
         GroupEventFactory(group=session.group,type='status_update')
@@ -7364,6 +7369,65 @@ def test_proceedings(self):
         self._assertMeetingHostsDisplayed(r, meeting)
         self._assertProceedingsMaterialsDisplayed(r, meeting)
 
+    def test_named_session(self):
+        """Session with a name should appear separately in the proceedings"""
+        meeting = MeetingFactory(type_id='ietf', number='100')
+        group = GroupFactory()
+        plain_session = SessionFactory(meeting=meeting, group=group)
+        named_session = SessionFactory(meeting=meeting, group=group, name='I Got a Name')
+        for doc_type_id in ('agenda', 'minutes', 'bluesheets', 'recording', 'slides', 'draft'):
+            # Set up sessions materials that will have distinct URLs for each session.
+            # This depends on settings.MEETING_DOC_HREFS and may need updating if that changes.
+            SessionPresentationFactory(
+                session=plain_session,
+                document__type_id=doc_type_id,
+                document__uploaded_filename=f'upload-{doc_type_id}-plain',
+                document__external_url=f'external_url-{doc_type_id}-plain',
+            )
+            SessionPresentationFactory(
+                session=named_session,
+                document__type_id=doc_type_id,
+                document__uploaded_filename=f'upload-{doc_type_id}-named',
+                document__external_url=f'external_url-{doc_type_id}-named',
+            )
+
+        url = urlreverse('ietf.meeting.views.proceedings', kwargs={'num': meeting.number})
+        r = self.client.get(url)
+        self.assertEqual(r.status_code, 200)
+        q = PyQuery(r.content)
+
+        plain_label = q(f'div#{group.acronym}')
+        self.assertEqual(plain_label.text(), group.acronym)
+        plain_row = plain_label.closest('tr')
+        self.assertTrue(plain_row)
+
+        named_label = q(f'div#{slugify(named_session.name)}')
+        self.assertEqual(named_label.text(), named_session.name)
+        named_row = named_label.closest('tr')
+        self.assertTrue(named_row)
+
+        for material in (sp.document for sp in plain_session.sessionpresentation_set.all()):
+            if material.type_id == 'draft':
+                expected_url = urlreverse(
+                    'ietf.doc.views_doc.document_main',
+                    kwargs={'name': material.canonical_name()},
+                )
+            else:
+                expected_url = material.get_href(meeting)
+            self.assertTrue(plain_row.find(f'a[href="{expected_url}"]'))
+            self.assertFalse(named_row.find(f'a[href="{expected_url}"]'))
+
+        for material in (sp.document for sp in named_session.sessionpresentation_set.all()):
+            if material.type_id == 'draft':
+                expected_url = urlreverse(
+                    'ietf.doc.views_doc.document_main',
+                    kwargs={'name': material.canonical_name()},
+                )
+            else:
+                expected_url = material.get_href(meeting)
+            self.assertFalse(plain_row.find(f'a[href="{expected_url}"]'))
+            self.assertTrue(named_row.find(f'a[href="{expected_url}"]'))
+
     def test_proceedings_no_agenda(self):
         # Meeting number must be larger than the last special-cased proceedings (currently 96)
         meeting = MeetingFactory(type_id='ietf',populate_schedule=False,date=date_today(), number='100')