internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: microsoft/PubSec-Info-Assistant

Issue:
Large files time out in Unstructure.io processing 
**Describe the bug**
Large files (unknown threshold) seem to take too long to chunk with Unstructured.io. Tested file was 879 page PDF, saved as Word file (docx). At the 1 hour mark, the system seems to assume the chunking failed and it restarts the chunking process. 

**To Reproduce**
Steps to reproduce the behavior:
1. Upload large .docx file (~900 pages?)
2. Note that it remains in queued state forever
3. Look at logs and note the "01:00:00 exceeded by function" message  

**Expected behavior**
Processing happens, in reasonable time. 

**Screenshots**
![Screenshot 2023-12-15 100904](https://github.com/microsoft/PubSec-Info-Assistant/assets/93607917/e38cc9fd-6070-43c0-a8d0-9b3cdf1857e8)
![Screenshot 2023-12-15 101234](https://github.com/microsoft/PubSec-Info-Assistant/assets/93607917/60f3f6d1-82d7-4796-88d6-0f0509748aed)
![Screenshot 2023-12-15 101259](https://github.com/microsoft/PubSec-Info-Assistant/assets/93607917/571dd772-e44c-49db-918b-e394e1374384)

**Version details**
 - GitHub branch: Ryon's autoscaling PR, and main (Delta)

**Additional context**
Ryon reproduced on Main. 



Discussion:

Resolving patch:
diff --git a/app/enrichment/app.py b/app/enrichment/app.py
--- a/app/enrichment/app.py
+++ b/app/enrichment/app.py
@@ -4,6 +4,8 @@
 import json
 import logging
 import os
+import threading
+import time
 import re
 from datetime import datetime
 from typing import List
@@ -276,9 +278,17 @@ def get_tags_and_upload_to_cosmos(blob_service_client, blob_path):
     tagsHelper.upsert_document(blob_path, tags_list)
     return tags_list
 
-        
 @app.on_event("startup") 
-@repeat_every(seconds=5, logger=log, raise_exceptions=True)
+def startup_event():
+    poll_thread = threading.Thread(target=poll_queue_thread)
+    poll_thread.daemon = True
+    poll_thread.start()
+
+def poll_queue_thread():
+    while True:
+        poll_queue()
+        time.sleep(5)     
+        
 def poll_queue() -> None:
     """Polls the queue for messages and embeds them"""
     
@@ -294,6 +304,11 @@ def poll_queue() -> None:
     response = queue_client.receive_messages(max_messages=int(ENV["DEQUEUE_MESSAGE_BATCH_SIZE"]))
     messages = [x for x in response]
 
+    if not messages:
+        log.debug("No messages to process. Waiting for a couple of minutes...")
+        time.sleep(120)  # Sleep for 2 minutes
+        return
+
     target_embeddings_model = re.sub(r'[^a-zA-Z0-9_\-.]', '_', ENV["TARGET_EMBEDDINGS_MODEL"])
 
     # Remove from queue to prevent duplicate processing from any additional instances


Test patch:
Repository: microsoft/PubSec-Info-Assistant

Issue:
Chunks number not correctly reported in statuscontainer
**Describe the bug**
Chunks number for non-PDF files not correctly reported in statuscontainer

**To Reproduce**
Steps to reproduce the behavior:
1. Upload a non-PDF file
2. Wait for the chunking process to complete
3. Go to CosmosDB -> statuscontainer -> item relative to the uploaded non-PDF file
4. Check the "status" field which will display something like: 
"FileLayoutParsingOther - chunking complete. <built-in method count of list object at 0x7713c4227440> chunks created" 

**Expected behavior**
It should show a number instead of "<built-in method count of list object at 0x7713c4227440>"

**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser: Edge
 - Version: 120.0.2210.121 (Official build) (64-bit)

**Alpha version details**
 - GitHub branch: main

**Additional context**
Minor problem
probably line 166, functions/FileLayoutParsingOther/__init__.py



Discussion:

Resolving patch:
diff --git a/functions/FileLayoutParsingOther/__init__.py b/functions/FileLayoutParsingOther/__init__.py
--- a/functions/FileLayoutParsingOther/__init__.py
+++ b/functions/FileLayoutParsingOther/__init__.py
@@ -163,7 +163,7 @@ def main(msg: func.QueueMessage) -> None:
         # chunks = chunk_by_title(elements, multipage_sections=True, new_after_n_chars=NEW_AFTER_N_CHARS, combine_under_n_chars=COMBINE_UNDER_N_CHARS)
         # chunks = chunk_by_title(elements, multipage_sections=True, new_after_n_chars=NEW_AFTER_N_CHARS, combine_under_n_chars=COMBINE_UNDER_N_CHARS, max_characters=MAX_CHARACTERS)   
         chunks = chunk_by_title(elements, multipage_sections=True, new_after_n_chars=NEW_AFTER_N_CHARS, combine_text_under_n_chars=COMBINE_UNDER_N_CHARS)
-        statusLog.upsert_document(blob_name, f'{function_name} - chunking complete. {str(chunks.count)} chunks created', StatusClassification.DEBUG)
+        statusLog.upsert_document(blob_name, f'{function_name} - chunking complete. {len(chunks)} chunks created', StatusClassification.DEBUG)
                 
         subtitle_name = ''
         section_name = ''


Test patch:
Repository: microsoft/PubSec-Info-Assistant

Issue:
Issues with documents filtering using tags
Hi, 

We uploaded some files with multiple tags as seen below (e.g., project name: Wegstunnels AWA, project code: MN004259,  tender outcome: Gewonnen)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/df2fb540-7306-4b5d-8169-2ee338c65664)

When using these tags on the app frontend before querying the model, we got diverse errors:
- When selecting only one of the three tags, the completion always states that no data source is found (see below the completion, in Dutch sorry)
 
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/2cea1d7f-4922-404f-977a-49273e034661)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/ed0187a8-0b7c-40ea-b09d-6339883f1f35)

- When selecting two or all three tags, we get instead an error:
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/8318b304-9054-4415-a264-b956d3479c8f)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/b182ea6f-5074-441f-a963-632a35f97489)

 We have looked into the code and are struggling to pinpoint the reason why it fails. Could we get some support on this? 

The Git commit of the source code from this repo is: https://github.com/microsoft/PubSec-Info-Assistant/commit/27df39089a0a203c41ae153dc2539315559d0eeb 

Thank you!



Discussion:

Resolving patch:
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -209,11 +209,10 @@ async def run(self, history: Sequence[dict[str, str]], overrides: dict[str, Any]
         else:
             search_filter = None
         if tags_filter != "" :
-            quoted_tags_filter = tags_filter.replace(",","','")
             if search_filter is not None:
-                search_filter = search_filter + f" and tags/any(t: search.in(t, '{quoted_tags_filter}', ','))"
+                search_filter = search_filter + f" and tags/any(t: search.in(t, '{tags_filter}', ','))"
             else:
-                search_filter = f"tags/any(t: search.in(t, '{quoted_tags_filter}', ','))"
+                search_filter = f"tags/any(t: search.in(t, '{tags_filter}', ','))"
 
         # Hybrid Search
         # r = self.search_client.search(generated_query, vector_queries =[vector], top=top)


Test patch:
Repository: microsoft/PubSec-Info-Assistant

Issue:
Issues with documents filtering using tags
Hi, 

We uploaded some files with multiple tags as seen below (e.g., project name: Wegstunnels AWA, project code: MN004259,  tender outcome: Gewonnen)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/df2fb540-7306-4b5d-8169-2ee338c65664)

When using these tags on the app frontend before querying the model, we got diverse errors:
- When selecting only one of the three tags, the completion always states that no data source is found (see below the completion, in Dutch sorry)
 
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/2cea1d7f-4922-404f-977a-49273e034661)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/ed0187a8-0b7c-40ea-b09d-6339883f1f35)

- When selecting two or all three tags, we get instead an error:
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/8318b304-9054-4415-a264-b956d3479c8f)
![image](https://github.com/microsoft/PubSec-Info-Assistant/assets/89200604/b182ea6f-5074-441f-a963-632a35f97489)

 We have looked into the code and are struggling to pinpoint the reason why it fails. Could we get some support on this? 

The Git commit of the source code from this repo is: https://github.com/microsoft/PubSec-Info-Assistant/commit/27df39089a0a203c41ae153dc2539315559d0eeb 

Thank you!



Discussion:
Hello - Found the issue here.
I have a solution in Pull Request https://github.com/microsoft/PubSec-Info-Assistant/pull/484
The issue being each tag was being quoted separately which was overloading the AZ Search function.
I made a change to normalize the parameter. 
That's great thank you, we'll try it out!
Hi, we've just deployed the latest version of vNext-Dev (commit 0909c4b). The "Invalid Expression" error when using multiple tag is indeed gone, thank you! However, with one or multiple tag(s), the issue that no document source is found remains. If we query the database without any tag selected, we get an answer with citations. However, when we ask the same question but this time with one/multiple tag(s) selected, we get the message that no data source could be found (see screenshot in Dutch in previous message).

Is it this working well for you or other users? 
I think I have a better understanding of that search API and have an idea that should make searching with multiple tags way more accurate. Will update here when I have more.

Resolving patch:
diff --git a/app/enrichment/app.py b/app/enrichment/app.py
--- a/app/enrichment/app.py
+++ b/app/enrichment/app.py
@@ -269,9 +269,9 @@ def get_tags_and_upload_to_cosmos(blob_service_client, blob_path):
     tags = blob_properties.metadata.get("tags")
     if tags is not None:
         if isinstance(tags, str):
-            tags_list = [unquote(tags)]
+            tags_list = [unquote(tag.strip()) for tag in tags.split(",")]
         else:
-            tags_list = [unquote(tag) for tag in tags.split(",")]
+            tags_list = [unquote(tag.strip()) for tag in tags]
     else:
         tags_list = []
     # Write the tags to cosmos db


Test patch:
Repository: ietf-tools/datatracker

Issue:
Redirects to stale link after renaming a schedule
`type_defect`   |    by jennifer@painless-security.com

___


After successfully changing the name of a schedule using the new meeting schedule editor, the browser is redirected to a URL containing the old name, causing a 404. The new name should be used for the redirect URL.

___
_Issue migrated from trac:3427 at 2022-03-04 09:16:10 +0000_


Discussion:
_@rjsparks@nostrum.com_ _changed status from `new` to `accepted`_

Resolving patch:
diff --git a/ietf/meeting/views.py b/ietf/meeting/views.py
--- a/ietf/meeting/views.py
+++ b/ietf/meeting/views.py
@@ -1344,12 +1344,13 @@ def edit_schedule_properties(request, num, owner, name):
         permission_denied(request, "You may not edit this schedule.")
 
     if request.method == 'POST':
-        form = SchedulePropertiesForm(meeting, instance=schedule, data=request.POST)
+        # use a new copy of the Schedule instance for the form so the template isn't fouled if validation fails
+        form = SchedulePropertiesForm(meeting, instance=Schedule.objects.get(pk=schedule.pk), data=request.POST)
         if form.is_valid():
-           form.save()
-           if request.GET.get('next'):
-               return HttpResponseRedirect(request.GET.get('next'))
-           return redirect('ietf.meeting.views.edit_meeting_schedule', num=num, owner=owner, name=name)
+            form.save()
+            if request.GET.get('next'):
+                return HttpResponseRedirect(request.GET.get('next'))
+            return redirect('ietf.meeting.views.edit_meeting_schedule', num=num, owner=owner, name=form.instance.name)
     else:
         form = SchedulePropertiesForm(meeting, instance=schedule)
 


Test patch:
diff --git a/ietf/meeting/tests_views.py b/ietf/meeting/tests_views.py
--- a/ietf/meeting/tests_views.py
+++ b/ietf/meeting/tests_views.py
@@ -851,7 +851,7 @@ def test_edit_schedule_properties(self):
 
         new_base = Schedule.objects.create(name="newbase", owner=schedule.owner, meeting=schedule.meeting)
         response = self.client.post(url, {
-                'name':schedule.name,
+                'name': 'some-other-name',
                 'visible':True,
                 'public':True,
                 'notes': "New Notes",
@@ -859,11 +859,19 @@ def test_edit_schedule_properties(self):
             }
         )
         self.assertNoFormPostErrors(response)
+        self.assertRedirects(
+            response,
+            urlreverse(
+                'ietf.meeting.views.edit_meeting_schedule',
+                kwargs={'num': schedule.meeting.number, 'owner': schedule.owner.email(), 'name': 'some-other-name'}
+            ),
+        )
         schedule.refresh_from_db()
         self.assertTrue(schedule.visible)
         self.assertTrue(schedule.public)
         self.assertEqual(schedule.notes, "New Notes")
         self.assertEqual(schedule.base_id, new_base.pk)
+        self.assertEqual(schedule.name, 'some-other-name')
 
     def test_agenda_by_type_ics(self):
         session=SessionFactory(meeting__type_id='ietf',type_id='lead')
Repository: ietf-tools/datatracker

Issue:
Poor choice of email used for identity when saving a copy of an agenda
`type_defect`   |    by rjsparks@nostrum.com

___


When I save a copy of an agenda in the drag-drop editor, it names it with an inactive email address:
```
/meeting/103/agenda/robert.sparks@tekelec.com/rjstest/edit
```
It should use the code that chooses the "best" address instead (which will return the address marked primary when there is one).

___
_Issue migrated from trac:2611 at 2022-03-04 06:54:05 +0000_


Discussion:
_@rjsparks@nostrum.com_ _changed status from `new` to `accepted`_

Resolving patch:
diff --git a/ietf/meeting/models.py b/ietf/meeting/models.py
--- a/ietf/meeting/models.py
+++ b/ietf/meeting/models.py
@@ -734,11 +734,7 @@ def base_url(self):
 #         return self.url_edit("")
 
     def owner_email(self):
-        email = self.owner.email_set.all().order_by('primary').first()
-        if email:
-            return email.address
-        else:
-            return "noemail"
+        return self.owner.email_address() or "noemail"
 
     @property
     def is_official(self):


Test patch:
Repository: ietf-tools/datatracker

Issue:
The non-wg list view contains things that do not belong.
The list contains things that do not belong.  For example, 'geopriv' is listed as a non-wg list, but it is a concluded wg.  Maybe this should be a separate issue.

_Originally posted by @russhousley in https://github.com/ietf-tools/datatracker/issues/3675#issuecomment-1075013354_


Discussion:
When the code builds the list of WG mail lists, the lists associated with BoFs and concluded WGs are not captured.  Clearly, the Concluded WGs should be in the list.  What about BoFs? I assume that they are non-WG mail lists, but of course, some of them will become WG mail lists.
Concluded WG lists **are** WG lists though. Maybe we need another category? (Agree on BOF lists.)

Resolving patch:
diff --git a/ietf/mailinglists/views.py b/ietf/mailinglists/views.py
--- a/ietf/mailinglists/views.py
+++ b/ietf/mailinglists/views.py
@@ -1,4 +1,4 @@
-# Copyright The IETF Trust 2007, All Rights Reserved
+# Copyright The IETF Trust 2007-2022, All Rights Reserved
 
 import re
 
@@ -15,7 +15,7 @@ def groups(request):
     return render(request, "mailinglists/group_archives.html", { "groups": groups } )
 
 def nonwg(request):
-    groups = Group.objects.filter(type__features__acts_like_wg=True).exclude(state__in=['bof', 'conclude']).order_by("acronym")
+    groups = Group.objects.filter(type__features__acts_like_wg=True).exclude(state__in=['bof']).order_by("acronym")
 
     #urls = [ g.list_archive for g in groups if '.ietf.org' in g.list_archive ]
 


Test patch:
diff --git a/ietf/mailinglists/tests.py b/ietf/mailinglists/tests.py
--- a/ietf/mailinglists/tests.py
+++ b/ietf/mailinglists/tests.py
@@ -32,7 +32,13 @@ def test_groups(self):
 
 
     def test_nonwg(self):
+        groups = list()
+        groups.append(GroupFactory(type_id='wg', acronym='mars', list_archive='https://ietf.org/mars'))
+        groups.append(GroupFactory(type_id='wg', acronym='ames', state_id='conclude', list_archive='https://ietf.org/ames'))
+        groups.append(GroupFactory(type_id='wg', acronym='newstuff', state_id='bof', list_archive='https://ietf.org/newstuff'))
+        groups.append(GroupFactory(type_id='rg', acronym='research', list_archive='https://irtf.org/research'))
         lists = ListFactory.create_batch(7)
+
         url = urlreverse("ietf.mailinglists.views.nonwg")
 
         r = self.client.get(url)
@@ -43,3 +49,6 @@ def test_nonwg(self):
             else:
                 self.assertNotContains(r, l.name, html=True)
                 self.assertNotContains(r, l.description, html=True)
+
+        for g in groups:
+            self.assertNotContains(r, g.acronym, html=True)
Repository: ietf-tools/datatracker

Issue:
Reevaluate which groups for which delegation is available
Delegation of chair responsibilities is possible for wg/rg groups, but not for rags. Likely not for ags, programs, and other groups that _should_ be able to delegate authority to manage the group.

A comprehensive review is needed, but delegation should be added to rags as soon as possible.


Discussion:

Resolving patch:
diff --git a/ietf/group/migrations/0054_enable_delegation.py b/ietf/group/migrations/0054_enable_delegation.py
new file mode 100644
--- /dev/null
+++ b/ietf/group/migrations/0054_enable_delegation.py
@@ -0,0 +1,29 @@
+# Copyright The IETF Trust 2022 All Rights Reserved
+
+from django.db import migrations
+
+def forward(apps, schema_editor):
+    GroupFeatures = apps.get_model('group','GroupFeatures')
+    for type_id in ('dir', 'iabasg', 'program', 'review', 'team'):
+        f = GroupFeatures.objects.get(type_id=type_id)
+        if 'delegate' not in f.groupman_roles:
+            f.groupman_roles.append('delegate')
+            f.save()
+    for type_id in ('adhoc', 'ag', 'iesg', 'irtf', 'ise', 'rag', 'dir', 'iabasg', 'program', 'review'):
+        f = GroupFeatures.objects.get(type_id=type_id)
+        if 'delegate' not in f.default_used_roles:
+            f.default_used_roles.append('delegate')
+            f.save()
+
+def reverse (apps, schema_editor):
+    pass
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('group', '0053_populate_groupfeatures_session_purposes'),
+    ]
+
+    operations = [
+        migrations.RunPython(forward,reverse),
+    ]


Test patch:
Repository: ietf-tools/datatracker

Issue:
`Adopt` button is misleading
The button that was labelled `Manage document adoption in group` is now labelled `Adopt`, which is causing chairs to wonder how to start the adoption process without actually adopting a document yet.

A better short label for the button is needed.


Discussion:
"Manage adoption"?

Resolving patch:
diff --git a/ietf/doc/views_doc.py b/ietf/doc/views_doc.py
--- a/ietf/doc/views_doc.py
+++ b/ietf/doc/views_doc.py
@@ -370,7 +370,7 @@ def document_main(request, name, rev=None):
                 # already adopted in one group
                 button_text = "Switch adoption"
             else:
-                button_text = "Adopt"
+                button_text = "Manage adoption"
             actions.append((button_text, urlreverse('ietf.doc.views_draft.adopt_draft', kwargs=dict(name=doc.name))))
 
         if can_unadopt_draft(request.user, doc) and not doc.get_state_slug() in ["rfc"] and not snapshot:
@@ -1990,4 +1990,4 @@ def rfcdiff_latest_json(request, name, rev=None):
                 response['previous'] = f'rfc{match.group(2)}'
     if not response:
         raise Http404
-    return HttpResponse(json.dumps(response), content_type='application/json')
\ No newline at end of file
+    return HttpResponse(json.dumps(response), content_type='application/json')


Test patch:
Repository: ietf-tools/datatracker

Issue:
Milestones with multiple associated drafts do not render those drafts
### What happened?

I created a new milestone, "Qlog documents to IESG" in the QUIC WG

The milestone has 3 associated drafts. But none of them render in the datatracker column.

![image](https://user-images.githubusercontent.com/6571445/167402519-1b2997c2-c523-4ab7-aa55-2f365f1e89e9.png)



### What browser(s) are you seeing the problem on?

Firefox

### Code of Conduct

- [X] I agree to follow the IETF's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ietf/utils/fields.py b/ietf/utils/fields.py
--- a/ietf/utils/fields.py
+++ b/ietf/utils/fields.py
@@ -264,10 +264,7 @@ def prepare_value(self, value):
                 value = value[0]
             else:
                 if not isinstance(value[0], self.model):
-                    qs = self.get_model_instances(value[0])
-                    for val in value[1:]:
-                        qs = qs.union(self.get_model_instances(val))
-                    value = qs
+                    value = self.get_model_instances(value)
         if isinstance(value, int):
             value = str(value)
         if isinstance(value, str):
@@ -343,4 +340,4 @@ def update_dimension_fields(self, *args, **kwargs):
         try:
             super().update_dimension_fields(*args, **kwargs)
         except FileNotFoundError:
-            pass  # don't do anything if the file has gone missing
\ No newline at end of file
+            pass  # don't do anything if the file has gone missing


Test patch: