Repository: laws-africa/peachjam
Pull request #2471: Similar documents for folders
Author: login: actlikewill
Created: 2025-05-02T14:08:18Z | Merged: 2025-05-07T09:47:56Z

This adds similar document recommendations to saved folders.

https://www.loom.com/share/af397a0d2da54c6faf53fcdb62fbe2d2?sid=ee2d8215-47c4-4d4e-83fa-76894d3f20b1


![image](https://github.com/user-attachments/assets/a0bc714a-d4b4-4d92-a392-ef76b2fc15f3)


closes https://github.com/laws-africa/peachjam/issues/2408


Diff:
diff --git a/peachjam/static/stylesheets/_global.scss b/peachjam/static/stylesheets/_global.scss
index eac346e6e..2357ee8d3 100644
--- a/peachjam/static/stylesheets/_global.scss
+++ b/peachjam/static/stylesheets/_global.scss
@@ -151,6 +151,10 @@ footer {
   font-size: 22px;
 }
 
+.text-ai {
+  color: $text-ai !important;
+}
+
 @each $state, $value in $theme-colors {
   .alert-#{$state} {
     $alert-color: shift-color($value, $alert-color-scale);
diff --git a/peachjam/static/stylesheets/_variables.scss b/peachjam/static/stylesheets/_variables.scss
index dd7510b08..192b076b2 100644
--- a/peachjam/static/stylesheets/_variables.scss
+++ b/peachjam/static/stylesheets/_variables.scss
@@ -19,6 +19,7 @@ $anntn-highlight-color: #a68bdc !default;
 $diff-green-color: #a6f3a6 !default;
 $diff-red-color: #f8cbcb !default;
 $visited-link: #1A77F2 !default;
+$text-ai : #6f42c1 !default;
 
 :root {
   --facebook-color: #{$facebook-color};
diff --git a/peachjam/static/stylesheets/components/_icons.scss b/peachjam/static/stylesheets/components/_icons.scss
index a0adb5a89..9c0f2609b 100644
--- a/peachjam/static/stylesheets/components/_icons.scss
+++ b/peachjam/static/stylesheets/components/_icons.scss
@@ -9,4 +9,6 @@
   &.pj-history::before { content: "\f292"; /* bi-file-text */ }
   &.pj-related::before { content: "\f548"; /* bi-signpost-split-fill */ }
   &.pj-taxonomies::before { content: "\f5b1"; /* bi-tags-fill */ }
+  &.pj-ai::before{ color: $text-ai; content: "\f589"; /* bi-stars */ }
+
 }
diff --git a/peachjam/templates/peachjam/saved_document/_folders_list.html b/peachjam/templates/peachjam/saved_document/_folders_list.html
index e9c3b819f..e11d01975 100644
--- a/peachjam/templates/peachjam/saved_document/_folders_list.html
+++ b/peachjam/templates/peachjam/saved_document/_folders_list.html
@@ -10,52 +10,58 @@ <h2>{% trans 'Saved documents' %}</h2>
     </button>
   </div>
   {% if folders or ungrouped_saved_documents %}
-    {% for folder in folders %}
-      <div class="card mb-5">
-        <h5 class="card-header">
-          <i class="bi bi-folder"></i> {{ folder.name }}
-        </h5>
-        <div class="card-body">
-          <div>
-            {% if folder.saved_documents.all %}
-              {% for saved_document in folder.saved_documents.all %}
-                {% include "peachjam/saved_document/_folder_list_item.html" %}
-              {% endfor %}
-            {% else %}
-              <p>{% trans 'No saved documents.' %}</p>
+    {% block folder-list %}
+      {% for folder in folders %}
+        <div class="card mb-5">
+          <div class="card-header">
+            {% block folder-list-card-header %}
+              <h5 class="mb-0">
+                <i class="bi bi-folder"></i> {{ folder.name }}
+              </h5>
+            {% endblock %}
+          </div>
+          <div class="card-body">
+            {% block folder-list-card-body %}
+              {% if folder.saved_documents.all %}
+                {% for saved_document in folder.saved_documents.all %}
+                  {% include "peachjam/saved_document/_folder_list_item.html" %}
+                {% endfor %}
+              {% else %}
+                <p>{% trans 'No saved documents.' %}</p>
+              {% endif %}
+            {% endblock %}
+          </div>
+          <div class="card-footer d-flex">
+            {% if perms.peachjam.download_folder %}
+              <a href="{% url 'folder_download' folder.id %}"
+                 target="_blank"
+                 class="btn btn-primary me-2">
+                {% trans 'Download to Excel' %}
+              </a>
             {% endif %}
+            <input id="folder_{{ forloop.counter }}"
+                   type="hidden"
+                   name="name"
+                   value="{{ folder.name }}"/>
+            <button class="btn btn-outline-danger me-2 ms-auto"
+                    type="button"
+                    hx-post="{% url 'folder_delete' folder.id %}"
+                    hx-target="#folders-list"
+                    hx-include="#folder_{{ forloop.counter }}"
+                    hx-confirm="{% trans "Are you sure you want to delete this folder? All saved documents in this folder will also be deleted." %}">
+              {% trans 'Delete' %}
+            </button>
+            <button class="btn btn-secondary"
+                    type="button"
+                    hx-post="{% url 'folder_update' folder.id %}"
+                    hx-target="#folders-list"
+                    hx-prompt="Edit folder name">
+              {% trans 'Rename' %}
+            </button>
           </div>
         </div>
-        <div class="card-footer d-flex">
-          {% if perms.peachjam.download_folder %}
-            <a href="{% url 'folder_download' folder.id %}"
-               target="_blank"
-               class="btn btn-primary me-2">
-              {% trans 'Download to Excel' %}
-            </a>
-          {% endif %}
-          <input id="folder_{{ forloop.counter }}"
-                 type="hidden"
-                 name="name"
-                 value="{{ folder.name }}"/>
-          <button class="btn btn-outline-danger me-2 ms-auto"
-                  type="button"
-                  hx-post="{% url 'folder_delete' folder.id %}"
-                  hx-target="#folders-list"
-                  hx-include="#folder_{{ forloop.counter }}"
-                  hx-confirm="{% trans "Are you sure you want to delete this folder? All saved documents in this folder will also be deleted." %}">
-            {% trans 'Delete' %}
-          </button>
-          <button class="btn btn-secondary"
-                  type="button"
-                  hx-post="{% url 'folder_update' folder.id %}"
-                  hx-target="#folders-list"
-                  hx-prompt="Edit folder name">
-            {% trans 'Rename' %}
-          </button>
-        </div>
-      </div>
-    {% endfor %}
+      {% endfor %}
+    {% endblock %}
     {% if ungrouped_saved_documents %}
       <div class="card">
         <h5 class="card-header">{% trans 'Ungrouped documents' %}</h5>
diff --git a/peachjam_ml/models.py b/peachjam_ml/models.py
index 0e531d81c..389fe21a3 100644
--- a/peachjam_ml/models.py
+++ b/peachjam_ml/models.py
@@ -5,10 +5,11 @@
 from django.conf import settings
 from django.contrib.postgres.fields import ArrayField
 from django.db import models
-from django.db.models import Avg
+from django.db.models import Avg, F
 from django.forms import model_to_dict
-from pgvector.django import HnswIndex, VectorField
+from pgvector.django import HnswIndex, MaxInnerProduct, VectorField
 
+from peachjam.models import Work
 from peachjam_ml.embeddings import TEXT_INJECTION_SEPARATOR, get_text_embedding_batch
 from peachjam_search.tasks import search_model_saved
 
@@ -147,6 +148,55 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
+            avg = normalize_vector(avg)
+
+        return avg
+
+    @classmethod
+    def get_similar_documents(cls, doc_ids, threshold=0.8, n_similar=10):
+        weight_similarity = 0.9
+        weight_authority = 0.1
+        top_k = 100
+        avg_embedding = cls.get_average_embedding(doc_ids)
+
+        similar_docs = (
+            DocumentEmbedding.objects.exclude(
+                document__work__in=Work.objects.filter(documents__in=doc_ids)
+            )
+            .exclude(text_embedding__isnull=True)
+            .annotate(
+                similarity=MaxInnerProduct("text_embedding", avg_embedding) * -1,
+                title=F("document__title"),
+                expression_frbr_uri=F("document__expression_frbr_uri"),
+                authority_score=F("document__work__authority_score"),
+            )
+            .filter(similarity__gt=threshold)
+            .values("title", "expression_frbr_uri", "similarity", "authority_score")
+            .order_by("-similarity")
+        )[:top_k]
+
+        # re-rank based on a weighted average of similarity and authority score, and keep the top 10
+        similar_docs = sorted(
+            similar_docs,
+            key=lambda x: (
+                x["similarity"] * weight_similarity
+                + x["authority_score"] * weight_authority
+            ),
+            reverse=True,
+        )[:n_similar]
+
+        return similar_docs
+
 
 class ContentChunk(models.Model):
     """A chunk of content that is indexed for semantic search. This has slightly different semantics for different
diff --git a/peachjam_ml/templates/peachjam/_similar_documents.html b/peachjam_ml/templates/peachjam/_similar_documents.html
deleted file mode 100644
index 193815437..000000000
--- a/peachjam_ml/templates/peachjam/_similar_documents.html
+++ /dev/null
@@ -1,11 +0,0 @@
-{% load i18n %}
-<h4>{% trans 'More documents like this one' %}</h4>
-<ol>
-  {% for doc in similar_documents %}
-    <li class="pb-2">
-      <a href="{{ doc.expression_frbr_uri }}">{{ doc.title }}
-        {% if request.user.is_superuser %}({{ doc.similarity }}){% endif %}
-      </a>
-    </li>
-  {% endfor %}
-</ol>
diff --git a/peachjam_ml/templates/peachjam/_similar_documents_doc_detail.html b/peachjam_ml/templates/peachjam/_similar_documents_doc_detail.html
new file mode 100644
index 000000000..086cd1f88
--- /dev/null
+++ b/peachjam_ml/templates/peachjam/_similar_documents_doc_detail.html
@@ -0,0 +1,18 @@
+{% load i18n %}
+<h4>{% trans 'More documents like this one' %}</h4>
+<p class="text-ai">
+  <i class="bi-pj pj-ai"></i>{% trans 'Suggestions generated by AI.' %}
+</p>
+{% if similar_documents %}
+  <ol>
+    {% for doc in similar_documents %}
+      <li class="pb-2">
+        <a href="{{ doc.expression_frbr_uri }}">{{ doc.title }}
+          {% if request.user.is_superuser %}({{ doc.similarity }}){% endif %}
+        </a>
+      </li>
+    {% endfor %}
+  </ol>
+{% else %}
+  <p>{% trans 'No documents found.' %}</p>
+{% endif %}
diff --git a/peachjam_ml/templates/peachjam/_similar_documents_folder.html b/peachjam_ml/templates/peachjam/_similar_documents_folder.html
new file mode 100644
index 000000000..9329d4fd4
--- /dev/null
+++ b/peachjam_ml/templates/peachjam/_similar_documents_folder.html
@@ -0,0 +1,17 @@
+{% load i18n %}
+<p class="mb-4 text-ai">
+  <i class="bi-pj pj-ai"></i>{% trans 'Suggestions generated by AI.' %}
+</p>
+{% if similar_documents %}
+  <ol>
+    {% for doc in similar_documents %}
+      <li class="pb-2">
+        <a href="{{ doc.expression_frbr_uri }}">{{ doc.title }}
+          {% if request.user.is_superuser %}({{ doc.similarity }}){% endif %}
+        </a>
+      </li>
+    {% endfor %}
+  </ol>
+{% else %}
+  {% trans 'No documents found.' %}
+{% endif %}
diff --git a/peachjam_ml/templates/peachjam/saved_document/_folders_list.html b/peachjam_ml/templates/peachjam/saved_document/_folders_list.html
new file mode 100644
index 000000000..85861c853
--- /dev/null
+++ b/peachjam_ml/templates/peachjam/saved_document/_folders_list.html
@@ -0,0 +1,56 @@
+{% extends "peachjam/saved_document/_folders_list.html" %}
+{% load i18n %}
+{% block folder-list-card-header %}
+  {{ block.super }}
+  {% if folder.saved_documents.exists %}
+    <ul class="nav nav-tabs card-header-tabs">
+      <li class="nav-item">
+        <button class="nav-link active"
+                id="saved-doc-tab-{{ folder.pk }}"
+                data-bs-toggle="tab"
+                data-bs-target="#saved-docs-{{ folder.pk }}"
+                type="button"
+                role="tab"
+                aria-controls="saved-docs-{{ folder.pk }}"
+                aria-selected="true">
+          {% trans 'Saved documents' %}
+        </button>
+      </li>
+      {% if perms.peachjam_ml.view_documentembedding %}
+        <li class="nav-item">
+          <button class="nav-link"
+                  id="similar-docs-tab-{{ folder.pk }}"
+                  data-bs-toggle="tab"
+                  data-bs-target="#similar-docs-{{ folder.pk }}"
+                  hx-target="#similar-docs-list-{{ folder.pk }}"
+                  hx-get="{% url 'folder_similar_docs' folder.pk %}"
+                  type="button"
+                  role="tab"
+                  aria-controls="similar-docs-{{ folder.pk }}"
+                  aria-selected="false">
+            {% trans 'Suggested for this folder' %}
+            <i class="bi-pj pj-ai"></i>
+          </button>
+        </li>
+      {% endif %}
+    </ul>
+  {% endif %}
+{% endblock %}
+{% block folder-list-card-body %}
+  <div class="tab-content">
+    <div class="tab-pane fade show active"
+         id="saved-docs-{{ folder.pk }}"
+         role="tabpanel"
+         aria-labelledby="saved-doc-tab-{{ folder.pk }}">
+      {{ block.super }}
+    </div>
+    {% if perms.peachjam_ml.view_documentembedding %}
+      <div class="tab-pane fade"
+           id="similar-docs-{{ folder.pk }}"
+           role="tabpanel"
+           aria-labelledby="similar-docs-tab-{{ folder.pk }}">
+        <div id="similar-docs-list-{{ folder.pk }}"></div>
+      </div>
+    {% endif %}
+  </div>
+{% endblock %}
diff --git a/peachjam_ml/urls.py b/peachjam_ml/urls.py
index dc12ec21b..7d858b7ee 100644
--- a/peachjam_ml/urls.py
+++ b/peachjam_ml/urls.py
@@ -1,14 +1,19 @@
-from django.urls import re_path
+from django.urls import path, re_path
 from django.views.decorators.cache import cache_page
 
 from peachjam.urls import CACHE_DURATION
 
-from .views import SimilarDocumentsView
+from .views import SimilarDocumentsDocumentDetailView, SimilarDocumentsFolderView
 
 urlpatterns = [
     re_path(
         r"^(?P<frbr_uri>akn/.*)/similar-documents$",
-        cache_page(CACHE_DURATION)(SimilarDocumentsView.as_view()),
+        cache_page(CACHE_DURATION)(SimilarDocumentsDocumentDetailView.as_view()),
         name="document_similar_docs",
     ),
+    path(
+        "folder/<int:pk>/similar-documents",
+        SimilarDocumentsFolderView.as_view(),
+        name="folder_similar_docs",
+    ),
 ]
diff --git a/peachjam_ml/views.py b/peachjam_ml/views.py
index d23239667..b0f293fb5 100644
--- a/peachjam_ml/views.py
+++ b/peachjam_ml/views.py
@@ -1,60 +1,35 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
 @method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class SimilarDocumentsDocumentDetailView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
-    template_name = "peachjam/_similar_documents.html"
+    template_name = "peachjam/_similar_documents_doc_detail.html"
     slug_url_kwarg = "frbr_uri"
     slug_field = "expression_frbr_uri"
     model = CoreDocument
-    similarity_threshold = 0.0
-    weight_similarity = 0.7
-    weight_authority = 0.3
-    # choose the best from this set, after re-ranking
-    top_k = 100
-    n_similar = 10
 
-    def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
-        )[: self.top_k]
+    def get_context_data(self, **kwargs):
+        context = super().get_context_data(**kwargs)
+        context["similar_documents"] = DocumentEmbedding.get_similar_documents(
+            [self.object.pk]
+        )
+        return context
 
-        # re-rank based on a weighted average of similarity and authority score, and keep the top 10
-        similar_docs = sorted(
-            similar_docs,
-            key=lambda x: (
-                x["similarity"] * self.weight_similarity
-                + x["authority_score"] * self.weight_authority
-            ),
-            reverse=True,
-        )[: self.n_similar]
 
-        return similar_docs
+class SimilarDocumentsFolderView(PermissionRequiredMixin, DetailView):
+    permission_required = "peachjam_ml.view_documentembedding"
+    template_name = "peachjam/_similar_documents_folder.html"
+    model = Folder
 
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)
-        context["similar_documents"] = self.get_similar_documents()
+        doc_ids = self.object.saved_documents.values_list("document_id", flat=True)
+        context["similar_documents"] = DocumentEmbedding.get_similar_documents(doc_ids)
         return context


Reviews:
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T08:54:51Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T08:55:44Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T08:56:10Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:17:19Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:20:49Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:22:19Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:24:40Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:26:13Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:29:09Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:32:17Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:35:05Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:39:16Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:39:22Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:41:00Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:49:00Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:52:39Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:53:06Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:54:42Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:55:07Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T01:56:07Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:27:41Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:33:12Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:36:51Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:37:57Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:39:06Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T07:07:10Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T07:07:41Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T14:58:13Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:14:02Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:15:25Z
user: login: longhotsummer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:15:32Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T07:25:11Z
user: login: actlikewill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T07:25:25Z
user: login: longhotsummer
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T08:36:10Z

Review comments:
diff_hunk: @@ -1,5 +1,5 @@
 {% load i18n %}
-<h4>{% trans 'More documents like this one' %}</h4>
+<h4>{% trans 'Recommended documents' %}</h4>
path: peachjam_ml/templates/peachjam/_similar_documents.html
user: login: actlikewill
body: we needed a more general title for these similar documents since they now appear in multiple places. I went with "Recommended documents". If this term is going to be used in other contexts we can change it
created_at: 2025-05-05T08:54:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-doc-tab-{{ folder.pk }}"
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: longhotsummer
body: this and the data-bs-target should be the same element AFAIK
created_at: 2025-05-05T08:55:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-doc-tab-{{ folder.pk }}"
+                      aria-selected="true">
+                Saved documents
+              </button>
+            </li>
+            <li class="nav-item">
+              <button class="nav-link"
+                      id="similar-docs-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#similar-docs-{{ folder.pk }}"
+                      hx-target="#similar-docs-list-{{ folder.pk }}"
+                      hx-get="{% url 'similar_documents' %}?doc_ids={% for pk in folder.get_doc_ids %}{{ pk }},{% endfor %}"
+                      type="button"
+                      role="tab"
+                      aria-controls="home"
+                      aria-selected="true">
+                Similar to these
+              </button>
+            </li>
+          </ul>
+        </div>
         <div class="card-body">
-          <div>
-            {% if folder.saved_documents.all %}
-              {% for saved_document in folder.saved_documents.all %}
-                {% include "peachjam/saved_document/_folder_list_item.html" %}
-              {% endfor %}
-            {% else %}
-              <p>{% trans 'No saved documents.' %}</p>
-            {% endif %}
+          <div class="tab-content">
+            <div class="tab-pane fade show active"
+                 id="saved-docs-{{ folder.pk }}"
+                 role="tabpanel"
+                 aria-labelledby="saved-doc-tab">
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: longhotsummer
body: needs to correspond to the ID of the tab
created_at: 2025-05-05T08:56:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,5 +1,5 @@
 {% load i18n %}
-<h4>{% trans 'More documents like this one' %}</h4>
+<h4>{% trans 'Recommended documents' %}</h4>
path: peachjam_ml/templates/peachjam/_similar_documents.html
user: login: longhotsummer
body: Another option is to move the heading out of this template so it can be customised where there's enough context to know how to phrase it 
created_at: 2025-05-05T18:17:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,45 +1,30 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
-@method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class BaseSimilarDocumentsView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
     template_name = "peachjam/_similar_documents.html"
-    slug_url_kwarg = "frbr_uri"
-    slug_field = "expression_frbr_uri"
-    model = CoreDocument
-    similarity_threshold = 0.0
+    similarity_threshold = 0.6
     weight_similarity = 0.7
     weight_authority = 0.3
     # choose the best from this set, after re-ranking
     top_k = 100
     n_similar = 10
 
+    def get_doc_ids(self):
+        raise NotImplementedError("Subclasses must implement get_doc_ids()")
+
     def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
+        doc_ids = self.get_doc_ids()
+        similar_docs = DocumentEmbedding.get_similar_documents(
+            doc_ids,
+            threshold=self.similarity_threshold,
         )[: self.top_k]
 
         # re-rank based on a weighted average of similarity and authority score, and keep the top 10
path: peachjam_ml/views.py
user: login: longhotsummer
body: I think you should push all of this into the document embedding class - that's the class that knows about ranking, scores, thresholds etc. The views don't need that and aren't customising it 
created_at: 2025-05-05T18:20:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,45 +1,30 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
-@method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class BaseSimilarDocumentsView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
     template_name = "peachjam/_similar_documents.html"
-    slug_url_kwarg = "frbr_uri"
-    slug_field = "expression_frbr_uri"
-    model = CoreDocument
-    similarity_threshold = 0.0
+    similarity_threshold = 0.6
     weight_similarity = 0.7
     weight_authority = 0.3
     # choose the best from this set, after re-ranking
     top_k = 100
     n_similar = 10
 
+    def get_doc_ids(self):
+        raise NotImplementedError("Subclasses must implement get_doc_ids()")
+
     def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
+        doc_ids = self.get_doc_ids()
+        similar_docs = DocumentEmbedding.get_similar_documents(
+            doc_ids,
+            threshold=self.similarity_threshold,
         )[: self.top_k]
 
         # re-rank based on a weighted average of similarity and authority score, and keep the top 10
path: peachjam_ml/views.py
user: login: longhotsummer
body: We're going to need it all later when sending recommendation emails, and it's domain logic not view logic 
created_at: 2025-05-05T18:22:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
path: peachjam_ml/models.py
user: login: longhotsummer
body: This is a premature optimisation. If there's only one, the db will handle it during Avg() etc 
created_at: 2025-05-05T18:24:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
path: peachjam_ml/models.py
user: login: longhotsummer
body: You don't need to guard against this. Everything below will work if there are no doc ids, since it has to work if there are no embeddings for the listed documents. 
created_at: 2025-05-05T18:26:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
+            avg = normalize_vector(avg)
+
+        return avg
+
+    @classmethod
+    def get_similar_documents(cls, doc_ids, threshold=0.0):
+        # always exclude the documents we are comparing against
+        exclude_ids = doc_ids
+        avg_embedding = cls.get_average_embedding(doc_ids)
+
+        similar_docs = (
+            DocumentEmbedding.objects.exclude(document__work__documents__in=exclude_ids)
path: peachjam_ml/models.py
user: login: longhotsummer
body: I don't think this works correctly? I think you need a sub query to find the works to exclude 
created_at: 2025-05-05T18:28:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-docs-{{ folder.pk }}"
+                      aria-selected="true">
+                {% trans 'Saved documents' %}
+              </button>
+            </li>
+            <li class="nav-item">
+              <button class="nav-link"
+                      id="similar-docs-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#similar-docs-{{ folder.pk }}"
+                      hx-target="#similar-docs-list-{{ folder.pk }}"
+                      hx-get="{% url 'folder_similar_docs' folder.pk %}"
+                      type="button"
+                      role="tab"
+                      aria-controls="similar-docs-{{ folder.pk }}"
+                      aria-selected="false">
+                {% trans 'Recommended documents' %}
+              </button>
+            </li>
+          </ul>
+        </div>
         <div class="card-body">
-          <div>
-            {% if folder.saved_documents.all %}
-              {% for saved_document in folder.saved_documents.all %}
-                {% include "peachjam/saved_document/_folder_list_item.html" %}
-              {% endfor %}
-            {% else %}
-              <p>{% trans 'No saved documents.' %}</p>
-            {% endif %}
+          <div class="tab-content">
+            <div class="tab-pane fade show active"
+                 id="saved-docs-{{ folder.pk }}"
+                 role="tabpanel"
+                 aria-labelledby="saved-doc-tab-{{ folder.pk }}">
+              {% if folder.saved_documents.all %}
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: longhotsummer
body: The .all goes to the db on this line and again on the next. Instead use a for loop and django templates '{% empty %}' tag at the end. 
created_at: 2025-05-05T18:31:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
path: peachjam_ml/models.py
user: login: longhotsummer
body: This is the same as:

If avg:
created_at: 2025-05-05T18:34:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-doc-tab-{{ folder.pk }}"
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: actlikewill
body: done
created_at: 2025-05-06T01:39:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-doc-tab-{{ folder.pk }}"
+                      aria-selected="true">
+                Saved documents
+              </button>
+            </li>
+            <li class="nav-item">
+              <button class="nav-link"
+                      id="similar-docs-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#similar-docs-{{ folder.pk }}"
+                      hx-target="#similar-docs-list-{{ folder.pk }}"
+                      hx-get="{% url 'similar_documents' %}?doc_ids={% for pk in folder.get_doc_ids %}{{ pk }},{% endfor %}"
+                      type="button"
+                      role="tab"
+                      aria-controls="home"
+                      aria-selected="true">
+                Similar to these
+              </button>
+            </li>
+          </ul>
+        </div>
         <div class="card-body">
-          <div>
-            {% if folder.saved_documents.all %}
-              {% for saved_document in folder.saved_documents.all %}
-                {% include "peachjam/saved_document/_folder_list_item.html" %}
-              {% endfor %}
-            {% else %}
-              <p>{% trans 'No saved documents.' %}</p>
-            {% endif %}
+          <div class="tab-content">
+            <div class="tab-pane fade show active"
+                 id="saved-docs-{{ folder.pk }}"
+                 role="tabpanel"
+                 aria-labelledby="saved-doc-tab">
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: actlikewill
body: done
created_at: 2025-05-06T01:39:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,45 +1,30 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
-@method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class BaseSimilarDocumentsView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
     template_name = "peachjam/_similar_documents.html"
-    slug_url_kwarg = "frbr_uri"
-    slug_field = "expression_frbr_uri"
-    model = CoreDocument
-    similarity_threshold = 0.0
+    similarity_threshold = 0.6
     weight_similarity = 0.7
     weight_authority = 0.3
     # choose the best from this set, after re-ranking
     top_k = 100
     n_similar = 10
 
+    def get_doc_ids(self):
+        raise NotImplementedError("Subclasses must implement get_doc_ids()")
+
     def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
+        doc_ids = self.get_doc_ids()
+        similar_docs = DocumentEmbedding.get_similar_documents(
+            doc_ids,
+            threshold=self.similarity_threshold,
         )[: self.top_k]
 
         # re-rank based on a weighted average of similarity and authority score, and keep the top 10
path: peachjam_ml/views.py
user: login: actlikewill
body: I've moved everything to the DocumentEmbedding class. I've put all the configurations in the method, with some defaults that can overriden in the method call
created_at: 2025-05-06T01:41:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,5 +1,5 @@
 {% load i18n %}
-<h4>{% trans 'More documents like this one' %}</h4>
+<h4>{% trans 'Recommended documents' %}</h4>
path: peachjam_ml/templates/peachjam/_similar_documents.html
user: login: actlikewill
body: since we are loading via htmx, we'd need to have separate templates so we can handle empty responses properly. Let me know if we need to do that. Using a common title is easier for now.
created_at: 2025-05-06T01:49:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
path: peachjam_ml/models.py
user: login: actlikewill
body: i think the more common case is the document detail view which will have only one embedding. we don't need to be calculating the average for one document each time we call that view.
created_at: 2025-05-06T01:52:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
path: peachjam_ml/models.py
user: login: actlikewill
body: ok, removed
created_at: 2025-05-06T01:53:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
+            avg = normalize_vector(avg)
+
+        return avg
+
+    @classmethod
+    def get_similar_documents(cls, doc_ids, threshold=0.0):
+        # always exclude the documents we are comparing against
+        exclude_ids = doc_ids
+        avg_embedding = cls.get_average_embedding(doc_ids)
+
+        similar_docs = (
+            DocumentEmbedding.objects.exclude(document__work__documents__in=exclude_ids)
path: peachjam_ml/models.py
user: login: actlikewill
body: this works as expected. it excludes all documents that we are using to calculate similarity 
created_at: 2025-05-06T01:54:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,56 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
+          <ul class="nav nav-tabs card-header-tabs">
+            <li class="nav-item">
+              <button class="nav-link active"
+                      id="saved-doc-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#saved-docs-{{ folder.pk }}"
+                      type="button"
+                      role="tab"
+                      aria-controls="saved-docs-{{ folder.pk }}"
+                      aria-selected="true">
+                {% trans 'Saved documents' %}
+              </button>
+            </li>
+            <li class="nav-item">
+              <button class="nav-link"
+                      id="similar-docs-tab-{{ folder.pk }}"
+                      data-bs-toggle="tab"
+                      data-bs-target="#similar-docs-{{ folder.pk }}"
+                      hx-target="#similar-docs-list-{{ folder.pk }}"
+                      hx-get="{% url 'folder_similar_docs' folder.pk %}"
+                      type="button"
+                      role="tab"
+                      aria-controls="similar-docs-{{ folder.pk }}"
+                      aria-selected="false">
+                {% trans 'Recommended documents' %}
+              </button>
+            </li>
+          </ul>
+        </div>
         <div class="card-body">
-          <div>
-            {% if folder.saved_documents.all %}
-              {% for saved_document in folder.saved_documents.all %}
-                {% include "peachjam/saved_document/_folder_list_item.html" %}
-              {% endfor %}
-            {% else %}
-              <p>{% trans 'No saved documents.' %}</p>
-            {% endif %}
+          <div class="tab-content">
+            <div class="tab-pane fade show active"
+                 id="saved-docs-{{ folder.pk }}"
+                 role="tabpanel"
+                 aria-labelledby="saved-doc-tab-{{ folder.pk }}">
+              {% if folder.saved_documents.all %}
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: actlikewill
body: neat, i've never used the {% empty %} tag before
created_at: 2025-05-06T01:55:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
path: peachjam_ml/models.py
user: login: actlikewill
body: the value here is a numpy array, not a list. this is how we can check for truthiness for it
created_at: 2025-05-06T01:56:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
path: peachjam_ml/models.py
user: login: longhotsummer
body: ah of course
created_at: 2025-05-06T06:27:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,60 +1,42 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
-@method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class BaseSimilarDocumentsView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
     template_name = "peachjam/_similar_documents.html"
-    slug_url_kwarg = "frbr_uri"
-    slug_field = "expression_frbr_uri"
-    model = CoreDocument
-    similarity_threshold = 0.0
-    weight_similarity = 0.7
-    weight_authority = 0.3
-    # choose the best from this set, after re-ranking
-    top_k = 100
-    n_similar = 10
 
-    def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
-        )[: self.top_k]
-
-        # re-rank based on a weighted average of similarity and authority score, and keep the top 10
-        similar_docs = sorted(
-            similar_docs,
-            key=lambda x: (
-                x["similarity"] * self.weight_similarity
-                + x["authority_score"] * self.weight_authority
-            ),
-            reverse=True,
-        )[: self.n_similar]
+    def get_doc_ids(self):
+        raise NotImplementedError("Subclasses must implement get_doc_ids()")
 
+    def get_similar_documents(self):
+        doc_ids = self.get_doc_ids()
+        similar_docs = DocumentEmbedding.get_similar_documents(doc_ids)
         return similar_docs
 
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)
         context["similar_documents"] = self.get_similar_documents()
         return context
+
+
+@method_decorator(add_slash_to_frbr_uri(), name="setup")
+class SimilarDocumentsView(BaseSimilarDocumentsView):
path: peachjam_ml/views.py
user: login: longhotsummer
body: This mixin doesn't really help any more -- it's more work to get the classes to use it than just to have a single line in their `get_context_data()`:

```
def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context["similar_documents"] = DocumentEmbedding.get_similar_documents([self.object.id])
        return context
```
created_at: 2025-05-06T06:33:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,15 +15,54 @@ <h2>{% trans 'Saved documents' %}</h2>
         <h5 class="card-header">
           <i class="bi bi-folder"></i> {{ folder.name }}
         </h5>
+        <div class="card-header">
path: peachjam/templates/peachjam/saved_document/_folders_list.html
user: login: longhotsummer
body: You've got a duplicate header - move the `ul` up into the header above and it'll look better. That's how it's designed to work.

Also, we should only show this if peachjam_ml is configured. We can either do template inheritance with overridden blocks, or we can assume that `peachjam` knows about the existence of `peachjam_ml` (like it knows about `peachjam_search`) and have something like `settings.PEACHJAM["PEACHJAM_ML"] = True` that gets injected into templates.

created_at: 2025-05-06T06:36:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
path: peachjam_ml/models.py
user: login: longhotsummer
body: It's not expensive to calculate the average of one, and it's all done in the DB. You can safely delete 6 lines of code -- less to test, less to go wrong.
created_at: 2025-05-06T06:37:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
+            # if there's only one document, we can just return its embedding
+            doc_embedding = cls.objects.filter(document__pk=pks[0]).first()
+            if doc_embedding:
+                return doc_embedding.text_embedding
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
+            avg = normalize_vector(avg)
+
+        return avg
+
+    @classmethod
+    def get_similar_documents(cls, doc_ids, threshold=0.0):
+        # always exclude the documents we are comparing against
+        exclude_ids = doc_ids
+        avg_embedding = cls.get_average_embedding(doc_ids)
+
+        similar_docs = (
+            DocumentEmbedding.objects.exclude(document__work__documents__in=exclude_ids)
path: peachjam_ml/models.py
user: login: longhotsummer
body: You'll have to show me -- we need to exclude all other documents that have the same works as the documents we're looking at. We don't want to include historical versions of a legislation document, for example.
created_at: 2025-05-06T06:39:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +147,50 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+        if not pks:
+            return None
+
+        if len(pks) == 1:
path: peachjam_ml/models.py
user: login: actlikewill
body: alright, changed
created_at: 2025-05-06T07:07:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,60 +1,42 @@
 from django.contrib.auth.mixins import PermissionRequiredMixin
-from django.db.models import F
-from django.shortcuts import get_object_or_404
 from django.utils.decorators import method_decorator
 from django.views.generic import DetailView
-from pgvector.django import MaxInnerProduct
 
 from peachjam.helpers import add_slash_to_frbr_uri
-from peachjam.models import CoreDocument
+from peachjam.models import CoreDocument, Folder
 from peachjam_ml.models import DocumentEmbedding
 
 
-@method_decorator(add_slash_to_frbr_uri(), name="setup")
-class SimilarDocumentsView(PermissionRequiredMixin, DetailView):
+class BaseSimilarDocumentsView(PermissionRequiredMixin, DetailView):
     permission_required = "peachjam_ml.view_documentembedding"
     template_name = "peachjam/_similar_documents.html"
-    slug_url_kwarg = "frbr_uri"
-    slug_field = "expression_frbr_uri"
-    model = CoreDocument
-    similarity_threshold = 0.0
-    weight_similarity = 0.7
-    weight_authority = 0.3
-    # choose the best from this set, after re-ranking
-    top_k = 100
-    n_similar = 10
 
-    def get_similar_documents(self):
-        embedding = get_object_or_404(DocumentEmbedding, document_id=self.object.pk)
-        similar_docs = (
-            DocumentEmbedding.objects.exclude(document__work=self.object.work)
-            .exclude(text_embedding__isnull=True)
-            .select_related("document")
-            .annotate(
-                similarity=MaxInnerProduct("text_embedding", embedding.text_embedding)
-                * -1,
-                title=F("document__title"),
-                expression_frbr_uri=F("document__expression_frbr_uri"),
-                authority_score=F("document__work__authority_score"),
-            )
-            .filter(similarity__gt=self.similarity_threshold)
-            .values("title", "expression_frbr_uri", "similarity", "authority_score")
-            .order_by("-similarity")
-        )[: self.top_k]
-
-        # re-rank based on a weighted average of similarity and authority score, and keep the top 10
-        similar_docs = sorted(
-            similar_docs,
-            key=lambda x: (
-                x["similarity"] * self.weight_similarity
-                + x["authority_score"] * self.weight_authority
-            ),
-            reverse=True,
-        )[: self.n_similar]
+    def get_doc_ids(self):
+        raise NotImplementedError("Subclasses must implement get_doc_ids()")
 
+    def get_similar_documents(self):
+        doc_ids = self.get_doc_ids()
+        similar_docs = DocumentEmbedding.get_similar_documents(doc_ids)
         return similar_docs
 
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)
         context["similar_documents"] = self.get_similar_documents()
         return context
+
+
+@method_decorator(add_slash_to_frbr_uri(), name="setup")
+class SimilarDocumentsView(BaseSimilarDocumentsView):
path: peachjam_ml/views.py
user: login: actlikewill
body: ok, i've removed the base class. much cleaner
created_at: 2025-05-06T07:07:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -147,6 +148,55 @@ def refresh_for_document(cls, document):
 
         return doc_embedding
 
+    @classmethod
+    def get_average_embedding(cls, pks):
+        """Get the average embedding for a set of documents."""
+
+        avg = (
+            DocumentEmbedding.objects.filter(document__in=pks)
+            .aggregate(avg=Avg("text_embedding"))
+            .get("avg")
+        )
+        if avg is not None and len(avg):
+            avg = normalize_vector(avg)
+
+        return avg
+
+    @classmethod
+    def get_similar_documents(cls, doc_ids, threshold=0.8, n_similar=10):
+        weight_similarity = 0.9
+        weight_authority = 0.1
path: peachjam_ml/models.py
user: login: actlikewill
body: adjusted the weights and threshold
created_at: 2025-05-06T14:58:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -151,6 +151,10 @@ footer {
   font-size: 22px;
 }
 
+.text-purple-500 {
path: peachjam/static/stylesheets/_global.scss
user: login: longhotsummer
body: Minor: I suggest making this semantic (eg `.text-ai`) and not the exact colour, and applying the same colour for the `pj-ai` icon, so that you can change that colour globally (eg. for ULII).
created_at: 2025-05-06T15:14:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,14 +1,19 @@
-from django.urls import re_path
+from django.urls import path, re_path
 from django.views.decorators.cache import cache_page
 
 from peachjam.urls import CACHE_DURATION
 
-from .views import SimilarDocumentsView
+from .views import SimilarDocumentsDocumentDetailView, SimilarDocumentsFolderView
 
 urlpatterns = [
     re_path(
         r"^(?P<frbr_uri>akn/.*)/similar-documents$",
-        cache_page(CACHE_DURATION)(SimilarDocumentsView.as_view()),
+        cache_page(CACHE_DURATION)(SimilarDocumentsDocumentDetailView.as_view()),
         name="document_similar_docs",
     ),
+    path(
+        "folder/<int:pk>/similar-documents",
+        cache_page(CACHE_DURATION)(SimilarDocumentsFolderView.as_view()),
path: peachjam_ml/urls.py
user: login: longhotsummer
body: Do we want to cache this? If I add a document or two to my folder, then the suggestions must update.
created_at: 2025-05-06T15:15:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,14 +1,19 @@
-from django.urls import re_path
+from django.urls import path, re_path
 from django.views.decorators.cache import cache_page
 
 from peachjam.urls import CACHE_DURATION
 
-from .views import SimilarDocumentsView
+from .views import SimilarDocumentsDocumentDetailView, SimilarDocumentsFolderView
 
 urlpatterns = [
     re_path(
         r"^(?P<frbr_uri>akn/.*)/similar-documents$",
-        cache_page(CACHE_DURATION)(SimilarDocumentsView.as_view()),
+        cache_page(CACHE_DURATION)(SimilarDocumentsDocumentDetailView.as_view()),
         name="document_similar_docs",
     ),
+    path(
+        "folder/<int:pk>/similar-documents",
+        cache_page(CACHE_DURATION)(SimilarDocumentsFolderView.as_view()),
path: peachjam_ml/urls.py
user: login: longhotsummer
body: It's also not horribly expensive.
created_at: 2025-05-06T15:15:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,14 +1,19 @@
-from django.urls import re_path
+from django.urls import path, re_path
 from django.views.decorators.cache import cache_page
 
 from peachjam.urls import CACHE_DURATION
 
-from .views import SimilarDocumentsView
+from .views import SimilarDocumentsDocumentDetailView, SimilarDocumentsFolderView
 
 urlpatterns = [
     re_path(
         r"^(?P<frbr_uri>akn/.*)/similar-documents$",
-        cache_page(CACHE_DURATION)(SimilarDocumentsView.as_view()),
+        cache_page(CACHE_DURATION)(SimilarDocumentsDocumentDetailView.as_view()),
         name="document_similar_docs",
     ),
+    path(
+        "folder/<int:pk>/similar-documents",
+        cache_page(CACHE_DURATION)(SimilarDocumentsFolderView.as_view()),
path: peachjam_ml/urls.py
user: login: actlikewill
body: makes sense, removed
created_at: 2025-05-07T07:25:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -151,6 +151,10 @@ footer {
   font-size: 22px;
 }
 
+.text-purple-500 {
path: peachjam/static/stylesheets/_global.scss
user: login: actlikewill
body: agreed, changed
created_at: 2025-05-07T07:25:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-02T14:11:12Z
body: ## Test Results
5 tests   - 1   5 ✅  - 1   0s ⏱️ ±0s
3 suites  - 1   0 💤 ±0 
3 files    - 1   0 ❌ ±0 

Results for commit 1f69c010. ± Comparison against base commit 2b069f6d.

<details>
  <summary>This pull request <b>removes</b> 3 and <b>adds</b> 2 tests. <i>Note that renamed tests count towards both.</i></summary>

```
liiweb.tests.test_legislation_views.LegislationViewsTest ‑ test_legislation_listing_national_only
liiweb.tests.test_mnc_matcher.MncMatcherTest ‑ test_html_matches
liiweb.tests.test_mnc_matcher.MncMatcherTest ‑ test_za_provincial_matches
```

```
africanlii.tests.test_views.AfricanliiViewsTest ‑ test_homepage
africanlii.tests.test_views.AfricanliiViewsTest ‑ test_legal_instrument_listing
```
</details>

[test-results]:data:application/gzip;base64,H4sIAKcLG2gC/1WMyw6CMBREf4V07aIPaYM/Q+ptb3IjUNPHivjvFhQtuzkzk7MypMknduvUpWOpUP6BK9FmCktFXrEOeZv6I4+pAJyLBz3b94iWplPhYwzx28SyHL4ttroP/207N7KdWxeEeaZcgQnUA3DBwQ1X9MbfFYheSmWkElqjkmCEdNaw1xu5PWdL+wAAAA==

:recycle: This comment has been updated with latest results.
author_association: NONE
user: login: longhotsummer
created_at: 2025-05-05T08:53:54Z
body: @actlikewill screenshots or loom, pls
author_association: CONTRIBUTOR