Repository: julep-ai/julep
Pull request #1319: fix: Speed up search
Author: login: whiterabbit1983
Labels: name: Review effort 4/5
Created: 2025-04-14T06:43:04Z | Merged: 2025-04-26T11:33:47Z

### **PR Type**
Enhancement


___

### **Description**
- Significantly optimize search performance in the database.
  - Add new indexes and adjust index settings for faster queries.
  - Increase statistics target for key columns to improve query planning.
  - Redesign and optimize the `search_by_text` function for efficiency.

- Provide up and down migrations for reversible changes.
  - Up migration applies optimizations and new constraints.
  - Down migration reverts index and statistics changes.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>000037_speed_up_search.up.sql</strong><dd><code>Major search optimization and function redesign migration</code></dd></summary>
<hr>

memory-store/migrations/000037_speed_up_search.up.sql

<li>Adds new primary and unique constraints to <code>docs</code> table.<br> <li> Creates and tunes indexes for faster search and lookup.<br> <li> Increases statistics targets for <code>title</code> and <code>content</code> columns.<br> <li> Redesigns <code>search_by_text</code> function for efficient FTS and trigram <br>search.<br> <li> Sets <code>work_mem</code> for improved query execution.


</details>


  </td>
  <td><a href="https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2e">+202/-0</a>&nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>000037_speed_up_search.down.sql</strong><dd><code>Revert search optimization changes and indexes</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

memory-store/migrations/000037_speed_up_search.down.sql

<li>Resets <code>work_mem</code> to default.<br> <li> Drops new indexes created in the up migration.<br> <li> Resets statistics targets for <code>docs</code> columns.<br> <li> Restores index settings to previous state.


</details>


  </td>
  <td><a href="https://github.com/julep-ai/julep/pull/1319/files#diff-35d5510843c975400960ab9bee1472f7a38034aa83e5a4f0192cc9ea19341069">+19/-0</a>&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about Qodo Merge usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>


Diff:
diff --git a/memory-store/migrations/000037_speed_up_search.down.sql b/memory-store/migrations/000037_speed_up_search.down.sql
new file mode 100644
index 000000000..1fd52c54e
--- /dev/null
+++ b/memory-store/migrations/000037_speed_up_search.down.sql
@@ -0,0 +1,19 @@
+BEGIN;
+-- Reset work_mem to default (or a reasonable value)
+RESET work_mem;
+
+-- Drop indexes created in the up migration
+DROP INDEX IF EXISTS idx_doc_owners_agent_lookup;
+DROP INDEX IF EXISTS idx_doc_owners_lookup;
+
+-- Reset statistics
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 100;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 100;
+
+-- Reset index settings
+ALTER INDEX idx_docs_metadata SET (fastupdate=on);
+ALTER INDEX idx_docs_search_tsv SET (fastupdate=on);
+ALTER INDEX idx_docs_title_trgm SET (fastupdate=on);
+ALTER INDEX idx_docs_content_trgm SET (fastupdate=on);
+
+END;
\ No newline at end of file
diff --git a/memory-store/migrations/000037_speed_up_search.up.sql b/memory-store/migrations/000037_speed_up_search.up.sql
new file mode 100644
index 000000000..5c7cc36ce
--- /dev/null
+++ b/memory-store/migrations/000037_speed_up_search.up.sql
@@ -0,0 +1,202 @@
+BEGIN;
+
+alter table docs drop constraint if exists pk_docs CASCADE;
+alter table docs drop constraint if exists uq_docs_dev_doc_idx;
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, index, created_at);
+alter table docs drop constraint if exists uq_doc_index;
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE,
+        migrate_data => TRUE
+    );
+
+alter index idx_docs_metadata set (fastupdate=off);
+alter index idx_docs_search_tsv set (fastupdate=off);
+alter index idx_docs_title_trgm set (fastupdate=off);
+alter index idx_docs_content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
+
+CREATE OR REPLACE FUNCTION search_by_text (
+    developer_id UUID,
+    query_text text,
+    owner_types TEXT[],
+    owner_ids UUID[],
+    search_language text DEFAULT 'english_unaccent',
+    k integer DEFAULT 3,
+    metadata_filter jsonb DEFAULT NULL,
+    similarity_threshold float DEFAULT 0.6
+) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
+DECLARE
+    ts_query tsquery;
+BEGIN
+    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
+        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
+    END IF;
+
+    -- Convert query to tsquery
+    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);
+
+    SET work_mem = '32MB';
+
+    RETURN QUERY EXECUTE 
+'WITH fts_results AS MATERIALIZED (
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
+    NULL::double precision AS trigram_score,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at,
+    /* Mark source=1 for FTS */
+    1 AS source
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2
+    -- Must pass FTS
+    AND d.search_tsv @@ $1
+    -- Filter by owners if needed
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 
+
+    
+),
+trgm_candidates AS MATERIALIZED (
+  /*
+   * Get docs that fail the FTS pass but pass a broad trigram test
+   * i.e. (title % :search_text) or (content % :search_text)
+   * so it can use your GIN/GIST trigram indexes.
+   */
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    d.search_tsv,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2::uuid
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
+    -- A broad trigram match (index-friendly):
+    AND (d.title  %> $6
+         OR d.content %> $6)
+    -- Exclude anything that was already in the FTS set:
+    AND NOT EXISTS (
+      SELECT 1
+      FROM fts_results f
+      WHERE f.doc_id = d.doc_id
+        AND f.index = d.index
+    )
+),
+trgm_scored AS (
+  /*
+   * Only now do the heavy comprehensive_similarity check,
+   * but only for these trigram candidates.
+   */
+  SELECT
+    t.developer_id,
+    t.doc_id,
+    t.index,
+    t.title,
+    t.content,
+    NULL::double precision AS tsv_score,
+    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
+    t.embedding,
+    t.metadata,
+    t.owner_type,
+    t.owner_id,
+    t.updated_at,
+    2 AS source
+  FROM trgm_candidates t
+  WHERE comprehensive_similarity(t.title, t.content, $6) > $7
+),
+combined AS (
+  /*
+   * Union the two sets:
+   *   - FTS matches (fts_results) => "score" is tsv_score
+   *   - Trigram-only matches => "score" is trigram_score
+   */
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id
+  FROM fts_results
+
+  UNION ALL
+
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id
+  FROM trgm_scored
+)
+SELECT *
+FROM combined
+ORDER BY score DESC
+LIMIT $8'
+
+    USING
+        ts_query,
+        developer_id,
+        owner_types,
+        owner_ids,
+        metadata_filter,
+        query_text,
+        similarity_threshold,
+        k;
+END;
+$$;
+
+END;
\ No newline at end of file


Reviews:
user: login: creatorrr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:47:38Z
user: login: creatorrr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:48:17Z
user: login: creatorrr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:49:25Z
user: login: creatorrr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:49:49Z
user: login: creatorrr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:50:22Z
user: login: whiterabbit1983
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T06:17:37Z
user: login: ellipsis-dev[bot]
body: > [!CAUTION]
> **Changes requested  :x:**
> <details>
> <summary>Reviewed everything up to c598edfb63f4f4d6824460b4a7e0eacacbbf02a1 in <ellipsistime>2 minutes and 20 seconds</ellipsistime>. Click for details.</summary> 
> 
> - Reviewed `235` lines of code in `2` files
> - Skipped `0` files when reviewing.
> - Skipped posting `9` draft comments. View those below.
> - Modify your [settings](https://app.ellipsis.dev/julep-ai/settings/code-review) and [rules](https://app.ellipsis.dev/julep-ai/code-review/rules) to customize what types of comments Ellipsis leaves. And don't forget to react with :+1: or :-1: to teach Ellipsis.
> 
> <details>
> <summary>1. <code>memory-store/migrations/000037_speed_up_search.down.sql:19</code></summary>
> 
> - **Draft comment:** 
> Add a newline at the end of the file for POSIX compliance.
> - **Reason this comment was not posted:**
> Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% While having a newline at the end of files is a common convention and technically POSIX-compliant, this is a very minor issue. It doesn't affect the functionality of the SQL migration. Most modern tools handle files without trailing newlines gracefully. This seems like an overly pedantic comment that doesn't require immediate action. The missing newline could potentially cause issues with some Unix tools or shell scripts that process the file. Some SQL clients might also behave differently without the trailing newline. While true, any serious SQL migration tool would handle this case properly, and the migration will work correctly either way. The benefit of fixing this is minimal compared to the noise it adds to the review. This comment should be removed as it's too minor and doesn't affect functionality. It violates the rule about not making obvious or unimportant comments.
> 
> 
> </details>
> 
> <details>
> <summary>2. <code>memory-store/migrations/000037_speed_up_search.up.sql:27</code></summary>
> 
> - **Draft comment:** 
> Review the use of 'SET work_mem' here and inside the function – the setting is being applied twice. Consider consolidating if possible.
> - **Reason this comment was not posted:**
> Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% Setting work_mem twice isn't necessarily wrong. The global setting affects the creation of the index on line 29, while the function setting ensures the memory is available for the complex query execution. The function setting also ensures the right memory is available even if the global setting changes later. Having both provides better isolation and predictability. I could be wrong about the scope and persistence of work_mem settings in PostgreSQL. Maybe one setting is redundant. Even if one setting is technically redundant, having both provides better guarantees about query performance and makes the code more maintainable and self-contained. The duplicate work_mem settings serve different purposes and provide better isolation. This isn't a clear problem that needs fixing.
> 
> 
> </details>
> 
> <details>
> <summary>3. <code>memory-store/migrations/000037_speed_up_search.up.sql:3</code></summary>
> 
> - **Draft comment:** 
> Dropping and re-adding primary key and unique constraints can impact data integrity and performance; ensure these changes are coordinated with application requirements.
> - **Reason this comment was not posted:**
> Comment did not seem useful. Confidence is useful = `0%` <= threshold `50%` The comment is advising the PR author to ensure that changes are coordinated with application requirements, which falls under asking the author to ensure behavior is intended or tested. This violates the rules.
> 
> 
> </details>
> 
> <details>
> <summary>4. <code>memory-store/migrations/000037_speed_up_search.up.sql:54</code></summary>
> 
> - **Draft comment:** 
> Dynamic SQL uses parameter placeholders; verify that the mapping in the USING clause matches the intended parameters to avoid misalignment.
> - **Reason this comment was not posted:**
> Comment did not seem useful. Confidence is useful = `0%` <= threshold `50%` The comment is asking the PR author to verify the mapping in the USING clause, which is against the rules as it asks for confirmation or verification. It doesn't provide a specific suggestion or point out a clear issue.
> 
> 
> </details>
> 
> <details>
> <summary>5. <code>memory-store/migrations/000037_speed_up_search.up.sql:31</code></summary>
> 
> - **Draft comment:** 
> Ensure that the return type 'doc_search_result' is correctly defined and matches the columns returned by the query.
> - **Reason this comment was not posted:**
> Comment did not seem useful. Confidence is useful = `0%` <= threshold `50%` This comment is asking the author to ensure that the return type matches the query result, which is a form of asking for confirmation or verification. It doesn't provide a specific suggestion or point out a clear issue, so it violates the rules.
> 
> 
> </details>
> 
> <details>
> <summary>6. <code>memory-store/migrations/000037_speed_up_search.up.sql:5</code></summary>
> 
> - **Draft comment:** 
> Using the column 'index' without quoting may lead to issues since it can be a reserved word; consider quoting it.
> - **Reason this comment was not posted:**
> Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% While 'index' is indeed a reserved word in SQL, PostgreSQL actually handles unquoted reserved words as column names in most contexts without issues. The code is a migration script that's already been written and tested. If there was an issue with the unquoted 'index', it would have caused immediate syntax errors. The fact that it's used consistently throughout without quotes suggests it's working fine in this codebase. The comment raises a valid point about SQL best practices. Quoting reserved words is generally safer and more explicit. However, PostgreSQL is known to handle this case, and the code is already working. Making this change would be purely stylistic and could introduce risks for no practical benefit. The comment should be removed as it suggests a change that's not strictly necessary and could introduce risk without clear benefit.
> 
> 
> </details>
> 
> <details>
> <summary>7. <code>memory-store/migrations/000037_speed_up_search.up.sql:119</code></summary>
> 
> - **Draft comment:** 
> Ensure that the operator %> used for trigram matching is defined and behaves as expected.
> - **Reason this comment was not posted:**
> Comment did not seem useful. Confidence is useful = `0%` <= threshold `50%` The comment is asking the author to ensure that the operator `%>` is defined and behaves as expected. This falls under asking the author to confirm or double-check something, which is against the rules.
> 
> 
> </details>
> 
> <details>
> <summary>8. <code>memory-store/migrations/000037_speed_up_search.down.sql:1</code></summary>
> 
> - **Draft comment:** 
> Down migration only resets work_mem, drops indexes, and adjusts statistics/fastupdate settings; it does not revert constraint changes, hypertable creation, or the search_by_text function. Confirm if this rollback behavior is intentional.
> - **Reason this comment was not posted:**
> Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% This is a speculative comment asking for confirmation about intended behavior. Without seeing the up migration, we can't know if there are actually missing reversals. The comment is asking the author to verify their intention, which violates our rules. Even if there are missing reversals, this type of "are you sure?" comment isn't actionable. What if there really are missing reversals that could cause problems in production? Maybe this is an important safety check. While database safety is important, asking for confirmation isn't actionable. If there are specific missing reversals that would cause problems, the comment should point those out directly with evidence. Delete the comment as it violates our rules against asking for confirmation of intentions and making speculative comments without clear evidence of problems.
> 
> 
> </details>
> 
> <details>
> <summary>9. <code>memory-store/migrations/000037_speed_up_search.down.sql:19</code></summary>
> 
> - **Draft comment:** 
> The file is missing a trailing newline at the end (after line 19). Adding a newline at the end of the file would be beneficial for adherence to conventions.
> - **Reason this comment was not posted:**
> Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% While trailing newlines are a common convention, this is a very minor formatting issue. It doesn't affect the functionality of the SQL migration. Most modern editors and git tools handle this automatically. This feels like the kind of trivial comment that clutters PR reviews without adding value. Missing newlines can cause issues with some tools and make diffs less clean. It's a widely accepted convention for a reason. While true, this is exactly the kind of obvious, minor formatting issue that we're instructed not to comment on. It's similar to commenting on whitespace or indentation. We should delete this comment as it's too minor and obvious to warrant a PR comment, falling under the rule about not making obvious or unimportant comments.
> 
> 
> </details>
> 
> 
> Workflow ID: <workflowid>`wflow_qt41S9836TthIP4t`</workflowid>
> </details>
> 
> <sup>You can customize </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=julep-ai%2Fjulep&utm_source=github&utm_medium=referral)<sup> by changing your [verbosity settings](https://app.ellipsis.dev/julep-ai/settings/code-review), reacting with :+1: or :-1:, [replying to comments](https://docs.ellipsis.dev/features/code-review#teaching-ellipsis), or adding [code review rules](https://app.ellipsis.dev/julep-ai/code-review/rules).</sup>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T10:08:45Z

Review comments:
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE
+    );
+
+SELECT
+    ai.create_vectorizer (
+        source => 'docs',
+        destination => 'docs_embeddings',
+        embedding => ai.embedding_openai ('text-embedding-3-large', 1024, 'document'), -- need to parameterize this
+        -- actual chunking is managed by the docs table
+        -- this is to prevent running out of context window
+        chunking => ai.chunking_recursive_character_text_splitter (
+            chunk_column => 'content',
+            chunk_size => 30000, -- 30k characters ~= 7.5k tokens
+            chunk_overlap => 600, -- 600 characters ~= 150 tokens
+            separators => ARRAY[ -- tries separators in order
+                -- markdown headers
+                E'\n#',
+                E'\n##',
+                E'\n###',
+                E'\n---',
+                E'\n***',
+                -- html tags
+                E'</article>', -- Split on major document sections
+                E'</div>', -- Split on div boundaries
+                E'</section>',
+                E'</p>', -- Split on paragraphs
+                E'<br>', -- Split on line breaks
+                -- other separators
+                E'\n\n', -- paragraphs
+                '. ',
+                '? ',
+                '! ',
+                '; ', -- sentences (note space after punctuation)
+                E'\n', -- line breaks
+                ' ' -- words (last resort)
+            ]
+        ),
+        scheduling => ai.scheduling_timescaledb (),
+        indexing => ai.indexing_diskann (),
+        formatting => ai.formatting_python_template (E'Title: $title\n\n$chunk'),
+        processing => ai.processing_default (),
+        enqueue_existing => TRUE
+    );
+
+alter index idx_docs_metadata|search_tsv|title_trgm|content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
+
+CREATE OR REPLACE FUNCTION search_by_text (
+    developer_id UUID,
+    query_text text,
+    owner_types TEXT[],
+    owner_ids UUID[],
+    search_language text DEFAULT 'english_unaccent',
+    k integer DEFAULT 3,
+    metadata_filter jsonb DEFAULT NULL,
+    similarity_threshold float DEFAULT 0.6
+) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
+DECLARE
+    ts_query tsquery;
+BEGIN
+    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
+        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
+    END IF;
+
+    -- Convert query to tsquery
+    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);
+
+    SET work_mem = '32MB';
+
+    RETURN QUERY EXECUTE 
+'WITH fts_results AS MATERIALIZED (
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
+    NULL::double precision AS trigram_score,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at,
+    /* Mark source=1 for FTS */
+    1 AS source
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2
+    -- Must pass FTS
+    AND d.search_tsv @@ $1
+    -- Filter by owners if needed
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 
+
+    
+),
+trgm_candidates AS MATERIALIZED (
+  /*
+   * Get docs that fail the FTS pass but pass a broad trigram test
+   * i.e. (title % :search_text) or (content % :search_text)
+   * so it can use your GIN/GIST trigram indexes.
+   */
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    d.search_tsv,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2::uuid
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
+    -- A broad trigram match (index-friendly):
+    AND (d.title  %> $6
+         OR d.content %> $6)
+    -- Exclude anything that was already in the FTS set:
+    AND NOT EXISTS (
+      SELECT 1
+      FROM fts_results f
+      WHERE f.doc_id = d.doc_id
+        AND f.index = d.index
+    )
+),
+trgm_scored AS (
+  /*
+   * Only now do the heavy comprehensive_similarity check,
+   * but only for these trigram candidates.
+   */
+  SELECT
+    t.developer_id,
+    t.doc_id,
+    t.index,
+    t.title,
+    t.content,
+    NULL::double precision AS tsv_score,
+    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
+    t.embedding,
+    t.metadata,
+    t.owner_type,
+    t.owner_id,
+    t.updated_at,
+    2 AS source
+  FROM trgm_candidates t
+  WHERE comprehensive_similarity(t.title, t.content, $6) > $7
+),
+combined AS (
+  /*
+   * Union the two sets:
+   *   - FTS matches (fts_results) => "score" is tsv_score
+   *   - Trigram-only matches => "score" is trigram_score
+   */
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id --,
+    -- updated_at,
+    -- source
+  FROM fts_results
+
+  UNION ALL
+
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id --,
+    -- updated_at,
+    -- source
+  FROM trgm_scored
+)
+SELECT *
+FROM combined
+ORDER BY score DESC -- , updated_at DESC
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: creatorrr
body: remove commented out code
created_at: 2025-04-15T14:47:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE
+    );
+
+SELECT
+    ai.create_vectorizer (
+        source => 'docs',
+        destination => 'docs_embeddings',
+        embedding => ai.embedding_openai ('text-embedding-3-large', 1024, 'document'), -- need to parameterize this
+        -- actual chunking is managed by the docs table
+        -- this is to prevent running out of context window
+        chunking => ai.chunking_recursive_character_text_splitter (
+            chunk_column => 'content',
+            chunk_size => 30000, -- 30k characters ~= 7.5k tokens
+            chunk_overlap => 600, -- 600 characters ~= 150 tokens
+            separators => ARRAY[ -- tries separators in order
+                -- markdown headers
+                E'\n#',
+                E'\n##',
+                E'\n###',
+                E'\n---',
+                E'\n***',
+                -- html tags
+                E'</article>', -- Split on major document sections
+                E'</div>', -- Split on div boundaries
+                E'</section>',
+                E'</p>', -- Split on paragraphs
+                E'<br>', -- Split on line breaks
+                -- other separators
+                E'\n\n', -- paragraphs
+                '. ',
+                '? ',
+                '! ',
+                '; ', -- sentences (note space after punctuation)
+                E'\n', -- line breaks
+                ' ' -- words (last resort)
+            ]
+        ),
+        scheduling => ai.scheduling_timescaledb (),
+        indexing => ai.indexing_diskann (),
+        formatting => ai.formatting_python_template (E'Title: $title\n\n$chunk'),
+        processing => ai.processing_default (),
+        enqueue_existing => TRUE
+    );
+
+alter index idx_docs_metadata|search_tsv|title_trgm|content_trgm set (fastupdate=off);
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: creatorrr
body: incorrect syntax, by the `|` chars I meant repeat for each of them, so `idx_docs_metadata`, `idx_docs_search_tsv`, ... and so on.
created_at: 2025-04-15T14:48:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: creatorrr
body: this _might_ not still work, more cleanup in internal vectorizer tables might be needed before you can recreate the vectorizer @whiterabbit1983 
created_at: 2025-04-15T14:49:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE
+    );
+
+SELECT
+    ai.create_vectorizer (
+        source => 'docs',
+        destination => 'docs_embeddings',
+        embedding => ai.embedding_openai ('text-embedding-3-large', 1024, 'document'), -- need to parameterize this
+        -- actual chunking is managed by the docs table
+        -- this is to prevent running out of context window
+        chunking => ai.chunking_recursive_character_text_splitter (
+            chunk_column => 'content',
+            chunk_size => 30000, -- 30k characters ~= 7.5k tokens
+            chunk_overlap => 600, -- 600 characters ~= 150 tokens
+            separators => ARRAY[ -- tries separators in order
+                -- markdown headers
+                E'\n#',
+                E'\n##',
+                E'\n###',
+                E'\n---',
+                E'\n***',
+                -- html tags
+                E'</article>', -- Split on major document sections
+                E'</div>', -- Split on div boundaries
+                E'</section>',
+                E'</p>', -- Split on paragraphs
+                E'<br>', -- Split on line breaks
+                -- other separators
+                E'\n\n', -- paragraphs
+                '. ',
+                '? ',
+                '! ',
+                '; ', -- sentences (note space after punctuation)
+                E'\n', -- line breaks
+                ' ' -- words (last resort)
+            ]
+        ),
+        scheduling => ai.scheduling_timescaledb (),
+        indexing => ai.indexing_diskann (),
+        formatting => ai.formatting_python_template (E'Title: $title\n\n$chunk'),
+        processing => ai.processing_default (),
+        enqueue_existing => TRUE
+    );
+
+alter index idx_docs_metadata|search_tsv|title_trgm|content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
+
+CREATE OR REPLACE FUNCTION search_by_text (
+    developer_id UUID,
+    query_text text,
+    owner_types TEXT[],
+    owner_ids UUID[],
+    search_language text DEFAULT 'english_unaccent',
+    k integer DEFAULT 3,
+    metadata_filter jsonb DEFAULT NULL,
+    similarity_threshold float DEFAULT 0.6
+) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
+DECLARE
+    ts_query tsquery;
+BEGIN
+    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
+        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
+    END IF;
+
+    -- Convert query to tsquery
+    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);
+
+    SET work_mem = '32MB';
+
+    RETURN QUERY EXECUTE 
+'WITH fts_results AS MATERIALIZED (
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
+    NULL::double precision AS trigram_score,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at,
+    /* Mark source=1 for FTS */
+    1 AS source
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2
+    -- Must pass FTS
+    AND d.search_tsv @@ $1
+    -- Filter by owners if needed
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 
+
+    
+),
+trgm_candidates AS MATERIALIZED (
+  /*
+   * Get docs that fail the FTS pass but pass a broad trigram test
+   * i.e. (title % :search_text) or (content % :search_text)
+   * so it can use your GIN/GIST trigram indexes.
+   */
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    d.search_tsv,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2::uuid
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
+    -- A broad trigram match (index-friendly):
+    AND (d.title  %> $6
+         OR d.content %> $6)
+    -- Exclude anything that was already in the FTS set:
+    AND NOT EXISTS (
+      SELECT 1
+      FROM fts_results f
+      WHERE f.doc_id = d.doc_id
+        AND f.index = d.index
+    )
+),
+trgm_scored AS (
+  /*
+   * Only now do the heavy comprehensive_similarity check,
+   * but only for these trigram candidates.
+   */
+  SELECT
+    t.developer_id,
+    t.doc_id,
+    t.index,
+    t.title,
+    t.content,
+    NULL::double precision AS tsv_score,
+    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
+    t.embedding,
+    t.metadata,
+    t.owner_type,
+    t.owner_id,
+    t.updated_at,
+    2 AS source
+  FROM trgm_candidates t
+  WHERE comprehensive_similarity(t.title, t.content, $6) > $7
+),
+combined AS (
+  /*
+   * Union the two sets:
+   *   - FTS matches (fts_results) => "score" is tsv_score
+   *   - Trigram-only matches => "score" is trigram_score
+   */
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id --,
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: creatorrr
body: remove commented out code
created_at: 2025-04-15T14:49:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE
+    );
+
+SELECT
+    ai.create_vectorizer (
+        source => 'docs',
+        destination => 'docs_embeddings',
+        embedding => ai.embedding_openai ('text-embedding-3-large', 1024, 'document'), -- need to parameterize this
+        -- actual chunking is managed by the docs table
+        -- this is to prevent running out of context window
+        chunking => ai.chunking_recursive_character_text_splitter (
+            chunk_column => 'content',
+            chunk_size => 30000, -- 30k characters ~= 7.5k tokens
+            chunk_overlap => 600, -- 600 characters ~= 150 tokens
+            separators => ARRAY[ -- tries separators in order
+                -- markdown headers
+                E'\n#',
+                E'\n##',
+                E'\n###',
+                E'\n---',
+                E'\n***',
+                -- html tags
+                E'</article>', -- Split on major document sections
+                E'</div>', -- Split on div boundaries
+                E'</section>',
+                E'</p>', -- Split on paragraphs
+                E'<br>', -- Split on line breaks
+                -- other separators
+                E'\n\n', -- paragraphs
+                '. ',
+                '? ',
+                '! ',
+                '; ', -- sentences (note space after punctuation)
+                E'\n', -- line breaks
+                ' ' -- words (last resort)
+            ]
+        ),
+        scheduling => ai.scheduling_timescaledb (),
+        indexing => ai.indexing_diskann (),
+        formatting => ai.formatting_python_template (E'Title: $title\n\n$chunk'),
+        processing => ai.processing_default (),
+        enqueue_existing => TRUE
+    );
+
+alter index idx_docs_metadata|search_tsv|title_trgm|content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
+
+CREATE OR REPLACE FUNCTION search_by_text (
+    developer_id UUID,
+    query_text text,
+    owner_types TEXT[],
+    owner_ids UUID[],
+    search_language text DEFAULT 'english_unaccent',
+    k integer DEFAULT 3,
+    metadata_filter jsonb DEFAULT NULL,
+    similarity_threshold float DEFAULT 0.6
+) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
+DECLARE
+    ts_query tsquery;
+BEGIN
+    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
+        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
+    END IF;
+
+    -- Convert query to tsquery
+    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);
+
+    SET work_mem = '32MB';
+
+    RETURN QUERY EXECUTE 
+'WITH fts_results AS MATERIALIZED (
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
+    NULL::double precision AS trigram_score,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at,
+    /* Mark source=1 for FTS */
+    1 AS source
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2
+    -- Must pass FTS
+    AND d.search_tsv @@ $1
+    -- Filter by owners if needed
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 
+
+    
+),
+trgm_candidates AS MATERIALIZED (
+  /*
+   * Get docs that fail the FTS pass but pass a broad trigram test
+   * i.e. (title % :search_text) or (content % :search_text)
+   * so it can use your GIN/GIST trigram indexes.
+   */
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    d.search_tsv,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2::uuid
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
+    -- A broad trigram match (index-friendly):
+    AND (d.title  %> $6
+         OR d.content %> $6)
+    -- Exclude anything that was already in the FTS set:
+    AND NOT EXISTS (
+      SELECT 1
+      FROM fts_results f
+      WHERE f.doc_id = d.doc_id
+        AND f.index = d.index
+    )
+),
+trgm_scored AS (
+  /*
+   * Only now do the heavy comprehensive_similarity check,
+   * but only for these trigram candidates.
+   */
+  SELECT
+    t.developer_id,
+    t.doc_id,
+    t.index,
+    t.title,
+    t.content,
+    NULL::double precision AS tsv_score,
+    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
+    t.embedding,
+    t.metadata,
+    t.owner_type,
+    t.owner_id,
+    t.updated_at,
+    2 AS source
+  FROM trgm_candidates t
+  WHERE comprehensive_similarity(t.title, t.content, $6) > $7
+),
+combined AS (
+  /*
+   * Union the two sets:
+   *   - FTS matches (fts_results) => "score" is tsv_score
+   *   - Trigram-only matches => "score" is trigram_score
+   */
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id --,
+    -- updated_at,
+    -- source
+  FROM fts_results
+
+  UNION ALL
+
+  SELECT
+    developer_id,
+    doc_id,
+    index,
+    title,
+    content,
+    COALESCE(tsv_score, trigram_score) AS score,
+    embedding,
+    metadata,
+    owner_type,
+    owner_id --,
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: creatorrr
body: remove commented out code
created_at: 2025-04-15T14:50:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,238 @@
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, created_at);
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+select ai.drop_vectorizer(id) from ai.vectorizer;
+
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: whiterabbit1983
body: what kind of cleanup?
created_at: 2025-04-16T06:17:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,202 @@
+BEGIN;
+
+alter table docs drop constraint if exists pk_docs CASCADE;
+alter table docs drop constraint if exists uq_docs_dev_doc_idx;
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, index, created_at);
+alter table docs drop constraint if exists uq_doc_index;
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE,
+        migrate_data => TRUE
+    );
+
+alter index idx_docs_metadata set (fastupdate=off);
+alter index idx_docs_search_tsv set (fastupdate=off);
+alter index idx_docs_title_trgm set (fastupdate=off);
+alter index idx_docs_content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: ellipsis-dev[bot]
body: Duplicate index on `doc_owners`: `idx_doc_owners_agent_lookup` duplicates `idx_doc_owners_lookup`. Consider removing one to avoid unnecessary overhead.
```suggestion

```

created_at: 2025-04-26T10:08:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,202 @@
+BEGIN;
+
+alter table docs drop constraint if exists pk_docs CASCADE;
+alter table docs drop constraint if exists uq_docs_dev_doc_idx;
+alter table docs add constraint pk_docs primary key (developer_id, doc_id, index, created_at);
+alter table docs drop constraint if exists uq_doc_index;
+alter table docs add constraint uq_doc_index unique (doc_id, index, created_at);
+
+SELECT
+    create_hypertable (
+        'docs',
+        by_range ('created_at', INTERVAL '1 day'),
+        if_not_exists => TRUE,
+        migrate_data => TRUE
+    );
+
+alter index idx_docs_metadata set (fastupdate=off);
+alter index idx_docs_search_tsv set (fastupdate=off);
+alter index idx_docs_title_trgm set (fastupdate=off);
+alter index idx_docs_content_trgm set (fastupdate=off);
+
+ALTER TABLE docs ALTER COLUMN title SET STATISTICS 1000;
+ALTER TABLE docs ALTER COLUMN content SET STATISTICS 1000;
+CREATE INDEX idx_doc_owners_lookup 
+ON doc_owners(owner_type, owner_id, doc_id);
+
+SET work_mem = '32MB';
+
+CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
+
+CREATE OR REPLACE FUNCTION search_by_text (
+    developer_id UUID,
+    query_text text,
+    owner_types TEXT[],
+    owner_ids UUID[],
+    search_language text DEFAULT 'english_unaccent',
+    k integer DEFAULT 3,
+    metadata_filter jsonb DEFAULT NULL,
+    similarity_threshold float DEFAULT 0.6
+) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
+DECLARE
+    ts_query tsquery;
+BEGIN
+    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
+        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
+    END IF;
+
+    -- Convert query to tsquery
+    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);
+
+    SET work_mem = '32MB';
+
+    RETURN QUERY EXECUTE 
+'WITH fts_results AS MATERIALIZED (
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
+    NULL::double precision AS trigram_score,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at,
+    /* Mark source=1 for FTS */
+    1 AS source
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2
+    -- Must pass FTS
+    AND d.search_tsv @@ $1
+    -- Filter by owners if needed
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 
+
+    
+),
+trgm_candidates AS MATERIALIZED (
+  /*
+   * Get docs that fail the FTS pass but pass a broad trigram test
+   * i.e. (title % :search_text) or (content % :search_text)
+   * so it can use your GIN/GIST trigram indexes.
+   */
+  SELECT
+    d.developer_id,
+    d.doc_id,
+    d.index,
+    d.title,
+    d.content,
+    d.search_tsv,
+    e.embedding,
+    d.metadata,
+    o.owner_type,
+    o.owner_id,
+    d.updated_at
+  FROM docs d
+  LEFT JOIN docs_embeddings e
+    ON e.developer_id = d.developer_id
+    AND e.doc_id = d.doc_id
+    AND e.index = d.index
+  LEFT JOIN doc_owners o
+    ON d.doc_id = o.doc_id
+  
+  WHERE d.developer_id = $2::uuid
+    AND o.owner_type = ANY($3)
+    AND o.owner_id   = ANY($4::uuid[])
+    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
+    -- A broad trigram match (index-friendly):
+    AND (d.title  %> $6
+         OR d.content %> $6)
+    -- Exclude anything that was already in the FTS set:
+    AND NOT EXISTS (
+      SELECT 1
+      FROM fts_results f
+      WHERE f.doc_id = d.doc_id
+        AND f.index = d.index
+    )
+),
+trgm_scored AS (
+  /*
+   * Only now do the heavy comprehensive_similarity check,
+   * but only for these trigram candidates.
+   */
+  SELECT
+    t.developer_id,
+    t.doc_id,
+    t.index,
+    t.title,
+    t.content,
+    NULL::double precision AS tsv_score,
+    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
path: memory-store/migrations/000037_speed_up_search.up.sql
user: login: ellipsis-dev[bot]
body: The function `comprehensive_similarity` is computed twice (in `SELECT` and in `WHERE`); consider computing it once for better performance.

created_at: 2025-04-26T10:08:45Z
author_association: CONTRIBUTOR

PR comments:
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-04-26T10:06:48Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 4 🔵🔵🔵🔵⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR24-R29'><strong>Duplicate Index</strong></a>

The same index `idx_doc_owners_agent_lookup` is created twice with identical columns (owner_type, owner_id, doc_id). First on line 24-25 and then again on line 29, which is redundant and could cause issues.
</summary>

```sql
CREATE INDEX idx_doc_owners_lookup 
ON doc_owners(owner_type, owner_id, doc_id);

SET work_mem = '32MB';

CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
```

</details>

<details><summary><a href='https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR27-R51'><strong>Work Memory Setting</strong></a>

The `work_mem` is set twice - once globally at line 27 and again inside the function at line 51. This could lead to confusion about which setting takes precedence and may affect other queries in the system.
</summary>

```sql
SET work_mem = '32MB';

CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);

CREATE OR REPLACE FUNCTION search_by_text (
    developer_id UUID,
    query_text text,
    owner_types TEXT[],
    owner_ids UUID[],
    search_language text DEFAULT 'english_unaccent',
    k integer DEFAULT 3,
    metadata_filter jsonb DEFAULT NULL,
    similarity_threshold float DEFAULT 0.6
) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
DECLARE
    ts_query tsquery;
BEGIN
    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
    END IF;

    -- Convert query to tsquery
    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);

    SET work_mem = '32MB';
```

</details>

<details><summary><a href='https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR31-R200'><strong>Error Handling</strong></a>

The function lacks error handling for cases where the trigram search might fail or when the materialized CTEs might consume excessive memory for large datasets.
</summary>

```sql
CREATE OR REPLACE FUNCTION search_by_text (
    developer_id UUID,
    query_text text,
    owner_types TEXT[],
    owner_ids UUID[],
    search_language text DEFAULT 'english_unaccent',
    k integer DEFAULT 3,
    metadata_filter jsonb DEFAULT NULL,
    similarity_threshold float DEFAULT 0.6
) RETURNS SETOF doc_search_result LANGUAGE plpgsql AS $$
DECLARE
    ts_query tsquery;
BEGIN
    IF array_length(owner_types, 1) != array_length(owner_ids, 1) THEN
        RAISE EXCEPTION 'owner_types and owner_ids must be the same length';
    END IF;

    -- Convert query to tsquery
    ts_query := websearch_to_tsquery(search_language::regconfig, query_text);

    SET work_mem = '32MB';

    RETURN QUERY EXECUTE 
'WITH fts_results AS MATERIALIZED (
  SELECT
    d.developer_id,
    d.doc_id,
    d.index,
    d.title,
    d.content,
    ts_rank_cd(d.search_tsv, $1, 32)::double precision AS tsv_score,
    NULL::double precision AS trigram_score,
    e.embedding,
    d.metadata,
    o.owner_type,
    o.owner_id,
    d.updated_at,
    /* Mark source=1 for FTS */
    1 AS source
  FROM docs d
  LEFT JOIN docs_embeddings e
    ON e.developer_id = d.developer_id
    AND e.doc_id = d.doc_id
    AND e.index = d.index
  LEFT JOIN doc_owners o
    ON d.doc_id = o.doc_id

  WHERE d.developer_id = $2
    -- Must pass FTS
    AND d.search_tsv @@ $1
    -- Filter by owners if needed
    AND o.owner_type = ANY($3)
    AND o.owner_id   = ANY($4::uuid[])
    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb) 


),
trgm_candidates AS MATERIALIZED (
  /*
   * Get docs that fail the FTS pass but pass a broad trigram test
   * i.e. (title % :search_text) or (content % :search_text)
   * so it can use your GIN/GIST trigram indexes.
   */
  SELECT
    d.developer_id,
    d.doc_id,
    d.index,
    d.title,
    d.content,
    d.search_tsv,
    e.embedding,
    d.metadata,
    o.owner_type,
    o.owner_id,
    d.updated_at
  FROM docs d
  LEFT JOIN docs_embeddings e
    ON e.developer_id = d.developer_id
    AND e.doc_id = d.doc_id
    AND e.index = d.index
  LEFT JOIN doc_owners o
    ON d.doc_id = o.doc_id

  WHERE d.developer_id = $2::uuid
    AND o.owner_type = ANY($3)
    AND o.owner_id   = ANY($4::uuid[])
    AND (($5)::jsonb IS NULL OR d.metadata @> ($5)::jsonb)
    -- A broad trigram match (index-friendly):
    AND (d.title  %> $6
         OR d.content %> $6)
    -- Exclude anything that was already in the FTS set:
    AND NOT EXISTS (
      SELECT 1
      FROM fts_results f
      WHERE f.doc_id = d.doc_id
        AND f.index = d.index
    )
),
trgm_scored AS (
  /*
   * Only now do the heavy comprehensive_similarity check,
   * but only for these trigram candidates.
   */
  SELECT
    t.developer_id,
    t.doc_id,
    t.index,
    t.title,
    t.content,
    NULL::double precision AS tsv_score,
    comprehensive_similarity(t.title, t.content, $6)::double precision AS trigram_score,
    t.embedding,
    t.metadata,
    t.owner_type,
    t.owner_id,
    t.updated_at,
    2 AS source
  FROM trgm_candidates t
  WHERE comprehensive_similarity(t.title, t.content, $6) > $7
),
combined AS (
  /*
   * Union the two sets:
   *   - FTS matches (fts_results) => "score" is tsv_score
   *   - Trigram-only matches => "score" is trigram_score
   */
  SELECT
    developer_id,
    doc_id,
    index,
    title,
    content,
    COALESCE(tsv_score, trigram_score) AS score,
    embedding,
    metadata,
    owner_type,
    owner_id
  FROM fts_results

  UNION ALL

  SELECT
    developer_id,
    doc_id,
    index,
    title,
    content,
    COALESCE(tsv_score, trigram_score) AS score,
    embedding,
    metadata,
    owner_type,
    owner_id
  FROM trgm_scored
)
SELECT *
FROM combined
ORDER BY score DESC
LIMIT $8'

    USING
        ts_query,
        developer_id,
        owner_types,
        owner_ids,
        metadata_filter,
        query_text,
        similarity_threshold,
        k;
END;
$$;
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-04-26T10:07:07Z
body: ## PR Code Suggestions ✨

<!-- c598edf -->

Explore these optional code suggestions:

<table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=3>General</td>
<td>



<details><summary>Remove duplicate index</summary>

___


**The <code>idx_doc_owners_agent_lookup</code> index is redundant as it has the exact same <br>columns as <code>idx_doc_owners_lookup</code> created just before it. Having duplicate <br>indexes wastes disk space and slows down write operations without providing any <br>performance benefit.**

[memory-store/migrations/000037_speed_up_search.up.sql [24-29]](https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR24-R29)

```diff
 CREATE INDEX idx_doc_owners_lookup 
 ON doc_owners(owner_type, owner_id, doc_id);
 
 SET work_mem = '32MB';
 
-CREATE INDEX idx_doc_owners_agent_lookup ON doc_owners(owner_type, owner_id, doc_id);
-
```

- [ ] **Apply this suggestion** <!-- /improve --apply_suggestion=0 -->

<details><summary>Suggestion importance[1-10]: 9</summary>

__

Why: The suggestion correctly identifies that idx_doc_owners_agent_lookup is an exact duplicate of idx_doc_owners_lookup with identical columns. Removing this redundant index would prevent wasted disk space and avoid unnecessary overhead during write operations.


</details></details></td><td align=center>High

</td></tr><tr><td>



<details><summary>Remove redundant configuration setting</summary>

___


**The <code>work_mem</code> parameter is set twice - once at the global level and again inside <br>the <code>search_by_text</code> function. The second setting is redundant and could cause <br>confusion about which setting is in effect. Consider removing the duplicate <br>setting inside the function.**

[memory-store/migrations/000037_speed_up_search.up.sql [27-51]](https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR27-R51)

```diff
 SET work_mem = '32MB';
 
--- Later in the function:
-SET work_mem = '32MB';
+-- Inside function, remove the duplicate:
+-- SET work_mem = '32MB';  -- Already set at transaction level
```

- [ ] **Apply this suggestion** <!-- /improve --apply_suggestion=1 -->

<details><summary>Suggestion importance[1-10]: 8</summary>

__

Why: The suggestion correctly identifies that work_mem is set twice - once at the transaction level (line 27) and again inside the function (line 51). This redundancy could cause confusion and the second setting is unnecessary since the parameter is already set.


</details></details></td><td align=center>Medium

</td></tr><tr><td>



<details><summary>Make configuration change permanent</summary>

___


**Setting <code>work_mem</code> at the transaction level will only affect the current <br>transaction. For a permanent configuration change, consider using <code>ALTER DATABASE</code> <br>or updating the PostgreSQL configuration file. Otherwise, the setting will <br>revert after the migration completes.**

[memory-store/migrations/000037_speed_up_search.up.sql [27]](https://github.com/julep-ai/julep/pull/1319/files#diff-e5c0b38e01b51f710894c8667c0b70c01350f46c25d1db5a60a12fdc775c6c2eR27-R27)

```diff
-SET work_mem = '32MB';
+ALTER DATABASE current_database() SET work_mem = '32MB';
```

- [ ] **Apply this suggestion** <!-- /improve --apply_suggestion=2 -->

<details><summary>Suggestion importance[1-10]: 7</summary>

__

Why: The suggestion correctly points out that SET work_mem only affects the current transaction and will revert after migration. Using ALTER DATABASE would make the change persistent, which is likely the intended behavior for a performance optimization migration.


</details></details></td><td align=center>Medium

</td></tr>
<tr><td align="center" colspan="2">

- [ ] More <!-- /improve --more_suggestions=true -->

</td><td></td></tr></tbody></table>


author_association: CONTRIBUTOR