Repository: sciencehistory/scihist_digicoll
Pull request #2927: Add all the works returned by a blacklight search to the user's cart
Author: login: eddierubeiz
Created: 2025-03-20T17:11:20Z | Merged: 2025-03-31T15:38:46Z

ref #2659.

We add a button to the search results page that goes to a new controller and method:

- All the works returned by the search (up to 10 million of them) will be added to the logged-in user's cart;
- Sort is preserved;
- Collections returned by the search are ignored, as are works already in the user's cart;
- We have a request test that asserts that the friendlier_ids are exactly the same as the ordinary search results. Not sure if we need more tests.
- The user is then forwarded back to the same results page.

For now, you can't do this on the collection search page, featured topics page, etc.

Does not load the actual items from the database; does not get any data from SOLR except the ids; does not instantiate the works at any point. Therefore, this is quite fast.




Diff:
diff --git a/app/controllers/all_search_result_ids_controller.rb b/app/controllers/all_search_result_ids_controller.rb
new file mode 100644
index 000000000..9f5611fbc
--- /dev/null
+++ b/app/controllers/all_search_result_ids_controller.rb
@@ -0,0 +1,45 @@
+# This controller (just used for #add_to_cart) runs a Blacklight search for all works and collections specified in the params.
+# and returns JUST the ids in the search results, in a JSON array.
+class AllSearchResultIdsController < CatalogController
+  before_action :authenticate_user! # need to be logged in
+
+  configure_blacklight do |config|
+    config.search_state_fields << :all_search_result_ids
+  end
+
+  # A CartItem doesn't contain a lot of data, so this goes fast, even for thousands of ids
+  def add_to_cart
+    CartItem.transaction do
+      all_ids_from_search.each_slice(500) do |ids|
+        CartItem.upsert_all( ids.map { |id| { user_id: user_id, work_id: id } } ,  unique_by: [:user_id, :work_id])
+      end
+    end
+    notice =  "Added these works to your cart."
+  rescue StandardError => e
+    notice = "Error adding these works to your cart: #{e.message}"
+    raise
+  ensure
+    redirect_back_or_to '/', allow_other_host: false, notice: notice
+  end
+
+
+  # tell AllSearchResultIdsBuilder to modify the solr params before connecting to solr
+  def search_service_context
+    super.merge!(all_search_result_ids: 'true')
+  end
+
+  # no need to bulk-load works; there could be thousands of them.
+  self.search_service_class =  Blacklight::SearchService
+
+  private
+
+  # Careful - these may contain Kithe::Collection ids, too.
+  def all_ids_from_search
+    search_service.search_results['response']['docs'].map { |doc| doc['model_pk_ssi'] }
+  end
+
+  def user_id
+    @user_id ||= current_user.id
+  end
+
+end
\ No newline at end of file
diff --git a/app/controllers/catalog_controller.rb b/app/controllers/catalog_controller.rb
index 3ca301201..4d640fa44 100644
--- a/app/controllers/catalog_controller.rb
+++ b/app/controllers/catalog_controller.rb
@@ -216,6 +216,22 @@ def active_sort_fields
     # config.add_results_document_tool(:bookmark, partial: 'bookmark_control', if: :render_bookmarks_control?)
 
     config.add_results_collection_tool(:sort_widget)
+
+
+    # See also:
+    # app/models/search_builder/all_search_result_ids_builder.rb
+    # app/controllers/all_search_result_ids_controller.rb
+    #
+    # Don't show this tool on e.g. the collection results page or the featured topics page.
+    config.add_results_collection_tool(
+      :all_search_results_add_to_cart,
+      opts = {
+        if: proc { |controller| controller.can?(:update, Kithe::Model) && controller.class.name == 'CatalogController' }
+      }
+    )
+
+
+
     # config.add_results_collection_tool(:per_page_widget)
     # config.add_results_collection_tool(:view_type_group)
 
diff --git a/app/models/search_builder.rb b/app/models/search_builder.rb
index 9e5e29c60..c9ba5f33c 100644
--- a/app/models/search_builder.rb
+++ b/app/models/search_builder.rb
@@ -13,6 +13,7 @@ class SearchBuilder < Blacklight::SearchBuilder
   include SearchBuilder::AccessControlFilter
   include SearchBuilder::AdminOnlySearchFields
   include SearchBuilder::CustomSortLogic
+  include SearchBuilder::AllSearchResultIdsBuilder
 
   ##
   # @example Adding a new step to the processor chain
diff --git a/app/models/search_builder/all_search_result_ids_builder.rb b/app/models/search_builder/all_search_result_ids_builder.rb
new file mode 100644
index 000000000..496d004ff
--- /dev/null
+++ b/app/models/search_builder/all_search_result_ids_builder.rb
@@ -0,0 +1,46 @@
+class SearchBuilder
+
+  module AllSearchResultIdsBuilder
+    extend ActiveSupport::Concern
+
+    included do
+      self.default_processor_chain += [:all_search_result_ids_processor]
+    end
+
+    # Return only friendlier_ids, and return up to ten million of them.
+    # We take some pains to request as little as possible from SOLR other than the actual friendlier_ids.
+    def all_search_result_ids_processor(solr_parameters)
+
+      # This method runs on all searches, but we exit early unless we actually want this.
+      return unless scope.context[:all_search_result_ids] == 'true'
+
+      solr_parameters.except!(*unneeded_keys)
+
+      # remove all facet limit info as well:
+      facet_limit_keys = solr_parameters.keys.select  {|k| k.include? 'facet.limit'}
+      solr_parameters.except!(*facet_limit_keys)
+  
+      solr_parameters.merge!({fl:"model_pk_ssi", rows: '10000000'})
+    end
+
+    def unneeded_keys
+      @unneeded_keys ||= [
+        # Keep the sort for now
+        # ['sort']
+
+        # don't need stats
+        ["stats", "stats.field"],
+
+        # don't need highlighting
+        [
+          "hl", "hl.method", "hl.fl", "hl.usePhraseHighlighter", "hl.snippets",
+          "hl.encoder", "hl.maxAnalyzedChars", "hl.bs.type",
+          "hl.fragsize", "hl.fragsizeIsMinimum"
+        ],
+
+        # Also think we don't need these:
+        ["facet", "facet.field", "facet.query"],
+      ].flatten!
+    end
+  end
+end
diff --git a/app/views/catalog/_all_search_results_add_to_cart.html.erb b/app/views/catalog/_all_search_results_add_to_cart.html.erb
new file mode 100644
index 000000000..2d18f33bb
--- /dev/null
+++ b/app/views/catalog/_all_search_results_add_to_cart.html.erb
@@ -0,0 +1,3 @@
+<div id="add-results-to-cart" class="add-results-to-cart">
+  <%= link_to 'ADD RESULTS TO CART', request.params.merge(action: "add_to_cart", controller: 'all_search_result_ids'), class:"btn btn-outline-secondary" %>
+</div>
diff --git a/config/routes.rb b/config/routes.rb
index 407d26962..a0842c19c 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -192,6 +192,10 @@ def self.matches?(request)
   get "focus/:slug/range_limit_panel" => "featured_topic#range_limit_panel"
   get "focus/:slug/facet" => "featured_topic#facet"
 
+  # Run a Blacklight search, but ONLY return the friendlier_ids, and return ALL of them.
+  # Add them to the user's cart.
+  get "all_search_result_ids/add_to_cart" => "all_search_result_ids#add_to_cart", as: :all_search_result_ids_add_to_cart
+
 
   # the file_category is in here solely so we can distinguish in robots.txt,
   # we want the file type in the URL. It's not actually used by controller.
diff --git a/spec/requests/admin/all_search_result_ids_spec.rb b/spec/requests/admin/all_search_result_ids_spec.rb
new file mode 100644
index 000000000..a05b21808
--- /dev/null
+++ b/spec/requests/admin/all_search_result_ids_spec.rb
@@ -0,0 +1,97 @@
+require 'rails_helper'
+describe AllSearchResultIdsController, type: :request, solr: true, queue_adapter: :test, indexable_callbacks: true, logged_in_user: true do
+
+  #let!(:user) { FactoryBot.create(:admin_user, email: 'the_user@sciencehistory.org', password: "goatgoat") }
+
+
+  let(:works_in_cart) {
+    User.all.first.works_in_cart.count
+  }
+
+  let(:search_results) {
+    controller.search_service.search_results
+  }
+
+
+  context "smoke test" do
+    let!(:works) { [
+      create(:work),
+      create(:work)
+    ] }
+    let!(:collection) { create(:collection)}
+
+    it "successfully adds works to the user's cart" do
+      get all_search_result_ids_add_to_cart_path
+      expect(response.code).to eq "302"
+      expect(works_in_cart).to eq 2
+    end
+  end
+
+  context "checking parameters" do
+
+
+    let(:complex_query_with_facets) {
+      {
+        "search_field" => "all_fields",
+        "f" => {
+          "creator_facet"=>["Dow Chemical Company"],
+          "department_facet"=>["Archives"],
+          "format_facet"=>["Image"],
+          "subject_facet"=>["Employees", "Industries"]
+        },
+        "q" => "Midland",
+        "range" => {
+            "year_facet_isim"=>{"begin"=>"1950", "end"=>"2023"}
+        },
+      }
+    }
+
+    let(:solr_params) {
+      controller.search_service.search_results['responseHeader']['params']
+    }
+
+    it "controller params unchanged" do
+      get all_search_result_ids_add_to_cart_path, params: complex_query_with_facets
+
+      expect(response.code).to eq "302"
+
+      incoming_params = controller.params.dup.permit!.to_h
+      incoming_params.delete('action')
+      incoming_params.delete('controller')
+      expect(incoming_params).to eq complex_query_with_facets
+    end
+
+    it "makes the edits we want before contacting solr" do
+      get all_search_result_ids_add_to_cart_path, params: complex_query_with_facets
+
+      # should return exclusively IDs:
+      expect(solr_params['fl']).to eq  "model_pk_ssi"
+
+      # should return ten million works, max:
+      expect(solr_params['rows']).to eq "10000000"
+
+      # all other solr params are preserved, so we know the IDS correspond exactly to the search results from the catalog controller:
+
+      # search phrase
+      expect(solr_params['q']).to eq "Midland"
+
+      # fields:
+      expect(solr_params['qf']).to eq "text1_tesim^1000 text2_tesim^500 text3_tesim^100 text4_tesim^50 description_text4_tesim^50 text_no_boost_tesim^10 friendlier_id_ssi id^10 searchable_fulltext_en^0.5 searchable_fulltext_de^0.5 searchable_fulltext_language_agnostic^0.5 admin_only_text_tesim admin_only_text_tesim"
+      expect(solr_params['pf']).to eq "text1_tesim^1500 text2_tesim^1200 text3_tesim^600 text4_tesim^120 description_text4_tesim^120 text_no_boost_tesim^55 friendlier_id_ssi id^55 searchable_fulltext_en^12 searchable_fulltext_de^12 searchable_fulltext_language_agnostic^12"
+
+      # facets:
+      expect(solr_params['fq']).to eq   [
+        "year_facet_isim:[1950 TO 2023]",
+        "{!term f=subject_facet}Employees",
+        "{!term f=subject_facet}Industries",
+        "{!term f=creator_facet}Dow Chemical Company",
+        "{!term f=format_facet}Image",
+        "{!term f=department_facet}Archives"
+       ]
+
+      # default sort:
+      expect(solr_params['sort']).to eq "score desc, date_created_dtsi desc"
+    end
+  end
+
+end


Reviews:
user: login: jrochkind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:49:10Z
user: login: jrochkind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:50:19Z
user: login: jrochkind
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:51:20Z
user: login: eddierubeiz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T16:31:35Z

Review comments:
diff_hunk: @@ -0,0 +1,72 @@
+# This controller (just used for #add_to_cart) runs a Blacklight search for all works and collections specified in the params.
+# and returns JUST the friendlier ids in the search results, in a JSON array.
+class AllSearchResultIdsController < CatalogController
+  before_action :authenticate_user! # need to be logged in
+
+  configure_blacklight do |config|
+    config.search_state_fields << :all_search_result_ids
+  end
+
+
+  # A CartItem doesn't contain a lot of data, so this goes fast, even for thousands of ids
+  def add_to_cart
+    conn = ActiveRecord::Base.connection
+    CartItem.transaction do
+      all_ids_from_search.each_slice(500) do |ids|
+        conn.execute add_friendlier_ids_to_cart_sql(ids)
path: app/controllers/all_search_result_ids_controller.rb
user: login: jrochkind
body: I still think there oughta be a way to do this with ActiveRecord insert_all and/or upsert_all that would be just as performant, and somewhat neater, without having to maintain our own SQL. 

But there's really nothing _wrong_ with doing it this way, and you have done it very tidily!  Not sure if you experimented with insert_all/upsert_all.    I wonder if we should try it -- are your tests good enough that if they pass we are confident? :). 

But also really nothing wrong with this way. 
created_at: 2025-03-26T22:49:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -216,6 +216,22 @@ def active_sort_fields
     # config.add_results_document_tool(:bookmark, partial: 'bookmark_control', if: :render_bookmarks_control?)
 
     config.add_results_collection_tool(:sort_widget)
+
+
+    # See also:
+    # app/models/search_builder/all_search_result_ids_builder.rb
+    # app/controllers/all_search_result_ids_controller.rb
+    #
+    # Don't show this tool on e.g. the collection results page or the featured topics page.
+    config.add_results_collection_tool(
+      :all_search_results_add_to_cart,
+      opts = {
+        if: proc { |controller| controller.can?(:update, Kithe::Model) && controller.class.name == 'CatalogController' }
+      }
+    )
path: app/controllers/catalog_controller.rb
user: login: jrochkind
body: Oh my, this is a part of Blacklight I have literally _never_ seen before! Good job figuring it out, that was some real code spelunking! 

I hope it doens't end up fragile towards future BL upgrades... I suppose we will just have to deal with it if it does. 
created_at: 2025-03-26T22:50:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,3 @@
+<div id="add-results-to-cart" class="add-results-to-cart">
+  <%= link_to 'ADD RESULTS TO CART', request.params.merge(action: "add_to_cart", controller: 'all_search_result_ids'), class:"btn btn-outline-secondary" %>
+</div>
path: app/views/catalog/_all_search_results_add_to_cart.html.erb
user: login: jrochkind
body: Should we use a ViewComponent instead of a partial?  Or wait, does it NEED to be a partial to work with the Blacklight API you found? Woah. 

We better have really good tests for this stuff, I don't trust Blacklight at all! 
created_at: 2025-03-26T22:51:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,3 @@
+<div id="add-results-to-cart" class="add-results-to-cart">
+  <%= link_to 'ADD RESULTS TO CART', request.params.merge(action: "add_to_cart", controller: 'all_search_result_ids'), class:"btn btn-outline-secondary" %>
+</div>
path: app/views/catalog/_all_search_results_add_to_cart.html.erb
user: login: eddierubeiz
body: I believe add_results_collection_tool will only take a symbol that is assumed to be the name of a partial. I did a bit of digging around but was unable to get it to accept a partial. 
created_at: 2025-03-28T16:31:35Z
author_association: CONTRIBUTOR

PR comments:
user: login: jrochkind
created_at: 2025-03-20T18:18:32Z
body: Nice start!

In our actual implementation, I think we'll definitely want to NOT send the IDs back to the front-end. There could of course be thousands of them -- there's no need to take that trip sending them to the frontend and then having the frontend send em BACK again with a "save to cart" directive. 

Instead the action `all_search_result_ids` can be changed to actually `save_search_to_cart` or something -- do the same fetch you are doing already (the hard part) then instead of returning hte IDs in json, actually do the work in the Rails action to save them to cart (a back-end DB operation) and then either just redirect the front-end to reload, or tell the front-end how many things are in cart now for front-end to update it's cart count without a reload. 

(the former just reloading the page seems easier to me and sufficient -- so maybe don't even make it a JS fetch, make it an actual form submit to the backend, which returns a redirect)
author_association: CONTRIBUTOR
user: login: jrochkind
created_at: 2025-03-26T22:52:13Z
body: Well done! Tricky stuff. Just a few things I'm wondering about. 
author_association: CONTRIBUTOR
user: login: jrochkind
created_at: 2025-03-31T15:38:42Z
body: nicely done!
author_association: CONTRIBUTOR