Repository: ministryofjustice/laa-court-data-ui
Pull request #2849: ACD-611: Remove PII from CDAPI request logs
Author: login: patrick-laa
Created: 2025-03-13T11:36:28Z | Merged: 2025-03-20T15:54:43Z

#### What
Users of VCD conduct searches of personally identifiable information (PII), including the names,  dates of birth and NI numbers of defendants. 

At present, when a search is conducted using CDAPI, VCD will put search terms in the logs that get sent to OpenSearch, meaning that OpenSearch contains, amongst its logs, a bunch of PII that it shouldn’t. 

#### Ticket

[Remove PII from logs in VCD and CDAPI](https://dsdmoj.atlassian.net/browse/ACD-611)

#### Why
We have responsibilities governing the storage and security of that information, so need to limit the number of places that information is stored.

#### How
Intercept log messages generated by ActiveResource, parse the request URI, and filter out any query params that are blacklisted Rails filter parameters (ensuring the appropriate terms are indeed blacklisted)



Diff:
diff --git a/config/initializers/active_resource_filtered_logging.rb b/config/initializers/active_resource_filtered_logging.rb
new file mode 100644
index 000000000..37929c978
--- /dev/null
+++ b/config/initializers/active_resource_filtered_logging.rb
@@ -0,0 +1,39 @@
+require 'active_resource/base'
+require 'active_resource/log_subscriber'
+
+# This monkey-patch exists because CdApi uses ActiveResource which, by default, logs
+# all query params of outgoing GET requests. However, our requests include PII that we
+# don't want ending up in the logs (e.g. defendant date of birth). So we need to intercept
+# the log commands and scrub out sensitive query params
+# (based on a gist found at https://gist.github.com/vraravam/1319094)
+module ActiveResource
+  class LogSubscriber < ActiveSupport::LogSubscriber
+    def request_with_filtered_logging(event)
+      request_uri = event.payload[:request_uri]
+      start_of_query_string = request_uri.index("?")
+
+      return request_without_filtered_logging(event) unless start_of_query_string
+
+      request_uri_without_query_string = request_uri[0, start_of_query_string]
+      query_string = request_uri[(start_of_query_string + 1)...]
+
+      event.payload[:request_uri] = "#{request_uri_without_query_string}?#{filter_query_string(query_string)}"
+      request_without_filtered_logging(event)
+    end
+
+    alias request_without_filtered_logging request
+    alias request request_with_filtered_logging
+
+    private
+
+    def filter_query_string(query_string)
+      query_hash = Rack::Utils.parse_nested_query(query_string).with_indifferent_access
+      Rails.configuration.filter_parameters.each do |param|
+        query_hash[param] = "FILTERED" if query_hash.key?(param)
+      end
+      query_hash.to_query
+    end
+  end
+end
+
+ActiveResource::Base.logger = Rails.logger
diff --git a/config/initializers/filter_parameter_logging.rb b/config/initializers/filter_parameter_logging.rb
index 7f9cf478d..1dc376d87 100644
--- a/config/initializers/filter_parameter_logging.rb
+++ b/config/initializers/filter_parameter_logging.rb
@@ -4,5 +4,5 @@
 # Use this to limit dissemination of sensitive information.
 # See the ActiveSupport::ParameterFilter documentation for supported notations and behaviors.
 Rails.application.config.filter_parameters += %i[
-  passw password email secret token _key crypt salt certificate otp ssn
+  passw password email secret token _key crypt salt certificate otp ssn dob name term nino
 ]
diff --git a/spec/lib/active_resource/log_subscriber_spec.rb b/spec/lib/active_resource/log_subscriber_spec.rb
new file mode 100644
index 000000000..f19bd66d0
--- /dev/null
+++ b/spec/lib/active_resource/log_subscriber_spec.rb
@@ -0,0 +1,39 @@
+class ActiveResourceLogSubscriberTestImplementation < ActiveResource::LogSubscriber
+  attr_reader :output
+
+  def request_without_filtered_logging(input)
+    @output = input
+  end
+end
+
+RSpec.describe ActiveResource::LogSubscriber do
+  subject(:filtered_log) do
+    ActiveResourceLogSubscriberTestImplementation.new.tap { _1.request_with_filtered_logging(event) }.output
+  end
+
+  let(:event) { instance_double(ActiveSupport::Notifications::Event, payload: { request_uri: input }) }
+
+  context "when the query string contains a blacklisted attribute" do
+    let(:input) { "https://example.com/some/endpoint?dob=private-data" }
+
+    it "filters out private data" do
+      expect(filtered_log.payload[:request_uri]).to eq "https://example.com/some/endpoint?dob=FILTERED"
+    end
+  end
+
+  context "when there is no query string" do
+    let(:input) { "https://example.com/some/endpoint" }
+
+    it "leaves the URI untouched" do
+      expect(filtered_log.payload[:request_uri]).to eq "https://example.com/some/endpoint"
+    end
+  end
+
+  context "when the query string has no exact match" do
+    let(:input) { "https://example.com/some/endpoint?a_dob=something&dob-2=something-else" }
+
+    it "leaves the URI untouched" do
+      expect(filtered_log.payload[:request_uri]).to eq "https://example.com/some/endpoint?a_dob=something&dob-2=something-else"
+    end
+  end
+end


Reviews:
user: login: alexdesi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T12:10:09Z
user: login: alexdesi
body: it's great you added tests,  
it looks good! ✅ 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-13T12:15:55Z

Review comments:
diff_hunk: @@ -0,0 +1,39 @@
+class ActiveResourceLogSubscriberTestImplementation < ActiveResource::LogSubscriber
+  attr_reader :output
+
+  def request_without_filtered_logging(input)
+    @output = input
+  end
+end
+
+RSpec.describe ActiveResource::LogSubscriber do
+  subject(:filtered_log) do
+    ActiveResourceLogSubscriberTestImplementation.new.tap { _1.request_with_filtered_logging(event) }.output
path: spec/lib/active_resource/log_subscriber_spec.rb
user: login: alexdesi
body: I am not a fan of `_1`, not very readable, but it's ok for now.
Ruby 3.4 introduces the `it` block param (like kotlin), which is pretty cool :)

created_at: 2025-03-13T12:10:09Z
author_association: COLLABORATOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-20T15:45:35Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=ministryofjustice_laa-court-data-ui&pullRequest=2849) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=ministryofjustice_laa-court-data-ui&pullRequest=2849&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=ministryofjustice_laa-court-data-ui&pullRequest=2849&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=ministryofjustice_laa-court-data-ui&pullRequest=2849&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=ministryofjustice_laa-court-data-ui&pullRequest=2849&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=ministryofjustice_laa-court-data-ui&pullRequest=2849&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=ministryofjustice_laa-court-data-ui&pullRequest=2849)


author_association: NONE