Repository: apache/solr
Pull request #3262: SOLR-10998: Obey 'Accept' header in v2 APIs
Author: login: gerlowskija
Labels: name: tests
name: cat:api
Created: 2025-03-13T11:25:52Z | Merged: 2025-03-21T13:20:28Z

'wt' takes precedence if specified for now, if not provided Solr (through our use of Jersey) will pick a response format in keeping with the specified 'Accept' header.

JSON responses remains the "default" if neither 'wt' nor an "Accept" header are specified.

https://issues.apache.org/jira/browse/SOLR-10998



# Description

The HTTP spec defines an ["Accept" header](https://httpwg.org/specs/rfc7231.html#header.accept) that users can provide to specify the response format (or formats, plural) that they're willing to "accept" in a response.  This is part of the HTTP specs support for "content negotiation" more generally.

Solr today doesn't support this, and opts to have users specify their response format in a "wt" (i.e. "writer type") query-parameter.  We should obey "Accept" headers, when they are provided.
 
# Solution

This PR modifies Solr to obey "Accept" headers on v2 API requests, unless an overriding "wt" parameter is also provided.  Jersey (which is used for v2 requests) actually obeys the "Accept" header out of the box, but a bug in our `MediaTypeOverridingFilter` (which implements the 'wt' override) was causing the value of "Accept" to be ignored altogether.

With this change, the order of precedence for v2 APIs is: `wt` parameter, then `Accept` header, and lastly "JSON" is used as a fallback/default.

# Tests

See test cases in V2ApiIntegrationTest

# Checklist

Please review the following and check all that apply:

- [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
- [x] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew check`.
- [x] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)



Diff:
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 66adcaf6291e..812ab8243a77 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -195,6 +195,9 @@ Improvements
 * SOLR-17607: SolrJ CloudSolrClient configured with HTTP URLs will no longer eagerly connect to
   anything. (David Smiley)
 
+* SOLR-10998: v2 APIs now obey the "Accept" request header for content-negotiation if 'wt' is unspecified.  JSON is still used as a default when
+  neither 'Accept' or 'wt' are specified. (Jason Gerlowski)
+
 Optimizations
 ---------------------
 * SOLR-17578: Remove ZkController internal core supplier, for slightly faster reconnection after Zookeeper session loss. (Pierre Salagnac)
diff --git a/solr/core/src/java/org/apache/solr/jersey/JerseyApplications.java b/solr/core/src/java/org/apache/solr/jersey/JerseyApplications.java
index 6bb4b07c4397..3ac1ef79600a 100644
--- a/solr/core/src/java/org/apache/solr/jersey/JerseyApplications.java
+++ b/solr/core/src/java/org/apache/solr/jersey/JerseyApplications.java
@@ -43,12 +43,12 @@ public CoreContainerApp() {
 
       // Request and response serialization/deserialization
       // TODO: could these be singletons to save per-request object creations?
-      register(MessageBodyWriters.JavabinMessageBodyWriter.class);
-      register(MessageBodyWriters.XmlMessageBodyWriter.class);
-      register(MessageBodyWriters.CsvMessageBodyWriter.class);
-      register(MessageBodyWriters.RawMessageBodyWriter.class);
-      register(JacksonJsonProvider.class, 5);
-      register(MessageBodyReaders.CachingJsonMessageBodyReader.class, 10);
+      register(JacksonJsonProvider.class, 1);
+      register(MessageBodyWriters.JavabinMessageBodyWriter.class, 5);
+      register(MessageBodyWriters.XmlMessageBodyWriter.class, 5);
+      register(MessageBodyWriters.CsvMessageBodyWriter.class, 5);
+      register(MessageBodyWriters.RawMessageBodyWriter.class, 5);
+      register(MessageBodyReaders.CachingJsonMessageBodyReader.class, 2);
       register(SolrJacksonMapper.class);
 
       // Request lifecycle logic
diff --git a/solr/core/src/java/org/apache/solr/jersey/MediaTypeOverridingFilter.java b/solr/core/src/java/org/apache/solr/jersey/MediaTypeOverridingFilter.java
index 44c08bff03ef..cc5c4f6ca800 100644
--- a/solr/core/src/java/org/apache/solr/jersey/MediaTypeOverridingFilter.java
+++ b/solr/core/src/java/org/apache/solr/jersey/MediaTypeOverridingFilter.java
@@ -17,7 +17,10 @@
 
 package org.apache.solr.jersey;
 
+import static jakarta.ws.rs.core.HttpHeaders.ACCEPT;
 import static jakarta.ws.rs.core.HttpHeaders.CONTENT_TYPE;
+import static jakarta.ws.rs.core.MediaType.APPLICATION_JSON;
+import static org.apache.solr.common.params.CommonParams.WT;
 import static org.apache.solr.jersey.RequestContextKeys.SOLR_QUERY_REQUEST;
 
 import jakarta.ws.rs.container.ContainerRequestContext;
@@ -25,19 +28,23 @@
 import jakarta.ws.rs.container.ContainerResponseFilter;
 import jakarta.ws.rs.container.ResourceInfo;
 import jakarta.ws.rs.core.Context;
-import jakarta.ws.rs.core.MediaType;
 import java.io.IOException;
+import java.lang.invoke.MethodHandles;
 import java.util.List;
 import org.apache.solr.api.JerseyResource;
 import org.apache.solr.handler.admin.ZookeeperRead;
 import org.apache.solr.handler.api.V2ApiUtils;
 import org.apache.solr.request.SolrQueryRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 // TODO Deprecate or remove support for the 'wt' parameter in the v2 APIs in favor of the more
 //  HTTP-compliant 'Accept' header
 /** Overrides the content-type of the response based on an optional user-provided 'wt' parameter */
 public class MediaTypeOverridingFilter implements ContainerResponseFilter {
 
+  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
   private static final List<Class<? extends JerseyResource>> EXEMPTED_RESOURCES =
       List.of(ZookeeperRead.class);
 
@@ -65,9 +72,16 @@ public void filter(
         (SolrQueryRequest) requestContext.getProperty(SOLR_QUERY_REQUEST);
     // TODO Is it valid for SQRequest to be null?
     final var params = (solrQueryRequest != null) ? solrQueryRequest.getParams() : null;
-    final String mediaType = V2ApiUtils.getMediaTypeFromWtParam(params, MediaType.APPLICATION_JSON);
-    if (mediaType != null) {
-      responseContext.getHeaders().putSingle(CONTENT_TYPE, mediaType);
+    if (params != null && params.get(WT) != null) { // Override for 'wt'
+      final String mediaType = V2ApiUtils.getMediaTypeFromWtParam(params, null);
+      if (mediaType != null) {
+        responseContext.getHeaders().putSingle(CONTENT_TYPE, mediaType);
+      }
+    } else if (!requestContext.getHeaders().containsKey(ACCEPT)
+        || "*/*"
+            .equals(requestContext.getHeaderString(ACCEPT))) { // Override default response to json
+      responseContext.getHeaders().putSingle(CONTENT_TYPE, APPLICATION_JSON);
     }
+    // Else, obey the user-provided 'Accept' header
   }
 }
diff --git a/solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java b/solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java
index a7f4606e1609..70ac4e89d8c9 100644
--- a/solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java
+++ b/solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java
@@ -22,11 +22,16 @@
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.client.utils.URIBuilder;
 import org.apache.solr.client.solrj.ResponseParser;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrRequest;
 import org.apache.solr.client.solrj.SolrServerException;
 import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.client.solrj.impl.CloudLegacySolrClient;
 import org.apache.solr.client.solrj.impl.CloudSolrClient;
 import org.apache.solr.client.solrj.impl.JsonMapResponseParser;
 import org.apache.solr.client.solrj.impl.NoOpResponseParser;
@@ -139,6 +144,52 @@ public void testWTParam() throws Exception {
     assertEquals(respString, 0, Utils.getObjectByPath(resp, true, "/responseHeader/status"));
   }
 
+  @Test
+  public void testObeysWtParameterWhenProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.setURI(
+        new URIBuilder(listCollRequest.getURI()).addParameter("wt", "xml").build());
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testObeysAcceptHeaderWhenWtParamNotProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.addHeader("Accept", "application/xml");
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testRespondsWithJsonWhenWtAndAcceptAreMissing() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/json", response.getFirstHeader("Content-type").getValue());
+  }
+
+  private HttpClient getRawClient() {
+    return ((CloudLegacySolrClient) cluster.getSolrClient()).getHttpClient();
+  }
+
+  private HttpRequestBase getListCollectionsRequest() {
+    final var v2BaseUrl = cluster.getJettySolrRunner(0).getBaseURLV2().toString();
+    final var listCollUrl = v2BaseUrl + "/collections";
+    return new HttpGet(listCollUrl);
+  }
+
   @Test
   public void testSingleWarning() throws Exception {
     NamedList<?> resp =


Reviews:
user: login: dsmiley
body: A forward-looking question: In SolrJ, assuming "V2", how might we compose an Accept header for multiple formats, and then on the receiving end, choose the right ResponseParser for the final format chosen by the server?

Even assuming Jackson, I see us supporting `application/json`, `application/cbor`, and could add "smile" to the list, as all 3 are supported with small changes thanks to Jackson's well-designed API.  

Our current abstractions are a bit rigid, with the ResponseParser that must supply a "wt" and that which consumes an inputStream without knowledge of the actual mime type in the response header, as it's not passed to the ResponseParser.  Maybe we basically just need to change ResponseParser, and make the SolrClients get content types from the RP (to put in "Accept") instead of the "wt" param?  Sounds pretty straight-forward actually.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T20:18:52Z
user: login: gerlowskija
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T19:07:41Z
user: login: dsmiley
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T19:33:09Z

Review comments:
diff_hunk: @@ -139,6 +144,52 @@ public void testWTParam() throws Exception {
     assertEquals(respString, 0, Utils.getObjectByPath(resp, true, "/responseHeader/status"));
   }
 
+  @Test
+  public void testObeysWtParameterWhenProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.setURI(
+        new URIBuilder(listCollRequest.getURI()).addParameter("wt", "xml").build());
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testObeysAcceptHeaderWhenWtParamNotProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.addHeader("Accept", "application/xml");
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testRespondsWithJsonWhenWtAndAcceptAreMissing() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/json", response.getFirstHeader("Content-type").getValue());
+  }
+
+  private HttpClient getRawClient() {
+    return ((CloudLegacySolrClient) cluster.getSolrClient()).getHttpClient();
+  }
path: solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java
user: login: dsmiley
body: We're trying to get away from Apache HttpClient.  Can you please try Jetty HttpClient instead?
created_at: 2025-03-17T20:10:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,6 +144,52 @@ public void testWTParam() throws Exception {
     assertEquals(respString, 0, Utils.getObjectByPath(resp, true, "/responseHeader/status"));
   }
 
+  @Test
+  public void testObeysWtParameterWhenProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.setURI(
+        new URIBuilder(listCollRequest.getURI()).addParameter("wt", "xml").build());
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testObeysAcceptHeaderWhenWtParamNotProvided() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+    listCollRequest.addHeader("Accept", "application/xml");
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/xml", response.getFirstHeader("Content-type").getValue());
+  }
+
+  @Test
+  public void testRespondsWithJsonWhenWtAndAcceptAreMissing() throws Exception {
+    final var httpClient = getRawClient();
+    final var listCollRequest = getListCollectionsRequest();
+
+    final var response = httpClient.execute(listCollRequest);
+
+    assertEquals(200, response.getStatusLine().getStatusCode());
+    assertEquals("application/json", response.getFirstHeader("Content-type").getValue());
+  }
+
+  private HttpClient getRawClient() {
+    return ((CloudLegacySolrClient) cluster.getSolrClient()).getHttpClient();
+  }
path: solr/core/src/test/org/apache/solr/handler/V2ApiIntegrationTest.java
user: login: gerlowskija
body: What's the best way to do that?  Is there one?

Per [the Slack discussion here](https://the-asf.slack.com/archives/CEKUCUNE9/p1741870697981609), there's not yet a general utility for creating a Jetty client.  And adding a well-thought-out version of that is conceptually unrelated and beyond the scope of this PR.

I can throw together a client-creation snippet just for this test, but that feels a little hacky and the sort of thing that'd be likely to run afoul of our test randomization and cause flaky failures down the line whenever SSL or whatever variable is randomly toggled.
created_at: 2025-03-20T19:07:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: epugh
created_at: 2025-03-13T11:32:38Z
body: This is some nice alignment with normal web expectations..    
author_association: CONTRIBUTOR
user: login: dsmiley
created_at: 2025-03-15T20:25:22Z
body: I love what you wrote -- what this PR accomplishes.  It's a mystery to me **how** this PR accomplishes that goal.  A test would be nice.
author_association: CONTRIBUTOR
user: login: gerlowskija
created_at: 2025-03-17T14:08:36Z
body: > a test would be nice

Tests added in `V2ApiIntegrationTest`

> It's a mystery to me how this PR accomplishes that goal.

Really it doesn't.  Jersey will parse the "Accept" header out of the box, and choose a "MessageBodyWriter" (akin to our ResponseWriter") based on that.  But I'd hidden this support in the past by creating a "response filter" (`MediaTypeOverridingFilter`) to key off of only the 'wt' parameter.

So what this PR is really doing is tweaking `MediaTypeOverridingFilter` to be slightly less aggressive.  It still obeys "wt" when provided, and makes sure that JSON is the default otherwise.  But now it only leans on that "JSON" default when there's no  "wt" AND there's no "Accept" header.  Jersey is still doing the "Accept" parsing and support and all that - we're just getting out of its way in the cases where we don't want our own custom behavior (e.g. "wt")
author_association: CONTRIBUTOR
user: login: gerlowskija
created_at: 2025-03-21T13:20:02Z
body: > In SolrJ, assuming "V2", how might we compose an Accept header for multiple formats, and then on the receiving end, choose the right ResponseParser for the final format chosen by the server?

Hmm... `SolrRequest` already has a ResponseParser getter/setter pair that is used by the clients to override the default parser on a per-request basis.  Ditto for request-headers - `SolrRequest` has methods to set/get those as well.  So it seems like we've got a lot of the puzzle pieces and plumbing already.

The one thing that's missing I think is some sort of "CompoundResponseParser" - something that supports a number of content types via delegation and then chooses one at response-parsing time based on the "Content-type" header in the response.
author_association: CONTRIBUTOR