Repository: internetarchive/openlibrary

Issue:
Decorate Availability API Requests to pass x-client-id
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->

When we hit archive.org ES directly from Open Library, we lookup the client-id and pass along w/ our request. We **don't** do this when we make requests for Bulk Availability to `http://archive.org/services/availability?identifier=`. This issue aims to address that by setting a `x-client-id` so ES can throttle Bulk Availability API where/when needed.

https://github.com/internetarchive/openlibrary/blob/b9ecda1075011eea8c12832d985b7d01b0041c0b/openlibrary/core/lending.py#L329

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@ximm @bfalling @cdrini @jimchamp 




Discussion:
I believe this is mostly closed by #6201? 
We could add the annotation here: https://git.archive.org/ia/petabox/-/merge_requests/3118/commits
Currently https://archive.org/services/availability?identifier=finnikinofrock0000marc is in petabox so it makes a direct request to `Search` API (as opposed to making a network request) so I think it's exempt from needed a `x-client-id` unless we there's a way to pass in a client-id into the `Search` function call itself (to pass-through rate limiting capabilities).

Deferring to @ximm to re-open but assuming we're OK for now (with just adding headers to openlibrary.org)
`preferred_client_id`,`preferred_client_ip`, and `application_id` may all be set with parameters in both the `Search.inc` constructor, and, via HTTP endpoints as CGI arguments, in addition to the latter accepting request headers.

If this `services` endpoint is called directly from clients themselves, `client_id` and `client_ip` will be inferred from the request itself (and/or any authenticated user credentials).

If the endpoint is called via HTTP from within e.g. OL infrastructure, _on behalf of clients who are themselves addressing a different OL endpoint/service_, these arguments should be populated with values harvested at the time of the initial request, then passed into the service endpoint [presumably through a new affordance to optionally provide them].

If on the other hand the endpoint is called from OL in a recurring way (e.g. via `cron` job) to do work for OL's own internal use, we will want to add a `service` parameter to `AvailabilityService::execute_search()`. requesting `metadata__pinned` or `metadata__unlimited` to be exempt from rate-limiting.

The model:
• requests _originating with third-party clients_ (of Archive.org directly, or Open Library and other projects which expose endpoints for their own clients) should pass through a characterization of the client (by IP; or if authenticated, by iD), which we key off of to perform session-concurrency rate-limiting
• requests made by services like Open Library on their own behalf may (still) be made exempt from rate-limiting; but that tier of service _must now be requested_ by specifying an appropriate service in the request parameters;
• access to un-rate-limited service is granted for "qualified" requests; requests originating inside our allowed-host range qualify (may require authentication sometime in the future)

These requests should all include an `application_id` parameter (or header), to provide analytics visibility.

In sum, if the availability service is 
• *directly queried by users/partners*, it only needs an `application_id` parameter (I can make this change myself);
• *used to satisfy user/partner queries _to_ OL*, we should add a mechanism to harvest and pass through to the Search constructor client characterization parameters (and add `application_id`)
• *used by OL for its own business logic*, we should add a `service` parameter to request un-rate-limited service (and add an `application_id`) 

I am available to make required changes to `AvailabilityService.inc`; but collaboration would be needed for the case where this service is used internally to help satisfy a reply to a third-party which arrived at e.g. an OL endpoint.

Add headers to (like https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/lending.py#L308-L312)

https://github.com/internetarchive/openlibrary/blob/27ec35622e6bcb549b0ad5edab507131a76ef26d/openlibrary/solr/data_provider.py#L141-L151 

Hi @mekarpeles this issue is reopened. Do you have any thoughts on that?
Ftr I would recommend adding:

- IAS3 authorization headers for an OL account, with an associated archive.org email address;
- an `x-application-id` header with the value `openlibrary` (_or_ a request parameter `application_id` with that value); abd
- a request parameter `service` with the value `metadata__pinned` (_or_ value `metadata__unlimited`)

The pinned metadata service always routes to one of the two clusters, no matter what IP address the request is seen from (load balancing is a function of IP). This provides consistency across multiple sessions/requests. (The unlimited metadata service is a new option which is load balanced across the two clusters as is our default, but, it is exempt from rate limiting. For cases like this where IA metadata search is serving as "ground truth" in some sense, the "pinned" service is the canonical solution.)
Update: reviewing traffic I see that it will be important to add the `service` parameter in https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/lending.py#L280-L288

`metadata__pinned` may be used but I would recommend `metadata__unlimited`

I believe these are the internal queries populating carousels; those should be exempt from rate limiting.

(This pair can just be added to the existing parameters list, and probably should be also added to the debugging output just below)

(bumping)
On Monday I'll add

* [ ] s3 keys to headers
* [ ] `service=metadata__unlimited` to our ES queries
* [x] `x-application-id` header with the value openlibrary

Should we be added `metadata__unlimited` to Bulk Availability API as well? Or no since that uses the `Search` php methods directly?
Great! Thanks Mek.

The model here should be:
- `unlimited` (or `pinned`) service is appropriate for **internal** use of search, e.g. to populate carousels, or for internal maintenance tasks, etc.–any place data is needed for OL for itself.
- it is _not_ appropriate, in cases where searches are done synchronously _on behalf_ of third-party clients, be they human users, or partner backends.

IIRC the Bulk Availability path does not "precalculate" values, I think it answers third-party client questions? Assuming so it should _not_ be routed to backends which don't rate-limit.

(The primary motivation for this work was to address a long-standing gap in our ability to insulate the search backends from third-party traffic, that any "pass through" traffic was getting "IP washed" for the perspectives of rate limiting, because we had a blanket exemption for rate-limiting for any request originating in our IP ranges.

The OL case is the most complex because the uses of search include both moments where exemption is appropriate; and, cases where rate limiting guardrails should apply.)

So! For each given case, the rule of thumb should be: if this is passing through traffic, it should **not** be routed to `unlimited` or `pinned`; it should be allowed to default to the rate-limited path. 
1. Is throttling turned on? @ximm 
2. If OL passes along IP (an anonymized one from nginx), does Archive.org compute the same anonymomized IP?
1. Throttling has _not_ been turned on yet. I was waiting for a dull moment, and aware of a few corners, one in TVS which has been resolved; and at least one with OL, IIRC there is a case I'm not sure about for partner users of the Bulk Availability API.
2. When qualified requests specify a `preferred_client_id` that value is used; otherwise, `client_id` is set to the userid if a user is authenticated; or (as a last resort), the client IP known to the PHP as `$_SERVER['REMOTE_ADDR']`,
Followup: checking if `get_available()` in `lending.py` needs to be exempt from session concurrency rate limiting, if so, https://github.com/internetarchive/openlibrary/blob/master/openlibrary/core/lending.py#L280-L301 should be extended with an additional argument, `service=metadata__unlimited`.

Whether this is appropriate is a function of usage pattern.

The guideline is:
- if this function is called _on behalf of partners_ in a synchronous fashion, i.e. it is "passing on load," it would typically be appropriate to leave it as-is, but set a `preferred_client_id` argument unique to the actual requesting client. 
- if this function is used internally to _prepare_ data for client consumption, i.e. it builds internal data caches, it would be appropriate to add a `service=metadata__unlimited` argument, to exempt it from rate-limiting.

As it is, this path ends up always using `ol-internal` for client identification, which means it goes into "one bin" for purposes of rate limiting; and because the unlimited service is not currently requested, this means those binned requests _are_ subject to rate limiting.

Under current use patterns I see about ~700-800~ requests [EDIT: that was wrong, more like 70,000 per day... about 1.1/s..] I believe are from this path in a 24 hour period. Rate limiting only applies to multiple *concurrent* requests, so I believe the exposure to adverse latency or timeouts is ~minimal~ real.

~But,~ So we should ~ideally~ definitely adopt one or the other of these additions. In either case the consequence will be to almost certainly prevent rate-limiting under any foreseen normal use.

Resolving patch:
diff --git a/openlibrary/core/lending.py b/openlibrary/core/lending.py
--- a/openlibrary/core/lending.py
+++ b/openlibrary/core/lending.py
@@ -116,6 +116,7 @@ def compose_ia_url(
     _type: Literal['authors', 'subjects'] = None,
     sorts=None,
     advanced=True,
+    rate_limit_exempt=True,
 ) -> Optional[str]:
     """This needs to be exposed by a generalized API endpoint within
     plugins/api/browse which lets lazy-load more items for
@@ -196,6 +197,8 @@ def compose_ia_url(
         ('page', page),
         ('output', 'json'),
     ]
+    if rate_limit_exempt:
+        params.append(('service', 'metadata__unlimited'))
     if not sorts or not isinstance(sorts, list):
         sorts = ['']
     for sort in sorts:


Test patch: