SLO/Search

Status: draft
== Organizational ==
=== Service ===
The various search components allow full-text search of Wiki articles as well as article title autocompletion and MediaSearch
=== Teams ===
Search Platform, a member of the Data Platforms working group, owns these services. The operational side is specifically owned by Data Platforms SRE.
== Architectural ==
=== Environmental dependencies ===
CirrusSearch is a MediaWiki extension that handles cross-communication between MediaWiki and Search Elasticsearch clusters.
Our search Elasticsearch clusters are ultimately responsible for serving the search queries forwarded by MediaWiki.
SLIs based around time to render clientside are dependent on the UI; performance regressions in the UI can lead to higher p95 times.
=== Service dependencies ===
MediaWiki depends on CirrusSearch being operational.
CirrusSearch depends on the underlying Elasticsearch clusters being healthy and responsive.
== Client-facing ==
=== Clients ===
Users submit queries on a given wiki. That wiki's MediaWiki processes generate Elasticsearch queries and, having received the result, display them through the UI.
=== Request Classes ===
There are various query types: full text, autocomplete, more_like.
There are also mediasearch requests as well as search preview requests.
== Service Level Indicators (SLIs) ==
All Search SLIs use a Latency SLI, defined as % of all requests that complete within X ms.
Full text p95 latency: <4000 ms
This represents the p95 latency of the time to render the results of a "normal" (full text) search query made via Special:Search (here's enwiki for example)
Autocomplete p95 latency: <600 ms
This represents the p95 latency of the time to render the results of an autocomplete query on the user's screen, also known as a CompSuggest query. This query type is used by Mediawiki and the Wikipedia UI to surface articles whose titles fuzzy match the text of the CompSuggest query.
Search preview p95 latency: <10000 ms
This represents the p95 latency of the time to render a "flyout" UI element corresponding to a search preview request.
MediaSearch p95 latency: <10000 ms
This represents the p95 latency of MediaSearch queries.
(TODO)
Availability: Some notion of the actual availability of Elasticsearch to respond to MediaWiki (the CirrusSearch extension). We can likely use Cirrus failed events (graph) to capture errors generated by lack of service availability as opposed to rejections which correlate with users sending improper queries.
== Operational ==
=== Monitoring ===
We get various graphite-based metrics that are plumbed through MediaWiki. These are the major metrics used as SLIs for this service.
Eventually we'll transition to prometheus-based metrics once MediaWiki has been cut over from Graphite to Prometheus.
=== Troubleshooting ===
Some issues, like the backend Elasticsearch clusters being temporarily overloaded, are quite simple. Other issues can require more extensive troubleshooting.
=== Deployment ===
The CirrusSearch extension for MediaWiki handles all the logic of communication between mediawiki and Elasticsearch.
Additionally, we have 2 "cirrus clusters", in eqiad and codfw respectively, each of which is composed of 3 different Elasticsearch clusters (the multiple ES clusters are for performance reasons; having all of our indices/shards in a single cluster results in poor performance due to Elasticsearch performance being inversely related to total number of shards (after some threshold has been passed).
== Service Level Objectives ==
=== Realistic targets ===
Realistic (perhaps overly lenient; we can tighten the non-mediasearch SLOs in future when we have more data) targets:
Full text p95 latency: <4000 ms 95% of the time
Autocomplete p95 latency: <600 ms 95% of the time
Search preview p95 latency: <10000 ms 95% of the time
MediaSearch p95 latency: <10000 ms 95% of the time
=== Ideal targets ===
The ideal targets for the SLOs are as follows:
Full text p95 latency: <4000 ms 99.9% of the time
Autocomplete p95 latency: <600 ms 99.9% of the time
Search preview p95 latency: <10000 ms 99.9% of the time
MediaSearch p95 latency: <10000 ms 95% of the time
=== Reconciliation ===
Currently we're using the realistic targets because our metrics have only been added in the last several months; once we have more data we should circle back and look if we want to tighten our SLOs accordingly.
Category:Published SLOs