nyuuzyou/wikis huggingface dataset:
Search/ElasticSearch/LoadTesting

Load testing of the elasticsearch clusters is done by recording real production GET requests during peak hours and playing it back against the hot-spare cluster at accelerated rates. The performance of elasticsearch depends heavily on how much of it's data is available in the kernel disk cache, and how much has to be pulled up from disk in real time. Having both the variety of query types and the variety of real terms provided by users ensures we are testing a realistic load.
== Overview ==
goreplay is used to record production traffic on the currently live servers for at least one hour of peak usage. The recorded data files are copied from the live cluster to the spare cluster, and then replayed at normal and accelerated rates. The most recent load testing run was run with gor_0.16.1_x86 and the commands below are based on that version.
== Recording traffic ==
On all servers currently running production traffic use the following command. This needs to run for at least 60 minutes to capture enough traffic that we can play it back at 2x speed and still have a sufficiently long test to have confidence in the results.
sudo ./goreplay --input-raw :9200 --http-allow-method GET --output-file requests.log --exit-after 60m
== Transfering recorded files ==
Gor will create files requests_0.log, requests_1.log, etc. In previous runs these files were transferred between elasticsearch servers using netcat. Since then the firewalls have been tightened up and this is no longer an option. Someone in SRE with agent forwarding capabilities can likely use scp to accomplish the same task.
== Replaying recorded files ==
When replaying recorded files there are two important parameters to tune: The number of concurrent requests, and the speed of replay. Without limiting the number of concurrent requests it is possible to bring down the entire cluster. CirrusSearch limits this using the PoolCounter, goreplay uses --output-http-workers to limit on a per-instance basis. 
./gor --input-file ./requests.gor'|150%' --input-file-repeat 3 --output-http http://search.svc.codfw.wmnet:9200 --stats --output-http-stats --output-http-workers 30
== Evaluating Performance ==
Both server load and latencies should be evaluated via the grafana dashboards.
TODO
== Scripts ==
Some scripts were hacked together during last load test (). Those are available on GitHub.
== Caveats ==
When we have done this before we had the same number of servers in both clusters, which meant we could do a 1-to-1 transfer of request logs between servers for replay. When the number of servers in the spare cluster is less than the number of servers in the cluster recorded from the playback needs to be sped up to account for the reduced number of servers. We make the assumption that LVS evenly distributes requests across the servers, so a simple ratio of server counts can be used to determine the speedup necessary.
Search/Elasticsearch Administration/Opensearch Plugin Deploy

== Opensearch Plugin Deploy ==
All steps necessary to deploy a new version of the WMF Opensearch Plugins Debian package. This documentation is targeted towards the Search Platform team, as it is their responsibility to deploy the See this Phab task for a list of what repos might need updating.
*
* https://gerrit.wikimedia.org/r/c/search/extra-analysis/+/1080352
* https://gerrit.wikimedia.org/r/c/search/highlighter/+/1080384
* https://gitlab.wikimedia.org/repos/search-platform/opensearch-analysis-stconvert
Plugin name 	Code Repository 	Artifact Repository
analysis-hebrew 	https://gitlab.wikimedia.org/repos/search-platform/opensearch-analysis-hebrew 	Gitlab
analysis-icu 	 	https://artifacts.opensearch.org
analysis-nori 	 	https://artifacts.opensearch.org
analysis-smartcn 	 	https://artifacts.opensearch.org
analysis-stconvert 	https://gitlab.wikimedia.org/repos/search-platform/opensearch-analysis-stconvert	
analysis-stempel 	 	https://artifacts.opensearch.org
analysis-ukrainian 	 	https://artifacts.opensearch.org
extra 	https://gerrit.wikimedia.org/r/c/search/extra/+/1080355 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-homoglyph	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-ukrainian	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-khmer 	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-slovak 	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-serbian 	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-esperanto	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-turkish 	 	https://mvnrepository.com/artifact/org.wikimedia.search
opensearch-extra-analysis-textify 	 	https://mvnrepository.com/artifact/org.wikimedia.search
ltr 	 	gitlab
repository-s3 	 	gitlab
 	 	
highlighter 	https://gerrit.wikimedia.org/r/q/project:search/highlighter 	
You can also check plugin_urls.lst in the plugins Debian package repo for a full list. If you're on a host with the packages installed, you can also do:
dpkg -L wmf-elasticsearch-search-plugins | awk -F/ '{print $6}' | sort | uniq
analysis-hebrew
analysis-icu
analysis-nori
analysis-smartcn
analysis-stconvert
analysis-stempel
analysis-ukrainian
changelog.Debian.gz
experimental-highlighter
extra
extra-analysis-esperanto
extra-analysis-homoglyph
extra-analysis-khmer
extra-analysis-serbian
extra-analysis-slovak
extra-analysis-textify
extra-analysis-turkish
extra-analysis-ukrainian
ltr
repository-s3
Search/Rebuilds

We'll use this to track rebuilds that are required.
* April 22
** Commons (aggressive splitting)
** All new wikis
*** Nik is monitoring. Slow progress because, well, they are big.
* April 24
** Group 0 (aggressive splitting, better analysis for some languages, auxiliary text)
*** Nik will do once (April 25) once rolling Elasticsearch upgrade is done
* April 29
** Group 1 (aggressive splitting, better analysis for some languages, auxiliary text)
* May 1
** Group 2 (aggressive splitting, better analysis for some languages, auxiliary text)
Search/S3 Plugin Enable/Beta Env

== S3 Plugin Test Environment (beta) ==
=== Hosts ===
[deployment_prep]
deployment-elastic09.deployment-prep.eqiad1.wikimedia.cloud
deployment-elastic10.deployment-prep.eqiad1.wikimedia.cloud
deployment-elastic11.deployment-prep.eqiad1.wikimedia.cloud
=== Minio ===
Minio is running in a container on deployment-elastic10.deployment-prep.eqiad1.wikimedia.cloud so we can have an S3 test environment.
Search/UDP Logger

== UDP Logger ==
There is a UDP logger running on searchidx1 that is collecting mesg's from each one of the search servers. All search queries are getting logged into the following format:
 db timestamp search
The udp logger was added in r51097 and r51324.
== Host ==
searchidx1
== Storage ==
Files are written to 
 /a/search/udplogger
=== Scripts ===
 (start/stop)logger.sh - start/stop the process
 udplogger.py - main worker for collection
=== Logs ===
All queries are written to searchqueries including public/private wiki's
== Additions Needed ==
* Public/Private filter - add a script to filter
* Log rotation/archiving - rainman_: now, ideally i think.. there would be another script that would at midnight: 1) run stoplogger.sh 2) rename searchqueries to searchqueries-CURRENTDATE 3) upload/move this file to some public location 4) run startlogger.sh
** Wrote a quick log splitter to break up the huge 40GB query file into day chunks. Threw away private searches and archived them into 7za format and will put them on download after getting approval . --Tomasz Finc 23:57, 31 July 2009 (UTC)
* Publish data to download.wikimedia.org
Category:Services
Category:Search
Search/articletopic

CirrusSearch supports searching Wikipedia main-namespace articles by article topic (provided by the language agnostic link-based article topic) using the articletopic: keyword. This requires loading LiftWing predictions into Elasticsearch. The rough data flow for this is:
:wiki edit -> changeprop -> LiftWing -> EventGate -> Cirrus Streaming Updater -> Elasticsearch
This is near real-time, typically within 10 minutes of an edit. This usually won't matter much since most edits won't change topic predictions drastically, but new articles become searchable after 10 minutes.
articletopic: uses its own set of keywords as the model provided topic labels are not search-expression-friendly; these can be found in ArticleTopicFeature.php.
== Details ==
* Whenever a page is edited, the EventBus extension (code) creates a mediawiki.page-change event through EventBus.
* changeprop configuration listens for those events and invokes LiftWing. 
* The LiftWing containers receive the requests from changeprop, run the prediction, and sends an event to mediawiki.page_outlink_topic_prediction_change.v1.
* The Cirrus Streaming Updater subscribes to the event stream in kafka. It merges edit-related updates like the article topic predictions into the initial edit event and updates the appropriate search indices. Any newly provided topic prediction overwrites the previous prediction. This doesn't do anything in the way of custom processing, topic predictions are stored for the page specified in the event.
* In Elasticsearch the data is stored under the Search/WeightedTags field with the classification.ores.articletopic tag family as a pretend word vector (e.g. if an article has a prediction of Linguistics -> 0.98, Literature -> 0.63, Elasticsearch will represent that with a field which has the word "Linguistics" 980 times and the word "Literature" 630 times). The articletopic: search keyword will do a text similarity search between this field and the topics provided. (field definition, search code)
 
== Management ==
When the model changes significantly the data has to be bulk-uploaded into Elasticsearch (since normally it's only updated for articles which are changed). This has not been done since the migration from ORES to liftwing and the change to streaming updates. Some method of bulk-importing predictions from monthly snapshots will be required.
In MediaWiki site configuration, the $wgCirrusSearchWMFExtraFeatures['weighted_tags']['build'] flag enables building the Elasticsearch index for the field, and $wgCirrusSearchWMFExtraFeatures['weighted_tags']['use'] enables the articletopic search keyword.
== See also ==
* phab:T240517 which was the epic for setting this up
* mw:ORES/Articletopic for the topic taxonomy (this is used for all wikis) and
* mw:Help:CirrusSearch#Articletopic for user docs
* Search/WeightedTags
Search/lsearchd inventory

In February 2014 we decided to look at what we're using in lsearchd with an eye to what can be reallocated to Elasticsearch. This page serves to document each server's status.
Server 	Lsearchd pool 	Job 	Utilized?
search1001 	Pool 1 	enwiki content search and spelling 	Yes
search1002 	Pool 1 	enwiki content search 	Yes
search1003 	Pool 1 	enwiki content search and general search 	Yes
search1004 	 	enwiki content highlighting 	Yes
search1005 	 	enwiki content highlighting 	Yes
search1006 	 	enwiki general search and spelling 	Yes
search1007 	Pool 2 	frwiki, jawiki, dewiki search 	Less
search1008 	Pool 2 	frwiki, jawiki, dewiki search 	Less
search1009 	 	frwiki, jawiki, dewiki highlighting and frwiki and dewiki spelling (but not jawiki's spelling) 	Less
search1010 	 	frwiki, jawiki, dewiki highlighting and frwiki and dewiki spelling (jawiki doesn't have spelling)	Less
search1011 	Pool 3 	largish wikis non-talk search, all eswiki search and highlighting 	Less
search1012 	Pool 3 	largish wikis non-talk search, all eswiki search and highlighting 	Less
search1013 	 	largish wikis highlighting and spelling 	Less
search1014 	 	largish wikis highlighting and spelling 	Less
search1015 	Pool 4 	everything else 	Just disk
search1016 	Pool 4 	everything else 	Just disk
search1017 	Pool "prefix" 	all prefix searches 	Just disk
search1018 	Pool "prefix" 	all prefix searches 	Just disk
search1019 	Pool 5 	commons, wikidata, enwiktionary, meta 	Just disk
search1020 	Pool 5 	commons, wikidata, enwiktionary, meta 	Just disk
search1021 	 	everything else highlighting 	Just disk
search1022 	 	everything else highlighting 	Just disk
search1023 	 	largish wikis highlighting and spelling 	Almost none
search1024 	 	largish wikis highlighting and spelling 	Almost none
searchidx1001	Search indexer for all wikis	 	Oh yeah
SearchShards

= what =
Search/Lucene index sharding.
== finest quality script to gather index distribution data ==
#!/usr/bin/perl
my $index;
my @hosts = (1..20);
for my $n (@hosts) {
 open SSH, "ssh root\@search$n \"find /a/search/indexes/update -ctime 0 -type d -name [0-9]* \"|";
 while (<SSH>) {
 if (/([^\/]+)\/\d+$/) {
 $index->{$1}->{$n} = 1;
 }
 }
 close SSH;
}
print "{| class=\"wikitable sortable\"\n|-\n! shard";
for my $n (@hosts) {
 print " !! $n";
}
print "\n";
for my $shard (sort keys %{$index}) {
 print "|-\n| $shard";
 for my $n (@hosts) {
 my $cell = (defined $index->{$shard}->{$n}) ? 'y' : '—';
 print " || $cell";
 }
 print "\n";
}
print "|}\n";
Search Platform/About

The Search Platform team (part of Wikimedia Technology) is responsible for maintaining and enhancing the various Search features and APIs for MediaWiki. This includes the CirrusSearch extension which relies on Elasticsearch, the search backend used at the Wikimedia Foundation to support Wikimedia projects and Wikidata Query Service, the SPARQL endpoint used to query Wikidata.
== Mission ==
Our mission is to help people easily discover knowledge on Wikipedia and its sister projects by providing tools and infrastructure for casual readers and expert users with precise needs, while maintaining a strong emphasis on privacy.
== Overview ==
* We operate and maintain a disparate collection of production services related to content discovery, enabling the wiki community to find information that is not available through simply following links. We also provide a platform on which other people can create tools to support editing and other workflows.
* We provide an open-source search engine, backed by an inverted index for non-structured on-wiki data. We work to develop more sophisticated searching with machine learning and natural language processing.
* We provide a SPARQL-based query service for Wikidata and Commons, encouraging users to capitalize on this vast store of computer-readable structured data for use on-wiki and in knowledge discovery.
* We endeavor to support underserved wiki communities, and we rely on those communities to help us understand their needs and evaluate potential solutions, especially with respect to underserved languages.
* We prioritize privacy for logged-in users and anonymity for logged-out users over almost everything else, even when it slows down or complicates development or hinders our ability to collect or use data.
Search Platform/Contact

The canonical version of this page is on OfficeWiki, please report all edits there.
= Which channels to use to contact the team =
* Urgent requests, breakage: 
* Simple questions, quick synchronous conversations: 
* Longer questions, request for help / clarification: discovery mailing list
* Requests for help / work from the team: phab task, tagged with "Discovery-Search" or "Wikidata-Query-Service" (TBD: add link to "make a request")
= Communication channels =
== Mailing lists ==
mail:discovery—A public mailing list about the Wikimedia Search Platform team and projects (formerly Discovery Department). Examples of topics would include:
* Announcements, including major upcoming initiatives, completed major releases, quarterly or annual plans, requests for feedback or input
* Technical discussions and brainstorming regarding our work:
** Search, Elastic, Cirrus, the Relevance Forge, and other relevant subjects
** Our dashboards or related analysis
* Other team news, such as changes to team structure, significant changes to processes, changes in how we use phabricator or other tools like gerrit
The Search Platform team is also active and watching the following lists:
* mail:wikidata: anything related to Wikidata Query Service
* mail:commons-l: anything related to Wikimedia Commons Query Service
* mail:wikitech-l: general technology
== Phabricator ==
Phabricator tasks tagged with either "Discovery-Search" or with "Wikidata-Query-Service" will be triaged every Monday.
== IRC channels ==
== Office Hours ==
The Search Platform Team usually holds office hours on the first Wednesday of each month. Come talk to us about anything related to Wikimedia search! 
* Feel free to add your items to the Etherpad Agenda for the next meeting.
== Weekly status updates ==
See Search Platform/Weekly Updates.
== Meetup groups ==
* San Francisco
** Directly relevant
*** Bay Area NLP (natural language processing, not neuro-linguistic programming)
*** San Francisco text
*** Elasticsearch San Francisco
** Indirectly related (these sorts of meetup groups attract smart/enthusiastic people who like to spend their free time learning and solving problems)
*** Silicon Valley Java user group
*** San Francisco PHP
*** Bay Area Haskell users group
*** Scala study group
*** SF JavaScript
*** Oakland advanced Scala study group
Search Platform/Goals

= Goals for each quarter =
* OKR Q3 2022-2023
* OKR Q4 2022-2023
* OKR Q1 2023-2024
* OKR Q2 2023-2024
* OKR Q1 2024-2025
= Other sources of information =
The high level epics that the team is currently working on can be found in the "Epics" column on our team Phabricator workboard.
A more detailed status of each current OKR can be found in Asana.
Search Platform/Roles and Responsibilities

Roles are not boundaries, don’t feel restricted by your role.
= Project Tech Lead =
A team member is assigned to each project as a tech lead. The tech lead is the focal point for a project. The tech lead is not expected to decide everything themselves, but to be the facilitator that allows the team to keep its coherence. Having the Tech Lead rotate per project allows sharing this responsibility and give everyone their chance to improve their skills.
* Work with engineering manager to define project plan and roadmap
* Primary point of contact to understand the state of the project, and the technical and organizational constraints
* Primary point of contact for communication with other teams, manage incoming and outgoing dependencies
* Responsible for breaking work into manageable pieces that can be shared across the team
* Identify risks and mitigations early
* Primary contact to solidify the technical constraints of the project (software and system architecture / gardening)
= Software Engineer =
* Contribute to discussion around software practices, architecture and delivery
* Ensure code quality through testing, design patterns and attention to code quality
* Ensure that value is delivered early, in an iterative process
* Support the tech lead by providing input, ideas, suggestions, and participating in discussions related to current projects
* Provide visibility on the work being done by using the appropriate communication channels (phab, etherpad, ...)
* Work with the engineering manager to propose new features and direction for the team
* Work collaboratively with team members to complete defined work
* Identify challenges, blocks and ensure that the team is aware of those
* Take a proactive role in team rituals
= Computational Linguist =
* Contribute to discussion around language support
* Ensure quality of language support through careful quantitative and qualitative analysis
* Ensure that value is delivered early, in an iterative process
* Provide visibility on the work being done by using the appropriate communication channels (phab, etherpad, ...)
* Work with the engineering manager to propose new features and direction for the team
* Work collaboratively with team members to complete defined work
* Identify challenges, blocks and ensure that the team is aware of those
* Take a proactive role in team rituals
= Engineering Manager =
The engineering manager's primary responsibility is supporting the team to deliver quality solutions that satisfy our users' needs. In the absence of a product manager, the engineering manager has some responsibility to advocate for our users and provide a long term product vision.
* Address impediments that block the team progress
* Facilitate team rituals, build effective internal processes for the team
* Ensure that the team is aligned and has a common understanding of goals and priorities
* Provide visibility on work of the team to our communities and higher management
* Tie breaker in case decisions are stuck at the team level
* Provide an escalation path in case of conflicts with other teams
Search Platform/Team

The canonical version of this page is on OfficeWiki, please report all edits there.
= Team members =
* Erik Bernhardson, Tech Lead, Staff Software Engineer
* David Causse, Staff Software Engineer
* Peter Fischer, Senior Software Engineer
* Trey Jones, Staff Computational Linguist
** Contact info on my user page.
* Ryan Kemper, Site Reliability Engineer
* Brian King, Senior Site Reliability Engineer
* Guillaume Lederrey, Engineering Manager
** Direct contact on IRC (gehel on ) or Slack
** Asynchronous conversations: email
** Emergency: phone
* Mike Pham, Senior Product Manager
Contact information, including phone numbers are available on Office wiki.
= Who to contact when? =
The tables below show who can be contacted for which subject. If you don't see the subject you are interested in, you can contact the team's engineering manager (Guillaume) or Product Manager (Mike).
== Organizational ==
 	Prioritization	Team processes	ERC
Guillaume Lederrey	X 	X 	X
 	 	 	
 	 	 	
== Search ==
=== Technical components ===
 	Operations	Elasticsearch	Update Pipelines	Airflow	Flink	Saneitizer	Mjolnir	Language support	Analysis chains	Weighted tags
Erik Bernhardson 	X 	X 	X 	X 	 	X 	X 	 	 	X
Trey Jones 	 	X 	 	 	 	 	 	X 	X 	
Guillaume Lederrey	X 	X 	 	 	 	 	 	 	 	
 	 	 	 	 	 	 	 	 	 	
=== Features ===
 	Machine Learning Ranking	More Like	Sister Search	Cross Language Search	Search Syntax	Did you mean	MediaSearch
Erik Bernhardson	X 	X 	X 	X 	X 	X 	
Trey Jones 	 	 	X 	X 	X 	X 	
 	 	 	 	 	 	 	
== Query Services ==
 	Operations	Blazegraph	Data reload	Update pipeline	SPARQL	Authentication	Traffic management (throttling, filtering)	Federation
Guillaume Lederrey	X 	X 	X 	X 	 	X 	X 	
 	 	 	 	 	 	 	 	
 	 	 	 	 	 	 	 	
== Misc skills ==
 	Puppet	Python	Java	Scala	Maven	Mediawiki	Docker	k8s	Flink	Airflow	Spark	Perl
Guillaume Lederrey	X 	X 	X 	 	X 	 	 	 	 	 	 	
Trey Jones 	 	 	 	 	 	 	 	 	 	 	 	🐪
Search Platform/Weekly Updates/2023-02-24

= Summary =
We finally have a successful data reload for WDQS. It still needs to be copied over to all servers. This process is still fragile and problematic: it took 2.5 month to complete a task that should take at most a couple of weeks.
= What we've accomplished =
== Spark 3 migration ==
* RDF (Java / Scala) jobs are running on Spark 3. This was one of the more complex job to be migrated - https://phabricator.wikimedia.org/T327381
* Migrated drop_old_data_daily from Airflow 1 to 2, needed some additional fixes to date handling.
== Search Update Pipeline ==
* Removed our use of a custom Swift plugin in Flink, now relying on the standard S3 interface.
* Preliminary work on tracking update lag to create a formal SLO - https://phabricator.wikimedia.org/T320408
== Operations / SRE ==
* Documentation for Elasticsearch is updated. The datacenter switch should be fully transparent, with no manual intervention this year - https://phabricator.wikimedia.org/T330417
* Initial data reload on WDQS completed, copy to all servers in progress - https://phabricator.wikimedia.org/T323096
* k8s upgrade went well, with our Flink application reacting as expected.
== Misc ==
* Ongoing work to align the Search Platform team documentation with ERC Team Interface template - https://office.wikimedia.org/wiki/ERC/Search
* Communication on the WDQS data reload issues went out (https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/thread/7QTJBRU2T3J22SNV4TGBRML4QNBGCEOU/), this prompted some discussion, mostly about modularity and loose coupling.O