nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2024-05-03

= Summary =
As the Search Update Pipeline work is ramping down, we're are onboarding more team members to work on the WDQS Graph Split.
Improve multilingual zero-results rate is going to be on pause until July, while one of our team member is on sabbatical.
We had to deal with the follow up of an incident with completion indices, root cause was tracked to a network issue.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Analysis of normalization of Arabic family languages, write up on https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Normalization_for_Arabic_Script_Across_Languages - https://phabricator.wikimedia.org/T72899
== WDQS graph splitting ==
* Work started on automating the data reload process - https://phabricator.wikimedia.org/T349069
* Onboarding a new team member on the project
== Operations ==
* Incident follow up on missing documents in the search completion indices in codfw, root cause seem to be a network issue (https://phabricator.wikimedia.org/T363516#9748908), but we should also increase the reliability of the indices update in face of errors - https://phabricator.wikimedia.org/T363516
* Late events in the Search update pipeline - https://phabricator.wikimedia.org/T359580
Search Platform/Weekly Updates/2024-05-10

= Summary =
Lot of public holidays this week across the team.
Work is ramping down on Search Update Pipeline and ramping up on the implementation of the WDQS Graph Split.
= What we've accomplished =
== WDQS graph splitting ==
* A Signpost update on the graph split and the size of the WDQS graph is drafted and forthcoming - https://en.wikipedia.org/wiki/User:Bluerasberry/signpost_wikicite
* Benchmarking of CPU governor and BlazeGraph configuration variable complete. It took 2.15 days to load the scholarly graph with this configuration, compared to the original configuration taking 5.875 days; the "main" (not scholarly) graph would be slightly slower, but in the same ballpark of 2-3 days. Although further performance gain would likely be achievable with an NVMe based on behavior on another computer, it isn't possible right now to install the NVMe to replicate this. The performance gain has provided a bigger buffer for ensuring imports in an even more timely fashion, though, already. https://phabricator.wikimedia.org/T362920
* Adapted the import_ttl dag to always do the graph split and parquet -> n3 transformation - https://phabricator.wikimedia.org/T362060
* Working on adapting the data-reload cookbook to source its data from HDFS - https://phabricator.wikimedia.org/T349069
== Search Metrics ==
* Notebooks posted at https://gitlab.wikimedia.org/repos/search-platform/notebooks/-/tree/main/cirrus covering a range of behavior on search-powered endpoints. https://phabricator.wikimedia.org/T358345
== Search Update Pipeline ==
* Quick hack on the SUP producer to workaround what seems to be a flink bug - https://issues.apache.org/jira/browse/FLINK-22425
== Operations ==
* WDQS updater misbehaving in codfw - https://phabricator.wikimedia.org/T362508
* WDQS lag propagation to wikidata not working as intended - https://phabricator.wikimedia.org/T360993
Search Platform/Weekly Updates/2024-05-17

= Summary =
Consultation period for the WDQS Graph Split proposal is over, we had a few comments, but no request to fundamentally change our approach. A Signpost article by Blueraspberry is adding some visibility to our efforts (https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2024-05-16/Op-Ed).
= What we've accomplished =
== Search Update Pipeline ==
* Some failures in Wikidata updates, linked to a bug in Flink. The bug is patched in our version and has been submitted (and merged) upstream - https://phabricator.wikimedia.org/T364837
* We started to shift updates to the new SUP for codfw.
== WDQS graph splitting ==
* Signpost article published. Only a few comments on the talk page at the moment, and it seems that the issue is mostly understood and accepted. It has generated some additional feedback on the project pages - https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2024-05-16/Op-Ed
* Massive refactor of the data-reload cookbook - https://phabricator.wikimedia.org/T349069
* Work started on Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs - https://phabricator.wikimedia.org/T361935
== Search Metrics ==
* Metric collection code moved from jupyter notebooks to our usual git repo - https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/33
* Dashboard proposal at https://phabricator.wikimedia.org/T364600#9801594
== Misc ==
* Peter is preparing a proposal for the Flink Forward conference - https://www.flink-forward.org/berlin-2024
* serviceops moving to calico for network policies, change applied to all k8s clusters running flink jobs, worked smoothly - https://phabricator.wikimedia.org/T287491
Search Platform/Weekly Updates/2024-05-24

= Summary =
100% of updates (excluding private wikis) are migrated to our new Search Update Pipeline for our codfw and cloudelastic clusters, 25% for the eqiad cluster. We expect to complete the migration next week.
The recent Signpost article (https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2024-05-16/Op-Ed) has generated some additional discussion around the graph split. We're following them and engaging with the discussion.
= What we've accomplished =
== Search Update Pipeline ==
* All updates (except private wikis) for codfw and cloudelastic, 25% for eqiad, have been migrated to the new search update pipeline, we expect to shift the rest of the traffic next week - https://phabricator.wikimedia.org/T363475
* small patch to stop shipping cirrusLinksUpdate jobs when there are no writable clusters available, these jobs would do nothing, this should save some resources on jobrunners, kafka and changeprop - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1035429
* Fixed some wikidata items not being parsed correctly due to integer overflow - https://phabricator.wikimedia.org/T364837Cannot provide empty array to wikis as $wgCirrusSearchWriteClusters - https://phabricator.wikimedia.org/T365190
== WDQS graph splitting ==
* Respond to comment on the refinement talk page - https://www.wikidata.org/wiki/Wikidata_talk:SPARQL_query_service/WDQS_graph_split/WDQS_Split_Refinement
* Deployed the airflow change to split the graph daily with spark
* data-reload cookbooks reviewed, we should be able to test them next week - https://phabricator.wikimedia.org/T349069
* Generalize ScholarlyArticleSplitter - https://phabricator.wikimedia.org/T362060
== Search Metrics ==
* We have a superset dashboard for search metrics (https://superset.wikimedia.org/superset/dashboard/search/) - https://phabricator.wikimedia.org/T358345 (and a number of subtasks)
== Misc ==
* Fixed failing tests on Glent with Java 11 - https://phabricator.wikimedia.org/T350974
* Fixed image suggestion pipeline that was failing on empty partitions - https://phabricator.wikimedia.org/T358472
Search Platform/Weekly Updates/2024-05-31

= Summary =
Slow week, with vacations and sabbatical.
= What we've accomplished =
== Search Update Pipeline ==
* New repository, https://gitlab.wikimedia.org/repos/search-platform/cirrus-reindex-orchestrator/. Orchestrates full-cluster reindex. Has completed cloudelastic, and is running against eqiad and codfw now. General idea is a one-touch reindex procedure. Interacting with SUP backfilling required re-thinking how we orchestrate this.
* Working on aligning the update schema with the cirrus doc - https://phabricator.wikimedia.org/T365692
== WDQS graph splitting ==
* New volunteers joining our regular graph split meeting
== Search Metrics ==
* Answering on reporting for https://superset.wikimedia.org/superset/dashboard/search/
== Operations ==
* Investigation of XML dumps issue. https://phabricator.wikimedia.org/T365155
Search Platform/Weekly Updates/2024-06-07

= Summary =
We're working on communication following the feedback period for the graph split, and on finalizing the implementation.
All writes (except private wikis) switched to the new search update pipeline, leading to a significant decrease in PHP worker saturation.
= What we've accomplished =
== WDQS graph splitting ==
* Update on-wiki pages related to the split:
** Added a note regarding the end of the feedback period https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/WDQS_Split_Refinement
** Created a simplified version of the definition of the split: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Rules
* Prepared a test of the new wdqs.data-reload cookbook: https://phabricator.wikimedia.org/P64016
* Continued work on update pipeline - https://phabricator.wikimedia.org/T361935
== Search Update Pipeline ==
* All writes (except private wikis) switched to the new pipeline - https://phabricator.wikimedia.org/T363475
** Significant decrease in PHP worker saturation related to this: https://grafana-rw.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s?forceLogin=true&from=1716422400000&orgId=1&to=1717027200000&var-container_name=All&var-dc=eqiad+prometheus%2Fk8s&var-namespace=mw-jobrunner&var-release=main&var-service=mediawiki&var-site=&viewPanel=84
== Search Metrics ==
* Automate search metrics notebooks and integrate with Airflow - https://phabricator.wikimedia.org/T364599
== Misc ==
* Fixed a small problem in the drop_old_data_daily dag failing because I misconsfigured a table that do not have a wiki partition
* Reindexing across all three clusters is completed with the new orchestration. Based on this run refactored the orchestration to operate on a per-index basis instead of a per-wiki basis. - https://phabricator.wikimedia.org/T363734 This completes the work on various search harmonization efforts:
** https://phabricator.wikimedia.org/T72899
** https://phabricator.wikimedia.org/T358495
** https://phabricator.wikimedia.org/T361377
** https://phabricator.wikimedia.org/T180387
** https://phabricator.wikimedia.org/T362501
Search Platform/Weekly Updates/2024-06-14

= Summary =
Search Update Pipeline is mostly done' with a few minor tuning still going on now that all wikis are using it.
First data reload of the WDQS graph split with the new automated process is ongoing. Work on the split update pipeline in progress.
= What we've accomplished =
== Search Update Pipeline ==
* wrote a small python script to send rerender events scanning the allpages API, reindexed all 1.3M lexemes using it - https://phabricator.wikimedia.org/T365692
* Rate limiting:
** Use specific user agent in mediawiki api requests - https://phabricator.wikimedia.org/T366363
** Tested back-fill of production-like volume of updates on staging: Requests exceeding 1k/s are blocked - https://phabricator.wikimedia.org/T362310
== WDQS graph splitting ==
* Testing of WDQS data reload. https://phabricator.wikimedia.org/P64016
* Ensure that WDQS query throttling does not interfere with federation - https://phabricator.wikimedia.org/T361950
== Search Metrics ==
* Collaboration with Product Analytics to improve metrics on mobile apps - https://phabricator.wikimedia.org/T259883
== Misc ==
* Fixed a bug in wikimedia-eventutilities to allow parsing iso8601 dates with trailing timezone offsets instead of a trailing 'Z' that caused the producer to fail after sending such formats
* Made all the subgraph analysis hive table readable by others (requested by product analytics)
Search Platform/Weekly Updates/2024-06-21

= Summary =
While we are making progress on a split updater and on the underlying infrastructure for new SPARQL endpoints, it is likely that those endpoint will only be available mid-July.
= What we've accomplished =
== WDQS graph splitting ==
* Data import completed on wdqs2023
** Loaded 15B triples (we expect 10B triples per subgraph for a 20B triple full graph)
** Took 7 days 15h for just the data reload, it still needs to catch up on initial lag
** Validates our automation process - https://phabricator.wikimedia.org/T364077 / https://phabricator.wikimedia.org/T349069
* Permission to create new Kafka topics for the split updater - https://phabricator.wikimedia.org/T367510
* Initial implementation of a split updater, working on a test suite - https://phabricator.wikimedia.org/T361935
= Misc =
* Started to migrate Cirrus statsd metrics to statsv - https://phabricator.wikimedia.org/T359033
Search Platform/Weekly Updates/2024-06-28

= Summary =
Vacations and sabbaticals makes this a slow week.
We still had to fix a few minor issues on the new Search Update Pipeline. Work on WDQS Graph Split is moving forward but it will still take a couple of weeks before we have endpoints publicly available.
= What we've accomplished =
== Search Update Pipeline ==
* Working on a fix for https://phabricator.wikimedia.org/T331127 (support cross-index page moves)
* Looking into https://phabricator.wikimedia.org/T368010 (wikidata entity schema)
** Thinking about adding support for page rerender upsert to allow refreshing non-indexed pages
*** opened a patch https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/143 to hopefully help this kind of maintainance task
** Root cause of the problem is that the entity schema was erroneously tagged as non-content
== WDQS graph splitting ==
* Discussion with traffic to solve our problem regarding how internal federation can bypass throttling (https://phabricator.wikimedia.org/T361950)
** Seems like we can rely on a header (X-Client-Ip is standard enough that it should be safe to use it)
** Going back to LVS is not ideal and they prefer that we use envoy as a load balancer, some research has to be done to see if and how this is done in other places in our infra
* Started reviewing the graph split code in the updater, added some more tests to illustrate comments
== Other work ==
* Dumps incident documentation for https://phabricator.wikimedia.org/T368098
* Mute helmfile apply notifications from cirrus-streaming-updater deploys - https://phabricator.wikimedia.org/T366346
Search Platform/Weekly Updates/2024-07-05

= Summary =
A new quarter is starting. Our goals for this quarter are not entirely clear yet, but will include:
* Continuing graph split work
* Supporting the migration from Archiva to Gitlab
* Supporting the work of the Web team as needed on WE3.1 (Improve general and personalized browsing)
It might also include:
* Decrease update lag in search for Wikidata edits - https://phabricator.wikimedia.org/T365831
* Deploying the new Search Update Pipeline to private wikis - https://phabricator.wikimedia.org/T341332
* Support the Kafka version upgrade
* Plan the migration from Elasticsearch to OpenSearch
* Define use cases and experiments around vector search
= What we've accomplished =
== Search Update Pipeline ==
* T331127 phantom redirects lingering in incategory searches after page moves - https://phabricator.wikimedia.org/T331127
== Improve multilingual zero-results rate ==
* looking at the results of deploying the dotted_i_fix - https://phabricator.wikimedia.org/T358495
== Misc ==
* Starting to migrate cirrus icinga alerts to alertmanager - https://phabricator.wikimedia.org/T359033
* was blocked by https://phabricator.wikimedia.org/T369080
* Re-deployed all flink jobs to support restricted PSS - https://phabricator.wikimedia.org/T362978
* looked into the failures of the export_queries_to_relforge dag, there was a missing partition and then relforge failed because of too many shards
** DPE (Andrew & Xabriel) fixed the missing partition
** cleaned up relforge
Search Platform/Weekly Updates/2024-07-12

= Summary =
The WDQS graph split updater has been running without issues for multiple days on Yarn (as a test environment). We will start deploying it in a production environment next week.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* The write up for "Harmonization Post-Reindex Evaluation, Part II" is available on MediaWiki: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Harmonization_Post-Reindex_Evaluation,_Part_II
== WDQS graph splitting ==
* the updater with the new split graph support is running fine in yarn
== Misc ==
* Resolve some itermittent failures in the cirrus browser testing suite that was making CI tedious
* Look into the cache hit rate of the cirrus cache of ParserOutputPageProperties. This should be seeing ~66% hit rate but instead is seeing <1%. This induces significant load on the mediawiki api clusters. Have added metrics and will re-review after deployment. Probably also add some alerting once fixed.
* Provide a potential fix for tracking search satisfaction events (frontend/UI) - https://phabricator.wikimedia.org/T286814
* Completion suggester can promote a bad build - https://phabricator.wikimedia.org/T363521
Search Platform/Weekly Updates/2024-07-19

= Summary =
WDQS Graph Split is late, but moving forward nicely. We have a working updater deployed, it will need a couple of weeks to generate enough data. We will then be able to expose the new SPARQL endpoints.
We are starting work on migrating private wikis to the new Search Update Pipeline.
We finally have a definite answer to how many languages are supported by our search engine, but that answer isn't simple: https://techblog.wikimedia.org/2024/07/18/how-many-languages-does-wikimedia-search-support/
= What we've accomplished =
== Search Update Pipeline / Private Wikis ==
* Delving into EventBus / EventStreams to understand how it works and ensure we can safely produce private streams without risk of producing private events into public streams (via guidence from Andrew Otto). Patches and configuration mostly complete, deployment will take ~ 2 weeks for train deploys and verification. - https://phabricator.wikimedia.org/T346046
== Search Update Pipeline ==
* Let wikidata updates bypass deduplication window to reduce lag - https://phabricator.wikimedia.org/T365831
* SUP: Retry 429 (rate limit) at HTTP client level - https://phabricator.wikimedia.org/T367691
== WDQS graph splitting ==
* The split graph updater is running in production
** https://phabricator.wikimedia.org/T361935
** https://phabricator.wikimedia.org/T367510
** https://phabricator.wikimedia.org/T369729
* Writing a Internal Federation Guide on wiki
* Query throttling and internal federation appears to work OK on experimental endpoints - https://phabricator.wikimedia.org/T361950
== Misc ==
* New blog post on how many languages are supported by Search: https://techblog.wikimedia.org/2024/07/18/how-many-languages-does-wikimedia-search-support/
* Homogenise jackson version in WDQS - https://phabricator.wikimedia.org/T365158
* Search and haswebstatement not working for EntitySchema - https://phabricator.wikimedia.org/T368010 / https://phabricator.wikimedia.org/T369495
* PHP Deprecated: Implicit conversion from float 75000.00000000001 to int loses precision - https://phabricator.wikimedia.org/T366589
* Completion suggester can promote a bad build (not actually fixed, but we have metrics in place in case the issue is reproduced) - https://phabricator.wikimedia.org/T363521
Search Platform/Weekly Updates/2024-07-26

= Summary =
We're getting very close to having SPARQL endpoints for the WDQS main and scholarly subgraphs. We're working on documentation and communication.
We are drafting a plan on replacing our aging Elasticsearch 7.x, which besides providing a supported platform for search should also allow us to experiment with vector search.
= What we've accomplished =
== WDQS graph splitting ==
* Drafted a guide about federation at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Internal_Federation_Guide
== Search backend replacement ==
* A decision record for the target technology for the search backend replacement was drafted for team review. After this initial review it will be circulated further - https://www.mediawiki.org/wiki/Draft:ABaso_(WMF)/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology
== Misc ==
* Investigate incident causing instabilities on the search clusters - https://phabricator.wikimedia.org/T370621
* Additional edits to https://techblog.wikimedia.org/2024/07/18/how-many-languages-does-wikimedia-search-support/ to clarify that we do support all languages communities, even if the level of support for specific languages varies.
* Error: Call to a member function audienceCan() on null (error related to optimization in Wikidata indexation) - https://phabricator.wikimedia.org/T370770
Search Platform/Weekly Updates/2024-08-08

== Summary ==
The WDQS graph split is loading onto its first production hosts. There were some hiccups during the first attempt, but the import is happening now and we'll be watching over the coming days to verify it worked.
Rollout of SUP for private wikis has some issues requiring debugging.
== What we've accomplished ==
=== WDQS graph splitting ===
* Graph split rolling out to production hosts. https://phabricator.wikimedia.org/T370754
* Some Grafana graphs will need some retooling to reflect new arrangement of clusters.
=== Search Update Pipeline / Private Wikis ===
* Shipped NetworkSession to production, with troubleshooting on network connections underway. https://phabricator.wikimedia.org/T355267
* Debugging deployment of events for private wikis affecting jobs daily. https://phabricator.wikimedia.org/T371767
* Added support for authorizing API requests. https://phabricator.wikimedia.org/T345185
* Added a dashboard tracking SUP lag. https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo?orgId=1
=== Improve multilingual zero-results rate ===
* Review of old notes, Phabricator tickets, and patch notes to understand current configuration. There will be refactoring to disentangle ICU and ASCII folding. https://phabricator.wikimedia.org/T332342
=== Search backend replacement ===
* Decision record for OpenSearch shown to SRE-at-large, review requested for the week. https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology
=== Misc ===
* Repair ability to run airflow-dags test suite locally on a Linux host. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/790
* Repair process_sparql_queries_hourly Airflow DAG. The task was running into out of memory errors, and it turns out it was also performing expensive work twice instead of reusing the result. https://phabricator.wikimedia.org/T371217
* Ship a fix for invalid events being generated from our frontend search logging events. Error rate has declined since train rolled forward, but not clear if this completely fixes the problem yet. https://phabricator.wikimedia.org/T286814
* Started a reindex of Wikidata on eqiad &codfw. https://phabricator.wikimedia.org/T371401
Search Platform/Weekly Updates/2024-08-16

= Summary =
All wikis, including private wikis, are now running on the new Search Update Pipeline. We might have a few minor tweaks in the following weeks, but this concludes a long effort to modernize our Search Update Pipeline!
= What we've accomplished =
== Search Update Pipeline / Private Wikis ==
* Private wikis are now migrated to the new Search Update Pipeline. This aligns all our wikis on the same update mechanism and concludes the work on this new SUP. The code for the previous update mechanism will stay in place and can still be used by 3rd parties as needed, but will not be maintained moving forward - https://phabricator.wikimedia.org/T341332
== Misc ==
* OOM issues when running QueriesProcessor (process_sparql_query_hourly) - https://phabricator.wikimedia.org/T371217
* '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814