nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2023-06-23

= Summary =
We've been dealing with more infrastructure complications this week. Search was broken over the weekend on Commons and Wikidata (https://wikitech.wikimedia.org/wiki/Incidents/2023-06-18_search_broken_on_wikidata_and_commons / https://phabricator.wikimedia.org/T339810). Missing canary events in EventGate created some additional issues in our downstream data pipelines that required manual intervention.
= What we've accomplished =
== Improve multilngual zero-results rate ==
* Progress on smarter handling of acronyms - https://phabricator.wikimedia.org/T170625
* Progress on aggressive splitting - https://phabricator.wikimedia.org/T219108
== Search Update Pipeline ==
* Progress on deduplication of the update stream and support for redirects -https://phabricator.wikimedia.org/T325672 / https://phabricator.wikimedia.org/T325315
== WDQS Split the graph ==
* Started documentation of the data analysis needs - https://docs.google.com/document/d/1QsV96LtpK5lDD2N2jy-6vaF_0d_Yf_HLb8uFARFMxJ8/edit
== Operations ==
* Canary events stopped being produced, causing downstream issues with most data pipelines. Manual steps were needed to recover. Route cause is tracked on the Data Engineering team (https://phabricator.wikimedia.org/T340166). Larger conversation is needed to make the system more robust and easier to recover from failures.
== Misc ==
*Search was broken on Commons and Wikidata over the weekend - https://wikitech.wikimedia.org/wiki/Incidents/2023-06-18_search_broken_on_wikidata_and_commons
Search Platform/Weekly Updates/2023-06-30

= Summary =
This is the last update of the quarter. Most of our projects are going to overflow to next quarter as expected.
Search update pipeline work is on track to be functionally complete by the end of next quarter, with some risks of the deployment strategy not being ready. Discussions are ongoing on how we can prioritize that work across Data Platform Engineering teams to get the support we need on time.
Improvements to multilingual zero-results rate are coming along. The most time consuming part was about measuring the impact, not about implementing the improvements. We will review the metrics used for next quarter to see if we can have metrics that are easier to compute, while still giving us a good sense of the impact of the improvements.
Search SLOs have been defined, we should be able to implement them during next quarter.
Note that we will have multiple team members on vacation during next quarter, so we will be in reduced capacity.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Ongoing work on aggressive splitting and word_break_helper - https://phabricator.wikimedia.org/T219108 / https://phabricator.wikimedia.org/T170625
== Search Update Pipeline ==
* Ongoing work on support for redirects - https://phabricator.wikimedia.org/T325315
* Ongoing work on support for reordering and optimization of change events - https://phabricator.wikimedia.org/T325672
== Operations / SRE ==
* Fix metric collection for Blazegraph on Debian Bullseye - https://phabricator.wikimedia.org/T336540
* Investigation of potential performance issues with new WDQS servers - https://phabricator.wikimedia.org/T336443
== Misc ==
* Progress on migrating from ORES to Liftwing for ingesting articletopics in Search -https://phabricator.wikimedia.org/T328276
* First version of an SLO dashboard to track update lag for WDQS - https://phabricator.wikimedia.org/T324811
* Optimize the elasticsearch analysis settings for wikibase - https://phabricator.wikimedia.org/T334194
Search Platform/Weekly Updates/2023-07-07

== Summary ==
Happy Fiscal New Year! 
== What we've accomplished ==
=== Search Update Pipeline ===
* Simplifying/optimizing CirrusSearch filters
* Squash cirrus doc caching bug
=== Operations/SRE ===
* Further improvements to data transfer cookbook
=== Misc ===
* Trey (our favorite computational linguist) to present analyzer unpacking at upcoming Tech & Product Meeting
Search Platform/Weekly Updates/2023-07-14

== Summary ==
Happy Bastille Day! 
== What we've accomplished ==
=== Search Update Pipeline ===
* Met with Service Ops, Data Engineering, and Event Platform team and decided to use Zookeeper for Flink HA. 
* Stop sending obviously broken queries to elasticsearch. Did this because of an automated client shipping the same bad query over and over and "massivily" (2.5 q/s) polluting our rejected metrics 
* Gather some numbers on the local_sites_with_dupe flag, it seems to me that this array contains mostly false positives (70% for enwiki). Discuss another approach that does not require any work at index time . Ideally, this should help us avoid using this feature in the new pipeline design. 
* Squash bug related to how we cache cirrus docs in memcached. 
=== Improve Multilingual Zero-Results Rate ===
* Continue work on work_break_helper in language analyzers
=== Misc ===
* Trey (our favorite computational linguist) presented analyzer unpacking at Tech & Product Meeting. It was both informative and entertaining! Selina has requested a brown bag on this topic.
* Chugging through word_break_helper candidates and looking at word_break_helper's interference with Korean/Nori and Chinese/SmartCN tokenizers.
Search Platform/Weekly Updates/2023-07-21

= Summary =
The team is working on the Search Update Pipeline, and improvements to multilingual zero-results rate.
We are formalizing the collaboration between Data Engineering and Search Platform around Flink deployments, which should help move forward better on the Search Update Pipeline work.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Some issues with testing and continuous integration (test coverage was reduced by moving unit tests to integration tests by another patch)
* Language, Harmony, and Unpacking Q&A session is scheduled for Monday, July 24, 2023 at 16:00 UTC (9 am PDT, Noon EDT, 6pm CEST). Have a look at the initial presentation (https://upload.wikimedia.org/wikipedia/commons/d/d5/Language%2C_Harmony%2C_and_Unpacking%E2%80%94Trey_Jones%2C_July_2023.webm) and add your questions to this etherpad: https://etherpad.wikimedia.org/p/Language,_Harmony,_and_Unpacking%E2%80%94Part_IS
* Handle variation in apostrophe-like characters better, see the full write up on wiki - https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Apostrophes_(T315118) / https://phabricator.wikimedia.org/T315118
* Investigate applying aggressive_splitting everywhere, not just on English-language wikis, see the full write up on wiki - https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#aggressive_splitting_(T219108) / https://phabricator.wikimedia.org/T219108
== Search Update Pipeline ==
* Ongoing work on adding support for page re-render to the pipeline: https://gitlab.wikimedia.org/dcausse/cirrus-streaming-updater/-/commits/page-rerender-wip
* Re-order and optimize change events - https://phabricator.wikimedia.org/T325672
* Summary of the use of Kafka topics is documented at https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD_64m0zQErny-9jUD5C6RGf_bU/edit#gid=1175302241 so that we can better coordinate with other teams on our requirements
== Misc ==
* Fixed bug in Wikidata that was preventing undeletes to reappear in WDQS - https://phabricator.wikimedia.org/T341905
* Adding tests to CirrusSearch to prevent an incident similar to https://wikitech.wikimedia.org/wiki/Incidents/2023-06-18_search_broken_on_wikidata_and_commons / https://phabricator.wikimedia.org/T339935
* Add outlink topic model predictions to CirrusSearch indices, this is done to support ML team in deprecating ORES - https://phabricator.wikimedia.org/T328276
Search Platform/Weekly Updates/2023-07-28

= Summary =
The team is focused on the Search Update Pipeline and Improvements to Multilingual Zero-Result Rate.
A new weekly sync meeting around the deployment of the Search Update Pipeline to Flink on k8s is helping unblock various questions and decisions (Thanks Luke!). Service Ops, Data Platform Engineering, Search Platform and Data Platform SRE are involved.
The integration tests for CirrusSearch are flaky (they have been for a long time) and causing delay in getting work merged. We might need to invest more time into reworking those instead of fixing individual errors as they come (and come again).
The Q&A session around the "unpacking analyzer" work was well received.
= What we've accomplished =
== Search Update Pipeline ==
* Flink Zookeeper cluster is up, still needs monitoring/alerts https://phabricator.wikimedia.org/T341792
* Monitor CirrusSearch update lag, we now have metrics and a dashboard tracking update lag: https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo - https://phabricator.wikimedia.org/T320408
* Add support for redirects in CirrusSearch - https://phabricator.wikimedia.org/T325315
== Improve multilingual zero-results rate ==
* Delays due to instability of integration tests for CirrusSearch
* Working on acronym/WBH write-up and while working through my explanation I realized I missed an edge case for the acronym_fixer that affects Brahmic scripts (Bengalis, Hindi, Khmer, Thai, etc.). Need to do a few more checks and then I'll have another small patch up, which will delay reindexing a little, alas. https://phabricator.wikimedia.org/T170625
== Operations ==
* Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki, related to problematic error handling in HTTP connection - https://phabricator.wikimedia.org/T341058
* Reimage wdqs20[13-22] servers to Bullseye - https://phabricator.wikimedia.org/T328325 / https://phabricator.wikimedia.org/T331300
* Undeleted Wikidata items do not reappear in WDQS: Wikidata was not sending revision_id for undelete events, this is fixed in EventBus - https://phabricator.wikimedia.org/T341905
* Investigate WDQS categories update failures on Bullseye hosts - https://phabricator.wikimedia.org/T342060
Search Platform/Weekly Updates/2023-08-04

= Summary =
The team is focused on the Search Update Pipeline and Improvements to Multilingual Zero-Result Rate.
= What we've accomplished =
== Search Update Pipeline ==
* Prep work and fixes required before implementing support for page re-render - https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/8
* Dashboard created to visualize inconsistencies between Mediawiki and search indices. It isn't clear yet how we can use this data as the basis for an SLO, we need more data points - https://superset.wikimedia.org/superset/dashboard/451/?native_filters_key=rY5v_sSw_7HBDwck-cepsTza3yzYSckh6HxJ7M6ZyrKcy9R5p3oW48lYXIb7MtKi
== Improve multilingual zero-results rate ==
* Ongoing work on standardization of ASCII folding and ICU folding - https://phabricator.wikimedia.org/T332342
== Create project plan for WDQS graph splitting ==
* Exciting early results from the WMDE analyst: https://phabricator.wikimedia.org/T342111 (TL/DR extracting scholarly article using a very simple rule is almost a 50-50 split with a very negligible number of overlapping tiples ~200k).
== Misc ==
* Removing dependencies in CirrusSearch integration tests - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/942620
Search Platform/Weekly Updates/2023-08-11

= Summary =
Mongolian is more complex than expected. Good read on https://threadreaderapp.com/thread/1407782738413817858.html if you want to follow the kind of challenges we have supporting languages in Search!
Work on metric collection for Search SLO is starting, with already a few metrics collected. We need to get up to speed on Vue.
Search update pipeline is moving forward. We're not confident that we will have a deployment strategy ready by the time the code is complete, please keep your fingers crossed. We found interesting complexity in how we handle search for files: Mediawiki automatically integrates with files on Commons, so when the same file is present both on the local wiki and on commons, strange things might happen. The logic in the current update pipeline is broken in various ways. This seems like a rare enough issue that we will probably just drop the feature from the new pipeline, after validating the rarity of the use case. The project manager for Commons has been notified.
= What we've accomplished =
== Finalize Search SLOs ==
* A number of metrics are collected for the Search SLOs - https://phabricator.wikimedia.org/T335499
== Create project plan for WDQS graph splitting ==
* Analysis of the split using simple instanceof rules done - https://phabricator.wikimedia.org/T342111
== Operations ==
* Migration of WCQS to Debian Bullseye completed, some more servers need to be migrated for WDQS - https://phabricator.wikimedia.org/T343124
== Misc ==
* Our integration tests have flagged a regression in Mediawiki Core related to externallinks. While those tests are a pain to maintain, it seems that they are useful, and not only for our team! - https://gerrit.wikimedia.org/r/c/mediawiki/core/+/945643
* Maintenance on our integration tests to unblock upgrading to more recent nodejs version - https://phabricator.wikimedia.org/T300793
Search Platform/Weekly Updates/2023-08-18

= Summary =
The Search Update Pipeline (SUP) is close to be feature complete and ready for a full scale test of production of changes. The deployment on a production environment is unlikely to be completed this quarter and the migration of all wikis will spill over to Q2. In particular, discussion about Kafka usage takes time and coordination between multiple teams (see https://phabricator.wikimedia.org/T341625).
A few team members are on vacation this week, reducing the capacity.
= What we've accomplished =
== Search Update Pipeline ==
* Schema of update documents is published - https://phabricator.wikimedia.org/T317202
== Search SLOs ==
* Some metrics already collected (MediaSearch, autocomplete, Search Result Page) - https://phabricator.wikimedia.org/T335499
== Operations ==
* WDQS Bullseye Migration 16/27 done - https://phabricator.wikimedia.org/T343124
Search Platform/Weekly Updates/2023-08-25

= Summary =
Search Update Pipeline is on track to a deployment on production for end to end testing, but it is unlikely that a full deployment on all wikis will be done before the end of Q1.
All major decisions in terms of deployment dependencies have been decided, with approval from supporting teams (which kafka cluster to use, what is the shape of events, storage of state in Swift / Zookeeper). We are experimenting with keeping a decision log on wiki: https://wikitech.wikimedia.org/wiki/Search/Update_Pipeline#Decision_records. Major thanks to Service Ops, Data Persistance and Data Engineering teams for their support!
= What we've accomplished =
== Search Update Platform ==
* Decision on usage of kafka-main - https://phabricator.wikimedia.org/T341625
== Operations / SRE ==
* Multiple reboots for Bullseye and Buster - https://phabricator.wikimedia.org/T344671 / https://phabricator.wikimedia.org/T344587
* Migration to Debian Bullseye for W[CD]QS servers (29 / 35 done), wdqs[1003-1007,1009].eqiad.wmnet still needs migration, all other servers are completed - https://phabricator.wikimedia.org/T343124
* Failure of the update pipeline for WCQS, related to someone editing the Main_Page which is in the entity namespace. Situation is currently resolved, but further work is necessary to ensure that this does not happen again - https://phabricator.wikimedia.org/T344882
* Lexeme dump failed this week, which failed some downstream processing. This has been manually recovered. No further action will be taken to improve stability, we're waiting on the general work on dumps.
* Elasticsearch reindex was started 10 days ago, but has only processed up to enwiki. This is much slower than usual, some investigation is needed. Note that this does not impact users, but reduces our ability to deploy changes to analyzer chains.
== Misc ==
* Support data analysis for the ChatGPT plugin to help find what topics are being returned.
Search Platform/Weekly Updates/2023-09-01

= Summary =
Not much happening this week.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Ongoing investigation in the increased indexing cost related to the new analysis configuration
== Search Update Pipeline ==
* We now have metrics on duplicate files in Search results (~300 duplicates/day vs ~30'000'000 search/day), which is low enough that we can ignore this use case at the moment. https://grafana-rw.wikimedia.org/d/hOONJnkIz/cirrus-search-deduplication - https://phabricator.wikimedia.org/T341227
== Misc ==
* Permissions are now correctly set on new snapshots discovery.wikibase_rdf https://phabricator.wikimedia.org/T342416
* New elasticsearch dev image is published, with the latest version of our custom plugins - https://phabricator.wikimedia.org/T344841
* We worked with Addshore to publish a new Wikidata Query Service journal, which simplifies the process for people who want to run their own query service. See the write up by Addshore: https://addshore.com/2023/08/wikidata-query-service-blazegraph-jnl-file-on-cloudflare-r2-and-internet-archive/ - https://phabricator.wikimedia.org/T344732
Search Platform/Weekly Updates/2023-09-08

= Summary =
We are close to complete the feature work on the Search Update Pipeline. We will shift our focus on deployment, which is unlikely to be completed this quarter (at risk).
The latest batch of improvements to zero-result rate seem to be working well, but have a significant impact on the indexing time. We need some work to optimize this.
= What we've accomplished =
== Search Update Pipeline ==
* Ongoing work on deployment helm charts
* routing writes to various elasticsearch clusters
* integration of Flink with Zookeeper
* Decision made on how to handle deduplication of events
== Improve multilingual zero-results rate ==
* Ongoing work on performance optimization of the new analysis chains. The current improvements in analysis do improve the functionality, but it comes at a fairly high cost during re-indexing. We need to find the right balance here.
== Finalize Search SLOs ==
* Patch for SLO collection from the SearchVue extension is blocked on review: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/SearchVue/+/948166
== Misc ==
* Reimport the image suggestions dataset https://phabricator.wikimedia.org/T345545
Search Platform/Weekly Updates/2023-09-15

= Summary =
We now track all the indicators we need for Search SLO. Next step, we will create dashboards.
The work on Search Update Pipeline needs to address a few remaining features, in particular dealing with late events and deduplication of events is a bit more complex than expected. We're moving forward on testing the deployment.
Performance improvements on Improve multilingual zero-results rate is ongoing.
We've finally removed the use of "whitelist" in the Wikidata Query Service code base, a good step in using inclusive language in our work.
= What we've accomplished =
== Search Update Pipeline (SUP) ==
* We're dropping support for Java 8 in the SUP. This will enable us to simplify the code a little bit (dropping compatibility libraries that we are currently using) and allow us to use newer features of Java. On the other hand, this will prevent us for deploying SUP on Yarn.
* We are being hit by a bug in Flink (https://issues.apache.org/jira/browse/FLINK-28758) that prevents us from properly testing the deployment of Flink with Zookeeper. This is raising the priority of migrating the WDQS Streaming Updater from older Kafka / Flink integration to newer APIs - https://phabricator.wikimedia.org/T326914
== Improve multilingual zero-results rate ==
* Smarter handling of acronyms for word_break_helper in language analyzers. Some performance optimization still need to be addressed. Write up on https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#word_break_helper_and_Acronyms_(T170625) - https://phabricator.wikimedia.org/T170625
* Ongoing performance testing of the various harmonisations - https://phabricator.wikimedia.org/T346051
== Search SLOs ==
* All defined SLI are tracked, we still need dashboards - https://phabricator.wikimedia.org/T335499
== Operations ==
* Migrate search-loader hosts to Bullseye or later - https://phabricator.wikimedia.org/T346039
* Allow WDQS federation with data.nlg.gr. Note that federation does not actually work, since data.nlg.gr has specific HTTP header requirements that we are not going to support. If requirements are relaxed, federation should start working - https://phabricator.wikimedia.org/T337296
* Retune enwiki_content shard settings - https://phabricator.wikimedia.org/T343820
== Misc ==
* Removed the usage of "whitelist" in Wikidata Query Service code, replacing it with "allowlist" - https://phabricator.wikimedia.org/T344284
* Partial fix to file uploads not being indexed properly. The root is a race condition in Mediawiki, which is identified and documented, but that isn't going to be fixed by our team (https://phabricator.wikimedia.org/T344285). Our partial fix reduces the frequency of this issue, but does not solve it entirely - https://phabricator.wikimedia.org/T342562
Search Platform/Weekly Updates/2023-09-22

= Summary =
We're moving from developing features to polishing then on the Search Update Pipeline. And testing the various requirements for our production deployment. Deployment of the Search Update Pipeline and migration of all wikis to the new pipeline will happen next quarter.
We are working through optimizing the performance of our improvement to multilingual zero-results rate.
= What we've accomplished =
== Search Update Pipeline ==
* Ongoing testing of Flink / Zookeeper integration - https://phabricator.wikimedia.org/T342149
* Ongoing refactoring to extract common configuration and graph factories
* Alignement of WDQS Streaming Updater with current best practices - https://phabricator.wikimedia.org/T326914
* Add a way to filter input events per wiki - https://phabricator.wikimedia.org/T345634
* Consider dropping support for java8 - https://phabricator.wikimedia.org/T346015
== Improve multilingual zero-results rate ==
* First draft of implementing the analysis improvemnts a
== Operations ==
* Search incident during the datacenter switch. Performance was decreased enough so that we dropped search queries. The system recovered mostly on its own once caches were warmed up. We're learning things and we will be better prepared for the next one. - https://wikitech.wikimedia.org/wiki/Incidents/2023-09-20_Elasticsearch_unavailable
== Misc ==
* Replaced the use of "whitelist" with "allowlist" in the WDQS code base (this follows a similar change to our puppet code - https://phabricator.wikimedia.org/T343856) - https://phabricator.wikimedia.org/T344284