nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2023-03-17

= Summary =
We are mostly on track to deliver what we planned for Q3.
* The Spark 3 upgrade might slip by a few weeks to finalize the deployment, but code should be ready before the end of the quarter.
* The Search Update Pipeline work was identified early as too ambitious for this quarter and will continue as planned in Q4.
* We are very close to having unpacked all of our Elasticsearch analysis chains, which will enable us to roll out future improvements to all languages at the same time.
= What we've accomplished =
== Search Update Pipeline ==
* Slow progress on validating the use of Flink k8s Operators - https://phabricator.wikimedia.org/T328675
* Added code to monitor update lag of most CirrusSearch updates (new revision/page refresh/page deletion/file uploads).
== Search Analysis==
* Unpack Romanian, Sorani Elasticsearch Analyzers. Notes on mediawiki.org: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Unpacking_Notes#Romanian_and_Sorani_Notes_(T325091) - https://phabricator.wikimedia.org/T325091
* Finished "better_apostrophe" filter for Turkish - https://phabricator.wikimedia.org/T329762
== Operations / SRE ==
* 10G networking enabled on all Elasticsearch servers, which should allow for faster maintenance and faster recovery in case of loosing servers.
== Misc ==
* Starting discussion on Q4 planning. It will be mostly about the Search Update Pipeline.
* WDQS Scaling workshops continue. The current focus is on splitting the graph. We are starting on getting a more detailed plan.
Search Platform/Weekly Updates/2023-03-24

= Summary =
The Spark 3 upgrade is close to be completed. We might have some overflow on Q4 to address the last few issues.
The focus is still on the Spark 3 upgrade over the Search Update Pipeline work. Progress on the Search Update Pipeline is minimal, but will pick up in Q4.
We're very close to have completed the unpacking of all Search Analyzers, with Brazilian and Estonian being the only analyzers left, and planned for Q4. This will enable us to implement improvements to Search uniformly across languages in the future.
= What we've accomplished =
== Spark 3 Upgrade ==
* Long list of DAGs migrated to Spark 3 (see the subtasks of https://phabricator.wikimedia.org/T318414 for details).
== Search Analyzers ==
* As part of the unpacking work, Romanian was improved by incorporating changes in how ș and ț are written. There is an interesting backstory (both on the ticket and on the write up on this work) about how technical limitations influence languages. This is also a good example of how small changes can have very positive impact on Search in specific languages (and how this aligns with aim to support underserved communities as well as mainstream languages). Note that a reindex is still needed before this work is available on wiki.
== Search Update Pipeline ==
* Made progress on deploying the WDQS streaming updater with the k8s operator, the JobManager starts, failure is when contacting swift, most probably secrets are not being passed properly. Lot of back and forth to find the right configuration.
* Extracting differences between the MySQL replicas (copied monthly to HDFS) and the CirrusSearch indices to create an SLI on the health of the update pipeline. This already uncovers a number of issues, mostly with redirects of JS/CSS and some missing updates.
== Operations / SRE ==
* Fixed WDQS Federation with AGROVOC - https://phabricator.wikimedia.org/T328625
* Added WDQS Federation with https://data.europa.eu/sparql - https://phabricator.wikimedia.org/T331271
* Cleanup /wmf/data/discovery/transfer_to_es folder in hdfs - https://phabricator.wikimedia.org/T323616
== Misc ==
* New dashboard on Search Preview usage, showing that mobile use is roughly twice that of desktop usage, likely because desktop has hovercards.
* Cindy (automated end to end testing for CirrusSearch) is having issues again. Lack of support for testing on docker with multiple wikis is a recurring issue for our team.
Search Platform/Weekly Updates/2023-04-07

= Summary =
We are starting a new quarter, with new goals. The plan is not entirely formalized yet, but:
* Now that Spark 3 migration is completed, we will focus engineering efforts on the Search Update Pipeline
* Once all search analysis chains are unpacked (should be completed shortly), we are going to address a few long standing issues, with the goal of reducing zero result rate or increasing the number of results returned for most languages.
* Defining and implementing SLOs for Search.
* Create a plan to split the WDQS graph.
= What we've accomplished =
== Search Analysis ==
* Brazilian Portuguese unpacking completed, still needs reindex -https://phabricator.wikimedia.org/T325092
== Search Update Pipeline ==
* Demonstrate how to run rdf-streaming-updater using the new flink-app helm chart - https://phabricator.wikimedia.org/T328675
* Starting to setup a Java CI pipeline on Gitlab. Multiple roadblocks on getting the proper docker images in place - https://phabricator.wikimedia.org/T326318
== Misc ==
* Start gathering hardware requests for next fiscal year - https://phabricator.wikimedia.org/T334207
* Fix broken multi-byte characters presented in the end of search snippets (third party usage of Mediawiki) - https://phabricator.wikimedia.org/T333653 (Thanks Func86 for the fix!)
* Discussion during the Search Platform Office Hours around the support of OpenRefine reconciliation API and the documentation of the WDQS data reload process.
Search Platform/Weekly Updates/2023-04-22

= Summary =
The focus this week has been on annual planning. Other projects are moving along.
= What we've accomplished =
== Search - Analysis ==
* Blog post written about the work that has been done around unpacking the analyzers. This should be published on Diff after editorial review.
* Brazilian Portuguese has been reindexed, enabling the unpacking of the Brazilian analyzer - https://phabricator.wikimedia.org/T333398
== Search - Update Pipeline ==
* rdf-streaming-updater is now running with Flink Operators on the DSE k8s cluster. This validate the underlying deployment process, so that we can focus on functional aspects again - https://phabricator.wikimedia.org/T328675
== Operations / SRE ==
* Decommission query-preview.wikidata.org, which was used to allow our users to validate the migration to the streaming updater. This migration is completed since long enough that this exposed service isn't needed anymore. - https://phabricator.wikimedia.org/T333656
* Cookbook created to ban / unban nodes, which is required during the upcoming network switch maintenance. Having this automated reduces the potential errors. - https://phabricator.wikimedia.org/T331303
== Misc ==
* Progress made on migrating our integration tests from vagrant to docker. Some polishing is needed, but it seems to be working - https://phabricator.wikimedia.org/T333183
* Reindex of all wikis to enable a new keyword to filter pages by "length" - https://phabricator.wikimedia.org/T328332
* Optimize queries for non-English users on Wikdiata - https://phabricator.wikimedia.org/T323628
Search Platform/Weekly Updates/2023-04-28

= Summary =
Annual planning took a reasonable amount of mind share this week. Search Update Pipeline is moving forward. Some unexpected work around operations and development support.
= What we've accomplished =
== Search Analysis ==
* Blog post is in the queue to be published. Hopefully in a couple of weeks.
* Enabling new Estonian analyzer, this seems to have a significant impact - https://phabricator.wikimedia.org/T332322
== Search Update Pipeline ==
* Write a small job that copies the cirrus index data in avro to a smaller dataset written in parquet, will be used to identify inconsistencies between the cirrus index the mysql page table.
* Upgrade WDQS Streaming Updater to Flink 1.16, testing is done, deployment to production is still pending - https://phabricator.wikimedia.org/T289836
== Search SLO ==
* Created phab tickets to track the work - https://phabricator.wikimedia.org/T335576
== Operations / SRE ==
* Hardware requests for next fiscal year
== Misc ==
* Add a new keyword to filter pages based on their "length", in support of article suggestion - https://phabricator.wikimedia.org/T328332
* Browser tests for CirrusSearch migrated from Vagrant to Docker. Documentation is updated https://wikitech.wikimedia.org/wiki/Cindy_The_Browser_Test_Bot. This still does not integrate as well as we would like into our CI environment (the tests run in our own WMCS instance), but it should significantly reduce the amount of ongoing work needed to keep the tests green - https://phabricator.wikimedia.org/T333183
* Search autocomplete was broken on enwiki for a couple of weeks. The problem has been fixed and new alert has been created. https://phabricator.wikimedia.org/T327199
Wrote a small patch to cleanup error handling of CirrusSearch writes (work initially started by Emmanuel).
* Tried to figure out what is going on with WCQS/commons next year, if anything.
* Met with Growth team to talk about product collaboration between reader/editor needs and search, disc, brows. There is a lot of shared thinking and overlap, and hopefully there's some room to collaborate with the team on reader discovery stuff.
Search Platform/Weekly Updates/2023-05-05

= Summary =
Planning, budget, vacations, sickness, not a lot of concrete outputs this week.
= What we've accomplished =
== Search Analysis ==
* Blog post about he work on language analysis: https://diff.wikimedia.org/2023/04/28/language-harmony-and-unpacking-a-year-in-the-life-of-a-search-nerd/. Lot of positive feedback and more ideas for the future.
== Operations / SRE ==
* Using a different library for WDQS data xfers https://phabricator.wikimedia.org/T321605
* Bootstrap WDQS on Bullseye https://phabricator.wikimedia.org/T331300
* Decommission query-preview.wikidata.org: this endpoint was exposed to allow our communities to test the WDQS streaming updater and not needed anymore. The full cleanup of any configuration related to this temporary service is now completed. https://phabricator.wikimedia.org/T333656
== Misc ==
* Mike had conversations with Chris (ML team) about Search and ML. There are interesting ideas around LLM-powered search experience. And collaboration opportunities, either around infrastructure or around features.
* Presentation from Pratt students on how editors use search. Nothing profoundly new, but a few notes on problems we already know:
** People don't know about the search function / page, or don't know how to find it.
** Advanced search is confusing, for example "search in:" is about namespaces, but some people don't even know what a namespace is. They expect a filter / facet function instead.
** Search results don't seem as relevant as Google when searching Wikipedia.
* Capex requests (server refresh and new servers for capacity expansion) completed. It looks like we are overall over budget as an organization, we need to cut some of the spending.
Search Platform/Weekly Updates/2023-05-12

= Summary =
We had a major lag issue on Wikidata Query Service codfw cluster not being updated (see below for details). This took significant time and focus to resolve. Work on the Search Update pipeline continues, with conversations with other consumers of the event stream to implement changes that are needed for search.
= What we've accomplished =
== Search - Analysis ==
* Starting work on putting in place the required infrastructure to measure the planned improvements.
* Estonian reindexing complete! It enabled a new stemmer (and other bits) and had a pretty big impact, roughly similar to the Bengali new stemmer—1 in 6 previous Estonian Wikipedia zero-results queries get results. Almost 1 in 3 of non-zero-results queries get more results, and more than 1 in 8 queries had their top result change. - https://phabricator.wikimedia.org/T335704
* A user asking about stop words prompted the creation of documentation - https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Stopwords_2023
== Operations / SRE ==
* Ironed out the last performance issues of the WDQS SLO dashbaord - https://phabricator.wikimedia.org/T313751
* We had a major lag incident on WDQS, where the codfw cluster stopped being updated from Friday to Monday. While it was still serving queries, it was answering them with outdated data. The incident report is still being written. Our current understanding is that the whole thing was first triggered by a bug in Flink (https://issues.apache.org/jira/browse/FLINK-22597), probably leading to higher memory usage during recovery, which led to the service not recovering as it should have. There are also a number of learnings on how to manage those kinds of incidents, where WDQS is serving traffic, but with outdated data. Follow up on https://phabricator.wikimedia.org/T336134 and https://wikitech.wikimedia.org/wiki/Incidents/2023-05-05_wdqs_not_updating_in_codfw
* Improve ban cookbook https://phabricator.wikimedia.org/T331303
* Decommission airflow (WIP) https://phabricator.wikimedia.org/T333697
* Review hardware requests for next fiscal year - https://phabricator.wikimedia.org/T334210
== Misc ==
* Some questions on wikitech-l about why Search isn't using word embedding / vector search (https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/PJGRN2H2CUSACZH4QSI2VOV3WBDZ3J6O/). The search platform team has been called out a few times on that thread. We are working on a reply - https://docs.google.com/document/d/1XLzHNqwEyD42mw3Zj4boJb9grh56UWhGYyJrm-g1VOg/edit#
Search Platform/Weekly Updates/2023-05-18

= Summary =
Working on the post-mortem of the WDQS outage, Search Update pipeline, and optimizing Wikibase index settings.
= What we've accomplished =
== Search - Analysis ==
* Continuing data analysis for apostrohpe-like characters (T315118). There are 22 candidate characters, and they get treated differently by different tokenizers (the Hebrew tokenizer straight up converts 5 of them to apostrophes—including Hebrew geresh—which I never noticed before!) and by ICU normalization and ICU folding. 
== Operations / SRE ==
* Ironed out the last performance issues of the WDQS SLO dashbaord - https://phabricator.wikimedia.org/T313751
* Deploying new Turkish analyzer https://phabricator.wikimedia.org/T332355 
* Deploy newer version of Flink to rdf-streaming-updater, this should eliminate the bug that caused last week's outage. https://phabricator.wikimedia.org/T334244
* Audit NIC firmware in preparation for Buster EOL https://phabricator.wikimedia.org/T331297
Search Platform/Weekly Updates/2023-05-23

= Summary =
Another WDQS incident this week disrupted our flow of work.
Dealing with page redirect in the context of the Search Update Pipeline is more complex than expected, and involves multiple teams (data engineering, mediawiki core, ML). Hopefully that additional work will benefit more teams, in particular ML.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* documentation and some implementation of the framework to evaluate impact https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes
== Search Update Pipeline =
* Discussions with Data Engineering and Mediawiki core about page redirects - https://phabricator.wikimedia.org/T325315#8871319
== Finalize Search SLOs ==
* Cleanup of the final list of SLOs - https://docs.google.com/document/d/1gYROXo8Fl7JSxReHAVI22EhcPvG-INVkq79a1C3tfK0/edit#heading=h.2v65rh3w6ii0
== Operations / SRE ==
* Dealing with the WDQS outage - https://wikitech.wikimedia.org/wiki/Incidents/2023-05-23_wdqs_CODFW_5xx_errors
* Fixed prometheus metrics not being reported by flink-1.16 - https://phabricator.wikimedia.org/T336872
== Misc ==
* Trying to make sense to the perceived limitations of Search when being used in the context of a ChatGPT plugin
* The language team finished the work on (https://phabricator.wikimedia.org/T322284) sent a patch to enable this new config. This should allow us in the end to quickly switch traffic from one DC to another (unblocking https://phabricator.wikimedia.org/T143553)
* Turkish is reindexed to enable unpacking, with write up on mediawiki https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Unpacking_Notes#better_apostrophe_Impact_on_Turkish_Wikipedia_(T337064) - https://phabricator.wikimedia.org/T337064
Search Platform/Weekly Updates/2023-06-02

= Summary =
We spent some time this week to deal with the aftermath of our recent WDQS outages. In particular, we are now equipped to diagnose and block problematic queries faster.
Our work on SLOs for Search is unlikely to be completed this quarter. We now have a good working definition of what we want to measure, but we will need help to implement metric collection and create the appropriate dashboards. This is unlikely to be done before the end of the quarter.
= What we've accomplished =
== Search Analysis ==
* Reindex Estonian wikis to enable new unpacked analyzer - https://phabricator.wikimedia.org/T335704
* Deploy Turkish Analyzer Plugin - https://phabricator.wikimedia.org/T332355
* Reindex Turkish wikis to enable improved apostrophe handling, full write up on Mediawiki https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Unpacking_Notes#better_apostrophe_Impact_on_Turkish_Wikipedia_(T337064) - https://phabricator.wikimedia.org/T337064
* [EPIC] Unpack all Elasticsearch analyzers - https://phabricator.wikimedia.org/T272606
== Search Update Pipeline ==
* The W[DC]QS flink job was upgraded to flink 1.16.1 in all k8s environments - https://phabricator.wikimedia.org/T334244 / https://phabricator.wikimedia.org/T289836
* Finalizing EventBus patch to encode redirect link informations https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/913030
* Deep dive into different types of links, to come up with a shared schema that supports almost all of them.
== WDQS graph splitting ==
* Started writing a small doc about a possible plan
== Search SLOs ==
* SLIs for Search SLOs are defined. Next steps: ensure metrics are collected, and create dashbaords. https://phabricator.wikimedia.org/T335498 (see parent task for more context)
== Operations / SRE ==
* Fixed missing metrics for WDQS rdf-streaming-updated - https://phabricator.wikimedia.org/T336872
* Investigate puppet failure on cirrus-integ03.search.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T336519
* Missing Cirrussearch dump (enwiki and wikidata) - https://phabricator.wikimedia.org/T330936
* Special:Search broken on Beta Wikidata for entity namespaces - https://phabricator.wikimedia.org/T335873
* on-wiki search is failing to find relatively newer titles on enwiki, adding a new check to alert if title suggester isn't updated every 24h - https://phabricator.wikimedia.org/T327199
* Federated queries to Lingua Libre time out in the Commons query service - https://phabricator.wikimedia.org/T334470
* Add https://opendata.aragon.es/sparql to the list of federated endpoints for WDQS and WCQS - https://phabricator.wikimedia.org/T334823
* Reduce the load of CirrusSearch update jobs on MW jobrunners - https://phabricator.wikimedia.org/T336698
== Misc ==
* Ongoing discussions around Search and ChatGPT. It seems that the current search ranking isn't appropriate for the needs, we need to clarify what the problem is and what we could be doing to help.
Search Platform/Weekly Updates/2023-06-09

= Summary =
* Hiring a new senior SRE for the Data Platforms SRE team is our highest priority.
* Work on language analysis and search update pipeline is moving forward.
= What we've accomplished =
== Language analysis harmonization ==
* Analysis of the use of multiple apostrophe like character on wiki completed, a patch will be created to harmonize the language analysis across languages. I encourage you to have a look at the write up for more context: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes
== Search Update Pipeline ==
* We are getting closer to have data on the inconsistencies between MySQL and CirrusSearch. Preliminary data indicates that there are not many inconsistencies in general, with some potential issues related to lexemes - https://phabricator.wikimedia.org/T328330 / https://phabricator.wikimedia.org/T338255
* Modified EventBus and related schema to handle page redirects
== Operations / SRE ==
* Improvements to the WDQS data transfer cookbook, which will make data reload more stable - https://phabricator.wikimedia.org/T321605
* Reboot of multiple servers for kernel upgrade completed - https://phabricator.wikimedia.org/T335835
== Misc ==
* Multiple interview for the Senior SRE position on the Data Platforms SRE team and the Senior Engineering Manager position on the Data Engineering team - https://boards.greenhouse.io/wikimedia/jobs/5070612?gh_src=aae3b0fb1us / https://boards.greenhouse.io/wikimedia/jobs/4990750?gh_src=4adc4c9d1us
* Discussions on the use of our search engine use in the context of a ChatGPT plugin. The initial understanding was that our search engine does not provide good enough ranking to be used. After reviewing the queries and results, it seems that it does indeed do a good enough job. We could address some of the limitations by improving the queries sent to the search engine.
Search Platform/Weekly Updates/2023-06-16

= Summary =
We're getting close to the end of the quarter. Improvements to the multilingual zero results rate for Search has made progress, with most of the infrastructure to evaluate changes in place and good progress on the handling of apostrophe. The project plan for a first experiment about splitting the WDQS graph is in good shape.
As expected, the Search Update Pipeline work will overflow to next quarter. As identified earlier in this quarter, the Search SLI have been defined, but the full implementation will not fit in this quarter and has been moved to the next one.
The Data Platform Engineering virtual offsite happened this week. This marks a transition point into our new team organization. In particular, have a look at this talk if you want an overview of the systems that the Search Platform manages: https://drive.google.com/file/d/1APye8qEdYP_OKBCnb0jsqSCDemobuUBT/view
= What we've accomplished =
== Search Update Pipeline ==
* Identified the kind of updates that flowed into cirrusSearchLinksPrioritized (these are "purge" api action that bots can trigger to force a LinksUpdate). We can now move those updates to a non-prioritized queue - https://phabricator.wikimedia.org/T320408
* Added event-time, meta.dt, meta.domain fields to ML revision-score based events - https://phabricator.wikimedia.org/T267648
== Improve multilngual zero-results rate ==
* Started looking into aggressive_splitting - https://phabricator.wikimedia.org/T219108
== Operations / SRE ==
* optimization of data transfer for WDQS - https://phabricator.wikimedia.org/T321605
* added federation with UNESCO Sparql endpoint - https://phabricator.wikimedia.org/T335994
* added federation with BNCF Sparql endpoint - https://phabricator.wikimedia.org/T336709
* migrate W[CD]QS puppet code to use profile::java - https://phabricator.wikimedia.org/T264181
== Misc ==
* Fix capitalization issue on search result page - https://phabricator.wikimedia.org/T335551
* update current search update pipeline to changes is ML generated revision scores - https://phabricator.wikimedia.org/T333468