nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2023-09-29

= Summary =
Last week of this quarter!
A Kafka outage led to WDQS not processing updates. This had some impact on our overall work, in particular we could not deploy some changes for testing. This also helped us discover an issue with lack of isolation between test and production pipelines.
Search Update Pipeline work is focused on deployment and operational improvements. We're waiting for a first production deployment to validate the full integration.
Improve multilingual zero-results rate work is about improving performances while not degrading the accuracy of the new filters. We will need a new plugin to distribute this work.
Are goals for next quarter (Oct-Dec) are being finalised and will be published shortly.
= What we've accomplished =
== Search Update Pipeline ==
* Upgrade to flink 1.17 and refactoring to extract code shared between aggregator and indexer - https://phabricator.wikimedia.org/T346719
* Testing various Flink operations on the DSE experimental cluster https://phabricator.wikimedia.org/T342149
* Improve concurrency limits configuration of the wdqs updater - https://phabricator.wikimedia.org/T346456
* Create a schema for fetch failures - https://phabricator.wikimedia.org/T317609
== Improve multilingual zero-results rate ==
* Decision on how to distribute new filters in a new plugin - https://phabricator.wikimedia.org/T346051#9204248
* Various accuracy and performance tests of the performance optimized filters vs the original regex. Both are looking good.
== Operations ==
* Small WDQS lag outage (<2hours), initially caused by mirrormaker failing between kafka-main and kafka-jumbo. WDQS was still available but not updated with the latest edits on Wikidata - https://wikitech.wikimedia.org/wiki/Incidents/2023-09-27_Kafka-jumbo_mirror-makers
* As part of the above outage, we discovered a lack of isolation between test and production pipelines - https://phabricator.wikimedia.org/T347515
* Fixed issue causing false-positive alerts on the non-active CirrusSearch datacenter due to low traffic volume: https://phabricator.wikimedia.org/T347341
* Tune process_sparql_query_hourly so that it does not get killed by yarn - https://phabricator.wikimedia.org/T347333
Search Platform/Weekly Updates/2023-10-06

= Summary =
We're starting a new quarter. Our goals for this quarter are almost ready and will be published on wiki shortly (keep an eye on https://wikitech.wikimedia.org/wiki/Search_Platform/Goals).
Overall, this was a short week, due to Wikimedia Connect. We're making good progress towards deploying the Search Update Pipeline, with the testing of standard operations completed. We've identified a number of performance improvements to our improvements to multilingual zero-results rate. And we're getting started on experimenting with WDQS graph split.
= What we've accomplished =
== Search Update Pipeline ==
* We have tested all relevant operations, we are ready for a production deployment of the Search Update Pipeline on Flink, with k8s operators - https://phabricator.wikimedia.org/T342149
* Migration of the WDQS updater to use newer Flink connectors
* Started to work on better isolation of wdqs updater error streams, quick patch to disable them to unblock testing the flink-k8s-op, better solution still WIP - https://phabricator.wikimedia.org/T347515
== Improve multilingual zero-results rate ==
* Performance optimization in progress. In particular, consolidating character mapping brings a 9.3% improvement to indexing times, implementing custom mapping code instead of the heavy weight elasticsearch machinery is ~50% faster.
* A new Elasticsearch plugin will be created to isolate this and allow for easier rollout
== WDQS Graph Split ==
* New servers being provisioned - https://phabricator.wikimedia.org/T347505
== Misc ==
* New partman recipe for cloudelastic, which will allow better use of disk space - https://phabricator.wikimedia.org/T342463
* Search Platform office hours had Andrea (https://wikitech.wikimedia.org/wiki/User:AndreaWest) joining and good discussions around WDQS, split the graph, the future of RDF and SPARQL.
* Document process for getting JNL files/consider automation - https://phabricator.wikimedia.org/T347605
* Review of Elasticsearch incident during DC switchover. We've identified an issue with RESTBase that might have contributed to overloading the cluster - https://wikitech.wikimedia.org/wiki/Incidents/2023-09-20_Elasticsearch_unavailable
Search Platform/Weekly Updates/2023-10-13

= Summary =
Search Update Pipeline (SUP) is getting ready for a first test deployment. We're working out through a list of small improvements needed to make the application production ready (various dependencies upgrades, better routing of requests, ...). The main focus is on getting WDQS Updater and SUP deployed, using the same underlying principles.
Work on Splitting the WDQS graph is starting. The test servers are in place, only missing data. We have a communication to our users ready to be out by next Monday. We are starting on the mechanics of splitting the graph, the current focus is on getting a development environment in place.
= What we've accomplished =
== Search Update Pipeline ==
* Almost all of operations are tested, we want to run a last test to simulate an upgrade of the k8s cluster itself - https://phabricator.wikimedia.org/T342149
* Ongoing work to deploy a test instance in k8s, we are learning about k8s, tweaking configurations at various level, checking connectivity, etc - https://phabricator.wikimedia.org/T347075
* Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink - https://phabricator.wikimedia.org/T326914
* Filtering added to avoid duplicate updates in a cross datacenter context - https://phabricator.wikimedia.org/T344357
* Make HTTP route configuration more flexible - https://phabricator.wikimedia.org/T345612
* Adding support for page re-render - https://phabricator.wikimedia.org/T325565
* Upgrade to flink 1.17.1 - https://phabricator.wikimedia.org/T346719
* Reference latest streams/schemas - https://phabricator.wikimedia.org/T346895
== WDQS graph splitting ==
* Ongoing work to get test servers ready for our experiment. Basic configuration is almost done, but no data is loaded yet - https://phabricator.wikimedia.org/T347505
== Improve multilingual zero-results rate ==
* Wrestling with Java to get filter parameters working. I've never actually had params for any of the plugins/filters I've made before. Talked about it as a future option, but it never actually came up. Need to put up a patch and a write up.
== Operations ==
* Manual reconciliation of a few deleted Wikidata items in WDQS - https://phabricator.wikimedia.org/T342593
* Cleanup of Swift buckets used for the WDQS Streaming Updater, recovering almost 1TB of space - https://grafana.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?from=1696938873511&orgId=1&to=1696952558553&viewPanel=14
* Image suggestion pipeline was blocked due to missing data. Current situation is resolved, but more discussion with Data Engineering team is needed to have a coherent strategy on how we deal with data quality and operatibility of our data pipelines - https://phabricator.wikimedia.org/T347832
Search Platform/Weekly Updates/2023-10-20

= Summary =
We've sent our first public communication about the WDQS graph split experiment (https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update). The project is starting, with infrastructure already well on the way and software engineering making sure we have a working development environment. Things should pick up speed soon.
Search Update Pipeline is being deployed on our staging wikikube environment, we're ironing out the last configuration, network routes, and optimizing the code for production.
Search analysis work is being optimized. Since deployment and reindexing is an expensive operation, we want to bundle all changes together.
= What we've accomplished =
== Search Update Pipeline ==
* All Flink operators are named and identified (part of the production recommendations) - https://phabricator.wikimedia.org/T346717
* Max parallelism is set on all operators with a state (part of the production recommendations) - https://phabricator.wikimedia.org/T346718
* We decided that we've learned enough about Flink deployment by using the WDQS updater as a test application. While we will still need to migrate it, we decided to remove this dependency from the Search Update Pipeline work. https://phabricator.wikimedia.org/T326409
== WDQS graph splitting ==
* Communication about the project sent to our communities. No reaction yet - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/October_2023_scaling_update
* We've contacted some people from Scholia, they are open to discussion, we will schedule a meeting.
* Tests servers are configured, initial (full) dataset is being loaded (which could take a few weeks / months)
* Development environment in place, we can start working on the graph split job itself.
== Operations ==
* Failure of WDQS updater after switching to the newer kafka connector API. The cause is related to a bug in older versions of kafka that forget consumer offsets on idle streams - https://phabricator.wikimedia.org/T349147
Search Platform/Weekly Updates/2023-11-03

= Summary =
Search Update pipeline: we're resolving deployment issues one by one, we're making progress, but we don't know yet what all the issues are going to be (https://phabricator.wikimedia.org/T347075)
Improve multilingual zero-results rate: Performance improvements are done (waiting to be deployed). Work continues on fixing tokens that are erroneously split by ICU.
WDQS graph splitting: work in picking up speed. We have a functional development environment and a first version of code that splits the graph. Collaboration with Scholia and with WMDE is picking up speed as well.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Refactoring of slow analysis components have been merged (still need to be deployed and reindexed). In the process, we now have better ways to catch performance issues earlier during analysis work - https://phabricator.wikimedia.org/T346051
* Work has started on ICU token repair (ICU erroneously splits token based on script, so words that contain multiple scripts are broken up). Good results for a first iteration of the implementation - https://phabricator.wikimedia.org/T332337
== WDQS graph splitting ==
* Meeting with Scholia to explore pain points and start collaboration - https://etherpad.wikimedia.org/p/20231024-wdqs-graph-split
* Experimentation on using Federation with Scholia queries, using the current WDQS endpoint. This is part of our collaboration with Scholia on better understanding the use case and how disruptive a graph split might be for Scholia - https://phabricator.wikimedia.org/T349911
== Search SLO ==
* We have a temporary dashboard (https://grafana.wikimedia.org/d/H6f-bA7Sk/rkemper-search-sli-test?forceLogin&from=1693181669857&orgId=1&to=1698246983235) that allow us to investigate what limits we want to put on those SLOs - https://phabricator.wikimedia.org/T338009
== Misc ==
* Fix search suggestions not showing exact match - https://phabricator.wikimedia.org/T347786
* Another failure of the Image Suggestion pipeline required manual intervention - https://phabricator.wikimedia.org/T349472
Search Platform/Weekly Updates/2023-11-10

= Summary =
Search Update Pipeline: we're resolving issues as they come up in deployment. We still don't have a stable data pipeline, but we're getting there.
WDQS graph split: We've started investigating the different test framework option so that we can validate the split on a subset of queries. The Graph split itself needs a bit more work on some corner cases, but we're getting there. Collaboration with WMDE is working well, we're getting support in terms of analysis.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* work started on repair multi-script tokens split by the ICU tokenizer - https://phabricator.wikimedia.org/T332337
== Search Update Pipeline ==
* Number of fixes and improvements to get production ready:
** Filter/drop canary events from update stream - https://phabricator.wikimedia.org/T349591
** logging: add correlating information - https://phabricator.wikimedia.org/T348211
** pick kafka consumer offset defaults - https://phabricator.wikimedia.org/T348112
** Use flink's AsyncRetryStrategy instead of custom retry logic - https://phabricator.wikimedia.org/T347545
** Fetch: Handle Timeout of AsyncAwaitOperator - https://phabricator.wikimedia.org/T347543
== WDQS graph splitting ==
* Initial data load of the full graph on test servers is still in progress (started on 2023-10-24). Roughly 70% done so far, but we expect the process to slow down over time - https://phabricator.wikimedia.org/T347504
* We will be hosting a session about Graph Split at Wikidata Data Modeling Days -https://www.wikidata.org/wiki/Wikidata:Events/Data_Modelling_Days_2023#Sessions
* Initial version of the graph split code, needs more work to support corner cases - https://phabricator.wikimedia.org/T347989
== Operations ==
* Java security restarts https://phabricator.wikimedia.org/T350703
== Misc ==
* rdf-streaming-updater migration: now running in staging https://phabricator.wikimedia.org/T349095
* Ensure mjolnir can work on Python 3.9 or later (as part of the migration to Debian Bullseye) - https://phabricator.wikimedia.org/T346373
* Some of our CI pipelines have been migrated to Java 11 - https://phabricator.wikimedia.org/T350587
Search Platform/Weekly Updates/2023-11-17

= Summary =
Year-end vacation season and deployment freeze are coming, work is expected to somewhat slow down until January. We're still on track to deliver what we expected.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Finished heuristics for merging Type and Script attributes (e.g., <ALPHANUM>/Latin + <NUM> = <ALPHANUM>/Latin; Latin + Cyrillic = Unknown; etc.). Abandoned making Script attributes merging more configurable (e.g., keep first, keep last, count characters), so every mixed token gets "Unknown" (we're limited to ICU script types, otherwise I'd go with "Mixed") - https://phabricator.wikimedia.org/T332337
* Lots of thinking about configurability of scripts & types to merge (e.g., don't merge <EMOJI> types; only merge <ALPHANUM> types; don't merge CJK scripts, etc.). Still thinking about "numbers only" option (because current behavior is an error wrt UAX #29) - https://phabricator.wikimedia.org/T332337
== WDQS graph splitting ==
* Investigated TFT (https://github.com/BorderCloud/TFT) and found that it might not suit our needs for the graph splitting analysis, it does not have a handy ways to generate "test scenarios" and seems to be designed to work only against the test created by the w3c working group (https://github.com/w3c/rdf-tests). It's written in PHP and I don't think it'd be wise to add such functionality there - https://phabricator.wikimedia.org/T349519
* Investigation started on Iguana (https://iguana-benchmark.eu/), looks more promising, but needs more investigation before decision.
* Scholarly Article Split job is now deployed via Airflow and generating a working graph split - https://phabricator.wikimedia.org/T347989
* Work started on converting the graph split to a format that can be exported and ingested into Blazegraph - https://phabricator.wikimedia.org/T350106
== Search Update Pipeline ==
* Starting backfilling test to validate functional correctness and that load on backend systems is appropriate - https://phabricator.wikimedia.org/T350826
* There are open questions about failure modes. Currently, some failures related to bad input data require manual intervention to recover. Automated recovery in a robust way isn't trivial. Note that at the moment, SUP has been running with production data for multiple days without issues, so failures due to data are at least somewhat rare.
* Helm charts created and validated by deployment - https://phabricator.wikimedia.org/T326328
* Improve the flink-app chart to provide more useful defaults - https://phabricator.wikimedia.org/T346315
== Misc ==
* Java restarts for security updates - https://phabricator.wikimedia.org/T350703
* Deployment of Mjolnir on Python 3.9 - https://phabricator.wikimedia.org/T346373
* Deployment of WDQS Streaming Updater with Flink / k8s Operators in staging (not in production yet) -https://phabricator.wikimedia.org/T326409
* Decommission search-loader VMs (part of the migration to Debian Bullseye) - https://phabricator.wikimedia.org/T351123
* VisualEditor's Add a link should suggest a redirect with exact case match - https://phabricator.wikimedia.org/T346920
Search Platform/Weekly Updates/2023-12-01

= Summary =
This update covers the last 2 weeks: there wasn't much progress due to holidays and vacations. With the end of year approaching, more vacations will reduce our output.
The end of year freeze implies that some of the work that is ready will still wait until January to be deployed in production. In particular, the Search Update Pipeline will wait until January to be deployed on the first production wiki.
= What we've accomplished =
==Improve multilingual zero-results rate==
* First implementation of ICU token repair is passing unit tests - https://phabricator.wikimedia.org/T332337
==WDQS graph splitting==
* Investigation of IGUANA as a SPARQL testing framework, writing a tool to convert IGUANA RDF results into a tabular CSV format - https://phabricator.wikimedia.org/T351894
* Create a tool that records and compares a set of sparql query results (in progress) - https://phabricator.wikimedia.org/T351819
* Completed evaluation of IGUANA and TFT as test frameworks. TFT is not fit for our purpose, but IGUANA might work. See ticket for more details - https://phabricator.wikimedia.org/T349519
* We've been observing performance degradation on the newer WDQS servers, which is probably linked to CPU starvation, with newer servers having more cores but slower - https://phabricator.wikimedia.org/T336443
==Search Update Pipeline==
* Pipeline has been stable for a week - https://phabricator.wikimedia.org/T347075
* The Flink Elasticsearch sink does not provide an easy way to ignore/retry individual flink failures (https://issues.apache.org/jira/browse/FLINK-32028) workaround added
* Found (and fixed) an issue in how EventBus processes changes involving redirects - https://phabricator.wikimedia.org/T350299
* Minor fixes to corner cases in the pipeline
* Create/adapt index comparing script for checking correctness - https://phabricator.wikimedia.org/T349838
==Search SLOs==
* Reviewed our current way to collect SLI and decide it is sufficient - https://phabricator.wikimedia.org/T335499
==Misc==
* Lexeme searches prefer forms over lemmas - https://phabricator.wikimedia.org/T348877
==Operations==
* Reduce the number of false positive alerts for p95 latency on Search - https://phabricator.wikimedia.org/T349340
Search Platform/Weekly Updates/2023-12-08

= Summary =
End of year is close, things are slowing down.
We've made good progress on WDQS Graph Split, with a completed split that is now being loaded onto our test servers. We've had a session at Wikidata Modeling days to present the project, it was well received.
We have a selection of test wikis for Search Update Pipeline (commons, fr, it, wikidata and testwiki) and we're updating Cloudelastic with the new pipeline as a low risk first end to end deployment.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* Integration tests and performance validation on the ICU token repair plugin. Performance overhead is between 4.5% and 6.5%, which is acceptable - https://phabricator.wikimedia.org/T332337
== Search Update Pipeline ==
* We're updating indices on Cloudelastic with the new update pipeline for our test wikis (commons, fr, it, wikidata and testwiki) - https://phabricator.wikimedia.org/T352335
== WDQS graph splitting ==
* Creation of split files that can be imported into test servers completed - https://phabricator.wikimedia.org/T350106
* Import of the split graph into test servers started - https://phabricator.wikimedia.org/T350465
* Write a tool that converts IGUANA test results into tabular data suited for analysis needs - https://phabricator.wikimedia.org/T351894
* We had a session about the graph split at the Wikidata Modeling Days (https://www.wikidata.org/wiki/Wikidata:Events/Data_Modelling_Days_2023#Sessions). The Graph Split proposal was well received. See the recording: https://www.youtube.com/watch?v=Krk1EcP0TyA
== Misc ==
* WDQS Streaming Updater has good concurrency limit configuration options - https://phabricator.wikimedia.org/T346456
* WDQS Streaming Updater has a way to tag side output events to allow reprocessing - https://phabricator.wikimedia.org/T347515
Search Platform/Weekly Updates/2023-12-15

= Summary =
We're fixing a few more bugs on the Search Update Pipeline and improving our understanding of operations.
Graph split is well on track, with data loaded into test servers and and the starting point of a test framework.
Next week is expected to be extremely low key, with multiple team members already starting their end of year vacation.
= What we've accomplished =
== Search Update Pipeline ==
* Discovered a new bug in Flink Elasticsearch integration related to a wrong estimation of the size of the update requests - https://phabricator.wikimedia.org/T353430
* Better understanding of throughput limitations of the pipeline - https://phabricator.wikimedia.org/T353460
* Fixed Envoy telemetry for SUP, we now have Envoy metrics - https://phabricator.wikimedia.org/T353224
* We now capture the http client metrics
== Improve multilingual zero-results rate ==
* Reviewing real world data for ICU tokenizer and the required repairs. Most results are good, but we might need to exclude some specific languages or scripts.
* Thinking about internal representation of allowed and denied language combinations for ICU token repair and how to specify them in the config - https://phabricator.wikimedia.org/T332337
== WDQS graph splitting ==
* We have the 3 test machines (wdqs102[234]) loaded with the data we need: full, wikidata-main, scholarly articles - https://phabricator.wikimedia.org/T350465
* Discussion with Traffic team about exposing test servers, we have an agreement on the strategy - https://phabricator.wikimedia.org/T351650
* First step on comparing queries between graph split and full graph: we have a Spark UDF that captures a sparql query output, writing a small comparison job that emits various stats comparing a set of queries queried against endpoint A and B. - https://phabricator.wikimedia.org/T351819
== Operations ==
* Check & Reset the image suggestion failed airflow task - https://phabricator.wikimedia.org/T353134
== Misc ==
* Review of Java / Scala documentation completed, as part of the JVM Languages Stewardship effort - https://phabricator.wikimedia.org/T344595