nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2023-12-22

= Summary =
Very slow week, most of the team is already on holiday.
Search Update Pipeline: We're ready to enable 3 more wikis (dewiki, frwiktionary, kuwikitionary) to our current test setup (commonswiki, frwiki, itwiki, testwiki, wikidatawiki). We will wait until after the holiday to actually enable them. Those tests are end to end, updating the production cloudelastic indices, but not yet updating the main search cluster. We are still working on understanding the throughput limitations of the pipeline, but this isn't blocking the rollout yet see https://phabricator.wikimedia.org/T353460).
WDQS Split the Graph: We had a meeting with WMDE about query analysis work, so that we can have a set of queries to validate the impact of a graph split. This work was delayed due to other priorities, but we should have enough to get started on validation in early January. As our test framework isn't entirely ready yet, this will have no impact on our timeline.
= What we've accomplished =
== Search Update Pipeline ==
* Second batch of wikis ready to be enabled on Cloudelastic - https://phabricator.wikimedia.org/T351503
* Pull request sent upstream to allow more flexibility in flushing bulk requests - https://issues.apache.org/jira/browse/FLINK-33857
== WDQS graph splitting ==
* Discussion with WMDE on query log analysis to extract queries for our tests, we should have input data for our test in January - https://phabricator.wikimedia.org/T349512
* We're seeing instability on our test servers, which we do not understand yet, we suspect something low level, since they are running the same code and configuration as the production machines - https://phabricator.wikimedia.org/T352878
* First version of a Jupyter notebook ready to record queries for analysis - https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/981550/7/rdf-spark-tools/docs/query_recorder.md
Search Platform/Weekly Updates/2024-01-12

= Summary =
We've started query analysis on the WDQS Graph Split. The first number look good, with > 95% of queries returning coherent results both on full graph and on the split. This first analysis covers only pywikibot, but as this is the basis for multiple different tools and workflows, the numbers might be representative of general use of WDQS.
We're still working on improving performance and stability of the Search Update Pipeline. Some of the improvements might be replicated to other projects, notably the use of compaction on Kafka.
= What we've accomplished =
==WDQS Graph Splitting==
* Analyst extracted a first set of test queries from logs, covering 5 important tools (pywikibot, wikidata integrator, listeria, mix&match, sparqlwrapper)
* first rough analysis for 10K queries from pywikibot
** 99% of which execute successfully on both split an full graph
** out of which 94% return the same results
** Identified "AuthorBot" as potentially problematic, we will contact the author
** More analysis is needed, but this looks like promising numbers!
* Removed throttling from the test servers to allow to run large amounts of test queries - https://phabricator.wikimedia.org/T354555
== Search Update Pipeline ==
* Track down envoy performance issues
* Fix throttled throughput by discarding order of processing rerenders
* Add gzip support
* Enable page_rerender event emission for 80% of wikis
* Process page_rerender and resulting update_pipeline events without writing to elasticsearch (using devnull consumer/indexer) to test fetching from mw-api
* Configure Kafka topics to be more space efficient (partitions + compaction) and enable parallel processing of multiple partitions
* Started work on supporting private wikis via blockingly executed jobs
Search Platform/Weekly Updates/2024-01-19

= Summary =
We've addressed all known performance issues on the Search Update Pipeline and are now generating re-render events for all public wikis. Some more work is required around private wikis, but that's not blocking the roll out.
Query analysis for WDQS Graph Split is showing some results and raising more questions. Some clients seem to have no issues (Pywikibot, SPARQLWrapper), others show queries returning different results (Listeria, MixNMatch, WikidataIntegrator, ...). More investigation are required to understand if this is a limitation of our testing strategy or of the graph split itself.
= What we've accomplished =
== Search Update Pipeline ==
* New NetworkSession extension is code ready. This is one step in allowing SUP to update also private wikis. We're going through the "new extension for deployment checklist". This is probably going to be blocked on security review, which are prioritized once per quater. - https://phabricator.wikimedia.org/T345185
* All known performance issues are addressed, we are seeing the throughput that we expect.
** https://phabricator.wikimedia.org/T354517
** https://phabricator.wikimedia.org/T353430
* The main SUP grafana dashboard now links to related dashboards - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater / https://phabricator.wikimedia.org/T354322
* All non-private wikis featuring the cirrussearch extension publish page_rerender events by default. - https://phabricator.wikimedia.org/T351503
== WDQS graph splitting ==
* Some progress on analyzing sparql query results differences - https://phabricator.wikimedia.org/T355040
** Our query logs do not only contains sparql queries and the sparql client used to collect the data has to be adapted to support these (ASK, CONSTRUCT, DESCRIBE) (https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/991622)
** Getting failures due to response size, bumped the limit to 16M but still getting problems, I might stop here and simply ignore these queries
** Getting very bad numbers from Listeria and MixNMatch (34% and 17% identical respectively), avg result size is 1.6k and 8k so might explain partly why getting identical results is difficult, need more investigation to understand the cause...
** Getting pretty meh numbers for WikidataIntegrator at 88% with very small avg result size at 8, more investigation needed
** Pywikibot and SPARQLWrapper are good at 99.4% for both
* Expose 3 new dedicated WDQS endpoints: DNS entries, SSL certificate and microsite configuration are ready, but not yet working. Investigation required, but we're almost there - https://phabricator.wikimedia.org/T351650
* Spark job to export the split graph from HDFS is completed, with appropriate tests - https://phabricator.wikimedia.org/T350106
== Operations ==
* Fix dags affected by canary events failure, it affected 4 dags + a downstream one so far on our side - https://phabricator.wikimedia.org/T337055#9470093
Search Platform/Weekly Updates/2024-01-26

= Summary =
We now have test endpoints for the WDQS Graph Split exposed to the internet (see below for links). We have not yet communicated those to our users, but we're technically ready to get more feedback from our communities about the WDQS graph split. Functional and performance testing is moving forward, we are identifying a few use cases that will require more attention.
Search Update Pipeline has a few more quality of life and performance improvements, our Cloudelastic cluster is ingesting updates without issues, we can move to our production search cluster soon!
= What we've accomplished =
== Improve multilingual zero-results rate ==
* ICU token repair plugin code is done, waiting for a last code review, write up is in progress, but most information is already available in the readme of the project https://gerrit.wikimedia.org/r/c/search/extra/+/972478/8/docs/icu_tok_repair.md - https://phabricator.wikimedia.org/T332337
== WDQS graph splitting ==
* Meeting with WMDE about sample of queries to analyze, new and larger query sample should be ready in a few days
* A few fixes to IGUANA to make tests less flaky and to integrate with WMF build system - https://gitlab.wikimedia.org/repos/search-platform/IGUANA/-/merge_requests/4
* Performance test with IGUANA is running with ~17K queries known to work on both the full and main graph - https://phabricator.wikimedia.org/T355037
* Experimental endpoints with full/main/scholarly graphs are publicly available - https://phabricator.wikimedia.org/T350464 (and sub tasks)
** https://query-full-experimental.wikidata.org/
** https://query-main-experimental.wikidata.org/
** https://query-scholarly-experimental.wikidata.org/
== Search Update Pipeline ==
* A few more quality of file / performance improvements:
** SUP: Process (large) JSON responses non-blocking to save memory, with a nice reduction in garbage collection after this change (see task for the graph) - https://phabricator.wikimedia.org/T355066
** SUP: deployment: allow passing integers - https://phabricator.wikimedia.org/T354197
** ConsumerApplicationIT should fail when the update request payload changed - https://phabricator.wikimedia.org/T353427
** The consumer job of the SUP does not achieve its expected throughput - https://phabricator.wikimedia.org/T353460
Search Platform/Weekly Updates/2024-02-09

= Summary =
Further investigation of failed queries on the WDQS main graph shows that most are coming for a few sources, which gives us some confidence that we can improve the situation significantly by focusing on a small number of use cases.
Other projects are moving along nicely.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* ICU token repair corpus is built and daily diffs are running. Reviewing diffs from enabling the ICU tokenizer. Mostly looks good, but there are a few things to track down. (Malayalam has the most unusual results and I'm having a little trouble figuring out what's going on—diffs from my regresion test set aren't reproducing easily in focused testing. I'll get to the bottom of it eventually.)
== WDQS graph splitting ==
* A draft of the analysis is available on wiki. We confirm the numbers from last week that showed some categories of queries having a high failure rate on a split graph. This is mitigated by the large majority of failed queries coming from a small number of user agents (the top 5 user agent account for >90% of failures), which indicates that it is likely that a targeted effort can reduce the number of failures significantly. This will need a qualitative analysis which is planned as a next step. https://wikitech.wikimedia.org/wiki/User:DCausse/WDQS_Graph_Split_Impact_Analysis
* Intermediary report on query performance is published. This analysis covers only queries that run on the main graph (without requiring federation) and shows a modest performance improvement compared to running on the full graph. This confirms our initial assumptions. https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/Graph_split_IGUANA_performance
* February update is published, inviting feedback from our communities: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/February_2024_scaling_update
* Project page is published: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split
== Misc ==
* Investigated, restarted and back filled failed data pipeline. https://phabricator.wikimedia.org/T356030
* We participated to a Unicode Consortium meeting about the Foundation's membership. Nothing concrete yet, but a lot of good will and promises to do introductions and work together in the future. This is especially timely with our current work on ICU token repair.
Search Platform/Weekly Updates/2024-02-16

= Summary =
Slow week, with multiple team members out this week.
= What we've accomplished =
== WDQS graph splitting ==
* Meet videoconferences with WDQS community members as follow ups to initial explorations on graph split
* Reviewing and responding to questions on the graph split while looking at sampled queries that have results different between the full graph and the main graph.
== Search Update Pipeline ==
* Create new NetworkSession mediawiki extension to integrate private wikis with SUP - https://phabricator.wikimedia.org/T354976
* Create tool and process to investigate Search update Pipeline failures - https://phabricator.wikimedia.org/T356655
Search Platform/Weekly Updates/2024-02-23

= Summary =
We are in the deployment phase of our multilingual zero-result rate improvements. The new elasticsearch plugins are deployed, the new configuration is ready. Once the configuration is deployed, we will need to reindex all wikis (which takes 2-3 weeks) and analyze the improvements.
We are in active conversation with people from Scholia around the WDQS graph split. The discussions are constructive, but we are identifying major impacts to Scholia queries. Some we might be able to resolve, some we might not. In particular, the authors and scholarly articles are on different graphs, which makes some queries complex and expensive to run. Scholia is not only about scholarly articles, but about other types of papers, which now need to be treated differently. This conversation and investigation needs to continue.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* The config for the ICU tokenizer + ICU token repair upgrades along with a few new character mappings has been committed. Just need wait for it to be deployed, then we can reindex. - https://phabricator.wikimedia.org/T356643
* Indexing times seem to increase significantly with ICU token repair, but are still within acceptable limites. Some preliminary notes are in our standup notes (https://etherpad.wikimedia.org/p/search-standup) and a full write up will be published soon on https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Timing_Tests
== WDQS graph splitting ==
* Meeting with Daniel Mietchen and Lane Raspberry
** Agreed to have regular meetings
** Still some questions about why we're doing this and what problem we try to solve
** Question regarding what is wikidata, is it only a wdqs problem or should wikidata stop accepting some data and ask communities to use another hosting solution
more notes: https://etherpad.wikimedia.org/p/wdqs-graph-split-2024-02-15-1
*Started a page with example queries found in the samples and how to rewrite them with federation - https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federated_Queries_Examples
= Operations =
* backfill w[dc]qs reconciliation dag after a failure of the canary events system, deployed a quick patch to stop creating this dag dynamically and sense both DC partitions via a single sensor, should decouple the use of the wmf_conf.eventgate_datacenters var in airflow that was set to codfw only to workaround the canary event issue (which I reverted batch ["eqiad", "codfw"] to do the backfill).
** We might expect some turbulences during the upcoming dc switch (march 19), DE might still rely on a manual switch of that config var in airflow.
Search Platform/Weekly Updates/2024-03-22

= Summary =
This was a slow week, with most of the team traveling and recovering after our in person offsite last week.
The full deployment of the Search Update Pipeline work is almost certainly going to spill over to next quarter. We don't expect further surprises, the deployment will roll out progressively to more wikis.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* The dotted I fix code is up and ready for review, but Gerrit forgot to tag any reviewers. (It's just random enough about it that I haven't learned to check every time!) The write up is also done (https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Generally_Enable_dotted_i_fix_(T358495)). There was a fair amount of refactoring, but the core of it was straightforward. - https://phabricator.wikimedia.org/T358495
* Reindexing completed on the main search clusters, which enables a number of apostrophe normalization, camelCase handling, acronym handling, word_break_helper, and icu_tokenizer/_repair) - https://phabricator.wikimedia.org/T342444
== WDQS graph splitting ==
*Documented limits of federation: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federation_Limits
* Further performance analysis of various hardware options - https://phabricator.wikimedia.org/T359062
Search Platform/Weekly Updates/2024-04-05

= Summary =
A new quarter is starting! We haven't yet published the list of goals for this quarter, but it's mostly a continuation from the previous one.
The last 2 known patches for Search Update Pipeline are ready and need to be merged.
We have a good write up on the last batch of multilingual search improvements: https://www.mediawiki.org/w/index.php?title=User%3ATJones_%28WMF%29%2FNotes%2FLanguage_Analyzer_Harmonization_Notes&wvprov=sticky-header#Harmonization_Post-Reindex_Evaluation,_Part_I_(T359100)
= What we've accomplished =
== WDQS graph splitting ==
* Our current plan to do one last iteration on how we split the graph was discussed with Scholia and seems to be accepted.
* Tentatively, it does appear that there is a configuration variable that may yield somewhat faster imports. This is being tested against a split graph side, which uses a different record format. https://phabricator.wikimedia.org/T359062#9686176
* The limitations of federation are documented: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federation_Limits
== Improve multilingual zero-results rate ==
* Analyze results of harmonization - https://phabricator.wikimedia.org/T359100
** Also see the full write up on MediaWiki: https://www.mediawiki.org/w/index.php?title=User%3ATJones_%28WMF%29%2FNotes%2FLanguage_Analyzer_Harmonization_Notes&wvprov=sticky-header#Harmonization_Post-Reindex_Evaluation,_Part_I_(T359100)
== Search Update Pipeline ==
* Stability improvements to elasticsearch writes - https://phabricator.wikimedia.org/T356933
* Integrate Saneitizer with SUP - https://phabricator.wikimedia.org/T358599
== Misc ==
* Update URLs on MediaWiki:Elastica-desc - https://phabricator.wikimedia.org/T355451
* Unable to find a file by filename while adding a Commons media file statement - https://phabricator.wikimedia.org/T353683
Search Platform/Weekly Updates/2024-04-12

= Summary =
Every so often, celestial bodies align just so, you look up, and you can see the moon set in front of the sun, and so many stars off in the distance in the middle of the day. One of our team members had a chance to see the eclipse this week.
In other movements, we're looking into some basic search metrics such as number of searches, pageviews from searches, and so on:
https://phabricator.wikimedia.org/T358345
Next week we will review some of the preliminary results.
= What we've accomplished =
== WDQS graph splitting ==
* Changes to WDQS stream updater design are being specified. https://phabricator.wikimedia.org/T361935
* Started to write a simple model for defining sugraphs and refactored the split Spark job to make it a bit more generic. https://phabricator.wikimedia.org/T362060
* A configuration variable change yields faster imports on the scholarly article entity graph import. We will be looking to try this configuration variable in a near future import and are looking into a faster hard drive, which also seems to make things faster. https://phabricator.wikimedia.org/T359062#9704104
== Improve multilingual zero-results rate ==
* Test & fixture refactoring is almost done. https://phabricator.wikimedia.org/T361377
* Started doing data collection and test set extraction for hiragana/katakana mapping. https://phabricator.wikimedia.org/T180387
* Language analysis test case refactoring merged. https://phabricator.wikimedia.org/T361377
* An Azerbaijani admin/volunteer dev (NMW03) found the dotted I (İ) task and was hoping it would take care of more general dotted I lowercasing problems on the user's home wiki. Alas, it will not, but the user received some insight into what is being seen in Javascript and what to ask for in terms of MediaWiki internationalization. https://phabricator.wikimedia.org/T358495
== Search Update Pipeline ==
* Saneitizer deployment for Cloudelastic. https://phabricator.wikimedia.org/T358599
== Misc ==
* There is discussion about Kafka upgrades. Search Platform's Search Satisfaction schema would probably be a good early adopter to verify performance characteristics.
Search Platform/Weekly Updates/2024-04-19

= Summary =
We had an incident on WDQS in the codfw datacenter, which required manual investigation and intervention, taking up significant time this week. The user impact was mitigated by depooling this data center and serving all traffic from the eqiad datacenter.
A few highlights for the week:
* Communication about the graph split was sent, we're waiting for feedback until May 15, after which we will freeze the graph split strategy
* We have a few stability fixes to the Search Update Pipeline. It has been deployed on all wikis on Cloudelastic and we will shortly start enabling it for the production search clsuter.
= What we've accomplished =
== Improve multilingual zero-results rate ==
* After investigation, we will not enable hiragana/katakana mapping on all wikis as planned, but instead disable it on all wikis in the spirit of language harmonization (and because it does not seem to bring much value). Full write up on https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Enab..._no,_wait.._Disable_Hiragana-to-Katakana_Mapping_(T180387) - https://phabricator.wikimedia.org/T180387
* Starting work on Yiddish ligatures that was reported recently by our users. This is similar to an issue with Arabic normalization that was reported almost 10 years ago, which we will address at the same time https://phabricator.wikimedia.org/T362501 / https://phabricator.wikimedia.org/T72899
* Reindex all wikis to enable apostrophe normalization, camelCase handling, acronym handling, word_break_helper, and icu_tokenizer/_repair - https://phabricator.wikimedia.org/T342444
== WDQS Graph Split ==
* Communication sent: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/April_2024_scaling_update
* Federation limits are documented: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federation_Limits
* Example queries with federation are published: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federated_Queries_Examples
== Search Update Pipeline ==
* Deploy streaming updater for 100% of writes to cloudelastic - https://phabricator.wikimedia.org/T358518
* Improvement to stability: Streaming Updater should still make forward progress when one index has problems - https://phabricator.wikimedia.org/T356933
* Improvement to stability: SUP: Allow precise control over connection pool size AND operator capacity - https://phabricator.wikimedia.org/T361900
* Saneitizer has been integrated and deployed as part of the new update pipeline. This is helping discover a few more issues with pages with error state - https://phabricator.wikimedia.org/T358599
== Operations ==
* Incident on WDQS@codfw where the streaming updater failed and reverted to an expensive reconciliation process for all changes, reducing the overall throughput enough that the updater could not catch up with the edit rate. We don't have a full understanding of the root cause, but we do have a few improvements to the general stability of the system - https://phabricator.wikimedia.org/T362508
Search Platform/Weekly Updates/2024-04-26

= Summary =
Things are in progress, but not much was completed this week. We're dealing with an incident on Search and spending significant time on planning for next fiscal year.
= What we've accomplished =
== Search Update Pipeline ==
* Rolled out SUP writing to cloudelastic for all wikis
** Backfill with five instances and up to 1k updates/s written to cloudelastic
* Ported saneitizer rerender loop (to continously rerender all pages over longer periods of time) - https://phabricator.wikimedia.org/T358599
== WDQS graph splitting ==
* First patch up to support stubs in splitting the graph, including some cleanup to the existing code - https://phabricator.wikimedia.org/T362060
== Misc ==
* Search roadmap, what do we need to do for next fiscal year
* Issue with title suggestion search on codfw, still being investigated - https://phabricator.wikimedia.org/T363516