nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2024-08-23

= Summary =
We've been tying up a few loose ends on the Search Update Pipeline. One of the improvements of that new pipeline is the introduction of a standardized way to process weighted tag updates (used for example by the Growth team for Link Recommendations). This is timely as we've seen a few issues indexing those recommendation and simplifying the process will make it more robust.
Data is being reloaded for the WDQS Graph Split, and most of the traffic routing is in place. We hope to announce those new endpoints next week.
= What we've accomplished =
== WDQS graph splitting ==
* Load balancing configuration for production is underway. https://phabricator.wikimedia.org/T373145 https://phabricator.wikimedia.org/T364368 https://phabricator.wikimedia.org/T364364
* More testing to be done as domain names and load balancing are connected. https://phabricator.wikimedia.org/T370754
== Search Update Pipeline / Private Wikis ==
* Fix redirect processing https://phabricator.wikimedia.org/T372446
** Adapt to CirrusSearch update job behavior: Only process intra-namspace redirects and those coming out of main (0)
** Avoid redundant prefixes, since redirect targets in page_change events already have prefixed titles
* Fixed orchestrator script: Run any pending backfills before shutting down for good - https://phabricator.wikimedia.org/T372128
* Proved the hypothesis that we loose events in the producer (https://phabricator.wikimedia.org/T372362) wrong. Turns out this suspicion was caused by wrongly counting events going in and out.
* Created followup tasks for adopting the new page_weighted_tags_change stream (https://phabricator.wikimedia.org/T366253) and arranged a meeting with Growth Team. Follow up on simplification of this architecture and visibility in https://phabricator.wikimedia.org/T372904 and https://phabricator.wikimedia.org/T373140
== Improve multilingual zero-results rate ==
* Finished refactoring the asciifolding/icu_folding code https://phabricator.wikimedia.org/T332342 / https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Analyzer_Harmonization_Notes#Part_4%E2%80%94Refactoring_&_Analysis_Notes
* Looking at expanding ICU folding to other languages
== Vector based search ==
* A follow up coordination meeting on a WE.3.1 focused technical spike for vector embeddings is scheduled.
== Misc ==
* Search backend replacement: Email sent for advance deprecation notice. https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/SXK2FODVJP3R3GRSYE2V2ODPUO74YLFO/
* CirrusSearch: Add support for sorting results by naturally sorting page titles (note that this will not be activated on WMF wikis, but is available for 3rd parties) - https://phabricator.wikimedia.org/T371458
* Decision on migrating from Elasticsearch to OpenSearch is documented on https://www.mediawiki.org/wiki/Wikimedia_Search_Platform/Decision_Records/Search_backend_replacement_technology - https://phabricator.wikimedia.org/T370661
Search Platform/Weekly Updates/2024-09-13

= Summary =
Now that WDQS graph split endpoints are public, we are seeing some discussions and user experimentation with those endpoints. Big thanks to David for following up on questions from our communities!
Elasticsearch is moving back to an open source license. We are reviewing our plan to migrate to OpenSearch in light of this licensing change. Both OpenSearch and a recent version of Elasticsearch would provide support for vector search, and a modern and supported software stack. We'll need time to let things settle, understand the implications of the new licensing of Elasticsearch, and reach a decision on which stack is the right one in our context.
= What we've accomplished =
== WDQS graph splitting ==
* Answering questions on wiki (https://www.wikidata.org/w/index.php?title=Special%3AContributions&target=DCausse+%28WMF%29&namespace=all&tagfilter=&start=&end=)
* New property proposal created by Charles https://www.wikidata.org/wiki/Wikidata:Property_proposal/publication_type_of_scholarly_article
== Search backend replacement ==
* Awaiting licensing feedback. Wrote high level checklist, tasks will need to be created. https://phabricator.wikimedia.org/T370148
== Misc ==
* Add flag revision_based to page_weighted_tags_changed schema - https://phabricator.wikimedia.org/T372914
* Started to investigate why the wdqs-updater misbehaved after kafka-main@codfw node replacements, suspecting some weird behaviors of the transactional kafka producer and the way we re-used the same broker.id on two different nodes - https://phabricator.wikimedia.org/T374628
* Some work to cleanup wdqs in puppet (better separation of the categories endpoint)
* Small patch to add support for SSL in the wdqs updater (long overdue)
* Data platform industry landscape slide preparation
Search Platform/Weekly Updates/2024-09-20

= Summary =
We closed a number of operational tasks and qulityof life improvements for our users this week.
Good progress was done on deploying ICU folding to most of the languages supported by Search.
= What we've accomplished =
== WDQS graph splitting ==
* Reproduce and fix connectivity issue with the internal http client of blazegraph (https://phabricator.wikimedia.org/T373812)
* Write a bunch of subtasks for https://phabricator.wikimedia.org/T294133 (expose the wikidata rdf update stream)
== Improve multilingual zero-results rate ==
General task: https://phabricator.wikimedia.org/T332342
* I have finished Nepali (after deciding not to do anything special with the occasional Tibetan script), Assamese, and Punjabi. Only Oriya is left.. then a quick re-check that everything works as expected, a quick code review, and a new patch Friday or Monday.
* ICU folding configs are done for Marathi, Burmese, Malayalam, Telugu, Sinhala, Kannada, & Gujarati.
* I was able to do some additional needed normalization for Marathi, Malayalam, Sinhala, and Gujarati, which is a very nice bonus. I did some minor refactoring so all those share a `case` statement, too.
* Nepali is in progress.. I think the configs for Devanagari are done, but I'm looking into the lesser-used Tibetan script (which occurs regularly on-wiki), and that config may be re-usable for the Tibetan langauge, too.
* After that Assamese, Punjabi, and Oriya are left. Each language config takes between 20 minutes and 2 days—though I'm averaging ~2½ configs per day, and hoping to finish the configs for the INdic languages by the end of the week.
== Search Metrics ==
* T372932 Add raw numbers to Search Metrics dashboard
== Misc / Operations ==
* Preparation of Q2 goals
* Preparation of Data Platform offsite
* T371929 Index all statements (without value) for all datatypes for haswbstatement - the change is deployed, but it will take up to 16 weeks until all items have been re-indexed
* T372030 Index statements in commons media datatype for haswbstatements - the change is deployed, but it will take up to 16 weeks until all items have been re-indexed
* T373778 NetworkSession and AbuseFilter may be spammy
* T369808 The Commons search "deepcategory" operator often does not work (Deep category query returned too many categories)
* T328330 Create SLI / SLO on Search update lag
* T371648 Run rebuildall.php on cswikivoyage
* T331127 phantom redirects lingering in incategory searches after page moves
* T374637 Decide how to make datasets owned by analytics-search-users also readable by analytics-privatedata-users
* T371401 Adapt search ranking for mul language code
Search Platform/Weekly Updates/2024-09-27

= Summary =
Lots of people out this week (including attending OpenSearch conference), we made only limited progress.
As part of the WDQS Graph Split work, we want to start exposing an stream of RDF updates publicly to that it can be used to test alternative implementations. We will focus on exposing a stream of changes, but we will not at this time provide additional tooling to consume it.
= What we've accomplished =
== WDQS Graph Split ==
* T373391 Create wdqs-main and wdqs-scholarly specific test queries
* Creating the work structure to expose the WDQS RDF change stream publicly - https://phabricator.wikimedia.org/T294133 (and subtasks)
== Misc ==
* Adding SSL support for the rdf flink job when it connects to kafka, will deploy this alongside the use of the better discovery technique for kafka brokers - https://phabricator.wikimedia.org/T333373
* First draft of our goals for Q2: https://docs.google.com/document/d/1A9ZwQI3mFPJyzR3GQUlYPo0eYbpqzJ3dTGpqD9FcLG0/edit
Search Platform/Weekly Updates/2024-10-04

= Summary =
We are starting work on exposing the stream of RDF updates that is used by WDQS publicly. This will allow 3rd party deployments of WDQS to keep up to date more easily and more crucially for us, it will allow external organizations working on RDF backends to test ingesting WDQS data live, which is the ground work needed before a potential migration away from Blazegraph to another RDF backend.
= What we've accomplished =
== Search Update Pipeline - Private Wikis ==
* Follow up on cleaning up how we use kafka to publish private events - T374335 The SUP producer should ship private wiki update events to a separate stream
* Follow up on how Closed Wikis interact with SUP - T374987 "Account autocreation denied for CirrusSearch Streaming Updater by ClosedWikiProvider"
== Improve multilingual zero-results rate ==
* T332342 Standardize ASCII-folding/ICU-folding across analyzers Full write up on MediaWiki.
== WDQS Expose RDF stream publicly ==
* Starting work on T374919 Adapt the rdf-streaming-updater flink job to use wikimedia-eventutilities-flink
== Misc / Operations ==
* T372446 Special:Search intitle search has weird redirect behavior
* T333373 The WDQS streaming updater should support connecting to kafka with SSL
Search Platform/Weekly Updates/2024-10-11

= Summary =
Now that the dust has settled on licensing changes between OpenSearch and Elasticsearch, we're starting preliminary work on migrating to OpenSearch.
Exposing the stream of RDF updates for WDQS is moving forward. The main step will be reworking the schema of those updates. We want to get this right before making it public, as it will be much harder to change it once it is used by more clients. Some level of bike shedding will occur.
= What we've accomplished =
== WDQS Expose RDF stream publicly ==
* Working on refactoring the input/output streams to use eventutilities - https://gerrit.wikimedia.org/r/q/topic:%22output-schema-v2%22
== Improve multilingual zero-results rate ==
* ICU folding before-and-after analysis is done! 100% of reasonably-sized targeted samples showed improvement in ZRR! 84% of general query samples showed improvements (mostly ZRR, some increase in results, a few top result changes). [TJ]
** Details in https://phabricator.wikimedia.org/T375557T375557. More details on-wiki:
*** https://www.mediawiki.org/w/index.php?title=User%3ATJones_%28WMF%29%2FNotes%2FLanguage_Analyzer_Harmonization_Notes&wvprov=sticky-header#ICU_Folding_Results
** Side note: weighted vs unweighted ZRR doesn't matter much, except for Vietnamese, which has some craaaaaazy bot traffic that gets through into the sample.
== Search backend replacement ==
* Have the basics of cindy's environment running with a new plugins.deb and cirrus-opensearch image. Getting the basics together having new repos created in gerrit, new branches where appropriate. https://phabricator.wikimedia.org/T372769
* Re-affirmed decision to migrate to OpenSearch. Update at https://www.mediawiki.org/w/index.php?title=Wikimedia_Search_Platform%2FDecision_Records%2FSearch_backend_replacement_technology&diff=6793828&oldid=6737032 and https://phabricator.wikimedia.org/T370661, with addition of a checklist item in https://phabricator.wikimedia.org/T370148 for planning a migration guide later.
Search Platform/Weekly Updates/2024-10-18

= Summary =
* Migration of Elasticsearch plugins continues, migrating to the gradle build system is challenging.
* Flink application processing RDF updates for WDQS is migrated to leverage Event_Platform/Event_Utilities.
= What we've accomplished =
== Flink updates ==
* We updated & deployed our flink images in response to security issues, see https://phabricator.wikimedia.org/T376886 and https://phabricator.wikimedia.org/T371874
== Weighted tags ==
* Migration of sources of weighted tags to use our our new consolidated event stream continues. Link recommendations is almost complete, despite some trouble after changing the API.
== Search backend replacement ==
* Migrating most Elasticsearch plugins works smoothly, however, Opensearch uses a different build system based on gradle which does not work for all migrated plugins yet.
Search Platform/Weekly Updates/2024-10-25

= Summary =
We made steady progress on various initiatives, though most efforts remain in progress: Release of a public RDF stream, Improvements for Japanese wikis, and consolidation of data flows for weighted tags.
= What's ongoing =
== WDQS: Public RDF Stream ==
* Continued migration of the RDF streaming updater (flink application) to leverage event utilities library.
== Language Stuff ==
* Continued analysis and tests of Kuromoji, a Japanese language analyzer, see T318269
== Spark Kafka Writer (Weighted tags/Dumps 2.0) ==
* Completed first test runs of the kafka wrapper with schema validation. Still needs work to simplify usage, see T374341 and T372912
= What we've completed =
* T375557	Reindex all wikis to enable folding harmonization and new functionality
* T372904	Use page_weighted_tags_changed stream
* T376715	TypeError: Argument 3 passed to CirrusSearch\DataSender::sendWeightedTagsUpdate() must be of the type array, null given, called in /srv/mediawiki/php-1.43.0-wmf.25/extensions/CirrusSearch/includes/Job/ElasticaWrite.php on line
* T376161	Classify fulltext search abandonment: sampling
Search Platform/Weekly Updates/2024-11-01

= Ongoing work =
== WDQS Graph Split ==
* Working on adapting the sparql query logging to the split graph endpoints - https://phabricator.wikimedia.org/T376134
== Language Stuff: Kuromoji ==
* I think I've optimized the Kuromoji settings to fix the known problems that don't require a native speaker to understand. I'm working on a tool to parse example sentences and queries from Japanese Wikipedia and Wiktionary. And because I want to refresh some skills, I'm doing it in Python. (It's going, but it's a little slow.) I'm also comparing the ICU tokenizer vs the Kuromoji tokenizer in case that can solve other problems once speakers are involved. - https://phabricator.wikimedia.org/T318269
** I hope to be done with the tokenizing and prepping the text for review tomorrow (Friday), and I'll make the initial request for help from native speakers within the Foundation, and hopefully have instructions to send out to them on Monday.
** Fun fact: Python uses lowercase \u for 4-digit UTF-8 codes: \u2620 (☠) and uppercase \U for 8-digit UTF-16 codes: \U0001F480 (💀). This solves the problem of high surrogates and low surrogates—but creates a problem for the programmer who doesn't know about it. (Of course I know him—he's me!)
= What we've accomplished =
== WDQS Graph Split ==
* New hackathon scheduled by scholia folks: https://www.wikidata.org/wiki/Wikidata:Scholia/Events/Hackathon_November_2024
* T373812 Internal federation sometimes fail with HttpConnectionOverHTTP
== WDQS Expose RDF stream publicly ==
* Finished the work to adapt the flink jobs to event platform best practices... deployment might be tedious, I switched to the page_change stream instead of the old rev-create/page-[delete|suppress] streams and thus might need a carefull/manual deploy... - https://phabricator.wikimedia.org/T374918
** still working on few config patches to declare the stream in mw-config
== Search Update Pipeline / Weighted tags ==
* Tansitioned processing of weighted tags, created by Growth, to Search Update Pipeline; no more direct writes from CirrusSearch to Elasticsearch - https://phabricator.wikimedia.org/T377150 / https://phabricator.wikimedia.org/T372904
* Finished work on spark-kafka-writer (with schema validation)
** Created upstream ticket (https://issues.apache.org/jira/browse/SPARK-50160) + PR (https://github.com/apache/spark/pull/48695) but neigther has seen any feedback yet
== Search Metrics ==
* T376161 Classify fulltext search abandonment: sampling - see https://superset.wikimedia.org/superset/dashboard/search/
== Misc / Operations ==
* T378227 Investigate failed Cirrus index build services on mwmaint2002
* T374118 Datahub - ingest Hive discovery database
* T375557 Reindex all wikis to enable folding harmonization and new functionality
* T376715 TypeError: Argument 3 passed to CirrusSearch\DataSender::sendWeightedTagsUpdate() must be of the type array, null given, called in /srv/mediawiki/php-1.43.0-wmf.25/extensions/CirrusSearch/includes/Job/ElasticaWrite.php on line
Search Platform/Weekly Updates/2024-11-08

= Ongoing work =
== Search Update Pipeline / Weighted tags ==
* Finalized schema-validating wrapper for spark kafka writer (released as part of eventutilities 1.4.0). However, first runs with real data revealed limitations in handling null values: Spark data frames may hold null that have to be excluded from the JSON validated against an event-stream-schema, since they do not support excplicit nulls (via union types). - https://phabricator.wikimedia.org/T374341
* Enabled stream processing for weighted tags created by Growth scripts. Failed due to wrong output format (not compliant with schema) and had to roll back, but a fix is shipped and should roll with the next train. https://phabricator.wikimedia.org/T378983 / https://phabricator.wikimedia.org/T378664 / https://phabricator.wikimedia.org/T377150
== Language Stuff: Kuromoji ==
* I've massaged all the Japanese tokenization data as much as I can and written up instructions for speaker evaluation. I've got one volunteer to look over the samples and evaluate the tokenization (both the Kuromoji and ICU tokenizers), and I've asked a couple more people.
** I'll keep asking around until I get at least 2 reviewers or run out of candidates. - https://phabricator.wikimedia.org/T318269
= What we've accomplished =
== Misc / Operations ==
* PHP Warning: array_merge(): Expected parameter 1 to be an array, null given - https://phabricator.wikimedia.org/T377479
Search Platform/Weekly Updates/2024-11-15

= Ongoing work =
== Language Stuff: Kuromoji ==
T318269 Test and analyze Kuromoji Japanese language analyzer
* 2 of my 3 volunteers have made some progress on reviewing tokenization by Kuromoji and the ICU tokenizer. Thomas suggested the Sudachi tokenizer, which has Elasticsearch and OpenSearch plugins available, so I spent a little time trying to get it to work. It threw errors as soon as I installed it, so I messed around a bit and asked David for some help and he got to the next sticking point (the dictionary has to be installed manually in a not really local place.. though to be fair it was right there in the docs—I should not try to install new software on a Friday afternoon before a 3-day weekend).
* I took the lack of progress in Sudachi installation and waiting on the volunteer reviewers as an opportunity to work on committing my last couple of years' worth of minor upgrades to my analysis analysis tools. The code was easy enough, but there have been so many upgrades to the analyzers themselves that I need to work through the docs again to make sure everything is up to date—because it is not currently up to date at all.
* Up next: submit a patch with the analysis analysis tools and then get back to trying to get Sudachi working while I wait for the reviewers.
= What we've accomplished =
== Search Update Pipeline / Weighted tags ==
* We can now produce events from Spark. This is also in support of the Dumps 2.0 work. T374341 Add support for Spark producers in Event Platform
== Search backend replacement ==
* We are confident that running a mixed Elasticsearch / OpenSearch cluster during migration will not cause issues: T379938 Evaluate mixed-cluster behavior of elasticsearch + opensearch
Search Platform/Weekly Updates/2024-12-06

= Ongoing work =
== Language Stuff: Kuromoji ==
* David helped me read the Sudachi documentation (which was in English.. it was a Friday, I dunno what to say) and I got it installed and running. It has some serious quirks, the most blatant of which is that is has a lot of multi-word English (and some French) tokens in its dictionary, so it tokenizes "prêt-à-porter" and "Organization for Economic Cooperation and Development" as one token each. It generally can generate some weird tokens on Latin script. Rumor has it that it is also very slow (which I need to check). If it were stupendous at parsing Japanese it might be worth it to hack around the quirks, but I think it's almost time to put it on hold until I finish analyzing the reviews for Kuromoji and the ICU tokenizer.
* 2 of my 3 volunteers have made some progress on reviewing tokenization by Kuromoji and the ICU tokenizer. Thomas suggested the Sudachi tokenizer, which has Elasticsearch and OpenSearch plugins available, so I spent a little time trying to get it to work. It threw errors as soon as I installed it, so I messed around a bit and asked David for some help and he got to the next sticking point (the dictionary has to be installed manually in a not really local place.. though to be fair it was right there in the docs—I should not try to install new software on a Friday afternoon before a 3-day weekend).
* I took the lack of progress in Sudachi installation and waiting on the volunteer reviewers as an opportunity to work on committing my last couple of years' worth of minor upgrades to my analysis analysis tools. The code was easy enough, but there have been so many upgrades to the analyzers themselves that I need to work through the docs again to make sure everything is up to date—because it is not currently up to date at all.
= What we've accomplished =
== WDQS Graph Split ==
* T376598 Adapt the split graph rules to take into account the new P13046 property
== Elasticsearch to OpenSearch migration ==
* T372769 Adjust cirrus-integration-test-runner to boot opensearch
== Search Update Pipeline / Weighted tags ==
* T377150 Config: enable CirrusSearchEnableEventBusWeightedTags
== Migrate Archiva to Gitlab ==
* T377153 Migrate Glent to Gitlab for publication of artifacts
== Misc / Operations ==
* T380343 Move oozie/util/swift/upload/ out of the refinery oozie folder (moving utility scripts to a central location now that we are removing the last pieces of Oozie)
* T359062 Assess Wikidata dump import hardware - a blog post will follow, tracked in T373338 Blazegraph import of Wikidata - tech blog post
* T379045 mjolnir fails with: Partition not found in table 'labeled_query_page' database 'mjolnir' (issue with the Machine Learning Ranking pipeline)
* T374628 Investigate why rdf-streaming-updater is unable to recover after replacing kafka-main@codfw nodes
Search Platform/Weekly Updates/2024-12-13

= Ongoing work =
== Language Stuff: Kuromoji+Sudachi ==
* Recently spent a fair amount of time improving analysis tools to deal with issues related to running and reviewing Sudachi tokenizer (suggested as an option by a reviewer).
* Determined reasonable Sudachi config and ran examples for review.
* Completed first analysis of evals of Kuromoji and ICU tokenizers. Kuromoji is better, and very good on Wikipedia articles and queries (at the sentence level). It is less good on Wiktionary articles and queries, so I need to look a little more closely.