nyuuzyou/wikis huggingface dataset:
Search Platform/Weekly Updates/2024-12-20

= Ongoing work =
== WDQS Expose RDF stream publicly ==
* Agreed on a strategy to expose "double compute" streams via the EventStreams HTTP service, patch uploaded and merged - https://phabricator.wikimedia.org/T382065
* WDQS update stream definitions deployed to the event stream config
* Add support for eventstream utilities is almost done, couple patches still in review (this should not block exposing the stream tho) - https://phabricator.wikimedia.org/T374919
== Search Update Pipeline / Weighted tags ==
* We have a new search keyword "inproject" whose data is populated via weighted_tags, the data is close to be fully populated (I measured the growth at https://docs.google.com/spreadsheets/d/14I8o4lJxqrT11oa1yyDGBbs01A7Lsp0a5QCz8TAuyIs/edit?usp=sharing and it seems to stabilize), the work has been done by a community member - https://phabricator.wikimedia.org/T378868
== Misc ==
*Migration of cirrus tools based on mwscript to mwscript-k8s done (https://phabricator.wikimedia.org/T378382):
** generalized one of our repo as cirrus-toolbox so that it can hold more than one script and can be re-used as library: https://gitlab.wikimedia.org/repos/search-platform/cirrus-toolbox
** cirrus-reindex-orchestrator almost done but blocked on a race between the php & tls-proxy containers: https://phabricator.wikimedia.org/T382398
* A/B test of new Mjolnir models (https://phabricator.wikimedia.org/T377128) [EB]
** Test reports have been generated. Results look very good for jawiki and kowiki, which did not have the ML based ranker yet. Results look either slightly positive or neutral for the other reports I've looked over. Generally it doesn't seem to get worse anywhere, it gets a little better in some places, and it gets much better in places where we were not yet applying the ML models.
** In the future I think we should publish these via Data Platform/Web publication, but i feel like it's currently premature. The next time we run a test we should tighten up the report a bit more, make sure it has what we want. There are probably some extra graphs that are unnecessary in the current report.
** Reports can be found at: https://people.wikimedia.org/~ebernhardson/T377128/
= What we've accomplished =
== WDQS Graph Split ==
* T376134 WDQS query logging should be separated per subgraph
== Search Update Pipeline / Weighted tags ==
* We have a new search keyword "inproject" whose data is populated via weighted_tags, the data is close to be fully populated (I measured the growth at https://docs.google.com/spreadsheets/d/14I8o4lJxqrT11oa1yyDGBbs01A7Lsp0a5QCz8TAuyIs/edit?usp=sharing and it seems to stabilize), the work has been done by a community member - https://phabricator.wikimedia.org/T378868
== Operations / Misc ==
* T378097 Investigation: why do statements on Senses and Forms not show up in searches using haswbstatement
* T375641 [ES-M3: Implement label and aliases search for EntitySchemas via the wbsearchentities API] (work done by WMDE)
Search Platform/Weekly Updates/2025-01-17

= Ongoing work =
== Language Stuff: Kuromoji/Sudachi ==
* Sorting through my Sudachi notes and working on the write up. In the spirit of sharing early and often—though it is not that early or that often—my Kuromoji notes ("Japan Eset OK en I Zat Ion") are on Mediawiki. [TJ] T318269 https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Japan_Eset_OK_en_I_Zat_Ion%E2%80%94Kuromoji,_ICU,_and_Sudachi
== WDQS graph splitting ==
* Attempted to deploy config changes to use the internal split graph endpoints [SM][DC]
** succeeded for the campaign extension (https://phabricator.wikimedia.org/T377956)
** failed for WikibaseQualityConstraints (https://phabricator.wikimedia.org/T374021)
*** Re-opened https://phabricator.wikimedia.org/T369079
== MLR Improvements ==
* Updated documentation on https://phabricator.wikimedia.org/T383048 on the general approach and what has been done so far
* Agreed with the team on next steps. I'll train a model with "vanilla" xgboost (instead of the full mjolnir's pipeline). I'll use the "exact match" heuristic as a baseline for "easy queries" identification. No blockers, but today I've been mostly focused on airflow / mjolnir troubleshooting
== Misc / Operations ==
* Airflow issues after the k8s migration
** While we ironed out a quite few issues, the airflow-search is not yet fully operational
** Investigated and resolved name resolution issues that resulted in pods getting killed (http://phabricator.wikimedia.org/T383651)
*** this triggered cascading issues whereby airflow would lose track of skein, resuliting in multiple instances of skein and spark job being spawned per task
** KubernetesPodExcutor: WIP on both SRE and SWE fronts to resume execution of refinery script (unblocks the drop_data_daily dag). Aim is to test a solution early next week 
** Fixed a skein/spark memory unit missfit bug, that resulted in spark job being erroneusly killed (http://phabricator.wikimedia.org/T383589)
** Fixed a race condition in mjolnir's use of refinery jars (http://phabricator.wikimedia.org/T383870)
** MRs in flight to reduce dataset size and tweak popularity_score 
* Helping out the Design System Team with a question on CSS hyphenation for languages with loooooong words. German compounds are just the tip of the iceberg! (That must be "deutschcompoundennounzericenbergertippe" or something, right?)
Search Platform/Weekly Updates/2025-01-31

= Ongoing work =
== MLR Improvements ==
* T383048 Investigate current MLR models for Search and identify improvements: the "easy queries" exploration is in review.
* There's a report, in notebook form, available at https://gitlab.wikimedia.org/repos/search-platform/notebooks/-/merge_requests/6. In short, reweighting training data with query "easiness" did not have the impact we expected. Nonetheless, data analysis shows that current mjolnir models perform well (ndcg) on both "hard" and "easy" queries. Following our discussion on Wednesday, the notebook has been update with info about low ndgc scores (queries are omitted from the renderer to avoid exposing PII). Given the outome of this work, there's three avenues for improving MLR that we are considering:
** incorporate abdandoned queries in the training pipeline (requires further research)
** extend MLR to more wikis
** invest in infra improvement to support more frequent model rollouts 
As discussed on Wednesday we'll have chance to touch base and plan next steps at the upcoming offiste
== Operations / Misc ==
* Investigate possible impact of changing some of the RDF prefixes in wikibase
** https://phabricator.wikimedia.org/T384344
** https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1114003
* Started working on migrating the search update pipeline to non-dev schema&stream names - https://phabricator.wikimedia.org/T375821
** this involves removing "development" from schema names, writing to/reading from ".v1" stream instead of ".rc0"
** might be a bit more tedious than anticipated because we have existing users that we don't want to break (i.e. the weighted tags ingestion stream)
* T384255 python_script_executor should run with KubernetesPodOperator: added support to airflow common libs to execute refinery scripts on k8s. [GM]
** removes a blocker for the search instance migration.
== Language Stuff: Kuromoji/Sudachi ==
* Spent a good bit of Friday synthesizing my Sudachi findings into a semi-objective list of either errors or reasonable options/features they could provide, and opened a ticket on their repo. No response yet... [TJ]
https://github.com/WorksApplications/elasticsearch-sudachi/issues/156
* Currently updating our config for Sudachi without a custom dictionary to get a sense of what can and can't be done, so I know how much hassle a custom dictionary is worth, and so I can commit a config that encapsulates all the investigation so far.
Search Platform/Weekly Updates/2025-02-14

= Ongoing work =
== MLR Improvements ==
* Deploy and test new MLR models (https://phabricator.wikimedia.org/T385972)
** We deployed the learning-to-rank (ml) models tested last quarter to all wikis with MLR enabled. We started a new A/B experiment to benchark those models again recently trained ones. 
== WDQS graph split ==
* Worked on a small patch to improve the parsing of RDF dumps in hadoop which might slighly change due to https://phabricator.wikimedia.org/T384344 
== WDQS Expose RDF stream publicly ==
* Started a documentation at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater/Public_Update_Stream
== Search Update Pipeline - Weighted Tags ==
* T374702 Cleanup: Remove deprecated weighted tag methods
== Elasticsearch -> OpenSearch migration ==
* Upgraded all the plugins to opensearch 1.3.20, updated the cirrus dev image to that same version, switched cindy to 1.3.20 too (https://phabricator.wikimedia.org/T385005) 
== Misc / Operations ==
* Multiple instabilities with Mjolnir (Machine Learning training pipeline) - T383218 Mjolnir is sometimes stuck in feature selection:
** T383870 mjolnir should pin refinery jar version explicitly
** implemented the workarounds proposed in T383218, to mitigate impact on dags sharing the same airflow pool. There's an issue with this implementation though, that is leading tasks to execute out of order.
** bumped Spark Driver memory for the offending task, hopefully this will help. If not, the next step will be digging into the codepath executed by this task. 
* Investigated categories query service lag issues, was due to a discrepancies in mariadb index names on some dbstore hosts, got fixed by Amir (https://phabricator.wikimedia.org/T386005)
= What we've accomplished =
== Misc / Operations ==
* We have completed our migration from Graphite to Prometheus for all Search related metrics!
** T359033 EPIC: Convert CirrusSearch metrics to statslib
** T369148 Replace usage of StatsdDataFactory with StatsFactory
Search servers relocation - 12 august 2010

Batch one: searchidx1, search1 - search12
Batch two: search13 - search20
== Before moving servers ==
* In /home/wikipedia/common/php/wmf-config/lucene.php comment out:
 $wgEnableLucenePrefixSearch = true;
This will re-route ajax suggestions from search boxes back to the database.
== Moving batch one ==
Before turning off and relocating search1-12 do the following:
* on lvs3, searchpool1 should be "search14, search18, search19, search20", searchpool2 should be "search16", searchpool3 should be "search17". Make sure to backup the old configuration.
* In /home/wikipedia/common/php/wmf-config/lucene.php replace 
 '10.0.3.11' # search 11
with
 '10.0.3.13' # search 13
* verify everything is ok, then searchidx1 and search1-12 are ready to move
== Moving batch two ==
* bring up search1-12 and make sure lsearchd is started ("sudo /etc/init.d/lsearchd start")
* bring up searchidx1 - lsearchd should *not* be running here - I'll start it later 
* revert the changes from previous section, i.e. revert changes on lvs3 and on /home/wikipedia/common/php/wmf-config/lucene.php. This will direct queries back to search1-12. 
* Try some test queries on en.wp, de.wp and commons. Allow up to 15 minutes for search to start working (it might take some time for linux to cache the appropriate pages, will see lots of I/O in ganglia stats). If it still doesn't work after 15 minutes, and no CPU is used on any of search1-12, revert the changes and wait for tomorrow to do the second batch. 
* if everything is fine, search13-20 are ready for relocation and can be switched off
== Switching back to original state ==
* Turn on search13-20
* Robert will restore original lsearchd configuration 
* uncomment $wgEnableLucenePrefixSearch = true; (revert #Before moving servers)
* verify everything works
Category:Search
Security

Wikitech is meant to host procedural or instructional material that is not training. Content on this wiki is likely to relate to the Wikimedia Foundation hosted wikis, and related ecosystem.
To report security bugs, vulnerabilities or other issues please follow our process.
= How To =
*How to deploy a WMF security fix
*How to perform a security update for a Debian package
= Security mailing lists to follow =
* oss-security mailing list, source of many initial reports/CVE requests, previously private reports on the linux-vendors lists are posted here
* Bugtraq (general security mailing list, high volume, includes advisories and announcements for many vendors)
* US-CERT alerts mailing list
* fulldisclosure
= Related Security Content = 
Project 	Use by Wikimedia Security Team
mediawiki.org 	General content for Policy, SOPs, etc. Official Security team page.
meta.wikimedia.org 	Policy and other content for translation.
office.wikimedia.org 	Sensitive or private content. Must have an NDA and appropriate access.
foundation.wikimedia.org	Canonical location for Policy
----
Understanding Wikimedia Security Team documentation structure
Category:wikimedia security team
Category:Security
Security/logging

= Security Logging Use Cases/Work =
== 01/22/2019 ==
* stage: there's the need to easily correlate logs with security incidents
* infra foundations has been taking logstash / logging infrastructure onboard and we can join efforts
* software involved first: phabricator / wikitech / gerrit
** phab error logs not working ATM https://phabricator.wikimedia.org/T214176
* other scenarios: e.g. gerrit SQL injection
* hard to correlate events e.g. by IP now
* Chase would like to increase ldap verbosity (e.g. for bruteforce attacks, inappropriate use of ldap as authentication backend)
* Filippo: problem, logs are one giant bucket access wise now (NDA)
* Multi tenancy exploration / stretch goal https://phabricator.wikimedia.org/T213902
* When investigating incidents security needs to persist relevant logs outside the retention policy (90d)
** e.g. different elasticsearch indices with longer retention
* Keith: what would provide the most value next?
** gerrit + phab + ldap
* Moritz: openldap logging is kinda insane and all-in, we need to gauge logging volume
* Chase: we could filter out uninteresting logs and send to logstash interesting authentication activity
* Keith: "password policy" could be investigated too to log interesting events
* Chase: password policy, there hasn't been one historically
Actions:
* phab error logs investigation https://phabricator.wikimedia.org/T214176
* phab access logs to be added
* gerrit access logs to be added
Security update

When you notice a security report on one of the following lists you should first check debmonitor to see if we are affected and if we are follow the process here
 * debian-security-announce
 * debian-security-announce
 * ubuntu-security-announce
bla
== Phabricator ticket ==
[] Check for reported regressions
[] Review: check the actually fix patches to ensure that the are sound
[] Review: Ensure correct patches have been backported and are applied e.g. debdiff
[] check reverse dependencies services which may need restarts i.e. `debdeploy query_deps --packages libgd3 -s all`
[] build debdeploy spec file i.e. `generate-debdeploy-spec`
[] confirm restarts with debdeploy
[] deploy to canaries
[] deploy to fleet
== Check for reported regressions ==
First we need to make sure that no regressions have been introduced in the new packed. Check the Debian Tracker and review any bugs which have been raised since the security release.
== Review: Review the fix ==
You should review all vulnerabilities reported to ensure you have a good understanding of the risk profile theses updates resolve. You should also check that the fix looks reasonable.
== Review: Review the package ==
We should check the package to ensure that no bugs have been introduced by the update.
Specifically we need to check:
 * all patches included match closely what has been produced upstream
 * all patches are applied i.e. debian/patches/series
To do this you need to use debdiff to compare the newsource package with the one installed. You should be able to get the new and old source packages from boron (or the current build server). 
$ apt-get source $package=$oldversion
$ apt-get source $package=$newversion
$ debdiff $package$oldversion.dsc $package$newversion.dsc
if you are unable to get the source from apt then download it from snapshot
== Build a spec file and deploy ==
WARNING: please try to avoid making deployments on a Friday!
To build a spec file and deploy consult the Software deployments page. You should deploy the update first to a canary node. If the security update is not critical its good practice to allow the system to run for some time to try and tickle any bugs, how much time is a call the engineer should make based on the severity of the security issue. if the package being installed requires you to restart a service then ensure you consulted the Service restarts page to ensure you correctly [de]pool the service before restarting.
Once you are happy to proceed roll the update out to all systems, it is advisable to do this in batches and again ensure that safely restart and services which require restarting.
Sel

The system event log is used to log hardware events directly to the BMC. This allows events to be logged even if the Os is unable to. Currently we have use ipmiseld. We also export the ipmi_sel_logs_count metric so we can detect when new events have been logged however at this point a manual investigations is required to investigate what action to take. 
= Addtional information =
To get additional information one is able to use either the ipmi-sel command on the server or the racadm command getsel. The later is more likely to give more precise information however we the former often gives the necessary information to further progress the resolution
== ipmi-sel ==
ipmi-sel comes from the freeipmi tools and tries to interpret the messages based on standardised codes. running the command without any commands will give you some information e.g.
$ sudo ipmi-sel 
ID | Date | Time | Name | Type | Event
67 | Feb-14-2023 | 22:45:19 | SBE Log Disabled | Event Logging Disabled | Correctable Memory Error Logging Disabled ; OEM Event Data2 code = C0h ; OEM Event Data3 code = 40h
however you can get ipmi-sel to try and infer additional information by adding the following parameters --interpret-oem-data and --entity-sensor-names e.g.
sudo ipmi-sel --interpret-oem-data --entity-sensor-names 
ID | Date | Time | Name | Type | Event
67 | Feb-14-2023 | 22:45:19 | System Firmware SBE Log Disabled | Event Logging Disabled | Correctable Memory Error Logging Disabled ; DIMM A7
== racadm ==
Using racadm is likely to give you more accurate human readable information as it is using the dell interface, however this requires logging into the idrac interface
$ ssh cloudvirt1036.mgmt.eqiad.wmnet 
root@cloudvirt1036.mgmt.eqiad.wmnet's password: 
racadm>>getsel
Record: 67
Date/Time: 02/14/2023 22:45:19
Source: system
Severity: Critical
Description: Correctable memory error logging disabled for a memory device at location DIMM_A7.
-------------------------------------------------------------------------------
= Alerts =
When you receive an alert you should check the SEL using one of the aforementioned methods creating phabricator ticket for each actionable event then clear the log down using 
== Memory Errors ==
See Memory correctable errors -EDAC-
== Clearing SEL ==
once all events have been logged you should clear the sel log down. this can be preformed either using the sudo ipmi-sel --clear
Selling to Wikimedia Foundation

=== A note to Information Technology Vendors ===
Thank you for your interest in the Wikimedia Foundation.
We are frequently approached by vendors hoping for some of our time to discuss why we should purchase a product/service/software/solution from that company.
If you are interested in selling to Wikimedia, please start by reading the "Get involved" page maintained by our operations team.  Understand, however, that the Wikimedia Foundation is very strongly biased toward free software, so before trying to making an aggressive sales pitch, please read the Wikimedia Foundation Guiding Principles published by the Wikimedia Foundation Board of Trustees.
At Wikimedia Foundation, we pride ourselves with our level of accessibility to skilled technical collaborators. Many vendors make it clear that they fail to understand our values and way of working, and attempt manipulative and/or pushy techniques to reach us. Your best strategy in reaching us is going to be by demonstrating the depth of your understanding of our values and preferred way of working. As such, any closed source solutions are largely frowned upon, and would only be used if there was no viable open source alternative. So far, the only place we have found we cannot use open source software effectively is within a servers BIOS/firmware and within our network routing/switching platform. All of our other operational services and projects are open source.
Additionally, all of our infrastructure is documented and available via the above links and sub-links from those pages. Any suggestion for software to replace an existing software solution for one of our services should be fully researched and detail how it specifically applies to our infrastructure. Unfortunately, we cannot entertain every offer to converse about every potential solution, as we get dozens of these offers weekly.
When we launch a search for a vendor solution, we often announce it on our blog at https://wikimediafoundation.org/news/. Please follow it for these announcements, as the announcements will frequently provide detailed instructions about how to effectively present your product. 
Like the vast majority of people using email, each employee at Wikimedia Foundation frequently receives a lot of unsolicited email. Even only two emails from your organization is enough for many of us to mentally flag you as a "spammer". As such, you should avoid placing any of our email addresses on your promotional/mailing/sales/contact lists.
=== Advice for Wikimedia Foundation employees ===
You may receive frequent contact from salespeople for technology vendors, hoping for a "bit of your time" and pressuring you to connect them to "the right person" at our organization. Be strong! These people are frequently paid by commission, and will frequently get more aggressive and/or desperate when it appears as though they will miss their sales targets for their company's fiscal quarter. Their problem is not your problem. We have a brief email (see Selling to Wikimedia Foundation/Pointer) which you are free to use in response to the vendor in question.
Selling to Wikimedia Foundation/Pointer

__NOEDITSECTION__
== Copy ==
Thank you for your interest in the Wikimedia Foundation.
We are frequently approached by vendors hoping for some of our time to discuss why we should purchase a product/service/software/solution from that company.
Before we continue this conversation, please read the instructions at the following link:
https://wikitech.wikimedia.org/wiki/Selling_to_Wikimedia_Foundation
At Wikimedia Foundation, we pride ourselves with our level of accessibility to skilled technical collaborators. Your best strategy in reaching us is going to be by demonstrating the depth of your understanding of our values and preferred way of working. Additionally, all of our infrastructure is documented and available via the link above and sub-links to that page. Unfortunately, we cannot accept every offer to converse about every potential solution, as we get dozens of these offers weekly.
The link above also describes how we search for new vendors. Please kindly remove my email address from your promotional/mailing/sales/contact lists.
Thank you for your interest in the Wikimedia Foundation,