WMDE/Wikidata/Scaling

== Current & Future ==
=== WDQS ===
Needed for: Increased edit rate. Increased WDQS queries
Right now "Disaster mitigations" are underway.
Long-term looks like evaluation of new services: https://phabricator.wikimedia.org/T206560
Also Wikidata_query_service/ScalingStrategy
=== Abuse filter ===
Needed for: Larger entity sizes.
Various things have been worked on in this area, including https://phabricator.wikimedia.org/T204109 and other tickets avoiding processing here.
Ultimately the AbuseFilter needs re working in order to make it work better for the Wikibase usecase, or other more specific solutions are needed.
=== Entity size / serialization ===
Needed for: Larger entity sizes.
As is highlighted in https://addshore.com/2021/07/what-happens-in-wikibase-when-you-make-a-new-item/ there is quote some overhead with editing larger and larger items.
=== Revision table? ===
Needed for: More revisions
In 3 years time we will likely have 2 billion rows in the mediawiki revision table.
Is this actually an issue? Do we just need some bigger hardware? Sharding?
See also T63111 for various 32-bit primary keys.
Other things come into play here, for example changing Wikidata editing patterns to mean that less revisions happen..
=== Dump generation ===
Weekly dump generation: https://phabricator.wikimedia.org/T206535
All revisions: https://phabricator.wikimedia.org/T221504
=== Recent Changes ===
MediaWiki is not optimized for having so many changes and also for having so many changes in a single NS and not many in the others..
On a social side RC is also quite unweildy.
== Past ==
=== wb_terms ===
Removed: https://phabricator.wikimedia.org/T208425
=== CacheAwarePropertyInfoStore ===
Needed for
*Increased property count. Increased request rate(/more app servers?)?
*Continued stability
Links
* Ticket: https://phabricator.wikimedia.org/T97368
* Monitoring: https://grafana.wikimedia.org/d/000000574/t204083-investigation?orgId=1
Timeline
* 28/04/2015 Hoo originally filed the ticket.
* 11/09/2018 Krinkle marked the ticket as high as we are slowly reaching the limit of the pipe for the memcached machines https://phabricator.wikimedia.org/T97368#4575476
* 08/10/2018 Added 10 seconds of apc cache, https://phabricator.wikimedia.org/T97368#4649272
* 30/10/2018 Switch from 10 seconds to 15 seconds on server cache
* 13/11/2018 Adding 15 seconds of on server cache reduced the traffic to the memcached server by 50%, but the issue is still there and will continue to grow https://phabricator.wikimedia.org/T97368#4741600
* 19/12/2018 ADR written about what was done
* 10/03/2019 Causing issues again, ticket back open not only because of reads, also writes see https://phabricator.wikimedia.org/T97368#5018549
* April 2019 WANObjectCache used https://phabricator.wikimedia.org/T218197
=== Rate of entity Id acquisition ===
Needed for: Increased entity creation rate
* Ticket: https://phabricator.wikimedia.org/T194299
* Monitoring: https://logstash.wikimedia.org/goto/b2a28cc42a9203bfe1d796929696fe55
fname:"Wikibase\\SqlIdGenerator::generateNewId" AND db_name:"wikidatawiki" AND channel:"DBQuery" AND error:"Lock wait timeout exceeded; try restarting transaction"
Timeline
* 19/12/2018 - Deployed upsert id generator
* Feb 2019 - Separate DB connection used for ID creation - https://phabricator.wikimedia.org/T213817
=== Change dispatching ===
Needed for: Edit rate
Change dispatching was slow and often backlogged.
This limits the rate of editing that can happen on wikidata.
Fix: https://phabricator.wikimedia.org/T205865
With the fix in place, we managed to switch the number of dispatchers from 4 to 2. We switched back to 3 a while later when turning on data access to a bunch of wiktionaries.