nyuuzyou/wikis huggingface dataset:
Superset

Apache Superset is a platform for data exploration, collaboration, and visualization.
== Instances ==
=== Wikimedia Cloud Services ===
A public instance of Superset is currently being tested. It has access to Wiki Replicas and to ToolsDB (public schemas only). Source code can be found here: https://github.com/toolforge/superset-deploy. Patches are welcome!
=== Data Engineering ===
The data engineering team runs a restricted access instance. For access, see Analytics/Data access#LDAP access.
== How do I ... ==
=== Get access to Superset? ===
Please read Superset Access.
=== See a list of recent queries from all users? ===
Look at the Query History under SQL Lab.
=== Run a SQL query? ===
Use SQL Lab, it can interactively build your query. See upstream documentation.
Database configuration has a map of which schema is in which database.
=== Collaborate with others? ===
Create a dashboard, and add additional owners. Publish the same dashboard and it will also be publicly visible. To publish, click the "Draft" text adjacent to the star icon. 
=== Create a visualization? ===
A chart can be created from any query. See the upstream documentation.
=== Use the API? ===
See https://superset.wmcloud.org/swagger/v1 and the upstream documentation.
=== Add a feature ===
For new upstream features, submit them to the Apache superset project: https://github.com/apache/superset. For specific configuration related to WMF deployments, see https://github.com/toolforge/superset-deploy.
Survey.wikimedia.org

This software is presently running on singer and is not using puppet. It will shortly move to hooper and use puppet for its apache vhost configuration shortly. The database resides on db9
== Usage ==
* In Production since 2008-11-14
== Installation instructions ==
* SVN checkout the current stable deployment:
 svn co https://limesurvey.svn.sourceforge.net/svnroot/limesurvey/source/limesurvey/ /srv/org/wikimedia/survey/
* Update install with our configuration file and interface translation (currently we have our own updated version of /srv/org/wikimedia/survey/locale/mk/)
:* Also backed up daily via singer's daily rsync to tridge.
* Update Puppet to have new server use misc::survey-wikimedia class.
* Update DNS to point the cname to the new server.
== OLD/OUTDATED: How to recover in a hurry ==
The entire web frontend is backed up to tridge from isidore once a week. It resides in the /data/isidore/ directory, along with other web facing software installed on isidore.
The database information resides on db9.
Category:Decommissioned services
Swap-watchdog

Quickie script Domas threw together for the PDF service to reboot the box when memory leaks start killing swap. Ideally we'll get the leak sorted out soon!
#!/usr/bin/python
import time
threshold=0.1
while True:
 swapused=0
 for line in open("/proc/swaps"):
 filename,type,size,used,priority=line.strip().split()
 if filename=="Filename": continue
 if int(used) > int(size)*threshold:
 open("/proc/sysrq-trigger","w").write("b\n")
 sys.exit(0)
 time.sleep(3)
Swift

Overview of Swift, as implemented in Wikimedia production.
== Quick Notes ==
* Swift cluster resource utilization: eqiad codfw
* Swift service dashboard: eqiad codfw
* Swift "four golden signal" dashboard: eqiad codfw
* How to do stuff to the cluster (add or remove nodes, etc.)
* Extra Related Processes - descriptions of the other processes running that interact with or affect swift
* Phabricator project "SRE-swift-storage"
* Logging and metrics - general overview related to logging and metrics
If you need a different document, most of the stuff written about swift in wikitech is part of the Swift Category.
== What is Swift ==
Openstack Swift is a scalable fault tolerant clustered object storage system. 
== Other documentation ==
* As Swift behaves as a general-purpose application layer, the CDN caching documentation will accurately reflect Swift caching behaviors
* Media storage is the documentation for the media storage infrastructure in general, including all other layers
== External links ==
* Swift developer documentation at OpenStack
Category:Swift
Swift/Dell R720XD setup

To set these up for use as backend hosts, you need to do the following extra steps before the PXE boot:
*If the system hangs forever on 'Scanning devices' and never proceeds to pxe boot, change the boot order in the BIOS to NIC first. Don't forget to change it back later after install.
*If they are going to be used for container listings, add two ssds. Due to their size etc, there's only one place they can be seated and cabled up, which will result in them showing up as disks 12 and 13 (numbering starts with 0) in the raid controller listing. You'll need to go into the raid controller bios and set disk 12 as the disk that boots. This option is found on the Control management screen, 'Select Boot Device'.
*In the raid controller bios, make sure all disks have been converted to non raid. This option is found on the VD management screen, F2 for operations (for the whole controller) or on the PD management screen, F2 for operations.
If on the boxes with ssds you see various bizarre errors like 'error creating raid' the safest thing is to dd out the first so many sectors on /dev/sdm and /dev/sdn from the installer shell, powercycle the box and try again. Preexisting md or other configuration information can horribly confuse the installer.
Swift/Deploy Plan - Shard Smaller Wikis

Category:Swift
This page describes the plan to introduce sharding containers for thumbnails to a number of wikis. Currently (2012-05-08) only commons and enwiki are sharded. We would like to shard all wikis with over 25 thousand objects to make sure there's plenty of headroom for growth. 
List of wikis to shard: 'commonswiki', 'dewiki', 'enwiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki'
List of containers to shard: wikipedia-commons-local-thumb,wikipedia-de-local-thumb,wikipedia-en-local-thumb,wikipedia-fi-local-thumb,wikipedia-fr-local-thumb,wikipedia-he-local-thumb,wikipedia-hu-local-thumb,wikipedia-id-local-thumb,wikipedia-it-local-thumb,wikipedia-ja-local-thumb,wikipedia-ro-local-thumb,wikipedia-ru-local-thumb,wikipedia-th-local-thumb,wikipedia-tr-local-thumb,wikipedia-uk-local-thumb,wikipedia-zh-local-thumb
Order of operations:
* create sharded containers in swift (following instructions) [DONE]
 pass='aoeu'
 for cont in wikipedia-de-local-thumb wikipedia-fi-local-thumb wikipedia-fr-local-thumb wikipedia-he-local-thumb wikipedia-hu-local-thumb wikipedia-id-local-thumb wikipedia-it-local-thumb wikipedia-ja-local-thumb wikipedia-ro-local-thumb wikipedia-ru-local-thumb wikipedia-th-local-thumb wikipedia-tr-local-thumb wikipedia-uk-local-thumb wikipedia-zh-local-thumb ; do
 for i in {0..9} {a..f} ; do for j in {0..9} {a..f} ; do 
 echo "creating container $cont.$i$j"; 
 swift -A http://127.0.0.1/auth/v1.0 -U mw:thumbnail -K $pass post -r '.r:*' ${cont}.$i$j
 done; done
 done
* change mediawiki configuration to look at the sharded containers for purging [DONE]
* change rewrite.py in swift to use the sharded containers [DONE]
* verify it's all working and happy [DONE]
** checked new objects get created and verified that purging still works [DONE]
* wait a day or two [DONE]
* delete all the content in the old unsharded containers and drop them [in progress]
We are not prepopulating the sharded containers; image latency for the newly sharded containers (all in the list except commons and enwiki) will degrade for a period while (a few days to a week is my guess)
Swift/Deploy Schedule - Originals

This page details the steps necessary to get originals running in swift, with more detail around the actual cutover process.
== Overall Plan ==
These steps are in the rough order of execution but is not strictly ordered.
* increase the size of the swift cluster with new hardware (waiting on new racks to get set up)
* deploy mediawiki code to write a log of all media changes ('changelog') to use to synchronize backend stores DONE
* deploy configuration to enable the media changelog DONE
* rewrite cloudfiles to allow parallel purging DONE
* FileBackend based thumbnailer (more detail) DONE
** aka make mediawiki write thumbnails to swift instead of the 404-handler rewrite module DONE
* start process to migrate all data from ms7 to swift (initial snapshot) (eta start time week of 5/1)
* write apache rewrite rule to allow image scaler apaches to handle upload.wikimedia style URLs (to let us take ms5 out of the loop)
* wait for data migration/population into swift to complete
* cut over to swift as the primary backend store
Things not in the above list that we should do
* test originals in swift in labs
* create and test (in labs) rollback plan
Things not in the above list that don't actually block originals but it would be nice if they happened first
* mirror swift cluster to eqiad
* upgrade swift version
== Mediawiki Writes Thumbnails ==
At the moment, the image scalers write thumbnails to NFS and swift's rewrite module writes thumbnails into swift. The goal for this change is to have Mediawiki write the thumbnails into swift directly in addition to NFS. ms5 will stay in the image request path. A new config parameter is added to rewrite to allow us to choose to write thumbnails based on the container.
Plan is to make the following changes for a few wikis Thursday 5/18 and the rest of them Tuesday 5/20:
* adjust rewrite.py config (in puppet); set write_thumbs to 'most' and the list to the containers for which MW will write the thumbs
* adjust MW config to list the containers (wikis) for which it will write thumbs
* test!
Category:Swift
Swift/Icehouse

Category:Swift
There are currently (July 2014) two swift clusters running (esams and eqiad). esams is used to sync files (manually) from eqiad every now and then (though it is currently pending expansion due to lack of disk space) whereas eqiad is in production to serve originals and thumbnails.
== upgrade to icehouse ==
As part of a overhaul of swift it is necessary to upgrade to the latest upstream version (v1.13.1, codename icehouse). The new version will let us, among other things, setup proper geocluster replication between eqiad and codfw (see the full changelog).
The upgrade has been tested on a minimal cluster in labs and subsequently in esams without any adverse effect. As per the recommended upgrade procedure, a backend has been upgraded and let run for some time, followed by all the remaining backends, followed by a frontend, followed by the remaining frontends. 
== proposed timeline for eqiad ==
eqiad will follow a similar upgrade procedure to what has been used in esams, with the more conservative timeline outlined below:
* 2014-07-02T08:00Z (Wed): upgrade ms-be1001
* 2014-07-07T08:00Z (Mon): upgrade ms-be1002/1008 (zone1) + ms-be1003/1004/1012 (zone2)
* 2014-07-09T08:00Z (Wed): upgrade ms-be1005/1006/1007 (zone3) + ms-be1009/1010/1011 (zone4) + ms-be1013/1014/1015 (zone5)
* 2014-07-14T08:00Z (Mon): upgrade ms-fe1001
* 2014-07-16T08:00Z (Wed): upgrade ms-fe1002/1003/1004
== upgrade procedure and rollback ==
the upgrade itself is straightforward enough because openstack makes debian packages available for precise, which we pin at a particular priority to make them installable along with all the dependencies. thus:
* get the proper pinning in place in puppet (something similar to https://gerrit.wikimedia.org/r/#/c/141677/)
* on each machine that is going to be upgraded:
 apt-get update
 apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install swift
 swift-init all restart
=== rollback ===
rollback is similarly straightforward, revert the pinning and execute the same commands to get the older version installed
== post-upgrade ==
As of 2014-07-18 the upgrade has been completed with no issues reported so far. However the backend bandwidth (ms-be* RX/TX) has seen a steady increase after the frontend have been upgraded to icehouse.
Swift/Load Thumbnail Data

Category:Swift
This page details the method of loading existing thumbnails into swift pre-deploy. 
= new method =
ms5 is currently suffering severe load. To avoid increasing that load, we don't want to run 'find' on ms5 (the old method). 
== capture incoming thumbnail requests ==
On ms5, we run tcpdump to capture incoming thumbnail requests. We ship them off to fenari, where they're processed and requested from squids (which have just received the object back from ms5) where they're served from cache
 root@ms5:~# tcpdump -i any -s 0 dst port 80 and dst host ms5.pmtpa.wmnet -A | grep GET \
 | grep -o "[^ ]*/thumb/[^ ]*" | nc fenari.wikimedia.org 29876
== stuff images into swift ==
on fenari, we run a listener that
* processes the incoming list of URLs from ms5's tcpdump
* holds on to each URL for about 30s to make sure it's present in squid
* requests the URL from swift, which falls through to upload.wikimedia.org (the squids) on 404
 ben@fenari:~/swift$ ./urllistener-fifo
== watching progress ==
Ganglia is graphing the total number of objects as well as the number of new objects per 30s time slice. If either of these metrics stagnate, we should verify the tcpdump and urllistener are still runnning (in a screen session ben/listener on fenari - you can connect as root)
----
----
----
= old method (as of 2012-01-30) =
== success condition ==
The goal is only 99% coverage; it's ok to not get 100% of thumbnails. Any that we miss will be picked up from ms5 or regenerated when they're requested.
additionally, only publicly visible thumbnails are getting retrieved for this first iteration
== get a list of existing thumbnails from ms5 ==
ionice -c 3 specifies the 'idle' priority.
cd /export/thumbs
for i in wikibooks wikimedia wikinews wikipedia wikiquote wikisource wikiversity wiktionary
do
 ionice -c 3 find $i -type f > /tmp/$i-filelist.txt
done
== exclude commons for test clusters ==
Until we clear out the google-generated stuff, it's just wasted space. We should skip anything on commons for testing. Note - for production; we should include commons, aka skip this step
for i in *-filelist.txt; do grep -v "/commons/" $i > ${i/.txt/-nocommons.txt}; done
== transform the list into URLs ==
Turn the list of paths into a list of URLs that should be provided by swift. Loading each URL will cause the file to be fetched from ms5 and saved to swift
swift_frontend="msfe-pmtpa-test.wikimedia.org:8080"
for i in *-nocommons.txt; do cat $i | sed -e "s/^/http:\/\/${swift_frontend}\//" > ${i/.txt/-urls.txt}; done
== load all these urls ==
fetch them all from swift - run this on hume or some other host in pmtpa
cd ~ben/swift
for i in *-urls.txt; do ./geturls.py -t 30 $i; done
Swift/ObjectExpiration

Investigation in how object expiration works in swift.
Swift supports expiring objects by passing in custom headers (X-Delete-At/X-Delete-After) (see expiring objects in swift
This process is implemented internally by having an "expirer account" where objects are added as they are asked to be deleted. If during a regular request the "TTL headers" are spotted by the proxy they will be forwarded to the backend which then does something like this:
 if delete_timestamp:
 add_to_account(".expired-objects", delete_timestamp, "%(timestamp)-%(account)/%(container)/%(object)")
so inside the .expired-objects account there will be several containers named after the timestamp at which the objects inside should be deleted. Inside each container there is an object that encodes which (account, container, object) to delete.
This "queue" is processed by swift-object-expirer in the background by reading the container list and deleting objects accordingly, the process itself can have multiple levels of concurrency (added in 209c5ec418):
* A single expirer process can run a pool of workers to farm out requests to delete objects
* Several expirer processes can work on the same queue via a simple modulo-based sharding
=== caveats ===
* We are running icehouse (1.13) which doesn't contain this fix for expiring objects not available 8d20e0e927d
* If done for all thumbnails this would increase the object count by whatever (TODO: how much?) thumbnail churn we have over a month (TODO: check if correct), which we might not want to do. Another approach would be to expire only non-standard thumbnail sizes to save some (TODO: how much?) space and not have a big expiry queue (assumption being that non-standard sizes are the minority, TODO: check this)
* Other noteworthy commits on the object expirer that we got during the upgrade:
** fixed bug 1187200 fef2afd9271
** Fixed bugs with internal client and object expirer f46a4d8a2f3f8
Category:Swift
Swift/Other Processes

== Current ==
=== Ganglia Logtailer ===
See swift::base in puppet for this.
* ganglia-logtailer process running on all swift nodes via cron (front and back ends)
* parses logs to gather performance and throughput information, feeds that into ganglia
=== swift-ganglia-report-global-stats ===
See role::swift::pmtpa-prod::ganglia_reporter in puppet for this.
* runs on one server per cluster, currently ms-fe1
* reports cluster-wide statistics to ganglia
** object count, bytes used, change in # objects
** these aren't related to a specific server but instead reflect cluster-wide stats
== Old ==
=== purgethumbs.py ===
Unused; can't find source to this anywhere; does it still exist?
* purges thumbnails from all media stores
* takes in a list of images to purge and attempts to purge all thumbnails for that list of images from all three locations: ms5, swift, and squid
* used to clean out unreferenced thumbnails generated by ariel's script
=== delete-containerless-objects ===
Can't find source to this anywhere; does it still exist?
* script to crawl the object contents in a swift backend, compare this to a list of what's in the container, and delete objects that exist but aren't in the container list.
** note that this shouldn't happen in the first place
=== Swift Cleaner ===
No longer running.
* compared thumbnails on swift to those on ms5. deleted things on swift that didn't match what's on ms5.
* two instances used to run on iron - full scan and incremental scan. 
** incremental took about 2 hours
** full took about 2 weeks
* was installed in /opt/swiftcleaner. canonical source in git:operations/software/swiftcleaner
Category:Swift
Swift/Preliminary test plan

Category:Swift
This page documents a preliminary test plan to get a basic handle on how swift performs, as well as a battery of functional tests to verify swift is working correctly.
== Background information ==
Some statistics from ms5, to be used as a starting point for the tests
* How many images per second does ms5 serve? about 110/s at peak (from ganglia
* How many requests per second does ms5 send back to the image scalers? about 50/s at peak (from ganglia
* How much storage does ms5 have? 10T, 8T used. (ms7: 27T, 18T used)
* What is the rate of disk consumption on ms5? Historic: 0.3T/mo. Current: 1.2T/mo (ms7: 1T/mo)
* How long does it take to request 1000 images on ms5 (sequentially)?
** note - this test is performed during normal load, so it's actually 1000 images plus regular load
* How long does it take to request 1000 images on ms5 (with 20 clients)?
** note - this test is performed during normal load, so it's actually 1000 images plus regular load
== Test platforms ==
=== First iteration ===
three machines using high performance misc servers
* X CPU
* Y RAM
* 2 disks
two machines are configured as proxy machines, three as storage nodes (the two proxy servers are also storage nodes)
=== Second iteration (proposed) ===
separate proxy nodes from storage nodes - five high performance misc servers (two proxy, three storage)
=== production config (proposed) ===
* three proxy nodes - high performance misc
* 6 storage nodes - ES hardware (grow to 10 over 1yr)
== Tests ==
=== speed ===
a list of 30,000 images (pulled from a 3min tcpdump off ms5) is at fenari:~ben/swift/list-of-images. This can serve as input to performance tests. 
(30,000 chosen because peak load on ms5 is ~100qps == 6000qpm. 5 minutes =~ 30k.)
* How many images per second can swift serve?
* How long does it take to request 30000 images on swift (sequentially)?
* How long does it take to request 30000 images on swift (with 20 clients)?
* How does swift maintenance (rebalancing, etc) affect performance? (fail out a disk and retest 1000 image retrieval)
=== functional against swift directly ===
* upload file
* retrieve file
* rename file
* delete file
=== functional through mediawiki - thumbnails ===
* request a thumb that exists on swift
* request a thumb that doesn't exist on swift (but whose original does exist), watch it get created and stored
** re-request, verify it comes from local disk
* request a thumb for an original image that doesn't exist, verify correct error handling
* purge the image cache, verify the image gets deleted from swift
=== functional through mediawiki - originals ===
* upload file
* upload new version of file
* revert to previous version of file
* rename file
* delete file
Swift/Server issues Aug-Sept 2012

We've seen some issues with the ms-be boxes (all C2100s) in tampa.
Current status as of this writing:
Hardware Issues and Troubleshooting.
*Per conversation with Dell, checked the jumpers on the backplane to ensure that j15 was indeed empty. After examining the backplanes on ms-be6,7,8, it is confirmed to have only the 3 jumpers and j15 is empty.
*ms-be6 replaced all 12 HDD w/different manufacturer .
*ms-be6 has a replaced main board, replaced, backplane, replaced sas2008 controller card
*ms-be6 replaced the LSI sas2008 controller that supports jbod with a LSI 9680 that does not support jbod. The card and configuration change did not fix the issue. (several disk had issues mounting)
These boxes are powered on, have had puppet disabled in /etc/default/puppet and the swift processes are shut off via swift-init stop all.
*ms-be6 has had ssds uncabled and power pulled, reinstall, latest LSI driver (mpt2sas0), latest controller firmware. After recreating all xfs filesystems on /dev/sdc and up, by hand, and manual remount, it boots and continues to see all drives and mount them. It reported a degraded raid array on /dev/md0 (= /dev/sda and /dev/sdb, /deb/sdb is the one that fell out of the raid), I repaired the raid, a couple hours later it reported degraded again. So we can use this box for testing the other drives but not put it back into the rings.
*ms-be7 has ssds which are cabled up. I followed the same procedure of recreating all xfs filesystems except for the os, remounting manually, rebooting, and it sees all drives. We can put this back into the swift rings next Monday. (After re-enabling puppet and restarting swift processes on the box)
*ms-be10 has ssds which are cabled up. After recreating all xfs filesystems except for the OS, remounting manually, on reboot it reports a few disks (not always the same ones) as not ready/not present, but if you wait it out they eventually mount. Obviously this is a problem. I don't think this should go back into production.
These boxes are powered off.
*ms-be8 replaced the sas disk with NL SATA disks. Once data was written to the disk and they were rebooted several times, the disk failed. "firmware status: failed"
These boxes have one problem disk.
*ms-be5 has one disk reporting errors, needs replaced once other servers are stable
*ms-be11 has one disk replaced but it shows the wrong logical id so it needs reboot once other servers are stable