nyuuzyou/wikis huggingface dataset:
Cassandra/Tools/cassandra-ca-manager

Easily manage Java keystores using a self-signed certificate authority.
== Project ==
* Github project page
== Usage ==
=== Step 1 ===
Create a YAML-formatted manifest describing the certificate authority and machine certificates.
# The top-level working directory
base_directory: /path/to/base/directory
# The Certificate Authority
authority:
 key:
 size: 2048
 cert:
 subject:
 organization: WMF
 country: US
 unit: Services
 valid: 365
 password: qwerty
# Java keystores
keystores:
 - name: restbase1001-a
 key:
 size: 2048
 cert:
 subject:
 organization: WMF
 country: US
 unit: Services
 valid: 365
 password: qwerty
 - name: restbase1001-b
 key:
 size: 2048
 cert:
 subject:
 organization: WMF
 country: US
 unit: Services
 valid: 365
 password: qwerty
 - name: restbase1002-a
 key:
 size: 2048
 cert:
 subject:
 organization: WMF
 country: US
 unit: Services
 valid: 365
 password: qwerty
=== Step 2 ===
Run the script with the manifest as its only argument.
$ cassandra-ca-manager manifest.yaml
$ tree /path/to/base/directory
/path/to/base/directory
├── restbase1001-a
│ ├── restbase1001-a.crt
│ └── restbase1001-a.csr
│ └── restbase1001-a.kst
├── restbase1001-b
│ ├── restbase1001-b.crt
│ └── restbase1001-b.csr
│ └── restbase1001-b.kst
├── restbase1002-a
│ ├── restbase1002-a.crt
│ └── restbase1002-a.csr
│ └── restbase1002-a.kst
├── rootCa.crt
├── rootCa.key
├── rootCa.srl
└── truststore
3 directories, 13 files
Note: cassandra-ca-manager is idempotent, it will not overwrite any existing material in the base directory. To create additional keystores later, simply add their definitions to the manifest, and re-run the script.
=== Step 3 ===
Copy the resulting truststore and *.kst files to their respective machines, and configure server and/or client encryption accordingly.
Cassandra/Tools/cassandra-metrics-collector

Discovers running instances of Cassandra on the local machine, collects performance metrics (via JMX), and writes them to a remote Carbon server in a format compatible with the Dropwizard metrics GraphiteReporter.
== Project ==
* Github project page
== Usage ==
$ java -jar target/cassandra-metrics-collector-<version>-jar-with-dependencies.jar --help
NAME
 cmcd - cassandra-metrics-collector daemon
SYNOPSIS
 cmcd [ {-di | --discovery-interval} <INTERVAL> ]
 [ {-f | --filter-config} <YAML> ] [ {-h | --help} ]
 [ {-H | --carbon-host | --graphite-host} <HOSTNAME> ]
 [ {-i | --interval} <INTERVAL> ]
 [ {-p | --carbon-port | --graphite-port} <PORT> ]
OPTIONS
 -di <INTERVAL>, --discovery-interval <INTERVAL>
 Interval (in seconds) to perform (re)discovery (default: 300
 seconds)
 -f <YAML>, --filter-config <YAML>
 Metric filter configuration
 -h, --help
 Display help information
 -H <HOSTNAME>, --carbon-host <HOSTNAME>, --graphite-host <HOSTNAME>
 Carbon hostname (default: localhost)
 -i <INTERVAL>, --interval <INTERVAL>
 Collection interval in seconds (default: 60 seconds)
 -p <PORT>, --carbon-port <PORT>, --graphite-port <PORT>
 Carbon port number (default: 2003)
Category:Metrics
Cassandra/Tools/cdsh

cdsh is a simple wrapper for dsh. It allows you to define a single hierarchical configuration for remote command execution, by cluster, data-center, and rack.
== Project ==
* GitHub project page
* Debian package download
== Configuration ==
# Save as ~/.cdsh
# The restbase cluster (-c restbase)
restbase:
 # Data-center 'eqiad' (-d eqiad)
 eqiad:
 # Rack a (-r a)
 a:
 - restbase1007.eqiad.wmnet
 - restbase1010.eqiad.wmnet
 - restbase1011.eqiad.wmnet
 # Rack b (-r b)
 b:
 - restbase1003.eqiad.wmnet
 - restbase1004.eqiad.wmnet
 - restbase1008.eqiad.wmnet
 # Rack d (-r d)
 d:
 - restbase1005.eqiad.wmnet
 - restbase1006.eqiad.wmnet
 - restbase1009.eqiad.wmnet
 # Data-center 'codfw' (-d codfw)
 codfw:
 # Rack b (-r b)
 b:
 - restbase2001.codfw.wmnet
 - restbase2002.codfw.wmnet
 - restbase2007.codfw.wmnet
 # Rack c (-r c)
 c:
 - restbase2003.codfw.wmnet
 - restbase2004.codfw.wmnet
 - restbase2008.codfw.wmnet
 # Rack d (-r d)
 d:
 - restbase2005.codfw.wmnet
 - restbase2006.codfw.wmnet
 - restbase2009.codfw.wmnet
aqs:
 datacenter1:
 rack1:
 - aqs1001.eqiad.wmnet
 - aqs1002.eqiad.wmnet
 - aqs1003.eqiad.wmnet
== Usage ==
$ cdsh -h
usage: cdsh [-h] [-c CLUSTER] [-d DATA_CENTER] [-r RACK] [--config CONFIG]
 [-P]
 [args [args ...]]
A dsh wrapper for Cassandra
positional arguments:
 args arguments to dsh
optional arguments:
 -h, --help show this help message and exit
 -c CLUSTER, --cluster CLUSTER
 cluster name
 -d DATA_CENTER, --data-center DATA_CENTER
 data-center name
 -r RACK, --rack RACK rack name
 --config CONFIG yaml configuration file
== Examples ==
Output kernel version for all machines in the restbase cluster, eqiad data-center, rack a:
$ cdsh -c restbase -d eqiad -r a -- uname -r
restbase1007.eqiad.wmnet: 4.4.0-1-amd64
restbase1010.eqiad.wmnet: 4.4.0-1-amd64
restbase1011.eqiad.wmnet: 4.4.0-1-amd64
To print all of the hosts in the aqs cluster:
$ cdsh -c aqs -P
aqs1001.eqiad.wmnet
aqs1002.eqiad.wmnet
aqs1003.eqiad.wmnet
Cassandra/Tuning

== Compaction ==
List current compactions: nodetool compactionstats
Check the compaction status of a particular keyspace:
 nodetool -h restbase1005.eqiad.wmnet cfstats local_group_wikipedia_T_parsoid_html.data
 (...)
 SSTables in each level: [23/4, 52/10, 190/100, 1269/1000, 2817, 0, 0, 0, 0]
This means that level 0 (new data, potentially overlapping) has 23 sstables, when it really should only have four (the /4 is only shown if the actual number is above the threshold). This means that compactions for this keyspace are behind. Since level 0 sstables can overlap, this can cause a high number of sstables to be read for a query, which is bad for latency. See this stack exchange post for more background on leveled compaction.
=== Concurrency ===
Set concurrent_compactors to 10 to avoid compaction being CPU-bound. With leveled compaction there can be many concurrent compactions on the same keyspace; we observed 10 concurrent compactions for a single, large keyspace in prod. 
=== Throughput ===
Compaction throughput affects GC pressure and iowait. Generally, compaction throughput should leave enough headroom for request processing without backing up. This means that iowait should average at most 5% when compacting at full concurrency. In production (3x Samsung SSD RAID0), this is achieved with compaction_throughput_mb_per_sec: 120. A good method to check this is to dynamically set a throughput using
 for i in `seq 1 6`; do
 nodetool -h restbase100$i.eqiad.wmnet setcompactionthroughput 120
 done
The throughput seems to only apply to new compactions, so wait until all ongoing compactions listed in nodetool compactionstats are done & replaced with new ones before starting to watch vmstat for iowait.
== Read and write concurrency ==
Limiting the concurrency of read and write request processing is a major load limiting and -shedding knob in Cassandra.
A moderate concurrent_reads (ex: 18) has a limiting effect on heap pressure induced by read bursts. At 96, we saw nodes spiraling into OOM death after heavy reads, which in turn led to writes backing up.
Similarly, a concurrent_writes of 18 or below limits memory used during write bursts. Additionally, Cassandra relies on relatively low (2s) write timeouts to avoid OOM. Setting that timeout to 5s significantly increased OOM probability under load, at least in combination with a high concurrent_writes setting of 96. It might be worth testing a low concurrent_writes of 16-18 in combination with a longer write timeout.
== SSTable indexing interval ==
Cassandra maintains index offsets per partition to speed up the lookup process in the case of key cache misses (see cassandra read path overview). By default it samples a subset of keys, somewhat similar to a skip list. The sampling interval is configurable with min_index_interval and max_index_interval CQL schema attributes (see describe table). For relatively large blobs like HTML pages we seem to get better read latencies by lowering the sampling interval from 128 min / 2048 max to 64 min / 512 max. For large tables like parsoid HTML with ~500G load per node this change adds a modest ~25mb off-heap memory.
Cassandra/Upgrades and testing

== Developing a Plan ==
Part of the test and evaluation of new versions is the development of a strategy for rolling out the new version to production. Some items to keep in mind when developing such a plan:
* The process should begin with the upgrade of a single instance, a so-called canary. When feasible, consider using a node from the stand-by data-center as the canary to limit user-facing impact in the event of a problem. 
* Establish in advance a comprehensive set of Go/No-Go criteria. Know what needs to be tested/measured beforehand, and what the results must before continuing, holding, or rolling back.
* Test the upgrade plan; When upgrading a development, beta, or staging environment, closely follow the process that will be used in production. You're not just vetting the software for production, you are vetting the process for rolling it out as well.
== Things to test before a major version upgrade ==
* Upgrade one node at a time, and perform the full set of tests in both mixed operation & after a full upgrade.
* Make sure to generate enough load to provoke failures. A simple technique is to run 2 or more html dumps from different offsets / for different projects.
* Decommissions & bootstraps in mixed operation & after upgrade
** Some bootstrap failures in this space only happen with large data sets / specific compaction strategies. In the past, we had consistent bootstrap failures on 2.1 when using Leveled Compaction and data sets greater than ~700G per instance.
*** Open question: Can we provoke common failure modes without having the full data set? 
* Performance: Benchmark app throughput before / after
** In-memory performance: ab -c 300 -n 100000 <url>
** More realistic performance: Multiple concurrent HTML dumps, as described above.
** Check IO metrics (disk read & write throughput, iowait), look for major changes
** Check network metrics, look for major changes
* Inspect logs
== Checklist ==
=== Node.js driver and native protocol versions ===
In the absence of an explicitly configured maxVersion, the Node.js driver will use a constant as the default. As the driver works through the contact points at startup, it will attempt to use this default, and failing that, decrement and try again. Once it has established the first successful connection, this negotiated protocol version is used for all subsequent connections, and any node that does not support it is skipped entirely. This can result in severe imbalances of client connections during an upgrade where the default native protocol version is higher than the version being upgraded.
When upgrading, be aware of any changes to the default native protocol version, and ensure that protocolOptions.maxVersion is explicitly configured for the least common denominator node until such time as the upgrade is complete.
See also: Mixed cluster versions and rolling upgrades
== Past Upgrades ==
=== 2.1.13 -to- 2.2.6 ===
==== RESTBase ====
* Incident documentation/20160531-RESTBase
* https://phabricator.wikimedia.org/T126629
Catalyst

Project Catalyst is a team working on a kubernetes cluster as a service explicitly for CI and exploratory testing. Our current effort is upgrading Patch Demo, to use kubernetes to provision and host wikis with custom extensions and skins. The Catalyst API wraps a small kubernetes cluster. A new instance of Patch Demo will consume this API. Our next effort will be to open the Catalyst API for more general workloads, including wikis which rely on (micro)services.
== The Need for Catalyst ==
The evolving complexities introduced with projects like Abstract Wikipedia require adaptable testing environments. Catalyst steps in to fill this void, offering a platform that can easily cater to a myriad of testing scenarios by offering test environments that resemble production configurations with extensions and services.
== Roadmap ==
* Summer 2024 - deploy a new kubernetes backed Patch Demo instance and begin sun-setting the legacy instance
* Fall 2024 - work with Abstract Wikipedia and other teams to bring their testing workloads onto the Catalyst kubernetes cluster
=== Project Updates ===
* Catalyst/Updates/2025-02-16
* Catalyst/Updates/2025-02-09
* Catalyst/Updates/2025-02-02
* Catalyst/Updates/2025-01-26
* Catalyst/Updates/2025-01-17
* Catalyst/Updates/2025-01-10
* Catalyst/Updates/2024-12-05
* Catalyst/Updates/2024-12-02
* Catalyst/Updates/2024-11-25
* Catalyst/Updates/2024-11-15
* Catalyst/Updates/2024-11-08
* Catalyst/Updates/2024-11-01
* Catalyst/Updates/2024-10-25
* Catalyst/Updates/2024-10-22
* Catalyst/Updates/2024-10-04
* Catalyst/Updates/2024-09-27
* Catalyst/Updates/2024-09-20
* Catalyst/Updates/2024-09-13
* Catalyst/Updates/2024-09-06
* Catalyst/Updates/2024-08-29
* Catalyst/Updates/2024-08-22
* Catalyst/Updates/2024-08-15
* Catalyst/Updates/2024-08-08
* Catalyst/Updates/2024-08-02
* Catalyst/Updates/2024-07-26
* Catalyst/Updates/2024-07-19
==== Decision Log ====
Catalyst/Decision log catalogues major decisions made by the team.
== Get Involved ==
We're fervently seeking community involvement in various ways:
* volunteer code contributors - the Project Catalyst team is very collaborative, and would welcome community involvement
* early adopters
** folks who want to try out the kubernetes backed Patch Demo
** teams interested in deploying and testing wikis with services in CI
If you're interested in getting involved, comment on Talk:Catalyst or email sdunlapwikimedia.org.
You can file bug reports in Phabricator under #Catalyst.
== Documentation ==
* Catalyst/Scaling
Catalyst/Decision log

Date 	Category 	Decision 	Why? 	Comments
2024-08-12	Tech solutions	Maintain an .htaccess of wikis previously created by legacy patchdemo that 301 redirects from k8s legacy patchdemo to the existing environment. 	We're doing this in the hopes of preserving Phabricator links to existing wikis in a way that is transparent to users. 	Decided in Catalyst planning meeting.
2024-08-12	Strategy 	Switch the existing https://patchdemo.wmcloud.org backed by WMCS VM to https://patchdemo-legacy.wmcloud.org on 2024-08-15. Point existing URL to k8s-backed, containerized legacy patchdemo.	End users exercise k8s-backed patchdemo. 	Decided in Catalyst planning meeting.
2024-11-21	Tech solutions	We will not be doing automatic skin/extension/service dependency management and linking during the demo creation process. 	Our users are knowledgeable enough to manage these requirements while creating the environments they need. Given this and the fact that users have been able to complete their objectives before without it, we have determined that the technical cost required to implement this cannot be justified.	Decided in Catalyst/Patchdemo planning meeting
Catalyst/Scaling

Project Catalyst will need to scale as it takes on more workloads. The original layout topology was 1 VM with 4 cores and 8 GB of memory. The general plan will be to add more kubernetes nodes (VMs) as needed but this creates some interesting considerations.
== How to add more nodes ==
Adding additional nodes to k3s is straight forward, but please note the "Unknowns" below.
# request additional resources from CloudVPS (if needed) for the catalyst project
## be sure to request enought storage for the new nodes (80 GB per node is the current recommendation)
# create a new VM
## on horizon:
### create a new 80 GB volume
### create new instance with
#### latest Debian image
#### 4 cores and 8 GB of memory
#### the http security group the the VM
#### the 80 GB volume attached
## on the new VM
### setup k3s-data partition
#### mkfs.ext4 /dev/vdb
#### mkdir /k3s-data
#### echo "/dev/vdb /k3s-data ext4 defaults,nofail 0 2" >> /etc/fstab
#### mount -a
### install k3s
#### rsync --rsync-path="sudo rsync" -L k3s.catalyst.eqiad1.wikimedia.cloud:/var/lib/rancher/k3s/server/node-token /tmp/node-token
#### curl -sfL https://get.k3s.io | sh - --data-dir /k3s-data --token-file /tmp/node-token --server k3s.catalyst.eqiad1.wikimedia.cloud
#### rm /tmp/node-token
#### sudo kubectl get nodes
== Unknowns ==
We use the Rancher's Local Path provisioner for persistent storage. Normally it is safe to add more nodes with the following caveats:
# data in the local-path provisioner will not be replicated to and cannot be shared with the new nodes
## we use have a "repository" pool for new environments to quickly get extensions, etc. by leveraging the multiple persistent volumes that point to the same location on the node's disk
## the pool is populated by patchdemo
## we have a story to have a separate service manage the pool phab:T376273, but we'll have to ensure that service is running on every node (or use an alternate method of populating the pool phab:T376273#10203342)
Catalyst/Updates/2024-07-19

* Progress made towards achieving the metric:
** exploration on storing and rotating logs of the Catalyst control plane
** work on porting Patchdemo environment provisioning scripts to the Catalyst wikimedia Helm chart
** work on streaming logs from catalyst containers
* Are we on track to achieve the KR metric this FY?
** Yes
* What is the confidence level?
** As confident as a kid looking for Sour Skittles in a candy store (8/10)
* Any unresolved or new blockers, dependencies or risks.
** we are hoping to use Observability's experimental ELK cluster for logs, so that we don't have to host our own ELK stack. We're waiting to hear back on their decisions if they'll support us
** we'll need to use SSO for the new Catalyst backed patchdemo, but our core engineers don't have experience
*** we're hoping to work with our principal engineer for guidance
* Any new hypotheses that are emerging for that KR
** no
* Progress update and new learnings for each hypothesis in the KR, as well as the lifecycle stage of each hypothesis.
** technical learnings
*** log streaming over the kubernetes golang api
*** log rotation in Kubernetes
*** custom data persistence in k3s, so we can put it on external volumes
** product learnings
*** Observability maintains an experiment ELK cluster that we may be able to use for logs instead of rolling our own
*** Abstract Wikipedia is actively looking to us to enable deploying and testing their patches
** lifecycle
*** we are still "pre-production" for the Catalyst API/patchdemo
*** we have a go-live in sights with an issue tracking all of the MVP tasks to be completed before go-live
Catalyst/Updates/2024-07-26

* Status: On Track
* Progress update on the hypothesis for the week
** Crafted and disseminated draft of go-live plan 
** Collected requirements for logging to Observability’s Beta-Log elk cluster
** Merge request submitted for kubernetes pod log streaming through catalyst api
*** This daunting task was completed by Jeena Huneidi
*** This enables patch deployers to see their environment being provisioned and help diagnose failed builds
** Merge request for adapting the legacy patchdemo provisioning toolchain to catalyst’s kubernetes helm chart
*** I cannot understate how big of a win this will be for the team thanks to Jaime Nuche
*** This unlocks nearly all remaining extensions and skins
*** Enables work for the last mile of tasks remaining for feature completeness with legacy patchdemo
* Any new metrics related to the hypothesis
** no
* Any emerging blockers or risks
** Potential Risk – work on go-live for patchdemo is blocking other work slated for this quarter (see next section)
* Any unresolved dependencies - do you depend on another team that hasn’t already given you what you need? Are you on the hook to give another team something you aren’t able to give right now?
** This quarter our goal is to bring on other service-oriented workloads
*** The first team which is actively waiting for us to deliver this is Abstract Wikipedia
*** As a team, we have opted to delay this work until after the patchdemo go-live
*** There is some risk that we will not be able to deliver this in this quarter
*** Our team is meeting with the Abstract Wikipedia team to create a strategy to mitigate this risk
* Have there been any new lessons from the hypothesis?
** We have learned that there are certain requirements for us integrating with the Beta-Logs elk stack
*** As a team we’ll decide if it is a priority to tackle that this quarter
* Have there been any changes to the hypothesis scope or timeline?
** no
Catalyst/Updates/2024-08-02

== Introduction ==
Project Catalyst is a team working on a kubernetes cluster as a service explicitly for CI and and exploratory testing. Our current effort is upgrading Patch Demo, to use kubernetes to provision and host wikis with custom extensions and skins. The Catalyst API wraps a small kubernetes cluster. A new instance of Patch Demo will consume this API. Our next effort will be to open the Catalyst API for more general workloads include wikis which rely on (micro)services.
== Update ==
* Status: On Track
* Progress update on the hypothesis for the week
** phab:T366971 - log streaming from test environment containers merged in, thanks Jeena Huneidi
** phab:T369393 - provisioning test environments using adapted legacy patch demo scripts has been merged in, thanks Jaime Nuche
** phab:T370078 environment data and logs are stored on and external volume, and logs written to files in the patch demo container are extracted to the kubernetes logging system
** team reached consensus on our go-live plan
*** the team has an internal goal of soft-launching our kubernetes backed patch demo 2024-08-12
*** this is an internal team goal, our public posture is still "within the next couple of months" because the soft launch will incrementally release over a month (see the go-live document)
*** the team decided to punt (deprioritize) logging to Beta-Logs ELK stack, so that we can focus on go live phab:T371056
* Any new metrics related to the hypothesis
** no
* Any emerging blockers or risks
** no new blockers or risks
* Any unresolved dependencies - do you depend on another team that hasn’t already given you what you need? Are you on the hook to give another team something you aren’t able to give right now?
** no new unresolved dependencies
*** meta:Abstract Wikipedia is still waiting for us to open up the Catalyst API for general consumption
**** this work is considered blocked behind our go-live
* Have there been any new lessons from the hypothesis?
** no new lessons
* Have there been any changes to the hypothesis scope or timeline?
** no
Catalyst/Updates/2024-08-08

== Introduction ==
Project Catalyst is a team working on a kubernetes cluster as a service explicitly for CI and and exploratory testing. Our current effort is upgrading Patch Demo, to use kubernetes to provision and host wikis with custom extensions and skins. The Catalyst API wraps a small kubernetes cluster. A new instance of Patch Demo will consume this API. Our next effort will be to open the Catalyst API for more general workloads include wikis which rely on (micro)services.
== Update ==
* Status: On Track
* Progress update on the hypothesis for the week
** team has been working on stories blocking the go-live phab:T370076
** our original go-live target of Monday 2024-08-08 has been pushed a short indeterminate time
*** concern of UX for users with existing patch environments
*** namely because existing links would break if we changed the URL of the legacy patch demo
* Any new metrics related to the hypothesis
** no
* Any emerging blockers or risks
** no new blockers or risks
* Any unresolved dependencies - do you depend on another team that hasn’t already given you what you need? Are you on the hook to give another team something you aren’t able to give right now?
** no new unresolved dependencies
*** meta:Abstract Wikipedia is still waiting for us to open up the Catalyst API for general consumption
**** this work is considered blocked behind our go-live
* Have there been any new lessons from the hypothesis?
** no new lessons
* Have there been any changes to the hypothesis scope or timeline?
** no