nyuuzyou/wikis huggingface dataset:
Istio

From the upstream website: Istio extends Kubernetes to establish a programmable, application-aware network using the powerful Envoy service proxy. Working with both Kubernetes and traditional workloads, Istio brings standard, universal traffic management, telemetry, and security to complex deployments.
== Istio binary sources ==
In order to deploy Istio to our Kubernetes clusters, we use three main sources for the binaries needed:
* istioctl Debian package (repo https://gerrit.wikimedia.org/r/admin/repos/operations/debs/istioctl)
* istio-cni Debian package (repo https://gerrit.wikimedia.org/r/admin/repos/operations/debs/istio)
* The istio-build Docker image (repo https://gerrit.wikimedia.org/r/admin/repos/operations/docker-images/production-images, docker-registry https://docker-registry.wikimedia.org/istio/build/tags/)
Let's review each source to understand why we need it and how it is used (at a very high level).
=== Istioctl ===
The istioctl binary is a Go-based tool shipped by upstream that replaces the need for a dedicated Istio Helm chart. We use it to deploy CRDs and k8s settings, bypassing helmfile, but keeping the configuration yamls in the same repo (see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/custom_deploy.d/istio/). We chose to follow what upstream recommends, since upgrades and general maintenance is easier with istioctl.
The tool is able to perform its job because, behind the scenes, it embeds helm3 and several charts that are maintained by upstream. Every version of istio ships with its own istioctl, so we needed a way to have multiple versions of istioctl on the deployment node (to support the fact that not all kubernetes nodes will run with the same Istio version at any given time). We created a Debian package that collects multiple releases of Istio from Github, and that packages istioctl binaries named istioctl-#version (for example, istioctl-195).
=== Istio build Docker image ===
This is where the Istio binaries are built (proxyv2 and pilot) to then be included in the Docker images that Istio needs to operate. We mimic what upstream does with few extra settings, since our policy is to avoid using untrusted repositories like Dockerhub directly. More information about the build process in the build Dockerfile, but it is basically calling some targets of a Makefile.
=== Istio-cni ===
In order to implement a service mesh, upstream provides a way to inject Envoy sidecars to kubernetes pod to intercept (transparently) traffic and apply routing / traffic policies over it (most useful ones are proxying and circuit breaking). The trick that Istio uses it to leverage special binaries implementing the CNI interface to inject iptables policies when the pod is being created (more info https://istio.io/latest/docs/setup/additional-setup/cni/).
The CNI binaries need to be shipped to the Kubernetes nodes, and we use a dedicated Debian package for it (we chose not to follow the install-cni Daemonset approach listed in the upstream page due to the security capabilities that need to be granted to the pods).
JVM Language Stewardship

= Abstract =
The Wikimedia Foundation uses Java and Scala for a number of projects. The Data Platform Engineering group is where most of the JVM based projects are.
Having good standards help switch between projects, reduce the cost of mental switching and reconfiguration of IDE. While we have some level of standardisation (see “links” section below), some improvements are needed, in particular around Scala, and around the standardisation of CI on Gitlab.
Nominating a few engineers as stewards of the language ecosystem empowers them to:
* Define the standards
* Work with various teams and projects to implement that standard or migrate existing projects to that standard
* implement the necessary infrastructure
Current Stewards: Joseph Allemandou, Peter Fischer.
= Organisational sponsor =
Olja Dimitrijevic
= Links =
* mw:Manual:Coding conventions/Java
* Maven parent pom
* Phabricator board
* Tool Forge: Using Java
* JVM Garbage Collection
* Analytics JMX
= Goals =
* Minimal mental burden when switching between projects
* Minimal effort to start new projects
* Simple onboarding of new engineers
* Maintaining a growing number of JVM based projects over time is sustainable
* Keeping the standard up to date and applying them over time is sustainable
== Scope ==
* Java and Scala as used at the Wikimedia Foundation
== Out of scope ==
* Kotlin isn’t included to keep the scope to mainly projects in the Data Platform Engineering group. Kotlin is sufficiently different in terms of build and deployment that it probably make sense to have a different stewardship group
= Who are the stewards =
Current Stewards: Joseph Allemandou, Peter Fischer.
Conversations so far show that there isn’t much divergence of opinions, but a lack of ownership: no one feels responsible to make decisions. We don’t need a lengthy information gathering process, involving many people, we need someone empowered to make and document decisions. We don’t want a committee, but we want a focal point for this effort. Two engineers seems like a good compromise: a small group is more efficient in this context, but more than one enables conversations.
Data Engineering and Search Platform are the two teams with the most JVM based projects. Those two teams should be represented.
= Responsibilities of the stewards =
JVM Stewards are expected to:
* Consult the engineers impacted by the decisions
* Define and document the required standard
* Provide shared assets (tooling and IDE configuration, CI templates, etc…)
* Decide if standardisation on Gitlab is necessary from a language stewardship perspective.
* Document the necessary steps to align JVM projects with the standard
* Support other engineers as needed to migrate to the standard
Juniper TLS certificate install

In the fundraising environment, we use certificates from the pre-existing Puppet
certificate authority to encrypt syslog traffic. We used the Puppet CA to
generate a certificate for the SRX routers, so they can log securely to the
fundraising central loggers.
== Preparation ==
Generate a client certificate
frpm1001:~$ sudo puppet cert generate pfw-codfw.wikimedia.org
Copy the relevant certificates to the router (this assumes read permissions to /var/lib/puppet/ssl/* and pre-existing destination directories on the pfw)
frpm1001:~$ scp /var/lib/puppet/ssl/certs/ca.pem pfw3-codfw.wikimedia.org:certs/
frpm1001:~$ scp /var/lib/puppet/ssl/certs/pfw-codfw.wikimedia.org.pem pfw3-codfw.wikimedia.org:certs/
frpm1001:~$ scp /var/lib/puppet/ssl/private_keys/pfw-codfw.wikimedia.org.pem pfw3-codfw.wikimedia.org:private_keys/
== Certificate reinstall/install ==
pfw3-codfw> clear security pki ca-certificate ca-profile frack-ca-profile
[no output]
pfw3-codfw> request security pki ca-certificate load ca-profile frack-ca-profile filename /var/tmp/ssl/certs/ca.pem
node0:
--------------------------------------------------------------------------
Fingerprint:
 49:98:40:62:4f:a2:f7:41:6f:4c:b2:5b:0e:81:6a:f5:0b:9a:49:ad (sha1)
 82:76:6e:43:ee:36:48:1c:c3:d2:ae:a3:fe:bd:2f:b2 (md5)
CA certificate for profile frack-ca-profile loaded successfully
pfw3-codfw> clear security pki local-certificate certificate-id pfw-codfw
[no output]
pfw3-codfw> clear security pki key-pair certificate-id pfw-codfw
node0:
--------------------------------------------------------------------------
Key pair deleted successfully
pfw3-codfw> request security pki local-certificate load certificate-id pfw-codfw filename /var/tmp/ssl/certs/pfw-codfw.wikimedia.org.pem key /var/tmp/ssl/private_keys/pfw-codfw.wikimedia.org.pem
node0:
--------------------------------------------------------------------------
Local certificate loaded successfully
pfw3-codfw> clear services ssl initiation counters
[no output]
== Verification ==
When certificate is loaded you show see it here:
cmooney@pfw1-eqiad> show security pki local-certificate 
node0:
--------------------------------------------------------------------------
LSYS: root-logical-system
Certificate identifier: pfw-eqiad
 Issued to: pfw-eqiad.wikimedia.org, Issued by: CN = Puppet CA: frpm1002.frack.eqiad.wmnet
 Validity:
 Not before: 11-12-2023 16:09 UTC
 Not after: 11-11-2028 16:09 UTC
 Public key algorithm: rsaEncryption(4096 bits)
 Keypair Location: Keypair generated locally
{primary:node0}
After this is loaded you can apply the configuration under 'set services ssl' which reference the CA and certificate. Once this is done you should see output as per below:
pfw3-codfw> show services ssl certificate detail
[long output]
== Cleanup ==
pfw3-codfw> file delete-directory /var/tmp/ssl recurse
[no output]
== See also ==
 - Fundraising pfw rsyslog TLS errors
 - Refresh client certificate for central logging on pfw's
Category:Juniper
Kafka HTTP purging

The current (2020) mechanism for purging objects from the CDN is based on a daemon running on all cache nodes called Purged. Purged can be configured to read purge messages using either the legacy Multicast HTCP purging mechanism, or via Kafka. Regardless of the source from which purge messages are read, Purged converts them into HTTP PURGE requests sent locally to both the ATS cache backend and to the Varnish cache frontend.
== Typical purge flow ==
* MediaWiki detects that a purge is needed. It produces a Kafka message on a given topic for each individual URI that needs to be purged
* Purged, the daemon running on every relevant cache machine, consumes the appropriate Kafka topics receives a copy of the purge message. Purged forwards the request to the Varnish and ATS instances on localhost over a persistent HTTP/1.1 connection, using the PURGE request method.
* PURGE requests are handled by ATS and Varnish and cause the cache object in question to be invalidated.
== MediaWiki ==
All CDN purges are generated in MediaWiki via CdnCacheUpdate::purge method. Currently MediaWiki is configured to send the generated purges to the EventRelayer under the cdn-url-purges key. EventBus extension provides an implementation of the EventRelayer, CdnPurgeEventRelayer that creates purge events and sends them to Kafka using normal EventBus flow - via eventgate service.
Relevant configuration:
// Configuration for the EventRelayer to send purges to resource-purge kafka topic
'wgEventRelayerConfig' => [
	'cdn-url-purges' => [
		'class' => \MediaWiki\Extension\EventBus\Adapters\EventRelayer\CdnPurgeEventRelayer::class,
		'stream' => 'resource-purge',
	],
	'default' => [
		'class' => EventRelayerNull::class,
	],
],
// EventBus stream configuration 
'wgEventServiceDefault' => 'eventgate-main' 
== One-off purge ==
On a deployment server, run: 
$ echo 'https://example.org/foo?x=y' | mwscript-k8s --attach -- purgeList.php
If the URL is under /static/, it must always be purged via hostname en.wikipedia.org. This is the shared virtual hostname under which Varnish caches content for /static/, regardless of requesting wiki hostname.
Note also that mobile hostnames are cached independently of desktop hostnames. For example, to purge all copies of enwiki's article about Foo, one must purge both https://en.wikipedia.org/wiki/Foo and https://en.m.wikipedia.org/wiki/Foo
Category:Caching
Category:SRE Traffic
Karapace

Karapace is a schema registry with the permissive Apache 2 license that we are using for the Datahub data catalog since the default confluent schema registry uses the more restrictive Confluent Community License.
See https://karapace.io/.
=== Administration ===
There is a karapace service which should provide the http endpoint at localhost:8081.
The commands to inspect the schemas are in the Karapace README.
The state is stored on a kafka topic called _schemas.
=== Instances ===
We have two instances of karapace.
* karapace1001 - This serves the production deployments of datahub
* karapace1002 - This serves the staging deployment of datahub
The instances are currently configured identically, except for the kafka cluster used to store the schemas. karapace1002 uses the kafka-test cluster, whereas karapace1001 uses the kafka-jumbo cluster.
Klaxon/Administration

== Development and deployment ==
Klaxon's source code is hosted on Gerrit, in the repository. Changes are automatically tested and merged by CI, triggered by a C+2 review. For now, merged changes are also automatically deployed to production via git pull on the next Puppet run (i.e., within 30 minutes).
Klaxon runs on the "alerting_host" role (i.e. alert1001/alert2001, where Icinga & Alertmanager run). It runs inside the gunicorn engine, fronted publicly by Apache. gunicorn is automatically restarted on any changes to code or configuration.
Puppet configuration is located at and 
== VictorOps integration ==
Klaxon uses two VictorOps APIs:
* For the front page, it fetches the list of recent incidents from the VictorOps v1 API.
* To send an alert, it calls the REST Endpoint, and also sends a message to via logmsgbot.
API secrets are stored in the private Puppet repository, in hieradata/role/common/alerting_host.yaml.
== Notes ==
:1.https://gerrit.wikimedia.org/g/operations/puppet/+/1b49ba0141b35c3bf253eb93479db572bd8d8c98/modules/klaxon/manifests/init.pp#13
:2.https://bash.toolforge.org/quip/AU7VTzhg6snAnmqnK_pc
Knsq15

== Defect ==
Seems to give a
 PCI error
 
 System halted!
during POST.
$ racadm getsel
Record: 1
Date/Time: 11/02/2006 06:37:05
Source: system
Severity: 2
Description: System Board SEL: event log sensor for System Board, log cleared was asserted
-------------------------------------------------------------------------------
Record: 2
Date/Time: 11/13/2006 19:16:13
Source: system
Severity: 4
Description: POST Err: post sensor, POST fatal error #254
-------------------------------------------------------------------------------
Record: 3
Date/Time: 11/13/2006 19:17:53
Source: system
Severity: 4
Description: POST Err: post sensor, POST fatal error #254
-------------------------------------------------------------------------------
Record: 4
Date/Time: 11/13/2006 20:00:35
Source: system
Severity: 4
Description: POST Err: post sensor, POST fatal error #254
-------------------------------------------------------------------------------
Record: 5
Date/Time: 11/13/2006 20:02:55
Source: system
Severity: 4
Description: POST Err: post sensor, POST fatal error #254
-------------------------------------------------------------------------------
Knsq6

== Hardware problems ==
Record: 1
Date/Time: 11/02/2006 06:52:49
Source: system
Severity: 2
Description: System Board SEL: event log sensor for System Board, log cleared was asserted
-------------------------------------------------------------------------------
Record: 2
Date/Time: 12/07/2006 19:47:31
Source: system
Severity: 3
Description: Mem ECC Warning: memory sensor, transition to non-critical from OK ( DIMM4 )
-------------------------------------------------------------------------------
Record: 3
Date/Time: 12/07/2006 19:47:53
Source: system
Severity: 4
Description: Mem ECC Warning: memory sensor, transition to critical from less severe ( DIMM4 )
-------------------------------------------------------------------------------
Category:Servers
Kohl.wikimedia.org

kohl.wikimedia.org was a web server hosting miscellaneous tools in a production environment. It existed from 2004 to 2006, and played a role in inspiring the creation of WMDE Toolserver, noc.wikimedia.org, and techblog.wikimedia.org.
== Kate's Tools ==
In 2004, the WMDE Toolserver did not yet exist. The technical community mainly hosted their tools on independently hosted servers, using only public APIs and data dumps. 
Kohl comprised only a single server, but had the unique privilege of direct and unrestricted internal network access to production MediaWiki databases. This enabled a kind of tool that, at the time, could not have worked through web APIs alone.
The following tools were included, as developed by Kate (also known by username river):
* Kate's Six Degrees of Wikipedia.
* User edit counter.
For future greppers: These tools stored some misc data in a database called katesdb, using tables such as sixdeg_recent, in a db section now known as s3.
== MediaWiki weblog ==
In November 2004, the then-called "MediaWiki weblog" was launched on the same server, and mainly posted to by Hashar (then under alias "Ashar Voultoiz"). The blog was powered using the Pivot, a PHP-based CMS software. See archive of MediaWiki weblog at kohl.wikimedia.org/blog/.
The weblog was later replaced by a Livejournal at https://wikitech.livejournal.com/, and likely inspired Techblog.wikimedia.org a few years after that.
== Network operation center ==
The landing page of Kohl initially linked primarily to Kate's Tools and the MediaWiki weblog. It was grew to also serve as entry point for various production monitoring tools, including:
* Ganglia graphs for production, then-known as the "Wikimedia Grid". This would in later years be replaced by Icinga, Graphite, Prometheus, and Grafana.
* Network traffic rate stats.
* Squid CDN service health metrics.
After the weblog and tools were relocated elsewhere, the remaining landing page was renamed to noc.wikimedia.org.
=== Nagios ===
First installed in Jan 2005 by Hashar.[Wikitech-l] nagios monitoring alpha, 9 Jan 2005.
== See also ==
*Developer activity (2005-2008).
*database (9 Dec 2005) a toolserver-l announcement by Kate about database replicas.
*Kate's-Tools, Bugzilla category (now Phabricator). 
== External links ==
* Initial version of kohl.wikimedia.org
* Last version of kohl.wikimedia.org as alias for noc.wikimedia.org (2005-12-24)
Krypton

krypton was a virtual machine in Ganeti. (phab:T105507)
Krypton ran miscellaneous PHP apps/services. (Previously ran on zirconium.)
== Services ==
* include role::wikimania_scholarships scholarships (service docs)
* include role::iegreview iegreview (service docs)
* include role::grafana grafana
== VM Info ==
[ganeti1003:~] $ sudo gnt-instance info bromine.eqiad.wmnet
..
- Instance name: krypton.eqiad.wmnet
 Creation time: 2015-07-15 15:08:58
 Modification time: 2015-08-06 09:16:28
 State: configured to be up, actual state is up
 Nodes: 
 - primary: ganeti1004.eqiad.wmnet
.. 
 Disks: 
 - disk/0: drbd, size 20.0G
 access mode: rw
 nodeA: ganeti1004.eqiad.wmnet, minor=1
 nodeB: ganeti1002.eqiad.wmnet, minor=4
...
Kubernetes/Administration/containerd migration

Tracking task for the prep work that lead to this: https://phabricator.wikimedia.org/T362408
You can migrate your cluster to containerd right away, by: 
* Update the partman recipe in preseed.yaml to use partman/custom/kubernetes-node-containerd.cfg instead of partman/custom/kubernetes-node-overlay.cfg
** Bigger /var/lib/kubelet partition (more space for emptyDir volumes and container logs)
** /var/lib/containerd instead of /var/lib/docker
* Update the nrpe_check_disk_options to also exclude the containerd partition:
profile::monitoring::nrpe_check_disk_options: -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(docker|kubelet|containerd)/*' --exclude-type=tracefs
* If your cluster has access to restricted container images, add the following to private hiera wherever you set profile::kubernetes::node::docker_kubernetes_user_password:
profile::containerd::registry_password: "%{lookup('kubernetes_docker_password')}"
* Update your puppet roles for control planes and workers, replacing include profile::docker::engine with include profile::kubernetes::container_runtime
** The include has to come after the include of profile::dragonfly::dfdaemon
** profile::kubernetes::container_runtime will set up containerd on bookworm hosts and docker on bullseye and below
** You can override that by hard coding the runtime on a role or host level: profile::kubernetes::container_runtime: docker
So ideally you have your nodes on bullseye still and everything above is more or less a noop until you kick of the reimage to bookworm. After that the node just comes back up with bookworm and containerd (no further action required).
There is a cookbook for rolling-reimage of stacked control planes: Add a cookbook to roll-reimage stacked k8s control planes (1081377) · Gerrit Code Review (probably not of interest to anyone outside ServiceOps as you don’t run stacked control planes).
When you're done with a cluster, please make sure to remove hiera keys that are no longer required (basically everything profile::docker::engine::* as well as profile::kubernetes::node::docker_kubernetes_user_password in private hiera.
Category:Kubernetes
Category:SRE Service Operations
Kubernetes/Benefits

Designing and writing an application to run inside Kubernetes can easily be more difficult than putting together something that just runs in a VM or on a baremetal server. However there are plenty of benefits to using Kubernetes at Wikimedia that make the tradeoff worth it, including: 
* centralized logging: logs from all pods end up in Logstash automatically 
* monitoring: icinga? alertmanager? 
* traffic encryption (HTTPS) via envoy 
* request metrics via envoy 
* deployment tooling: via helmfile and helm on the deployment server 
* flexible resource limits: can increase CPUs/memory/replicas without needing to resize a VM or order more hardware
Kubernetes/CRE

== Intro ==
Kubernetes doesn't manage the low level details of running a container itself. It delegates that function to what is called a Container Runtime Engine (CRE). As a TL;DR, the project started supporting only Docker but has helped morph a standard, the Container Runtime Interface, which allowed interested parties to create container runtime engines to be used with Kubernetes.
== Docker deprecation ==
In late 2020, the kubernetes project announced that they would be deprecating and replacing Docker as a Container Runtime Engine by version 1.23. This was tracked in T269684
=== Moving away from Docker ===
In 2023 the Service Operations SRE team run a container runtime engine selection process to move away from Docker. The result of the evaluation was to target containerd, the container runtime engine powering Docker (which is under the hood utilizing it as of 2024). The team crafted a set of criteria for the evaluation and using a simple grading approach.
Kubernetes/Education

🎓 A collection of resources for learning Kubernetes
* Introduction @ WMF video by Alex
* Alex/Giuseppe at Kubecon 2018 video on Kubernetes at the foundation
* Kubernetes Workshop, 10 steps from Docker and Minikube to blubber and WMF production Kubernetes
* Docker Tutorial for Beginners - FULL COURSE in 3 Hours
* Kubernetes Tutorial for Beginners - FULL COURSE in 4 Hours
* https://labs.play-with-k8s.com/
* https://www.katacoda.com/courses/kubernetes/playground
* https://wikitech.wikimedia.org/wiki/Deployments_on_kubernetes