nyuuzyou/wikis huggingface dataset:
Kubernetes/Kubectl

kubectl is the Kubernetes command-line tool to deploy and manage applications on a Kubernetes cluster.
For tasks like doing deployments kubectl should not be used. Instead please refer to helm and helmfile.
On Debian 9 (also known as Debian Stretch) it is included in the package kubernetes-client.
kubectl can be used to view and troubleshoot cluster components and to do certain maintenance tasks. See also the kubectl cheatsheet/.
Run kubectl from the deploy servers
== Setting KUBECONFIG ==
kubectl needs to know how to connect to the cluster and what credentials to use. This information can be stored in the kubeconfig file. By default kubectl uses a variable called KUBECONFIG to find this kubeconfig file. If the variable is not set, it looks in ~/.kube/config.https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable
In order to have kubectl working with your service the KUBECONFIG variable has to be set. This is of the form:
 KUBECONFIG="/etc/kubernetes/${SERVICE}-${CLUSTER}.config"
It is easier to use kube_env for this purpose, see below.
== Running kubectl Commands ==
For example to get a list of pods for a service:
 KUBECONFIG="/etc/kubernetes/termbox-staging.config" kubectl get pods
== Using kube_env ==
The tool kube_env (present on deploy hosts) simplifies the configuration of kubectl and the access to different clusters and services. Use the following command to setup the KUBECONFIG variable (and many other useful variables like namespace and helm variables):
kube_env $SERVICE $CLUSTER
Then you can use kubectl to access the service:
kubectl get pods
=== kube_env related helpers ===
kube_env is aliased to kube-env for consistency with usual CLI tools
kube-environments and kube-services will list for you the available environments and services, which you can also access through the $KUBERNETES_ENV_ENVIRONMENTS and $KUBERNETES_ENV_SERVICES env variables.
=== Privileged user (for admin access) ===
The "usual" users have restricted privileges. If more access is needed, root (on deploy hosts) has access to the admin user:
user@deploy1002:~$ sudo -i
root@deploy1002:~# kube-env admin <cluster>
== Describe a pod ==
If there is an issue with a pod not starting and logs don't say anything, use "describe" to get more info. For example:
 kubectl describe pod P -n sessionstore
== Also see ==
* upstream docs - Install and Set Up kubectl
* Kubernetes#Managing_pods,_jobs_and_cronjobs
* kubectl cheatsheet/
Kubernetes/Kubernetes Workshop/Autoscaling in Kubernetes

== Overview ==
At the end of this module, you should be able to:
* Define Autoscaling.
* Set up Horizontal Pod Autoscaling (HPA)
== Hands-on Demo: Horizontal Pod Autoscaling (HPA) ==
Kubernetes has basic autoscaling built-in with the Horizontal Pod Autoscaling capability. Using the official docs you will run a demo where you scale a sample application using HPA.
Checking YAML
There are several ways to validate the syntax of the K8s YAML files. The kubeval program contains the basic K8s syntax, and kube-score offers a more opinionated take. Other options, such as Polaris, have a base set of rules but can be easily extended with custom rules (you want to limit repository usage to only approved repositories).
== minikube Dashboard and Lens ==
minikube dashboard is a web-based Kubernetes user interface. While Lens provides you with an opportunity to gain insight into clusters.
Shutdown all services and deployments and stop minikube at the end of this workshop module.
<syntaxhighlight lang="bash">
$ minikube stop
<syntaxhighlight>
== Next Module ==
Module 5: Refactoring code using Kubernetes features
== Previous Module ==
 Module 3: Setting up Infrastructure as Code (IaC) in Kubernetes 
Category:Kubernetes Category: How-To
Kubernetes/Packages

== Intro ==
For a variety of reasons, we deploy our kubernetes components in WMF production (Tools/Toolforge is a completely different environment) using Debian packages. Those are:
* kubernetes-client
* kubernetes-master
* kubernetes-node
kubernetes-client one has kubectl in it, kubernetes-master has kube-apiserver, kube-scheduler, kube-controller-manager and kuberetes-node has kubelet and kube-proxy components.
== Packaging ==
We don't actually build kubernetes but package it's components from upstream binary releases as described above.
Part of the process is to download the release tarball and verify its sha512 hash against the one found in the current git master CHANGELOG.
Because of that, you will need to set HTTP proxy variables for internet access on the build host.
The general process to follow is:
* Check out on your workstation
* Switch to the major version branch if it already exists or create a new one (vX.Y)
* Create a patch to bump the debian changelog
export NEW_VERSION=1.19.3 # Kubernetes version you want to packae
dch -v ${NEW_VERSION}-1 -D bookworm-wikimedia "Update to v${NEW_VERSION}"
git commit debian/changelog
# Make sure to submit the patch to the correct branch
git review vX.Y
* Merge
* Check out on the build host
* Build the packages:
git checkout vX.Y
# Ensure you allow networking in pbuilder
# This option needs to be in the file, an environment variable will *not* work!
echo "USENETWORK=yes" >> ~/.pbuilderrc
# Build the package
https_proxy=http://webproxy.$(hostname -d):8080 DIST=bullseye pdebuild
== Publishing ==
# On apt1001, copy the packages from the build host
rsync -vaz build2001.codfw.wmnet::pbuilder-result/bullseye-amd64/kubernetes*<PACKAGE VERSION>* .
# Import packages to the corresponding kubernetes apt component
sudo -i reprepro -C component/kubernetesXY --ignore=wrongdistribution include bullseye-wikimedia /path/to/<PACKAGE>.changes
# Kubernetes packages can be copied between distros:
sudo -i reprepro -C component/kubernetesXY copysrc buster-wikimedia bullseye-wikimedia kubernetes
sudo -i reprepro -C component/kubernetesXY copysrc bookworm-wikimedia bullseye-wikimedia kubernetes
Kubernetes/Packages/kubernetes-client

kubernetes-client - Kubernetes client tools is the Debian package containing the Kubernetes client kubectl.
The Wikimedia APT repo provides a version maintained by User:Akosiaris. The same is true for kubernetes-master.
apt-cache show kubernetes-client
Package: kubernetes-client
Source: kubernetes
..
Provides: kubernetes-utils
Homepage: http://kubernetes.io/
..
Description: Kubernetes client tools
 Kubernetes command line utilities:
 * kubectl -- Kubernetes cluster management utility
Category:Kubernetes
Kubernetes/Packages/kubernetes-master

kubernetes-master is the Debian package containing the Kubernetes services for a master host.
The Wikimedia APT repo provides a version maintained by User:Akosiaris. The same is true for kubernetes-client.
apt-cache show kubernetes-master
Package: kubernetes-master
Source: kubernetes
..
Description: Kubernetes services for master host
 Container Cluster Manager from Google. Kubernetes is an open source system
 for managing containerized applications across multiple hosts, providing
 basic mechanisms for deployment, maintenance, and scaling of applications.
 .
 Linux kernel version 3.8 or above is required for proper operation of the
 daemon process, and that any lower versions may have subtle and/or glaring
 issues.
 .
 This package provides "kube-apiserver", "kube-controller-manager" and
 "kube-scheduler" daemons.
Category:Kubernetes
Kubernetes/Service ips

This page is intended to list the ClusterIP addresses we allocate statically on the wikikube cluster. We just list the last octet. Given we assign a /24 subnet as a ClusterIP range, the first 16 addresses should be reserved for static IPs allocation.
However, some other IPs might be assigned inside that range. You should always verify your address isn't already allocated by checking (as admin):
(admin eqiad) # kubectl get service --all-namespaces | awk '{if ($4 ~ /^[0-9]/) {print $4}}' | sort -t. -n +3.0
10.64.72.1
10.64.72.3
...
Once you've checked that your desired IP is not already allocated live on the cluster or reserved here, please add it to the table below.
Namespace 	Service 	IP
default 	kubernetes 	.1
kube-dns 	coredns 	.3
mw-debug 	statsd-exporter	.5
mw-web 	statsd-exporter	.6
mw-api-ext 	statsd-exporter	.7
mw-api-int 	statsd-exporter	.8
mw-jobrunner	statsd-exporter	.9
mw-parsoid 	statsd-exporter	.11
mw-mcrouter 	mcrouter 	.12
Kubernetes/Service ports

In general, new services should reserve TLS ports in the 4000-5000 port range. Old services will be moved over that range as soon as practically possible.
== Service ports ==
Port 	Service
3031 	termbox-test
4001 	mathoid
4002 	cxserver
4003 	citoid
4004 	termbox
4005 	linkrecommendation
4006 	linkrecommendation-external
4007 	rdf-streaming-updater
4008 	shellbox
4009 	image-suggestion
4010 	shellbox-constraints
4011 	toolhub
4012 	shellbox-timeline
4014 	shellbox-syntaxhighlight
4015 	shellbox-media
4016 	developer-portal
4017 	machinetranslation
4018 	termbox-test-tls
4030 	proton / chromium-render
4080 	shellbox-video
4101 	wikifeeds
4102 	mobileapps
4104 	push-notifications
4105 	tegola-vector-tiles
4106 	Maps/Kartotherian
4110 	similarusers
4111 	miscweb
4112 	thumbor (k8s)
4113 	rest-gateway
4200 	xhgui
4250 	iPoid
4392 	eventgate-logging-external
4442 	mw-mcrouter (ds)
4444 	mw-debug
4446 	mw-api-int
4447 	mw-api-ext
4448 	mw-jobrunner
4449 	mw-videoscaling
4450 	mw-web
4451 	mw-wikifunctions
4452 	mw-parsoid
4453 	mw-debug-next
4454 	mw-web-next
4455 	mw-api-ext-next
4456-4490	reserved for MediaWiki
4492 	eventgate-main
4592 	eventgate-analytics
4666 	blubberoid
4667 	calculator-service
4684 	chart-renderer
4692 	eventgate-analytics-external
4737 	apertium
4892 	eventstreams
4632 	recommendation-api
4969 	zotero
4971 	function-orchestrator
4972 	device-analytics
4992 	eventstreams-internal
6543 	kartotherian
7200 	changeprop-jobqueue
7272 	changeprop
8081 	sessionstore
8082 	echostore
8087 	api-gateway
8800 	thumbor (temporary)
9632 	recommendation-api (legacy)
34192 	eventgate-main (legacy)
35192 	eventgate-analytics (legacy)
== aux-k8s cluster services ==
Port 	Service
16686	jaeger
4195 	benthos-mw-accesslog-metrics
== Infrastructure services ==
Services considered part of the underlying infrastructure may derive from the above standard to stand out
Port 	Service
30012	Istio Ingressgateway (tcp-istiod; not open, just blocked)
30021	Istio Ingressgateway (status-port; not open, just blocked)
30080	Istio Ingressgateway (HTTP)
30443	Istio Ingressgateway (HTTPS)
31443	Istio Ingressgateway (TLS, SNI)
LVS/ha.cf

<- back to LVS
# debugfile /var/log/ha-debug
# logfile /var/log/ha-log
# logfacility local0
# send hearbeat every # seconds
keepalive 2
# timeout before considered dead
deadtime 30
# late heartbeat warning
#warntime 10
# In case of a reboot, a server takes sometime
# to bring network back.
initdead 120
udpport 694
#baud 19200
#serial /dev/ttyS0
# Network interface on which heartbeats broadcast are sent.
# bcast eth0
# Multicast: dev mcastgroup port ttl loop
# mcast eth0 225.0.0.1 694 1 0
# unicast: dev peer-ip
# ucast eth0 192.168.0.2
auto_failback on
# nodes in the cluster (uname -n)
#node foo
#node bar
Ldapsearch

ldapsearch is a unix command line tool for querying an LDAP directory.
== Example queries ==
Show a user by shell account name (laner):
ldapsearch -x uid=laner
Show a user by wiki name (underscore "_" becomes space " ") (User:Ryan_Lane):
ldapsearch -x "cn=Ryan Lane"
You can also add \* + to the above queries to return all user attributes
ldapsearch -x uid=laner \* +
ldapsearch -x "cn=Ryan Lane" \*
Alternatively you can return only the attributes you are interested in
ldapsearch -x uid=laner mail,uid
ldapsearch -x "cn=Ryan Lane" uid createTimestamp mail cn sn
Search for all users with a wikimedia.org email address
ldapsearch -x 'mail=*@wikimedia.org'
Search all users created after a specific date
ldapsearch -x "createTimestamp>=20190316100000Z" uid createTimestamp mail
Search for all users with a wikimedia.org email address created after a specific date
ldapsearch -x "(&(mail=*@wikimedia.org)(createTimestamp>=20190316100000Z))" uid createTimestamp mail cn sn
== Helper aliases ==
When using ldapsearch from a Cloud VPS instance, these shell aliases can help make things work better:
alias ldap='ldapsearch -xLLL -P 3 -E pr=40000/noprompt -o ldif-wrap=no -b"dc=wikimedia,dc=org"'
alias un64='awk '\BEGIN{FS=":: ";c="base64 -d"}{if(/\w+:: /) {print $2 |& c; close(c,"to"); c |& getline $2; close(c); printf("%s:: \"%s\"\n", $1, $2); next} print $0 }'\'
The ldap alias configures ldapsearch to make paginated queries by default and to page through the results printing them all to the screen. It also disables wrapping of long output lines and sets a base for the search matching the base used for Developer accounts and other records in the LDAP directory used for authn/authz by Cloud VPS projects.
The un64 alias can be used to decode the base64 encoding used by our LDAP server when storing and returning non-ascii characters. This will most commonly be needed for cn attributes which store MediaWiki usernames containing unicode characters. These base64 encoded values will be displayed by ldapsearch with a "::" marker following the field name:
$ ldap uid=matmarex cn
dn: uid=matmarex,ou=people,dc=wikimedia,dc=org
cn:: QmFydG9zeiBEemlld2/FhHNraQ==
# pagedresults: cookie=
$ ldap uid=matmarex cn | un64
dn: uid=matmarex,ou=people,dc=wikimedia,dc=org
cn:: "Bartosz Dziewoński"
# pagedresults: cookie=
Category:Operations
Category:How-To
Lifetime of an issue

Issues are handled like this:
# User mails noc@wikimedia.org with the issue. (please, please get them to do this if possible. it provides a central place to contact them)
# Someone looks at the issue, and decides if it's a real issue. If it's not (for example, caused by slow site, or user error) send a mail to the user explaining that.
# If it is a real, decide if it's a duplicate. If it is, send a mail to the user saying it's a known issue. Use the "issues-Known bug" template, and include the bugzilla URL. Add a note of the new ticket to the existing bug.
# If it's not a duplicate, do the following:
## Open a new bugzilla in the "Issues" product. Assign it to a particular person if appropriate, otherwise assign it to keturner@livejournal.com. Please always cc: keturner on the bug.
##Add a "Note" to the ticket containing the bugzilla URL
##Use the "issues-Bug opened" template to mail the user. Include the URL
##Send a mail to issues-l containing a description of the issue and the bugzilla URL.
##Fix the problem :-)
when the problem is resolved:
#Close the bug
#Send a mail to all tickets listed in the bug, using the "issues-Bug fixed" template.
#Send a mail to issues-l saying that the bug is fixed.
Link aggregation

There are several ways to aggregate multiple links into one, bigger, more redundant link.
== Server link aggregation ==
Before we did (sporadically) aggregate links using manual configuration on Linux servers, usually statically, without LACP. Now, we have a Puppet definition for it: interface_aggregate in generic-definitions.pp. This sets up an LACP aggregated link.
Use the following switch side configuration:
=== Juniper ===
[edit interfaces]
ge-4/0/22 {
 apply-groups-except access-port;
 description arsenic;
 ether-options {
 802.3ad {
 lacp {
 force-up;
 }
 ae3;
 }
 }
}
...
ge-4/0/30 {
 apply-groups-except access-port;
 description arsenic:eth1;
 ether-options {
 802.3ad ae3;
 }
}
ge-4/0/31 {
 apply-groups-except access-port;
 description arsenic:eth2;
 ether-options {
 802.3ad ae3;
 }
}
ge-4/0/32 {
 apply-groups-except access-port;
 description arsenic:eth3;
 ether-options {
 802.3ad ae3;
 }
}
...
ae3 {
 description arsenic:bond0;
 mtu 9192;
 aggregated-ether-options {
 lacp {
 periodic fast;
 }
 }
 unit 0 {
 family ethernet-switching {
 port-mode access;
 vlan {
 members public1-a-eqiad;
 }
 }
 }
}
* Remove the individual (ge-x/x/x or whatever) interfaces from any interface-range lists that put them into vlans, since the aggregated port is added directly. Look at 
*:interfaces interface-range 
* Check and increment if you added a new aggregated device:
*:chassis aggregated-devices ethernet device-count
The apply-group access-port needs to deactivated with apply-groups-except access-port on the individual interfaces used for the aggregated link, as individual MTU and 802.1q configuration is now allowed.
Note that we define the interface connected to eth0 as force-up (always up). When Linux isn't running, for example during PXE boot or during an installation, LACP is not active, and then the switch accepts traffic from eth0 (only) as if there is no aggregated link.
=== Foundry ===
The following configuration sets up an LACP (dynamic) aggregate with fast PDU rate:
lag "nas1-a" dynamic
 ports e 15/5 to 15/6 
 primary-port 15/5
 lacp-timeout short
 deploy
 port-name "nas1-a:e0a" ethernet 15/5 
 port-name "nas1-a:e0b" ethernet 15/6
!
Unfortunately, Foundry doesn't seem to support a way to force one interface to up in the absence of LACP packets. Therefore the LAG will need to be undeployed during PXE boots. :-(
=== Linux/Puppet ===
In Puppet, the interface_aggregate definition can be used, like this:
interface_aggregate { "bond0":
	orig_interface => "eth0",
	members => [ "eth0", "eth1", "eth2", "eth3" ],
	lacp_rate => "fast"
}
orig_interface contains the original interface from which IP configuration should be taken. This interface is converted to the bonding interface (bond0 here). lacp_rate, default fast defines the LACP PDU transmission rate. Our Juniper and Foundry switches support fast, but some may only support "slow".
This sets up file /etc/network/interfaces. Check /sys/class/net/bond0/bonding/ for debugging.
Category:Network
Linode

== Restarting the Linode server (contains wikitech) ==
* Login it to the Linode Management site (The login info is on fenari in /h/w/doc/linode)
* Click on Wikitech (name of the server)
* Click on the reboot button next to the following line: My Ubuntu 8.04 LTS 64bit Profile (2.6.18.8-x86_64-linode1) - Last Booted 
Category:Linode
Linux Apache2 modsetproctitle

/* The code for Apache1 can be found at dammit.lt */
 /*
 ** mod_proctitle.c -- Apache sample proctitle module
 ** [Autogenerated via ``apxs -n proctitle -g]
 **
 ** To play with this sample module first compile it into a
 ** DSO file and install it into Apache's modules directory
 ** by running:
 **
 ** $ apxs -c -i mod_proctitle.c
 **
 ** Then activate it in Apache's httpd.conf file for instance
 ** for the URL /proctitle in as follows:
 **
 ** # httpd.conf
 ** LoadModule proctitle_module modules/mod_proctitle.so
 **
 ** Then after restarting Apache via
 **
 ** $ apachectl restart
 */
 #include "httpd.h"
 #include "http_config.h"
 #include "http_protocol.h"
 #include "ap_config.h"
 extern char *ap_server_argv0;
 char proctitle_buf[128];
 /* The sample content handler */
 static int proctitle_handler(request_rec *r)
 {
 if (strcmp(r->handler, "proctitle")) {
 return DECLINED;
 }
 r->content_type = "text/html";
 if (!r->header_only)
 ap_rputs("The sample page from mod_proctitle.c\n", r);
 return OK;
 }
 static int proctitle_child_init(request_rec *r)
 {
 strncpy(ap_server_argv0,"httpd: init-child",64);
 return DECLINED;
 }
 static int proctitle_clear(request_rec *r)
 {
 strncpy(ap_server_argv0,"httpd: idle",64);
 return DECLINED;
 }
 static int proctitle_ft(request_rec *r)
 {
 bzero(proctitle_buf,128);
 snprintf(proctitle_buf,127,"httpd: %16s [%s] %s",
 r->connection->remote_ip,r->hostname,
 r->the_request);
 strncpy(ap_server_argv0,proctitle_buf,128);
 return DECLINED;
 }
 static void proctitle_register_hooks(apr_pool_t *p)
 {
 /* ap_hook_handler(proctitle_handler, NULL, NULL, APR_HOOK_MIDDLE); */
 ap_hook_child_init(proctitle_child_init, NULL, NULL, APR_HOOK_FIRST);
 ap_hook_translate_name(proctitle_ft,NULL,NULL, APR_HOOK_FIRST);
 ap_hook_log_transaction(proctitle_clear,NULL,NULL, APR_HOOK_LAST);
 strncpy(ap_server_argv0,"httpd: master",60);
 }
 /* Dispatch list for API hooks */
 module AP_MODULE_DECLARE_DATA proctitle_module = {
 STANDARD20_MODULE_STUFF,
 NULL, /* create per-dir config structures */
 NULL, /* merge per-dir config structures */
 NULL, /* create per-server config structures */
 NULL, /* merge per-server config structures */
 NULL, /* table of config file commands */
 proctitle_register_hooks /* register hooks */
 };
Category:How-To
Live-1.5

The /home/wikipedia/common/live-1.5 web tree holds the "live" files which will be exposed to the web via http://xx.wikipedia.org/w/.
This allows us to restrict the entry points to MediaWiki to just the files we control via symlinks or stub files, as well as giving us a place to deploy one off web entry points.
== Configuration and updates ==
sync-common-all is required to push an update to this directory.
MediaWiki entry points are wrapped in stub scripts:
* index.php / wiki.phtml
* img_auth.php
* opensearch_desc.php
* redirect.php / redirect.phtml
* thumb.php
These stubs include MWVersion.php to tell it which base directory to pull MediaWiki from. Originally this was used during the transition from MediaWiki 1.4 to 1.5 to send migrated wikis to the new code base; currently it just sends test.wikipedia.org to the NFS master copy instead of the local copy of deployment files.
== Example ==
So for instance when pushing out /home/wikipedia/common/live-1.5/mobileRedirect.php with sync-common-all the file would be request-able from 
http://en.wikipedia.org/w/mobileRedirect.php?to=http://en.wikipedia.org/wiki/Manor
Category:Operations
Live hacks/internalParse

Index: includes/Parser.php
===================================================================
--- includes/Parser.php (revision 30164)
+++ includes/Parser.php (working copy)
@@ -328,8 +328,10 @@
 
 global $wgUseTidy, $wgAlwaysUseTidy, $wgContLang;
 $fname = 'Parser::parse-' . wfGetCaller();
- wfProfileIn( __METHOD__ );
- wfProfileIn( $fname );
+ if ( $clearState ) {
+ wfProfileIn( __METHOD__ );
+ wfProfileIn( $fname );
+ }
 
 if ( $clearState ) {
 $this->clearState();
@@ -347,7 +349,7 @@
 wfRunHooks( 'ParserBeforeStrip', array( &$this, &$text, &$this->mStripState ) );
 # No more strip!
 wfRunHooks( 'ParserAfterStrip', array( &$this, &$text, &$this->mStripState ) );
- $text = $this->internalParse( $text );
+ $text = $this->internalParse( $text, $clearState );
 $text = $this->mStripState->unstripGeneral( $text );
 
 # Clean up special characters, only run once, next-to-last before doBlockLevels
@@ -443,9 +445,12 @@
 $this->mOutput->setText( $text );
 $this->mRevisionId = $oldRevisionId;
 $this->mRevisionTimestamp = $oldRevisionTimestamp;
- wfProfileOut( $fname );
- wfProfileOut( __METHOD__ );
 
+ if ( $clearState ) {
+ wfProfileOut( $fname );
+ wfProfileOut( __METHOD__ );
+ }
+
 return $this->mOutput;
 }
 
@@ -967,7 +972,7 @@
 *
 * @private
 */
- function internalParse( $text ) {
+ function internalParse( $text, $clearState ) {
 $isMain = true;
 $fname = 'Parser::internalParse';
 wfProfileIn( $fname );
@@ -978,7 +983,13 @@
 return $text ;
 }
 
+ if ( $clearState ) {
+ wfProfileIn( __METHOD__.'-rv' );
+ }
 $text = $this->replaceVariables( $text );
+ if ( $clearState ) {
+ wfProfileOut( __METHOD__.'-rv' );
+ }
 $text = Sanitizer::removeHTMLtags( $text, array( &$this, 'attributeStripCallback' ), false, array_keys( $this->mTransparentTagHooks ) );
 wfRunHooks( 'InternalParseBeforeLinks', array( &$this, &$text, &$this->mStripState ) );