nyuuzyou/wikis huggingface dataset:
Network equipment lifecycle

== Procurement ==
== Provisioning ==
ZTP Network Device Provision
SONiC/initial-setup
Juniper TLS certificate install (Fundraising SRXs)
Network telemetry#Network devices (exporters) (Production)
* New eqiad/codfw leaf switches need to be added to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/admin_ng/values/common-bgp.yaml
== Upgrade ==
Juniper router upgrade
Juniper switch upgrade
Juniper RE i40e firmware
== Migration ==
Juniper virtual chassis member replace
Migrate from VC switch stack to EVPN
== Decommissioning ==
# Make sure the device is not handling any production traffic anymore
# Remove from https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/admin_ng/values/common-bgp.yaml if present
# Downtime in monitoring
# Remove its production facing links
# Remove from monitoring (LibreNMS, Puppet, Rancid)
# Remove from automation (Homer public & private)
# Zeroize
# Update Netbox (inc. change status)
# Update DNS (dns cookbook)
# Remove management/console links (optionally power & unrack)
Category:Network
News

Cloud Services news and major change announcements
== Current announcements ==
; Migrating Wikitech accounts to SUL
: We are asking folks with Developer accounts to connect their accounts with a SUL account to ease an upcoming Wikitech configuration change.
; Cloud VPS VXLAN and IPv6 migration
: Cloud VPS will gain IPv6 support, and will transition its underlying overlay network to VXLAN.
== Recently completed ==
; Toolforge Grid Engine Deprecation
: Toolforge Grid Engine was shut down in March 2024 after a multi-year deprecation process
; 2024 Cloud VPS Purge
: We are asking project members to claim their active Cloud VPS projects and also explicitly tell us about projects that were no longer needed.
; Buster deprecation
: Debian Buster is being removed from Cloud VPS.
== Archive ==
== See also ==
* Toolforge change log for minor changes
Category:Cloud Services
News/2017 Tools Precise deprecation

Hosts running Ubuntu Precise (12.04) as an operating system were removed from the Toolforge infrastructure on 2017-03-14. The login bastions were migrated to Trusty (14.04) in 2015. 
== Why did we do this? ==
Ubuntu Precise was released in April 2012, and support for it (including security updates) ends in April 2017. We needed to shut down all Precise hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration took many months because many people were still using the Precise hosts and our users are working on tools in their spare time.
== What changed? ==
; early September 2016 : Jobs started without -l release=trusty and webservices started with --release=precise will print a warning to standard error. This will end up in the job's err log file for most jobs.
; 26 October 2016 : Jobs started with jsub will no longer run on Precise hosts by default, and will instead run on Trusty hosts. This means jobs running via cron will migrate to trusty automatically.
; February 2017 : Tools starting jobs with jsub -l release=precise will start receiving emails urging them to migrate to Trusty.
; March 14th 2017 : Jobs started with jsub -l release=precise will no longer function and will crash with an error.
== What should I do if my jobs or webservice are not running now? ==
If you:
* Ran jobs with jsub -l release=precise:
** test whether your process works on a Trusty host by submitting the job with -l release=trusty
* Ran a webservice with --release=precise:
** test whether your process works on a Trusty host by submitting the job with --release=trusty
* Ran a webservice without specifying the release
** Run webservice start to migrate it to Trusty.
If your job fails to run on Trusty and you need help figuring out why, file a Phabricator task asking for help.
== What are the primary changes with moving from Precise to Trusty? ==
You get newer versions of most software, which is great - but also means you can run into compatibility issues. Common issues probably include:
 
# Newer version of PHP (5.5 rather than 5.3). Some old deprecated functions / interfaces were dropped in 5.4, so watch out if your code uses any of them. A comprehensive listing can be found in https://secure.php.net/manual/en/migration54.php - specifically in https://secure.php.net/manual/en/migration54.incompatible.php. There are also a few additional breaking changes in the 5.5 that can be seen at https://secure.php.net/manual/en/migration55.incompatible.php.
# Slightly newer version of Python 2 and much newer version of Python3 (3.2 -> 3.4). If you had virtualenvs set up to run in Precise, you will have to recreate them.
== Useful links ==
* https://tools.wmflabs.org/precise-tools/
* https://tools.wmflabs.org/?status
News/2019 Jessie deprecation

This page keeps track of our ongoing effort towards Debian Jessie deprecation before April 2020.
== Tell me exactly what I need to do ==
=== Cloud VPS project maintainers ===
* Check https://os-deprecation.toolforge.org/ for reports about your project(s)
* For each reported deprecated instance:
** Create a new instance using Debian Buster (longest time until future deprecation) or Debian Stretch
** Move content and configuration from the old Jessie instance to the new instance
** If you are using a web proxy on the old instance, delete the proxy and the add a new one with the same name pointing at the new instance
** Shutdown the old Jessie instance
** Update any maintainer documentation to reference the new instance rather than the old Jessie instance
** When you are sure you have all the content and configuration moved, delete the old Jessie instance
If you need additional quota to create new virtual machine instances, follow the instructions on Phabricator to request a temporary quota increase.
=== Toolforge tool maintainers ===
TBD; that said, the following webservice types for the kubernetes backend are already based on Stretch containers:
* php7.2
* tcl
* python3.5
* golang
* jdk8
* node10
== Why are we doing this? ==
On June 30, 2020 the Debian project will stop providing security patch support for the Debian 8 "Jessie" release.
== Timelines. What will change? What has changed already? ==
=== Cloud VPS projects ===
In March 2019, instance creation was disabled for new Debian Jessie virtual machines (VMs).
The timeline for the remainder of the process is:
* In September 2019 (2019-09-11), create a Phabricator task, a page on Wikitech, and announce deprecation plan to Cloud VPS community via the cloud-announce mailing list. 
* In September 2019, create a scoreboard tool to list all Cloud VPS projects and instances believed to be running Debian Jessie.
* In October 2019, create Phabricator tasks for every Cloud VPS project having Jessie instances running.
* In November 2019, ping project administrator (same Phabricator tasks) if they are still using Jessie. 
* In December 2019, deadline. Evaluate if Jessie VMs not migrated are actually in use and why they weren't migrated.
* In January 2020, shutdown all Jessie instances (unless special arrangements have been made for extension of deadline).
=== Toolforge ===
* Toolforge Bastions and Grid Engine environments are already using Debian Stretch, so no change is needed there.
* Some Kubernetes runtimes are currently using Debian Jessie as a base layer. The Toolforge administration team will be working to ensure that suitable Stretch/Buster replacements are available.
* The Toolforge Kubernetes cluster used Debian Jessie instances for its control plane and runtime environment. A migration to a newer cluster using Debian Buster instances is in the design and testing phase (phab:T214513).
== See also ==
* Operating system upgrade policy
* News/2018 Trusty deprecation
News/2020 HTTPS enforcement at shared proxy

The web proxy service for Cloud VPS projects is enforcing TLS encryption by automatically redirecting from HTTP to HTTPS.
== What is changing? ==
* Requests to *.wmcloud.org and *.wmflabs.org hosts via the domain proxy service enforce TLS encryption
* Strict-Transport-Security header added to TLS secured responses instructing user-agents to automatically upgrade http:// requests to https:// for the next 24 hours.
== Timeline ==
* 2020-08-18: TLS enforced for GET and HEAD requests and Strict-Transport-Security header with one day duration sent to clients.
* 2021-02-02: TLS enforced for all requests and Strict-Transport-Security header with one year duration sent to clients.
News/2020 Phasing out the .wmflabs domain

Cloud VPS instances created on or after 2020-09-08 will stop using the .eqiad.wmflabs domain, and be found only under .eqiad1.wikimedia.cloud.
== What's changing ==
* New instances in Cloud VPS projects will get one DNS entry named <host>.<project>.eqiad1.wikimedia.cloud.
* IP addresses assigned to instances in Cloud VPS projects will continue to have one reverse DNS entry returning <host>.<project>.eqiad1.wikimedia.cloud.
* New instances will be configured to treat <host>.<project>.eqiad1.wikimedia.cloud as their canonical name.
In order to avoid breaking existing systems, these changes will NOT be applied retroactively to existing instances. Old DNS entries will live on until the instance is deleted and should be largely harmless. 
== Timeline ==
* 2020-02-18: Start creating <host>.<project>.eqiad1.wikimedia.cloud hostnames for Cloud VPS instances
* 2020-08-19: Announced deprecation of <host>.<project>.eqiad.wmflabs and <host>.eqiad.wmflabs hostnames
* 2020-09-08: Stop creating legacy <host>.<project>.eqiad.wmflabs and <host>.eqiad.wmflabs hostnames for new Cloud VPS instances
== More information ==
From 2020-02-18 to 2020-09-08, Cloud VPS instances had a hybrid hostnaming system:
* New instances were given three different DNS entries: 
** <host>.<project>.eqiad1.wikimedia.cloud
** <host>.<project>.eqiad.wmflabs
** <host>.eqiad.wmflabs
* Reverse DNS lookups of instance IP addresses returned hostnames of <host>.<project>.eqiad1.wikimedia.cloud
* Instances themselves believed (for example via hostname -f) that they were canonically named <host>.<project>.eqiad.wmflabs
This hybrid system was originally implemented to maintain backwards compatibility as the domain name changes rolled out across the Cloud VPS infrastructure and projects. In the interest of simplifying, standardizing, and eliminating another use of the term 'labs', the Cloud VPS admin team started this project to phase out the use of the wmflabs domain name.
== Frequently asked questions ==
=== Renaming existing instances ===
If you find yourself rewriting code in order to deal with instances under both domains (due to the change in hostname -f behavior), adjusting an old instance to identify as part of <project>.eqiad1.wikimedia.cloud only requires a simple change to /etc/hosts. The Cloud VPS admin team can assist in this type of renaming.
=== What about *.svc.eqiad.wmflabs? ===
A number of services, notably the Wiki replicas, use "service names" under the domain svc.eqiad.wmflabs. These names are not being replaced with equivalent names under svc.eqiad1.wikimedia.cloud at this time. These names will change in the future, but there are some other projects underway in the Cloud Services environment which need to progress further before changing our legacy service names.
== See also ==
* Announcement of change on cloud-announce@ mailing list
* New names for everyone! blog post
News/2020 Toolforge Elasticsearch v7 migration

== What is changing? ==
* New Elasticsearch cluster version (v7.4)
* New client connection URL: http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud
* Improved service availability and scaling capabilities
== Timeline ==
* 2020-03-04: Version 7 Elasticsearch cluster general availability for migration on an opt-in basis.
* 2020-04-06: deadline Evaluate if all tools have been migrated to the new Elasticsearch cluster.
* 2020-04-20: shutdown All tools are fully migrated to the new Elasticsearch cluster, version 5 cluster is retired from service.
== What should I do? ==
=== Update client connection strings ===
A new service address (http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud) has been configured for the Elasticsearch V7 cluster. This service address is configured to provide high availability and load-balancing features across all Elasticsearch V7 cluster members.
=== Update your Tools support Elasticsearch version 7 APIs ===
==== Deprecated index document types ====
The ability to specify a custom document type has been deprecated in Elasticsearch version 7. A full list of mapping changes can be found at https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html#breaking_70_mappings_changes
Deprecated version 5 example with a custom document type:
curl -X PUT 'http://<elasticsearch>/index_name' -d '{
 "mappings": {
 "custom_type": {
 "properties": {
 "foo": { "type": "keyword" }
 },
 }
 }
 }'
Supported version 7 example:
curl -X PUT 'http://<elasticsearch>/index_name' -d '{
 "mappings": {
 "doc": {
 "properties": {
 "foo": { "type": "keyword" },
 },
 }
 }
 }'
This mapping change also effects the path to your index documents.
Example index document paths
Deprecated version 5
PUT <index>/<custom type>/<id>
version 7
PUT <index>/_doc/<id>
== See also ==
* Help:Toolforge/Elasticsearch
* News/2019 Jessie deprecation
News/2021 Stretch deprecation

This page keeps track of our ongoing effort towards Debian Stretch deprecation, which happened upstream on July 6, 2021.
== Tell me exactly what I need to do ==
=== Cloud VPS project maintainers ===
* Check :toolforge:os-deprecation for reports about your project(s)
* For each reported deprecated instance:
** Create a new instance using Debian Bullseye (longest time until future deprecation) or Debian Buster
** Move content and configuration from the old Stretch instance to the new instance
** If you are using a web proxy on the old instance, delete the proxy and the add a new one with the same name pointing at the new instance
** Shutdown the old Stretch instance
** Update any maintainer documentation to reference the new instance rather than the old Stretch instance
** When you are sure you have all the content and configuration moved, delete the old Stretch instance
If you need additional quota to create new virtual machine instances, follow the instructions on Phabricator to request a temporary quota increase.
=== Toolforge tool maintainers ===
Please see the information at News/2022 Toolforge Stretch deprecation#What should I do?.
== Why are we doing this? ==
On July 6, 2021 the Debian project stopped providing security patch support for the Debian 9 "Stretch" release. It did, however, move to LTS support and will receive updates other than backports from the LTS team until end of June 2022.
== Timelines. What will change? What has changed already? ==
=== Cloud VPS projects ===
In March 2021, instance creation was disabled by default for new Debian Stretch virtual machines (VMs).
The timeline for the remainder of the process is:
* In August 2021, create a Phabricator task, a page on Wikitech. An announcement needs to be scheduled and set up.
* January 1st, 2022: Stretch VM creation disabled in all projects, deprecation nagging begins in earnest.
* May 1, 2022: All active Stretch VMs will be shut down (but not deleted) by WMCS admins.
* June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will be deleted by WMCS admins
=== Toolforge ===
Please see main article News/2022 Toolforge Stretch deprecation.
== See also ==
* Operating system upgrade policy
* News/2019 Jessie deprecation
* Debian Stretch release information
* News/2022 Toolforge Stretch deprecation
News/2023 Cloud VPS metrics changes

This page details information about the changes to Cloud VPS metrics systems in mid 2023.
== What is changing? ==
=== Relevant changes that happened previously ===
* Debian 11 (released mid 2021) no longer includes Diamond, which produced machine-level stats for Graphite.
* From November 2022, a Prometheus instance (at prometheus.wmcloud.org) has been collecting machine-level statistics using prometheus-node-exporter on all VMs.
=== What is changing now? ===
* Monday, July 17th: The remaining Cloud VPS instances will have Diamond removed. This means Graphite-based per-instance metrics will stop working. 
* Tuesday, August 1st
** The Cloud VPS Graphite and statsd servers will be shut down. 
** The Grafana server at grafana-cloud.wikimedia.org and grafana-labs.wikimedia.org will be shut down and replaced by a new server running on grafana.wmcloud.org. 
== What should I do? ==
=== Migrate dashboards (and other tools) to use Prometheus data ===
You should migrate any affected Grafana dashboards and other tools you find useful to load data from Prometheus instead of loading it from Graphite.
If your project needs Graphite for a reason other than per-machine statistics and can't move to Prometheus, you need to run your own Graphite instance instead of relying on the WMCS-hosted instance.
=== Migrate dashboards from grafana-cloud.wikimedia.org to grafana.wmcloud.org ===
You should move any Grafana dashboards you find useful on grafana-cloud.wikimedia.org to grafana.wmcloud.org.
There is a dump of all of the dashboards that existed on grafana-cloud before its shutdown.
== Why are we doing this? ==
* We are replacing Diamond metrics with Prometheus based ones because Diamond is written in Python 2, which is no longer supported on latest versions of the Debian operating system.
* We are shutting down the Graphite server because its primary purpose was to collect Diamond data.
* We are replacing the old Grafana instance because new versions of Grafana no longer supports the setup we currently have.
== See also ==
* Help:Cloud VPS managed monitoring
* Prometheus
* Grafana
* Graphite
* Statsd
News/2023 ToolsDB upgrade

In April 2023, ToolsDB was migrated to a new Cloud VPS instance, and is now running MariaDB 10.4 with Ceph-based storage.
== Timeline ==
* 2023-03 New MariaDB server created and set to replicate data from the old primary
* 2023-04-06 New server promoted to primary, DNS records pointed to the new primary server (T333471)
== What you need to do ==
Hopefully nothing. If your tool lost connectivity to the database, please try restarting it. Note: ideally all tools should automatically reconnect when they lose database connectivity.
== What you should be aware of ==
The new version of MariaDB (10.4) introduced some breaking changes from the one we were running on the old server (10.1).
So far the ones listed below are the ones that we know have required manual changes from users:
* 10.4 defaults to STRICT_TRANS_TABLES, which means (among other things) that ToolsDB no longer accepts certain datetime formats. You must use one of the documented Date and Time Literals.
== Performance ==
The new storage is slightly less performant than the old one, because it's using network storage (Ceph) and no longer physically-attached storage. We believe the slight performance penalty is compensated by the vastly improved reliability and flexibility. We are not observing significant issues after the migration, but we expect some queries could be (marginally) slower.
== Linkwatcher ==
We decided NOT to migrate the database s51230__linkwatcher (by far the biggest database in the old ToolsDB server) and instead move it to a dedicated Trove instance.
News/2024 Buster deprecation

Support for Debian Buster ends on June 30th, 2024. This page tracks our efforts to stop using Buster on cloud-vps. Tasks tracking this work can be found on the Cloud-VPS (Debian Buster Deprecation) workboard.
== Tell me exactly what I need to do ==
=== Cloud VPS project maintainers ===
* Check :toolforge:os-deprecation for reports about your project(s)
* For each reported deprecated instance:
** Create a new instance using Debian Bookworm (longest time until future deprecation) or Debian Bullseye
** Move content and configuration from the old Buster instance to the new instance. Consider using a Cinder volume to store and move your files.
** If you are using a web proxy on the old instance, delete the proxy and the add a new one with the same name pointing at the new instance
** Shutdown the old Buster instance
** Update any maintainer documentation to reference the new instance rather than the old Buster instance
** When you are sure you have all the content and configuration moved, delete the old Buster instance
If you need additional quota to create new virtual machine instances, follow the instructions on Phabricator to request a temporary quota increase.
=== Toolforge tool maintainers ===
No action is needed for Toolforge users.
== Why are we doing this? ==
On September 10, 2022 the Debian project stopped providing security patch support for the Debian 10 "Buster" release. It did, however, move to LTS support and will receive updates other than backports from the LTS team until end of June 2024.
Once Buster is no longer supported, Servers using Buster will become subject to ever more security vulnerabilities and increasingly difficult to maintain. Eventually the cloud-vps staff will find it necessary to stop and delete such VMs in order to prevent them from become security risks to cloud-vps and the greater internet.
== See also ==
* Operating system upgrade policy
* News/2021 Stretch deprecation
* Debian Buster release information
Noc.wikimedia.org

noc.wikimedia.org provides public copies of our configuration files. This should not be confused with the noc at wikimedia dot org email address that goes to all of SRE. 
You can use in pages on this wiki to link to a config file in https://noc.wikimedia.org/conf.
== Service ==
noc is now served from the "mw-misc" cluster in MediaWiki On Kubernetes.
For internal access, it may be queried directly with e.g. curl -s -H 'Host: noc.wikimedia.org' https://mw-misc.discovery.wmnet:30443/conf/dblists/open.dblist
Alternatively, mw-misc is available via port localhost:6508 when using the Service Proxy - See operations/puppet/+/1114383 for where this was added.
== History ==
To learn about the creation of noc.wikimedia.org and earlier pre-2010 interations, see kohl.wikimedia.org.
In 2010, noc.wikimedia.org was served by the deployment server (at the time, fenari) as one of the virtual hosts co-located with the MediaWiki install for test.wikipedia.org. The noc site also used to serve public_html from personal home to serve ad-hoc files and data. These would later be moved to people.wikimedia.org.
By 2015, the noc.wikimedia.org and people.wikimedia.org sites were moved away from the deployment server to a dedicated maintenance server (at the time, terbium). The maintenance servers did not serve any MediaWiki sites, with the webserver serving only the noc app from the wmf-config repository as a mostly standalone web app.
By 2022, it became multi-dc aware, and served from the then-current maintenance servers mwmaint1002 (eqiad) and mwmaint2002 (codfw).
In 2023, the noc app was moved to a new "mw-misc" cluster in MediaWiki on Kubernetes (T341859).
== See also ==
* people.wikimedia.org: The public_html home directories used to be served from the deployment server, under the noc.wikimedia.org site.
* Data Platform/Internal API requests: Background information on host to access services using our service mesh.
== External links ==
* https://noc.wikimedia.org
* Gerrit repositories for the files:
** 
Category:Services