News/2019 Toolforge Trusty deprecation

This page details information about deprecating and removing hosts running Ubuntu Trusty (14.04) as an operating system from the Toolforge infrastructure. The login bastions and Grid execution hosts are still running Trusty and must be replaced with new instances.
== What is changing? ==
* New job grid running Son of Grid Engine on Debian Stretch instances
* New limits on concurrent job execution and job submission by a single tool
* New bastion hosts running Debian Stretch with connectivity to the new job grid
* New versions of PHP, Python2, Python3, and other language runtimes
* New versions of various support libraries
== Timeline ==
* 2019-01-11: Availability of Debian Stretch grid announced to community
* Week of 2019-02-04: Weekly reminders via email to tool maintainers for tools still running on Trusty
* Week of 2019-03-04:
** Daily reminders via email to tool maintainers for tools still running on Trusty
** Switch login.tools.wmflabs.org to point to Stretch bastion
* 2019-03-25: Shutdown Trusty grid
== What should I do? ==
=== SSH to the Stretch bastion ===
login.tools.wmflabs.org connects to the new Debian Stretch bastion.
=== Move a grid engine webservice ===
When possible, we recommend migrating web services to Kubernetes instead of the new grid:
:# Connect to the Stretch bastion
$ ssh <your-shell-name>@login.tools.wmflabs.org
:# Become your tool account
$ become YOUR_TOOL
:# Start the webservice as a Kubernetes container rather than a grid job
:# <type> is one of: php7.2, php5.6, python, python2, nodejs, golang, jdk8, ruby2, tcl
$ webservice --backend=kubernetes <type> start
:# -- OR --
:# Start the webservice as a Stretch grid job
:# <type> is one of: lighttpd, uwsgi-python, tomcat, generic, lighttpd-plain, nodejs, uwsgi-plain
$ webservice --backend=gridengine <type> start
See Help:Toolforge/Web#Backends for more information on migrating from grid engine to Kubernetes.
=== Move a continuous job ===
:# Connect to the Stretch bastion
$ ssh <your-shell-name>@login.tools.wmflabs.org
:# Become your tool account
$ become YOUR_TOOL
:# Start your job on the Stretch job grid
$ jstart ...
The exact commands needed to start each continuous job vary greatly from tool to tool. This would be a great time to make a page of reference material for yourself and other maintainers here on Wikitech in the namespace and using the Tool template if you haven't already.
=== Move a cron job ===
The crontab data for all tools which still had a cron registered on the Trusty grid were backed up to $HOME/crontab.trusty.save before the Trusty cron server was shutdown. This backup can be used to setup your crontab on the Stretch grid.
:# Connect to the Stretch bastion
$ ssh <your-shell-name>@login.tools.wmflabs.org
:# Become your tool account
$ become YOUR_TOOL
:# Load the backup of your crontab on the Stretch job grid
$ crontab $HOME/crontab.trusty.save
If your workload permits, please avoid scheduling cronjobs from midnight to 3am so you're not competing with other cronjobs for system resources. That time window is currently very crowded.
== What are the primary changes with moving to Stretch? ==
=== Language runtime and library versions ===
The vast majority of the language runtimes and libraries installed on the grid nodes are upgraded in Stretch.
Runtime	Trusty Version	Stretch Version
Python3	3.4.0 	3.5.3
PHP 	5.5.9 	7.2
Python2	2.7.5 	2.7.13
NodeJS 	0.10.25 	8.11.1
Perl 	5.18.2 	5.24.1
Java 	1.7.0 	11.0.1
Ruby 	1.9.3 	2.3.3
Mono 	5.12.0 	5.12.0
TCL 	8.6.1 	8.6.0
R 	3.2.3 	3.3.3
Also note that the system-installed phpunit is not going to be present due to lack of current packages for recent versions of PHP. To use phpunit, please install via composer (instructions for setting up composer are included here Help:Toolforge#Installing_MediaWiki_core)
A table of the primary packages that users are likely to notice changes in is below.
Package 	Trusty Version 	Stretch Version
fonts-arabeyes 	2.1-3 	2.1-3
fonts-arphic-ukai 	0.2.20080216.2-4ubuntu2 	0.2.20080216.2-4
fonts-arphic-uming 	0.2.20080216.2-7ubuntu2 	0.2.20080216.2-7
fonts-farsiweb 	0.4.dfsg-12 	0.4.dfsg-12
fonts-kacst 	2.01+mry-10 	2.01+mry-12
fonts-khmeros 	5.0-7ubuntu1 	5.0-7
fonts-lao 	0.0.20060226-9 	0.0.20060226-9
fonts-liberation 	1.07.3-3 	1:1.07.4-2
fonts-linuxlibertine 	5.3.0-2 	5.3.0-2
fonts-manchufont 	2.007.svn0068-4 	2.007.svn0068-4
fonts-nafees 	1.2-5ubuntu1 	1.2-5
fonts-sil-abyssinica 	1.200-6 	1.500-1
fonts-sil-ezra 	2.51-8 	2.51-8
fonts-sil-padauk 	2.80-2 	2.80-2
fonts-sil-scheherazade 	1.001-8 	2.100-1
fonts-takao-gothic 	003.02.01-9ubuntu2 	00303.01-1
fonts-takao-mincho 	003.02.01-9ubuntu2 	00303.01-1
fonts-thai-tlwg 	1:0.5.1-3 	1:0.6.3-1
fonts-tibetan-machine 	1.901b-5 	1.901b-5
fonts-unfonts-core 	1.0.3.is.1.0.2-080608-10ubuntu1 	1.0.2-080608-13
fonts-unfonts-extra 	1.0.3.is.1.0.2-080608-4ubuntu2 	1.0.2-080608+dfsg-6
texlive-fonts-recommended 	2013.20140215-1ubuntu0.1 	2016.20170123-5
texlive-full 	2013.20140215-1ubuntu0.1 	2016.20170123-5
ttf-alee 	13 	13.1
ttf-ubuntu-font-family 	0.80-0ubuntu6 	0.80-0ubuntu6+wmf1
ttf-wqy-zenhei 	0.9.45-5ubuntu1 	0.9.45-6
xfonts-100dpi 	1:1.0.3 	1:1.0.4+nmu1
xfonts-75dpi 	1:1.0.3 	1:1.0.4+nmu1
xfonts-base 	1:1.0.3 	1:1.0.4+nmu1
xfonts-mplus 	2.2.4-1 	1:2.2.4-2
xfonts-scalable 	1:1.0.3-1 	1:1.0.3-1.1
fonts-sil-nuosusil 	2.1.1-7 	2.1.1-8
culmus 	0.130-1 	0.130-1
culmus-fancy 	0.0.20051018-3 	0.0.20051018-3
fonts-lklug-sinhala 	0.6-3 	0.6-3
fonts-vlgothic 	20130607-1ubuntu1 	20141206-3
fonts-dejavu-core 	2.34-1ubuntu1 	2.37-1
fonts-dejavu-extra 	2.34-1ubuntu1 	2.37-1
fonts-lyx 	2.0.8.1-0ubuntu1 	2.2.2-1
fonts-crosextra-carlito 	20130920-1 	20130920-1
fonts-crosextra-caladea 	20130214-1 	20130214-1
fonts-smc 	6.0-1 	6.0-1
fonts-hosny-amiri 	0.107-1 	0.107-1
fonts-taml-tscu 	1.0-2 	1.0-2
qpdf 	8.0.2-3~14.04.1 	6.0.0-2
pngquant 	2.0.1-1 	2.5.0-2
unpaper 	0.4.2-1 	6.1-2+b1
ttf-bengali-fonts 	1:0.5.14ubuntu1 	x
ttf-devanagari-fonts 	1:0.5.14ubuntu1 	x
ttf-gujarati-fonts 	1:0.5.14ubuntu1 	x
ttf-kannada-fonts 	1:0.5.14ubuntu1 	x
ttf-oriya-fonts 	1:0.5.14ubuntu1 	x
ttf-punjabi-fonts 	1:0.5.14ubuntu1 	x
ttf-tamil-fonts 	1:0.5.14ubuntu1 	x
ttf-telugu-fonts 	1:0.5.14ubuntu1 	x
ttf-kochi-gothic 	20030809-15 	x
ttf-kochi-mincho 	20030809-15 	x
fonts-mgopen 	1.1-9 	x
fonts-beng 	x 	2:1.2
fonts-deva 	x 	2:1.2
fonts-gujr 	x 	2:1.2
fonts-knda 	x 	2:1.2
fonts-mlym 	x 	2:1.2
fonts-orya 	x 	2:1.2
fonts-guru 	x 	2:1.2
fonts-taml 	x 	2:1.3
fonts-telu 	x 	2:1.2
fonts-gujr-extra 	x 	1.0-5
fonts-noto-cjk 	x 	1:1.004+repack2-2
fonts-sil-lateef 	x 	1.001-1
fonts-ipafont-gothic 	x 	00303-16
fonts-ipafont-mincho 	x 	00303-16
language-pack-ar 	1:14.04+20160720 	x
language-pack-bn 	1:14.04+20160720 	x
language-pack-ca 	1:14.04+20160720 	x
language-pack-de 	1:14.04+20160720 	x
language-pack-en 	1:14.04+20160720 	x
language-pack-es 	1:14.04+20160720 	x
language-pack-eu 	1:14.04+20160720 	x
language-pack-fr 	1:14.04+20160720 	x
language-pack-he 	1:14.04+20160720 	x
language-pack-hi 	1:14.04+20160720 	x
language-pack-ja 	1:14.04+20160720 	x
language-pack-ko 	1:14.04+20160720 	x
language-pack-mr 	1:14.04+20160720 	x
language-pack-nl 	1:14.04+20160720 	x
language-pack-pa 	1:14.04+20160720 	x
language-pack-pt 	1:14.04+20160720 	x
language-pack-ru 	1:14.04+20160720 	x
language-pack-uk 	1:14.04+20160720 	x
language-pack-zh-hans 	1:14.04+20160720 	x
language-pack-zh-hant 	1:14.04+20160720 	x
gcj-jre 	4:4.8.2-1ubuntu6 	4:6.3.0-4
golang 	2:1.3.3-1ubuntu4~ubuntu14.04.1 	2:1.7~5
luarocks 	2.0.9-1 	2.4.2+dfsg-1
mono-complete 	5.12.0.226-0xamarin3+ubuntu1404b1 	5.12.0.226-0xamarin3+debian9b1
mono-fastcgi-server 	4.5-0xamarin3+ubuntu1404b1 	4.5-0xamarin3+debian9b1
mono-vbnc 	4.7-0xamarin1+ubuntu1404b1 	4.7-0xamarin1+debian9b1
npm 	1.3.10~dfsg-1 	5.8.0
nodejs 	0.10.25~dfsg2-2ubuntu1+wmf1 	8.11.1~dfsg-2~bpo9+1
openjdk-7-jre-headless 	7u181-2.6.14-0ubuntu0.3 	x
openjdk-11-jre-headless 	x 	11.0.1+13-2~bpo9+1
icedtea-7-jre-jamvm 	7u181-2.6.14-0ubuntu0.3 	x
r-base 	3.2.3-1trusty0 	3.3.3-1
ruby 	1.9.3.484-2ubuntu1.13 	2.3.3
tcl8.6 	8.6.1-4ubuntu1 	8.6.0+9
libberkeleydb-perl 	0.54-1 	0.55-1+b2
libbot-basicbot-perl 	0.89-1 	0.89-1
libbsd-resource-perl 	1.2907-1build1 	1.2910-1+b1
libcache-memcached-fast-perl 	0.21-1build1 	0.23-1+b2
libcgi-fast-perl 	5.18.2-2ubuntu1.7 	1:2.12-1
libclass-data-inheritable-perl 	0.08-2 	0.08-2
libcommon-sense-perl 	3.72-2build1 	3.74-2
libcrypt-gcrypt-perl 	1.26-1build1 	1.26-5+b1
libcrypt-openssl-bignum-perl 	0.04-4build1 	0.07-2
libcrypt-openssl-rsa-perl 	0.28-1build1 	0.28-5
libdata-compare-perl 	1.22-0.1 	1.25-0.1
libdata-dumper-simple-perl 	0.11-4 	0.11-5
libdatetime-format-duration-perl	1.03a-1 	1.03a-1.2
libdatetime-format-strptime-perl	1.5400-1 	1.7200-1
libdbd-mysql-perl 	4.025-1ubuntu0.1 	4.041-2
libdbd-sqlite2-perl 	2:0.36-1build1 	2:0.37-1+b2
libdbd-sqlite3-perl 	1.40-3 	1.54-1
libdbi-perl 	1.630-1 	1.636-1+b1
libdigest-crc-perl 	0.18-1build1 	0.21-1+b2
libdigest-hmac-perl 	1.03+dfsg-1 	1.03+dfsg-1
libfile-nfslock-perl 	1.21-1 	1.27-1
libgd-gd2-perl 	2.46-3.1build1 	2.53-3
libhtml-format-perl 	2.11-1 	2.12-1
libhtml-html5-entities-perl 	0.003-2 	0.004-1
libhtml-parser-perl 	3.71-1build1 	3.72-3
libhtml-template-perl 	2.95-1 	2.95-2
libhttp-message-perl 	6.06-1 	6.11-1
libimage-exiftool-perl 	9.46-1 	10.40-1
libio-socket-ssl-perl 	1.965-1ubuntu1 	2.044-1
libipc-run-perl 	0.92-1 	0.94-1+deb9u1
libirc-utils-perl 	0.12-1 	0.12-1
libjson-perl 	2.61-1 	2.90-1
libjson-xs-perl 	2.340-1build1 	3.030-1
liblog-log4perl-perl 	1.41-1.1ubuntu1 	1.48-1+deb9u1
liblwp-protocol-https-perl 	6.04-2ubuntu0.1 	6.06-2
libmediawiki-api-perl 	0.39-1 	0.41-1
libmediawiki-bot-perl 	5.005006-1 	5.006003-1
libnet-netmask-perl 	1.9016-1 	1.9022-1
libnet-oauth-perl 	0.28-1 	0.28-2
libnet-ssleay-perl 	1.58-1 	1.80-1
libnetaddr-ip-perl 	4.071+dfsg-1 	4.079+dfsg-1+b1
libobject-pluggable-perl 	1.29-1 	1.29-1
libparse-mediawikidump-perl 	1.0.6-1 	1.0.6-2
libpod-simple-wiki-perl 	0.15-1 	0.20-1
libpoe-component-irc-perl 	6.83+dfsg-1 	6.88+dfsg-1
libpoe-component-syndicator-perl	0.06-1 	0.06-1
libpoe-filter-ircd-perl 	2.44-1 	2.44-1
libpoe-perl 	2:1.3580-1 	2:1.3670-1
libppix-regexp-perl 	0.036-1 	0.050-1
libreadonly-perl 	1.04-1 	2.050-1
libredis-perl 	2:1.9670-1 	2:1.9910-1
libregexp-common-perl 	2013031301-1 	2016060801-1
libsocket-getaddrinfo-perl 	0.22-1build1 	0.22-3
libsort-fields-perl 	0.90-2 	0.90-2
libstring-diff-perl 	0.06-1 	0.07-2
libtask-weaken-perl 	1.04-1 	1.04-1
libtest-exception-perl 	0.32-1 	0.43-1
libtext-diff-perl 	1.41-1 	1.44-1
libtimedate-perl 	2.3000-1 	2.3000-2
liburi-encode-perl 	0.090-1 	1.1.1-1
liburi-perl 	1.60-1 	1.71-1
libwww-mechanize-perl 	1.73-1 	1.83-1
libwww-perl 	6.05-2 	6.15-1
libxml-libxml-perl 	2.0108+dfsg-1ubuntu0.2 	2.0128+dfsg-1+deb9u1
libxml-parser-perl 	2.41-1build3 	2.44-2+b1
libxml-simple-perl 	2.20-1 	2.22-1
libxml-xpathengine-perl 	0.13-1 	0.13-1
perl-modules 	5.18.2-2ubuntu1.7 	5.24.1-3+deb9u5
python-flask-oauth 	0.12-1 	x
python-mwparserfromhell 	0.3.3-1 	0.4.2-1.1+b1
python-oursql 	0.9.3.1-1 	x
python-socketio-client 	0.5.3-1 	0.6.5-0.1
python-wikitools 	1.1.1-1 	x
python-mwclient 	0.8.4-1 	x
python-pymysql 	0.6.2-2 	0.7.10-1
python3-pymysql 	0.7.2-1 	0.7.10-1
php5-cli 	5.5.9+dfsg-1ubuntu4.26 	x
php5-common 	5.5.9+dfsg-1ubuntu4.26 	x
php5-curl 	5.5.9+dfsg-1ubuntu4.26 	x
php5-gd 	5.5.9+dfsg-1ubuntu4.26 	x
php5-imagick 	3.1.2-1build1 	x
php5-intl 	5.5.9+dfsg-1ubuntu4.26 	x
php5-mcrypt 	5.4.6-0ubuntu5 	x
php5-mysqlnd 	5.5.9+dfsg-1ubuntu4.26 	x
php5-pgsql 	5.5.9+dfsg-1ubuntu4.26 	x
php5-redis 	2.2.4-1build2 	x
php5-sqlite 	5.5.9+dfsg-1ubuntu4.26 	x
php5-xdebug 	2.2.3-2build1 	x
php5-xhprof 	0.9.4-1build1 	x
php5-xsl 	5.5.9+dfsg-1ubuntu4.26 	x
php5-readline 	5.5.9+dfsg-1ubuntu4.26 	x
php7.2-cli 	x 	7.2.13
php7.2-common 	x 	7.2.13
php7.2-curl 	x 	7.2.13
php7.2-gd 	x 	7.2.13
php-imagick 	x 	3.4.3~rc2-2
php7.2-intl 	x 	7.2.13
php7.2-mysql 	x 	7.2.13
php7.2-pgsql 	x 	7.2.13
php-redis 	x 	4.1.1
php7.2-sqlite3 	x 	7.2.13
php-xdebug 	x 	2.7.0
php7.2-xml 	x 	7.2.13
php7.2-readline 	x 	7.2.13
ttf-indic-fonts-core 	1:0.5.14ubuntu1 	x
fonts-indic 	x 	2:1.2
texlive-fonts-extra 	2013.20140215-2 	2016.20170123-5
mysqltcl 	3.051-1 	3.052-2
tcl-tls 	1.6+dfsg-3 	1.6.7+dfsg-1.2
tcl-trf 	2.1.4-dfsg2-1 	2.1.4-dfsg3-2+b1
tclcurl 	7.22.0-1 	7.22.0+hg20160822-1
tcllib 	1.15-dfsg-2 	1.18-dfsg-3
tdom 	0.8.3-1 	0.8.3-1
tesseract-ocr 	3.03.02-3 	3.04.01-5
tesseract-ocr-afr 	3.02-2 	3.04.00-1
tesseract-ocr-ara 	3.02-2 	3.04.00-1
tesseract-ocr-aze 	3.02-2 	3.04.00-1
tesseract-ocr-bel 	3.02-2 	3.04.00-1
tesseract-ocr-ben 	3.02-2 	3.04.00-1
tesseract-ocr-bul 	3.02-2 	3.04.00-1
tesseract-ocr-cat 	3.02-2 	3.04.00-1
tesseract-ocr-ces 	3.02-2 	3.04.00-1
tesseract-ocr-chi-sim 	3.02-1 	3.04.00-1
tesseract-ocr-chi-tra 	3.02-1 	3.04.00-1
tesseract-ocr-chr 	3.02-2 	3.04.00-1
tesseract-ocr-dan 	3.02-2 	3.04.00-1
tesseract-ocr-deu 	3.02-2 	3.04.00-1
tesseract-ocr-deu-frak 	3.02-4 	3.04.00-1
tesseract-ocr-ell 	3.02-2 	3.04.00-1
tesseract-ocr-eng 	3.02-2 	3.04.00-1
tesseract-ocr-enm 	3.02-2 	3.04.00-1
tesseract-ocr-epo 	3.02-2 	3.04.00-1
tesseract-ocr-equ 	3.02-2 	3.04.00-1
tesseract-ocr-est 	3.02-2 	3.04.00-1
tesseract-ocr-eus 	3.02-2 	3.04.00-1
tesseract-ocr-fin 	3.02-2 	3.04.00-1
tesseract-ocr-fra 	3.02-2 	3.04.00-1
tesseract-ocr-frk 	3.02-2 	3.04.00-1
tesseract-ocr-frm 	3.02-2 	3.04.00-1
tesseract-ocr-glg 	3.02-2 	3.04.00-1
tesseract-ocr-heb 	3.02-2 	3.04.00-1
tesseract-ocr-hin 	3.02-2 	3.04.00-1
tesseract-ocr-hrv 	3.02-2 	3.04.00-1
tesseract-ocr-hun 	3.02-2 	3.04.00-1
tesseract-ocr-ind 	3.02-2 	3.04.00-1
tesseract-ocr-isl 	3.02-2 	3.04.00-1
tesseract-ocr-ita 	3.02-2 	3.04.00-1
tesseract-ocr-ita-old 	3.02-1 	3.04.00-1
tesseract-ocr-jpn 	3.02-2 	3.04.00-1
tesseract-ocr-kan 	3.02-2 	3.04.00-1
tesseract-ocr-kor 	3.02-2 	3.04.00-1
tesseract-ocr-lav 	3.02-2 	3.04.00-1
tesseract-ocr-lit 	3.02-2 	3.04.00-1
tesseract-ocr-mal 	3.02-2 	3.04.00-1
tesseract-ocr-mkd 	3.02-2 	3.04.00-1
tesseract-ocr-mlt 	3.02-2 	3.04.00-1
tesseract-ocr-msa 	3.02-2 	3.04.00-1
tesseract-ocr-nld 	3.02-2 	3.04.00-1
tesseract-ocr-nor 	3.02-2 	3.04.00-1
tesseract-ocr-osd 	3.02-2 	3.04.00-1
tesseract-ocr-pol 	3.02-2 	3.04.00-1
tesseract-ocr-por 	3.02-2 	3.04.00-1
tesseract-ocr-ron 	3.02-2 	3.04.00-2
tesseract-ocr-rus 	3.02.02-1 	3.04.00-1
tesseract-ocr-slk 	3.02-2 	3.04.00-1
tesseract-ocr-slk-frak 	3.02-2 	3.04.00-1
tesseract-ocr-slv 	3.02-2 	3.04.00-1
tesseract-ocr-spa 	3.02-2 	3.04.00-1
tesseract-ocr-spa-old 	3.02-1 	3.04.00-1
tesseract-ocr-sqi 	3.02-2 	3.04.00-1
tesseract-ocr-srp 	3.02-2 	3.04.00-1
tesseract-ocr-swa 	3.02-2 	3.04.00-1
tesseract-ocr-swe 	3.02-2 	3.04.00-1
tesseract-ocr-tam 	3.02-2 	3.04.00-1
tesseract-ocr-tel 	3.02-2 	3.04.00-1
tesseract-ocr-tgl 	3.02-2 	3.04.00-1
tesseract-ocr-tha 	3.02-2 	3.04.00-1
tesseract-ocr-tur 	3.02-2 	3.04.00-1
tesseract-ocr-ukr 	3.02-2 	3.04.00-1
tesseract-ocr-vie 	3.02.02-1 	3.04.00-1
bison 	2:3.0.2.dfsg-2 	2:3.0.4.dfsg-1+b1
calibre 	1.25.0+dfsg-1ubuntu1.2 	2.75.1+dfsg-1
csh 	20110502-2ubuntu2 	20110502-2.2+b1
csvtool 	1.3.2-1 	1.5-1+b1
dc 	1.06.95-8ubuntu1 	1.06.95-9+b3
djvulibre-bin 	3.5.25.4-3 	3.5.27.1-7
djvulibre-plugin 	4.9-4ubuntu3 	4.10.6-1
doxygen 	1.8.6-2 	1.8.13-4+b1
doxygen-latex 	1.8.6-2 	1.8.13-4
expect 	5.45-5ubuntu1 	5.45-7+deb9u1
fabric 	1.8.2-1 	1.13.1-2
gawk 	1:4.0.1+dfsg-2.1ubuntu2 	1:4.1.4+dfsg-1
gdal-bin 	1.10.1+dfsg-5ubuntu1 	2.1.2+dfsg-5
git-review 	1.23-1 	1.25.0-2
git-svn 	1:1.9.1-1ubuntu0.10 	1:2.11.0-3+deb9u4
gnuplot-nox 	4.6.4-2 	5.0.5+dfsg1-6+deb9u1
graphicsmagick 	1.3.18-1ubuntu3.1 	1.3.30+hg15796-1~deb9u2
graphviz 	2.36.0-0ubuntu3.2 	2.38.0-17
grep 	2.16-1 	2.27-2
hugin-tools 	2013.0.0+dfsg-1ubuntu2 	2016.2.0+dfsg-1
hunspell 	1.3.2-6ubuntu2.1 	1.4.1-2+b2
inkscape 	0.48.4-3ubuntu2 	0.92.1-1
iotop 	0.6-1 	0.6-2
ksh 	93u+20120801-1ubuntu0.14.04.1 	93u+20120801-3.1
lame 	3.99.5+repack1-3ubuntu1 	3.99.5+repack1-9+b2
libaio1 	0.3.109-4 	0.3.110-3
libav-tools 	6:9.20-0ubuntu0.14.04.1 	7:3.2.12-1~deb9u1
libdmtx0a 	0.7.4-2 	0.7.4-2
libfcgi0ldbl 	2.4.0-8.1ubuntu5 	2.4.0-8.4+b1
libffi-dev 	3.1~rc1+r3.0.13-12ubuntu0.2 	3.2.1-6
libfreetype6 	2.5.2-1ubuntu2.8 	2.6.3-3.2
libgdbm-dev 	1.8.3-12build1 	1.8.3-14
libgeoip1 	1.6.0-1 	1.6.9-4
libhunspell-dev 	1.3.2-6ubuntu2.1 	1.4.1-2+b2
libjpeg-turbo-progs 	1.3.0-0ubuntu2.1 	1:1.5.1-2
libmpfr4 	3.1.2-1 	3.1.5-1
libncurses5-dev 	5.9+20140118-1ubuntu1 	6.0+20161126-1+deb9u2
libneon27-gnutls 	0.30.0-1ubuntu1 	0.30.2-2
libnfnetlink0 	1.0.1-2 	1.0.1-3
libnspr4 	2:4.13.1-0ubuntu0.14.04.1 	2:4.12-6
libnss3 	2:3.28.4-0ubuntu0.14.04.3 	2:3.26.2-1.1+deb9u1
libnss3-1d 	2:3.28.4-0ubuntu0.14.04.3 	x
libotf0 	0.9.13-1ubuntu1 	0.9.13-3+b1
libpcsclite1 	1.8.10-1ubuntu1.1 	1.8.20-1
libpng3/libpng16-16 	1.2.50-1ubuntu2.14.04.3 	1.6.28-1
libproj0/libproj12 	4.8.0-2ubuntu2 	4.9.3-1
libquadmath0 	4.8.4-2ubuntu1~14.04.4 	6.3.0-18+deb9u1
librsvg2-bin 	2.40.2-1+wm2 	2.40.16-1+b1
libsvn1 	1.8.8-1ubuntu3.3 	1.9.5-1+deb9u3
libvips-tools 	7.38.5-2 	8.4.5-1+b1
libxml2-utils 	2.9.1+dfsg1-3ubuntu4.13 	2.9.4+dfsg1-2.2+deb9u2
libyaml-dev 	0.1.4-3ubuntu3.1 	0.1.7-2
libzbar0 	0.10+doc-9build1 	0.10+doc-10.1
mailutils 	1:2.99.98-1.1 	1:3.1.1-1
mdbtools 	0.7.1-1ubuntu1 	0.7.1-5
melt 	0.9.0-3 	6.4.1-4
mktorrent 	1.0-4 	1.0-4+b2
openbabel 	2.3.2+dfsg-1.1 	2.3.2+dfsg-3
p7zip-full 	9.20.1~dfsg.1-4+deb7u3build0.14.04.1	16.02+dfsg-3+deb9u1
pdf2svg 	0.2.2-1 	0.2.2-1+b1
pdf2djvu 	0.7.17-3ubuntu2 	0.9.4-1+b3
pdftk 	2.01-1 	2.02-4+b2
phantomjs 	1.9.0-1 	2.1.1+dfsg-2
phpunit 	3.7.28-1 	x
poppler-utils 	0.24.5-2ubuntu4.14 	0.48.0-2+deb9u2
postgis 	2.1.2+dfsg-2ubuntu0.1 	2.3.1+dfsg-2
postgresql-client 	9.3+154ubuntu1.1 	9.6+181+deb9u2
pstoedit 	3.62-1ubuntu1 	3.70-3+b2
rrdtool 	1.4.7-2ubuntu5 	1.6.0-1+b2
socat 	1.7.2.3-1 	1.7.3.1-2+deb9u1
sqlite3 	3.8.2-1ubuntu2.1 	3.16.2-5+deb9u1
supybot 	0.83.4.1.ds-2 	0.83.4.1.ds-3
tabix 	0.2.6-2 	1.3.2-2
texinfo 	5.2.0.dfsg.1-2 	6.3.0.dfsg.1-1+b2
texlive 	2013.20140215-1ubuntu0.1 	2016.20170123-5
ufraw-batch 	0.19.2-2ubuntu2 	0.22-1.1
virtualenvwrapper 	4.1.1-1 	4.3.1-2
whois 	5.1.1 	5.2.17~deb9u1
xml2 	0.4-3.1 	0.4-3.1+deb9u1
xsltproc 	1.1.28-2ubuntu0.1 	1.1.29-2.1
xvfb 	2:1.15.1-0ubuntu2.11 	2:1.19.2-1+deb9u5
zbar-tools 	0.10+doc-9build1 	0.10+doc-10.1
debootstrap 	1.0.59ubuntu0.10 	1.0.89
fakechroot 	2.17.2-1 	2.19-3
hhvm 	3.12.7+dfsg-1+wmf1~trusty1 	3.18.5+dfsg-1+wmf8+deb9u1
libboost-python1.##.0 	1.54.0-4ubuntu3.1 	1.62.0+dfsg-4
libmpc3 	1.0.1-1ubuntu1 	1.0.3-1+b2
libprotobuf8/10 	2.5.0-9ubuntu1 	3.0.0-9
libbytes-random-secure-perl 	0.28-1 	0.28-1
libvips37/42 	7.38.5-2 	8.4.5-1+b1
nodejs-legacy 	0.10.25~dfsg2-2ubuntu1+wmf1 	6.11.0~dfsg-1+wmf5
mariadb-client 	5.5.61-1ubuntu0.14.04.1 	10.1.37-0+deb9u1
opencv-data 	2.4.8+dfsg1-2ubuntu1.2 	2.4.9.1+dfsg1-2
python-flake8 	2.1.0-1ubuntu1 	3.2.1-1
python3-flake8 	2.1.0-1ubuntu1 	3.2.1-1
tcl-thread 	1:2.7.0-1 	1:2.8.0-1
python-redis 	2.7.2-1build1 	2.10.5-2
python 	2.7.5-5ubuntu3 	2.7.13-2
python3 	3.4.0-0ubuntu2 	3.5.3-1
python-babel 	1.3+dfsg.1-2ubuntu2 	2.3.4+dfsg.1-2
python3-babel 	1.3+dfsg.1-2ubuntu2 	2.3.4+dfsg.1-2
python-beautifulsoup 	3.2.1 	3.2.1-1
python-bottle 	0.12.0-1 	0.12.13-1
python3-bottle 	0.12.0-1 	0.12.13-1
python-celery 	3.1.6-1ubuntu1 	3.1.23-7
python3-celery 	x 	3.1.23-7
python-cffi 	0.8.2-0ubuntu1 	1.9.1-2
python3-cffi 	0.8.2-0ubuntu1 	1.9.1-2
python-egenix-mxdatetime 	3.2.7-1build1 	3.2.9-1
python-egenix-mxtools 	3.2.7-1build1 	3.2.9-1
python-enum34 	0.9.23-1 	1.1.6-1
python3-enum34 	0.9.23-1 	
python-flask 	0.10.1-2build1 	0.12.1-1
python3-flask 	0.10.1-2build1 	0.12.1-1
python-flask-login 	0.2.6-1 	x
python3-flask-login 	x 	0.4.0-1
python-flickrapi 	1.2-3 	2.1.2-5
python3-flickrapi 	x 	2.1.2-5
python-flup 	1.0.2-4 	1.0.2-5
python-gdal 	1.10.1+dfsg-5ubuntu1 	2.1.2+dfsg-5
python3-gdal 	1.10.1+dfsg-5ubuntu1 	2.1.2+dfsg-5
python-gdbm 	2.7.5-1ubuntu1 	2.7.13-1
python3-gdbm 	3.4.3-1~14.04.2 	3.5.3-1
python-genshi 	0.7-3 	0.7-5
python3-genshi 	0.7-3 	0.7-5
python-genshi-doc 	0.7-3 	0.7-5
python-geoip 	1.2.4-2ubuntu5 	1.3.2-1+b2
python3-geoip 	x 	1.3.2-1+b2
python-gevent 	1.1b6-1~trusty1 	1.1.2-1
python3-gevent 	x 	1.1.2-1
python-gi 	3.12.0-1ubuntu1 	3.22.0-2
python3-gi 	3.12.0-1ubuntu1 	3.22.0-2
python-greenlet 	0.4.2-1ubuntu0.1 	0.4.11-1
python3-greenlet 	0.4.2-1ubuntu0.1 	0.4.11-1
python-httplib2 	0.8-2build1 	0.9.2+dfsg-1
python3-httplib2 	0.8-2build1 	0.9.2+dfsg-1
python-imaging 	2.3.0-1ubuntu3.4 	4.0.0-4
python3-imaging 	2.3.0-1ubuntu3.4 	x
python-ipaddr 	2.1.10-1 	2.1.11-2
python3-ipaddr 	2.1.10-1 	x
python-irclib 	0.4.8-1.1 	x
python-keyring 	3.5-1 	10.1-1
python3-keyring 	3.5-1 	10.1-1
python-launchpadlib 	1.10.2+ds-2 	1.10.4-1
python3-launchpadlib 	x 	1.10.4-1
python-lxml 	3.3.3-1ubuntu0.2 	3.7.1-1
python3-lxml 	3.3.3-1ubuntu0.2 	3.7.1-1
python-magic 	1:5.14-2ubuntu3.4 	1:5.30-1+deb9u2
python3-magic 	1:5.14-2ubuntu3.4 	1:5.30-1+deb9u2
python-matplotlib 	1.3.1-1ubuntu5.1 	2.0.0+dfsg1-2
python3-matplotlib 	1.3.1-1ubuntu5.1 	2.0.0+dfsg1-2
python-mysql.connector 	1.1.6-1 	2.1.6-1
python3-mysql.connector 	1.1.6-1 	2.1.6-1
python-mysqldb 	1.2.3-2ubuntu1 	1.3.7-1.1
python3-mysqldb 	x 	1.3.7-1.1
python-newt 	0.52.15-2ubuntu5 	0.52.19-1+b1
python3-newt 	0.52.15-2ubuntu5 	0.52.19-1+b1
python-nose 	1.3.1-2 	1.3.7-2
python3-nose 	1.3.1-2 	1.3.7-2
python-numpy 	1:1.8.2-0ubuntu0.1 	1:1.12.1-3
python3-numpy 	1:1.8.2-0ubuntu0.1 	1:1.12.1-3
python-opencv 	2.4.8+dfsg1-2ubuntu1.2 	2.4.9.1+dfsg1-2
python-pandas 	0.13.1-2ubuntu2 	0.19.2-5.1
python3-pandas 	0.13.1-2ubuntu2 	0.19.2-5.1
python-pil 	2.3.0-1ubuntu3.4 	4.0.0-4
python3-pil 	2.3.0-1ubuntu3.4 	4.0.0-4
python-problem-report 	2.14.1-0ubuntu3.29 	x
python3-problem-report 	2.14.1-0ubuntu3.29 	x
python-psycopg2 	2.4.5-1build5 	2.6.2-1
python3-psycopg2 	2.4.5-1build5 	2.6.2-1
python-pycountry 	0.14.1+ds1-3build1 	1.8+ds1-0.2
python3-pycountry 	x 	1.8+ds1-0.2
python-pydot 	1.0.28-0ubuntu1 	1.0.28-2
python3-pydot 	x 	1.0.28-2
python-pyexiv2 	0.3.2-5ubuntu3 	0.3.2-8+b3
python-pygments 	1.6+dfsg-1ubuntu1.1 	2.2.0+dfsg-1
python3-pygments 	1.6+dfsg-1ubuntu1.1 	2.2.0+dfsg-1
python-pyicu 	1.5-2ubuntu4 	1.9.5-1
python3-pyicu 	1.5-2ubuntu4 	x
python-pyinotify 	0.9.4-1build1 	0.9.6-1
python3-pyinotify 	0.9.4-1build1 	0.9.6-1
python-requests 	2.2.1-1ubuntu0.4 	2.12.4-1
python3-requests 	2.2.1-1ubuntu0.4 	2.12.4-1
python-requests-oauthlib 	0.4.0-1 	0.7.0-0.1
python3-requests-oauthlib 	0.4.0-1 	0.7.0-0.1
python-rsvg 	2.32.0+dfsg-3 	2.32.0+dfsg-3
python-scipy 	0.13.3-1build1 	0.18.1-2
python3-scipy 	0.13.3-1build1 	0.18.1-2
python-sqlalchemy 	0.8.4-1build1 	1.0.15+ds1-1
python3-sqlalchemy 	0.8.4-1build1 	1.0.15+ds1-1
python-svn 	1.7.8-0.2 	1.9.4-2
python3-svn 	x 	1.9.4-2
python-tk 	2.7.5-1ubuntu1 	2.7.13-1
python3-tk 	3.4.3-1~14.04.2 	3.5.3-1
python-twisted 	13.2.0-1ubuntu1.2 	16.6.0-2
python3-twisted 	x 	16.6.0-2
python-twitter 	1.1+git20131227-2 	1.1+git20131227-2
python-unicodecsv 	0.9.4-1 	0.14.1-1
python3-unicodecsv 	x 	0.14.1-1
python-unittest2 	0.5.1-1ubuntu1 	1.1.0-6.1
python3-unittest2 	x 	1.1.0-6.1
python-virtualenv 	1.11.4-1ubuntu1 	15.1.0+ds-1
python3-virtualenv 	x 	15.1.0+ds-1
python-wadllib 	1.3.2-2build1 	1.3.2-3
python3-wadllib 	1.3.2-2build1 	1.3.2-3
python-webpy 	1:0.37+20120626-1 	1:0.38-1
python-werkzeug 	0.9.4+dfsg-1.1ubuntu2.1 	0.11.15+dfsg1-1
python3-werkzeug 	0.9.4+dfsg-1.1ubuntu2.1 	0.11.15+dfsg1-1
python-zbar 	0.10+doc-9build1 	0.10+doc-10.1
python-zmq 	14.0.1-1build2 	16.0.2-2
python3-zmq 	14.0.1-1build2 	16.0.2-2
python-stdeb 	0.6.0+20100620-4ubuntu1 	0.8.5-1
python3-stdeb 	x 	0.8.5-1
python-coverage 	3.7.1 	4.2
python3-coverage 	3.7.1 	4.2
=== Concurrency limits ===
* Maximum of 16 active jobs simultaneously allowed per tool user
** The scheduler will hold additional job submissions in the qw (queued/waiting) until an active slot is available. 
* Maximum of 50 active and queued jobs simultaneously allowed per tool user
** The scheduler will reject additional job submissions by exiting with a status code of 25 and writing "Unable to run job: job rejected: only 50 jobs are allowed per user (current job count: 50)" to stderr
Implementing these limits has allowed us enable job submission from the continuous and and task job queues.
== Solutions to common problems ==
Having trouble with the new grid? If the answer to your problem isn't here, ask for help in or file a bug in Phabricator.
=== Rebuild virtualenv for python users ===
Since the python executables and libraries are updated in stretch, local virtualenvs will need to be deleted and re-created on the new bastion for anything that runs from those virtualenvs to work. Several errors are likely to be caused by old virtualenvs with one obvious one being an unexpected ImportError.
Using a requirements file may make this simpler in many cases, if your project doesn't already use one. You can create one in your local directory by running pip freeze > requirements.txt in your tool folder with your virtualenv activated. Then later on, you can simply use pip install -r requirements.txt to install the new environment after you deleted the old virtualenv and created a new one. For more information on this option, see pip's documentation on requirements files.
===== Example 1: Upgrading a Trusty grid engine based tool to the Stretch grid =====
Follow these steps if you manually submit jobs using jsub, or if you submit jobs using a crontab.
$ ssh <your-shell-name>@login-stretch.tools.wmflabs.org
$ become YOUR_TOOL
$ rm -rf venv # This will destroy the virtualenv and all libraries, so make sure you know what you will need to install later!
$ virtualenv venv
$ source venv/bin/activate
$ pip install --upgrade pip # upgrade pip itself to avoid problems with older versions
$ pip install ... # Here you'd use the requirements file syntax if you have one, or you'd manually install each needed library.
===== Example 2: Upgrading a uWSGI webservice into a Kubernetes container =====
If you are currently running your uWSGI webservice under the Grid Engine backend (i.e., webservice uwsgi-python command), and you want to upgrade to a uWSGI webservice running under Kubernetes (i.e., webservice --backend=kubernetes python command), you should rebuild your virtualenv as follows:
$ ssh <your-shell-name>@login-stretch.tools.wmflabs.org
$ become YOUR-TOOL
$ webservice --backend=kubernetes python stop
$ webservice --backend=kubernetes python shell # do not skip this step – setting up the venv directly from the bastion may result in serious performance issues, compare T214086
$ rm -rf www/python/venv/ # this will destroy the virtualenv and all libraries, so make sure you know what you will need to install later!
$ python3 -m venv www/python/venv/
$ source www/python/venv/bin/activate
$ pip install --upgrade pip # upgrade pip itself to avoid problems with older versions
$ pip install -r www/python/src/requirements.txt # assuming your tool has a requirements.txt file
$ webservice --backend=kubernetes python start
===== Example 3: Upgrading a Kubernetes uWSGI webservice =====
If you are already using the Kubernetes backend, there is nothing you need to do -- the container will use the same Debian Jessie-based image as before.
=== PyYAML fails to install in Debian Stretch Python3 virtualenv ===
The new bastions are using systemd resource control to restrict the amount of RAM and CPU resources that a user can consume. We do this to attempt to keep a single user from using all of the shared resources of the bastion accidentally and thus making the bastion slow for everyone. The initial limits we had set were overly restrictive and caused gcc to fail when compiling PyYAML. This has been corrected by increasing the limits.
=== BotPassword or OAuth grant does not work from new job grid ===
Bot passwords and OAuth registrations can both include allowed IP range restrictions. The defaults for both are to allow usage from any IPv4 and IPv6 address. If you have changed this when creating the bot password or OAuth consumer registration to restrict access to specific IP address ranges you may have issues using the password or OAuth consumer from the new job grid. The Cloud VPS environment is nearing the end of a process of moving from the 10.0.0.0/8 private address range that is shared with other internal servers operated by the Wikimedia Foundation to a new 172.16.0.0/21 private subnet. The new job grid is the first end-user facing portion of Toolforge to be migrated to the new range.
The allowed IP ranges for bot passwords can be changed by the owner of the account using Special:BotPasswords. Either add the 172.16.0.0/21 CIDR to the list of allowed ranges or reset them to the defaults of 0.0.0.0/0 and ::/0.
The allowed IP ranges for an OAuth consumer registration can be changed by the original proposer of the registration using Special:OAuthConsumerRegistration/list. Either add the 172.16.0.0/21 CIDR to the list of allowed ranges or reset them to the defaults of 0.0.0.0/0 and ::/0.
=== Lighttpd crashes on startup with message "parser failed somehow near here: (EOL)" ===
Lighttpd 1.4.40 made overriding keys in an existing array a fatal error. The Stretch version of lighttpd is 1.4.45. This change in the upstream application makes the advice at Help:Toolforge/Web/Lighttpd#Header, mimetype, character_encoding, error_handler for replacing existing mime-type mappings with new local versions obsolete.
Look for a $HOME/error.log line similar to Duplicate array-key '.js' just prior to the parser failure error message to help you find the entry in your $HOME/.lighttpd.conf file that needs to be removed.
=== 'webservice stop' says service is not running, but 'webservice start' says service is running ===
 has this advice:
* webservice stop
* rm $HOME/service.manifest
* webservice [add other args here as needed] start
It is not completely well understood what causes webservice to become confused about the state of the process, but deleting the service.manifest file generally seems to fix the issue.
=== Python: redis.exceptions.ResponseError: value is not an integer or out of range ===
The Python Redis client made a breaking change in v3.0.0 vs older versions in renaming the prior StrictRedis class to Redis. The new behavior expects a different order of arguments for calls such as setex(). The expected order of arguments now matches the Redis protocol docs rather than the more "pythonic" order that the prior implementation used. Typically this means that you need to swap the order of the time and value arguments in your calling code. See the library documentation for more breaking changes.
=== Delete a tool ===
Some tools were experiments that are done, others were made obsolete by other tools, some are just things that the original maintainer is tired of caring for. Maintainers can mark their tools for deletion using the "Disable tool" button on the tool's detail page on https://toolsadmin.wikimedia.org/. Disabling a tool will immediately stop any running jobs including webservices and prevent maintainers from logging in as the tool. Disabled tools are archived and deleted after 40 days. Disabled tools can be re-enabled at any time prior to being archived and deleted.
=== Python 'oursql' package fails to compile ===
The latest official release of the Python 'oursql' package will not compile against MariaDB client libraries. See upstream bug report at https://github.com/python-oursql/oursql/issues/5. Oursql can be installed from a fork maintained at https://github.com/sqlobject/oursql, but the recommended long term solution is to migrate application code to the PyMySQL package instead.
=== SSH to login-stretch.tools.wmflabs.org fails with 'Permission denied (publickey)'===
This is typically an issue with the newer Debian Stretch provided version of sshd on the server side refusing to authenticate an insecure or deprecated public key type. Specifically, support for DSA (ssh-dss) keys was deprecated in Openssh 7.0. If your ssh public key starts with the string "ssh-dss" you will be impacted by this. RSA keys smaller than 1024 bits are also deprecated.
First make sure that you are passing a valid key by attempting to ssh to login-trusty.tools.wmflabs.org using the same public key and username. If this also fails, the problem is likely something other than the ssh key type. Join us in for interactive debugging help.
If you can ssh to login-trusty.tools.wmflabs.org with no errors, your key is probably of an unsupported type. Generate a new ssh key pair and upload the public key using the form at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys. We currently recommend using either ed25519 or 4096-bit RSA keys. See Production shell access#Generating your SSH key for more information.
=== SSH to login-stretch.tools.wmflabs.org fails with 'Permission denied (publickey,hostbased)' ===
In case you face this problem, make sure to use the right shell name located on your User Preferences called **Instance shell account name**. It's supposed to be used in logging into the Toolforge server when need be, whether Trusty or Stretch.
=== "Unable to run job: Error reading answer list from qmaster" ===
Attempting to start a job with a name including non-ASCII characters using jsub, jstart, qcronsub, etc may fail with an error message written to the job's err file like "Unable to run job: Error reading answer list from qmaster". This is a known bug in Son of Grid Engine.
== Monitoring tools ==
* Tools running jobs on Trusty grid engine in last 7 days
** This report updates once per hour and will not report jobs that have been seen running on the Stretch grid in the same 7 day period.
** Report has drill down pages for each maintainer and tool. Examples: bd808's tools, gridengine-status tool
** Webservices that move from the Trusty grid directly to the Kubernetes cluster will not be removed from the report automatically.
* Jobs running on the Stretch grid in the last 7 days
* Stretch grid status
== Why are we doing this? ==
Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time. 
During past operating system updates we were able to create a mixed grid which contained hosts running multiple operating systems and control which was used to run each job using command line arguments to jsub and webservice. The current version of Sun Grid Engine (v6.2u5) that exists in Ubuntu Trusty is incompatible with "Son of" Grid Engine (v8.1.9) from Debian Stretch. Therefore the two grids must be entirely separate environments. Any cron jobs that exist or web services in the old grid (submitted from one of the current bastions) will not currently exist in the new grid. To schedule any job or service on the new Son of Grid Engine grid, one must log into a bastion dedicated to that grid (currently tools-sgebastion-06.tools.eqiad.wmflabs) to submit them.
== See also ==
* [Cloud-announce] Toolforge: Trusty deprecation and grid engine migration
* [Cloud-announce] Toolforge: Trusty job grid deprecation reminders start today
* [Cloud-announce] Toolforge: Bastion changes and Trusty deprecation final steps