UESPWiki:Server Setup/New Servers Feb2011

Files1
:* Ran setup script uesp-setup-common files1 10.2.212.14 (no issues)
:* Ran setup script uesp-install-memcached 10.2.212.10 (no issues)
:* Ran setup script uesp-install-lighttpd (no issues)
:* Ran setup script uesp-setup-shares (no issues)
:* Copy files from current files server: rsync -avz dave@files1.uesp.net:/shared/* /shared/
:* Changed password for user dave
:* Ran setup script uesp-update-sshd (no issues)
:* Note: Make sure that /etc/init.d/nfslock is running.
:* Note: Add 127.0.0.1 to NFS entries in /etc/host.allow.
:* Installed Apache (for access to ab).
:* Edit /etc/init.d/memcached to start service using the private IP and restart.

Content3
:* Ran setup script uesp-setup-content content3 10.2.212.14 (failed to mount file shares)
:* Manually mounted shares from the new files1, for example:
::* mount newfiles1.uesp.net:/shared/uesp/wikiimages /mnt/test/wikiimages
::* Setup script failed to mount due to using files1.uesp.net which points to the current file server.
::* Added the new content3 to the entries in /etc/exports and /etc/hosts.allow on old files1.
::* Re-ran setup script uesp-setup-mounts (successfully mounted the uesp shares)
::* Tested shares as user apache to ensure read-write access (succeeded).
::* Added entry files1  10.2.212.14 to the /etc/hosts file on the new files1 (this can be removed after the files1 sub domain points to the new server).
::* Re-ran uesp-setup-mounts (no issues)
:* Changed password for user dave
:* Apache failed to start. Added entry for content3 in /etc/hosts and restarted.
:* Ran setup script uesp-setup-zabbixserver 174.142.196.117
:* Copied /etc/my.cnf from the old content3.
:* Copied /etc/httpd/conf/httpd.conf from the old content3.
:* Copied files from /etc/httpd/conf.d/ from the old content3.
:* Copied /home/dave from the old content3.
:* Copied /home/uesp from the old content3.
:* Created the directory /var/log/mysql/
:* Ran mysql_install_db
:* Created symbolic link: ln -s /tmp/mysql.sock /var/lib/mysql/mysql.sock
:* Disabled MySQL slow query log (was preventing the server from starting for some reason).
:* Changed ownership in /var/lib/mysql/ to mysql:mysql.
:* Started mysql: mysqld_safe --user=mysql
:* Ran mysql_secure_installation
:* Copied the latest full database dump from db1.
:* Ran setup script uesp-update-sshd (no issues)
:* Note: Make sure that /etc/init.d/nfslock is running.
:* Ran command: chkconfig --level 345 netfs on
:* Copied mysql tables from old content3.
:* Started zabbix_server.
:* Added zabbix_server to /etc/rc.local
:* Created link in /home/www/ to relevant web paths.
:* Changed Apache ServerName to *:80 and restarted.
:* Added date.timezone to php.ini and restarted Apache.
:* Changed host IPs in Zabbix

Files1 Migration
:* Turned off Wiki uploads.
:* Checked that all shared NFS paths on newfiles1 could be mounted by content1/2 and files1.
:* Confirmed that files could be locked on newfiles1, for example:
    cd /mnt/phpsessions
    su apache
    touch test
    flock test -c ls
::If NFS locking is working on the mount then the above script should immediately display the directory listing. Otherwise it will do nothing and eventually time out.
:* Mounted /mnt/phpsessions on content1/2 to newfiles1:/shared/phpsessions
:* Synced the phpsessions on newfiles1 from files1.
:* Changed the PHP session path on content1/2 to /mnt/phpsessions and restarted Apache.
:* Checked content1/2 to ensure pages were still loading correctly.
:* Moved /shared/uesp to /shared/uesp-old on files1.
:* Mounted UESP shares on files1 from newfiles1.
:* Unmounted files1 shares on content1/2.
:* Mounted newfiles1 shares on content1/2.
:* Changed DNS entries of maps/skins/images/files to the newfiles1.
:* Edited local hosts file to check that pages still load fine from the new server.
:* Enabled Wiki uploads.
:* Tested upload and file deletion.
:* Restarted lighttpd on files1.


Content2
:* Ran setup script uesp-setup-content content2 10.2.212.14 (didn't run uesp-setup-mounts automatically)
:* Ran setup script uesp-setup-mounts (no issues)
:* Manually started nfslock: /etc/init.d/nfslock start
:* Tested read/write/locking of file shares (no issues)
:* Changed password for user dave.
:* Copy UESP content files from old content2 into /home/uesp/www.
:* Create link from /home/www/uesp pointing to /home/uesp/www.
:* Create link from /home/www/uesp.net pointing to /home/uesp/www.
:* Install httpd: yum install httpd
:* Setup chkconfig: /sbin/chkconfig --level 345 httpd on
:* Re-ran setup script: uesp-install-contentapps (no issues)
:* Change permissions in UESP content path: chown -R apache:uespadmin /home/uesp/www/*
:* Copy/create/modify Apache configuration in /etc/httpd/ (old configuration broken apart into multiple files for easier editing).
:* Check Apache config: /etc/init.d/httpd configtest (re-enable several modules until no errors listed)
:* Add entry for server in /etc/hosts to ensure Apache binds to public IP address.
:* Reboot server to ensure everything comes back up.
:* Edit /etc/fstab and change nfs4 entries to nfs and reboot server (files1 shares still not mounted).
:* Run the command chkconfig --level 345 netfs on and reboot server (files1 shares now mounted).


Content2 Migration
:* Test the new content2 UESP wiki pages for read/write access.
:* Changed content2 entries in Squid to point to new IP.
:* Restarted Squid.
:* Changed DNS entries to point to new server.


Content1
:* Ran setup script uesp-setup-content content1 10.2.212.14 (no issues)
:* Set passwd for dave
:* Rebooted server (success)
:* Test read/write/lock of mounts (no issues)
:* Changed permissions for backup directory: chmod og+rx /home/backup
:* Changed permissions for uesp directory: chmod og+rx /home/uesp
:* Copied files from old content1 server
:* Copied UESP content into /home/uesp
:* Created links to /home/uesp/www/ in /home/www/
:* Copied Apache configuration from the new content2
:* Added entry for content1 in /etc/hosts
:* Restarted httpd (no issues)
:* Test web pages
::* /server-status OK
::* /wiki/Main_Page blank page
::* Changed ownership of UESP content: chown -R apache:uespadmin /home/uesp/www/*
::* /wiki/Main_Page OK
::* /wiki/Special:RecentChanges OK
::* Edit page OK
::* Benchmark pages OK
:* Copy other site configurations (eqwiki, ocp, forum, blog, dhack)
:* Copy other site contents
:* Update file/directory ownerships for other sites
:* Restart Apache
:* Test hosting of other sites on new content1

Content1 Migration
:* Test the new content1 UESP wiki pages for read/write access.
:* Changed content1 entries in Squid to point to new IP.
:* Restarted Squid.
:* Changed DNS entries to point to new server.
:* Changed DNS entries for other sites to point to new server (ocp, forum, blog, dhack, eqwiki).
:* Check Apache error log (no errors relevant to migration found).

Squid1
:* Changed main file system from ext3 to ext2.
:* Added noatime option to main file system.
:* Ran setup script uesp-setup-common squid1 10.2.212.14 (no issues)
:* Set password for dave
:* Installed Squid: yum install squid
:* Changed Squid initialization levels: /sbin/chkconfig --level 345 squid on 
:* Copied Squid configuration from oldsquid1
:* Added local host name to /etc/hosts
:* Modified configuration for new Squid server
::* Changed content1/2 host IPs 
::* Changed squid1 IP
::* Added visible_hostname squid1.uesp.net
::* Added cache_swap_log /var/log/squid/cache_swap_log
::* Modified cache_dir coss /var/spool/squid 50000 max-size=1000000 maxfullbufs=4 membufs=20 block-size=4096
::* Created swap directories: squid -z
:* Reboot system
::* Squid not started. Starting manually...failed.
::* System log message: Cannot open HTTP port
::* Modified http_port line to remove specific IP address: http_port 80 vhost defaultsite=www.uesp.net
::* Squid started successfully
::* Disabled Squid logging
::* Restarted Squid
:* Installed Apache: yum install httpd (for access to ab)
:* Test Squid cache with ab (no issues)
:* Installed PHP (for Zabbix custom scripts)

Squid1 Migration
:* Change DNS entries for uesp.net and www.uesp.net to new squid1.
:* Added new squid1 IP to content1/2/3 wiki configuration.


Db1
:* Ran setup script uesp-setup-common db1 10.2.212.14 (no issues)
:* Added entry for host in /etc/hosts
:* Ran setup script uesp-setup-db (no issues)
:* Reverted PHP session path to default (set to mounted share by previous step)
:* Ran MySQL install script mysql_install_db
:* Started MySQL /etc/init.d/mysqld start (failed)
:* Changed ownership of MySQL directory: chown -R mysql:mysql /var/lib/mysql
:* Started MySQL /etc/init.d/mysqld start (no issues)
:* Ran script mysql_secure_installation (no issues)
:* Set password for dave
:* Rebooted server (no issues)
:* Copied and restored database dumps from old db1.
:* Changed server-id to 142 and enabled bin-log. Restarted mysqld.
:* Setup and started slave replication on new db1.


Db1 Migration
:* Checked new db1 replication to ensure no errors.
:* Locked Wiki on content3.
:* Changed Wiki database host on content3 and tested page loads.
:* Locked Wiki on content1/2 and forums.
:* Stopped slave on new db1.
:* Tested page loads on content3.
:* Made test edit on content3.
:* Changed Wiki database host on content1/2 and tested page loads.
:* Change forum database host, re-enabled forum, and made test post.
:* Unlocked content1/2 and made test edits.
:* Changed database host on all secondary services: blog, DaveWiki, EQWiki, simap, obmap, mwmap
:* Shutdown MySQL on old db1.
:* Tested all services.
:* Changed db1 DNS entry.