The Missing Sysadmin Notes

This is just a place for me to store some of the less encountered notes on administering a server. Unless specified otherwise, most notes are for an Ubuntu Server. It will be updated continuously.

#### Consider using xip.io or nip.io during development (Wildcard DNS)

This way you can avoid editing your hosts file. Quite useful if you didn't own a domain name as well as you can use this temporarily in your Nginx/Apache server configs. Examples from xip.io site:

```
10.0.0.1.xip.io resolves to 10.0.0.1
www.10.0.0.1.xip.io resolves to 10.0.0.1
mysite.10.0.0.1.xip.io resolves to 10.0.0.1
foo.bar.10.0.0.1.xip.io resolves to 10.0.0.1
```
#### Consider vtop as an alternative to htop

https://parall.ax/blog/view/3071/introducing-vtop-a-terminal-activity-monitor-in-node-js

#### Consider setting the timezone to UTC for standardization

- https://serverfault.com/questions/191331/should-servers-have-their-timezone-set-to-gmt-utc
- http://yellerapp.com/posts/2015-01-12-the-worst-server-setup-you-can-make.html

```
sudo ln -sf /usr/share/zoneinfo/UTC /etc/localtime
```

#### Locale Issue on EC2 Ubuntu

When you first SSH into your EC2 instances, it is quite likely to see some locale error message. Here's how to resolve it.

- http://m-schmidt.eu/2013/03/29/fix-locale-setting-on-ubuntu-ec2-instances/
- https://stackoverflow.com/questions/12016318/aws-ec2-en-us-utf-8-issue

```
sudo apt install language-pack-en

# nano /etc/environment and add in these 2 lines
LANG=en_US.utf-8
LC_ALL=en_US.utf-8
```

#### Consider using Nginx mainline branch instead of the stable branch
The mainline branch is stable branch + bug fixes. Nginx maintainers recommend it over the stable branch actually.

- https://serverfault.com/questions/715049/what-s-the-difference-between-the-mainline-and-stable-branches-of-nginx

```
sudo add-apt-repository -y ppa:nginx/development
sudo apt update
sudo apt install -y nginx
```

#### Consider upgrading your RSA/RSA 1024 SSH keys to Ed25519

Your old keys may not be as secure anymore, time to upgrade

https://blog.g3rt.nl/upgrade-your-ssh-keys.html

```
# New Keys
ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_key_name -C you@company.com

# Upgrade Old keys
for keyfile in ~/.ssh/id_*; do ssh-keygen -l -f "${keyfile}"; done | uniq
```

#### Database

##### Consider using utf8mb4 character set in MySQL

UTF8mb4 fully supports Unicode and may even be better for security compared to utf8

https://mathiasbynens.be/notes/mysql-utf8mb4

##### MySQL Indexing

https://use-the-index-luke.com/

Indexes are usually added to fields that you use in the WHERE section of a query. Though sometimes the SELECT fields too.

When creating indexes, the columns order is important. A different order could meant a different in the query performance.

The more specific your indexes are, the better they tend to perform. But the more indexes you create, the more data it takes up in memory & disk space. It's a trade-off.

When a query is slow, first use EXPLAIN on that query to find out more information about it.

```
EXPLAIN SELECT * FROM users;
```

To show a table indexes

```
SHOW INDEXES FROM users;
```

##### Generate a more performant MySQL configuration file using Percona Tools

Just go to https://tools.percona.com/wizard, login or create an account and follow the step by step wizard instructions and download your new configuration file. Then compare against your current MYSQL server configuration file (/etc/mysql/mysql.conf.d/mysqld.cnf in Ubuntu) and combine the new parts you wanted.

Percona is the company that wrote the famous "High-Performance MySQL" book. Also check out their [Xtrabackup](https://www.percona.com/software/mysql-database/percona-xtrabackup) for backing up your database.

#### Redis

##### Getting the latest stable version of Redis

```
# For Ubuntu
sudo apt-add-repository -y ppa:chris-lea/redis-server
sudo apt update
sudo apt install -y redis-server
```

#### Apache

##### Installing the latest stable version of apache

```
# Ubuntu
sudo add-apt-repository -y ppa:ondrej/apache2
sudo apt update
sudo apt install -y apache2

# Note if you use ppa:ondrej/apache2, the default MPM is 'event' not 'prefork'
```

#### Consider HTML 5 Boilerplate Project Server Configs Recommendations

- https://github.com/h5bp/server-configs (Nginx, Apache, IIS, GAE.etc)

#### Don't forget your strace

strace is a powerful debugging tool when you don't know what's going on on a production server. It has served me well since learning it in my school days. However, I often hesitate to recommend it to co-workers given my position as a junior. It's so useful I bet pretty much every sysadmin knows already.

```
# Get a list of running php-fpm processes
ps ax | grep php-fpm

# strace on 1 of the pids in this case 12345 to see what system calls pid 12345 makes
strace -r -p 12345
```

#### Consider pigz instead of gzip

pigz is the parallel implementation of gzip. Next time you want to gzip files, consider installing pigz if you have multicores

```
apt install pigz
```

#### PHP Specific (PHP-FPM, Opcache.etc). May move to another post

##### Disable xDebug on production server

Although xDebug is definitely an awesome tool for debugging your PHP code, it has an impact on performance and should be disabled on the production server most of the time.

```
sudo phpdismod xdebug

# Reload PHP-FPM afterwards
sudo service php7.2-fpm reload
```
#### CentOS Specific

##### Consider installing EPEL, IUS and Webtatic repositories

"EPEL provides only software that is not in the official CentOS and Red Hat repositories, IUS provides newer versions of software (like MySQL and PHP) that exist in the official repositories." - Rackspace Support

For me, I personally prefer Webtatic repositories as they appears to update more frequently for me for Nginx, PHP and MySQL.

Instructions:
- https://support.rackspace.com/how-to/install-epel-and-additional-repositories-on-centos-and-red-hat/ (EPEL, IUS)
- https://webtatic.com/projects/yum-repository/