Pixel/Setup server

== Install a fresh pixel server ==
Here's an example on how the server at https://horizon.wikimedia.org/ is setup.
=== Install Docker ===
We need to have Docker installed to be able to run pixel.
sudo apt update
sudo apt install apt-transport-https ca-certificates curl gnupg -y
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/debian bookworm stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce containerd.io docker-compose-plugin -y
=== Create a user for all pixel related work ===
We want to avoid using root, it's better to have specific user that we can switch to.
sudo adduser pixel
sudo usermod -aG sudo pixel
sudo echo "pixel ALL=(ALL:ALL) NOPASSWD:ALL" | sudo tee "/etc/sudoers.d/pixel"
Make sure the user can run Docker without sudo.
sudo su - pixel
sudo usermod -aG docker $USER
Disable login for the pixel user. Open the configuration: sudo nano /etc/ssh/sshd_config and add:
// Add the following line at the end, it need to be a tab between DenyUusers and pixel
DenyUsers	pixel
Restart the SSH process so pixel is locked out: sudo systemctl restart sshd
=== Setup two volumes to separate the data ===
We use two volumes, one for pixel data and one for Docker. 
Create the volumes at https://horizon.wikimedia.org. I created 45 gb for data and 35 for docker and attached them to the server.
Login to your server and check using lsblk. When I do it I can see the disks like this:
lsblk
> sdb 8:16 0 45G 0 disk 
> sdc 8:32 0 35G 0 disk 
Next step is to partion the disks
sudo fdisk /dev/sdb 
You will get multiple questions. Start by choosing n, then p and then choose the default on the rest. As a last step choose w to write. Do the same thing for the other disk sudo fdisk /dev/sdc.
Run lsblk and you will see something like this:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
|-sda14 8:14 0 3M 0 part 
`-sdb1 8:17 0 45G 0 part 
sdc 8:32 0 35G 0 disk 
`-sdc1 8:33 0 35G 0 part
Next step is to format the disks.
sudo mkfs.ext4 /dev/sdb1
sudo mkfs.ext4 /dev/sdc1
And then mount them. Create a mount point and mount them:
sudo mkdir /mnt/pixel-data
sudo mount /dev/sdb1 /mnt/pixel-data
sudo mkdir /mnt/docker
sudo mount /dev/sdc1 /mnt/docker
Now they are mounted. Next step is to make sure they are mounted after reboot. Here it is important to use the disks UUID so they will be mounted correct after a reboot.
Get the UUDI for each disk:
sudo blkid /dev/sdc1
sudo blkid /dev/sdb1
Then add the volumes to fstab: sudo nano /etc/fstab
And two new lines. Make sure to use your UUID and mount points:
UUID=ee39933b-5369-454e-aeee-c2cae69b3eec /mnt/pixel-data ext4 defaults,nofail,discard,noatime 0 2
UUID=df60bbbd-fada-45ae-8e1f-3c714c694e96 /mnt/docker ext4 defaults,nofail,discard,noatime 0 2
You can verify that everything in fstab is correct by running: sudo findmnt --verify --verbose
If it's correct, reload the fstab: sudo systemctl daemon-reload
Now the volumes should be there after a reboot. As a last step, make pixel the owners of the directories:
sudo chown pixel:pixel /mnt/pixel-data
sudo chown pixel:pixel /mnt/docker
And now as a last step, reboot and check if the mounts exists after the reboot: sudo reboot
=== Install NodeJS ===
The current version of Pixel runs using NodeJS so you need to install it:
sudo su - pixel
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
nvm install 20
=== Install crontab ===
All jobs are executed from the crontab: sudo apt-get install cron -y
=== Setup apache2 ===
Apache is used to serve the result: sudo apt-get install apache2 -y
Then configure it to serve the pixel reports: sudo nano /etc/apache2/sites-enabled/000-default.conf
At the top of the file, change the document root to the following:
DocumentRoot /mnt/pixel-data
And within the same virtual host block, add a redirect to the pixel index file:
<Directory /mnt/pixel-data>
 Options Indexes FollowSymLinks
 AllowOverride None
 Require all granted
</Directory>
Redirect 302 /index.html /reports/index.html
Restart apache: sudo systemctl restart apache2
=== Setup up logs ===
Setup up log rotation. First create a new home for our logs:
sudo mkdir /var/log/pixel
sudo chown -R pixel:pixel /var/log/pixel
Then setup the rotation: sudo nano /etc/logrotate.d/pixel
And add the following:
/var/log/pixel/pixel.log {
 size 20M
 rotate 2
 compress
 missingok
 notifempty
 create 0644 pixel pixel
}
And then do the same for the cleanup log: sudo nano /etc/logrotate.d/pixel-clean
And add:
/var/log/pixel/pixel-clean.log {
 size 20M
 rotate 2
 compress
 missingok
 notifempty
 create 0644 pixel pixel
}
=== Setup collectd ===
Send disk usage to Graphite (we should change this to Prometheus) using collectd.
Install: sudo apt-get install collectd collectd-utils
Disable plugins and then configure the df plugin. This is done a little differently depending on which disks and what collectd version you use.
Open the configuration file: nano /etc/collectd/collectd.conf
Remove all LoadPlugin and add:
LoadPlugin df
LoadPlugin write_graphite
<Plugin df>
 Device "/dev/sdc1"
 MountPoint "/mnt/pixel-data"
 ValuesPercentage true
 </Plugin>
<Plugin df>
 Device "/dev/sdb1"
 MountPoint "/mnt/docker"
 ValuesPercentage true
 </Plugin>
<Plugin df>
 Device "/dev/sda1"
 MountPoint "/"
 ValuesPercentage true
 </Plugin>
And follow the Graphite plugin setup in Performance/Synthetic testing/Bare metal#Install collectd
Then restart the service: sudo service collectd restart