Your Own Server

This page describes how I set up a web server and a database server on my Raspberry Pi computer.
For more information about running MX in a Linux environment please see MX on Linux.
For more information about Raspberry Pi computers see Raspberry Pi computer page.
For more information on log files that you might want to convert to database tables, please see MX Weather Diary, Legacy Weather Diary, daily summary log (dayfile.txt), Speciallog.txt, Standard log files, Extra Sensor Files, and Air Link Log.txt.
=Introduction=
This page records the experience of setting up a web server and database server on my computer.
Most web servers run on Linux, although you can still install them on a Microsoft Windows PC.
It is important to say that many different web servers exist, I have chosen to mention a few, and I have tried various ones. Since “apache2” is very well known, and proabably the most widely used server, I will mostly focus on that. Sometimes, you might choose a simple server that is what is described as light, it uses the minimum resources, but also probably offers the least functionality. In contrast, you might choose apache, because it offers the most functionality, and because it is used so widely there are many people who can help you.
=Web server and database server built into Cumulus MX=
==Web Server built into Cumulus MX==
The Admin Interface is a set of web pages that allow you to look at your weather data and change settings.
The MX code will create the necessary web server, it uses the Internet Protocol (ip) address of the computer that is running MX and a port (:8998 by default, but MX has an optional parameter to make it use a different port). If you are viewing the interface on the same computer as runs MX, “localhost” can replace the ip address.
As an aside, there are two different ways to specify ip addresses (IPv4 and IPv6), but this is not place to explain the difference.
==Databases built into Cumulus MX==
You do not need to know what is in these databases to use MX. I mention them here to explain MX has a built in database server.
Cumulus MX includes two SQLite databases;
#The first database was added by Steve Loft, but he never documented what the first table in that database was to be used for (see Cumulusmx.db). The second table in that database is used for Recent history.
#The second database is the Weather Diary documented here added by Mark Crossley. Also see in the support forum for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries. 
#* The Admin Interface provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).
The legacy Cumulus software used a XML file to store the weather diary.
===Reading content of SQLite databases===
Obviously, Cumulus MX can read and update both of these databases, but there are a number of software packages available that can be used outside MX to access these database files, and to minimise technical detail, two will be named as possible candidates, you will find others online.
On Microsoft Windows, install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above. 
In Linux use sudo apt install sqlitebrowser to install easy-to-use software to read/edit these databases.
You may need to change file permissions.
Note that sqlite databases use sequential file structure, not a relational database design. This implies for updates, rows are physically identified by row number (order of creation), (not by the primary key that SQL accesses use with relational database designs).
= Adding a web server and a database server for your own purposes =
MX has (optional) functionality that can upload to a web server, it also has (optional) functionality that can update one or more database tables on a database server.
Frequently, people will use an external web server. This might be one of the Wunderground, Windy, Awekas, WeatherCloud, PWSweathe, WOW, APRS, OpenWeatherMap, and WindGuru external sites that MX can automatically update to. It might be you have subscribed to a commercial web server (and optionally database server). that an external provider supplies to you to run either the 3.10.1 onwards default web site, or it might be for pages you or a third party have designed.
If you have a computer that is left running all the time without consuming a lot of electricity), you might want to add a web server and database server to that device. For example, see Raspberry_Pi_computer_page for a computer with low power consumption that a number of Cumulus users have used for their web server and database server. 
As with a commercial web server, you can use your own web server to run the default web pages available in the MX download package. Alternatively, you might want this option, if you are creating your own web pages, and want to try them out without exposure to the public over the internet.
You might select this option to save the subscription costs of a commercial web server (and optionally database server).
== All in one versus separate components ==
To get maximum benefit when using with MX, you need multiple components:
# web server – there are various options listed below; beware "apache2" is huge as it is very complicated, but secure; other options consume fewer resources, and may be easier to use, but they will also be more restrictive
# PHP script - from - this provides a way of controlling what SQL is run depending on other factors
# Database server - MySQL Community edition from [mysql.com] or MariaDB from .
# Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from which combines PHP and MySQL
There are a number of packages that combine all of these:
*If designed for Microsoft Windows and including Apache2, PHP, and either MariaDB or MySQL, the package is often described as ‘’’WAMP’’’
*If designed for UNIX (including Linux), then XAMP will include Apache2, PHP, and either MariaDB or MySQL
Obviously, there are other web servers, some come on their own, others come as a package.
I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.
This Wiki page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.
=== Install a web server ===
I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):
#You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space on your PC, install it following instructions at the official site (or one of many other web sites that give instructions for installing apache2 on Microsoft Windows).
#You might choose ‘’’Uniform Server’’’, which is extremely simple as it comes as a package including PHP script, and MariaDB database in one package from , although you may need to update to a newer release of PHP from and a newer phpMyAdmin from as Uniform Server tends to issue upgrades very rarely.
#You might choose to install the Abyss web server from [aprelium.com]
If you get components separately, then there is extra work to get them to work together, but you will find clear explanations on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).
I will next mention 3 possible web servers than can run on a Linux computer:
#You might choose nginx, as it is a quite popular web server for small computers (like the RPi), install this web server with sudo apt-get install nginx -y.
#You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space, install it using sudo apt install apache2 -y. 
#*You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
#* To add this use sudo apt install libapache2-mod-fcgid.
#You might choose lighttpd, as a web server that is designed to use as little space on your Linux computer as possible, install it with sudo apt-get install lighttpd -y.
For a Raspberry Pi, installing two of these is covered at 
=== Install PHP Hypertext Pre-processor ===
As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.
At time of updating this Wiki page, PHP 8.1 is stable version, PHP 8.2 is available to test, and PHP 9.0 is planned. Older versions 7.x, and 5.x are largely obsolete, although some web providers may still make them available.
* PHP is not the only script language available, but it is quite comprehensive 
** PHP can be used either in a fairly simple way by non-technical people 
** or in an object-oriented way for those more technical to achieve success with more complex scripts.
* To check which PHP modules have been installed by any command, type php -m. 
* To test that php is installed, type php -v into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
* If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least mysqli and mbstring. 
====Microsoft Windows - download/install PHP====
Straight forward, just down load a zip from from . You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.
*PHP is not the only script language available, but it is quite comprehensive 
**being able to be used either in a fairly simple way by non-technical people 
**or in an object-oriented way for those more technical to achieve success with more complex scripts.
*To check which PHP modules have been installed by the above command, type php -m. 
*If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least mysqli and mbstring. 
To test that php is installed, type php -v into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
====Linux operating systems - best repository====
The default repositories (see here) may not contain up-to-date versions of all software, or may contain some software that is not compatible with other software.
You can add another repository that generally has reasonably new PHP versions (a few weeks behind the official site) using
sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update
* On Linux, the simplest instruction to install it is sudo apt install php -y, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the respositories your computer has in its source list, so you may end up with the oldest!
* A better variation is to actually list the version we want:
sudo apt install phpV.x (where V is major version, and x is minor version e.g. 7.4 or 8.1)
** This may still not give us the modules we want, but it does control which version is obtained
* Therefore the best variation is to list both the modules we want, and the version we want (here "x" denotes minor version, adjust to what is latest stable (e.g. change to php8.1-common etc.):
sudo apt install -y php8.x-common php8.x-cli php8.x-json php8.x-xml php8.x-readline php8.x-mbstring php8.xmysql php8.x-curl php8.x-gd pdp8.x-psqlite php8.x-pdo -y
* The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
** Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't automatically integrate the PHP with your web server (if you are using apache 2 for web server, add "libapache2-mod-php8.x" to list); 
====Linux operating systems - standard repository====
* On Linux, the simplest instruction to install PHP is sudo apt install php -y, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the respositories your computer has in its source list, so you may end up with the oldest!
* A better variation is to actually list the version we want:
sudo apt install php7.4
** This still does not give us the modules we want, but it does control which version is obtained
*A still better variation is to list the components we want:
sudo apt install php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd php7.4-sqlite php7.4-pdo -y
** Add ”libapache2-mod-php7.4” as just part of what integrates apache 2 and PHP
** The included “php7.4-cli” is what allows us to run scripts in batch
** “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
** “php7.4-json” (see here and :Category:JSON_Files#Data_Transfer_Format and “php7.4-xml” (see this section) are used to handle particular file types
** “php7.4mysql” integrates php and MariaSQL/MySQL
** "php7.4-sqlite php7.4-pdo" are needed to access the weather diary or recent history databases
** “php7.4-curl” “php7.4-gd” may be needed as these are used by some third-party routines designed by Cumulus contributors
* There are advantages in using default repositories:
** installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.
===Creating a Home web page on your web server===
You may wish to create a index.php web page at (Linux Apache server) /var/www/html which is the web server root for browsing, or of course you may wish to copy or FTP here all your existing web pages. To view a php web page, go into a browser and type in a url with the same IPv4 address as you use for the admin interface, omit the port (:8998) and instead type in your web page name (e.g. //192.168.1.xy/index.php, where you need to determine digits that replace xy).
===Finding your PHP configuration file===
For sake of simplicity in this article, from now on I will assume the web server you installed is "apache2", change that segment (in the paths quoted below here), if you installed a different web server.
The web server main php.ini is found at /etc/php/m.n/apache2/php.ini (where 'm.n' depends on which PHP version was installed, that was found out in last sub-section). 
You may need to edit this file for example to specify where your include files are stored (if not in same folder as script with require/include). Replace "apache2" by "cli" in the path for the batch php.ini file that you may also need to edit.
To run a php script in batch, type php - f <file_name>. You can redirect the output by adding a greater than symbol and the destination file (i.e. > log_file) on the same terminal line. 
If you want the MX external commands to run a PHP script for you, use something like "sh" as the program to run (i.e. run shell script); and in the parameters something like
/home/pi/CumulusMX/MXutils/autoEOD.sh will run a script "autoEOD.sh" you have added to the "MXutils" folder. In that script, you put something like (assuming you have added a folder 'batch' with a subfolder 'log'):
#!/bin/sh
# This MX batch command file is initiated automatically by Cumulus MX software during last stage of processing the end of a meteorological day
echo "It stores feedback in log file CumulusMXDailyBatch (file name ends with day of month)"
sudo php -f the_path_and_filename_goes_here.php > /home/pi/CumulusMX/batch/log/DailyBatch_Day$(date +%d).log
=== Install MariaDB database ===
MySQL database software is controlled by Oracle and not made available for inclusion in Raspberry Pi repository.
MariaDB is an alternative that has largely similar command syntax so is likely to work with code (like MX) designed to work with MySQL. Since the MX developer (Mark Crossley) actually uses MariaDB, we can have plenty of confidence it is suitable. 
To install this database server, we type sudo apt install mariadb-server php-mysql -y.
Our database files will be stored at (Linux) /var/lib/mysql by default. Our MariaDB configuration is stored at /etc/mysql/mariadb.conf.d/50-server.cnf, and it is the datadir= entry that controls where the database files are stored, so edit that to store files somewhere else, but make sure the storage location has right ownership permissions. This Wiki page will not give you the full technical details, so only those with some technical knowledge do such changes.
=== Making your database secure ===
We need to assign passwords to control access to the database by typing into terminal sudo mysql_secure_installation. That brings up a screen:
# where we are asked to type current password for the root (as no password has yet been set, simply press Enter),
# next type Y to signify we are going to set a new password for root,
# next type in a new password that you will not forget,
# next as explained on the screen we are going to say whether users must select a user name as well as a password, type Y to Remove anonymous users
# next we have to decide whether we will only be logging into the database on our Pi (using Localhost) or we might be logging in remotely; type Y or N respectively, but if you choose N remember someone other than you might guess there is a root user and might guess the password you set,
# next we have another option of whether to retain or delete a test database, answer N or Y respectively, I would keep the test database for now as you can play with it and then remove it later,
# finally you type in another Y as that will Reload the privilege tables and ensure all is set up for your access to the database later.
The message, when the process successfully completes, is "Thanks for using MariDB".
===Getting web and database servers ready for use===
We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single root@localhost user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariaDB monitor where we can insert some SQL commands to create these two users. 
Most tools, like PhpMyAdmin, on first start up will ask for username (here I choose "admin") and password, thereafter the tool should use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it. 
The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the MySQL_settings in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).
Obviously, these names might not be what you use, but you can amend commands below accordingly. For each line with SQL, it must end with a semicolon (;) as shown. After you press "Enter" key you will get a response saying "OK" if you have remembered the semi-colon. You can actually use "\G" or "\g" instead of a semi-colon, but here we will keep it simple and stick to semi-colon.
The SQL lines have a prompt of a greater than symbol (>) while the command lines have a prompt showing current path. Note that "identified by" is followed by a password enclosed in single quotes.
sudo mysql --user=root --password=InitialPassword
create user admin@localhost identified by 'PhpMyAdminPassword';
create user weather@localhost identified by 'MXPassword';
grant all privileges on *.* to admin@localhost;
grant all privileges on *.* to weather@localhost;
FLUSH PRIVILEGES;
create database cumulus;
exit;
As I type this, Cumulus MX has no exception handling if the username and password defined in the settings do not exist in the database, therefore in this situation it will crash out (with message press Enter to close).
=== Installing Adminer, or PhpMyAdmin ===
ExportMySQL.exe and CumulusMX.exe (see Cumulus_MX#Executables for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.
This option is about installing a package to do the tasks that go with operating a database, that MX may not be able to do in the way you require.
====PhpMyAdmin====
PhpMyAdmin is one tool that can be used to manipulate your MySQL like database (that is the significance of the "My" bit in the middle of the tools's name).
You may like PhpMyAdmin as this offers:
* a graphical approach (you see a table on screen and navigate to the row or cell you want to work on)
* a SQL approach (you can try out any SQL here, before adding it to a script that you might use in a web page)
* a selection approach (you select a database, then select a table, then select an action)
WINDOWS:
#Download from official download page 
#You need to do some configuration (PLEASE EXPAND THIS). 
LINUX:
#The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
# Visit the phpMyAdmin official download page and look for the .tar.gz URL and download it using wget. 
# The following steps assume version 5.2.0, released in 2022 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
# Decide whether you just want English, or need another language
#* For English only, the wget command is sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-english.tar.gz
#* For other languages, the wget command is sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-all-languages.tar.gz
# Change to folder where we will extract our download, this could be cd /usr/share/ or some other directory perhaps on an extra storage drive
#* For English only, extract your download into the current folder using sudo tar xzf phpMyAdmin-5.2.0-english.tar.gz
#* For other languages, extract your download into a sub-folder of the current folder using sudo tar xzf phpMyAdmin-5.2.0-all-languages.tar.gz
# Create a folder to hold the download within the folder where we did our download sudo mkdir phpmyadmin_5.2.0
# Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
#* For English only, the move command is sudo mv phpMyAdmin-5.2.0-english/* ./phpmyadmin_5.2.0
#* For other languages, the move command is sudo mv phpMyAdmin-5.2.0-all-languages/* ./phpmyadmin_5.2.0
# There is some configuration to do, for simplicity here assume the destination directory ("./phpmyadmin_5.2.0") is actually "/usr/share/phpmyadmin" and that we will use '/var/lib/' and '/etc/' to store other directories used by phpmyadmin:
#* First sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php to edit (with nano) the vendor configuration
#** change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes define('TEMP_DIR', '/var/lib/phpmyadmin_5.2.0/tmp/');
#** change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes define('CONFIG_DIR', '/etc/phpmyadmin/');
#The following sequence of commands will
#* add the mysqli module to our php install, sudo phpenmod mysqli
#*will restart apache (change if you are using a different web server), sudo service apache2 restart
#*will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin, note the first path is to directory where we stored our phpmyadmin files, and the second path/file is the default web directory and the symbolic link file name, adjust these paths to what is needed for you.
#*will give the standard user (pi) ownership of the database files and the web pages: sudo chown -R pi:www-data /var/www/html/
# phpMyAdmin now has the files where it expects to see them, and should work
#* To log in requires a username (default "admin") and password (defined in your database set-up - see previous section)
#* The remaining configuration can be done within phpMyAdmin, follow cues on each screen as work through all configuration tabs.
If you have any difficulty, there are some instructions at https://devanswers.co/manually-upgrade-phpmyadmin/ and some at https://docs.phpmyadmin.net/en/latest/setup.html#quick-install
====adminer====
You may prefer to install a different tool, perhaps adminer that works using a drill down approach. A drill down approach is when you select the database, then select the table, then select the row, then select the column, then select the action. This logical step by step approach is a popular approach, but does not suit everyone. 
Install this drill down package with sudo apt install adminer (I leave you to work out the commands needed after that).
===Viewing web pages on our new web server===
You can view any index.php or PHPMyAdmin web page in your browser by prefixing the address with your Pi IPv4 address e.g. a URL like http://192.168.1.xyz/phpmyadmin where xyz is 2 or 3 digits you look up as mentioned before where FTP was described. If your Pi has both Ethernet and wireless connection, there will be two different values of xyz for you to choose one from. On first run of PhpMyAdmin, as already mentioned, you will see a PHP MyAdmin log-on page where you type username and password we have just set up.
===Populating your database tables on your Pi===
Cumulus MX has functionality to update database tables at one of several intervals:
# fixed SQL at real-time
# fixed SQL at standard interval
# fixed or custom SQL at end of day
# custom number of seconds
# custom number of minutes
MX now has ability to run SQL updates when you edit a log file, provided you use the predefined column and table names. MX also has the ability to store failed SQL and to try to run it again at the next rollover. This will fail again and again until MX is stopped, if the SQL is specified wrongly, but is designed to cope with correct SQL, that fails because the database server is down, and assumes the database server may be working again by next rollover.
The database tables can use the column names in the schema pre-defined by Cumulus MX for the predefined queries, or in a custom schema (where you specify the column names) if you specify the required SQL. The settings are all in SQL settings, so read that cross-reference to find out more.
You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the first 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.
There is an optional utility that can be installed. Type cd CumulusMX && sudo mono ExportToMySQL.exe parameter to run the executable. The mandatory parameter specifies the log file name to read (dayfile.txt or standard monthly log file).
# for each line in the log file the executable will try to insert a row in the database table
#*the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.
There is no way to use this executable for insert of past rows into the real-time table.
If you have been running Cumulus on another device previously and already have database tables, the next section explains how you can create SQL to export your old database and use that SQL to populate the new table.
===Transferring database tables to your Pi=== 
If you have been using Cumulus before (and already have a database) then you can use PhpMyAdmin on your old device to export out all the Cumulus tables as SQL in a zip file, FTP that zip file across to your Pi, then use PhpMyAdmin on your Pi to import that zip file. 
Providing you selected the right options for what SQL you created in your export, the import will contain SQL to create the tables and to insert all the rows into each Cumulus table on your Pi. Please note that there is a limit of around 1000 rows that can be imported in one action, so for your bigger tables, you will only create the table once, but you will do several Replace row actions (export from old device, then import to Pi) each transferring just one thousand rows, until the whole table is on the Pi. You may prefer to use "ExportMySQL.exe" as described in previous section to recreate your bigger tables from the Cumulus log files.
You might want to also export/import the tables in the PhpMyAdmin database (as they contain your preferences for each of your tables) in a separate zip, although these might need some extra transformations, as they are specific to a particular version of the database server (and the old database server version may not match the MariaDB version on your Pi). Some PhpMyAdmin tables do change for different versions of the tool, so that too may make export/import of its tables more complicated.
=== Restarting Web Server ===
After all these installs, we need to restart your web server (so it loads the PHP and MariaDB), if you have installed apache 2 do this by typing sudo systemctl restart apache2 or (if we just want the Apache configuration reloaded) by typing sudo systemctl reload apache2. Similar commands apply for other web servers.
We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.
Category:WebTips