Repository: chr0mag/geoipsets (language: python)

Issue:
[ci] add nftables integration tests for dbip and maxmind
Both the _dbip_ and _maxmind_ datasets are updated weekly. It would be nice to have an integration test automatically run weekly that does the following:

- build _dbip_ and _maxmind_ `nft` sets for both _ipv4_ and _ipv6_
- load each set into _nftables_, use them as part of a rule, and start the service to ensure the sets are compatible

The following python-nftables bindings might help:
- https://ral-arturo.org/2020/11/22/python-nftables-tutorial.html
- https://github.com/aborrero/python-nftables-tutorial
- https://git.netfilter.org/nftables/tree/py/nftables.py

This library is included as part of the [Archlinux nftables package](https://archlinux.org/packages/extra/x86_64/nftables/), and [packaged separately in Ubuntu](https://packages.ubuntu.com/hirsute/python3-nftables). Executing the tests from Python would make it simple to include in the existing _python-tests_ as part of the `pytest` run. 

Alternatively a Bash script could be used to perform the same task.
[ci] add iptables integration tests for dbip and maxmind
Both the _dbip_ and _maxmind_ datasets are updated weekly. It would be nice to have an integration test automatically run weekly that does the following:

- build _dbip_ and _maxmind_ `ipsets` for both _ipv4_ and _ipv6_
- load each set into `iptables/ip6tables`, use them as part of a rule, and start the service to ensure the sets are compatible

The following python modules might help:
- https://github.com/ldx/python-iptables
- https://github.com/sanyi/ipsetpy

The `iptables` library is available on [Pypi](https://pypi.org/project/python-iptables/), but the ipsetpy module hasn't had a commit since 2017 and only supports Python3.4. Recent `ipset` versions have a [libipset](https://ipset.netfilter.org/libipset.man.html) library that could be used to build Python bindings.

Executing the tests from Python would make it simple to include in the existing python-tests as part of the pytest run.

Alternatively a Bash script could be used to perform the same task.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
new file mode 100644
index 0000000..fd2a27a
--- /dev/null
+++ b/.github/workflows/ci.yaml
@@ -0,0 +1,65 @@
+# This workflow will build all geoipsets, load each set into nftables/iptables and use the set in a rule
+
+name: ci
+
+on:
+  push:
+    branches: [ '*' ]
+  workflow_dispatch:
+  schedule:
+    - cron: '14 14 * * 3'
+
+jobs:
+  generate-geoipsets:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+      - name: Generate geoipsets.conf
+        working-directory: ./scripts
+        env:
+          MAXMIND_KEY: ${{ secrets.MAXMIND_KEY }}
+        run: |
+          bash generate_geoipsets_conf.sh
+      - name: Build geoipsets
+        run: |
+          python -m geoipsets --config-file /tmp/geoipsets.conf
+          if [ "$?" -ne 0 ]; then exit 1; fi
+          ls -R /tmp/geoipsets
+        working-directory: ./python
+      - name: Upload geoipsets data
+        uses: actions/upload-artifact@v4
+        with:
+          name: geoipsets-data
+          path: /tmp/geoipsets
+
+  run-integration-tests:
+    needs: generate-geoipsets
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        provider: [ dbip, maxmind ]
+        firewall: [ nftset, ipset ]
+        address-family: [ ipv4, ipv6 ]
+    steps:
+      - name: Install dependencies
+        run: sudo apt-get install -y ipset jq
+      - name: Download geoipsets data
+        uses: actions/download-artifact@v4
+        with:
+          name: geoipsets-data
+          path: /tmp/geoipsets
+      - uses: actions/checkout@v4
+      - name: Run integration tests
+        working-directory: ./scripts
+        run: |
+          sudo bash integration_test.sh ${{ matrix.provider }} ${{ matrix.firewall }} ${{ matrix.address-family }}
+          if [ "$?" -ne 0 ]; then exit 1; fi
diff --git a/README.md b/README.md
index 4d17d73..7e20f92 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 geoipsets
 ============
-![badge](https://github.com/chr0mag/geoipsets/actions/workflows/python-tests.yaml/badge.svg) ![PyPI](https://img.shields.io/pypi/v/geoipsets) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/geoipsets) [![Downloads](https://pepy.tech/badge/geoipsets)](https://pepy.tech/project/geoipsets) ![GitHub](https://img.shields.io/github/license/chr0mag/geoipsets)
+![badge](https://github.com/chr0mag/geoipsets/actions/workflows/ci.yaml/badge.svg) ![PyPI](https://img.shields.io/pypi/v/geoipsets) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/geoipsets) [![Downloads](https://pepy.tech/badge/geoipsets)](https://pepy.tech/project/geoipsets) ![GitHub](https://img.shields.io/github/license/chr0mag/geoipsets)
 
 Utility to generate country-specific IPv4/IPv6 network ranges consumable by both *iptables/ipset* and *nftables*. Also included is a *systemd* service and timer to periodically update the IP sets.
 
diff --git a/python/README.md b/python/README.md
index 72c3ef5..d332848 100644
--- a/python/README.md
+++ b/python/README.md
@@ -1,4 +1,4 @@
-![badge](https://github.com/chr0mag/geoipsets/actions/workflows/python-tests.yaml/badge.svg) ![PyPI](https://img.shields.io/pypi/v/geoipsets) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/geoipsets) [![Downloads](https://pepy.tech/badge/geoipsets)](https://pepy.tech/project/geoipsets) ![GitHub](https://img.shields.io/github/license/chr0mag/geoipsets)
+![badge](https://github.com/chr0mag/geoipsets/actions/workflows/ci.yaml/badge.svg) ![PyPI](https://img.shields.io/pypi/v/geoipsets) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/geoipsets) [![Downloads](https://pepy.tech/badge/geoipsets)](https://pepy.tech/project/geoipsets) ![GitHub](https://img.shields.io/github/license/chr0mag/geoipsets)
 
 Installation
 ------------
diff --git a/scripts/generate_geoipsets_conf.sh b/scripts/generate_geoipsets_conf.sh
new file mode 100644
index 0000000..34c5615
--- /dev/null
+++ b/scripts/generate_geoipsets_conf.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+cat << EOF > /tmp/geoipsets.conf
+[general]
+# list of providers from which to acquire IP ranges
+# options are:
+# 'maxmind': www.maxmind.com
+# 'dbip': https://db-ip.com/
+provider=maxmind,dbip
+
+# list of firewalls to build sets for
+# valid values are: 'iptables', 'nftables'
+# iptables: builds 'ipset' compatible sets
+# nftables: builds nftables compatible sets
+# if the property doesn't exist, or exists but is empty both ip and nft sets are generated (default)
+firewall=iptables,nftables
+
+# list of IP protocols to build sets for
+# valid values are: 'ipv4', 'ipv6'
+# if the property doesn't exist or exists, but is empty both ipv4 and ipv6 sets will be generated (default)
+address-family=ipv4,ipv6
+
+# specify which countries to build sets for
+# countries are specified using the 2-character country codes, one per line
+# https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
+# if section doesn't exist, or exists but is empty, sets for all countries will be generated (default)
+[countries]
+#RU
+#CN
+#KP
+#BV
+
+[maxmind]
+# specify MaxMind license key needed to download data
+# required for provider type 'maxmind', ignored by other provider types
+license-key=${MAXMIND_KEY}
+EOF
\ No newline at end of file


Test patch:
diff --git a/.github/workflows/python-tests.yaml b/.github/workflows/python-tests.yaml
index 74f40c0..62e10aa 100644
--- a/.github/workflows/python-tests.yaml
+++ b/.github/workflows/python-tests.yaml
@@ -1,6 +1,6 @@
 # This workflow will install Python dependencies, run tests and lint with a variety of Python versions
 
-name: ci
+name: python-tests
 
 on:
   push:
@@ -9,19 +9,16 @@ on:
     branches: [ 'main' ]
 
 jobs:
-  build:
-
+  run-python-tests:
     runs-on: ubuntu-latest
-
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.8', '3.9', '3.10']
-
+        python-version: ['3.8', '3.9', '3.10', '3.11']
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
diff --git a/scripts/integration_test.sh b/scripts/integration_test.sh
new file mode 100644
index 0000000..14260ba
--- /dev/null
+++ b/scripts/integration_test.sh
@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+
+# usage: ./integration_test.sh <provider> <firewall> <address_family>
+# eg. ./integration_test.sh dbip nftset ipv4
+# assume default set location under /tmp/geoipsets/.
+# possible paths are:
+#% ls -lRd /tmp/geoipsets/*/*/*
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/dbip/ipset/ipv4
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/dbip/ipset/ipv6
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/dbip/nftset/ipv4
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/dbip/nftset/ipv6
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/maxmind/ipset/ipv4
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/maxmind/ipset/ipv6
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/maxmind/nftset/ipv4
+#drwxr-xr-x 2 root root 4096 Dec 28 14:19 /tmp/geoipsets/maxmind/nftset/ipv6
+
+# print an error and exit with failure
+# $1: error message
+function error() {
+  echo "$0: error: $1" >&2
+  exit 1
+}
+
+# ensure the programs needed to execute are available
+function check_progs() {
+  local PROGS="jq ipset nft iptables ip6tables"
+  which ${PROGS} > /dev/null 2>&1 || error "Searching PATH fails to find executables among: ${PROGS}"
+}
+
+function test_nftables() {
+  set_path="/tmp/geoipsets/${1}/${2}/${3}"
+  printf "Set location: %s.\n" "$set_path"
+  set_list=("$set_path"/*."$3")
+  printf "Sets to test: %s.\n" "${#set_list[@]}"
+
+  for s in "${set_list[@]}"; do
+    printf "Set: %s\t\t" "$s"
+    file_items=$(($(wc --lines < "$s") - 2))  # first & last lines are not subnets
+    set_name=$(basename "$s")
+    if [ "$3" = "ipv6" ]; then
+      ip_version="ip6"
+    else
+      ip_version="ip"
+    fi
+
+cat << EOF > /tmp/nftables.conf
+#!/usr/bin/nft -f
+
+# clear all prior state
+flush ruleset
+
+include "${s}"
+
+table inet filter
+delete table inet filter
+
+# IPv4/IPv6 filter table
+table inet filter {
+  set blacklist {
+    type ${3}_addr
+    flags interval
+    elements = \$$set_name
+  }
+  chain input {
+    type filter hook input priority 0; policy accept;
+    $ip_version saddr @blacklist counter drop
+  }
+}
+EOF
+
+    nft --file /tmp/nftables.conf
+    nft_ret_val=$?
+    loaded_items=$(nft --json list set inet filter blacklist | jq '.nftables[1].set.elem' | jq 'length')
+    printf "entries: %s\t loaded: %s\t\t" "$file_items" "$loaded_items"
+    if [ "$nft_ret_val" -eq 0 ] && [ "$file_items" = "$loaded_items" ]; then
+      printf "pass\n"
+    else
+      printf "fail\n"
+      error "Set '${s} failed to load."
+    fi
+  done
+}
+
+function test_ipset() {
+  # support running on both Arch & Ubuntu (22.04)
+  source "/etc/os-release"
+  if [ "$ID" = "arch" ]; then
+    iptables --table filter --flush
+    ip6tables --table filter --flush
+    if [ "$3" = "ipv4" ]; then
+      ipt_binary="iptables"
+    else
+      ipt_binary="ip6tables"
+    fi
+  elif [ "$ID" = "ubuntu" ]; then
+    iptables-legacy --table filter --flush
+    ip6tables-legacy --table filter --flush
+    if [ "$3" = "ipv4" ]; then
+      ipt_binary="iptables-legacy"
+    else
+      ipt_binary="ip6tables-legacy"
+    fi
+  fi
+  printf "Using iptables binary: %s\n" "$ipt_binary"
+  ipset destroy
+
+  set_path="/tmp/geoipsets/${1}/${2}/${3}"
+  printf "Set location: %s.\n" "$set_path"
+  set_list=("$set_path"/*."$3")
+  printf "Sets to test: %s.\n" "${#set_list[@]}"
+
+  for s in "${set_list[@]}"; do
+    printf "Set: %s\t\t" "$s"
+    file_items=$(($(wc --lines < "$s") - 1))  # first line is not a subnet
+    set_name=$(basename "$s")
+    ipset restore --file "$s"
+    ipset_ret_val=$?
+    loaded_items=$(ipset list --terse | grep "Number of entries" | awk '{print $4}')
+    printf "entries: %s\t loaded: %s\t\t" "$file_items" "$loaded_items"
+    $ipt_binary --table filter --insert INPUT --match set --match-set "$set_name" src -j DROP
+    iptables_ret_val=$?
+    if [ "$ipset_ret_val" -eq 0 ] && [ "$iptables_ret_val" -eq 0 ] && [ "$file_items" = "$loaded_items" ]; then
+      printf "pass\n"
+    else
+      printf "fail\n"
+      error "Set '${s} failed to load."
+    fi
+    $ipt_binary --table filter --flush
+    ipset destroy
+
+  done
+}
+
+function main() {
+  check_progs
+  if [ "$2" = "nftset" ]; then
+    test_nftables "$@"
+  else
+    test_ipset  "$@"
+  fi
+}
+
+main "$@"
+
+