npm package: @polymech/gadm
Description: Pure TypeScript interface to the GADM v4.1 administrative boundaries database.
Latest version: 1.0.0

# @polymech/gadm

![npm version](https://img.shields.io/npm/v/@polymech/gadm) 
![TypeScript](https://img.shields.io/badge/TypeScript-Strict-%233178C6?logo=typescript)

**[Homepage](https://service.polymech.info/)** &nbsp;&middot;&nbsp; **[Source Code](https://git.polymech.info/polymech/gadm-ts)**

Pure TypeScript interface to the [GADM](https://gadm.org) v4.1 administrative boundaries database.  
Zero Python dependencies — parquet data, tree construction, iterators, and caching all run in Node.js.

## Overview

| Feature | Description |
|---------|-------------|
| **Database** | 356K rows from GADM 4.1, stored as a 6 MB Parquet file |
| **Admin Levels** | L0 (country) → L5 (municipality/commune) |
| **Tree API** | Build hierarchical trees, walk with DFS/BFS/level iterators |
| **Name Search** | Fuzzy search across all levels with Levenshtein suggestions |
| **GeoJSON** | Fetch boundaries from GADM CDN with corrected names |
| **Caching** | File-based JSON cache for trees and API results |
| **VARNAME** | Alternate names / English translations via `VARNAME_1..5` columns |

![PoolyPress GADM based picker](./docs/gadm-inspector.png)


---

## Installation

```bash
npm install @polymech/gadm
```

Internal monorepo — referenced via workspace protocol in `package.json`.

---

## Acknowledgments & PyGADM Port

This package is a direct Node.js/TypeScript port of the excellent Python library [pygadm](https://github.com/gee-community/pygadm) (which powers the core parquet-based data structure and fetching methodology). 

While bringing these capabilities natively to the javascript ecosystem, we built and added several critical enhancements designed specifically for web applications and browser performance:

- **Aggressive Geometry Simplification:** Natively integrates `@turf/simplify` (0.001 tolerance, high-quality) to instantly compress raw unoptimized 25MB boundary polygons down to ~1MB browser-friendly payloads.
- **Unified Cascading Caches:** Intelligent caching ladders that auto-resolve across global `process.env.GADM_CACHE`, active `process.cwd()`, and local workspace `../cache` mounts.
- **Target-Level Subdivision Extraction:** A unified `targetLevel` API design that distinctly differentiates between extracting an outer merged geographic perimeter vs. an array of granular inner subdivided states natively derived from recursive `.merge()` operations.
- **Smart Pre-cacher Script:** Includes `boundaries.ts`, an auto-resuming build script that iterates downwards to pre-calculate, dissolve, and aggressively compress hierarchy layers 0–5 for instant sub-ms API delivery, bypassing heavy mathematical geometry intersections at runtime.

---

## Quick Start

```ts
import { buildTree, walkDFS, findNode, searchRegions, getNames } from '@polymech/gadm';

// Build a tree for Spain
const tree = await buildTree({ admin: 'ESP', cacheDir: './cache/gadm' });
console.log(tree.root.children.length); // 18 (comunidades)

// Find a specific region
const bcn = findNode(tree.root, 'Barcelona');
console.log(bcn?.gid); // ESP.6.1_1

// Walk all nodes
for (const node of walkDFS(tree.root)) {
    console.log('  '.repeat(node.level) + node.name);
}

// Search via wrapper API
const result = await searchRegions({ query: 'France', contentLevel: 2 });
console.log(result.data?.length); // ~101 departments
```

---

## API Reference

### Tree Module

#### `buildTree(opts: BuildTreeOptions): Promise<GADMTree>`

Builds a hierarchical tree from the flat parquet data. Results are cached to disk when `cacheDir` is set.

```ts
interface BuildTreeOptions {
    name?: string;      // Region name: "Spain", "Cataluña", "Bayern"
    admin?: string;     // GADM code: "ESP", "DEU.2_1", "FRA.11_1"
    cacheDir?: string;  // Path for JSON cache files (optional)
}
```

Either `name` or `admin` must be set (not both).  
Throws if the region is not found in the database.

#### `GADMTree` and `GADMNode`

```ts
interface GADMTree {
    root: GADMNode;     // Root node of the tree
    maxLevel: number;   // Deepest admin level reached (0–5)
    nodeCount: number;  // Total nodes across all levels
}

interface GADMNode {
    name: string;           // Display name: "Barcelona"
    gid: string;            // GADM ID: "ESP.6.1_1"
    level: number;          // Admin level 0–5
    children: GADMNode[];   // Sub-regions (sorted alphabetically)
}
```

#### Iterators

All iterators are generators — use `for...of` or spread into arrays.

| Function | Description |
|----------|-------------|
| `walkDFS(node)` | Depth-first traversal, top-down |
| `walkBFS(node)` | Breadth-first, level by level |
| `walkLevel(node, level)` | Only nodes at a specific admin level |
| `leaves(node)` | Only leaf nodes (deepest, no children) |
| `findNode(root, query)` | First DFS match by name or GID (case-insensitive) |

```ts
// Get all provinces (level 2) under Cataluña
const provinces = [...walkLevel(tree.root, 2)];
// → [{ name: 'Barcelona', ... }, { name: 'Girona', ... }, ...]

// Count municipalities
const municipios = [...leaves(tree.root)];
console.log(municipios.length); // 955

// Find by GID
const girona = findNode(tree.root, 'ESP.6.2_1');
```

---

### Names Module

#### `getNames(opts: NamesOptions): Promise<NamesResult>`

Searches the parquet database for admin areas. Returns deduplicated rows with fuzzy match suggestions on miss.

```ts
interface NamesOptions {
    name?: string;          // Search by name
    admin?: string;         // Search by GADM code
    contentLevel?: number;  // Target level (0–5), -1 = auto
    complete?: boolean;     // Return all columns up to contentLevel
}

interface NamesResult {
    rows: GadmRow[];    // Matched records
    level: number;      // Resolved content level
    columns: string[];  // Column names in result
}
```

On miss, throws with Levenshtein-based suggestions:
```
The requested "Franec" is not part of GADM.
The closest matches are: France, Franca, Franco, ...
```

---

### Items Module

#### `getItems(opts: ItemsOptions): Promise<GeoJSONCollection>`

Fetches GeoJSON boundaries from the GADM CDN, with name correction from the local parquet database (workaround for camelCase bug in GADM GeoJSON responses).

```ts
interface ItemsOptions {
    name?: string | string[];   // Region name(s)
    admin?: string | string[];  // GADM code(s)
    contentLevel?: number;      // Target level, -1 = auto
    includeOuter?: boolean;     // Also include the containing region's external perimeter
    geojson?: boolean;          // Return geometries instead of just properties (metadata)
}
```

Supports continent expansion: `getItems({ name: ['europe'] })` fetches all European countries.

---

### Wrapper Module (Server API)

Higher-level API designed for HTTP handlers. Includes file-based caching via `GADM_CACHE` env var (default: `./cache/gadm`).

| Function | Description |
|----------|-------------|
| `searchRegions(opts)` | Search by name, returns metadata or GeoJSON |
| `getBoundary(gadmId, contentLevel?, cache?)` | Get GeoJSON boundary for a GADM ID |
| `getRegionNames(opts)` | List sub-region names with depth control |

#### Integration Example (Server API)

Here is a real-world example of wrapping the GADM engine inside an HTTP handler (like Hono or Express) to fetch dynamically chunked boundaries and enrich their GeoJSON metadata on the fly:

```ts
import { getBoundary } from '@polymech/gadm';
import * as turf from '@turf/turf';

async function handleGetRegionBoundary(c) {
    const id = c.req.param('id'); // e.g. "DEU" or "ESP.6_1"
    const targetLevel = c.req.query('targetLevel'); // e.g. "1" for inner states 
    const enrich = c.req.query('enrich') === 'true';

    try {
        const parsedTargetLevel = targetLevel !== undefined ? parseInt(targetLevel) : undefined;
        
        // Instantly fetches Boundary FeatureCollection (already cached and compressed)
        const result = await getBoundary(id, parsedTargetLevel);

        if ('error' in result) {
            return c.json({ error: result.error }, 404);
        }

        // On-the-fly Geometry Enrichment
        if (enrich && result.features) {
            for (const feature of result.features) {
                // Calculate geographical square kilometers organically using Turf
                const areaSqkm = Math.round(turf.area(feature as any) / 1000000);
                feature.properties.areaSqkm = areaSqkm;
                
                // Construct bounding box for client camera tracking
                const bbox = turf.bbox(feature as any);
                feature.properties.bbox = bbox;
            }
        }

        return c.json(result, 200);

    } catch (error) {
        return c.json({ error: error.message }, 500);
    }
}
```

---

## Boundary Geometries & Caching

Fetching complex geospatial polygons (like country borders or district subdivisions) requires merging and calculating hundreds of complex geometries. Doing this mathematically at runtime for a user request is too slow, so `@polymech/gadm` handles this with pre-compiled caches and aggressive size compression.

### Resolving Boundary Target Levels

When building interactive user interfaces or fetching boundaries through the top-level API (`handleGetRegionBoundary`), the returned `FeatureCollection` granularity is controlled strictly through the `targetLevel` (or programmatic `contentLevel`).

- **Outer Boundary**: Set `targetLevel` exactly equal to the region's intrinsic level (e.g., Targetting `Level 0` for Spain). The engine uses `turf` to automatically dissolve internal geometries, returning a single merged bounding polygon mimicking the total region envelope.
- **Inner Subdivisions**: Provide a `targetLevel` deeper than the intrinsic level (e.g., Targetting `Level 1` for Spain). The engine filters for the exact constituent parts and returns a `FeatureCollection` where each active sub-group (the 17 Spanish States) is a distinctly preserved geometry feature.

### Smart Caching & Geometry Compression

To ensure instantaneous delivery (sub-10ms) of these polygons to your HTTP APIs:

1. **Pre-Caching Script**: You can run `npm run boundaries -- --country=all` manually. This script iterates downwards to compute and compress hierarchical layers 0 through 5 for each invoked country automatically. If a file already exists, it intelligently skips it allowing easy resume.
2. **Cascading Cache Lookups**: The package resolves caches intelligently by searching matching environments: First checking `process.env.GADM_CACHE` (used by explicit production APIs), then `process.cwd()/cache/gadm`, and finally its own local `../cache/gadm` workspace path. You can also actively inject an external Redis or memory cache interface providing `{ get(key), set(key, val) }` as the 3rd argument to `getBoundary`.
3. **Payload Compression (~25MB -> ~1MB)**: Boundary geometries are natively compressed using `@turf/simplify` (with `0.001` tolerance mapping to ~100m geospatial fidelity) immediately prior to caching, ensuring React mapping payloads stay tightly optimized and smooth without crashing browsers.

---

### Database Module (Low-Level)

| Function | Description |
|----------|-------------|
| `loadDatabase()` | Load parquet into memory (lazy, singleton) |
| `getColumns()` | Return column names |
| `resetCache()` | Clear the in-memory row cache |

`GadmRow` is `Record<string, string>` — all values normalized to strings.

---

## Types

All types are exported from the package entry point:

```ts
import type {
    GADMNode, GADMTree, BuildTreeOptions,     // tree
    NamesOptions, NamesResult, GadmRow,       // names + database
    ItemsOptions, GeoJSONFeature, GeoJSONCollection,  // items
    SearchRegionsOptions, SearchRegionsResult, RegionNamesOptions,  // wrapper
} from '@polymech/gadm';
```

---

## Data Layout

### Parquet File

`data/gadm_database.parquet` — **356,508 rows**, **6.29 MB**

| Column Group | Columns | Description |
|--------------|---------|-------------|
| GID | `GID_0` … `GID_5` | GADM identifiers per level |
| NAME | `NAME_0` … `NAME_5` | Display names per level |
| VARNAME | `VARNAME_1` … `VARNAME_5` | Alternate names / translations |

129,448 rows have `VARNAME_1` values (e.g. `Badakhshān`, `Bavière`).

### GADM Levels

| Level | Typical Meaning | Example (Spain) |
|-------|----------------|-----------------|
| 0 | Country | Spain |
| 1 | State / Region | Cataluña |
| 2 | Province / Department | Barcelona |
| 3 | District / Comarca | Baix Llobregat |
| 4 | Municipality | Castelldefels |
| 5 | Sub-municipality | *(rare, not all countries)* |

> **Note:** GADM does not include neighborhood/Stadtteil-level data.  
> For sub-city resolution (e.g. Johannstadt in Dresden), OSM/Nominatim would be needed.

---

## Caching

### Tree Cache (`cacheDir`)

When `cacheDir` is passed to `buildTree()`, the full tree is saved as `tree_{md5}.json`.  
Subsequent calls with the same `name`/`admin` return the cached tree instantly (~1ms).

### Wrapper Cache (`GADM_CACHE`)

The wrapper module caches search results, boundaries, and region names in `$GADM_CACHE/` (default `./cache/gadm`).  
Files are keyed by MD5 hash of the query parameters.

### In-Memory Cache

`loadDatabase()` is a singleton — the 356K-row array is loaded once per process.  
Call `resetCache()` to force a reload (useful in tests).

### Precalculating Boundaries

To improve runtime performance (especially for large geographies which take time to dissolve), you can precalculate and cache standard admin boundaries using the included CLI script:

```bash
cd packages/gadm

# Precalculate the outer boundary for a specific country
npm run boundaries -- --country=DEU

# Precalculate inner boundaries for a specific level
npm run boundaries -- --country=DEU --level=1

# Precalculate the outer boundary for ALL countries worldwide
npm run boundaries -- --country=all
```

Precalculated boundaries are saved as native `.json` artifacts inside the configured cache directory (`./cache/gadm/boundary_{md5}.json`).

---

## Data Refresh

Regenerate `data/gadm_database.parquet` from a GADM GeoPackage source file.

### Prerequisites

Download one of the core GeoPackage database files. You can point the package to your `gpkg` location using the `GADM_GPKG_PATH` environment variable, or store it in your working directory at `cache/gadm/gadm_410.gpkg`:

```bash
https://geodata.ucdavis.edu/gadm/gadm4.1/gadm_410-gpkg.zip  → unzip → gadm_410.gpkg
https://geodata.ucdavis.edu/gadm/gadm4.1/gadm_410-raw.gpkg
```

### Run

```bash
cd packages/gadm
npm run refresh
```

The script (`scripts/refresh-database.ts`):
1. Opens the GeoPackage (SQLite) via `better-sqlite3`
2. Auto-detects table format (per-level `ADM_x` tables or single flat table)
3. Extracts GID, NAME, and VARNAME columns for levels 0–5
4. Writes to `data/gadm_database.parquet` via `hyparquet-writer`

### Dev Dependencies (refresh only)

| Package | Purpose |
|---------|---------|
| `better-sqlite3` | Read GeoPackage (SQLite) files |
| `hyparquet-writer` | Write Parquet output |

These are `devDependencies` — not needed at runtime.

---

## Tests

```bash
cd packages/gadm
npx vitest run                              # all tests
npx vitest run src/__tests__/tree.test.ts   # tree tests only
```

### Tree Tests

JSON outputs saved to `tests/tree/` for inspection:

| File | Content |
|------|---------|
| `test-cataluna.json` | Full Cataluña tree (1,000 nodes, 955 leaves) |
| `test-germany-summary.json` | Germany L1 summary (16 Bundesländer, 16,402 nodes) |
| `test-dresden.json` | Sachsen → Dresden subtree with all children |
| `test-iterators.json` | DFS/BFS/walkLevel/findNode verification data |

### Name Tests

`src/__tests__/province-names.test.ts` — tests `getNames()` for France departments, exact matches, fuzzy suggestions.

---

## Architecture

```
packages/gadm/
├── data/
│   ├── gadm_database.parquet     # 356K rows, 6.29 MB
│   └── gadm_continent.json       # Continent → ISO3 mapping
├── scripts/
│   └── refresh-database.ts       # GeoPackage → Parquet converter
├── src/
│   ├── database.ts               # Parquet reader (hyparquet)
│   ├── names.ts                  # Name/code lookup + fuzzy match
│   ├── items.ts                  # GeoJSON boundaries from CDN
│   ├── wrapper.ts                # Server-facing API with cache
│   ├── tree.ts                   # Tree builder + iterators
│   ├── index.ts                  # Barrel exports
│   └── __tests__/
│       ├── tree.test.ts          # Tree building + iterator tests
│       └── province-names.test.ts
├── tests/
│   ├── tree/                     # Test output JSONs
│   └── cache/gadm/               # Tree cache files
└── package.json
```

## Dependencies

| Package | Type | Purpose |
|---------|------|---------|
| `hyparquet` | runtime | Read Parquet files (zero native deps) |
| `zod` | runtime | Schema validation |
| `better-sqlite3` | dev | GeoPackage reader (refresh only) |
| `hyparquet-writer` | dev | Parquet writer (refresh only) |
| `vitest` | dev | Test runner |
| `typescript` | dev | Build |