Creating vector tiles from Esri Feature Service using nodejs and tippecanoe

# Introduction
This article is written for my colleague M.L. and G.L.S. This article explains how I made vector tiles from Esri Feature services.

Originally, this method was developed in December 2022 and that effort was summrized in this article (in Japanese): https://qiita.com/T-ubu/items/17cb65754a54aa5ffb21

If you have the data in GeoJSON, just use it because it is much easy. This is a work flow for a case that you only have an access to the data as FeatureService.

This is a part of my work as activities of DWG7 under the UN Open GIS.


# Environment

My working einvironment;
* nodejs version 18
* Windows 10

I like working in the linux server, however, for demonstration purpose, I will use docker so that I can show it in the windows PC.

* Docker (unvt/nanban)
    * nodejs version 18
    * felt/tippecanoe


Targeted Feature service:
* Overview URL: https://geoportal.un.org/arcgis/home/item.html?id=f9acac44779f400985a7a04f4d0c68ac
* FeatureServer URL: https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer?f=pjson
* Targeted layers:
  * Layer 0: BNDL https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0?f=pjson
  * Layer 3: BNDA https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/3?f=pjson
* Features in a layer (example):
  * BNDL: https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0/1?f=pjson
  * BNDA: https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/3/1?f=pjson

If you read ArcGIS REST API, we can understand where we can get information of a layer (or a feature in a layer). Unfortunately, it seems that EsriFeature service does not provide all features in a layer as GeoJSON unless a GeoJSON file is provided as an item in that server.


# Procedure
## Step 1. check the number of features manually
Because the server does not provide whole feature of a layer as a single file, we first will check the number of each layer.

For the layer 0 (BNDL), check the following URL for example.
* https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0/0?f=pjson
* https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0/1?f=pjson
* ...
* https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0/242?f=pjson
* https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/UN_Geodata_Stylized/FeatureServer/0/743?f=pjson

You will find the data exists from 0/1 to 0/742, meaning that there are 742 features in the BNDL layer.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/296171a8-aa4e-6533-8195-27dd02b05b44.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/f633ed55-5f5c-0969-77b4-ffcfd6f46992.png)

With the same manner, we can find that there are 246 features in the BNDA layer. This is important information for the next step.

|layer# | layer name | Number of features |
|---|---|---|
|0|BNDL|742|
|3|BNDA|246|


## Step 2: Export GeoJSON sequence from FeatureServer using nodejs script.
Clone the repository (https://github.com/unvt/featureserviceVT) from GitHub. Then, run "npm install" to install necessary npm packages.

```
git clone https://github.com/unvt/featureserviceVT
cd featureserviceVT
npm install
node index.js
```
Make sure that you can use nodejs and npm. Please note that "index.js" can run even without tippecanoe. So, you may be able to run it on WindowsPowerShell if you installed nodejs (and npm).
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/d3a24da3-765c-dc49-ecf6-2ccce9cb549b.png)


Once you run index.js, you will obtain "temp-json.json" (or any other name you specified in config/default.hjson) that in cludes all features from all layers listed in the config file. 
Sometimes, communication to Esri FeatureServer fails returning an error, so make sure that you do not have any error message (see below as an example of success).
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/28b7ba54-76c9-da49-f224-f1d38be74a4f.png)

Exported GeoJSON sequence would be like the below one. In this case, I confrimed that there were 988 features in a file. That is the result of 742 + 246, so both BNDL and BNDA layers were properly exported. 
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/df45041b-4c68-3d51-1462-48f83369d1d1.png)

I do not explain about the index.js in detail, but this is the script I made so that we can adjust the JSON structure for the vector tile conversion. Please read my Japanese article using translation APP for detail (see the references for the link).

## Step 3: Runnint tippecanoe to convert GeoJSON sequence into vector tile
Then, we run tippecanoe to create vector tiles from GeoJSON seq. Here, for demonstration purpose, I used Docker (unvt/nanban) because it is really easy to use tippecanoe even in the Windows environment. 

```
docker run -it --rm -v ${PWD}:/data unvt/nanban
cd /data
tippecanoe -o tile.pmtiles --projection=EPSG:3857 --force --no-tile-compression --no-feature-limit --no-tile-size-limit --drop-rate=1 temp-json.json
```

If the version of tippecanoe is v2.17 or later, you can export vector tiles in PMTiles format as above. If you use older version, let's just exprort vector tiles as pbf files using the following command.


```
tippecanoe -e tile-zxy --projection=EPSG:3857 --force --no-tile-compression --no-feature-limit
--no-tile-size-limit --drop-rate=1 temp-json.json
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/d50460f6-a1e9-192e-3acc-66b3add444e7.png)

:::note info
You may notice that I did some tricky thing here. The projection of the source is the world Eckert IV, but I specified projection as EPSG:3857 (Web Mercator), because we wanted to keep the original shape from the world Eckert IV projection even if the vector tiles lose their accurate geographic coordination. 
This kind of arrangement is not so recommended because you cannot overlay any other layers over this base map. 
:::

Thus, we successfully obtained the vector tiles from Esri FeatureServices.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/9f19801c-a036-117b-e06c-3e9bbb5ab175.png)


# Summary
I explained how I made vector tile from Esri Feature Service.

The key is that we compiled the json structure for each feature and compiled them so that we can have the JSON structure that is compatible with the GeoJSON sequence. This is because the vector tile conversion tool tippecanoe uses GeoJSON sequence as its input while Esri Feature Service does not provide feature information as GeoJSON structure.

I hope it helps your work.

# References
* Our working repository: https://github.com/unvt/featureserviceVT
* My previous document: https://qiita.com/T-ubu/items/17cb65754a54aa5ffb21
* ArcGIS REST API -Feature Service: https://developers.arcgis.com/rest/services-reference/enterprise/feature-service.htm
* UN Open GIS: http://unopengis.org/unopengis/main/main.php