Converting PMtiles from pbf tiles using felt/tippecanoe (tile-join)

# Introduction
PMtiles is a single-file archive format for tile data. I think it would be also useful for vector tiles. Therefore, with this articule, I will introduce my test of the data conversion from existing pbf vector tiles in the Internet to PMtiles.

> PMTiles is a single-file archive format for tiled data. A PMTiles archive can be hosted on a commodity storage platform such as S3, and enables low-cost, zero-maintenance map applications that are "serverless" - free of a custom tile backend or third party provider. (from [protomaps/pmtiles](https://github.com/protomaps/PMTiles))

(I am working with the vector tile over 150GM in total, and I am not yet sure if PMtiles is good to deal with these large data base. I am currently deal with such vector tiles by separating them into several hundred mbtiles files and host it by using mabox/mbtiles module and nodejs/express server routing. )


# My environment
In order to use tippecanoe, we use a linux server. However, with this open demonstration, I used docker container so that I can show it easily.

* Windows 10
* Docker: unvt/nanban (pulled in January 2023)
  * Ubuntu based container 
  * felt/tippecanoe version 2.17.0

# Procedure
I tried to access the vector tile from my colleagues' Esri server. This time, my wokring repository is here: https://github.com/ubukawa/109

If you would like to follow the process, make sure that you confirm your vector tile URL. **If you already have your pbf tiles, you can just start from the step 2.**

In the windows Powershell, I started a Docker container unvt/nanban.

```
docker run -it --rm -v ${PWD}:/data unvt/nanban
cd /data
```


## Step 1: Downloading vector tiles (pbf tiles)

### 1-1. Making directory for pbf tiles
Under the 0_dl folder, I have created directories for pbf tiles. My target was from zoom level 0 to 6 with global coverage, so I used the following command. Then, there is a folder structure for pbf tiles.

```bash:mkdir.sh
rm -rf 0_dl; mkdir 0_dl;for z in {0..6}; do mkdir 0_dl/${z};for x in `seq $((2 ** z))`; do mkdir 0_dl/${z}/$((x-1));done;done
```

### 1-2. Downloading tiles
Then, using curl, I have downloaded the files. It took for a while. In my environment, it tool about one hour.

```bash:download.sh
for z in {0..6}; do echo $z;for x in `seq $((2 ** z))`; do echo $z/$((x-1));for y in `seq $((2 ** z))`; do echo $z/$((x-1))/$((y-1)).pbf; curl https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/Clearmap_Webplain/VectorTileServer/tile/$z/$((y-1))/$((x-1)).pbf --output 0_dl/$z/$((x-1))/$((y-1)).pbf;done;done;done
```
**Please pay enough attention that the order of tile URL for ArcGIS REST API is z/y/x while other tile schema often uses z/x/y order.** (I forgot about it and did the process again.)
If you would like to download pbf tiles from non-Esri vector tile server, you may need to change z/y/x order to z/x/y order. Sh script would be as follows.

```bash:download-nonEsri.sh
for z in {0..6}; do echo $z;for x in `seq $((2 ** z))`; do echo $z/$((x-1));for y in `seq $((2 ** z))`; do echo $z/$((x-1))/$((y-1)).pbf; curl https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/Clearmap_Webplain/VectorTileServer/tile/$z/$((x-1))/$((y-1)).pbf --output 0_dl/$z/$((x-1))/$((y-1)).pbf;done;done;done
```
 

If you are good at bash, the above two sh scripts can be merged, and you can do it with one script.


### 1-3. Deleting empty tile
If there is no tile, it seems that Esri server returns the following response. I confirned that its size is 590 byte.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/f17e8145-e253-7935-3a2b-d8279224d340.png)

Let's check the files in the directory.

```bash
find ./0_dl -type f -size 590c | xargs ls -al
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/539e4c09-e44f-8f16-8095-d200634db788.png)


Then, I confirmed that the data size of all the found files was 590 byte. I thought all are the same file, and deleted them all. And, let's also delete empty direcroty as they will disturb the next process.
I realized that there is no tile in zoom level 6, so I did not need to download these 4,096 (64 * 64) files at all!!

```bash:delete-null.sh
find ./0_dl -type f -size 590c | xargs rm
find 0_dl -type d -empty -delete
```


## Step 2: Making PM tiles from the downloaded pbf tiles

The vector tile conversion tool, felt/tippecanoe, now supprts the PMtiles format since its vversion 2.17.

I have confired the version of my tippecanoe and run **tile-join**, a tool included in tippecanoe, to obtain PMtiles fild from a series of pbf tiles in the **0_dl** directory.

```bash
tippecanoe --version
tile-join -pk -o test123.pmtiles 0_dl
```
At first, I did not add "-pk (--no-tile-size-limit)" option and lost some tiles. So, it would be good to add -pk option.


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/1e3a12b5-d9f7-0655-4bb1-9811e859d22b.png)


If you have empty directory, you will fail data  conversion as below.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/cbc4aad2-1c65-27c2-4396-70e64483fddc.png)


If you would like to convert pmtiles from mbtiles, the following command would work.

```
tile-join -o output.pmtiles input.mbtiles
```
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/7fcb53ad-7264-0a6d-af0e-5c9af6487f89.png)




## Step 3: Review of PMtiles
I heard that we can easily review PMtiles with online PMTiles Viewer at https://protomaps.github.io/PMTiles/

I draged and dropped a file and saw the following. It looked okay.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/679143/d1048246-bd94-07cc-994f-de57a8e88149.png)




# Conclusion
Exisitng pbf tiles can be easily downloaded into PMtiles with felt/tippecanoe. However, if we have a lot of pbf tiles, we may need some time to download them. I agree that PMtiles would help vector tile hosing, in particular for middle sized data set. I meant that I am not sure about the tiles over several hundred GB at this moment. I would like to keep watching the development of PMtiles.


# Acknowledgement
 I respect the great work by colleagues. I also respect my colleagues who developed data.





 




# References
https://github.com/protomaps/PMTiles

https://github.com/felt/tippecanoe

https://geoservices.un.org/Html5Viewer/index.html?viewer=clearmap

https://developers.arcgis.com/rest/services-reference/enterprise/vector-tile-service.htm