Package: pg-data-etl
pg-data-etlETL tools for spatial data stored in postgres.AboutThis module exists to make life easier when working with geospatial data in a Postgres database.You should have the following command-line tools installed, preferably on your system path:psqlpg_dumpshp2postgisogr2ogrIf you want to use the optional vector tile functions you'll also need:tippecanoeInstallationpip install pg_data_etlExampleThe following code blocks import spatial data into Postgres and runs a spatial query:1) Connect to the database and create it>>>frompg_data_etlimportDatabase>>>credentials={..."db_name":"sample_database",..."host":"localhost",..."un":"username",..."pw":"my-password",..."super_un":"postgres",..."super_pw":"superuser-password"...}>>>db=Database.from_parameters(**credentials)>>>db.admin("CREATE")2) Import GIS data from the web>>>data_to_import=[...("philly.high_injury_network","https://phl.carto.com/api/v2/sql?filename=high_injury_network_2020&format=geojson&skipfields=cartodb_id&q=SELECT+*+FROM+high_injury_network_2020"),...("philly.playgrounds","https://opendata.arcgis.com/datasets/899c807e205244278b3f39421be8489c_0.geojson")...]>>>forsql_tablename,source_urlindata_to_import:...kwargs={..."filepath":source_url,..."sql_tablename":sql_tablename,..."gpd_kwargs":{"if_exists":"replace"}...}...db.import_gis(**kwargs)3) Run a query and get the result as ageopandas.GeoDataFrame>>># Define a SQL query as a string in Python>>>query="""... select * from philly.high_injury_network... where st_dwithin(...     st_transform(geom, 26918),...     (select st_transform(st_collect(geom), 26918) from philly.playgrounds),...     100... )... order by st_length(geom) DESC """>>># Get a geodataframe from the db using the query>>>gdf=db.gdf(query)>>>gdf.head()indexobjectidstreet_namebuffergeomuid0234189BUSTLETONAVE75feetLINESTRING(-75.0708140.03528,-75.0705240.0...236165385THST50feetLINESTRING(-75.1452839.96913,-75.1450239.9...662223179ARAMINGOAVE75feetLINESTRING(-75.1221239.97449,-75.1213239.9...2243148215KELLYDR75feetLINESTRING(-75.1847039.96934,-75.1851339.9...1504156224MARTINLUTHERKINGDR75feetLINESTRING(-75.1771339.96327,-75.1777539.9...159To save time and typing, database credentials can be stored in a text file. You can place this file wherever you want,
but by default it's placed into/USERHOME/.pg-data-etl/database_connections.cfg.To generate one for the first time, run the following from a terminal prompt:>pgmake-config-fileThis file uses the following format:[DEFAULT]
pw = this-is-a-placeholder-password
port = 5432
super_db = postgres
super_un = postgres
super_pw = this-is-another-placeholder-password

[localhost]
host = localhost
un = postgres
pw = your-password-hereEach entry in square brackets is a named connection, and any parameters not explicitly defined are inherited fromDEFAULT.
You can have as many connections defined as you'd like, and you can use them like this:>>>frompg_data_etlimportDatabase>>>db=Database.from_config("sample_database","localhost")DevelopmentClone or fork this repo:gitclonehttps://github.com/aaronfraint/pg-data-etl.gitcdpg-data-etlInstall an editable version withpoetry:poetryinstallWindows users who prefer to usecondacan use the includedenvironment.ymlfile:condaenvcreate-fenvironment.yml