Package: nycdb
# NYCDBa tool for building a database of NYC housing dataThis is a Python library and cli tool for installing, updating and managing NYCDB, a postgres database of NYC Housing Data.For more background information on this project and links to download copies of full database dump visit:https://github.com/nycdb/nycdb. We use the termnycdbto refer to both the python software and the running copy of the postgres database.## Using the cli toolYou will need python 3.6+ and Postgres. The latest version can be installed from pypi with pip:python3 -m pip install nycdbIf the installation is successful, you can view a summary of the tool’s options by runningnycdb –helpTo print a list of datasets: ` nycdb –list-datasets`nycdb’s main job is to download datasets and import them into postgres. It does not manage the database for you. You can use the flags-U/–user,-D/–database,-P/–password, and-H/–hostto instruct nycdb to connect to the correct database. Seenycdb –helpfor the defaults.Example: downloading, loading, and verifying the datasethpd_violations:` sh nycdb--downloadhpd_violations nycdb--loadhpd_violations nycdb--verifyhpd_violations `To delete a previously imported dataset (for example, if you’d like to import an updated version of it), use the–dropoption. For example, to delete the datasethpd_violations:nycdb –drop hpd_violations.You can also verify all datasets: ` nycdb –verify-all `By default the downloaded data files are is stored in./data. Use–root-dirto change the location of the data directory.You can export a.sqlfile for any dataset by using the–dumpcommand## DevelopmentThere are two development workflows: one using python virtual environments and one using docker.### Using docker and docker-composeClone the nycdb repository to your computer, open the terminal, and set your working directory to the location of the cloned nycdb folder usingcd <filepath>To get started all you have to do is rundocker-compose up.On the first run Docker will take longer to downloads and build the images. It
will start a Postgres server on port 5432 of your local machineYou can also press
<kbd>CTRL</kbd>-<kbd>C</kbd> at any point to stop the server.In a separate terminal, you will be able to now use the nycdb cli:docker-compose run nycdb –help.You will not have any data loaded when you create your local instance of the db. Use functions like–downloadand–loadto add datasets to your local database, for example:docker-compose run nycdb –download <dataset>You can also open a python3 shell:docker-compose run –entrypoint=python3 nycdbor run the test suitdocker-compose run –entrypoint=”pytest tests” nycdbYou may also develop on nycdb itself:Any changes you make to the tool’s source code will automatically be reflected
in future invocations ofnycdband the test suite.The postgres database server is forwarded to localhost:5432 which you can connect to via a desktop client if you like.If you don’t have a desktop Postgres client, you can always runnycdb –dbshellto interactively inspect the database with [psql](http://postgresguide.com/utilities/psql.html).To update the database after adding new packages or dev dependencies, just rundocker-compose up –build –force-recreate –no-deps. This command will take a bit longer than the regulardocker-compose upcommand, but will reinstall packages within the docker container without removing any downloaded files or database data from the docker database.To stop the database rundocker-compose down. The downloaded files and database data are stored in docker volumes and are not automatically removed.However, if you ever want to wipe the database, rundocker-compose down -v.### Python3 virtual environmentsIf you have postgres installed separately, you can use this alternative method without docker:Setup and active a virtual environment:` sh python3-mvenv venv source venv/bin/activate `Install nycdb: ` pip install -e ./src`As long as the virtual environment is activated, you can usenycdbdirectly in your shell.###  Adding New DatasetsSee the [guide here](ADDING_NEW_DATASETS.md) for the steps to add a new dataset### A Note on PostGISTo use datasets with spatial information such asboundaries, you will need to install the extension [PostGIS](https://postgis.net/) and have the programshp2pgsqlinstalled and available on your path.shp2pgsqlis typically installed alongside the PostGIS installation.The extension can be enabled in your database by executing the SQLCREATE EXTENSION postgis.Whendocker compose upis executed for the first time, the PostGIS extension will be enabled automatically.