Package: wrds2postgres
WRDS (or SAS) to PostgreSQLThis software has two functions:Download tables fromWRDSand feeds them to a PostgreSQL database. (Requires access to WRDS and to the data in question.)Import a SAS file (*.sas7dbat) into a PostgreSQL database.Requirements1. PythonThe software uses Python 3 and depends on Pandas, SQLAlchemy and Paramiko. In addition, the Python scripts generally interact with PostgreSQL using SQLAlchemy and thePsycopglibrary.
These dependencies are installed when you use Pip (see instructions below).2. A WRDS IDTo use public-key authentication to access WRDS, follow hints taken fromhereto set up a public key.
Copy that key to the WRDS server from the terminal on my computer.
(Note that this code assumes you have a directory.sshin your home directory. If not, log into WRDS via SSH, then typemkdir ~/.sshto create this.)
Here's code to create the key and send it to WRDS (for me):ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub | ssh iangow@wrds-cloud.wharton.upenn.edu "cat >> ~/.ssh/authorized_keys"Use an empty passphrase in setting up the key so that the scripts can run without user intervention.3. PostgreSQLYou should have a PostgreSQL database to store the data.4. Environment variablesEnvironment variables that the code can use include:PGDATABASE: The name of the PostgreSQL database you use.PGUSER: Your username on the PostgreSQL database.PGHOST: Where the PostgreSQL database is to be found (this will belocalhostif its on the same machine as you're running the code on)WRDS_ID: YourWRDSID.I set these environment variables in~/.profile:export PGHOST="localhost"
export PGDATABASE="crsp"
export WRDS_ID="iangow"
export PGUSER="igow"Using the functionwrds_update.Two argumentstable_nameandschemaare required.1. WRDS SettingsSetWRDS_IDusing eitherwrds_id=your_wrds_idin the function call or the environment variableWRDS_ID.2. PG SettingsThe software will use the environment variablesPGHOST,PGDATABASE, andPGUSERif you If you have set them. Otherwise, you need to provide values as arguments towrds_udpate(). DefaultPGPORTis5432.3. Table SettingsTo tailor your request, specify the following arguments:fix_missing: set toTrueto fix missing values. This addresses special missign values, which SAS'sPROC EXPORTdumps as strings. Default value isFalse.fix_cr: set toTrueto fix characters. Default value isFalse.drop: add column names to be dropped (e.g.,drop="id name"will drop columnsidandname).obs: add maxium number of observations (e.g.,obs=10will import the first 10 rows from the table on WRDS).rename: rename columns (e.g.,rename="fee=mngt_fee"renamesfeetomngt_fee).force: set toTrueto force update. Default value isFalse.Importing SAS data into PostgreSQLThe software can also upload SAS file directly to PostgreSQL.
You need to have local SAS in order to use this function.
Usefpathto specify the path to the file to be importedExamplesHere are some examples.If you are at the home directory of this git repo, you can import and use the software as shown below.To install it from Github:sudo -H pip3 install git+https://github.com/iangow/wrds2pg --upgradeThis software is also available from PyPI. To install it fromPyPI:pip3 install wrds2pgExample usage:fromwrds2pg.wrds2pgimportwrds_update# 1. Download crsp.mcti from wrds and upload to pg as crps.mcti# Simplest versionwrds_update(table_name="mcti",schema="crsp")# Tailored argumentswrds_update(table_name="mcti",schema="crsp",host=your_pghost,dbname=your_pg_database,fix_missing=True,fix_cr=True,drop="b30ret b30ind",obs=10,rename="caldt=calendar_date",force=True)# 2. Upload test.sas7dbat to pg as crsp.mctiwrds_update(table_name="mcti",schema="crsp",fpath="your_path/test.sas7dbat")Report BugsAuthor: Ian Gow,iandgow@gmail.comContributor: Jingyu Zhang,jingyu.zhang@chicagobooth.edu