Package: jupancon
jupanconDatabase Connectors and SQL magics forJupyter.jupancon= Jupyter + Pandas + Connectors.FeaturesConnector to RedshiftUsing user/passUsing IAM profileConnector to Bigquery (using google profile)Connector to DatabricksOptional automatic tunnel setting through an SSH BastionQuerying capabilitiesIPython kernel magics for queryingAlways returnsPandas DataFramesSome hidden stuff I rather not document just yet so you don't nuke your Warehouse :) Will do when it's safer to useInstallpipinstalljupanconConfigureWrite a~/.jupancon/config.ymlYAML file that looks similar to the following C&P from my actual config file (heavily censored for obvious reasons):default:my-redshift-clustermy-redshift-cluster:type:redshifthost:XXXXXX.XXXXXX.XXXXXXX.redshift.amazonaws.com# explicitly setting redshift port (optional)port:5439user:XXXXXXXXpass:XXXXXXXXdbname:XXXXXXmy-redshift-using-iamprofile:type:redshifthost:XXXXXX.XXXXXX.XXXXXXX.redshift.amazonaws.comprofile:XXXXXXXXXdbname:XXXXXX# NOTE: you can choose dbuser and it will be auto-created if it doesn't existdbuser:XXXXXXcluster:XXXXXXmy-gcp:type:bigqueryproject:XXXXX-XXXXX-123456my-databricks:type:databricksserver_hostname:XXXXXX.cloud.databricks.comhttp_path:/sql/XXX/XXXX/XXXXXXXXXX# optionalcatalog:XXXXXXXtoken:XXXXXXXXXmy-redshift-behind-sshbastion:type:redshiftuse_bastion:truebastion_server:censored.bastion.server.combastion_user:XXXXXXbastion_host:XXXXXX.XXXXXX.XXXXXX.redshift.amazonaws.comhost:censored.main.server.comuser:XXXXXXXXpass:XXXXXXXXdbname:XXXXXXJupancon will also pick environment variables (which have preference over theconfig.yml).JPC_DB_TYPE:redshiftorbigqueryJPC_HOST: for example,XXXXXX.XXXXXX.XXXXXX.redshift.amazonaws.comJPC_USER: User nameJPC_DB: Database nameJPC_PASS: PasswordJPC_USE_BASTION:trueor leave blankJPC_BASTION_SERVERJPC_BASTION_HOSTJPC_PROFILE: IAM profile (for IAM connection only)JPC_CLUSTER: Redshift cluster (for IAM connection only)JPC_DBUSER: Redshift user (for IAM connection only)How to useThis library is developed primarily for usage withinJupyter Lab. It's likely to work in Jupyter Notebook and Ipython, but untested and unsupported at this stage. It also works and is being used in regular scripts, butit obviously loses its magic.Regular usagefromjupanconimportquery,list_schemas,list_tableslist_schemas()list_tables()query("select * from foo")Magical usagefromjupanconimportload_magicsload_magics()select*fromfoodf=%select*fromfoo%%sqlselect*fromfoowherecond=1andlabel='my nice label'DevelopmentCurrent status: Jupancon has enough basic features that it's worth open sourcing, but the documentation is still lacking.TODO listlist_table("schema")to detect if schema doesn't exist and return errorAdd query monitoring and cancelling functionalityComplete docs (low level stuff, exhaustive features, maybe sphinx/rdd?)Add animated gifs to docsFeatures that aren't worth adding right nowAutocomplete and autodiscover of databases is possible, but not trivial at all. In addition, I'll like to find a way of not adding any extra configuration. Regardless, not worth it until the TODO list above is tackled. Seethis projectfor a successful example.Because of the current architecture of Jupyter Lab, syntax highlighting is not feasible to add (as it was in Jupyter Notebook). This might change in the future. See thisgit issuefor more details.A note about Unit TestingI would like to publish decent unit testing, but this library is hard to test because all the databases currently queried for it's development are either tests that cost me money or private (my clients') databases. Any ideas on how to write an open source, non exploitable set of unit tests for Redshift or BigQuery are very welcome.