Package: ipython-sparksql
Author:Catherine Devlin,http://catherinedevlin.blogspot.comIntroduces a %sql (or %%sql) magic.Connect to a database, using SQLAlchemy connect strings, then issue SQL
commands within IPython or IPython Notebook.ExamplesIn[1]:%load_extsqlIn[2]:%%sqlpostgresql://will:longliveliz@localhost/shakes...:select*fromcharacter...:whereabbrev='ALICE'...:Out[2]:[(u'Alice',u'Alice',u'ALICE',u'a lady attending on Princess Katherine',22)]In[3]:result=_In[4]:print(result)charidcharnameabbrevdescriptionspeechcount=================================================================================AliceAliceALICEaladyattendingonPrincessKatherine22In[4]:result.keysOut[5]:[u'charid',u'charname',u'abbrev',u'description',u'speechcount']In[6]:result[0][0]Out[6]:u'Alice'In[7]:result[0].descriptionOut[7]:u'a lady attending on Princess Katherine'After the first connection, connect info can be omitted:In [8]: %sql select count(*) from work
Out[8]: [(43L,)]Connections to multiple databases can be maintained.  You can refer to
an existing connection byusername@databaseIn[9]:%%sqlwill@shakes...:selectcharname,speechcountfromcharacter...:wherespeechcount=(selectmax(speechcount)...:fromcharacter);...:Out[9]:[(u'Poet',733)]In[10]:print(_)charnamespeechcount======================Poet733If no connect string is supplied,%sqlwill provide a list of existing connections;
however, if no connections have yet been made and the environment variableDATABASE_URLis available, that will be used.For secure access, you may dynamically access your credentials (e.g. from your system environment orgetpass.getpass) to avoid storing your password in the notebook itself. Use the$before any variable to access it in your%sqlcommand.In[11]:user=os.getenv('SOME_USER')....:password=os.getenv('SOME_PASSWORD')....:connection_string="postgresql://{user}:{password}@localhost/some_database".format(user=user,password=password)....:%sql$connection_stringOut[11]:u'Connected: some_user@some_database'You may use multiple SQL statements inside a single cell, but you will
only see any query results from the last of them, so this really only
makes sense for statements with no outputIn[11]:%%sqlsqlite://....:CREATETABLEwriter(first_name,last_name,year_of_death);....:INSERTINTOwriterVALUES('William','Shakespeare',1616);....:INSERTINTOwriterVALUES('Bertold','Brecht',1956);....:Out[11]:[]Bind variables (bind parameters) can be used in the “named” (:x) style.
The variable names used should be defined in the local namespaceIn[12]:name='Countess'In[13]:%sqlselectdescriptionfromcharacterwherecharname=:nameOut[13]:[(u'mother to Bertram',)]As a convenience, dict-style access for result sets is supported, with the
leftmost column serving as key, for unique values.In[14]:result=%sqlselect*fromwork43rowsaffected.In[15]:result['richard2']Out[15]:(u'richard2',u'Richard II',u'History of Richard II',1595,u'h',None,u'Moby',22411,628)Results can also be retrieved as an iterator of dictionaries (result.dicts())
or a single dictionary with a tuple of scalar values per key (result.dict())AssignmentOrdinary IPython assignment works for single-line%sqlqueries:In[16]:works=%sqlSELECTtitle,yearFROMwork43rowsaffected.The<<operator captures query results in a local variable, and
can be used in multi-line%%sql:In[17]:%%sqlworks<<SELECTtitle,year...:FROMwork...:43rowsaffected.ReturningdatatolocalvariableworksConnectingConnection strings areSQLAlchemystandard.Some example connection strings:mysql+pymysql://scott:tiger@localhost/foo
oracle://scott:tiger@127.0.0.1:1521/sidname
sqlite://
sqlite:///foo.db
mssql+pyodbc://username:password@host/database?driver=SQL+Server+Native+Client+11.0Note thatmysqlandmysql+pymysqlconnections (and perhaps others)
don’t read your client character set information from .my.cnf.  You need
to specify it in the connection string:mysql+pymysql://scott:tiger@localhost/foo?charset=utf8Note that animpalaconnection withimpylafor HiveServer2 requires disabling autocommit:%config SqlMagic.autocommit=False
%sql impala://hserverhost:port/default?kerberos_service_name=hive&auth_mechanism=GSSAPIConfigurationQuery results are loaded as lists, so very large result sets may use up
your system’s memory and/or hang your browser.  There is no autolimit
by default.  However,autolimit(if set) limits the size of the result
set (usually with aLIMITclause in the SQL).displaylimitis similar,
but the entire result set is still pulled into memory (for later analysis);
only the screen display is truncated.In[2]:%configSqlMagicSqlMagicoptions--------------SqlMagic.autocommit=<Bool>Current:TrueSetautocommitmodeSqlMagic.autolimit=<Int>Current:0AutomaticallylimitthesizeofthereturnedresultsetsSqlMagic.autopandas=<Bool>Current:FalseReturnPandasDataFramesinsteadofregularresultsetsSqlMagic.displaylimit=<Int>Current:0Automaticallylimitthenumberofrowsdisplayed(fullresultsetisstillstored)SqlMagic.feedback=<Bool>Current:TruePrintnumberofrowsaffectedbyDMLSqlMagic.short_errors=<Bool>Current:TrueDon't display the full traceback on SQL Programming ErrorSqlMagic.style=<Unicode>Current:'DEFAULT'Setthetableprintingstyletoanyofprettytable's defined styles(currentlyDEFAULT,MSWORD_FRIENDLY,PLAIN_COLUMNS,RANDOM)In[3]:%configSqlMagic.feedback=FalsePlease note: if you have autopandas set to true, the displaylimit option will not apply. You can set the pandas display limit by using the pandasmax_rowsoption as described in thepandas documentation.PandasIf you have installedpandas, you can use a result set’s.DataFrame()methodIn[3]:result=%sqlSELECT*FROMcharacterWHEREspeechcount>25In[4]:dataframe=result.DataFrame()The bogus non-standard pseudo-SQL commandPERSISTwill create a table name
in the database from the named DataFrame.In[5]:%sqlPERSISTdataframeIn[6]:%sqlSELECT*FROMdataframe;GraphingIf you have installedmatplotlib, you can use a result set’s.plot(),.pie(), and.bar()methods for quick plottingIn[5]:result=%sqlSELECTtitle,totalwordsFROMworkWHEREgenretype='c'In[6]:%matplotlibinlineIn[7]:result.pie()DumpingResult sets come with a.csv(filename=None)method.  This generates
comma-separated text either as a return value (iffilenameis not
specified) or in a file of the given name.In[8]:result=%sqlSELECTtitle,totalwordsFROMworkWHEREgenretype='c'In[9]:result.csv(filename='work.csv')PostgreSQL featurespsql-style “backslash”meta-commandscommands (\d,\dt, etc.)
are provided byPGSpecial.  Example:In[9]:%sql\dInstallingInstall the lastest release with:pip install ipython-sqlor download fromhttps://github.com/catherinedevlin/ipython-sqland:cd ipython-sql
sudo python setup.py installDevelopmenthttps://github.com/catherinedevlin/ipython-sqlCreditsMatthias Bussonnier for help with configurationOlivier Le Thanh Duong for%configfixes and improvementsDistributeBuildoutmodern-package-templateMike Wilson for bind variable codeThomas Kluyver and Steve Holden for debugging helpBerton Earnshaw for DSN connection syntaxAndrés Celis for SQL Server bugfixMichael Erasmus for DataFrame truth bugfixNoam Finkelstein for README clarificationXiaochuan Yu for<<operator, syntax colorizationAmjith Ramanujam for PGSpecial and incorporating it hereNews0.1Release date: 21-Mar-2013Initial release0.1.1Release date: 29-Mar-2013Release to PyPIResults returned as listsprint(_) to get table form in text consoleset autolimit and text wrap in configuration0.1.2Release date: 29-Mar-2013Python 3 compatibilityuse prettyprint packageallow multiple SQL per cell0.2.0Release date: 30-May-2013Accept bind variables (Thanks Mike Wilson!)0.2.1Release date: 15-June-2013Recognize socket connection stringsBugfix - issue 4 (remember existing connections by case)0.2.2Release date: 30-July-2013Converted from an IPython Plugin to an Extension for 1.0 compatibility0.2.2.1Release date: 01-Aug-2013Deleted Plugin import left behind in 0.2.20.2.3Release date: 20-Sep-2013Contributions from Olivier Le Thanh Duong:SQL errors reported without internal IPython error stackProper handling of configurationAdded .DataFrame(), .pie(), .plot(), and .bar() methods to
result sets0.3.0Release date: 13-Oct-2013displaylimit config parameterreports number of rows affected by each querytest suite working againdict-style access for result sets by primary key0.3.1Reporting of number of rows affected configurable withfeedbackLocal variables usable as SQL bind variables0.3.2.csv(filename=None)method added to result sets0.3.3Python 3 compatibility restoredDSN access supported (thanks Berton Earnshaw)0.3.4PERSIST pseudo-SQL command added0.3.5Indentations visible in HTML cellsCOMMIT each SQL statement immediately - prevent locks0.3.6Fixed issue #30, commit failures for sqlite (thanks stonebig, jandot)0.3.7Newcolumn_local_varsconfig option submitted by darikgAvoid contaminating user namespace from locals (thanks alope107)0.3.7.1Avoid “connection busy” error for SQL Server (thanks Andrés Celis)0.3.8Stop warnings for deprecated use of IPython 3 traitlets in IPython 4 (thanks graphaelli; also stonebig, aebrahim, mccahill)README update for keeping connection info private, from eshilts0.3.9Fix truth value of DataFrame error (thanks michael-erasmus)<<operator (thanks xiaochuanyu)added README example (thanks tanhuil)bugfix in executing column_local_vars (thanks tebeka)pgspecial installation optional (thanks jstoebel and arjoe)conceal passwords in connection strings (thanks jstoebel)0.3.9Restored Python 2 compatibility (thanks tokenmathguy)