Package: ipython-sql
Author:Catherine Devlin,http://catherinedevlin.blogspot.comIntroduces a %sql (or %%sql) magic.Connect to a database, usingSQLAlchemy URLconnect strings, then issue SQL
commands within IPython or IPython Notebook.ExamplesIn[1]:%load_extsqlIn[2]:%%sqlpostgresql://will:longliveliz@localhost/shakes...:select*fromcharacter...:whereabbrev='ALICE'...:Out[2]:[(u'Alice',u'Alice',u'ALICE',u'a lady attending on Princess Katherine',22)]In[3]:result=_In[4]:print(result)charidcharnameabbrevdescriptionspeechcount=================================================================================AliceAliceALICEaladyattendingonPrincessKatherine22In[4]:result.keysOut[5]:[u'charid',u'charname',u'abbrev',u'description',u'speechcount']In[6]:result[0][0]Out[6]:u'Alice'In[7]:result[0].descriptionOut[7]:u'a lady attending on Princess Katherine'After the first connection, connect info can be omitted:In [8]: %sql select count(*) from work
Out[8]: [(43L,)]Connections to multiple databases can be maintained.  You can refer to
an existing connection byusername@databaseIn[9]:%%sqlwill@shakes...:selectcharname,speechcountfromcharacter...:wherespeechcount=(selectmax(speechcount)...:fromcharacter);...:Out[9]:[(u'Poet',733)]In[10]:print(_)charnamespeechcount======================Poet733If no connect string is supplied,%sqlwill provide a list of existing connections;
however, if no connections have yet been made and the environment variableDATABASE_URLis available, that will be used.For secure access, you may dynamically access your credentials (e.g. from your system environment orgetpass.getpass) to avoid storing your password in the notebook itself. Use the$before any variable to access it in your%sqlcommand.In[11]:user=os.getenv('SOME_USER')....:password=os.getenv('SOME_PASSWORD')....:connection_string="postgresql://{user}:{password}@localhost/some_database".format(user=user,password=password)....:%sql$connection_stringOut[11]:u'Connected: some_user@some_database'You may use multiple SQL statements inside a single cell, but you will
only see any query results from the last of them, so this really only
makes sense for statements with no outputIn[11]:%%sqlsqlite://....:CREATETABLEwriter(first_name,last_name,year_of_death);....:INSERTINTOwriterVALUES('William','Shakespeare',1616);....:INSERTINTOwriterVALUES('Bertold','Brecht',1956);....:Out[11]:[]As a convenience, dict-style access for result sets is supported, with the
leftmost column serving as key, for unique values.In[12]:result=%sqlselect*fromwork43rowsaffected.In[13]:result['richard2']Out[14]:(u'richard2',u'Richard II',u'History of Richard II',1595,u'h',None,u'Moby',22411,628)Results can also be retrieved as an iterator of dictionaries (result.dicts())
or a single dictionary with a tuple of scalar values per key (result.dict())Variable substitutionBind variables (bind parameters) can be used in the “named” (:x) style.
The variable names used should be defined in the local namespace.In[15]:name='Countess'In[16]:%sqlselectdescriptionfromcharacterwherecharname=:nameOut[16]:[(u'mother to Bertram',)]Alternately,$variable_nameor{variable_name}can be
used to inject variables from the local namespace into the SQL
statement before it is formed and passed to the SQL engine.
(Using$and{}together, as in${variable_name},
is not supported.)In [17]: %sql select description from character where charname = ‘{name}’
Out[17]: [(u’mother to Bertram’,)]Bind variables are passed through to the SQL engine and can only
be used to replace strings passed to SQL.$and{}are
substituted before passing to SQL and can be used to form SQL
statements dynamically.AssignmentOrdinary IPython assignment works for single-line%sqlqueries:In[18]:works=%sqlSELECTtitle,yearFROMwork43rowsaffected.The<<operator captures query results in a local variable, and
can be used in multi-line%%sql:In[19]:%%sqlworks<<SELECTtitle,year...:FROMwork...:43rowsaffected.ReturningdatatolocalvariableworksConnectingConnection strings areSQLAlchemy URLstandard.Some example connection strings:mysql+pymysql://scott:tiger@localhost/foo
oracle://scott:tiger@127.0.0.1:1521/sidname
sqlite://
sqlite:///foo.db
mssql+pyodbc://username:password@host/database?driver=SQL+Server+Native+Client+11.0Note thatmysqlandmysql+pymysqlconnections (and perhaps others)
don’t read your client character set information from .my.cnf.  You need
to specify it in the connection string:mysql+pymysql://scott:tiger@localhost/foo?charset=utf8Note that animpalaconnection withimpylafor HiveServer2 requires disabling autocommit:%config SqlMagic.autocommit=False
%sql impala://hserverhost:port/default?kerberos_service_name=hive&auth_mechanism=GSSAPIConnection arguments not whitelisted by SQLALchemy can be provided as
a flag with (-a|–connection_arguments)the connection string as a JSON string.
SeeSQLAlchemy Args.%sql –connection_arguments {“timeout”:10,”mode”:”ro”} sqlite:// SELECT * FROM work;%sql -a ‘{“timeout”:10, “mode”:”ro”}’ sqlite:// SELECT * from work;DSN connectionsAlternately, you can store connection info in a
configuration file, under a section name chosen to
refer to your database.For example, if dsn.ini contains[DB_CONFIG_1]drivername=postgreshost=my.remote.hostport=5433database=mydatabaseusername=myuserpassword=1234then you can%config SqlMagic.dsn_filename=’./dsn.ini’%sql –section DB_CONFIG_1ConfigurationQuery results are loaded as lists, so very large result sets may use up
your system’s memory and/or hang your browser.  There is no autolimit
by default.  However,autolimit(if set) limits the size of the result
set (usually with aLIMITclause in the SQL).displaylimitis similar,
but the entire result set is still pulled into memory (for later analysis);
only the screen display is truncated.In[2]:%configSqlMagicSqlMagicoptions--------------SqlMagic.autocommit=<Bool>Current:TrueSetautocommitmodeSqlMagic.autolimit=<Int>Current:0AutomaticallylimitthesizeofthereturnedresultsetsSqlMagic.autopandas=<Bool>Current:FalseReturnPandasDataFramesinsteadofregularresultsetsSqlMagic.column_local_vars=<Bool>Current:FalseReturndataintolocalvariablesfromcolumnnamesSqlMagic.displaycon=<Bool>Current:FalseShowconnectionstringafterexecuteSqlMagic.displaylimit=<Int>Current:NoneAutomaticallylimitthenumberofrowsdisplayed(fullresultsetisstillstored)SqlMagic.dsn_filename=<Unicode>Current:'odbc.ini'PathtoDSNfile.Whenthefirstargumentisoftheform[section],asqlalchemyconnectionstringisformedfromthematchingsectionintheDSNfile.SqlMagic.feedback=<Bool>Current:FalsePrintnumberofrowsaffectedbyDMLSqlMagic.short_errors=<Bool>Current:TrueDon't display the full traceback on SQL Programming ErrorSqlMagic.style=<Unicode>Current:'DEFAULT'Setthetableprintingstyletoanyofprettytable's defined styles(currentlyDEFAULT,MSWORD_FRIENDLY,PLAIN_COLUMNS,RANDOM)In[3]:%configSqlMagic.feedback=FalsePlease note: if you have autopandas set to true, the displaylimit option will not apply. You can set the pandas display limit by using the pandasmax_rowsoption as described in thepandas documentation.PandasIf you have installedpandas, you can use a result set’s.DataFrame()methodIn[3]:result=%sqlSELECT*FROMcharacterWHEREspeechcount>25In[4]:dataframe=result.DataFrame()The--persistargument, with the name of a
DataFrame object in memory,
will create a table name
in the database from the named DataFrame.
Or use--appendto add rows to an existing
table by that name.In[5]:%sql--persistdataframeIn[6]:%sqlSELECT*FROMdataframe;GraphingIf you have installedmatplotlib, you can use a result set’s.plot(),.pie(), and.bar()methods for quick plottingIn[5]:result=%sqlSELECTtitle,totalwordsFROMworkWHEREgenretype='c'In[6]:%matplotlibinlineIn[7]:result.pie()DumpingResult sets come with a.csv(filename=None)method.  This generates
comma-separated text either as a return value (iffilenameis not
specified) or in a file of the given name.In[8]:result=%sqlSELECTtitle,totalwordsFROMworkWHEREgenretype='c'In[9]:result.csv(filename='work.csv')PostgreSQL featurespsql-style “backslash”meta-commandscommands (\d,\dt, etc.)
are provided byPGSpecial.  Example:In[9]:%sql\dOptions-l/--connectionsList all active connections-x/--close<session-name>Close named connection-c/--creator<creator-function>Specify creator function for new connection-s/--section<section-name>Section of dsn_file to be used for generating a connection string-p/--persistCreate a table name in the database from the named DataFrame--appendLike--persist, but appends to the table if it already exists-a/--connection_arguments<"{connectionarguments}">Specify dictionary of connection arguments to pass to SQL driver-f/--file<path>Run SQL from file at this pathCautionCommentsBecause ipyton-sql accepts---delimited options like--persist, but--is also the syntax to denote a SQL comment, the parser needs to make some assumptions.If you try to pass an unsupported argument, like--lutefisk, it will
be interpreted as a SQL comment and will not throw an unsupported argument
exception.If the SQL statement begins with a first-line comment that looks like one
of the accepted arguments - like%sql--persistis great!- it will be
parsed like an argument, not a comment.  Moving the comment to the second
line or later will avoid this.InstallingInstall the latest release with:pip install ipython-sqlor download fromhttps://github.com/catherinedevlin/ipython-sqland:cd ipython-sql
sudo python setup.py installDevelopmenthttps://github.com/catherinedevlin/ipython-sqlCreditsMatthias Bussonnier for help with configurationOlivier Le Thanh Duong for%configfixes and improvementsDistributeBuildoutmodern-package-templateMike Wilson for bind variable codeThomas Kluyver and Steve Holden for debugging helpBerton Earnshaw for DSN connection syntaxBruno Harbulot for DSN exampleAndrés Celis for SQL Server bugfixMichael Erasmus for DataFrame truth bugfixNoam Finkelstein for README clarificationXiaochuan Yu for<<operator, syntax colorizationAmjith Ramanujam for PGSpecial and incorporating it hereAlexander Maznev for better arg parsing, connections accepting specified creatorJonathan Larkin for configurable displayconJared Moore forconnection-argumentssupportGilbert Brault for--appendLucas Zeer for multi-line bugfixes for var substitution,<<vkk800 for--fileJens Albrecht for MySQL DatabaseError bugfixmeihkv for connection-closing bugfixAbhinav C for SQLAlchemy 2.0 compatibilityNews0.1Release date: 21-Mar-2013Initial release0.1.1Release date: 29-Mar-2013Release to PyPIResults returned as listsprint(_) to get table form in text consoleset autolimit and text wrap in configuration0.1.2Release date: 29-Mar-2013Python 3 compatibilityuse prettyprint packageallow multiple SQL per cell0.2.0Release date: 30-May-2013Accept bind variables (Thanks Mike Wilson!)0.2.1Release date: 15-June-2013Recognize socket connection stringsBugfix - issue 4 (remember existing connections by case)0.2.2Release date: 30-July-2013Converted from an IPython Plugin to an Extension for 1.0 compatibility0.2.2.1Release date: 01-Aug-2013Deleted Plugin import left behind in 0.2.20.2.3Release date: 20-Sep-2013Contributions from Olivier Le Thanh Duong:SQL errors reported without internal IPython error stackProper handling of configurationAdded .DataFrame(), .pie(), .plot(), and .bar() methods to
result sets0.3.0Release date: 13-Oct-2013displaylimit config parameterreports number of rows affected by each querytest suite working againdict-style access for result sets by primary key0.3.1Reporting of number of rows affected configurable withfeedbackLocal variables usable as SQL bind variables0.3.2.csv(filename=None)method added to result sets0.3.3Python 3 compatibility restoredDSN access supported (thanks Berton Earnshaw)0.3.4PERSIST pseudo-SQL command added0.3.5Indentations visible in HTML cellsCOMMIT each SQL statement immediately - prevent locks0.3.6Fixed issue #30, commit failures for sqlite (thanks stonebig, jandot)0.3.7Newcolumn_local_varsconfig option submitted by darikgAvoid contaminating user namespace from locals (thanks alope107)0.3.7.1Avoid “connection busy” error for SQL Server (thanks Andrés Celis)0.3.8Stop warnings for deprecated use of IPython 3 traitlets in IPython 4 (thanks graphaelli; also stonebig, aebrahim, mccahill)README update for keeping connection info private, from eshilts0.3.9Fix truth value of DataFrame error (thanks michael-erasmus)<<operator (thanks xiaochuanyu)added README example (thanks tanhuil)bugfix in executing column_local_vars (thanks tebeka)pgspecial installation optional (thanks jstoebel and arjoe)conceal passwords in connection strings (thanks jstoebel)0.3.9Restored Python 2 compatibility (thanks tokenmathguy)0.4.0Changed most non-SQL commands to argparse arguments (thanks pik)User can specify a creator for connections (thanks pik)Bogus pseudo-SQL commandPERSISTremoved, replaced with–persistargTurn off echo of connection information withdisplayconin configConsistent support for {} variables (thanks Lucas)0.4.1Fixed .rst file location in MANIFEST.inParse SQL comments in first lineBugfixes for DSN,–close, others0.5.0Use SQLAlchemy 2.0Drop undocumented support for dict-style access to raw row instances