Package: shillelagh
Shillelagh (ʃɪˈleɪlɪ) is a Python library and CLI that allows you to query many resources (APIs, files, in memory objects) using SQL. It’s both user and developer friendly, making it trivial to access resources and easy to add support for new ones.Learn more on thedocumentation.The library is an implementation of thePython DB API 2.0based onSQLite(using theAPSWlibrary):fromshillelagh.backends.apsw.dbimportconnectconnection=connect(":memory:")cursor=connection.cursor()query="SELECT * FROM a_table"forrowincursor.execute(query):print(row)There is also aSQLAlchemydialect:fromsqlalchemy.engineimportcreate_engineengine=create_engine("shillelagh://")connection=engine.connect()query="SELECT * FROM a_table"forrowinconnection.execute(query):print(row)And a command-line utility:$shillelaghsql>SELECT*FROMa_tableWhy SQL?Sharks have been around for a long time. They’re older than trees and the rings of Saturn, actually! The reason they haven’t changed that much in hundreds of millions of years is because they’re really good at what they do.SQL has been around for some 50 years for the same reason: it’s really good at what it does.Why “Shillelagh”?Picture a leprechaun hitting APIs with a big stick so that they accept SQL.How is it different?Shillelagh allows you to easily query non-SQL resources. For example, if you have aGoogle Spreadsheetyou can query it directly as if it were a table in a database:SELECTcountry,SUM(cnt)FROM"https://docs.google.com/spreadsheets/d/1_rN3lm0R_bU3NemO0s9pbFkY5LQPcuy1pscv8ZXPtg8/edit#gid=0"WHEREcnt>0GROUPBYcountryYou can even runINSERT/DELETE/UPDATEqueries against the spreadsheet:UPDATE"https://docs.google.com/spreadsheets/d/1_rN3lm0R_bU3NemO0s9pbFkY5LQPcuy1pscv8ZXPtg8/edit#gid=0"SETcnt=cnt+1WHEREcountry!='BR'Queries like this are supported byadapters. Currently Shillelagh has the following adapters:NameTypeURI patternExample URICSVFile/API/path/to/file.csv;http(s)://*/home/user/sample_data.csvDatasetteAPIhttp(s)://*https://global-power-plants.datasettes.com/global-power-plants/global-power-plantsGeneric JSONAPIhttp(s)://*https://api.stlouisfed.org/fred/series?series_id=GNPCA&api_key=XXX&file_type=json#$.seriess[*]Generic XMLAPIhttp(s)://*https://api.congress.gov/v3/bill/118?format=xml&offset=0&limit=2&api_key=XXX#.//billGitHubAPIhttps://api.github.com/repos/${owner}/{$repo}/pullshttps://api.github.com/repos/apache/superset/pullsGSheetsAPIhttps://docs.google.com/spreadsheets/d/${id}/edit#gid=${sheet_id}https://docs.google.com/spreadsheets/d/1LcWZMsdCl92g7nA-D6qGRqg1T5TiHyuKJUY1u9XAnsk/edit#gid=0HTML tableAPIhttp(s)://*https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_populationPandasIn memoryAny variable name (local or global)my_dfS3APIs3://bucket/path/to/files3://shillelagh/sample_data.csvSocrataAPIhttps://${domain}/resource/${dataset-id}.jsonhttps://data.cdc.gov/resource/unsk-b7fc.jsonSystemAPIsystem://${resource}system://cpu?interval=2WeatherAPIAPIhttps://api.weatherapi.com/v1/history.json?key=${key}&q=${location}https://api.weatherapi.com/v1/history.json?key=XXX&q=LondonThere are also 3rd-party adapters:AirtableGraphQLA query can combine data from multiple adapters:INSERTINTO"/tmp/file.csv"SELECTtime,chance_of_rainFROM"https://api.weatherapi.com/v1/history.json?q=London"WHEREtimeIN(SELECTdatetimeFROM"https://docs.google.com/spreadsheets/d/1_rN3lm0R_bU3NemO0s9pbFkY5LQPcuy1pscv8ZXPtg8/edit#gid=1648320094")The query above reads timestamps from a Google sheet, uses them to filter weather data fromWeatherAPI, and writes the chance of rain into a (pre-existing) CSV file.New adapters are relatively easy to implement. There’s astep-by-step tutorialthat explains how to create a new adapter to an API or filetype.InstallationInstall Shillelagh withpip:$pipinstall'shillelagh'You also need to install optional dependencies, depending on the adapter you want to use:$pipinstall'shillelagh[console]'# to use the CLI$pipinstall'shillelagh[genericjsonapi]'# for Generic JSON$pipinstall'shillelagh[genericxmlapi]'# for Generic XML$pipinstall'shillelagh[githubapi]'# for GitHub$pipinstall'shillelagh[gsheetsapi]'# for GSheets$pipinstall'shillelagh[htmltableapi]'# for HTML tables$pipinstall'shillelagh[pandasmemory]'# for Pandas in memory$pipinstall'shillelagh[s3selectapi]'# for S3 files$pipinstall'shillelagh[systemapi]'# for CPU informationAlternatively, you can install everything with:$pipinstall'shillelagh[all]'