Package: rest-db-api
REST-DB-APIThis is a plugin to be used with apache-superset, for importing data via REST APIs.It builds on top ofshillelagh'sgeneric json adapter.To get started, just do:pipinstallrest-db-apiMotivationGeneric json adapter did not support request bodies and headers.Independence to specify http/httpsrestdialect enables this adapter to be used with apache superset.The dialect enables us to set a base URL, and query multiple endpoints with the same 'connection'.ExamplesGET requestsQueryingweather apiLets assume I am querying for 3 days weather forecast for Bangalore. The response gives the results by hour.https://api.weatherapi.com/v1/forecast.json?key={{your_key}}&q=Bangalore&days=3&aqi=no&alerts=noYou can get a free key by creating an account there.You can referthis fileto check the response structure.
We can query this withrest-db-api:fromsqlalchemyimportcreate_enginefromrest_db_api.utilsimportget_virtual_tableengine=create_engine("rest://api.weatherapi.com?ishttps=1")endpoint='/v1/forecast.json'params={'key':'your_key','q':'Bangalore','days':5}jsonpath="$.forecast.forecastday[*]"virtual_table=get_virtual_table(endpoint=endpoint,params=params,jsonpath=jsonpath)connection=engine.connect()foriinconnection.execute(f'SELECT * FROM "{virtual_table}"'):print(i)The response should return an array of objects/primitives. If not, we need to specify where in the response the array is (usingjsonpath). In this case that is at$.forecast.forecastday[*]As Shillelagh'sAdapterclass uses in memory storage -sqllite, we can query the data usingsqllitesyntax.query=f"""SELECTdate,json_extract(day, "$.maxtemp_c") as max_temp_celsius,json_extract(hour, "$[6].temp_c") as six_am_celsius,json_extract(hour, "$[6].will_it_rain") as will_it_rainFROM"{virtual_table}""""foriinconnection.execute(query):print(i)POST request with headers and request bodyConsider this sample requestcurl--location-g--requestPOST'https://some.api.com/some/api/path?a=60&c=someQuery&b=-50#$[*]'\--header'Content-Type: application/json'\--header'IAM_ID: satvik'\--header'ENVIRONMENT: staging:1.5.3'\--header'NAME: MY-REST-SERVICE'\--data-raw'{"name": "satvik","interests": [{"name": "badminton","category": "sports","stats": {"racket": "intermediate","shuttle": "yonex mavis 500"}},{"name": "programming","category": "computers","stats": {"laptop": "yw","mouse": "5D ergonomic","keyboard": "broken"}}]}'To query this with db-api, follow the snippet:fromsqlalchemyimportcreate_enginefromrest_db_api.utilsimportget_virtual_tableengine=create_engine("rest://some.api.com?ishttps=1")endpoint='/some/api/path'params={"a":60,"b":-50,"c":"someQuery"}headers={'Content-Type':'application/json','IAM_ID':'satvik','ENVIRONMENT':'staging:1.5.3','NAME':'MY-REST-SERVICE',}body={"name":"satvik","interests":[{"name":"badminton","category":"sports","stats":{"racket":"intermediate","shuttle":"yonex mavis 500"}},{"name":"programming","category":"computers","stats":{"laptop":"mac book pro","mouse":"5D ergonomic","keyboard":"broken"}}]}jsonpath="#$[*]"# set this according to your responsevirtual_table=get_virtual_table(endpoint=endpoint,params=params,headers=headers,body=body,jsonpath=jsonpath)foriinconnection.execute(f'SELECT * FROM "{virtual_table}"'):print(i)Usage with apache-supersetGo to Connect database and add databaseSelect Shillelaghadd the connection string withrest://prefix
eg:rest://api.weatherapi.com?ishttps=1Gice your connection a name: egrest-weather-apiClick test connection and then addGo to SQL lab and selectrest-weather-apifrom database.You can leave schema empty and query directly!!Query is:SELECTdate,json_extract(day,"$.maxtemp_c")asmax_temp_celsius,json_extract(hour,"$[6].temp_c")assix_am_celsius,json_extract(hour,"$[6].will_it_rain")aswill_it_rainFROM"/v1/forecast.json?key={your_key}&q=Bangalore&days=5#$.forecast.forecastday[*]";Tables and schema is empty, because there's no concept for tables in REST APIs.
It returns a default message. That message is configured inrest_api_dialect.pyGetting the virtual tableIn superset's SQL lab, we're directly using/v1/forecast.json?key={your_key}&q=Bangalore&days=5#$.forecast.forecastday[*]To get the similar virtual table address for your endpoint (it may have headers or even body), use the utilityrest_db_api.utils.get_virtual_tableand pass in your configs.POST requests (request body)headersadding write support to adapter (for PUT/DELETE requests)