Package: datadotworld
A python library for working with data.world datasets.This library makes it easy for data.world users to pull and work with data stored on data.world.
Additionally, the library provides convenient wrappers for data.world APIs, allowing users to create and update
datasets, add and modify files, etc, and possibly implement entire apps on top of data.world.Quick startInstallYou can install it usingpipdirectly from PyPI:pip install datadotworldOptionally, you can install the library including pandas support:pip install datadotworld[pandas]If you usecondato manage your python distribution, you can install from the community-maintained [conda-forge](https://conda-forge.github.io/) channel:conda install -c conda-forge datadotworld-pyConfigureThis library requires a data.world API authentication token to work.Your authentication token can be obtained on data.world once you enable Python underIntegrations > PythonTo configure the library, run the following command:dw configureAlternatively, tokens can be provided via theDW_AUTH_TOKENenvironment variable.
On MacOS or Unix machines, run (replacing<YOUR_TOKEN>>below with the token obtained earlier):export DW_AUTH_TOKEN=<YOUR_TOKEN>Load a datasetTheload_dataset()function facilitates maintaining copies of datasets on the local filesystem.
It will download a given dataset’sdatapackageand store it under~/.dw/cache. When used subsequently,load_dataset()will use the copy stored on disk and will
work offline, unless it’s called withforce_update=Trueorauto_update=True.force_update=Truewill overwrite your local copy unconditionally.auto_update=Truewill only overwrite your local copy if a newer version of the dataset is available on data.world.Once loaded, a dataset (data and metadata) can be conveniently accessed via the object returned byload_dataset().Start by importing thedatadotworldmodule:importdatadotworldasdwThen, invoke theload_dataset()function, to download a dataset and work with it locally.
For example:intro_dataset=dw.load_dataset('jonloyens/an-intro-to-dataworld-dataset')Dataset objects allow access to data via three different propertiesraw_data,tablesanddataframes.
Each of these properties is a mapping (dict) whose values are of typebytes,listandpandas.DataFrame,
respectively. Values are lazy loaded and cached once loaded. Their keys are the names of the files
contained in the dataset.For example:>>>intro_dataset.dataframesLazyLoadedDict({'changelog':LazyLoadedValue(<pandas.DataFrame>),'datadotworldbballstats':LazyLoadedValue(<pandas.DataFrame>),'datadotworldbballteam':LazyLoadedValue(<pandas.DataFrame>)})IMPORTANT: Not all files in a dataset are tabular, therefore some will be exposed viaraw_dataonly.Tables are lists of rows, each represented by a mapping (dict) of column names to their respective values.For example:>>>stats_table=intro_dataset.tables['datadotworldbballstats']>>>stats_table[0]OrderedDict([('Name','Jon'),('PointsPerGame',Decimal('20.4')),('AssistsPerGame',Decimal('1.3'))])You can also review the metadata associated with a file or the entire dataset, using thedescribefunction.
For example:>>>intro_dataset.describe(){'homepage':'https://data.world/jonloyens/an-intro-to-dataworld-dataset','name':'jonloyens_an-intro-to-dataworld-dataset','resources':[{'format':'csv','name':'changelog','path':'data/ChangeLog.csv'},{'format':'csv','name':'datadotworldbballstats','path':'data/DataDotWorldBBallStats.csv'},{'format':'csv','name':'datadotworldbballteam','path':'data/DataDotWorldBBallTeam.csv'}]}>>>intro_dataset.describe('datadotworldbballstats'){'format':'csv','name':'datadotworldbballstats','path':'data/DataDotWorldBBallStats.csv','schema':{'fields':[{'name':'Name','title':'Name','type':'string'},{'name':'PointsPerGame','title':'PointsPerGame','type':'number'},{'name':'AssistsPerGame','title':'AssistsPerGame','type':'number'}]}}Query a datasetThequery()function allows datasets to be queried live usingSQLorSPARQLquery languages.To query a dataset, invoke thequery()function.
For example:results=dw.query('jonloyens/an-intro-to-dataworld-dataset','SELECT * FROM DataDotWorldBBallStats')Query result objects allow access to the data viaraw_data,tableanddataframeproperties, of typejson,listandpandas.DataFrame, respectively.For example:>>>results.dataframeNamePointsPerGameAssistsPerGame0Jon20.41.31Rob15.58.02Sharon30.111.23Alex8.20.54Rebecca12.317.05Ariane18.13.06Bryon16.08.57Matt13.02.1Tables are lists of rows, each represented by a mapping (dict) of column names to their respective values.
For example:>>>results.table[0]OrderedDict([('Name','Jon'),('PointsPerGame',Decimal('20.4')),('AssistsPerGame',Decimal('1.3'))])To query usingSPARQLinvokequery()usingquery_type='sparql', or else, it will assume
the query to be aSQLquery.Just like in the dataset case, you can view the metadata associated with a query result using thedescribe()function.For example:>>>results.describe(){'fields':[{'name':'Name','type':'string'},{'name':'PointsPerGame','type':'number'},{'name':'AssistsPerGame','type':'number'}]}Work with filesTheopen_remote_file()function allows you to write data to or read data from a file in a
data.world dataset.Writing filesThe object that is returned from theopen_remote_file()call is similar to a file handle that
would be used to write to a local file - it has awrite()method, and contents sent to that
method will be written to the file remotely.>>>importdatadotworldasdw>>>>>>withdw.open_remote_file('username/test-dataset','test.txt')asw:...w.write("this is a test.")>>>Of course, writing a text file isn’t the primary use case for data.world - you want to write your
data!  The return object fromopen_remote_file()should be usable anywhere you could normally
use a local file handle in write mode - so you can use it to serialize the contents of a PANDASDataFrameto a CSV file…>>>importpandasaspd>>>df=pd.DataFrame({'foo':[1,2,3,4],'bar':['a','b','c','d']})>>>withdw.open_remote_file('username/test-dataset','dataframe.csv')asw:...df.to_csv(w,index=False)Or, to write a series ofdictobjects as a JSON Lines file…>>>importjson>>>withdw.open_remote_file('username/test-dataset','test.jsonl')asw:...json.dump({'foo':42,'bar':"A"},w)...json.dump({'foo':13,'bar':"B"},w)>>>Or to write a series ofdictobjects as a CSV…>>>importcsv>>>withdw.open_remote_file('username/test-dataset','test.csv')asw:...csvw=csv.DictWriter(w,fieldnames=['foo','bar'])...csvw.writeheader()...csvw.writerow({'foo':42,'bar':"A"})...csvw.writerow({'foo':13,'bar':"B"})>>>And finally, you can write binary data by streamingbytesorbytearrayobjects, if you open the
file in binary mode…>>>withdw.open_remote_file('username/test-dataset','test.txt',mode='wb')asw:...w.write(bytes([100,97,116,97,46,119,111,114,108,100]))Reading filesYou can also read data from a file in a similar fashion>>>withdw.open_remote_file('username/test-dataset','test.txt',mode='r')asr:...print(r.read)Reading from the file into common parsing libraries works naturally, too - when opened in ‘r’ mode, the
file object acts as an Iterator of the lines in the file:>>>withdw.open_remote_file('username/test-dataset','test.txt',mode='r')asr:...csvr=csv.DictReader(r)...forrowincsvr:...print(row['column a'],row['column b'])Reading binary files works naturally, too - when opened in ‘rb’ mode,read()returns the contents of
the file as a byte array, and the file object acts as an iterator of bytes:>>>withdw.open_remote_file('username/test-dataset','test',mode='rb')asr:...bytes=r.read()Additional API FeaturesFor a complete list of available API operations, seeofficial documentation.Python wrappers are implemented by theApiClientclass. To obtain an instance, simply callapi_client.
For example:client=dw.api_clientThe client currently implements the following functions:create_datasetupdate_datasetreplace_datasetget_datasetdelete_datasetadd_files_via_urlappend_recordsupload_filesupload_filedelete_filessync_filesdownload_datasetdownload_fileget_user_datafetch_contributing_datasetsfetch_liked_datasetsfetch_datasetsfetch_contributing_projectsfetch_liked_projectsfetch_projectsget_projectcreate_projectupdate_projectreplace_projectadd_linked_datasetremove_linked_datasetdelete_projectget_insightget_insights_for_projectcreate_insightreplace_insightupdate_insightdelete_insightsearch_resourcescreate_new_tablescreate_new_connectionsFor a few examples of what theApiClientcan be used for, see below.Add files from URLTheadd_files_via_url()function can be used to add files to a dataset from a URL.
This can be done by specifyingfilesas a dictionary where the keys are the desired file name and each item is an object containingurl,descriptionandlabels.For example:>>>client=dw.api_client()>>>client.add_files_via_url('username/test-dataset',files={'sample.xls':{'url':'http://www.sample.com/sample.xls','description':'sample doc','labels':['raw data']}})Append records to streamTheappend_record()function allows you to append JSON data to a data stream associated with a dataset. Streams do not need to be created in advance. Streams are automatically created the first time astreamIdis used in an append operation.For example:>>>client=dw.api_client()>>>client.append_records('username/test-dataset','streamId',{'data':'data'})Contents of a stream will appear as part of the respective dataset as a .jsonl file.You can find more about those functions usinghelp(client)