Package: haggle
HaggleSimple exampleBy the way...Search results and dataset metadata.metaCached search infoThe boring stuffInstallAPI credentialsF.A.Q.What if I don't want a zip file anymore?Do you have any jupyter notebooks demoing this.A little snippet of code to test?Table of contents generated with markdown-tocHaggleA simple facade toKaggledata.Essentially, instantiate aKaggleDatasetsobject, and from it search datasets, see their metadata, download the data
(automatically caching it in well organized folders), and all from an interface that looks like
a humble dict withowner/datasetkeys, and that's the coolest bit.Haggle:/ˈhaɡəl/an instance of intense argument (as in bargaining)wrangle (over a price, terms of an agreement, etc.)rhymes with Kaggle and is not taken on pypi (well, now it is)Simple examplefromhaggleimportKaggleDatasetsrootdir=None# define (or not) where you want the data to be cached/downloadeds=KaggleDatasets(rootdir)# make an instanceif'rtatman/english-word-frequency'ins:dels['rtatman/english-word-frequency']# just to prepare for the demolist(s)# see what you have locally['uciml/human-activity-recognition-with-smartphones',
 'sitsawek/phonetics-articles-on-plos']Let's search something (you can also search onkaggle, I was kidding about it being lame!)results=s.search('word frequency')print(f"{len(results)=}")list(results)[:10]len(results)=180




['rtatman/english-word-frequency',
 'yekenot/fasttext-crawl-300d-2m',
 'rtatman/japanese-lemma-frequency',
 'rtatman/glove-global-vectors-for-word-representation',
 'averkij/lingtrain-hungarian-word-frequency',
 'lukevanhaezebrouck/subtlex-word-frequency',
 'facebook/fatsttext-common-crawl',
 'facebook/fasttext-wikinews',
 'facebook/fasttext-english-word-vectors-including-subwords',
 'kushtej/kannada-word-frequency']Chose what you want? Good, now do this:v=s['rtatman/english-word-frequency']type(v)py2store.slib.s_zipfile.ZipReaderOkay, let's slow down a moment. What happened? What's thisZipReaderthingy?Well, what happened is that this downloaded the zip file of the data for you and saved it inROOTDIR/rtatman/english-word-frequency.zip. Don't believe me? Go have a look.But then it also returns this object calledZipReaderthat points to it.If you don't like it, you don't have to use it. But I think you should like it.Look at what it can do!List the contents of file (that's in the zip... okay there's just one here, it's a bit boring)list(v)['unigram_freq.csv']Retrieve the data for any given file of the zip without ever having to unzip it!Oh, and still pretending to be a dict.b=v['unigram_freq.csv']print(f"b is a{type(b)}and has{len(b)}bytes")b is a <class 'bytes'> and has 4956252 bytesNow the data is given in bytes by default, since that's the basis of everything.From there you can go everywhere. Here for example, say we'd like to go topandas.DataFrame...importpandasaspdfromioimportBytesIOdf=pd.read_csv(BytesIO(b))df.shape(333333, 2)print(df.head(7).to_string())word        count
0  the  23135851162
1   of  13151942776
2  and  12997637966
3   to  12136980858
4    a   9081174698
5   in   8469404971
6  for   5933321709And as mentioned, it caches the data to your local drive. You know, download, so that the next time you ask fors['rtatman/english-word-frequency'], it'll be faster to get those bytes.See, let's list the contents ofsagain and see that we now have that'rtatman/english-word-frequency'key we didn't have before.list(s)['uciml/human-activity-recognition-with-smartphones',
 'rtatman/english-word-frequency',
 'sitsawek/phonetics-articles-on-plos']By the way...So aKaggleDatasetsis a store with a dict-like interface.Listing happens locally. Remote listing is done through.search(...).Getting happens locally first, and if not, will get remotely (and cache locally).Where are the zips stored? Ask.zips_dir:s.zips_dir'/D/Dropbox/_odata/kaggle/zips'Search results and dataset metadataLet's have a closer look at those search results. All we did is alen(results)and alist(results). What else can you do with that object?Well, as is so happens, you can do whatever (read-only) operation you can do on a -- take a wild guess -- a dict.Namely, you can get a value for the keys we've listedfrompprintimportpprintpprint(results['rtatman/english-word-frequency']){'creatorName': 'Rachael Tatman',
 'creatorUrl': 'rtatman',
 'currentVersionNumber': 1,
 'description': None,
 'downloadCount': 3079,
 'files': [],
 'id': 2367,
 'isFeatured': False,
 'isPrivate': False,
 'isReviewed': True,
 'kernelCount': 12,
 'lastUpdated': '2017-09-06T18:21:27.18Z',
 'licenseName': 'Other (specified in description)',
 'ownerName': 'Rachael Tatman',
 'ownerRef': 'rtatman',
 'ref': 'rtatman/english-word-frequency',
 'subtitle': '⅓ Million Most Frequent English Words on the Web',
 'tags': [{'competitionCount': 3,
           'datasetCount': 231,
           'description': 'Language is a method of communication that consists '
                          'of using words arranged into meaningful patterns. '
                          'This is a good place to find natural language '
                          'processing datasets and kernels to study languages '
                          'and train your chat bots.',
           'fullPath': 'topic > culture and humanities > languages',
           'isAutomatic': False,
           'name': 'languages',
           'ref': 'languages',
           'scriptCount': 77,
           'totalCount': 311},
          {'competitionCount': 6,
           'datasetCount': 445,
           'description': 'The linguistics tag contains datasets and kernels '
                          'that you can use for text analytics, sentiment '
                          'analyses, and making clever jokes like this: Let me '
                          "tell you a little about myself. It's a reflexive "
                          'pronoun that means "me."',
           'fullPath': 'topic > people and society > social science > '
                       'linguistics',
           'isAutomatic': False,
           'name': 'linguistics',
           'ref': 'linguistics',
           'scriptCount': 122,
           'totalCount': 573},
          {'competitionCount': 18,
           'datasetCount': 4172,
           'description': 'An interconnected network of tubes that connects '
                          'the entire world together. This tag covers a broad '
                          'range of tags; anything from cryptocurrency to '
                          'website analytics.',
           'fullPath': 'topic > science and technology > internet',
           'isAutomatic': False,
           'name': 'internet',
           'ref': 'internet',
           'scriptCount': 198,
           'totalCount': 4388}],
 'title': 'English Word Frequency',
 'topicCount': 1,
 'totalBytes': 2236581,
 'url': 'https://www.kaggle.com/rtatman/english-word-frequency',
 'usabilityRating': 0.8235294,
 'versions': [],
 'viewCount': 21726,
 'voteCount': 105}You get description, size, tags, download count... Useful stuff to make your choice.Personally, I like transform those results in aDataFramethat I can subsequently interrogate:importpandasaspddf=pd.DataFrame(results.values())[['ref','title','subtitle','downloadCount','totalBytes']]df=df.set_index('ref').sort_values('downloadCount',ascending=False)# print(df.head(10).to_markdown())  # markdown for first 10 rowsreftitlesubtitledownloadCounttotalBytesjealousleopard/goodreadsbooksGoodreads-bookscomprehensive list of all books listed in goodreads23640637338uciml/zoo-animal-classificationZoo Animal ClassificationUse Machine Learning Methods to Correctly Classify Animals Based Upon Attributes165971898yekenot/fasttext-crawl-300d-2mFastText crawl 300d 2M2 million word vectors trained on Common Crawl (600B tokens)82751.54555e+09rtatman/sentiment-lexicons-for-81-languagesSentiment Lexicons for 81 LanguagesSentiment Polarity Lexicons (Positive vs. Negative)79601.62176e+06rtatman/glove-global-vectors-for-word-representationGloVe: Global Vectors for Word RepresentationPre-trained word vectors from Wikipedia 2014 + Gigaword 574324.80173e+08mozillaorg/common-voiceCommon Voice500 hours of speech recordings, with speaker demographics60751.29315e+10arathee2/demonetization-in-india-twitter-dataDemonetization in India Twitter DataData extracted from Twitter regarding the recent currency demonetization5761919578eibriel/rdany-conversationsrDany Chat157 chats & 6300+ messages with a (fake) virtual companion3983916724mrisdal/2016-us-presidential-debates2016 US Presidential DebatesFull transcripts of the face-off between Clinton & Trump3920123161nobelfoundation/nobel-laureatesNobel Laureates, 1901-PresentWhich country has won the most prizes in each category?319267763.meta.metais your access to metadata about datasets.It works the same way things work with the zips of datasets: It will:list: will list locally store dataset meta information (in location specified bys.meta_dir)get: when a value (metadata dict) is requested, (1) the key is searched locally first, and if not found, (2) will request it remotely (through the kaggle api), and (3) the value will be cached (stored) locallyCached search infoWait, it's not all:KaggleDatasetswill (by default) also cache these results locally in individual json files.Where? Askmeta_dir:s.meta_dir'/D/Dropbox/_odata/kaggle/meta'You can access these files with your favorite dict-like interface, through the.metaattributelen(s.meta)358list(s.meta)[:7]['emmabel/word-occurrences-in-mr-robot',
 'bitsnpieces/covid19-country-data',
 'johnwdata/coronavirus-covid19-cases-by-us-state',
 'johnwdata/coronavirus-covid19-cases-by-us-county',
 'andradaolteanu/bing-nrc-afinn-lexicons',
 'rahulloha/covid19',
 'nltkdata/word2vec-sample']pprint(s.meta['emmabel/word-occurrences-in-mr-robot']){'creatorName': 'Emma',
 'creatorUrl': 'emmabel',
 'currentVersionNumber': 1,
 'description': None,
 'downloadCount': 116,
 'files': [],
 'id': 4288,
 'isFeatured': False,
 'isPrivate': False,
 'isReviewed': False,
 'kernelCount': 1,
 'lastUpdated': '2017-11-09T18:30:15.733Z',
 'licenseName': 'CC0: Public Domain',
 'ownerName': 'Emma',
 'ownerRef': 'emmabel',
 'ref': 'emmabel/word-occurrences-in-mr-robot',
 'subtitle': "Find out F-Society's favorite lingo",
 'tags': [{'competitionCount': 0,
           'datasetCount': 7525,
           'description': 'Activities that holds the attention and interest of '
                          'an audience, or gives pleasure and delight. It can '
                          'be an idea or a task, but is more likely to be one '
                          'of the activities or events that have developed '
                          'over thousands of years specifically for the '
                          "purpose of keeping an audience's attention.",
           'fullPath': 'topic > arts and entertainment',
           'isAutomatic': False,
           'name': 'arts and entertainment',
           'ref': 'arts and entertainment',
           'scriptCount': 35,
           'totalCount': 7560},
          {'competitionCount': 0,
           'datasetCount': 1227,
           'description': 'One of the hallmarks of intelligence is the use of '
                          'games and toys to occupy free time and develop '
                          "intellectually. Often stored in Mom's basement.",
           'fullPath': 'topic > culture and humanities > games',
           'isAutomatic': False,
           'name': 'games',
           'ref': 'games',
           'scriptCount': 40,
           'totalCount': 1267}],
 'title': 'Word Occurrences in Mr. Robot',
 'topicCount': 0,
 'totalBytes': 119466,
 'url': 'https://www.kaggle.com/emmabel/word-occurrences-in-mr-robot',
 'usabilityRating': 0.7058824,
 'versions': [],
 'viewCount': 1028,
 'voteCount': 5}So if you want to search locally for information (again, information about your searches, not your data zips!), you can get them in aDataFramelike so:>>>df=pd.DataFrame(s.meta.values())>>>df.shape(358,36)Markdown for the 10 first rows...t=df.head(10).dropna(axis=1)delt['tags']print(t.to_markdown())idrefsubtitlecreatorNamecreatorUrltotalBytesurllastUpdateddownloadCountisPrivateisReviewedisFeaturedlicenseNameownerNameownerRefkernelCounttitletopicCountviewCountvoteCountcurrentVersionNumberfilesversionsusabilityRating04288emmabel/word-occurrences-in-mr-robotFind out F-Society's favorite lingoEmmaemmabel119466https://www.kaggle.com/emmabel/word-occurrences-in-mr-robot2017-11-09T18:30:15.733Z116FalseFalseFalseCC0: Public DomainEmmaemmabel1Word Occurrences in Mr. Robot0102851[][]0.7058821576036bitsnpieces/covid19-country-dataCountry level metadata that includes temperature, COVID-19 and H1N1 cases, etc.Patrickbitsnpieces190821https://www.kaggle.com/bitsnpieces/covid19-country-data2020-05-03T23:51:55.5Z939FalseFalseFalseDatabase: Open Database, Contents: Database ContentsPatrickbitsnpieces3COVID-19 Country Data054192630[][]0.8823532575937johnwdata/coronavirus-covid19-cases-by-us-stateNYTimes Coronavirus DatasetJohn Wackerowjohnwdata82582https://www.kaggle.com/johnwdata/coronavirus-covid19-cases-by-us-state2020-09-23T12:43:05.76Z59FalseFalseFalseOther (specified in description)John Wackerowjohnwdata2Coronavirus COVID-19 Cases By US State01276383[][]13575883johnwdata/coronavirus-covid19-cases-by-us-countyNYTimes Coronavirus DatasetJohn Wackerowjohnwdata3.50819e+06https://www.kaggle.com/johnwdata/coronavirus-covid19-cases-by-us-county2020-07-23T18:47:16.543Z37FalseFalseFalseUnknownJohn Wackerowjohnwdata2Coronavirus COVID-19 Cases By US County061036[][]0.7647064507452andradaolteanu/bing-nrc-afinn-lexiconsthe lexicons are in CSV formatAndrada Olteanuandradaolteanu83965https://www.kaggle.com/andradaolteanu/bing-nrc-afinn-lexicons2020-02-09T18:39:13.343Z135FalseFalseFalseUnknownAndrada Olteanuandradaolteanu11Bing, NRC, Afinn Lexicons0629121[][]0.8823535599303rahulloha/covid19Covid-19 dataset (updates daily at 10:00 pm pst)Rahul Loharahulloha1.60678e+07https://www.kaggle.com/rahulloha/covid192020-04-12T22:50:26.857Z65FalseFalseFalseUnknownRahul Loharahulloha1Global Coronavirus (COVID-19) Data (Johns Hopkins)061561[][]0.64705962045nltkdata/word2vec-sampleSample Word2Vec ModelLiling Tanalvations1.01479e+08https://www.kaggle.com/nltkdata/word2vec-sample2017-08-20T03:14:39.623Z772FalseTrueFalseOther (specified in description)NLTK Datanltkdata3Word2Vec Sample08073151[][]0.757585107nxpnsv/country-health-indicatorsHealth indicator relevant to covid19 death and infection risknxpnsvnxpnsv40146https://www.kaggle.com/nxpnsv/country-health-indicators2020-04-07T11:12:41.91Z185FalseFalseFalseCC BY-NC-SA 4.0nxpnsvnxpnsv6country health indicators02126116[][]18585591joelhanson/coronavirus-covid19-data-in-the-united-statesThe New York Times data on coronavirus cases and deaths in the U.SJoel Hansonjoelhanson5.83676e+06https://www.kaggle.com/joelhanson/coronavirus-covid19-data-in-the-united-states2020-09-21T15:51:45.933Z164FalseFalseFalseAttribution-NonCommercial 4.0 International (CC BY-NC 4.0)Joel Hansonjoelhanson3Coronavirus (Covid-19) Data of United States (USA)01724529[][]0.88235398352vsmolyakov/fasttextembeddings with sub-word informationvsmolyakovvsmolyakov1.1168e+08https://www.kaggle.com/vsmolyakov/fasttext2017-12-31T17:32:30.94Z947FalseFalseFalseCC0: Public Domainvsmolyakovvsmolyakov14FastText0380951[][]0.875Note: If you don't want all your search results to be cached you can just specify it.s=KaggleDatasets(rootdir,cache_metas_on_search=False)# make an instanceThe boring stuffInstallpip install haggleYou'll need a kaggle api token to use thisIf you do, you probably can just start using.If you don't got get one! Go seethisfor detailed instructions, it essentially says:API credentialsTo use the Kaggle API, sign up for a Kaggle account athttps://www.kaggle.com.
Then go to the 'Account' tab of your user profile (https://www.kaggle.com/<username>/account) and select 'Create API Token'.
This will trigger the download ofkaggle.json, a file containing your API credentials.
Place this file in the location~/.kaggle/kaggle.json(on Windows in the locationC:\Users\<Windows-username>\.kaggle\kaggle.json- you can check the exact location, sans drive, withecho %HOMEPATH%).
You can define a shell environment variableKAGGLE_CONFIG_DIRto change this location to$KAGGLE_CONFIG_DIR/kaggle.json(on Windows it will be%KAGGLE_CONFIG_DIR%\kaggle.json).For your security, ensure that other users of your computer do not have read access to your credentials. On Unix-based systems you can do this with the following command:chmod 600 ~/.kaggle/kaggle.jsonYou can also choose to export your Kaggle username and token to the environment:exportKAGGLE_USERNAME=datadinosaurexportKAGGLE_KEY=xxxxxxxxxxxxxxIn addition, you can export any other configuration value that normally would be in
the$HOME/.kaggle/kaggle.jsonin the format 'KAGGLE_' (note uppercase).For example, if the file had the variable "proxy" you would exportKAGGLE_PROXYand it would be discovered by the client.F.A.Q.What if I don't want a zip file anymore?Just delete it, like you do with any file you don't want anymore. You know the one.Or... you can be cool and dodel s['owner/dataset']for that key (note a key doesn't include the rootdir or the.zipextension), just like you would with a...dict, once again.Do you have any jupyter notebooks demoing this.Sure, you can find somehere on github.A little snippet of code to test?fromhaggleimportKaggleDatasetss=KaggleDatasets()# make an instance# results = s.search('coronavirus')# ref = next(iter(results))  # see that results iteration worksref='sanchman/coronavirus-present-data'ifrefins:# delete if existsdels[ref]assertrefnotins# see, not therev=s[ref]# redownloadassertrefins# see, not thereassert'PresentData.xlsx'inset(v)# and it has stuff in vimportpandasaspdimportiodf=pd.read_excel(io.BytesIO(v['PresentData.xlsx']))assert'Total Confirmed'indf.columnsassert'Country'indf.columnsassert'France'indf['Country'].values