Package: tapyoca
tapyocaA medley of small projectsparquet_deformationsI'm calling theseParquet deformationsbut purest would lynch me.Really, I just wanted to transform one word into another word, gradually, as I've seen in some ofEscher'swork, so I looked it up, and saw that it's called parquet deformations. The math looked enticing, but I had no time for that, so I did the first way I could think of: Mapping pixels to pixels (in some fashion -- but nearest neighbors is the method that yields nicest results, under the pixel-level restriction).Of course, this can be applied to any image (that will be transformed to B/W (not even gray -- I mean actual B/W), and there's several ways you can perform the parquet (I like the gif rendering).The main function (exposed as a script) ismk_deformation_image. All you need is to specify two images (or words). If you want, of course, you can specify:n_steps: Number of steps from start to end imagesave_to_file: path to file to save too (if not given, will just return the image object)kind: 'gif', 'horizontal_stack', or 'vertical_stack'coordinate_mapping_maker: A function that will return the mapping between start and end.
This function should return a pair (from_coord,to_coord) of aligned matrices whose 2 columns are the the(x, y)coordinates, and the rows represent aligned positions that should be mapped.ExamplesTwo words...fit_to_size=400start_im=image_of_text('sensor').rotate(90,expand=1)end_im=image_of_text('meaning').rotate(90,expand=1)start_and_end_image(start_im,end_im)im=mk_deformation_image(start_im,end_im,15,kind='h').resize((500,200))imim=mk_deformation_image(start_im.transpose(4),end_im.transpose(4),5,kind='v').resize((200,200))imf='sensor_meaning_knn.gif'mk_deformation_image(start_im.transpose(4),end_im.transpose(4),n_steps=20,save_to_file=f)display_gif(f)f='sensor_meaning_scan.gif'mk_deformation_image(start_im.transpose(4),end_im.transpose(4),n_steps=20,save_to_file=f,coordinate_mapping_maker='scan')display_gif(f)f='sensor_meaning_random.gif'mk_deformation_image(start_im.transpose(4),end_im.transpose(4),n_steps=20,save_to_file=f,coordinate_mapping_maker='random')display_gif(f)From a list of wordsstart_words=['sensor','vibration','tempature']end_words=['sense','meaning','detection']start_im,end_im=make_start_and_end_images_with_words(start_words,end_words,perm=True,repeat=2,size=150)start_and_end_image(start_im,end_im).resize((600,200))im=mk_deformation_image(start_im,end_im,5)imf='bunch_of_words.gif'mk_deformation_image(start_im,end_im,n_steps=20,save_to_file=f)display_gif(f)From filesstart_im=Image.open('sensor_strip_01.png')end_im=Image.open('sense_strip_01.png')start_and_end_image(start_im.resize((200,500)),end_im.resize((200,500)))im=mk_deformation_image(start_im,end_im,7)imf='medley.gif'mk_deformation_image(start_im,end_im,n_steps=20,save_to_file=f)display_gif(f)mk_deformation_image(start_im,end_im,n_steps=20,save_to_file=f,coordinate_mapping_maker='scan')display_gif(f)an image and some textstart_im='img/waveform_01.png'# will first look for a file, and if not consider as textend_im='makes sense'mk_gif_of_deformations(start_im,end_im,n_steps=20,save_to_file='image_and_text.gif')display_gif('image_and_text.gif')demonysWhat do we think about other peoples?This project is meant to get an idea of what people think of people for different nations, as seen by what they ask google about them.Here I use python code to acquire, clean up, and analyze the data.DemonymIf you're like me and enjoy the false and fleeting impression of superiority that comes when you know a word someone else doesn't. If you're like me and go to parties for the sole purpose of seeking victims to get a one-up on, here's a cool word to add to your arsenal:demonym: a noun used to denote the natives or inhabitants of a particular country, state, city, etc."he struggled for the correct demonym for the people of Manchester"Back-story of this analysisDuring a discussion (about traveling in Europe) someone said "why are the swiss so miserable". Now, I wouldn't say that the swiss were especially miserable (a couple of ex-girlfriends aside), but to be fair he was contrasting with Italians, so perhaps he has a point. I apologize if you are swiss, or one of the two ex-girlfriends -- nothing personal, this is all for effect.We googled "why are the swiss so ", and sure enough, "why are the swiss so miserable" came up as one of the suggestions. So we got curious and started googling other peoples: the French, the Germans, etc.That's the back-story of this analysis. This analysis is meant to get an idea of what we think of peoples from other countries. Of course, one can rightfully critique the approach I'll take to gauge "what we think" -- all three of these words should, but will not, be defined. I'm just going to see what google'scurrentauto-suggest comes back with when I enter "why are the X so " (where X will be a noun that denotes the natives of inhabitants of a particular country; ademonymif you will).WarningAgain, word of warning: All data and analyses are biased.
Take everything you'll read here (and to be fair, what you read anywhere) with a grain of salt.
For simplicitly I'll saying things like "what we think of..." or "who do we most...", etc.
But I don'treallymean that.Resourceshttp://www.geography-site.co.uk/pages/countries/demonyms.htmlfor my list of demonyms.google for my suggestion engine, using the url prefix:http://suggestqueries.google.com/complete/search?client=chrome&q=The resultsIn a nutshellBelow is listed 73 demonyms along with words extracted from the very first google suggestion when you type.why are the DEMONYM soafghan    	                eyes beautiful
albanian  	                     beautiful
american  	          girl dolls expensive
australian	                          tall
belgian   	                    fries good
bhutanese 	                         happy
brazilian 	              good at football
british   	     full of grief and despair
bulgarian 	              properties cheap
burmese   	             cats affectionate
cambodian 	                   cows skinny
canadian  	                          nice
chinese   	                       healthy
colombian 	                  avocados big
cuban     	                   cigars good
czech     	                          tall
dominican 	  republic and haiti different
egyptian  	                gods important
english   	                      reserved
eritrean  	                     beautiful
ethiopian 	                     beautiful
filipino  	                         proud
finn      	               shoes expensive
french    	                       healthy
german    	                          tall
greek     	                gods messed up
haitian   	                parents strict
hungarian 	                    words long
indian    	            tv debates chaotic
indonesian	                         smart
iranian   	                     beautiful
israeli   	           startups successful
italian   	                         short
jamaican  	                sprinters fast
japanese  	                        polite
kenyan    	                  runners good
lebanese  	                          rich
malagasy  	                    names long
malaysian 	                   drivers bad
maltese   	                          rude
mongolian 	                  horses small
moroccan  	                rugs expensive
nepalese  	                     beautiful
nigerian  	                          tall
north korean	                      hats big
norwegian 	                 flights cheap
pakistani 	                          fair
peruvian  	               blueberries big
pole      	                  vaulters hot
portuguese	                         short
puerto rican	       and cuban flags similar
romanian  	                     beautiful
russian   	                  good at math
samoan    	                           big
saudi     	                      arrogant
scottish  	                        bitter
senegalese	                          tall
serbian   	                          tall
singaporean	                          rude
somali    	                parents strict
south african	                     plugs big
south korean	                          tall
sri lankan	                          dark
sudanese  	                          tall
swiss     	        good at making watches
syrian    	                families large
taiwanese 	                        pretty
thai      	                        pretty
tongan    	                           big
ukrainian 	                     beautiful
vietnamese	        fiercely nationalistic
welsh     	                          dark
zambian   	                emeralds cheapNotes:The queries actually have a space after the "so", which matters so as to omit suggestions containing words that start with so.Only the tail of the suggestion is shown -- minus prefix (why are the DEMONYMorwhy are DEMONYM) as well as theso, where ever it lands in the suggestion.
For example, the first suggestion for the american demonym was "why are american dolls so expensive", which results in the "dolls expensive" association.Who do we most talk/ask about?The original list contained 217 demonyms, but many of these yielded no suggestions (to the specific query format I used, that is).
Only 73 demonyms gave me at least one suggestion.
But within those, number of suggestions range between 1 and 20 (which is probably the default maximum number of suggestions for the API I used).
So, pretending that the number of suggestions is an indicator of how much we have to say, or how many different opinions we have, of each of the covered nationalities,
here's the top 15 demonyms people talk about, with the corresponding number of suggestions
(proxy for "the number of different things people ask about the said nationality).french         20
singaporean    20
german         20
british        20
swiss          20
english        19
italian        18
cuban          18
canadian       18
welsh          18
australian     17
maltese        16
american       16
japanese       14
scottish       14Who do we least talk/ask about?Conversely, here are the 19 demonyms that came back with only one suggestion.somali          1
bhutanese       1
syrian          1
tongan          1
cambodian       1
malagasy        1
saudi           1
serbian         1
czech           1
eritrean        1
finn            1
puerto rican    1
pole            1
haitian         1
hungarian       1
peruvian        1
moroccan        1
mongolian       1
zambian         1What do we think about people?Why are the French so...How would you (if you're (un)lucky enough to know the French) finish this sentence?
You might even have several opinions about the French, and any other group of people you've rubbed shoulders with.
What words would your palette contain to describe different nationalities?
What words would others (at least those that ask questions to google) use?Well, here's what my auto-suggest search gave me. A set of 357 unique words and expressions to describe the 72 nationalities.
So a long tail of words use only for one nationality. But some words occur for more than one nationality.
Here are the top 12 words/expressions used to describe people of the world.beautiful         11
tall              11
short              9
names long         8
proud              8
parents strict     8
smart              8
nice               7
boring             6
rich               5
dark               5
successful         5Who is beautiful? Who is tall? Who is short? Who is smart?beautiful      : albanian, eritrean, ethiopian, filipino, iranian, lebanese, nepalese, pakistani, romanian, ukrainian, vietnamese
tall           : australian, czech, german, nigerian, pakistani, samoan, senegalese, serbian, south korean, sudanese, taiwanese
short          : filipino, indonesian, italian, maltese, nepalese, pakistani, portuguese, singaporean, welsh
names long     : indian, malagasy, nigerian, portuguese, russian, sri lankan, thai, welsh
proud          : albanian, ethiopian, filipino, iranian, lebanese, portuguese, scottish, welsh
parents strict : albanian, ethiopian, haitian, indian, lebanese, pakistani, somali, sri lankan
smart          : indonesian, iranian, lebanese, pakistani, romanian, singaporean, taiwanese, vietnamese
nice           : canadian, english, filipino, nepalese, portuguese, taiwanese, thai
boring         : british, english, french, german, singaporean, swiss
rich           : lebanese, pakistani, singaporean, taiwanese, vietnamese
dark           : filipino, senegalese, sri lankan, vietnamese, welsh
successful     : chinese, english, japanese, lebanese, swissHow did I do it?I scraped a list of (country, demonym) pairs from a table inhttp://www.geography-site.co.uk/pages/countries/demonyms.html.Then I diagnosed these and manually made a mapping to simplify some "complex" entries,
such as mapping an entry such as "Irishman or Irishwoman or Irish" to "Irish".Using the google suggest API (http://suggestqueries.google.com/complete/search?client=chrome&q=), I requested what the suggestions
forwhy are the $demonym soquery pattern, for$demonymrunning through all 217 demonyms from the list above,
storing the results for each if the results were non-empty.Then, it was just a matter of pulling this data into memory, formatting it a bit, and creating a pandas dataframe that I could then interrogate.Resources you can find hereThe code to do this analysis yourself, from scratch here:data_acquisition.py.The jupyter notebook I actually used when I developed this:01 - Demonyms and adjectives - why are the french so....ipynbNote you'll need to pip install py2store if you haven't already.In thedatafolder you'll findcountry_demonym.p: A pickle of a dataframe of countries and corresponding demonymscountry_demonym.xlsx: The same as above, but in excel formdemonym_suggested_characteristics.p: A pickle of 73 demonyms and auto-suggestion information, including characteristics.what_we_think_about_demonyns.xlsx: An excel containing various statistics about demonyms and their (perceived) characteristicsAgglutinationsInspired from atweetfrom Raymond Hettinger this morning:Resist the urge to elide the underscore in multiword function or method namesSo I wondered...GluglusThe gluglu of a word is the number of partitions you can make of that word into words (of length at least 2 (so no using a or i)).
(No "gluglu" isn't an actual term -- unless everyone starts using it from now on.
But it was inspired from an actuallinguistic term.)For example, the gluglu ofnewspaperis 4:newspaper
    new spa per
    news pa per
    news paperEvery (valid) word has gluglu at least 1.How many standard library names have gluglus at last 2?108Here'sthe listof all of them.The winner has a gluglu of 6 (not 7 because formatannotationrelativeto isn't in the dictionary)formatannotationrelativeto
	for mat an not at ion relative to
	for mat annotation relative to
	form at an not at ion relative to
	form at annotation relative to
	format an not at ion relative to
	format annotation relative toDetailsDictionaryReally it depends on what dictionary we use.
Here, I used a very conservative one.
The intersection of two lists: Thecorncoband thegoogle10000word lists.
Additionally, I only kept of those, those that had at least 2 letters, and had only letters (no hyphens or disturbing diacritics).Diacritics. Look it up. Impress your next nerd date.Im left with 8116 words. You can find themhere.Standard Lib NamesSurprisingly, that was the hardest part. I know I'm missing some, but that's enough rabbit-holing.What I did (modulo some exceptions I won't look into) was to walk the standard lib modules (even that list wasn't a given!)
extracting (recursively( the names of any (non-underscored) attributes if they were modules or callables,
as well as extracting the arguments of these callables (when they had signatures).You can find the code I used to extract these nameshereand the actual listthere.covidBar Chart Races (applied to covid-19 spread)The module will show is how to make these:Confirmed cases (by country):https://public.flourish.studio/visualisation/1704821/Deaths (by country):https://public.flourish.studio/visualisation/1705644/US Confirmed cases (by state):https://public.flourish.studio/visualisation/1794768/US Deaths (by state):https://public.flourish.studio/visualisation/1794797/The scriptIf you just want to run this as a script to get the job done, you have one here:https://raw.githubusercontent.com/thorwhalen/tapyoca/master/covid/covid_bar_chart_race.pyRun like this$ python covid_bar_chart_race.py -h
usage: covid_bar_chart_race.py [-h] {mk-and-save-covid-data,update-covid-data,instructions-to-make-bar-chart-race} ...

positional arguments:
  {mk-and-save-covid-data,update-covid-data,instructions-to-make-bar-chart-race}
    mk-and-save-covid-data
                        :param data_sources: Dirpath or py2store Store where the data is :param kinds: The kinds of data you want to compute and save :param
                        skip_first_days: :param verbose: :return:
    update-covid-data   update the coronavirus data
    instructions-to-make-bar-chart-race

optional arguments:
  -h, --help            show this help message and exitThe jupyter notebookThe notebook (the .ipynb file) shows you how to do it step by step in case you want to reuse the methods for other stuff.Getting and preparing the dataCorona virus data here:https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset(direct download:https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset/download). It's currently updated daily, so download a fresh copy if you want.Population data here:http://api.worldbank.org/v2/en/indicator/SP.POP.TOTL?downloadformat=csvIt comes under the form of a zip file (currently namednovel-corona-virus-2019-dataset.zipwith several.csvfiles in them. We usepy2store(To install:pip install py2store. Project lives here:https://github.com/i2mint/py2store) to access and pre-prepare it. It allows us to not have to unzip the file and replace the older folder with it every time we download a new one. It also gives us the csvs aspandas.DataFramealready.importpandasaspdfromioimportBytesIOfrompy2storeimportkv_wrap,ZipReader# google it and pip install itfrompy2store.cachingimportmk_cached_storefrompy2storeimportQuickPickleStorefrompy2store.sourcesimportFuncReaderdefcountry_flag_image_url():importpandasaspdreturnpd.read_csv('https://raw.githubusercontent.com/i2mint/examples/master/data/country_flag_image_url.csv')defkaggle_coronavirus_dataset():importkagglefromioimportBytesIO# didn't find the pure binary download function, so using temp dir to emulatefromtempfileimportmkdtempdownload_dir=mkdtemp()filename='novel-corona-virus-2019-dataset.zip'zip_file=os.path.join(download_dir,filename)dataset='sudalairajkumar/novel-corona-virus-2019-dataset'kaggle.api.dataset_download_files(dataset,download_dir)withopen(zip_file,'rb')asfp:b=fp.read()returnBytesIO(b)defcity_population_in_time():importpandasaspdreturnpd.read_csv('https://gist.githubusercontent.com/johnburnmurdoch/''4199dbe55095c3e13de8d5b2e5e5307a/raw/fa018b25c24b7b5f47fd0568937ff6c04e384786/city_populations')defcountry_flag_image_url_prep(df:pd.DataFrame):# delete the region col (we don't need it)deldf['region']# rewriting a few (not all) of the country names to match those found in kaggle covid data# Note: The list is not complete! Add to it as neededold_and_new=[('USA','US'),('Iran, Islamic Rep.','Iran'),('UK','United Kingdom'),('Korea, Rep.','Korea, South')]forold,newinold_and_new:df['country']=df['country'].replace(old,new)returndf@kv_wrap.outcoming_vals(lambdax:pd.read_csv(BytesIO(x)))# this is to format the data as a dataframeclassZippedCsvs(ZipReader):pass# equivalent to ZippedCsvs = kv_wrap.outcoming_vals(lambda x: pd.read_csv(BytesIO(x)))(ZipReader)# Enter here the place you want to cache your datamy_local_cache=os.path.expanduser('~/ddir/my_sources')CachedFuncReader=mk_cached_store(FuncReader,QuickPickleStore(my_local_cache))data_sources=CachedFuncReader([country_flag_image_url,kaggle_coronavirus_dataset,city_population_in_time])list(data_sources)['country_flag_image_url',
 'kaggle_coronavirus_dataset',
 'city_population_in_time']covid_datasets=ZippedCsvs(data_sources['kaggle_coronavirus_dataset'])list(covid_datasets)['COVID19_line_list_data.csv',
 'COVID19_open_line_list.csv',
 'covid_19_data.csv',
 'time_series_covid_19_confirmed.csv',
 'time_series_covid_19_confirmed_US.csv',
 'time_series_covid_19_deaths.csv',
 'time_series_covid_19_deaths_US.csv',
 'time_series_covid_19_recovered.csv']covid_datasets['time_series_covid_19_confirmed.csv'].head()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>Province/StateCountry/RegionLatLong1/22/201/23/201/24/201/25/201/26/201/27/20...3/24/203/25/203/26/203/27/203/28/203/29/203/30/203/31/204/1/204/2/200NaNAfghanistan33.000065.0000000000...7484941101101201701742372731NaNAlbania41.153320.1683000000...1231461741861972122232432592772NaNAlgeria28.03391.6596000000...2643023674094545115847168479863NaNAndorra42.50631.5218000000...1641882242673083343703763904284NaNAngola-11.202717.8739000000...33445777885 rows × 76 columnscountry_flag_image_url=data_sources['country_flag_image_url']country_flag_image_url.head()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countryregionflag_image_url0AngolaAfricahttps://www.countryflags.io/ao/flat/64.png1BurundiAfricahttps://www.countryflags.io/bi/flat/64.png2BeninAfricahttps://www.countryflags.io/bj/flat/64.png3Burkina FasoAfricahttps://www.countryflags.io/bf/flat/64.png4BotswanaAfricahttps://www.countryflags.io/bw/flat/64.pngfromIPython.displayimportImageflag_image_url_of_country=country_flag_image_url.set_index('country')['flag_image_url']Image(url=flag_image_url_of_country['Australia'])Update coronavirus data# To update the coronavirus data:defupdate_covid_data(data_sources):"""update the coronavirus data"""if'kaggle_coronavirus_dataset'indata_sources._caching_store:deldata_sources._caching_store['kaggle_coronavirus_dataset']# delete the cached item_=data_sources['kaggle_coronavirus_dataset']# update_covid_data(data_sources)  # uncomment here when you want to updatePrepare data for flourish uploadimportredefprint_if_verbose(verbose,*args,**kwargs):ifverbose:print(*args,**kwargs)defcountry_data_for_data_kind(data_sources,kind='confirmed',skip_first_days=0,verbose=False):"""kind can be 'confirmed', 'deaths', 'confirmed_US', 'confirmed_US', 'recovered'"""covid_datasets=ZippedCsvs(data_sources['kaggle_coronavirus_dataset'])df=covid_datasets[f'time_series_covid_19_{kind}.csv']# df = s['time_series_covid_19_deaths.csv']if'Province/State'indf.columns:df.loc[df['Province/State'].isna(),'Province/State']='n/a'# to avoid problems arising from NaNsprint_if_verbose(verbose,f"Before data shape:{df.shape}")# drop some columns we don't needp=re.compile('\d+/\d+/\d+')assertall(isinstance(x,str)forxindf.columns)date_cols=[xforxindf.columnsifp.match(x)]ifnotkind.endswith('US'):df=df.loc[:,['Country/Region']+date_cols]# group countries and sum up the contributions of their states/regions/pargsdf['country']=df.pop('Country/Region')df=df.groupby('country').sum()else:df=df.loc[:,['Province_State']+date_cols]df['state']=df.pop('Province_State')df=df.groupby('state').sum()print_if_verbose(verbose,f"After data shape:{df.shape}")df=df.iloc[:,skip_first_days:]ifnotkind.endswith('US'):# Joining with the country image urls and saving as an xlscountry_image_url=country_flag_image_url_prep(data_sources['country_flag_image_url'])t=df.copy()t.columns=[str(x)[:10]forxint.columns]t=t.reset_index(drop=False)t=country_image_url.merge(t,how='outer')t=t.set_index('country')df=telse:passreturndfdefmk_and_save_country_data_for_data_kind(data_sources,kind='confirmed',skip_first_days=0,verbose=False):t=country_data_for_data_kind(data_sources,kind,skip_first_days,verbose)filepath=f'country_covid_{kind}.xlsx't.to_excel(filepath)print_if_verbose(verbose,f"Was saved here:{filepath}")# for kind in ['confirmed', 'deaths', 'recovered', 'confirmed_US', 'deaths_US']:forkindin['confirmed','deaths','recovered','confirmed_US','deaths_US']:mk_and_save_country_data_for_data_kind(data_sources,kind=kind,skip_first_days=39,verbose=True)Before data shape: (262, 79)
After data shape: (183, 75)
Was saved here: country_covid_confirmed.xlsx
Before data shape: (262, 79)
After data shape: (183, 75)
Was saved here: country_covid_deaths.xlsx
Before data shape: (248, 79)
After data shape: (183, 75)
Was saved here: country_covid_recovered.xlsx
Before data shape: (3253, 86)
After data shape: (58, 75)
Was saved here: country_covid_confirmed_US.xlsx
Before data shape: (3253, 87)
After data shape: (58, 75)
Was saved here: country_covid_deaths_US.xlsxUpload to Flourish, tune, and publishGo tohttps://public.flourish.studio/, get a free account, and play.Got tohttps://app.flourish.studio/templatesChoose "Bar chart race". At the time of writing this, it was here:https://app.flourish.studio/visualisation/1706060/... and then play with the settingsDiscussion of the methodsfrompy2storeimport*fromIPython.displayimportImagecountry flags imagesThe manual data prep looks something like this.importpandasaspd# get the csv data from the urlcountry_image_url_source=\'https://raw.githubusercontent.com/i2mint/examples/master/data/country_flag_image_url.csv'country_image_url=pd.read_csv(country_image_url_source)# delete the region col (we don't need it)delcountry_image_url['region']# rewriting a few (not all) of the country names to match those found in kaggle covid data# Note: The list is not complete! Add to it as needed# TODO: (Wishful) Using a general smart soft-matching algorithm to do this automatically.# TODO:    This could use edit-distance, synonyms, acronym generation, etc.old_and_new=[('USA','US'),('Iran, Islamic Rep.','Iran'),('UK','United Kingdom'),('Korea, Rep.','Korea, South')]forold,newinold_and_new:country_image_url['country']=country_image_url['country'].replace(old,new)image_url_of_country=country_image_url.set_index('country')['flag_image_url']country_image_url.head()<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countryflag_image_url0Angolahttps://www.countryflags.io/ao/flat/64.png1Burundihttps://www.countryflags.io/bi/flat/64.png2Beninhttps://www.countryflags.io/bj/flat/64.png3Burkina Fasohttps://www.countryflags.io/bf/flat/64.png4Botswanahttps://www.countryflags.io/bw/flat/64.pngImage(url=image_url_of_country['Australia'])Caching the flag images dataDownloading our data sources every time we need them is not sustainable. What if they're big? What if you're offline or have slow internet (yes, dear future reader, even in the US, during coronavirus times!)?Caching. A "cache aside" read-cache. That's the word. py2store has tools for that (most of which are are caching.py).So let's say we're going to have a local folder where we'll store various datas we download. The principle is as follows:frompy2store.cachingimportmk_cached_storeclassTheSource(dict):...the_cache={}TheCacheSource=mk_cached_store(TheSource,the_cache)the_source=TheSource({'green':'eggs','and':'ham'})the_cached_source=TheCacheSource(the_source)print(f"the_cache:{the_cache}")print(f"Getting green...")the_cached_source['green']print(f"the_cache:{the_cache}")print("... so the next time the_cached_source will get it's green from that the_cache")the_cache: {}
Getting green...
the_cache: {'green': 'eggs'}
... so the next time the_cached_source will get it's green from that the_cacheBut now, you'll notice a slight problem ahead. What exactly does our source store (or rather reader) looks like? In it's raw form it would take urls as it's keys, and the response of a request as it's value. That store wouldn't have an__iter__for sure (unless you're Google). But more to the point here, themk_cached_storetool uses the same key for the source and the cache, and we can't just use the url as is, to be a local file path.There's many ways we could solve this. One way is to add a key map layer on the cache store, so externally, it speaks the url key language, but internally it will map that url to a valid local file path. We've been there, we got the T-shirt!But what we're going to do is a bit different: We're going to do the key mapping in the source store itself. It seems to make more sense in our context: We have a data source ofname: datapairs, and if we impose that the name should be a valid file name, we don't need to have a key map in the cache store.So let's start by building thisMyDataStorestore. We'll start by defining the functions that get us the data we want.defcountry_flag_image_url():importpandasaspdreturnpd.read_csv('https://raw.githubusercontent.com/i2mint/examples/master/data/country_flag_image_url.csv')defkaggle_coronavirus_dataset():importkagglefromioimportBytesIO# didn't find the pure binary download function, so using temp dir to emulatefromtempfileimportmkdtempdownload_dir=mkdtemp()filename='novel-corona-virus-2019-dataset.zip'zip_file=os.path.join(download_dir,filename)dataset='sudalairajkumar/novel-corona-virus-2019-dataset'kaggle.api.dataset_download_files(dataset,download_dir)withopen(zip_file,'rb')asfp:b=fp.read()returnBytesIO(b)defcity_population_in_time():importpandasaspdreturnpd.read_csv('https://gist.githubusercontent.com/johnburnmurdoch/''4199dbe55095c3e13de8d5b2e5e5307a/raw/fa018b25c24b7b5f47fd0568937ff6c04e384786/city_populations')Now we can make a store that simply uses these function names as the keys, and their returned value as the values.frompy2store.baseimportKvReaderfromfunctoolsimportlru_cacheclassFuncReader(KvReader):_getitem_cache_size=999def__init__(self,funcs):# TODO: assert no free arguments (arguments are allowed but must all have defaults)self.funcs=funcsself._func_of_name={func.__name__:funcforfuncinfuncs}def__contains__(self,k):returnkinself._func_of_namedef__iter__(self):yield fromself._func_of_namedef__len__(self):returnlen(self._func_of_name)@lru_cache(maxsize=_getitem_cache_size)def__getitem__(self,k):returnself._func_of_name[k]()# call the funcdef__hash__(self):return1data_sources=FuncReader([country_flag_image_url,kaggle_coronavirus_dataset,city_population_in_time])list(data_sources)['country_flag_image_url',
 'kaggle_coronavirus_dataset',
 'city_population_in_time']data_sources['country_flag_image_url']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countryregionflag_image_url0AngolaAfricahttps://www.countryflags.io/ao/flat/64.png1BurundiAfricahttps://www.countryflags.io/bi/flat/64.png2BeninAfricahttps://www.countryflags.io/bj/flat/64.png3Burkina FasoAfricahttps://www.countryflags.io/bf/flat/64.png4BotswanaAfricahttps://www.countryflags.io/bw/flat/64.png............210Solomon IslandsOceaniahttps://www.countryflags.io/sb/flat/64.png211TongaOceaniahttps://www.countryflags.io/to/flat/64.png212TuvaluOceaniahttps://www.countryflags.io/tv/flat/64.png213VanuatuOceaniahttps://www.countryflags.io/vu/flat/64.png214SamoaOceaniahttps://www.countryflags.io/ws/flat/64.png215 rows × 3 columnsdata_sources['country_flag_image_url']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countryregionflag_image_url0AngolaAfricahttps://www.countryflags.io/ao/flat/64.png1BurundiAfricahttps://www.countryflags.io/bi/flat/64.png2BeninAfricahttps://www.countryflags.io/bj/flat/64.png3Burkina FasoAfricahttps://www.countryflags.io/bf/flat/64.png4BotswanaAfricahttps://www.countryflags.io/bw/flat/64.png............210Solomon IslandsOceaniahttps://www.countryflags.io/sb/flat/64.png211TongaOceaniahttps://www.countryflags.io/to/flat/64.png212TuvaluOceaniahttps://www.countryflags.io/tv/flat/64.png213VanuatuOceaniahttps://www.countryflags.io/vu/flat/64.png214SamoaOceaniahttps://www.countryflags.io/ws/flat/64.png215 rows × 3 columnsdata_sources['city_population_in_time']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>namegroupyearvaluesubGroupcity_idlastValuelatlon0AgraIndia1575200.0IndiaAgra - India200.027.1833378.016671AgraIndia1576212.0IndiaAgra - India200.027.1833378.016672AgraIndia1577224.0IndiaAgra - India212.027.1833378.016673AgraIndia1578236.0IndiaAgra - India224.027.1833378.016674AgraIndia1579248.0IndiaAgra - India236.027.1833378.01667..............................6247VijayanagarIndia1561480.0IndiaVijayanagar - India480.015.3350076.462006248VijayanagarIndia1562480.0IndiaVijayanagar - India480.015.3350076.462006249VijayanagarIndia1563480.0IndiaVijayanagar - India480.015.3350076.462006250VijayanagarIndia1564480.0IndiaVijayanagar - India480.015.3350076.462006251VijayanagarIndia1565480.0IndiaVijayanagar - India480.015.3350076.462006252 rows × 9 columnsBut we wanted this all to be cached locally, right? So a few more lines to do that!frompy2store.cachingimportmk_cached_storefrompy2storeimportQuickPickleStoremy_local_cache=os.path.expanduser('~/ddir/my_sources')CachedFuncReader=mk_cached_store(FuncReader,QuickPickleStore(my_local_cache))data_sources=CachedFuncReader([country_flag_image_url,kaggle_coronavirus_dataset,city_population_in_time])list(data_sources)['country_flag_image_url',
 'kaggle_coronavirus_dataset',
 'city_population_in_time']data_sources['country_flag_image_url']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>countryregionflag_image_url0AngolaAfricahttps://www.countryflags.io/ao/flat/64.png1BurundiAfricahttps://www.countryflags.io/bi/flat/64.png2BeninAfricahttps://www.countryflags.io/bj/flat/64.png3Burkina FasoAfricahttps://www.countryflags.io/bf/flat/64.png4BotswanaAfricahttps://www.countryflags.io/bw/flat/64.png............210Solomon IslandsOceaniahttps://www.countryflags.io/sb/flat/64.png211TongaOceaniahttps://www.countryflags.io/to/flat/64.png212TuvaluOceaniahttps://www.countryflags.io/tv/flat/64.png213VanuatuOceaniahttps://www.countryflags.io/vu/flat/64.png214SamoaOceaniahttps://www.countryflags.io/ws/flat/64.png215 rows × 3 columnsdata_sources['city_population_in_time']<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>namegroupyearvaluesubGroupcity_idlastValuelatlon0AgraIndia1575200.0IndiaAgra - India200.027.1833378.016671AgraIndia1576212.0IndiaAgra - India200.027.1833378.016672AgraIndia1577224.0IndiaAgra - India212.027.1833378.016673AgraIndia1578236.0IndiaAgra - India224.027.1833378.016674AgraIndia1579248.0IndiaAgra - India236.027.1833378.01667..............................6247VijayanagarIndia1561480.0IndiaVijayanagar - India480.015.3350076.462006248VijayanagarIndia1562480.0IndiaVijayanagar - India480.015.3350076.462006249VijayanagarIndia1563480.0IndiaVijayanagar - India480.015.3350076.462006250VijayanagarIndia1564480.0IndiaVijayanagar - India480.015.3350076.462006251VijayanagarIndia1565480.0IndiaVijayanagar - India480.015.3350076.462006252 rows × 9 columnsz=ZippedCsvs(data_sources['kaggle_coronavirus_dataset'])list(z)