Package: cloupy
cloupy is a Python package for climatological data downloading, processing and visualizing. The main goal of the package is to help its author in writing a BA thesis. The package is well adapted to academic work - used data sources are reliable and graphs are easy to modify. What's more, cloupy is intuitive and really easy to use, so even users who are not related to the climatological environment should have no problems with the package usage.InstallationRun the following command on the command line:Windows:pipinstallcloupyLinux:pip3installcloupyUsage/ExamplesThis is just a demonstration of the cloupy functionality and some (in my opinion the most important) featuresof the functions/classes. However, every function and class has many arguments which are not used in this tutorial, socheck the docstrings to see all featuresTable of contentsData downloading and processinga.Download climatological data from the IMGW databaseb.Download specific climatological elements from the IMGW databasec.Useful functions for working with the IMGW databased.Download climatological data from the WMO databasee.Useful functions for working with the WMO databasef.Check data continuityMapsa.Exemplary interpolation mapb.The same exemplary map, but without setting the map style and data filteringc.Zoom a map to a specific location, show the difference between default style and retro styled.Draw a map from the manually provided data, select a non-default shapefile to draw borders of the country, plot additional shapes from an additional shapefilee.Import the data for drawing the interpolation map from the global dataframe and draw multi-country map. Zoom to a specific area without changing the extrapolation pointsGraphsa.Download climatological data for the station in Poznań (WMO ID: 12330) from the IMGW and draw a Walter-Lieth diagramb.Download climatological data for the station in Harare (WMO ID: 67775) from the WMO databasec.Download data, set as global dataframe and draw a Walter-Lieth diagram based on the global dataframed.The graph style that better fits a scientific articlee.Select which graph elements are to be drawnf.Provide drawing data manuallySee alsorecap of the drawing processandbrief documentationData downloading and processingDownload climatological data from theIMGW databaseUse thed_imgw_data()function to download the data from the IMGW database. In the code below, the data will be downloaded from the synoptic stations of the IMGW database. In this case, the selected data interval is monthly, the data string starts in 1966 and ends in 2020. The coordinates will be added to the stations in the downloaded data:importcloupyascldata=cl.d_imgw_data(interval='monthly',stations_kind='synop',years_range=range(1966,2021),return_coordinates=True)Download specific climatological elements from theIMGW databaseYou can specify which climatological elements from the IMGW database will be downloaded by setting thekeywordsargument of thed_imgw_data()function. Thekeywordsargument must be a single string or a list of strings. If the given string is in a column name, the column will be added to the dataframe that will be returned. In the example below, the returned dataframe will contain the columns whose names contain the given words ('temperatura' means temperature, 'opad' means precipitation):importcloupyascldata=cl.d_imgw_data(interval='monthly',stations_kind='synop',years_range=range(1966,2021),keywords=['temperatura','opad']# the function is not case-sensitive (try: ['tEmPeRaTURA', 'oPad']))Useful functions for working with theIMGW databaseSee available file formats from the IMGW database for the given arguments:importcloupyasclcl.i_imgw_get_file_formats(interval='monthly',stations_kind='climat',file_format_index='all')See column names in the given file format, see column names in all file formats, search for a keyword in all file formats:# column names in the 's_m_d' file format (monthly interval, synoptic station, the first file format from the available file formats)colnames_in_sdt=cl.i_imgw_search_keywords_in_columns(keywords='',file_format='s_m_d')# column names in all file formats (if the 'file_format' argument is None, then search in every file format)colnames_in_all=cl.i_imgw_search_keywords_in_columns(keywords='',file_format=None)# search for a keyword in all file formatskeywords_in_all=cl.i_imgw_search_keywords_in_columns(keywords='temperatura',file_format=None)Download climatological data from theWMO databaseYou can download data from the WMO database for a single station or for an entire country. If you would like to download the data for a single station, pass the station name (usually the city where the station is located) to thestation_nameargument. If you would like to download the data for an entire country, pass the country name prefixed withcouto thestation_nameargument (e.g. 'couGERMANY'). There are 5 elements that can be downloaded: mean air temperature ('mean'), precipitation ('preci'), minimum air temperature ('min_temp'), maximum air temperature ('max_temp') and sea level pressure ('sl_press'). To specify which element you would like to download, pass a single string or a list of strings to theelements_to_scrapeargument:importcloupyasclsingle_station=cl.d_wmo_data(station_name='POZNAN',elements_to_scrape=['temp','preci','sl_press'])entire_country=cl.d_wmo_data(station_name='couUnited Arab Emirates',elements_to_scrape='temp',return_coordinates=True)Useful functions for working with theWMO databaseYou can see all available station and country names with thecl.i_wmo_get_stations()function:importcloupyasclinfo_df=cl.i_wmo_get_stations()You can search for the nearest stations to the specified points with thecl.i_wmo_search_near_station()function:importcloupyasclclose_to_london=cl.i_wmo_search_near_station(lon=0,lat=51)higher_acceptable_distance=cl.i_wmo_search_near_station(lon=0,lat=51,degrees_range=3)Check data continuityWith thecl.check_data_continuity()function you can remove all stations that do not have satisfactory data continuity. The function will group a dataframe by unique values, count number of records for every single station and remove all stations that have too few number of records:importcloupyascldf=cl.d_imgw_data(interval='monthly',stations_kind='synop',years_range=range(1966,2021))df=cl.check_data_continuity(df=df,main_column=0,precision=1)In the example above, thecheck_data_continuity()groups the dataframe by the unique values in the first column (0 is the index of the dataframe columns), counts the number of records for each station, takes the highest number of records (high_records) and removes stations with the number of records less than the high_records. Theprecisionargument is set to 1, which means that the required number of records to keep a station is equal to 1 * high_records (if high_records is 1000, then the station must have 1000 records to be kept). If theprecisionargument was 0.5, then the required number of records to keep a station would be equal to 0.5 * high_records (if high_records is 1000, then the station must have at least 500 records to be kept)MapsExemplary interpolation mapimportcloupyasclimportnumpyasnpimap=cl.m_MapInterpolation(country='POLAND')imap.d_imgw_data(years_range=range(1966,2021),column_with_values='temp',check_continuity=True,continuity_precision=1)imap.dataframe=imap.dataframe[imap.dataframe.iloc[:,0]>5]# filter the dataframe to exclude# alpine stations that negatively# affect the interpolation resultimap.draw(levels=np.arange(5,10.01,0.05),# (start, end, step)show_contours=True,contours_levels=[5,6,7,8,9,10],clabels_add=[(17.5,54),(19.8,53.5),(23,54),(21,50.8),(17,50.8),(15,53),(15.5,51.1),(20,49.5)],cbar_ticks=[5,6,7,8,9,10],cbar_title='Air temperature [°C]',title='Fig. N. Spatial layout of the mean temperature in Poland\n(1966-2020)',title_ha='center',title_x_position=0.5,save='air_temperature_poland.png')The same exemplary map, but without setting the map style and data filteringNote thatin the above code the most of the arguments are not necessary and are only optional for setting the map style - the code for drawing may be significantly shorter. However, in this case the map style may not be satisfactory and not filtered data may affect negatively the interpolation result:importcloupyascl# The map based on the same dataset, but without data filtering and style settingimap=cl.m_MapInterpolation(country='POLAND')imap.d_imgw_data(years_range=range(1966,2021),column_with_values='temp')imap.draw(save='ugly_map.png')# The 'save' argument does not have to be specified either - the map will be displayed in your app with# a lower DPI value, so the quality of the image may be poor. When the map is being saved, the DPI# value is higher, so the image quality is much better. Different DPIs are used to enhance the# work - lower DPI makes the map creation process much faster, so it is used to preview the mapIn the figure on the left, no levels were specified and no data was filtered. In the figure on the right, the data was not filtered(alpine stations were still in the dataset, stations with poor data continuity were also in the dataset),but the levels were specified by passinglevels=np.arange(5, 10.5, 0.5)to theimap.draw()method.The red spots in the lower parts of the map are the result of the alpine stations in the dataset - due to the significantly deviated values, the interpolation effect is somewhat distorted. The values inside the red spots are much higher than 12°C, so they are not within the specified levels, which would result in the appearance of white spots. However, to theimap.draw()method another argument was passed to force the interpolation process to interpolate within the given levels -interpolation_within_levels=True. When theinterpolation_within_levelsis set to True, the white spots are replaced with the lowest or highest color level (depending on whether the values are below the specified levels or above the specified levels)Data filtering is not always necessary, but specifying manually levels is almost always necessary! Nevertheless, to get high-quality interpolation effect,it is recommended to always check the dataset and specify the interpolation levelsZoom a map to a specific location, show the difference between default style and retro styleSometimes you may want to highlight a specific region of a country and it is possible by passing thezoom_inargument to theimap.draw()method. This argument is also handy when you are working with a country that has overseas territories (e.g. France) and would like to show only a specific part of the countryThezoom_inargument accepts a list of tuples where the first tuple holds the x-values to which the view will be adjusted, and the second tuple holds the y-values to which the view will be adjusted, e.g.zoom_in=[(16, 18), (50, 51)]will zoom into the region of Poland that is located between 16 and 18 degree of the east longitude and between 50 and 51 degree of the north latitudeThe exemplary maps of France with thezoom_inargument in the default and retro styles:importcloupyasclimportnumpyasnpimportmatplotlib.pyplotasplt# create the interpolation map class and prepare the dataimap=cl.m_MapInterpolation('FRANCE')imap.d_wmo_data('couFRANCE','temp',check_continuity=True)imap.dataframe=imap.dataframe[imap.dataframe.iloc[:,0]>9]# remove alpine stations# set the style, modify the map, drawstyles=['default','retro']forstyleinstyles:cl.set_diagStyle(style)# set global style for cloupyimap.draw(zoom_in=[(-5.5,10),(41,51.5)],# zoom into the european part of Francelevels=np.arange(9.4,16.5,0.2),show_grid=True,show_contours=True,contours_levels=np.arange(6,16,1),clabels_add=[(5,49),(3,47),(2.5,45.3),(2,44),(2,43),(6.7,43.8)],clabels_inline_spacing=-7,xticks=np.arange(-4,9,2),cbar_ticks=np.arange(9,17,1),cbar_title='Air temperature [°C]',title=f'Fig. N. Spatial layout of the mean temperature in France\n({style}style)',save=f'france_mean_temperature_layout_{style}.png')# merge the mapsfig,(ax1,ax2)=plt.subplots(nrows=1,ncols=2,figsize=(8,4))ax1.axis('off')ax2.axis('off')img1=plt.imread('france_mean_temperature_layout_default.png')img2=plt.imread('france_mean_temperature_layout_retro.png')ax1.imshow(img1)ax2.imshow(img2)plt.subplots_adjust(wspace=-0.3)plt.savefig('france_styles_comp.png',dpi=300,bbox_inches='tight',pad_inches=0.05,facecolor='white')In this case, if thezoom_inargument was not specified, the maps would be completely indecipherable. This is due to the overseas territories of France which affects negatively the interpolation process and the default map zooming (by default the map is being zoomed into the extreme points of the shapefile):Draw a map from the manually provided data, select a non-default shapefile to draw borders of the country, plot additional shapes from an additional shapefileThe country borders do not have to be drawn from the default cloupy shapefile - you can provide your own non-default shapefile by passing its path to theshapefile_pathargument of thecl.m_MapInterpolation()classWhat is more, it is also possible to draw additional shapes (e.g. states, voivodeships, rivers, lakes) by passing a dictionary to theadd_shapesargument of theimap.draw()method. The dictionary keys must be shapes' paths and the dictionary values must be style/coordinates system set arguments. The dictionary values must be a singlestrin which arguments are separated by commas, e.g.:add_shapes={'path/to/the/shapefile':'crs=epsg:4326, linestyle=solid, linewidth=0.5','path/to/another/shapefile':'crs=epsg:2180,color=red,fill_color=blue'}The data for a map can be provided manually by passing apandas.DataFrame()object to thedataframeargument of thecl.m_MapInterpolation():importcloupyascldf=cl.d_imgw_data(interval='monthly',stations_kind='synop',years_range=range(1966,2021),return_coordinates=True)df=df.iloc[:,[1,12,-3,-2]]# remove unnecessary columns from the datadf=cl.check_data_continuity(df=df,main_column=0,precision=1)# check data continuitydf=df.groupby('Nazwa stacji').mean()# calc the mean values for the stationsdf=df[df.iloc[:,0]>5]# remove alpine stations which deviates significantly from the restWhen the dataframe meets the required data structure (seecl.m_MapInterpolationdocs for more detailed information) for the interpolation map, it can be passed to thecl.m_MapInterpolation()class. So, if you want to provide the data manually, draw non-default borders and additional shapes, the code will look like this:importnumpyasnpimap=cl.m_MapInterpolation(shapefile_path='path/to/shapefile',# specify the path to your non-default shapefilecrs='epsg:4326',# specify the shapefile coordinates systemdataframe=df# pass manually the dataframe)imap.draw(levels=np.arange(5,10.2,0.2),add_shape={'path/to/addit/shapefile':'crs=epsg:4326, linewidth=0.5, linestyle=solid, color=black'},# specify the path to the additional shapefile (as key) and its settings (as value)show_coordinates=False,show_frame=False,cbar_ticks=np.arange(5,11,1),cbar_title='Air temperature [°C]',title='Fig. N. Spatial layout of the mean temperature in Poland.',title_y_position=0.15,save='non-default_borders_and_additional_shp.png')For the above map, the shapefiles were downloaded from theGADM websiteunderGADM license.Import the data for drawing the interpolation map from the global dataframe and draw multi-country map. Zoom to a specific area without changing the extrapolation pointsimportcloupyasclimportpandasaspdimportnumpyasnpcountries=['ITALY','SWITZERLAND']global_df=pd.DataFrame(columns=['station','year','month','temp','lon','lat','elv'])forcountryincountries:data_for_country=cl.d_wmo_data(station_name='cou'+country,elements_to_scrape='temp',return_coordinates=True)global_df=global_df.append(data_for_country)cl.set_global_df(global_df)imap=cl.m_MapInterpolation(country=countries)imap.import_global_df(columns_order=[0,3,4,5],check_continuity=True,continuity_precision=0.2)# in this case, there is no need to remove the alpine stations (they are significantly relevant)imap.draw(levels=np.arange(-5,20.5,0.5),cmap='coolwarm',save='multiple_countries.png')If you wished to take a closer look at the situation on the Italian-Swiss border, you could use thezoom_inargument of theimap.draw()method. You may notice some slight differences in the colors' layout between the map before zooming in and the map after zooming in. The differences result from different extrapolation points, which are always in the corners of the plot (when you zoom into a specific area, it means that you change the corner points of the plot, which affects the extrapolation result).You can force theimap.draw()method to extrapolate the values to the points of the original plot (before zooming) by setting theextrapolation_into_zoomed_areaargument to False:importmatplotlib.pyplotasplt# draw maps (1 = True, 0 = False)forbooleanin[1,0]:imap.draw(levels=np.arange(-5,20.5,0.5),cmap='coolwarm',zoom_in=[(5,12),(45,48)],extrapolation_into_zoomed_area=boolean,figsize=(5,2.5),title=f'extrapolation_into_zoomed_area={boolean}',title_ha='center',title_x_position=0.5,title_y_position=0,title_bold=True,save=f'multiple_countries_zoomed_in_{boolean}.png')# merge the mapsfig,(ax1,ax2)=plt.subplots(nrows=2,ncols=1,figsize=(4,8))ax1.axis('off')ax2.axis('off')img1=plt.imread('multiple_countries_zoomed_in_1.png')img2=plt.imread('multiple_countries_zoomed_in_0.png')ax1.imshow(img1)ax2.imshow(img2)plt.subplots_adjust(hspace=-0.5)plt.savefig('multpile_countries_extr_into_comp.png',dpi=300,bbox_inches='tight',pad_inches=0.05,facecolor='white')As you can see, the slight differences are noticeable in the color layout. Theextrapolation_into_zoomed_areaargument must be set to True if thezoom_inargument is used for zooming into a specific part of the country that has overseas territories (like in the example of France above). If theextrapolation_into_zoomed_areaargument was set to False in the case of France, the method would extrapolate values to the corners of the original plot (the plot before zooming, so the interpolation result in the European France would be completely distorted). In the case of Italy and Switzerland, some slight differences are noticeable, but these differences are still acceptable. However,if you want to zoom into a specific part of the map with preserving the original color/contours layout, you can set theextrapolation_into_zoomed_areaargument to FalseGraphsDownload climatological data for the station in Poznań (WMO ID: 12330) from theIMGW databaseand draw a Walter-Lieth diagramimportcloupyasclwl=cl.g_WalterLieth(station_name='POZNAŃ')wl.d_imgw_data(years_range=range(1966,2020))wl.draw()Download climatological data for the station in Harare (WMO ID: 67775) from theWMO databaseand draw a Walter-Lieth diagramimportcloupyasclwl=cl.g_WalterLieth(station_name='HARARE')wl.d_wmo_data()wl.draw()Download data, set as global dataframe and draw a Walter-Lieth diagram based on the global dataframeimportcloupyasclglobal_df=cl.d_imgw_data(interval='monthly',stations_kind='synop',years_range=range(1966,2020))cl.set_global_df(global_df)wl=cl.g_WalterLieth('WARSZAWA')wl.import_global_df(columns_order='imgw_monthly')wl.draw()The graph style that better fits a scientific articlecl.set_diagStyle('retro')wl.draw()Select which graph elements are to be drawnAs you can see, the graph for POZNAŃ displays information about the coordinates, while the graphs for WARSZAWA do not. The coordinates for POZNAŃ are automatically imported when thewl.d_imgw_data()method is used. However, when we import data from the global dataframe for WARSZAWA, thewl.import_global_df()method does not add coordinates automatically and the coordinates have to be added manually when thecl.g_WalterLieth()object is being created. In our case, for the graph for WARSZAWA it would be:wl=cl.g_WalterLieth(station_name='WARSZAWA',lat=52.2,lon=21.0,elevation=100)Now thewl.draw()method will display the coordinates box. So,if thecl.g_WalterLieth()object does not find the unnecessary data, it will just simply not display missing values. What is more,you can manually decide which elements on the graph are to be drawnThewl.draw()method takes several arguments that let you decide which element on the graph will be displayed. For example, if you do not want to display the title (which actually is the station name), the yearly means box and the bottom freeze rectangles, you have to pass the following arguments to thewl.draw()method:wl.draw(title_text=False,yearly_means_box=False,freeze_rectangles=False)Provide drawing data manuallycloupy graphs can be drawn from the data provided manually. Every graph has its required data structure which must be preserved in thepandas.DataFrame()object. For a Walter-Lieth graph, thepandas.DataFrame()object must contain 5 or 6 columns, depending on the data interval (5 for a monthly interval, 6 for a daily interval). Data can be passed to thedataframeargument in thecl.g_WalterLieth()object. For example, the process might look like this:importcloupyasclimportpandasaspddata=pd.DataFrame({'months':[1,2,3,4,5,6,7,8,9,10,11,12],'temp':[-2,-1,0,7,15,18,19,20,18,14,8,3],'preci':[50,25,55,60,70,80,90,80,68,50,45,49],'max_temp':[10,15,17,18,19,20,35,34,25,20,15,10],'min_temp':[-36,-29,-20,-15,-5,-1,1,2,-1,-4,-18,-22]})wl=cl.g_WalterLieth(station_name='TEST',dataframe=data)wl.draw()More detailed information on the required data structure is available in the graphs classes docstringsRecap of the drawing processcloupy drawing system is easy and can be summarized as follows:create a graph/map class, provide data for further processing and drawing (follow required data structure)optionally, use the graph/map class methods to download and process datause thedraw()method to specify the graph/map styleBrief Documentation (the most important functions and classes)DATA PROCESSING FUNCTIONS/CLASSEScheck_data_continuity-> check data continuity and return a dataframe with the filtered valuesset_global_df(...)-> set global data frame from which data can be imported in any timeread_global_df(...)-> return the global data frame as pandas.DataFrameDATA SCRAPING FUNCTIONSd_imgw_data(...)-> download IMGW data files from the IMGW database and return it as one merged pd.DataFramei_imgw_get_file_formats(...)-> return the available file formats for the given 'interval' and 'stations_kind' in the IMGW database (different file formats contain different data)i_imgw_search_keywords_in_columns(...)-> search for the given keywords in the column names and return a dictionary with the file formats in which the keywords were foundd_wmo_data(...)-> download climatological data for specified station/stations from the WMO websitei_wmo_get_stations(...)-> return pandas.DataFrame with WMO stations information (WMO ids, coordinates, etc.)i_wmo_search_near_station(...)-> return the nearest stations from WMO database for the given coordinatesDATA VISUALIZATION FUNCTIONSset_diagStyle(...)-> choose global style for diagramschange_diagStyle_params(...)-> change global parameters for drawing diagramsDATA VISUALIZATION CLASSESNote thatevery class for drawing diagrams contains some of the above functions as its methods (for data scraping and processing)m_MapInterpolation(...)-> create a MapInterpolation class where the data for drawing an interpolation map can be downloaded, modified, manually providedg_WalterLieth(...)-> create a WalterLieth object where the data for drawing a Walter-Lieth diagram can be downloaded, modified, manually providedMore detailed documentation for every function, class and method is available in the cloupy's Python filesFuture Featuresmore climatological data processing functions (e.g. completing missing data)more climatological graphsDependenciesPackages/libraries:Pandasversion: 1.1.4; 1.3.5 or higherMatplotlibversion: 3.3.2; 3.4.3 or higherRequestsversion: 2.24.0; 2.26.0 or higherbs4version: 4.9.3 or higherpyshpversion: 2.1.3pyprojversion: 3.2.1 or higherscipyversion: 1.7.2 or higherNumpyversion: 1.19.4; 1.21.4 or higherPillowversion: 8.4.0 or highercyclerversion: 0.11.0Pytestversion: 6.2.5 or higher (only for running tests)Mockversion: 4.0.3 or higher (only for running tests)Python version: 3.8.2; 3.9.6, 3.10.2OS: Windows; LinuxAll the above versions of packages/Python have been tested.Note thatcloupy should also be compatible with the versions between mentioned (e.g. cloupy should work fine on any Pandas' version between 1.1.4 and 1.3.5; any Python version between 3.8.2 and 3.9.6). However, it has not been tested andit is recommended to use the most recent version of the packages/libraries and Python.Running TestsTo run tests, run the following command from the root directory:# change directory to the cloupy root directory, for example:cdc:\Python\Lib\site-packages\cloupy

python-mpytestIf you want to run only unit/integration testing:python-mpytesttest\test_unit

python-mpytesttest\test_integrationNote thatthe integration testing may take some time (during the process, data downloading functions are being tested, so duration depends on several factors). The process usually takes about 9-10 minutesIf you want to skip data downloading tests, run the following command:python-mpytest-k"not downloading"SupportFor support, please contact me via email:kamil.grala32466@gmail.comIf you want to report a bug, seeCONTRIBUTING.mdLicensecloupy is licensed underMIT.Author@Kamil Grala