Package: data-patterns
data-patternsPackage for generating and evaluating data-patterns in quantitative reportsFree software: MIT/X licenseDocumentation:https://data-patterns.readthedocs.io.FeaturesHere is what the package does:Generating and evaluating patterns in structured datasets and exporting to Excel and JSONTransforming generated patterns into Pandas codeQuick overviewTo install the packagepip install data_patternsTo introduce the features of the this package define the following Pandas DataFrame:df = pd.DataFrame(columns = ['Name',       'Type',             'Assets', 'TV-life', 'TV-nonlife' , 'Own funds', 'Excess'],
                  data   = [['Insurer  1', 'life insurer',     1000,     800,       0,             200,         200],
                            ['Insurer  2', 'non-life insurer', 4000,     0,         3200,          800,         800],
                            ['Insurer  3', 'non-life insurer', 800,      0,         700,           100,         100],
                            ['Insurer  4', 'life insurer',     2500,     1800,      0,             700,         700],
                            ['Insurer  5', 'non-life insurer', 2100,     0,         2200,          200,         200],
                            ['Insurer  6', 'life insurer',     9000,     8800,      0,             200,         200],
                            ['Insurer  7', 'life insurer',     9000,     0,         8800,          200,         200],
                            ['Insurer  8', 'life insurer',     9000,     8800,      0,             200,         200],
                            ['Insurer  9', 'non-life insurer', 9000,     0,         8800,          200,         200],
                            ['Insurer 10', 'non-life insurer', 9000,     0,         8800,          200,         199.99]])
df.set_index('Name', inplace = True)Start by defining a PatternMiner:miner = data_patterns.PatternMiner(df)To generate patterns use the find-function of this object:df_patterns = miner.find({'name'      : 'equal values',
                          'pattern'   : '=',
                          'parameters': {"min_confidence": 0.5,
                                         "min_support"   : 2,
                                         "decimal" : 8}})The result is a DataFrame with the patterns that were found. The first part of the DataFrame now containsidpattern_idpattern_defsupportexceptionsconfidence0equal values{Own funds} = {Excess}910.9The miner finds one patterns; it states that the ‘Own funds’-column is identical to the ‘Excess’-column in 9 of the 10 cases (with a confidence of 90 %, there is one case where the equal-pattern does not hold).To analyze data with the generated set of data-patterns use the analyze function with the dataframe with the data as input:df_results = miner.analyze(df)The result is a DataFrame with the results. If we selectresult_type = Falsethen the first part of the output containsindexresult_typepattern_idpattern_defsupportexceptionsconfidenceP valuesQ valuesInsurer 10Falseequal values{Own funds} = {Excess}910.9200199.99Other patterns you can use are ‘>’, ‘<’, ‘<=’, ‘>=’, ‘!=’, ‘sum’, and ‘–>’.Read the documentation for more features.Upload to Pypi (for developers)Change the version in setup.py and setup.cfgGo to github.com and navigate to the repository. Next, click on the tab “releases” and then on “Create a new release”. Now, define a Tag verion (it is best to use the same number as you used in your setup.py version-field: v0.1.15 for example). Then click on “publish release”.Make a Pypi account here:https://pypi.org/manage/projects/Download twine by typing in your command prompt:pip install twineGet admin rights of the owner of the data_patterns package.Delete the old files in the dist folderOpen your command prompt and go to the folder of data_patterns. Then typepython setup.py sdisttwine upload dist/*A good reference is here:https://medium.com/@joel.barmettler/how-to-upload-your-python-package-to-pypi-65edc5fe9c56History0.1.0 (2019-10-27)Development release.0.1.11 (2019-11-6)First release on PyPI.< 0.1.17 (2020-10-6)ExpressionYou can now use expressions to find patterns. This is a string such as ‘{.*}={.*}’ (this one will find columns that are equal to eachother). See example in usage as how to do it, also with unknown values.Patterns of the for IF THEN will be done through a pandas expression and quantitative patterns will be found using numpy (quicker). Expression will be split up in parts if it is quantitativeFunctionAdded the function correct_data. This corrects data based on the most common value if grouped with another column, e.g. changes the names in a column if there are multiple names per LEI code.OtherAdded P and Q values to analyzehighest_conf option to find the pattern with the highest conf based on P value.Possible to use with EVA2 rules0.1.17 (2020-10-6)Parameters‘window’ (boolean): Only compares columns in a window of n, so [column-n, column+n].‘disable’ (boolean): If you set this to True, it will disable all tqdm progress bars for finding and analyzing patterns.‘expres’ (boolean): If you use an expression, it will only directly work with the expression if it is an IF THEN statement. Otherwise it is a quantitative pattern and it will be split up in parts and it uses numpy to find the patterns (this is quicker). However sometimes you want to work with an expression directly, such as the difference between two columns is lower than 5%. If you set expres to True, it will work directly with the expression.ExpressionYou can use ABS in expressions. This calculates the absolute value. So something like ‘ABS({‘X’} - {‘Y’}) = {‘Z’})’clusterYou can now add the column name on which you want to clusterFunctionConvert_to_time: merge periodes together by adding suffix to columns (t-1) and (t).convert_columns_to_time: Make the periods into columns so that you have years as columns.OtherAdd tqdm progress bars0.1.18 (16-11-2020)variables to minerYou can now add a boolean to the miner. If you give the boolean True to the miner, it will get rid of all the “ and ‘ in the string data. This is needed for some data where name have those characters in their name. This will give errors later on if not removed.Function to read overzichtChanged the IF THEN expression so that we can use decimals when numericParameters‘notNaN’ (boolean): Only takes not NaN columnsFunction changesConvert_to_time: add boolean set_year. If true then only use the years (this is for yearly data), otherwise keep whole date. Set to True standardupdate_statistics: Remove patterns that contain columns which are not in the data. This is necessary for some insurers so that they do not get errors0.1.19 (10-2-2020)Bug fixes with expressions including regex0.1.20 (29-4-2021)Suppress Pandas slice error is some casesDeleted logging.basicConfig (to avoid that initial config is overwritten)