Package: conect2py
Conect2Ai - TAC python packageConect2Py-Package is the name given for the Conect2ai Python software package. The package contains the implementation of TAC, an algorithm for data compression using TAC (Tiny Anomaly Compression). The TAC algorithm is based on the concept the data eccentricity and does not require previously established mathematical models or any assumptions about the underlying data distribution.  Additionally, it uses recursive equations, which enables an efficient computation with low computational cost, using little memory and processing power.Currente version:DependenciesPandas,Numpy,Matplotlib,Seaborn,Scikit-learn,IpythonInstallationYou can download our package from the PyPi repository using the following command:pipinstallconect2pyIf you want to install it locally you download the Wheel distribution fromBuild Distribution.First navigate to the folder where you downloaded the file and run the following command:pipinstallconect2py-0.1.1-py3-none-any.whlExample of UseTo begin you can import TACpy using# FULL PACKAGEimporttacOr try each of our implemented functionalities# MODEL FUNCTIONSfromconect2ai.models.TACimportTACfromconect2ai.models.AutoTACimportAutoTAC# RUN FUNCTIONSfromconect2ai.run.singleimport(print_run_details)fromconect2ai.run.multipleimport(run_multiple_instances,get_optimal_params,display_multirun_optimal_values,run_optimal_combination)# UTILS FUNCTIONSfromconect2ai.utils.format_saveimport(create_param_combinations,create_compressor_list,create_eval_df)fromconect2ai.utils.metricsimport(get_compression_report,print_compression_report,calc_statistics)fromconect2ai.utils.plotsimport(plot_curve_comparison,plot_dist_comparison,plot_multirun_metric_results)Running Multiple tests with TACSetting up the initial variablesmodel_name='TAC_Compression'params={'window_size':np.arange(2,30,1),'m':np.round(np.arange(0.1,2.1,0.1),2),}param_combination=create_param_combinations(params)compressor_list=create_compressor_list(param_combination)Once you created the list of compressors you can runresult_df=run_multiple_instances(compressor_list=compressor_list,param_list=param_combination,series_to_compress=dataframe['sensor_data'].dropna(),cf_score_beta=2)This function returns a pandas Dataframe containing the results of all compression methods. You can expect something like:paramreduction_ratereduction_factormsermsenrmsemaepsnrncccf_score0(2, 0.1)0.45071.82040.06480.25450.06090.012739.98240.99820.80311(2, 0.2)0.45071.82040.06480.25450.06090.012739.98230.99820.80312(2, 0.3)0.45071.82040.06480.25450.06090.012739.98230.99820.80313(2, 0.4)0.45081.82090.06480.25450.06090.012739.98240.99820.80324(2, 0.5)0.45111.82170.06480.25450.06090.012839.98230.99820.8033You can also check the optimal combination by running the following code:display_multirun_optimal_values(result_df=result_df)Parameter combinations for  MAX CF_SCOREparam    reduction_rate  reduction_factor     mse      rmse    nrmse  \
      440  (24, 0.1)          0.9224           12.8919    0.6085  0.7801  0.1867   

             mae    psnr     ncc    cf_score  
      440  0.1294  30.254  0.9825    0.9698Parameter combinations for NEAR  MAX CF_SCOREparam  reduction_rate    reduction_factor     mse    rmse   nrmse  \
 521  (28, 0.2)          0.9336           15.0531  1.1504  1.0726  0.2567   
 364  (20, 0.5)          0.9118           11.3396  0.9458  0.9725  0.2328   
 262  (15, 0.3)          0.8810            8.4029  0.6337  0.7960  0.1905   
 363  (20, 0.4)          0.9102           11.1352  0.9084  0.9531  0.2281   
 543  (29, 0.4)          0.9372           15.9222  1.1474  1.0712  0.2564   

       mae     psnr     ncc     cf_score  
 521  0.1810  27.4883  0.9666    0.9598  
 364  0.1431  28.3388  0.9726    0.9598  
 262  0.0907  30.0780  0.9817    0.9598  
 363  0.1323  28.5140  0.9737    0.9603  
 543  0.1925  27.4996  0.9667    0.9607Visualize multirun results with a plotBy default this plot returns a visualization for the metricsreduction_rate,nccandcf_score.plot_multirun_metric_results(result_df=result_df)The result should look like this;Running a single complession with the optimal parameter foundYou don't need to run the visualization and thedisplay_multirun_optimal_valuesin order to get the optimal compressor created, by running the following code it's possible to get the best result:optimal_param_list=get_optimal_params(result_df=result_df)print("Best compressor param combination: ",optimal_param_list)With the list of optimal parameter (There is a possibility that multiple compressors are considered the best) run the function below to get get the compression result.points_to_keep,optimal_results_details=run_optimal_combination(optimal_list=optimal_param_list,serie_to_compress=dataframe['sensor_data'].dropna(),model='TAC')If you want to see the result details use:print_run_details(optimal_results_details)POINTS:total checked:  30889total kept:  1199percentage discaded:  96.12 %POINT EVALUATION TIMES (ms):mean:  0.003636738161744472std:  0.15511020000857362median:  0.0max:  13.513565063476562min:  0.0total:  112.335205078125RUN TIME (ms):total:  124.2864Evaluating the ResultsNow, to finish the process of the compression, you should follow the next steps:1. Step - Create the evaluation dataframe:evaluation_df=create_eval_df(original=dataframe['sensor_data'].dropna(),flag=points_to_keep)evaluation_df.info()2. Step - Evaluate the performance:report=get_compression_report(original=evaluation_df['original'],compressed=evaluation_df['compressed'],decompressed=evaluation_df['decompressed'],cf_score_beta=2)print_compression_report(report,model_name=model_name,cf_score_beta=2,model_params=optimal_param_list)After that you expect to see something like the following informations:RUN INFOModel:  TAC_CompressionOptimal Params:  [(24, 0.1)]CF-Score Beta:  2RESULTSSAMPLES NUMBER reductionOriginal length:  30889  samplesReduced length:  1199  samplesSamples reduced by a factor of 25.76 timesSample reduction rate: 96.12%FILE SIZE compressionOriginal size:  385549  BytesCompressed size:  14974  Bytesfile compressed by a factor of 25.75 timesfile compression rate: 96.12%METRICSMSE:  0.622RMSE:  0.7886NRMSE:  0.1888MAE:  0.1384PSNR:  30.1591NCC:  0.9821CF-Score:  0.97783. Step - Create the model visualizations:# plot the curves comparison (original vs decompressed)plot_curve_comparison(evaluation_df.original,evaluation_df.decompressed,show=True)And finally here is a example of the result:Literature referenceSignoretti, G.; Silva, M.; Andrade, P.; Silva, I.; Sisinni, E.; Ferrari, P. "An Evolving TinyML Compression Algorithm for IoT Environments Based on Data Eccentricity". Sensors 2021, 21, 4153.https://doi.org/10.3390/s21124153Medeiros, T.; Amaral, M.; Targino, M; Silva, M.; Silva, I.; Sisinni, E.; Ferrari, P.; "TinyML Custom AI Algorithms for Low-Power IoT Data Compression: A Bridge Monitoring Case Study" - 2023 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4.0&IoT), 2023.10.1109/MetroInd4.0IoT57462.2023.10180152