Package: anomalytics
AnomalyticsYour Ultimate Anomaly Detection & Analytics ToolIntroductionanomalyticsis a Python library that aims to implement all statistical methods for the purpose of detecting any sort of anomaly e.g. extreme events, high or low anomalies, etc. This library utilises external dependencies such as:Pandas 2.1.1NumPy 1.26.0SciPy 1.11.3Matplotlib 3.8.2Pytest-Cov 4.1.0.Black 23.10.0Isort 5.12.0MyPy 1.6.1Bandit 1.7.5anomalyticssupports the following Python's versions:3.10.x,3.11.x,3.12.0.InstallationTo use the library, you can install as follow:# Install without openpyxl$pip3installanomalytics# Install with openpyxl$pip3install"anomalytics[extra]"As a contributor/collaborator, you may want to consider installing all external dependencies for development purposes:# Install bandit, black, isort, mypy, openpyxl, pre-commit, and pytest-cov$pip3install"anomalytics[codequality,docs,security,testcov,extra]"Use Caseanomalyticscan be used to analyze anomalies in your dataset (both aspandas.DataFrameorpandas.Series). To start, let's follow along with this minimum example where we want to detect extremely high anomalies in our dataset.Read the walkthrough below, or the concrete examples here:Extreme Anomaly Analysis - DataFrameBattery Water Level Analysis - Time SeriesAnomaly Detection via theDetectorInstanceImportanomalyticsand initialise our time series of 100_002 rows:importanomalyticsasaticsdf=atics.read_ts("./ad_impressions.csv","csv")df.head()datetimexandrgamadobe02023-10-1809:01:0052.48357171.02113135.68191512023-10-1809:02:0049.30867873.65199660.34724622023-10-1809:03:0053.23844365.69081348.12080532023-10-1809:04:0057.61514980.94439359.55077542023-10-1809:05:0048.82923376.44509926.710413Initialize the needed detector object. Each detector utilises a different statistical method for detecting anomalies. In this example, we'll use POT method and a high anomaly type. Pay attention to the time period that is directly created where thet2is 1 by default because "real-time" always targets the "now" period hence 1 (sec, min, hour, day, week, month, etc.):pot_detector=atics.get_detector(method="POT",dataset=ts,anomaly_type="high")print(f"T0:{pot_detector.t0}")print(f"T1:{pot_detector.t1}")print(f"T2:{pot_detector.t2}")pot_detector.plot(ptype="line-dataset-df",title=f"Page Impressions Dataset",xlabel="Minute",ylabel="Impressions",alpha=1.0)T0:42705T1:16425T2:6570The purpose of using the detector object instead the standalone is to have a simple fix detection flow. In case you want to customize the time window, we can call thereset_time_window()to resett2value, even though that will beat the purpose of using a detector object. Pay attention to the period parameters because the method expects a percentage representation of the distribution of period (ranging 0.0 to 1.0):pot_detector.reset_time_window("historical",t0_pct=0.65,t1_pct=0.25,t2_pct=0.1)print(f"T0:{pot_detector.t0}")print(f"T1:{pot_detector.t1}")print(f"T2:{pot_detector.t2}")pot_detector.plot(ptype="hist-dataset-df",title="Dataset Distributions",xlabel="Distributions",ylabel="Page Impressions",alpha=1.0,bins=100)T0:65001T1:25001T2:10000Now, we can extract exceedances by giving the expectedquantile:pot_detector.get_extremes(0.95)pot_detector.exeedance_thresholds.head()xandrgamadobedatetime058.22465385.17702960.3623062023-10-1809:01:00158.22465385.17702960.3623062023-10-1809:02:00258.22465385.17702960.3623062023-10-1809:03:00358.22465385.17702960.3623062023-10-1809:04:00458.22465385.17702960.3623062023-10-1809:05:00Let's visualize the exceedances and its threshold to have a clearer understanding of our dataset:pot_detector.plot(ptype="line-exceedance-df",title="Peaks Over Threshold",xlabel="Minute",ylabel="Page Impressions",alpha=1.0)Now that we have the exceedances, we can fit our data into the chosen distribution, in this example the "Generalized Pareto Distribution". The first couple rows will be zeroes which is normal because we only fit data that are greater than zero into the wanted distribution:pot_detector.fit()pot_detector.fit_result.head()xandr_anomaly_scoregam_anomaly_scoreadobe_anomaly_scoretotal_anomaly_scoredatetime01.0871470.0000000.0000001.0871472023-11-1700:46:0010.0000000.0000000.0000000.0000002023-11-1700:47:0020.0000000.0000000.0000000.0000002023-11-1700:48:0030.0000001.8158750.0000001.8158752023-11-1700:49:0040.0000000.0000000.0000000.0000002023-11-1700:50:00
...Let's inspect the GPD distributions to get the intuition of our pareto distribution:pot_detector.plot(ptype="hist-gpd-df",title="GPD - PDF",xlabel="Page Impressions",ylabel="Density",alpha=1.0,bins=100)The parameters are stored inside the detector class:pot_detector.params{0:{'xandr':{'c':-0.11675297447288158,'loc':0,'scale':2.3129766056305603,'p_value':0.9198385927065513,'anomaly_score':1.0871472537998},'gam':{'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,'anomaly_score':0.0},'adobe':{'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,'anomaly_score':0.0},'total_anomaly_score':1.0871472537998},1:{'xandr':{'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,'anomaly_score':0.0},'gam':{'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,
...'scale':0.0,'p_value':0.0,'anomaly_score':0.0},'total_anomaly_score':0.0},
...}Last but not least, we can now detect the extremely large (high) anomalies:pot_detector.detect(0.95)pot_detector.detection_result16425False16426False16427False16428False16429False...22990False22991False22992False22993False22994False
Name:detecteddata,Length:6570,dtype:boolNow we can visualize the anomaly scores from the fitting with the anomaly threshold to get the sense of the extremely large values:pot_detector.plot(ptype="line-anomaly-score-df",title="Anomaly Score",xlabel="Minute",ylabel="Page Impressions",alpha=1.0)Now what? Well, while the detection process seems quite straight forward, in most cases getting the details of each anomalous data is quite tidious! That's whyanomalyticsprovides a comfortable method to get the summary of the detection so we can see when, in which row, and how the actual anomalous data look like:pot_detector.detection_summary.head(5)rowxandrgamadobexandr_anomaly_scoregam_anomaly_scoreadobe_anomaly_scoretotal_anomaly_scoreanomaly_threshold2023-11-2812:06:005922564.11713576.42592547.77292921.4457590.0000000.00000021.44575919.6898852023-11-2812:25:005924440.51341594.52602165.9216440.00000019.5579622.68533722.24329919.6898852023-11-2812:45:005926452.36203954.19171979.9728600.0000000.00000072.31327372.31327319.6898852023-11-2816:48:005950764.75320370.34414242.54016832.5430210.0000000.00000032.54302119.6898852023-11-2816:53:005951235.91222152.57293975.6210030.0000000.00000022.19950522.19950519.689885In every good analysis there is a test! We can evaluate our analysis result with "Kolmogorov Smirnov" 1 sample test to see how far the statistical distance between the observed sample distributions to the theoretical distributions via the fitting parameters (the smaller thestats_distancethe better!):pot_detector.evaluate(method="ks")pot_detector.evaluation_resultcolumntotal_nonzero_exceedancesstats_distancep_valueclocscale0xandr33110.0129010.635246-0.12856102.3290051gam32790.0110060.817674-0.14047903.8525742adobe32980.0194790.161510-0.13301906.007833If 1 test is not enough for evaluation, we can also visually test our analysis result with "Quantile-Quantile Plot" method to observed the sample quantile vs. the theoretical quantile:# Use the last non-zero parameterspot_detector.evaluate(method="qq")# Use a random non-zero parameterspot_detector.evaluate(method="qq",is_random=True)Anomaly Detection via Standalone FunctionsYou have a project that only needs to be fitted? To be detected? Don't worry!anomalyticsalso provides standalone functions as well in case users want to start the anomaly analysis from a different starting points. It is more flexible, but many processing needs to be done by you. LEt's take an example with a different dataset, thistime the water level Time Series!Importanomalyticsand initialise your time series:importanomalyticsasaticsts=atics.read_ts("water_level.csv","csv")ts.head()2008-11-0306:00:000.2192008-11-0307:00:00-0.0412008-11-0308:00:00-0.2822008-11-0309:00:00-0.3682008-11-0310:00:00-0.400
Name:WaterLevel,dtype:float64Set the time windows of t0, t1, and t2 to compute dynamic expanding period for calculating the threshold via quantile:t0,t1,t2=atics.set_time_window(total_rows=ts.shape[0],method="POT",analysis_type="historical",t0_pct=0.65,t1_pct=0.25,t2_pct=0.1)print(f"T0:{t0}")print(f"T1:{t1}")print(f"T2:{t2}")T0:65001T1:25001T2:10000Extract exceedances and indicate that it is a"high"anomaly type and what's thequantile:pot_thresholds=get_threshold_peaks_over_threshold(dataset=ts,t0=t0,"high",q=0.90)pot_exceedances=atics.get_exceedance_peaks_over_threshold(dataset=ts,threshold_dataset=pot_thresholds,anomaly_type="high")exceedances.head()2008-11-0306:00:000.8592008-11-0307:00:000.8592008-11-0308:00:000.8592008-11-0309:00:000.8592008-11-0310:00:000.859
Name:WaterLevel,dtype:float64Compute the anomaly scores for each exceedance and initialize a params for further analysis and evaluation:params={}anomaly_scores=atics.get_anomaly_score(exceedance_dataset=pot_exceedances,t0=t0,gpd_params=params)anomaly_scores.head()2016-04-0315:00:000.02016-04-0316:00:000.02016-04-0317:00:000.02016-04-0318:00:000.02016-04-0319:00:000.0
Name:anomalyscores,dtype:float64
...Inspect the parameters:params{0:{'index':Timestamp('2016-04-03 15:00:00'),'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,'anomaly_score':0.0},1:{'index':Timestamp('2016-04-03 16:00:00'),
...'c':0.0,'loc':0.0,'scale':0.0,'p_value':0.0,'anomaly_score':0.0},
...}Detect anomalies:anomaly_threshold=get_anomaly_threshold(anomaly_score_dataset=anomaly_scores,t1=t1,q=0.90)detection_result=get_anomaly(anomaly_score_dataset=anomaly_scores,threshold=anomaly_threshold,t1=t1)detection_result.head()2020-03-3119:00:00False2020-03-3120:00:00False2020-03-3121:00:00False2020-03-3122:00:00False2020-03-3123:00:00False
Name:anomalies,dtype:boolFor the test, kolmogorov-smirnov and qq plot are also accessible via standalone functions, but the params need to be processed so it only contains a non-zero parameters since there are no reasons to calculate a zero 😂nonzero_params=[]forrowinrange(0,t1+t2):if(params[row]["c"]!=0orparams[row]["loc"]!=0orparams[row]["scale"]!=0):nonzero_params.append(params[row])ks_result=atics.evals.ks_1sample(dataset=pot_exceedances,stats_method="POT",fit_params=nonzero_params)ks_result{'total_nonzero_exceedances':[5028],'stats_distance':[0.0284]'p_value':[0.8987],'c':[0.003566],'loc':[0],'scale':[0.140657]}Visualize via qq plot:nonzero_exceedances=exceedances[exceedances.values>0]visualize_qq_plot(dataset=nonzero_exceedances,stats_method="POT",fit_params=nonzero_params,)Sending Anomaly NotificationWe have anomaly you said? Don't worry,anomalyticshas the implementation to send an alert via E-Mail or Slack. Just ensure that you have your email password or Slack webhook ready. This example shows both application (please read the comments 😎):Initialize the wanted platform:# Gmailgmail=atics.get_notification(platform="email",sender_address="my-cool-email@gmail.com",password="AIUEA13",recipient_addresses=["my-recipient-1@gmail.com","my-recipient-2@web.de"],smtp_host="smtp.gmail.com",smtp_port=876,)# Slackslack=atics.get_notification(platform="slack",webhook_url="https://slack.com/my-slack/YOUR/SLACK/WEBHOOK",)print(gmail)print(slack)'Email Notification''Slack Notification'Prepare the data for the notification! If you use standalone, you need to process thedetection_resultto become a DataFrame withrow, ``# Standalonedetected_anomalies=detection_result[detection_result.values==True]anomalous_data=ts[detected_anomalies.index]standalone_detection_summary=pd.DataFrame(index=anomalous.index.flatten(),data=dict(row=[ts.index.get_loc(index)+1forindexinanomalous.index],anomalous_data=[datafordatainanomalous.values],anomaly_score=[scoreforscoreinanomaly_score[anomalous.index].values],anomaly_threshold=[anomaly_threshold]*anomalous.shape[0],))# Detector Instancedetector_detection_summary=pot_detector.detection_summaryPrepare the notification payload and a custome message if needed:# Emailgmail.setup(detection_summary=detection_summary,message="Extremely large anomaly detected! From Ad Impressions Dataset!")# Slackslack.setup(detection_summary=detection_summary,message="Extremely large anomaly detected! From Ad Impressions Dataset!")Send your notification! Beware that the scheduling is not implemented since it always depends on the logic of the use case:# Emailgmail.send# Slackslack.send'Notification sent successfully.'Check your email or slack, this example produces the following notification via Slack:ReferenceNakamura, C. (2021, July 13). On Choice of Hyper-parameter in Extreme Value Theory Based on Machine Learning Techniques. arXiv:2107.06074 [cs.LG].https://doi.org/10.48550/arXiv.2107.06074Davis, N., Raina, G., & Jagannathan, K. (2019). LSTM-Based Anomaly Detection: Detection Rules from Extreme Value Theory. In Proceedings of the EPIA Conference on Artificial Intelligence 2019.https://doi.org/10.48550/arXiv.1909.06041Arian, H., Poorvasei, H., Sharifi, A., & Zamani, S. (2020, November 13). The Uncertain Shape of Grey Swans: Extreme Value Theory with Uncertain Threshold. arXiv:2011.06693v1 [econ.GN].https://doi.org/10.48550/arXiv.2011.06693Yiannis Kalliantzis. (n.d.). Detect Outliers: Expert Outlier Detection and Insights. Retrieved [23-12-04T15:10:12.000Z], fromhttps://detectoutliers.com/Wall of FameI am deeply grateful to have met and guided by wonderful people who inspired me to finish my capstone project for my study at CODE university of applied sciences in Berlin (2023). Thank you so much for being you!Sabrina LindenbergAdam RoeAlessandro DolciChristian LeschinskiJohanna KokocinskiPeter Krauß