Tidy Anomaly Detection using R. Imagine, You run an online business... | by AbdulMajedRaja RS | Towards Data Science

Imagine, You run an online business like Amazon.com and you want to plan Server Resources for the next year — It is imperative that you need to know when your load is going to spike (or at least when did it spike in retrospective to believe it’ll repeat again) and that is where Time Series Anomaly Detection is what you are in need of. While there are some packages like Twitter’s AnomalyDetection that has been doing this job, there is another good candidate — anomalize — that does something specific which no other Anomaly Detection packages were doing. That is Tidy Anomaly Detection.
Please note, The purpose of this article is to help you perform Anomaly Detection in R — The Tidy Way and not to teach you the principles and concepts of Anomaly Detection or Time Series Data.
Sorry to say this! Data Scientists who use R are known to write clumsy code — code that is not very readable and code that is not very efficient but this trend has been changing because of the tidy principle popularized by Hadley Wickham who supposedly doesn’t need any introduction in R universe, because his tidyverse is what contributes to the efficiency and work of a lot of R Data scientists. Now, this new package anomalize open-sourced by Business Science does Time Series Anomaly Detection that goes inline with other Tidyverse packages (or packages supporting tidy data) – with one of the most used Tidyverse functionality – compatibility with the pipe %>% operator to write readable and reproducible data pipeline.
The Stable version of the R package anomalize is available on CRAN that could be installed like below:
install.packages('anomalize')
The latest development version of anomalize is available on github that could be installed like below:
#install.packages('devtools') devtools::install_github("business-science/anomalize")
Considering that the development version doesn’t require compiling tools, It’s better to install the development version from github that would be more bug-free and with latest features.
It’s easier to learn a new concept or code piece by actually doing and relating it to what we are of. So, to understand the Tidy Anomaly Detection in R, We will try to detect anomalies in Bitcoin Price since 2017.
We use the following 3 packages for to solve the above case:
library(anomalize) #tidy anomaly detectiomlibrary(tidyverse) #tidyverse packages like dplyr, ggplot, tidyrlibrary(coindeskr) #bitcoin price extraction from coindesk
We use get_historic_price() from coindeskr to extract historic bitcoin price from Coindesk. The resulting dataframe is stored in the object btc
btc <- get_historic_price(start = "2017-01-01")
For Anomaly Detection using anomalize, we need to have either a tibble or tibbletime object. Hence we have to convert the dataframe btc into a tibble object that follows a time series shape and store it in btc_ts.
btc_ts <- btc %>% rownames_to_column() %>% as.tibble() %>%   mutate(date = as.Date(rowname)) %>% select(-one_of('rowname'))
Just looking at the head of btc_ts to see sample data:
head(btc_ts) Price date      1  998. 2017-01-012 1018. 2017-01-023 1031. 2017-01-034 1130. 2017-01-045 1006. 2017-01-056  896. 2017-01-06
One of the important things to do with Time Series data before starting with Time Series forecasting or Modelling is Time Series Decomposition where the Time series data is decomposed into Seasonal, Trend and remainder components. anomalize has got a function time_decompose() to perform the same. Once the components are decomposed, anomalize can detect and flag anomalies in the decomposed data of the reminder component which then could be visualized with plot_anomaly_decomposition() .
btc_ts %>%   time_decompose(Price, method = "stl", frequency = "auto", trend = "auto") %>%  anomalize(remainder, method = "gesd", alpha = 0.05, max_anoms = 0.2) %>%  plot_anomaly_decomposition()
Gives this plot:
As you can see from the above code, the decomposition happens based on ‘stl’ method which is the common method of time series decomposition but if you have been using Twitter’s AnomalyDetection, then the same can be implemented in anomalize by combining time_decompose(method = “twitter”) with anomalize(method = "gesd"). Also the ‘stl’ method of decomposition can also be combined with anomalize(method = "iqr") for a different IQR based anomaly detection.
Anomaly Detection and Plotting the detected anomalies are almost similar to what we saw above with Time Series Decomposition. It’s just that decomposed components after anomaly detection are recomposed back with time_recompose() and plotted with plot_anomalies() . The package itself automatically takes care of a lot of parameter setting like index, frequency and trend, making it easier to run anomaly detection out of the box with less prior expertise in the same domain.
btc_ts %>%   time_decompose(Price) %>%  anomalize(remainder) %>%  time_recompose() %>%  plot_anomalies(time_recomposed = TRUE, ncol = 3, alpha_dots = 0.5)
Gives this plot:
It could be very well inferred from the given plot how accurate the anomaly detection is finding out the Bitcoin Price madness that happened during the early 2018.
If you are interested in extracting the actual datapoints which are anomalies, the following code could be used:
btc_ts %>%   time_decompose(Price) %>%  anomalize(remainder) %>%  time_recompose() %>%  filter(anomaly == 'Yes') Converting from tbl_df to tbl_time.Auto-index message: index = datefrequency = 7 daystrend = 90.5 days# A time tibble: 58 x 10# Index: date   date       observed   season trend remainder remainder_l1 remainder_l2 anomaly recomposed_l1                                                  1 2017-11-12    5857.  36.0    7599.    -1778.       -1551.        1672. Yes             6085. 2 2017-12-04   11617.  11.2    9690.     1916.       -1551.        1672. Yes             8150. 3 2017-12-05   11696.  -2.01   9790.     1908.       -1551.        1672. Yes             8237. 4 2017-12-06   13709.  -9.11   9890.     3828.       -1551.        1672. Yes             8330. 5 2017-12-07   16858.   0.0509 9990.     6868.       -1551.        1672. Yes             8439. 6 2017-12-08   16057. -28.1    9971.     6114.       -1551.        1672. Yes             8393. 7 2017-12-09   14913.  -8.03   9953.     4969.       -1551.        1672. Yes             8394. 8 2017-12-10   15037.  36.0    9934.     5067.       -1551.        1672. Yes             8420. 9 2017-12-11   16700.  11.2    9916.     6773.       -1551.        1672. Yes             8376.10 2017-12-12   17178.  -2.01   9897.     7283.       -1551.        1672. Yes             8345.# ... with 48 more rows, and 1 more variable: recomposed_l2
Thus, anomalize makes it easier to perform anomaly detection in R with cleaner code that also could be used in any data pipeline built using tidyverse. The code used here are available on my github. If you would like to know more about Time Series Forecasting in R, Check out Professor Rob Hyndman’s course on Datacamp.