Outlier Detection: Median Absolute Deviation in SAS
By: Nicholas Marey
Tags: Data Science, Anomaly Detection, Sas, Statistics, Exploratory Data Analysis

Photo by Randy Fath on Unsplash

Median absolute deviation is a robust way to identify outliers. It replaces standard deviation or variance with median deviation and the mean with the median. The result is a method that isn’t as affected by outliers as using the mean and standard deviation.

To calculate the median absolute deviation you must follow these steps:

1) Calculate the median of the population, median(population)

2) Calculate the deviation from the median, using the absolute value of the deviations, |x — median(population)|= deviation

3) Find the median of the absolute deviations, median(|deviations|), called scaling factor

4) Add the scaling factor to the median of the population for the upper bound

5) Subtract the scaling factor from the median of the population for the lower bound

6) Anything outside of the bounds is an outlier

Depending on the data it is strongly suggested that you multiply the scaling factor when determining the upper and lower bounds. For example: median(population) + 3*median(|deviations|) = upper bound.

To implement this in SAS I have made the following macro that outputs two data sets. The first data set contains only outliers, while the second data set is observations where the scaling factor is zero. A scaling factor of zero means the median deviation is zero and any value that is not the median of the population will be returned as an outlier. Therefore, I put those observations into heir own data set to examine separately. Additionally, this macro will subset the population based on whatever value is selected for group.

%macro MAD_DEV(dat /* input data set */,

new_dat /* name of outlier data set */,

var /* outlier variable */,

group /* how the data is grouped */,

num_devs /* number of deviations away from the median

*/); *** calculate the median ***;

proc univariate data=&dat._sort noprint;

var &var.;

by &group.;

output out=med

n=n_first

nmiss=num_miss

max=max_value

min=min_value

mean=mean

median=median

std=stdev;

run; *** determine the absolute deviations from the sample median ***;

data devmed;

merge &dat._sort

med(keep=&group. median);

by &group.;

deviate = abs(&var. - median);

run; *** median of the absolute deviations ***;

proc univariate data=devmed noprint;

by &group.;

var deviate;

output out=devmed2 median=scale;

run; *** data set that shows if outlier or not outlier ***;

data devmed3;

merge devmed

devmed2(keep=&group. scale);

by &group.;

if scale ^= 0;

if deviate > (median + scale*&num_devs.) then outlier_flag=1;

else if deviate < (median - scale*&num_devs. )

then outlier_flag=1;

else outlier_flag = 0;

run; *** subset on data that is an outlier ***;

data &new_dat.;

set devmed3;

if outlier_flag = 1;

run; *** subset on data that is not an outlier ***;

*** and scaling factor is 0 ***;

data scale_zero;

merge devmed

devmed2(keep=&group. scale);

by &group.;

if scale = 0;

run; *** Remove unwanted data sets ***;

proc datasets lib=work nolist nodetails;

delete &dat._sort

med

devmed

devmed2

devmed3;

run; %mend;

Thanks for reading and good luck investigating those outliers!