Package: coupled-biased-random-walks
Coupled-Biased-Random-WalksOutlier detection for categorical dataOverviewPython implementation of the Coupled Biased Random Walks (CBRW) outlier detection algorithm described by Pang, Cao, and Chen inhttps://www.ijcai.org/Proceedings/16/Papers/272.pdf.NOTE: Only Python>=3.7 is supported as of version 2.0.0.This implementation operates on Python dicts rather than Pandas DataFrames.  This has the advantage of allowing the model to be updated with new observations in a trivial manner and is more efficient in certain aspects.  However, these advantages come at the cost of iterating a (potentially large) dict of observed values more times than might otherwise be necessary using an underlying DataFrame implementation.If one is working with data previously loaded into a DataFrame, simply use the result ofpandas.DataFrame.to_dict(orient='records')instead of the DataFrame itself to add observations to the model.  Note that because it is common for a DataFrame to fill missing values withnan, the detector will ignore features with valuenanin any observation record.  Therefore, there is no need to further preprocess the DataFrame before using itsto_dictmethod to create records.InstallationThis package is hosted on PyPI and can be installed viapip:$ pip install coupled-biased-random-walksTo instead install from source:$ git clone git@github.com:dkaslovsky/Coupled-Biased-Random-Walks.git
$ cd Coupled-Biased-Random-Walks
$ python setup.py installExampleLet's run the CBRW detection algorithm on the authors' example data set from the paper:This data is saved as aCSV filein this repository and is loaded into memory as a list of dicts byexample.py.  Note that we drop theCheat?column when loading the data, as this is essentially the target variable indicating the anomalous activity to be detected.  The detector is instantiated and observations are added as follows:>>> detector = CBRW()
>>> detector.add_observations(observations)whereobservationsis an iterable of dicts such as the one loaded from the example .CSV file.  Once all of the observations are loaded, the detector can be finalized for scoring by callingfit()and observations can then be scored.>>> detector.fit()
>>> scores = detector.score(observations)Even after fitting and scoring, more observations can be added viaadd_observationsand the detector can again be fit to be used for scoring.  The advantage of this implementation is this ability to incrementally update with new observations.The results of scoring the example data are shown below.  Note that the only observation (ID=1) where fraud was present (Cheat? = yes) received the largest anomaly score.Scores:
Observation ID 1: 0.1055
Observation ID 2: 0.0797
Observation ID 3: 0.0741
Observation ID 4: 0.0805
Observation ID 5: 0.0992
Observation ID 6: 0.0752
Observation ID 7: 0.0741
Observation ID 8: 0.0815
Observation ID 9: 0.0728
Observation ID 10: 0.0979
Observation ID 11: 0.0812
Observation ID 12: 0.0887The "value scores" (scores per attribute) for each observation can also be calculated>>> value_scores(observations)and the results for the example data are shown below.Value scores:
Observation ID 1: {'Gender': 0.0088, 'Education': 0.0195, 'Marriage': 0.0379, 'Income': 0.0393}
Observation ID 2: {'Gender': 0.0171, 'Education': 0.0195, 'Marriage': 0.0208, 'Income': 0.0223}
Observation ID 3: {'Gender': 0.0088, 'Education': 0.0195, 'Marriage': 0.0212, 'Income': 0.0247}
Observation ID 4: {'Gender': 0.0088, 'Education': 0.0286, 'Marriage': 0.0208, 'Income': 0.0223}
Observation ID 5: {'Gender': 0.0171, 'Education': 0.0195, 'Marriage': 0.0379, 'Income': 0.0247}
Observation ID 6: {'Gender': 0.0088, 'Education': 0.0209, 'Marriage': 0.0208, 'Income': 0.0247}
Observation ID 7: {'Gender': 0.0088, 'Education': 0.0195, 'Marriage': 0.0212, 'Income': 0.0247}
Observation ID 8: {'Gender': 0.0171, 'Education': 0.0209, 'Marriage': 0.0212, 'Income': 0.0223}
Observation ID 9: {'Gender': 0.0088, 'Education': 0.0209, 'Marriage': 0.0208, 'Income': 0.0223}
Observation ID 10: {'Gender': 0.0088, 'Education': 0.0286, 'Marriage': 0.0212, 'Income': 0.0393}
Observation ID 11: {'Gender': 0.0171, 'Education': 0.0209, 'Marriage': 0.0208, 'Income': 0.0223}
Observation ID 12: {'Gender': 0.0088, 'Education': 0.0195, 'Marriage': 0.0212, 'Income': 0.0393}The entire example can be reproduced by running:$ python example.pyThe CBRW algorithm can also be used to calculate feature weights.  These weights are calculated when the detector is fit and are used during scoring, but can also be used by any other outlier detection algorithm.  Thus, the CBRW algorithm can be used simply to calculate feature weights and need not score observations.  Feature weights are stored as a property of the detector after the detector'sfitmethod has been called:>>> detector = CBRW()
>>> detector.add_observations(observations)
>>> detector.fit()
>>> detector.feature_weightsFor the example data, the computed feature weights areFeature weights:
{'Gender': 0.1608, 'Education': 0.2627, 'Marriage': 0.2826, 'Income': 0.2939}Implementation NotesFor efficiency, the detector state is only (re)computed upon calling.fit().  Therefore adding new observations (.add_observations()) will not affect scoring until.fit()is called.  Refitting overwrites previous state but includes contribution from all added observations.The.add_observations()and.fit()methods can be chained together if one-line training is desired:detector.add_observations(observations).fit().An observation containing a feature name or feature value that has not been previously fit will be scored asnan.  To instead ignore any such "new" features and score an observation based on known features only, initialize the detector withignore_unknown=True.TestsTo run unit tests:$ python -m unittest discover -v