Expected outcomes discussed during GSoC Community Bonding period #1
By: Pratik Mishra
Tags: Chaoss, Machine Learning, Augur, Gsoc 2020, Google Summer Of Code

So , I got selected in Google Summer of Code 2020 and I will work with CHAOSS organisation on project “Machine Learning for Anomaly Detection in Open Source Communities” associated with Augur.

My work will be to automate selection of metrics for Insight_worker which is responsible for sending insights of different repo_groups and repos to the user and to apply Machine Learning models which will best fit on different categories of data mainly on time_series data for detecting outliers.

As my mentors suggested I will first work on growing coverage of Insight_worker for Machine Learning specially automating selection of time_series metrics and then I will apply best fit ML models.I thought about the structure of Insight_worker and then came up with below structure.

This is not totally different from the current Insight_worker structure , I just added few more files to separate the preprocessing of the metrics and ML models from the worker.This will also be helpful for the new developers to integrate their ML models easily with the Insight_worker.

preprocess_metrics.py will select metric endpoints category wise and then it will create dataframe which will passed to the ml_model.py module where best fit ML_model will be applied and outliers will be inserted in the insight_records_table.Here we can also create top_insights metrics based on each model in the metric section of augur.worker.py will decide which outliers will be deleted or kept in insight_records_table based on the frequency , score , date etc.. and it will also send selected outliers to the augur’s slack bot “Auggie” which will notify the user about the most recent outliers.

Use case of preprocess_metrics module

Preprocessing and analysis of data is the first and most vital step before performing ML model and so this module will collect data from endpoints and after categorically separating the data, it will be mapped to situable ML models in ml_models.py module.This module will consider all the metrics and separate the metrics given by the insight_worker according to the task.

Use case of ml_models.py module

This module will apply best fit Machine Learning algorithms on mapped data and will discover insights and outliers.It will be helpful in generating different types of insights based on ML model and presenting in form of insight_metrics to the user.

With this setup I can easily process multiple Machine Learning models simultaneously using Python ProcessPoolExecutor and thus processing time will be reduced.

Machine Learning models

This is the main part of the project where we have to select ML algorithms to detect outliers and also at the same time allow users to tune its parameters easily and effectively.I would like to mention numenta/NAB and pyod are two great open source tool which is helpful in analysing different ML models for anomaly detection.

Currently I am excited about using Contextual Anomaly Detection , its quite interesting to apply an algorithm which will detect outliers based on both content and context.I read one such article Contextual anomaly detection framework for big sensor data , here they mentioned how CAD is performing contextually using KNN .

Prediction of active_issues using LSTM

We have currently more than 15–20 time series metrics from various fields like issues, pull_requests ,contributors and code changes where CAD might perform well by clustering metrics of same fields.I thought about a model consisting of LSTM(Long Short Term Memory) for finding patterns in time series data and then integrating it with RandomForestClassifier which i think will do kind of clustering of these metrics based on fields and will detect outliers.However other clustering algorithms may also be used.It involves kind of researching and applying different ML models and then select which will best fit.I found one thing quite challenging that different open source projects has different workflow and pace of their projects development like some projects has increasing patterns , some has static , some follows oscillatory patterns.

Analysis of Time_series data reflects the workflow development in the project and so it is very useful in tracking the development of project with time.Augur has a slack bot named “Auggie” which sends anomalies detected by the Insight_worker directly to the user on slack , this is very interesting and I would like to work on this too.

I will work mainly with Augur this GSoC and you can check out my project work here.

If you have any suggestions you can comment below or can contact me on LinkedIn.