Package: elb-log-analyzer
ELB Log AnalyzerTool for analyzing ELB logs for automating steps to retreive details of ip's user agent, total request count, to which urls requests were made along with their total count, and http methods in json format.S3 Bucket Log DownloaderDownloads S3 bucket objects that we created in specified time window.InstallationUsing Pippython3-mpipinstallelb-log-analyzerAWS configurationCreate IAM policy with below configuration{"Version":"2012-10-17","Statement":[{"Sid":"S3ListSpecificDirectory","Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::alb-log-bucket-name"},{"Sid":"S3GetSpecificDirectory","Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*"}]}Note: above policy will allow user to list all contents in the bucket but download objects only froms3://alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*Create AWS access keysUse aws cli to configure access key for boto3awsconfigureS3 Bucket Log Downloader UsagePrint Help Menu.python3-melb_log_analyzer.s3_log-hDownload all log files generated in 10 hours from now.python3-melb_log_analyzer.s3_log-belb-log-bucket-p'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/'-H10Download all log files generated in 40 mins from now.python3-melb_log_analyzer.s3_log-belb-log-bucket-p'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/'-m40Download all log files generated in 20 secs from now.python3-melb_log_analyzer.s3_log-belb-log-bucket-p'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/'-s20Download all log files generated in 10 hours, 40 mins and 20 secs from now and store in a directory.python3-melb_log_analyzer.s3_log-belb-log-bucket-p'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/'--hours10--minutes40--seconds20-o'./logs/downloads'AnalyzerAnalyzes downloaded log files.Analyzer UsagePrint Help Menupython3-melb_log_analyzer-hPrint json data on consolepython3-melb_log_analyzer-i[INPUT_LOG_FILE_PATH]Store json data in a filepython3-melb_log_analyzer-i[INPUT_LOG_FILE_PATH]-o[OUTPUT_FILE_PATH]Note:INPUT_LOG_FILE_PATHcan be log file or a directory containing all log files ending with.logextensionGet IP details from IPAbuseDBpython3-melb_log_analyzer-i[LOG_FILE_PATH]-t[REQUESTS_THRESHOLD_VALUE]-k[IP_ABUSE_DB_API_KEY]-o[OUTPUT_FILE_PATH]AlertsSend alert to slack channel with abusive ip details.UsageSend alert from analyzed filepythonelb_log_analyzer.alerts-w[SLACK_WEBHOOK]-f[ANALYZED_LOG_FILE_LOCATION]DashboardDashboard to visualize data.Dashboard InstallationInstall requirementspython3-mpipinstalldashboard/requirements.txtUsageStart Appstreamlitrundashboard/app.pyEnter Log File/Directory PathPublish package to pypiUsing poetrypython3-mpoetrypublish--build--username[PYPI_USERNAME]--password[PYPI_PASSWORD]Usage SummaryDownload log filespython3-melb_log_analyzer.s3_log-belb-log-bucket-p'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/'-H[HOURS]-ologsAnalyze Log Filespython3-melb_log_analyzer-ilogs-olog.json-t[REQUEST_THRESHOLD]-k[IP_ABUSE_API_KEY]Send Alert to slack with client ips having total number of requests greater than threshold requestspython-melb_log_analyzer.alerts-w[SLACK_WEBHOOK]-f[ANALYZED_LOG_FILE_LOCATION]Visualize Analyzed Logs using Dashboardstreamlitrundashboard/app.pyDockerPull imagedockerpulldmdhrumilmistry/elb-log-analyzerStart Containerdockerrun-it--rmdmdhrumilmistry/elb-log-analyzer"elb_log_analyzer -h"