Project: Matano (Provisioning / Security & Compliance)
File: README.md

<p align="center"> <a href="https://www.matano.dev"><img src="assets/cover_wide.png" width=600></a> </p> <p align="center"> <!-- <a href="#"><img src="https://img.shields.io/badge/Deploys%20to-AWS-%23FF9900.svg?style=for-the-badge&logo=amazon-aws&logoColor=white&labelColor=232F3E"/></a> <a href="#"><img src="https://img.shields.io/badge/rust-%233A3B3C.svg?style=for-the-badge&logo=rust&labelColor=B1513E&logoColor=white"/></a> <br/> --> <a href="https://discord.gg/YSYfHMbfZQ" target="_blank"><img src="https://img.shields.io/badge/rust-%233A3B3C.svg?label=built with&logo=rust&logoColor=ffffff&color=B1513E&labelColor=0d1117"/></a> <a href="#"><img src="https://img.shields.io/badge/deploys%20to-aws-%23FF9900.svg?logo=amazon-aws&logoColor=white&labelColor=232F3E"/></a> <a href="https://discord.gg/YSYfHMbfZQ" target="_blank"><img src="https://img.shields.io/discord/996484553290022973.svg?label=join us&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2"/></a> <a href="https://twitter.com/intent/follow?screenname=matanolabs" target="blank"><img src="https://img.shields.io/twitter/follow/matanolabs?style=social" alt="Twitter Follow"/></a> <a href="/LICENSE" target="_blank"><img src="https://img.shields.io/github/license/matanolabs/matano?style=flat"/></a> <a href="https://bestpractices.coreinfrastructure.org/projects/6478"><img src="https://bestpractices.coreinfrastructure.org/projects/6478/badge"></a> </p> Matano Open Source Security data lake is an open source cloud-native security data lake, built for security teams on AWS. [!NOTE] Matano offers a commercial managed Cloud SIEM for a complete enterprise Security Operations platform. . <div> <h3 align="center"> <a href="https://www.matano.dev/docs">Docs</a> <span> | </span> <a href="https://www.matano.dev">Website</a> <span> | </span> <a href="https://discord.gg/YSYfHMbfZQ">Community</a> </h3> </div> <div align="center"> <br> <img src="assets/matano_athena.png" width="650"> </div> <br> Security Data Lake: Normalize unstructured security logs into a structured realtime data lake in your AWS account. Collect All Your Logs: Integrates out of the box with for security logs and can easily be extended with custom sources. Detection-as-Code: Use Python to build realtime detections as code. Support for automatic import of detections to Matano. Log Transformation Pipeline: Supports custom VRL () scripting to parse, enrich, normalize and transform your logs as they are ingested without managing any servers. No Vendor Lock-In: Uses an open table format () and open schema standards (), to give you full ownership of your security data in a vendor-neutral format. Bring Your Own Analytics: Query your security lake directly from any Iceberg-compatible engine (AWS Athena, Snowflake, Spark, Trino etc.) without having to copy data around. Serverless: Fully serverless and designed specifically for AWS and focuses on enabling high scale, low cost, and zero-ops. <div align="center"> <br> <img src="assets/diagram.png" width="600"> </div> Reduce SIEM costs. Augment your SIEM with a security data lake for additional context during investigations. Write detections-as-code using Python to detect suspicious behavior & create contextualized alerts. ECS-compatible serverless alternative to ELK / Elastic Security stack. (default) (preview) Install the matano CLI to deploy Matano into your AWS account, and manage your deployment. Linux ```bash curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-linux-x64.sh chmod +x matano-linux-x64.sh sudo ./matano-linux-x64.sh ``` macOS ```bash curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-macos-x64.sh chmod +x matano-macos-x64.sh sudo ./matano-macos-x64.sh ``` To get started, run the `matano init` command. Make sure you have AWS credentials in your environment (or in an AWS CLI profile). The interactive CLI wizard will walk you through getting started by generating an initial for you, initializing your AWS account, and deploying into your AWS account. Initial deployment takes a few minutes. <div align="center"> <img src="assets/matano-init.gif" width="600"> </div> <br> Once initialized, your is used to control & manage all resources in your project e.g. log sources, detections, and other configuration. It is structured as follows: ```bash example-matano-dir git:(main) tree detections awsrootcredentials detect.py detection.yml log_sources cloudtrail log_source.yml tables default.yml zeek log_source.yml tables dns.yml matano.config.yml

``` When onboarding a new log source or authoring a detection, run `matano deploy` from anywhere in your project to deploy the changes to your account. , allows you to easily onboard custom log sources and encourages you to normalize fields according to the to enable enhanced pivoting and bulk search for IOCs across your security data lake. Users can define custom VRL programs to parse and transform unstructured logs as they are being ingested through one of the supported mechanisms for a log source (e.g. S3, SQS). VRL is an expression-oriented language designed for transforming observability data (e.g. logs) in a safe and performant manner. It features a simple syntax and a rich set of built-in functions tailored specifically to observability use cases. Let's have a look at a simple example. Imagine that you're working with HTTP log events that look like this: ```json { "line": "{\"status\":200,\"srcIpAddress\":\"1.1.1.1\",\"message\":\"SUCCESS\",\"username\":\"ub40fan4life\"}" } ``` You want to apply these changes to each event: Parse the raw `line` string into JSON, and explode the fields to the top level Rename `srcIpAddress` to the `source.ip` ECS field Remove the `username` field Convert the `message` to lowercase Adding this VRL program to your log source as a `transform` step would accomplish all of that: ```yml transform: | . = object!(parse_json!(string!(.json.line))) .source.ip = del(.srcIpAddress) del(.username) .message = downcase(string!(.message)) schema: ecsfieldnames: source.ip http.status ``` The resulting event : ```json { "message": "success", "status": 200, "source": { "ip": "1.1.1.1" } } ``` Use detections to define rules that can alert on threats in your security logs. A detection is a Python program that is invoked with data from a log source in realtime and can create an alert. ```python def detect(record): return ( record.deepget("event.action") == "CreateInstanceExportTask" and record.deepget("event.provider") == "ec2.amazonaws.com" and record.deepget("event.outcome") == "failure" ) ``` ```python def detect(r): return ( "authentication" in r.deepget("event.category", []) and r.deepget("event.outcome") == "failure" ) def title(r): return f"Multiple failed logins from {r.deepget('user.full_name')} - {r.deepget('source.ip')}" def dedupe(r): return r.deepget("source.ip") ``` ```yaml tables: aws_cloudtrail okta_system o365_audit alert: severity: medium threshold: 5 deduplicationwindowminutes: 15 destinations: slackmyteam ``` ```python from detection import remotecache usertoips = remotecache("user_ip") def detect(record): if ( record.deepget("event.action") == "ConsoleLogin" and record.deepget("event.outcome") == "success" ): user = record.deepget("user.name") existingips = userto_ips[user] or [] updatedips = usertoips.addtostringset( user, record.deepget("source.ip") ) newips = set(updatedips) - set(existing_ips) if existingips and newips: return True ``` All alerts are automatically stored in a Matano table named `matano_alerts`. The alerts and rule matches are normalized to ECS and contain context about the original event that triggered the rule match, along with the alert and rule data. Example Queries Summarize alerts in the last week that are activated (exceeded the threshold) ```sql select matano.alert.id as alert_id, matano.alert.rule.name as rule_name, max(matano.alert.title) as title, count(*) as match_count, min(matano.alert.firstmatchedat) as firstmatchedat, max(ts) as lastmatchedat, arraydistinct(flatten(arrayagg(related.ip))) as related_ip, arraydistinct(flatten(arrayagg(related.user))) as related_user, arraydistinct(flatten(arrayagg(related.hosts))) as related_hosts, arraydistinct(flatten(arrayagg(related.hash))) as related_hash from matano_alerts where matano.alert.firstmatchedat > (current_timestamp - interval '7' day) and matano.alert.activated = true group by matano.alert.rule.name,

order by lastmatchedat desc ``` You can deliver alerts to external systems. You can use the alerting SNS topic to deliver alerts to Email, Slack, and other services. <div align="center"> <br> <img src="assets/matanoslackalert.png" width="600"> <br> <i>A medium severity alert delivered to Slack</i> </div> For general help on usage, please refer to the official . For additional help, feel free to use one of these channels to ask a question: \(Come join the family, and hang out with the team and community\) \(For deeper conversations about features, the project, or problems\) \(Bug reports, Contributions\) \(Get news hot off the press\) Thanks go to these wonderful people (): <!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --> <!-- prettier-ignore-start --> <!-- markdownlint-disable --> <table> <tbody> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/shaeqahmed"><img src="https://avatars.githubusercontent.com/u/13088492?v=4?s=100" width="100px;" alt="Shaeq Ahmed"/><br /><sub><b>Shaeq Ahmed</b></sub></a><br /><a href="#maintenance-shaeqahmed" title="Maintenance"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://www.matano.dev/"><img src="https://avatars.githubusercontent.com/u/9027301?v=4?s=100" width="100px;" alt="Samrose"/><br /><sub><b>Samrose</b></sub></a><br /><a href="#maintenance-Samrose-Ahmed" title="Maintenance"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/kai-ten"><img src="https://avatars.githubusercontent.com/u/11355908?v=4?s=100" width="100px;" alt="Kai Herrera"/><br /><sub><b>Kai Herrera</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=kai-ten" title="Code"></a> <a href="#ideas-kai-ten" title="Ideas, Planning, & Feedback"></a> <a href="#infra-kai-ten" title="Infrastructure (Hosting, Build-Tools, etc)"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/rams3sh"><img src="https://avatars.githubusercontent.com/u/5143597?v=4?s=100" width="100px;" alt="Ram"/><br /><sub><b>Ram</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Arams3sh" title="Bug reports"></a> <a href="#ideas-rams3sh" title="Ideas, Planning, & Feedback"></a> <a href="#userTesting-rams3sh" title="User Testing"></a></td> <td align="center" valign="top" width="14.28%"><a href="http://zbmowrey.com/"><img src="https://avatars.githubusercontent.com/u/14931610?v=4?s=100" width="100px;" alt="Zach Mowrey"/><br /><sub><b>Zach Mowrey</b></sub></a><br /><a href="#ideas-zbmowrey" title="Ideas, Planning, & Feedback"></a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Azbmowrey" title="Bug reports"></a> <a href="#userTesting-zbmowrey" title="User Testing"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/marcin-kwasnicki"><img src="https://avatars.githubusercontent.com/u/91739800?v=4?s=100" width="100px;" alt="marcin-kwasnicki"/><br /><sub><b>marcin-kwasnicki</b></sub></a><br /><a href="#userTesting-marcin-kwasnicki" title="User Testing"></a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Amarcin-kwasnicki" title="Bug reports"></a> <a href="#ideas-marcin-kwasnicki" title="Ideas, Planning, & Feedback"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/gdrapp"><img src="https://avatars.githubusercontent.com/u/346463?v=4?s=100" width="100px;" alt="Greg Rapp"/><br /><sub><b>Greg Rapp</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Agdrapp" title="Bug reports"></a> <a href="#ideas-gdrapp" title="Ideas, Planning, & Feedback"></a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/niheconomoum"><img src="https://avatars.githubusercontent.com/u/22075648?v=4?s=100" width="100px;" alt="Matthew X. Economou"/><br /><sub><b>Matthew X. Economou</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Aniheconomoum" title="Bug reports"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/jarretraim"><img src="https://avatars.githubusercontent.com/u/981154?v=4?s=100" width="100px;" alt="Jarret Raim"/><br /><sub><b>Jarret Raim</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Ajarretraim" title="Bug reports"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://mdfranz.dev/"><img src="https://avatars.githubusercontent.com/u/47213?v=4?s=100" width="100px;" alt="Matt Franz"/><br /><sub><b>Matt Franz</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Amdfranz" title="Bug reports"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://www.linkedin.com/in/francescofaenzi/"><img src="https://avatars.githubusercontent.com/u/45026063?v=4?s=100" width="100px;" alt="Francesco Faenzi"/><br /><sub><b>Francesco Faenzi</b></sub></a><br /><a href="#ideas-FrancescoFaenzi" title="Ideas, Planning, & Feedback"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://nishant.daspatnaik.com/"><img src="https://avatars.githubusercontent.com/u/1339669?v=4?s=100" width="100px;" alt="Nishant Das Patnaik"/><br /><sub><b>Nishant Das Patnaik</b></sub></a><br /><a href="#ideas-dpnishant" title="Ideas, Planning, & Feedback"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/timoguin"><img src="https://avatars.githubusercontent.com/u/671968?v=4?s=100" width="100px;" alt="Tim O'Guin"/><br /><sub><b>Tim O'Guin</b></sub></a><br /><a href="#ideas-timoguin" title="Ideas, Planning, & Feedback"></a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Atimoguin" title="Bug reports"></a> <a href="https://github.com/matanolabs/matano/commits?author=timoguin" title="Code"></a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/francescor"><img src="https://avatars.githubusercontent.com/u/424577?v=4?s=100" width="100px;" alt="Francesco R."/><br /><sub><b>Francesco R.</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Afrancescor" title="Bug reports"></a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="http://grue.io"><img src="https://avatars.githubusercontent.com/u/555914?v=4?s=100" width="100px;" alt="Joshua Sorenson"/><br /><sub><b>Joshua Sorenson</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=grue" title="Code"></a> <a href="https://github.com/matanolabs/matano/commits?author=grue" title="Documentation"></a></td> <td align="center" valign="top" width="14.28%"><a href="http://www.nevermind.co.nz"><img src="https://avatars.githubusercontent.com/u/171317?v=4?s=100" width="100px;" alt="Chris Smith"/><br /><sub><b>Chris Smith</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=chrismsnz" title="Code"></a></td> </tr> </tbody> </table> <!-- markdownlint-restore --> <!-- prettier-ignore-end --> <!-- ALL-CONTRIBUTORS-LIST:END --> <!-- prettier-ignore-start --> <!-- markdownlint-disable --> <!-- markdownlint-restore --> <!-- prettier-ignore-end --> <!-- ALL-CONTRIBUTORS-LIST:END --> This project follows the specification. Contributions of any kind are welcome! <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=03c989f6-90f5-4982-b002-a48635f10b5d"/>