Package: hawk-scanner
🦅 Hawk-eyeFind PII & Secrets like never before across your entire infrastructure with same tool!Description•Installation•Features•Configuration•Acknowledgements🦅 HAWK Eye - Highly Advanced Watchful Keeper EyeHAWK Eye is a powerful and versatile CLI (Command-Line Interface) tool designed to be your vigilant watchkeeper, guarding against potential data breaches and cyber threats across various platforms. Inspired by the precision and vision of majestic birds of prey, HAWK Eye swiftly scans multiple data sources, including S3, MySQL, PostgreSQL, MongoDB, CouchDB, Google Drive, Slack, Redis, Firebase, filesystem, and Google Cloud buckets (GCS), for Personally Identifiable Information (PII) and secrets. It uses text analysis and OCR techniques to go throug most of the documents, database and different file types like  docx, xlsx, pptx, pdf, jpg, png, gif, zip, tar, rar, etc.Why "HAWK Eye"?The name "HAWK Eye" captures the essence of a Highly Advanced Watchful Keeper. Like the keen vision of a hawk, this tool enables you to monitor and safeguard your data with precision and accuracy, ensuring data privacy and security.HAWK Eye in ActionSee how this works on Youtube -https://youtu.be/LuPXE7UJKOYInstallation via pip or pip3pip3installhawk-scannerExample working command (Use all/fs/s3/gcs etc...)hawk_scannerall--connectionconnection.yml--fingerprintfingerprint.yml--jsonoutput.json--debugNote: Scanning Postgresql?, then you have to install some extra dependencies.For scanning postgresql source, this tool requires psycopg2-binary dependency, we can't ship this dependency with main package because psycopg2-binary not works with most of the systems espically with Windows, so you have to install it manually.pip3installpsycopg2-binaryBuilding or running from sourceHAWK Eye is a Python-based CLI tool that can be installed using the following steps:Clone the HAWK Eye repository to your local machine.gitclonehttps://github.com/rohitcoder/hawk-eye.gitNavigate to the HAWK Eye directory.Run the following command to install the required dependencies:pip3install-rrequirements.txtCreate a connection.yml file in the root directory and add your connection profiles (see the "How to Configure HAWK Eye Connections" section for details).Run the following command to install HAWK Eye:python3hawk_scanner/main.pyKey featuresSwiftly scans multiple data sources (S3, MySQL, PostgreSQL, Redis, Firebase, filesystem, and GCS) for PII data and malware exposure.Advanced algorithms and deep scanning capabilities provide thorough security auditing.Real-time alerts and notifications keep you informed of potential data vulnerabilities using Slack and other integrations, with more coming soon.New command support for S3, MySQL, PostgreSQL, Redis, Firebase, filesystem, and GCS expands the tool's capabilities.--debugflag enables printing of all debugging output for comprehensive troubleshooting.Save output in JSON format using the --json flag and specify a file name like --json output.json.Proudly crafted with love and a sense of humor to make your security journey enjoyable and stress-free.UsageTo unleash the power of HAWK Eye, simply follow the steps mentioned in the "Usage" section of the "README.md" file.OptionsNote: If you don't provide any command, it will run all commands (firebase, fs, gcs, mysql, text, couchdb, gdrive, gdrive workspace, slack, postgresql, redis, s3) by default.OptionDescriptionfirebaseScan Firebase profiles for PII and secrets data.fsScan filesystem profiles for PII and secrets data.gcsScan GCS (Google Cloud Storage) profiles for PII and secrets data.textScan text or string for PII and secrets data.mysqlScan MySQL profiles for PII and secrets data.mongodbScan MongoDB profiles for PII and secrets data.couchdbScan CouchDB profiles for PII and secrets data.slackScan slack profiles for PII and secrets data.postgresqlScan postgreSQL profiles for PII and secrets data.redisScan Redis profiles for PII and secrets data.s3Scan S3 profiles for PII and secrets data.gdriveScan Google drive profiles for PII and secrets data.gdrive_workspaceScan Google drive Workspace profiles for PII and secrets data.--connectionProvide a connection YAML local file path like --connection connection.yml, this file will contain all creds and configs for different sources and other configurations.--fingerprintProvide a fingerprint file path like --fingerprint fingerprint.yml, this file will override default fingerprints.--debugEnable Debug mode.--stdoutPrint output on stdout or terminal.--quietUse --quiet flag if you want to hide all logs from your terminal.--jsonProvide --json file name to save output in json file like --json output.json--shutupUse --shutup flag if you want to hide Hawk ASCII art from your terminal 😁How to Configure HAWK Eye Connections (Profiles in connection.yml)HAWK Eye uses a YAML file to store connection profiles for various data sources. The connection.yml file is located in the config directory. You can add new profiles to this file to enable HAWK Eye to scan additional data sources. The following sections describe the process for adding new profiles to the connection.yml file.Your connection fille will look like thisnotify:redacted:Truesuppress_duplicates:Trueslack:webhook_url:https://hooks.slack.com/services/T0XXXXXXXXXXX/BXXXXXXXX/1CIyXXXXXXXXXXXXXXXsources:redis:redis_example:host:YOUR_REDIS_HOSTs3:s3_example:access_key:YOUR_S3_ACCESS_KEYsecret_key:YOUR_S3_SECRET_KEYbucket_name:YOUR_S3_BUCKET_NAMEcache:truegcs:gcs_example:credentials_file:/path/to/your/credential_file.jsonbucket_name:YOUR_GCS_BUCKET_NAMEcache:trueexclude_patterns:-.pdf-.docxfirebase:firebase_example:credentials_file:/path/to/your/credential_file.jsonbucket_name:YOUR_FIREBASE_BUCKET_NAMEcache:trueexclude_patterns:-.pdf-.docxmysql:mysql_example:host:YOUR_MYSQL_HOSTport:YOUR_MYSQL_PORTuser:YOUR_MYSQL_USERNAMEpassword:YOUR_MYSQL_PASSWORDdatabase:YOUR_MYSQL_DATABASE_NAMElimit_start:0# Specify the starting limit for the rangelimit_end:500# Specify the ending limit for the rangetables:-table1-table2postgresql:postgresql_example:host:YOUR_POSTGRESQL_HOSTport:YOUR_POSTGRESQL_PORTuser:YOUR_POSTGRESQL_USERNAMEpassword:YOUR_POSTGRESQL_PASSWORDdatabase:YOUR_POSTGRESQL_DATABASE_NAMElimit_start:0# Specify the starting limit for the rangelimit_end:500# Specify the ending limit for the rangetables:-table1-table2mongodb:mongodb_example:uri:YOUR_MONGODB_URIhost:YOUR_MONGODB_HOSTport:YOUR_MONGODB_PORTusername:YOUR_MONGODB_USERNAMEpassword:YOUR_MONGODB_PASSWORDdatabase:YOUR_MONGODB_DATABASE_NAMEuri:YOUR_MONGODB_URI# Use either URI or individual connection parameterslimit_start:0# Specify the starting limit for the rangelimit_end:500# Specify the ending limit for the rangecollections:-collection1-collection2fs:fs_example:path:/path/to/your/filesystem/directoryexclude_patterns:-.pdf-.docx-private-venv-node_modulesgdrive:drive_example:folder_name:credentials_file:/Users/kumarohit/Downloads/client_secret.json## this will be oauth app json filecache:trueexclude_patterns:-.pdf-.docxgdrive_workspace:drive_example:folder_name:credentials_file:/Users/kumarohit/Downloads/client_secret.json## this will be service account json fileimpersonate_users:-usera@amce.org-userb@amce.orgcache:trueexclude_patterns:-.pdf-.docxtext:profile1:text:"HelloWorldHHWPK6943Q"slack:slack_example:token:xoxp-XXXXXXXXXXXXXXXXXXXXXXXXX# get your slack app these permissiosn https://api.slack.com/methods/team.info and https://api.slack.com/methods/conversations.listchannel_types:"public_channel,private_channel"# Optional: List of channel names to check# channel_names:#   - general#   - randomYou can add or remove profiles from the connection.yml file as needed. You can also configure only one or two data sources if you don't need to scan all of them.Adding New CommandsHAWK Eye's extensibility empowers developers to contribute new security commands. Here's how:Fork the HAWK Eye repository to your GitHub account.Create a new Python file for your security command inside the commands directory, with a descriptive name.Define a function execute(args) within the new Python file, containing the logic for your command.Provide clear documentation and comments explaining the purpose and usage of the new command.Thoroughly test your command to ensure it works seamlessly and aligns with the existing features.Submit a pull request from your branch to the main HAWK Eye repository.The maintainers will review your contribution, provide feedback if needed, and merge your changes.Contribution GuidelinesWe welcome contributions from the open-source community to enhance HAWK Eye's capabilities in securing data sources. To contribute:Fork the HAWK Eye repository to your GitHub account.Create a new branch from the main branch for your changes.Adhere to the project's coding standards and style guidelines.Write clear and concise commit messages for your changes.Include appropriate test cases for new features or modifications.Update the "README.md" file to reflect any changes or new features.Submit a pull request from your branch to the main branch of the HAWK Eye repository.The maintainers will review your pull request and work with you to address any concerns.After approval, your contributions will be merged into the main codebase.Join the HAWK Eye community and contribute to data source security worldwide. For any questions or assistance, feel free to open an issue on the repository.If you find HAWK Eye useful and would like to support the project, please consider making a donation. All 100% of the donations will be distributed to charities focused on education welfare and animal help.Conferences and TalksBlack Hat SecTor 2023 [Arsenal]Black Hat Middle East and Africa 2023 [Arsenal]Black Hat Europe 2023 [Arsenal]💪 ContributorsWe extend our heartfelt appreciation to all contributors who continuously improve this tool! Your efforts are essential in strengthening the security landscape. 🙏DonationHow to DonateFeel free to make a donation directly to the charities of your choice or send it to us, and we'll ensure it reaches the deserving causes. Just reach out to us onLinkedInorTwitterto let us know about your contribution. Your generosity and support mean the world to us, and we can't wait to express our heartfelt gratitude.Your donations will play a significant role in making a positive impact in the lives of those in need. Thank you for considering supporting our cause!