Package: threatingestor
ThreatIngestorAn extendable tool to extract and aggregateIOCsfrom threat feeds.Integrates out-of-the-box withThreatKBandMISP, and can fit seamlessly into any existing workflow withSQS,Beanstalk, andcustom plugins.Currently used by InQuest Labs IOC-DB:https://labs.inquest.net/iocdbOverviewThreatIngestor can be configured to watch Twitter, RSS feeds, sitemap (XML) feeds, or other sources, and extract meaningful information such as malicious IPs/domains and YARA signatures, and send that information to another system for analysis.Try it out now with thisquick walkthrough, read moreThreatIngestor walkthroughson the InQuest blog, and check outlabs.inquest.net/iocdb, an IOC aggregation and querying tool powered by ThreatIngestor.InstallationThreatIngestor requires Python 3.6+, with development headers.Install ThreatIngestor from PyPI:pipinstallthreatingestorInstall optional dependencies for using some plugins, as needed:pipinstallthreatingestor[all]View thefull installation instructionsfor more information.UsageCreate a newconfig.ymlfile, and configure each source and operator module you want to use. (Seeconfig.example.ymlfor layout.) Then run the script:threatingestorconfig.ymlBy default, it will run forever, polling each configured source every 15 minutes.If you'd like to run the image extraction source, or include the image extraction functionality for other sources, you will need to be running Python 3.7 >= due to the dependencies:pipinstallopencv-pythonpytesseractnumpyView thefull ThreatIngestor documentationfor more information.PluginsThreatIngestor uses a plugin architecture with "source" (input) and "operator" (output) plugins. The currently supported integrations are:SourcesGit repositoriesGitHub repository searchGists by usernameRSS feedsSitemap feedsImage extractionAmazon SQS queuesTwitterGeneric web pagesOperatorsCSV filesMISPMySQL tableSQLite databaseAmazon SQS queuesThreatKBTwitterView thefull ThreatIngestor documentationfor more information on included plugins, and how to create your own.Threat Intel SourcesLooking for some threat intel sources to get started? InQuest has a Twitter List with several accounts that post C2 domains and IPs:https://twitter.com/InQuest/lists/ioc-feed. Note that you will need to apply for a Twitter developer account to use the ThreatIngestor Twitter Source. Take a look atconfig.example.ymlto see how to set this list up as a source.For quicker setup, RSS feeds can be a great source of intelligence. Check out this exampleRSS config filefor a few pre-configured security blogs.SupportIf you need help getting set up, or run into any issues, feel free to open anissue. You can also reach out to@InQueston Twitter or read more about us on the web athttps://www.inquest.net.We'd love to hear any feedback you have on ThreatIngestor, its documentation, or how you're putting it to work for you!ContributingIssues and pull requests are welcomed. Please keep Python code PEP8 compliant. By submitting a pull request you agree to release your submissions under the terms of theLICENSE.DockerProductionADockerfileis available for running ThreatIngestor within a Docker container.First, you'll need to build the container:dockerbuild.-tthreatingestorAfter that, you can mount the container by using this command:dockerrun-it--mounttype=bind,source=/,target=/dockthreatingestor/bin/bashAfter you've mounted the container and you're inside the/bin/bashshell, you can run ThreatIngestor like normal:threatingestorconfig.ymlDevelopmentThere is also a Dockerfile.dev for building a development version of ThreatIngestor. All you need is an available .whl file, which can be generated with the following command:python3-mbuildAfter you've built the project, you can build the container:dockerbuild.-tthreatingestor-fDockerfile.devNOTE: If you run into any issues while building the development environment or running ThreatIngestor within the container, you may need to comment out the following lines inDockerfile.devto work properly:FROM ubuntu:18.04
...
# RUN apt-get install tesseract-ocr -y
...
# RUN pip3 install opencv-python pytesseract numpy
...Extra ScriptsSome scripts are now provided to help with your local configuration of ThreatIngestor.A README.md with additional information is availablehere.