Package: mist-lang
When you need to create complex Workflows and need to communicate different tools working together, maybe you needMIST.What is MISTMISTis a high level programming language for defining executions workflows easily.MISTis interpreted. So, you can use their command line interpreter for running.mistprograms.MISTinterpreter will create the workflow graph, execute each tool, manage executions and synchronization fo you.A quick example about how to run aMISTprogram:>mistrunmy_program.mistInstalling>pipinstallmist-langQuick StartRequirementsBefore start, we should install some command line tools used by catalog functions in the Demos:dnsrecon (for searchDomains)Mac & Linux:pip install git+https://github.com/cr0hn/dnsreconnmap (fir findOpenPorts)Mac:brew install nmapUbuntu:sudo apt install nmapkafka-console-consumer & kafka-console-producerMac:brew install kafkaUbuntu:sudo apt install kafkaNOTE: For Demo 3 to 5 a Kafka server is expected to be running at localhostfestinMac & Linux:pip install festinNOTE: Is also recommended to installtorin order to prevent being banned when using festinaws (for S3Store)Mac:brew install awscliUbuntu:sudo apt install awscliDemo 1 - The simplest scenarioExplanationIn this scenario we'll do:CLI Input- Read a domain as a parameter from CLI.Search Domains- Use MIST function for search related domains / sub-domains from a start domain.Fin OpenPorts- Search open port for each new domain / sub-domain found.Screen (Pring)- Displays the results into the screen (by using MIST 'print' function).Use case diagramMIST code(examples/demo/scenario-01.mist)include"searchDomains""findOpenPorts"searchDomains(%domain)=>findOpenPorts("80,443")=>print()Execute>mistrunexamples/demo/scenario-01.mistdomain=example.comDemo 2 - Sending results to KafkaExplanationIn this scenario we'll do:CLI Input- Read a domain as a parameter from CLI.Search Domains- Use MIST function for search related domains / sub-domains from a start domain.FindOpenPorts- Search open port for each new domain / sub-domain found.Kafka output- Send results to a Kafka topic.Use case diagramMIST code(examples/demo/scenario-02.mist)include"searchDomains""findOpenPorts""kafkaProducer"searchDomains(%domain)=>findOpenPorts("80,443")=>kafkaProducer($KAFKA_SERVER,"domainsTopic")Execute>mistrunexamples/demo/scenario-02.mistdomain=example.comDemo 3 - Adding new tool and remove duplicate domainsExplanationIn this scenario we'll do:CLI Input- Read a domain as a parameter from CLI.Search domains:Search Domains- Use MIST function for search related domains / sub-domains from a start domain.Festin- Use MIST integration forFestinfor search related domains / sub-domains from a start domain.Filter Repeated- Use MIST function to detect and remove repeated found domains.Fin OpenPorts- Search open port for each new domain / sub-domain get fromFitler Repeated.Kafka output- Send results to a Kafka topic.Use case diagramMIST code(examples/demo/scenario-03.mist)include"searchDomains""festin""findOpenPorts""filterRepeated""kafkaProducer"searchDomains(%domain)=>foundDomains
festin(%domain,$DNS_SERVER,True)=>foundDomainsfoundDomains=>filterRepeated(False)=>findOpenPorts("80,443")=>kafkaProducer($KAFKA_SERVER,"domainsTopic")Execute>mistrunexamples/demo/scenario-03.mistdomain=example.comDemo 4 - Send results to Kafka and S3 through a dispatcherExplanationIn this scenario we'll do:CLI Input- Read a domain as a parameter from CLI.Search domains:Search Domains- Use MIST function for search related domains / sub-domains from a start domain.Festin- Use MIST integration forFestinfor search related domains / sub-domains from a start domain.Filter Repeated- Use MIST function to detect and remove repeated found domains.Find OpenPorts- Search open port for each new domain / sub-domain get fromFitler Repeated.Dispatcher (80 / 443)- Split results and send each port to a different queue.Send results:Kafka output- Send found 80 ports to a Kafka topic.S3 output- Send found 443 ports to a AWS S3 bucket.Use case diagramMIST code(examples/demo/scenario-04.mist)include"searchDomains""festin""findOpenPorts""filterRepeated""kafkaProducer""S3Store"functiondispatcher(p)=>kafka,S3{if(isEqual(p.port,"80")){p=>kafka}else{p=>S3}}searchDomains(%domain)=>foundDomains
festin(%domain,$DNS_SERVER,True)=>foundDomainsfoundDomains=>filterRepeated(False)=>findOpenPorts("80,443")=>dispatcher()=>kafkaOutput,S3OutputkafkaOutput=>kafkaProducer($KAFKA_SERVER,"domainsTopic")S3Output=>S3Store($BUCKET_URI)Execute>mistrunexamples/demo/scenario-04.mistdomain=example.comDemo 5 - Read from Kafka and a FileExplanationIn this scenario we'll do:1 Input from multiple sources:File Input- Read domains from an external file.Kafka Input- Read domains from Kafka topics.CLI Input- Read domains from CLI.Search domains:Search Domains- Use MIST function for search related domains / sub-domains from a start domain.Festin- Use MIST integration forFestinfor search related domains / sub-domains from a start domain.Filter Repeated- Use MIST function to detect and remove repeated found domains.Find OpenPorts- Search open port for each new domain / sub-domain get fromFitler Repeated.Dispatcher (80 / 443)- Split results and send each port to a different queue.Send results:Kafka output- Send found 80 ports to a Kafka topic.S3 output- Send found 443 ports to a AWS S3 bucket.Use case diagramMIST code(examples/demo/scenario-05.mist)include"searchDomains""festin""findOpenPorts""filterRepeated""kafkaProducer""S3Store""kafkaConsumer""tail"functiondispatcher(p)=>kafka,S3{if(isEqual(p.port,"80")){p=>kafka}else{p=>S3}}kafkaConsumer($KAFKA_SERVER,"inputTopic","*END*",False)=>inputDomains
tail("domains.txt","*END*")=>inputDomains
%domain=>inputDomainsinputDomains=>searchDomains()=>foundDomainsinputDomains=>festin($DNS_SERVER,True)=>foundDomainsfoundDomains=>filterRepeated(False)=>findOpenPorts("80,443")=>dispatcher()=>kafkaOutput,S3OutputkafkaOutput=>kafkaProducer($KAFKA_SERVER,"domainsTopic")S3Output=>S3Store($BUCKET_URI)Execute>mistrunexamples/demo/scenario-05.mistdomain=example.comAuthorsMIST is being developed by BBVA-Labs Security team members.ContributionsContributions are of course welcome. SeeCONTRIBUTINGor skim existing tickets to see where you could help out.LicenseMIST is Open Source Software and available under theApache 2 license