Package: hamburglar
The HamburglarSetupThere are 2 versions of hamburglar, full and lite. The main branch is the full version, and hamburglar lite is on a separate branch.HamburglarFull fledged scraping tool for artifact retrieval from multiple sources. There are some dependencies, so install them first:pip3 install -r requirements.txtHamburglar also has the option of checking against file signatures during a hexdump. It will get skipped if not set up. To get it working, you will need to first create the database and a user:CREATEDATABASECREATEUSER'hamman'@'localhost'IDENTIFIEDBY'deadbeef';GRANTALLPRIVILEGESONfileSign.signaturesTO'hamman'@'localhost';Then, run magic_sig_scraper. This can be run on a cronjob to regularly update it, or just run it once:python3 magic_sig_scraper.pyHamburglar LiteMultithreaded and recursive directory scraping script. Stores useful information with the filepath and finding. Hamburglar lite will never require external packages, and will always remain as a single script. Setup is as simple as requesting the file and using it:wget https://raw.githubusercontent.com/needmorecowbell/Hamburglar/hamburglar-lite/hamburglar-lite.pyThis is designed to be quickly downloaded and executed on a machine.Operationusage: hamburglar.py [-h] [-g] [-x] [-v] [-w] [-i] [-o FILE] [-y YARA] path

positional arguments:
  path                  path to directory, url, or file, depending on flag
                        used

optional arguments:
  -h, --help            show this help message and exit
  -g, --git             sets hamburglar into git mode
  -x, --hexdump         give hexdump of file
  -v, --verbose         increase output verbosity
  -w, --web             sets Hamburgler to web request mode, enter url as path
  -i, --ioc             uses iocextract to parse contents
  -o FILE, --out FILE   write results to FILE
  -y YARA, --yara YARA  use yara ruleset for checkingDirectory Traversalpython3 hamburglar.py ~/Directory/This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filtersSingle File Analysispython3 hamburglar.py ~/Directory/file.txtThis will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filtersYARA Rule Based Analysispython3 hamburglar.py -y rules/ ~/DirectoryThis will compile the yara rule files in the rules directory and then check them against every item in Directory.Git Scraping Modepython3 hamburglar.py -g https://www.github.com/needmorecowbell/HamburglarAdding-y <rulepath>will allow the repo to be scraped using yara rulesWeb Request Modepython3 hamburglar.py -w https://google.comAdding a-wto hamburgler.py tells the script to handle the path as a url.Currently this does not spider the page, it just analyzes the requested html contentIOC Extractionpython3 hamburglar.py -w -i https://pastebin.com/SYisR95mAdding a-iwill use iocextract to extract any ioc's from the requested urlHex Dump Modepython3 hamburglar.py -x ~/file-to-dumpThis just does a hex dump and nothing more right now -- could be piped into a fileThis will eventually be used for binary analysisTipsAdding-vwill set the script into verbose mode, and-hwill show details of available argumentsAdding-o FILENAMEwill set the results filename, this is especially useful in scripting situations where you might want multiple results tables (ie github repo spidering)SettingswhitelistOn: turns on or off whitelist checkingmaxWorkers: number of worker threads to run concurrently when reading file stackwhitelist: list of files or directories to exclusively scan for (if whitelistOn=True)blacklist: list of files, extensions, or directories to block in scanregexList: dictionary of regex filters with filter type as the keyThe Hamburglar can findipv4 addresses (public and local)emailsprivate keysurlsioc's (using iocextract)cryptocurrency addressesanything you can imagine using regex filters and yara rulesExample output:{"/home/adam/Dev/test/email.txt":{"emails":"{'testingtesting@gmail.com'}"},"/home/adam/Dev/test/email2.txt":{"emails":"{'loall@gmail.com'}"},"/home/adam/Dev/test/ips.txt":{"ipv4":"{'10.0.11.2', '192.168.1.1'}"},"/home/adam/Dev/test/test2/email.txt":{"emails":"{'ntested4@gmail.com', 'ntested@gmail.com'}"},"/home/adam/Dev/test/test2/ips.txt":{"ipv4":"{'10.0.11.2', '192.168.1.1'}"},"/home/adam/Dev/test/test2/links.txt":{"site":"{'http://login.web.com'}"}}ContributionsPlease contribute! If there's an error let me know -- even better if you can fix it :)A big thank you to anyone who has helped:adi928jaeger-2601tijkojoanbonoandXumeiquerfor the rules fromyara-forensics