Package: cewler
CeWLeR - Custom Word List generator RedefinedCeWLeRcrawls from a specified URL and collects words to create a custom wordlist.It's a great tool for security testers and bug bounty hunters. The lists can be used for password cracking, subdomain enumeration, directory and file brute forcing, API endpoint discovery, etc. It's good to have an additional target specific wordlist that is different than what everybody else use.CeWLeRwas sort of originally inspired by the really nice toolCeWL. I had some challenges withCeWLon a site I wanted a wordlist from, but without any Ruby experience I didn't know how to contribute or work around it. So instead I created a custom wordlist generator in Python to get the job done.At a glanceFeaturesGenerates custom wordlists by scraping words from web sitesA lot of options:Output to screen or fileCan stay within subdomain, or visit sibling and child subdomains, or visit anything within the same top domainCan stay within a certain depth of a websiteSpeed can be controlledWord length and casing can be configuredJavaScript and CSS can be includedCrawled URLs can be output to separate fileScraped e-mail addresses can also be output to separate file++Using the excellentScrapyframework for scraping and using the beautifulrichlibrary for terminal outputCommands and optionsQuick examplesOutput to fileWill output to file unless a file is specified.cewler --output wordlist.txt https://example.comControl speed and depthThe rate is specified in requests per second. Please play nicely and don't don't break any rules.cewler --output wordlist.txt --rate 5 --depth 2 https://example.comChange User-Agent headerThe default User-Agent is a common browser.cewler --output wordlist.txt --user-agent "Cewler" https://example.comControl casing, word length and charactersUnless specified the words will have mixed case and be of at least 5 in length.cewler --output wordlist.txt --lowercase --min-word-length 2 --without-numbers https://example.comVisit all domains - including parent, children and siblingsThe default is to just visit exactly the same (sub)domain as specified.cewler --output wordlist.txt -s all https://example.comVisit same (sub)domain + any belonging child subdomainscewler --output wordlist.txt -s children https://example.comInclude JavaScript and/or CSSIf you want you can include links from<script>and<link>tags, plus words from within JavaScript and CSS.cewler --output wordlist.txt --include-js --include-css https://example.comOutput visited URLs to fileIt's also possible to store the crawled files to a file.cewler --output wordlist.txt --output-urls urls.txt https://example.comOutput e-mails to fileIt's also possible to store the scraped e-mail addresses to a separate file (they are always added to the wordlist).cewler --output wordlist.txt --output-emails emails.txt https://example.comNinja trick 🥷If it just takes too long to crawl a site you can pressctrl + conce(!) and wait while the spider finishes the current requests and then whatever words have been found so far is stored to the output file.All optionscewler -h
usage: cewler [-h] [-d DEPTH] [-js] [-l] [-m MIN_WORD_LENGTH] [-o OUTPUT] [-ou OUTPUT_URLS] [-r RATE] [-s {all,children,exact}] [--stream] [-u USER_AGENT] [-v] [-w] url

CeWLeR - Custom Word List generator Redefined

positional arguments:
  url                   URL to start crawling from

options:
  -h, --help            show this help message and exit
  -d DEPTH, --depth DEPTH
                        directory path depth to crawl, 0 for unlimited (default: 2)
  -css, --include-css   include CSS from external files and <style> tags
  -js, --include-js     include JavaScript from external files and <script> tags
  -l, --lowercase       lowercase all parsed words
  -m MIN_WORD_LENGTH, --min-word-length MIN_WORD_LENGTH
  -o OUTPUT, --output OUTPUT
                        file were to stream and store wordlist instead of screen (default: screen)
  -oe OUTPUT_EMAILS, --output-emails OUTPUT_EMAILS
                        file were to stream and store e-mail addresses found (they will always be outputted in the wordlist)
  -ou OUTPUT_URLS, --output-urls OUTPUT_URLS
                        file were to stream and store URLs visited (default: not outputted)
  -r RATE, --rate RATE  requests per second (default: 20)
  -s {all,children,exact}, --subdomain_strategy {all,children,exact}
                        allow crawling [all] domains, including children and siblings, only [exact] the same (sub)domain (default), or same domain and any belonging [children]
  --stream              writes to file after each request (may produce duplicates because of threading) (default: false)
  -u USER_AGENT, --user-agent USER_AGENT
                        User-Agent header to send (default: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36)
  -v, --verbose         A bit more detailed output
  -w, --without-numbers
                        ignore words are numbers or contain numbersSubdomain strategiesExample URL to scanhttps://sub.example.com:-s exact*-s children-s allsub.example.com✅✅✅child.sub.example.com❌✅✅sibling.example.com❌❌✅example.com❌❌✅* Default strategyDigging into the codeIf you want to do some tweaking you yourself you can probably find what you want insrc/cewler/constants.pyandsrc/cewler/spider.pyInstallation and upgradeAlternative 1 - installing from PyPIPackage homepage:https://pypi.org/project/cewler/python3 -m pip install cewlerUpgradepython3 -m pip install cewler --upgradeAlternative 2 - installing from GitHub1. Clone repositorygit clone https://github.com/roys/cewler.git --depth 12. Install dependenciescd cewler
python3 -m pip install -r requirements.txt3. Shortcut on Un*x based system (optional)cd src/cewler
chmod +x cewler.py
ln -s $(pwd)/cewler.py /usr/local/bin/cewler
cewler -hUpgradegit pullPronunciationCeWLeRis pronounced"cooler".LicenseAttribution-NonCommercial 4.0 International (CC BY-NC 4.0)