Package: unja
UnjaFetch Known UrlsWhat's Unja?Unja is a fast & light tool for fetching known URLs from Wayback Machine, Common Crawl, Virus Total, UrlScan.io & AlienVault's Otx it uses a separate thread for each provider to optimize its speed and use Wayback resumption key to divide scan into multiple parts to handle a large scan & it uses direct filters on API to get only filtered data from API to do less work on your system.Why Unja?SupportsWayback/Common-Crawl/Virus-Total/Otx/UrlScan.ioAutomatically handles rate limits and timeoutsExport results: text or detailed output with status,mime,length in JSONMultiThreading: separate thread for each provider to fetch data simultaneouslyFilters: apply filters dirtly on provider to avoid unnecessary dataInstalling UnjaYou can installUnjawith pip as following:pip3 install unjaor, by downloading this repository and runningpython3 setup.py installUpdating UnjaYou can updateUnjawith pip as following:pip3 install unja -UUsageunja-hThis will display help for the tool.FlagDescriptionExample-ddoimainunja -d ninjhacks.com-fList of domains file seprated by new lineunja -f domains.txt--subInclude subdomainunja --sub-pProviders (wayback,commoncrawl,otx,virustotal,urlscan)unja -p wayback--wbf(default : statuscode:200 ~mimetype:html)unja --wbf statuscode:200--ccf(default : =status:200 ~mime:.*html)unja --ccf =status:200--wblWayback results per request (default : 10000)unja --wbl 1000--otxlOtx results per request (default : 500)unja --otxl 500-rAmount of retries for http client (default : 3)unja -r 3-vEnable verbose mode to show errorsunja -v-jEnable json mode for detailed output in json formatunja -j-sSilent mode don't print headerunja -s--ucciUpdate CommonCrawl Indexunja --ucci--vtkeyChange VirusTotal Api in configunja --vtkey--uskeyChange UrlScan Api in configunja --uskeyOutput Methodstext = ( default ) Output urls only.json = ( -j ) Output url,status,mime,length in json format it's can help you later filtering result based on those variables.FiltersFilters directly apply on providers to get only useful filtered data from provider.WaybackCommoncrawlDescriptionstatuscode:200=status:200return only those urls which status code is 200!statuscode:200!=status:200return only non 200 status codemimetype:text/htmlmime:text/htmlreturn only those url which response type is text/html!mimetype:text/html!=mime:text/htmlreturn only non text/html response type~mimetype:html~mime:.*htmlreturn all those url which have html word in response type~original:unja~url:.*unjareturn all those url which have unja word in urlOnelinersGet only urls with parameters & status code 200unja -s -d target.com --sub -p wayback,commoncrawl --wbf 'statuscode:200 ~original:=' --ccf '=status:200 ~url:.*=' | anew | tee outputLooking for open redirectsunja -s -d target.com --sub -p wayback,commoncrawl --wbf '~statuscode:30 ~original:=http' --ccf '~status:30 ~url:.*=http' | anew | tee outputClean result ( Exclude images,css,javascripts,woff & 404)unja -s -d target.com --sub -p wayback,commoncrawl --wbf '!statuscode:404 ~!mimetype:image ~!mimetype:javascript ~!mimetype:css ~!mimetype:woff' --ccf '!=status:404 !~mime:.*image !~mime:.*javascript !~mime:.*css !~mime:.*woff' | anew | tee outputLet me know if you have any other good oneliner ./