How to Scan Websites for Interesting Directories & Files with Gobuster « Null Byte :: WonderHowTo

One of the first steps in attacking aweb applicationis enumerating hidden directories and files. Doing so can often yield valuable information that makes it easier to execute a precise attack, leaving less room for errors and wasted time. There are many tools available to do this, but not all of them are created equally. Gobuster, a directory scanner written inGo, is definitely worth exploring.Traditional directorybrute-forcescanners likeDirBusterandDIRBwork just fine, but can often be slow and prone to errors.Gobusteris a Go implementation of these tools and is offered in a convenient command-line format.The main advantage Gobuster has over other directory scanners is speed. As a programming language, Go is known to be fast. It also has excellent support forconcurrencyso that Gobuster can take advantage of multiple threads for faster processing.The one downfall of Gobuster, though, is the lack ofrecursivedirectory searching. For directories more than one level deep, another scan will be needed, unfortunately. Often this isn't that big of a deal, and other scanners can step up and fill in the gaps for Gobuster in this area.Don't Miss:Use Websploit to Scan Websites for Hidden DirectoriesGobuster offers a simple command-line interface that just works. It has some useful options, but not so many that it's easy to get bogged down in the details. All in all, it's a great tool that is effective and fast. In this tutorial, we'll be exploring it withDVWA(Damn Vulnerable Web App) as the target andKali Linuxas the attacking machine. You can follow along with those or use a similar testing configuration.Step 1: Install GobusterThe first thing we can do iscreate a working directoryto keep things neat, thenchangeinto it.~# mkdir gobuster
~# cd gobuster/Next, we need to install Gobuster since it is not included on Kali by default.~/gobuster# apt-get install gobuster

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  gobuster
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/1,532 kB of archives.
After this operation, 4,963 kB of additional disk space will be used.
Selecting previously unselected package gobuster.
(Reading database ... 412624 files and directories currently installed.)
Preparing to unpack .../gobuster_2.0.1-1_amd64.deb ...
Unpacking gobuster (2.0.1-1) ...
Setting up gobuster (2.0.1-1) ...
Processing triggers for man-db (2.8.5-2) ...Then, simply typegobusterin the terminal to run the tool.~/gobuster# gobuster

2019/05/06 11:43:08 [!] 2 errors occurred:
    * WordList (-w): Must be specified (use `-w -` for stdin)
    * Url/Domain (-u): Must be specifiedWe can see it gives us a couple of errors. It needs at least two parameters (-ufor the URL, and-wfor the wordlist) to run properly. We can also display the help menu with the-hflag.~/gobuster# gobuster -h

Usage of gobuster:
  -P string
        Password for Basic Auth (dir mode only)
  -U string
        Username for Basic Auth (dir mode only)
  -a string
        Set the User-Agent string (dir mode only)
  -c string
        Cookies to use for the requests (dir mode only)
  -cn
        Show CNAME records (dns mode only, cannot be used with '-i' option)
  -e    Expanded mode, print full URLs
  -f    Append a forward-slash to each directory request (dir mode only)
  -fw
        Force continued operation when wildcard found
  -i    Show IP addresses (dns mode only)
  -k    Skip SSL certificate verification
  -l    Include the length of the body in the output (dir mode only)
  -m string
        Directory/File mode (dir) or DNS mode (dns) (default "dir")
  -n    Don't print status codes
  -np
        Don't display progress
  -o string
        Output file to write results to (defaults to stdout)
  -p string
        Proxy to use for requests [http(s)://host:port] (dir mode only)
  -q    Don't print the banner and other noise
  -r    Follow redirects
  -s string
        Positive status codes (dir mode only) (default "200,204,301,302,307,403")
  -t int
        Number of concurrent threads (default 10)
  -to duration
        HTTP Timeout in seconds (dir mode only) (default 10s)
  -u string
        The target URL or Domain
  -v    Verbose output (errors)
  -w string
        Path to the wordlist
  -x string
        File extension(s) to search for (dir mode only)Step 2: Install Some Extra WordlistsWordlistson Kali are located in the/usr/share/wordlistsdirectory.~/gobuster# ls /usr/share/wordlists/

dirb  dirbuster  dnsmap.txt  fasttrack.txt  fern-wifi  metasploit  nmap.lst  rockyou.txt.gz  sqlmap.txt  wfuzzThe dirb and dirbuster ones are fine, but there is another wordlist I like to use for directory brute-forcing. There is a whole repository of useful wordlists on GitHub calledSecLists.The "common.txt" wordlist contains a good number of common directory names.We can download the raw file into our current directory using thewgetutility.~/gobuster# wget https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt

--2019-05-06 11:46:40--  https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.148.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.148.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35744 (35K) [text/plain]
Saving to: ‘common.txt’

common.txt                                            100%[======================================================================================================================>]  34.91K  --.-KB/s    in 0.03s

2019-05-06 11:46:40 (1.24 MB/s) - ‘common.txt’ saved [35744/35744]Alternatively, if we wanted to install the whole SecLists repository, we can do so with thepackage manager.~/gobuster# apt-get install seclists

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  seclists
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/221 MB of archives.
After this operation, 795 MB of additional disk space will be used.
Selecting previously unselected package seclists.
(Reading database ... 412629 files and directories currently installed.)
Preparing to unpack .../seclists_2019.1-0kali1_all.deb ...
Unpacking seclists (2019.1-0kali1) ...
Setting up seclists (2019.1-0kali1) ...That will install the entirety of SecLists in the/usr/sharedirectory. It contains a ton of content, and if you're looking for a go-to set of wordlists, SecLists will likely be the last thing you'll ever need./usr/share/seclists# ls

Discovery  Fuzzing  IOCs  Miscellaneous  Passwords  Pattern-Matching  Payloads  README.md  Usernames  Web-ShellsStep 3: Scan Directories & FilesNow that everything is set up and installed, we're ready to use Gobuster. Let's run it against our target with the default parameters.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Timeout      : 10s
=====================================================
2019/05/06 11:50:25 Starting gobuster
=====================================================
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/.hta (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/about (Status: 302)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/index (Status: 302)
/index.php (Status: 302)
/php.ini (Status: 200)
/instructions (Status: 302)
/logout (Status: 302)
/robots (Status: 200)
/robots.txt (Status: 200)
/login (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/setup (Status: 200)
/security (Status: 302)
=====================================================
2019/05/06 11:50:38 Finished
=====================================================We can see it gives us some information about the tool in the banner, and then it kicks off the directory discovery process. It returns the names of the directories and files as well as theirstatus codes.We can have it return the length of the body with the-lflag, which can be useful for enumeration.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -l

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Show length  : true
[+] Timeout      : 10s
=====================================================
2019/05/06 11:52:41 Starting gobuster
=====================================================
/.hta (Status: 403) [Size: 292]
/.htaccess (Status: 403) [Size: 297]
/.htpasswd (Status: 403) [Size: 297]
/README (Status: 200) [Size: 4934]
/config (Status: 301) [Size: 319]
/docs (Status: 301) [Size: 317]
/external (Status: 301) [Size: 321]
/favicon.ico (Status: 200) [Size: 1406]
2019/05/06 11:52:52 [!] Get http://10.10.0.50/dvwa/about: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/php.ini (Status: 200) [Size: 148]
2019/05/06 11:52:54 [!] Get http://10.10.0.50/dvwa/cmd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/robots (Status: 200) [Size: 26]
/robots.txt (Status: 200) [Size: 26]
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index.php: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/instructions: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/login: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/logout: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/phpinfo.php (Status: 302) [Size: 0]
/phpinfo (Status: 302) [Size: 0]
/security (Status: 302) [Size: 0]
/setup (Status: 200) [Size: 3549]
=====================================================
2019/05/06 11:53:01 Finished
=====================================================Usually, if something is zero bytes, it isn't even worth looking into. It can save loads of time, especially when dealing with a large website or a large number of directories. If we only want specific status codes to be displayed, we can do that using the-sflag followed by the code we want.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:16 Starting gobuster
=====================================================
/README (Status: 200)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:54:27 Finished
=====================================================Use commas to specify multiple codes.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200,301

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,301
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:58 Starting gobuster
=====================================================
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:55:10 Finished
=====================================================Let's say we just wanted a quick way to view the directories, without the extra noise of the banner and status codes. Use the-qflag to hide the banner, and the-nflag to hide the status codes.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -q -n

/.hta
/.htpasswd
/.htaccess
/README
/config
/docs
/external
/favicon.ico
/about
/cmd
/php.ini
/index
/index.php
/instructions
/logout
/robots
/robots.txt
/login
/phpinfo.php
/phpinfo
/setup
/securityAnother useful feature is the ability to save the results to a file. Use the-oflag to specify the output file.~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -o resultsNow the results are saved and can be viewed at a later time.~/gobuster# cat results

/.hta (Status: 403)
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/about (Status: 302)
/cmd (Status: 200)
/php.ini (Status: 200)
/instructions (Status: 302)
/index (Status: 302)
/logout (Status: 302)
/index.php (Status: 302)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/security (Status: 302)
/setup (Status: 200)Wrapping UpIn this tutorial, we learned about Gobuster, a directory brute-force scanner written inthe Go programming language. First, we learned how to install the tool, as well as some useful wordlists not found on Kali by default. Next, we ran it against our target and explored some of the various options it ships with. The bottom line: Gobuster is a fast and powerful directory scanner that should be an essential part of any hacker's repertoire, and now you know how to use it. Go!Don't Miss:Punchabunch Just Made SSH Local Forwarding Stupid EasyWant to start making money as a white hat hacker?Jump-start your hacking career with our2020 Premium Ethical Hacking Certification Training Bundlefrom the newNull Byte Shopand get over 60 hours of training from cybersecurity professionals.Buy Now (90% off) >Other worthwhile deals to check out:97% off The Ultimate 2021 White Hat Hacker Certification Bundle99% off The 2021 All-in-One Data Scientist Mega Bundle98% off The 2021 Premium Learn To Code Certification Bundle62% off MindMaster Mind Mapping Software: Perpetual LicenseCover image byPixabay/Pexels; Screenshots by drd_/Null ByteRelatedHow To:Find Hidden Web Directories with DirsearchHow To:Use Websploit to Scan Websites for Hidden DirectoriesHack Like a Pro:How to Find Directories in Websites Using DirBusterHow To:Anti-Virus in Kali LinuxHack Like a Pro:Metasploit for the Aspiring Hacker, Part 10 (Finding Deleted Webpages)Hack Like a Pro:How to Find Website Vulnerabilities Using WiktoHow To:Detect Vulnerabilities in a Web Application with UniscanHow To:Discover & Attack Services on Web Apps or Networks with SpartaHow To:Perform Directory Traversal & Extract Sensitive InformationHack Like a Pro:How to Hack Web Apps, Part 7 (Finding Hidden Objects with DIRB)Android for Hackers:How to Scan Websites for Vulnerabilities Using an Android Phone Without RootHow To:Use Metasploit's WMAP Module to Scan Web Applications for Common VulnerabilitiesHow To:Conduct Recon on a Web Target with Python ToolsHack Like a Pro:How to Scan the Internet for Heartbleed VulnerabilitiesHow To:Hack Apache Tomcat via Malicious WAR File UploadHow To:Scan for Vulnerabilities on Any Website Using NiktoHack Like a Pro:Linux Basics for the Aspiring Hacker, Part 21 (GRUB Bootloader)How To:Build a Directory Brute Forcing Tool in PythonHow To:Detect Misconfigurations in 'Anonymous' Dark Web Sites with OnionScanHow To:Improve Battery Life on Android by Optimizing Your Media ScannerHack Like a Pro:Linux Basics for the Aspiring Hacker, Part 2 (Creating Directories & Files)How To:How Hackers Use Your IP Address to Hack Your Computer & How to Stop ItHow To:Scan for Viruses in Windows Using a Linux Live CD/USBNews:First Steps of Compiling a Program in LinuxHow To:Download and Install the Minecraft 1.8 Pre-ReleaseGoodnight Byte:HackThisSite Walkthrough, Part 10 - Legal Hacker TrainingCommunity Contest:Code the Best Hacking Tool, Win Bragging RightsGoodnight Byte:HackThisSite Walkthrough, Part 6 - Legal Hacker TrainingGoodnight Byte:HackThisSite Walkthrough, Part 8 - Legal Hacker TrainingCongress:The Law MakersHow To:Recover Deleted Files in LinuxHow To:Start With Site Setting For Snoft Article Directory ScriptGoogle Dorking:AmIDoinItRite?Rainbow Tables:How to Create & Use Them to Crack PasswordsHack Logs and Linux Commands:What's Going On Here?News:"How To Make Compost" Zine (Download)How To:Install "Incompatible" Firefox Add-Ons After Upgrading to the New Firefox