Repository: epi052/feroxbuster (rust)

Issue:
[BUG] thread 'tokio-runtime-worker' panicked at 'a parsed Url should always be a valid Uri: InvalidUri(InvalidUriChar)'
Hello,

In the middle of scanning a machine (Hack The Box "Awkward" reseted machine), feroxbuster stopped while displaying this error:

```
$ feroxbuster --no-state -egBE -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,sh,txt,cgi,html,js,css,py,log,conf,json,ini,pdf,bak,old,zip,tar,gz,tgz,rar,docx,rtf,xlsx,pptx,asp,jsp -C 403,404 -u 'http://hat-valley.htb/'

 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.7.1
───────────────────────────┬──────────────────────
 🎯  Target Url            │ http://hat-valley.htb/
 🚀  Threads               │ 50
 📖  Wordlist              │ /usr/share/seclists/Discovery/Web-Content/common.txt
 💢  Status Code Filters   │ [403, 404]
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.7.1
 💉  Config File           │ /etc/feroxbuster/ferox-config.toml
 🔎  Extract Links         │ true
 💲  Extensions            │ [php, sh, txt, cgi, html, js, css, py, log, conf, json, ini, pdf, bak, old, zip, tar, gz, tgz, rar, docx, rtf, xlsx, pptx, asp, jsp]
 💰  Collect Extensions    │ true
 💸  Ignored Extensions    │ [Images, Movies, Audio, etc...]
 🏦  Collect Backups       │ true
 🤑  Collect Words         │ true
 🏁  HTTP methods          │ [GET]
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
WLD      GET       54l      163w     2881c Got 200 for http://hat-valley.htb/f9ca3e24ff5f4bcba810a75032bb2fcb (url length: 32)
WLD      GET         -         -         - Wildcard response is static; auto-filtering 2881 responses; toggle this behavior by using --dont-filter
301      GET       10l       16w      173c http://hat-valley.htb/css => /css/
200      GET        1l       35w     4286c http://hat-valley.htb/favicon.ico
301      GET       10l       16w      171c http://hat-valley.htb/js => /js/
[#######>------------] - 10m   497584/1404538 19m     found:4       errors:2      
[############>-------] - 10m    80354/127278  127/s   http://hat-valley.htb/ 
[############>-------] - 10m    80163/127278  127/s   http://hat-valley.htb/.git/logs/ 
[########>-----------] - 9m     53325/127278  93/s    http://hat-valley.htb/cgi-bin/ 
[########>-----------] - 9m     52920/127278  93/s    http://hat-valley.htb/.git/logs/cgi-bin/ 
[########>-----------] - 9m     52866/127278  93/s    http://hat-valley.htb/cgi-bin/.git/logs/ 
[#######>------------] - 8m     45495/127278  87/s    http://hat-valley.htb/css 
[#######>------------] - 8m     45009/127278  87/s    http://hat-valley.htb/css/.git/logs/ 
[####>---------------] - 5m     25704/127278  74/s    http://hat-valley.htb/cgi-bin/cgi-bin/ 
[##>-----------------] - 4m     18225/127278  65/s    http://hat-valley.htb/css/cgi-bin/ 
[##>-----------------] - 4m     17550/127278  66/s    http://hat-valley.htb/js 
[##>-----------------] - 4m     17550/127278  66/s    http://hat-valley.htb/js/.git/logs/ 
thread 'tokio-runtime-worker' panicked at 'a parsed Url should always be a valid Uri: InvalidUri(InvalidUriChar)', /build/feroxbuster-quLTbv/feroxbuster-2.7.1/"/build/feroxbuster-quLTbv/feroxbuster-2.7.1/target/cargo"/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.10/src/into_url.rs:70:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
zsh: IOT instruction  feroxbuster --no-state -egBE -w  -x  -C 403,404 -u 'http://hat-valley.htb/'
```

Stack backtrace with the `RUST_BACKTRACE=1 RUST_BACKTRACE=full` variables:
```
thread 'tokio-runtime-worker' panicked at 'a parsed Url should always be a valid Uri: InvalidUri(InvalidUriChar)', /build/feroxbuster-quLTbv/feroxbuster-2.7.1/"/build/feroxbuster-quLTbv/feroxbuster-2.7.1/target/cargo"/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.10/src/into_url.rs:70:14
stack backtrace:
   0:     0x55b61c079e6f - <unknown>
   1:     0x55b61bda9dfc - <unknown>
   2:     0x55b61c05e704 - <unknown>
   3:     0x55b61c061a1b - <unknown>
   4:     0x55b61c062603 - <unknown>
   5:     0x55b61c07a262 - <unknown>
   6:     0x55b61c07a1d6 - <unknown>
   7:     0x55b61c061ef2 - <unknown>
   8:     0x55b61bd0a742 - <unknown>
   9:     0x55b61bd0ab02 - <unknown>
  10:     0x55b61c00cff1 - <unknown>
  11:     0x55b61be45a26 - <unknown>
  12:     0x55b61be6be0b - <unknown>
  13:     0x55b61be6a356 - <unknown>
  14:     0x55b61be8232c - <unknown>
  15:     0x55b61be80075 - <unknown>
  16:     0x55b61be7afca - <unknown>
  17:     0x55b61c096f39 - <unknown>
  18:     0x55b61c094abf - <unknown>
  19:     0x55b61c08a5f7 - <unknown>
  20:     0x55b61c08a42f - <unknown>
  21:     0x55b61c05fad5 - <unknown>
  22:     0x7f91c488784a - <unknown>
  23:     0x7f91c490b2cc - <unknown>
  24:                0x0 - <unknown>
```

Running the scan again still shows the issue.

Temporary workaround is to reduce recursion depth or straight up disable it with `-n`.

 - feroxbuster version: 2.7.1
 - OS: Kali Linux


Pull request description:
fixed invalid uri exception during extraction
closes #689 

# Landing a Pull Request (PR)

Long form explanations of most of the items below can be found in the [CONTRIBUTING](https://github.com/epi052/feroxbuster/blob/master/CONTRIBUTING.md) guide.

## Branching checklist
- [ ] There is an issue associated with your PR (bug, feature, etc.. if not, create one)
- [ ] Your PR description references the associated issue (i.e. fixes #123456)
- [ ] Code is in its own branch
- [ ] Branch name is related to the PR contents
- [ ] PR targets main

## Static analysis checks
- [ ] All rust files are formatted using `cargo fmt`
- [ ] All `clippy` checks pass when running `cargo clippy --all-targets --all-features -- -D warnings -A clippy::mutex-atomic`
- [ ] All existing tests pass

## Documentation
- [ ] New code is documented using [doc comments](https://doc.rust-lang.org/stable/rust-by-example/meta/doc.html)
- [ ] Documentation about your PR is included in the `docs`, as needed. The docs live in a [separate repository](https://epi052.github.io/feroxbuster-docs/docs/). Update the appropriate pages at the links below.
  - [ ] update [example config file section](https://epi052.github.io/feroxbuster-docs/docs/configuration/ferox-config-toml/)
  - [ ] update [help output section](https://epi052.github.io/feroxbuster-docs/docs/configuration/command-line/)
  - [ ] add an [example](https://epi052.github.io/feroxbuster-docs/docs/examples/)
  - [ ] update [comparison table](https://epi052.github.io/feroxbuster-docs/docs/compare/)

## Additional Tests
- [ ] New code is unit tested
- [ ] New code is integration tested, as needed
- [ ] New tests pass


Patch:
diff --git a/src/extractor/builder.rs b/src/extractor/builder.rs
index 6349607..05e4cab 100644
--- a/src/extractor/builder.rs
+++ b/src/extractor/builder.rs
@@ -13,6 +13,11 @@ pub(super) const LINKFINDER_REGEX: &str = r#"(?:"|')(((?:[a-zA-Z]{1,10}://|//)[^
 pub(super) const ROBOTS_TXT_REGEX: &str =
     r#"(?m)^ *(Allow|Disallow): *(?P<url_path>[a-zA-Z0-9._/?#@!&'()+,;%=-]+?)$"#; // multi-line (?m)
 
+/// Regular expression to filter bad characters from extracted url paths
+///
+/// ref: https://www.rfc-editor.org/rfc/rfc3986#section-2
+pub(super) const URL_CHARS_REGEX: &str = r#"["<>\\^`{|} ]"#;
+
 /// Which type of extraction should be performed
 #[derive(Debug, Copy, Clone)]
 pub enum ExtractionTarget {
@@ -90,6 +95,7 @@ impl<'a> ExtractorBuilder<'a> {
         Ok(Extractor {
             links_regex: Regex::new(LINKFINDER_REGEX).unwrap(),
             robots_regex: Regex::new(ROBOTS_TXT_REGEX).unwrap(),
+            url_regex: Regex::new(URL_CHARS_REGEX).unwrap(),
             response: if self.response.is_some() {
                 Some(self.response.unwrap())
             } else {
diff --git a/src/extractor/container.rs b/src/extractor/container.rs
index 115badd..98236f9 100644
--- a/src/extractor/container.rs
+++ b/src/extractor/container.rs
@@ -17,7 +17,7 @@ use crate::{
 use anyhow::{bail, Context, Result};
 use reqwest::{Client, StatusCode, Url};
 use scraper::{Html, Selector};
-use std::collections::HashSet;
+use std::{borrow::Cow, collections::HashSet};
 
 /// Whether an active scan is recursive or not
 #[derive(Debug)]
@@ -38,6 +38,9 @@ pub struct Extractor<'a> {
     /// `ROBOTS_TXT_REGEX` as a regex::Regex type
     pub(super) robots_regex: Regex,
 
+    /// regex to validate a url
+    pub(super) url_regex: Regex,
+
     /// Response from which to extract links
     pub(super) response: Option<&'a FeroxResponse>,
 
@@ -332,8 +335,9 @@ impl<'a> Extractor<'a> {
         let normalized_path = self.normalize_url_path(path);
 
         // filter out any empty strings caused by .split
-        let mut parts: Vec<&str> = normalized_path
+        let mut parts: Vec<Cow<_>> = normalized_path
             .split('/')
+            .map(|s| self.url_regex.replace_all(s, ""))
             .filter(|s| !s.is_empty())
             .collect();
 
@@ -392,6 +396,17 @@ impl<'a> Extractor<'a> {
             .join(link)
             .with_context(|| format!("Could not join {} with {}", old_url, link))?;
 
+        if old_url.domain() != new_url.domain() || old_url.host() != old_url.host() {
+            // domains/ips are not the same, don't scan things that aren't part of the original
+            // target url
+            log::debug!(
+                "Skipping {} because it's not part of the original target",
+                new_url
+            );
+            log::trace!("exit: add_link_to_set_of_links");
+            return Ok(());
+        }
+
         links.insert(new_url.to_string());
 
         log::trace!("exit: add_link_to_set_of_links");


Test patch:
diff --git a/src/extractor/tests.rs b/src/extractor/tests.rs
index e8e9756..b485cd0 100644
--- a/src/extractor/tests.rs
+++ b/src/extractor/tests.rs
@@ -1,4 +1,4 @@
-use super::builder::{LINKFINDER_REGEX, ROBOTS_TXT_REGEX};
+use super::builder::{LINKFINDER_REGEX, ROBOTS_TXT_REGEX, URL_CHARS_REGEX};
 use super::*;
 use crate::config::{Configuration, OutputLevel};
 use crate::scan_manager::ScanOrder;
@@ -273,6 +273,7 @@ async fn extractor_get_links_with_absolute_url_that_differs_from_target_domain()
     let extractor = Extractor {
         links_regex: Regex::new(LINKFINDER_REGEX).unwrap(),
         robots_regex: Regex::new(ROBOTS_TXT_REGEX).unwrap(),
+        url_regex: Regex::new(URL_CHARS_REGEX).unwrap(),
         response: Some(&ferox_response),
         url: String::new(),
         target: ExtractionTarget::ResponseBody,
@@ -301,6 +302,7 @@ async fn request_robots_txt_without_proxy() -> Result<()> {
     let extractor = Extractor {
         links_regex: Regex::new(LINKFINDER_REGEX).unwrap(),
         robots_regex: Regex::new(ROBOTS_TXT_REGEX).unwrap(),
+        url_regex: Regex::new(URL_CHARS_REGEX).unwrap(),
         response: None,
         url: srv.url("/api/users/stuff/things"),
         target: ExtractionTarget::RobotsTxt,