Package: bypass-url-parser
Bypass Url ParserTool that testsMANYurl bypasses to reach a40X protected page.If you wonder why this code isnothing but a dirty curl wrapper, here's why:Most of the python requests do url/path/parameter encoding/decoding, and I hate this.If I submit raw chars, I want raw chars to be sent.If I send a weird path, I want it weird, not normalized.This issurprisingly hardto achieve in python without loosing all of the lib goodies like parsing, ssl/tls encapsulation and so on.So, be like me, usecurl as a backend, it's gonna be just fine.Also, this tool can be used as a library, seelib_sample_usage.pyInstallationWe recommend usingpipxto install this tool:pipxinstallbypass-url-parser# or for the latest dev versionpipxinstallgit+https://github.com/laluka/bypass-url-parserAlternatively, you can usepip:pipinstallbypass-url-parserUsageBypass Url Parser, made with love by @TheLaluka
A tool that tests MANY url bypasses to reach a 40X protected page.

Usage:
    bypass-url-parser (-u <URL> | -R <file>) [-m <mode>] [-o <outdir>] [-S <level>] [ (-H <header>)...] [-r <num>]
                           [-s <ip>] [--spoofip-replace] [-p <port>] [--spoofport-replace] [--request-tls]
                           [--dump-payloads] [-t <threads>] [-T <timeout>] [-x <proxy_url>] [-v | -d | -dd]

Program options:
    -u, --url <URL>           URL (path is optional) to run bypasses against
    -R, --request <file>      Load HTTP raw request from a file
    -H, --header <header>     Header(s) to use, format: "Cookie: can_i_haz=fire"
    -m, --mode <mode>         Bypass modes. See 'Bypasser.BYPASS_MODES' in code [Default: all]
    -o, --outdir <outdir>     Output directory for results
    -x, --proxy <proxy_url>   Set a proxy in the format http://proxy_ip:port.
    -S, --save-level <level>  Save results level. From 0 (DISABLE) to 3 (FULL) [Default: 1]
    -s, --spoofip <ip>        IP(s) to inject in ip-specific headers
    -p, --spoofport <port>    Port(s) to inject in port-specific headers
    -r, --retry <num>         Retry attempts of failed requests. Set 0 to disable all retry tentatives [Default: 3]
    -t, --threads <threads>   Scan with N parallel threads [Default: 1]
    -T, --timeout <timeout>   Request times out after N seconds [Default: 5]

General options:
    -h, --help                Show help, you are here :)
    -v, --verbose             Verbose output
    -d, --debug               Show more details like curl commands generated by this tool
    -dd, --debug              Print Debug level 2 (with all classes debug_class output)
    -V, --version             Show version info

Misc options:
    --spoofip-replace         Disable list of default internal IPs in 'http_headers_ip' bypass mode
    --spoofport-replace       Disable list of default internal ports in 'http_headers_port' bypass mode
    --request-tls             Force usage of TLS/HTTPS for the request load with the '-R, --request' option
    --dump-payloads           Print all payloads (curls) generated by this tool.

Examples:
    bypass-url-parser -u "http://127.0.0.1/juicy_403_endpoint/" -s 8.8.8.8 -d
    bypass-url-parser -u /path/urls -t 30 -T 5 -H "Cookie: me_iz=admin" -H "User-agent: test"
    bypass-url-parser -R /path/request_file --request-tls -m "mid_paths, end_paths"Expected resultbypass-url-parser-uhttp://127.0.0.1:8000/foo/bar2022-08-0914:52:40lalu-persobup[361559]WARNINGTryingtobypass'http://127.0.0.1:8000/foo/bar'url(3213payloads)...2022-08-0914:52:40lalu-persobup[361559]INFODoing:50/3213[...]2022-08-0914:52:54lalu-persobup[361559]INFODoing:3200/32132022-08-0914:52:54lalu-persobup[361559]INFORetry(1/3)the'16'failedcurlcommandswith10threadsand10stimeout2022-08-0914:52:54lalu-persobup[361559]INFORetry(2/3)the'16'failedcurlcommandswith5threadsand20stimeout2022-08-0914:52:54lalu-persobup[361559]INFORetry(3/3)the'16'failedcurlcommandswith1threadsand30stimeout2022-08-0914:52:55lalu-persobup[361559]INFO[#####] [bypass_method] [payload] => [status_code] [content_type] [content_length] [lines_count] [word_counts] [title] [server] [redirect_url][GROUP(1587)][original_request][http://127.0.0.1:8000/foo/bar]=>[404][text/html][469][14][95][Errorresponse][SimpleHTTP/0.6Python/3.8.10][][GROUP(10)][http_methods][-XCONNECThttp://127.0.0.1:8000/foo/bar]=>[501][text/html][500][14][96][Errorresponse][SimpleHTTP/0.6Python/3.8.10][][SINGLE][mid_paths][http://127.0.0.1:8000/???foo/bar]=>[200][text/html][913][26][27][Directorylistingfor/???foo/bar][SimpleHTTP/0.6Python/3.8.10][][SINGLE][mid_paths][http://127.0.0.1:8000//???foo/bar]=>[301][][][0][0][][SimpleHTTP/0.6Python/3.8.10][/???foo/bar][SINGLE][mid_paths][http://127.0.0.1:8000/??foo/bar]=>[200][text/html][911][26][27][Directorylistingfor/??foo/bar][SimpleHTTP/0.6Python/3.8.10][][SINGLE][mid_paths][http://127.0.0.1:8000//??foo/bar]=>[301][][][0][0][][SimpleHTTP/0.6Python/3.8.10][/??foo/bar][SINGLE][mid_paths][http://127.0.0.1:8000/?foo/bar]=>[200][text/html][909][26][27][Directorylistingfor/?foo/bar][SimpleHTTP/0.6Python/3.8.10][][SINGLE][mid_paths][http://127.0.0.1:8000//?foo/bar]=>[301][][][0][0][][SimpleHTTP/0.6Python/3.8.10][/?foo/bar][SINGLE][mid_paths][http://127.0.0.1:8000///?anythingfoo/bar]=>[200][text/html][929][26][27][Directorylistingfor///?anythingfoo/bar][SimpleHTTP/0.6Python/3.8.10][][SINGLE][mid_paths][http://127.0.0.1:8000////?anythingfoo/bar]=>[200][text/html][931][26][27][Directorylistingfor////?anythingfoo/bar][SimpleHTTP/0.6Python/3.8.10][][GROUP(2)][mid_paths][http://127.0.0.1:8000/#?foo/bar]=>[200][text/html][893][26][27][Directorylistingfor/][SimpleHTTP/0.6Python/3.8.10][][GROUP(2)][mid_paths][http://127.0.0.1:8000//#?foo/bar]=>[301][][][0][0][][SimpleHTTP/0.6Python/3.8.10][/]SetupLINUX# Depssudoaptinstall-ybatcurlvirtualenvpython3# Toolvirtualenv-ppython3.py3source.py3/bin/activate
pipinstall.
bypass-url-parser-uhttp://thinkloveshare.com/juicy_403_endpoint/
cat/tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json|jq-r'.results[].request_curl_cmd'cat/tmp/tmpRANDOM-bypass-url-parser/triaged-bypass.json|jq-r'.results[].response_data'DOCKERdockerbuild-tthelaluka/bypass-url-parser:latest.
dockerrun--rm-it--net=host-v"$PWD:/host"thelaluka/bypass-url-parser:latest-uhttp://127.0.0.1:8000/dummy
dockerrun--rm-it--net=host-v"$PWD:/host"thelaluka/bypass-url-parser:latest-u/host/urls.lst# TODO later, tag & push releases to dockerhub :)# -t thelaluka/bypass-url-parser:0.2.0# docker push thelaluka/bypass-url-parser:latestMore about supported argumentsArguments parsingBypass_url_parser allows to define some arguments in many ways:-m, --mode,-s, --spoofipand-p, --spoofportarguments can be a filename, a string, a comma-separated string list or a list (whenBypasseris used as a library);-u, --urlargument can be a filename, a string or a list (whenBypasseris used as a library);stdin(with-) is supported for all these arguments.For example, if you want to define several target urls (-u, --url), all the following commands produce the same result:bypass-url-parser-uhttp://thinkloveshare.com/test
bypass-url-parser-u/path/urls
cat/path/urls|bypass-url-parser-u-echo'http://thinkloveshare.com/test'|bypass-url-parser-u-Target definitionA target must be defined for the tool to work. 2 options:-u, --url: URL(s), in GET-R, --request: Request file. The protocol can't be guessed from file, sohttpby default orhttpsif--request-tlsoption is present.Bypass modeIf-m, --modeis specified, you can select the desired bypass mode to run a specific test (or tests) and reduce the number of requests sent by the tool.For now, the following bypass mode(s) are supported:all, mid_paths, end_paths, case_substitution, char_encode, http_methods, http_versions, http_headers_method, http_headers_scheme, http_headers_ip, http_headers_port, http_headers_urlExample:bypass-url-parser-u/path/urls-m"case_substitution, char_encode, http_headers_scheme"Spoofip / SpoofportIn order to customize the ip addresses and ports used in bypass attempts, the tool supports the following options:With-s, --spoofipyou can set some IP(s) to inject intoip-specificheaders (X-Forwarded-For,X-Real-Ip, etc.)With-p, --spoofportyou can set some ports to inject intoport-specificheaders (X-Forwarded-Port)By default, these custom entries are added to the internal IP/port lists. If you want to use only your IP(s)/port(s), you can use--spoofip-replaceand/or--spoofport-replacearguments.Example:bypass-url-parser-u/path/urls-s/path/custom_ip--spoofip-replace
bypass-url-parser-u/path/urls-p"3000, 9443, 10443"Results savingBy default, if target url is unique, the tool saves a copy of the results in/tmp/tmpXXX-bypass-url-parser/directory.Notes:If multiple target urls are passed to-u, results are prefixed with the url as directory (/tmp/tmpXXX-bypass-url-parser/http-target-com-8080-api-users/).There are two arguments to customize this behavior:-o, --outdirto set a custom output directory-S, --save-levelto choose a saving levelThe saving levels are:0(NONE): Disable output saving and output directory creation;1(MINIMAL): Only save the program log file which contains the results:triaged-bypass.log;2(PERTINENT): Save the program log filetriaged-bypass.logandpertinent (results)curl responses intriaged-bypass.jsonfile and separate html files (Default);3(FULL): Save the program log filetriaged-bypass.logandallcurl responses intriaged-bypass.jsonfile and separate html files.Examplebypass-url-parser-S0bypass-url-parser-S1-o/tmp/bypass-res
bypass-url-parser-S2-o/tmp/bypass-res2-H"User-Agent: curl 7.74.0"-uhttp://thinkloveshare.com/juicy_403_endpoint/
tree/tmp/bypass-res2/
├──bypass-2469eecf6c38b5817d2248e911ad4382.html
├──bypass-6f7cce7caf0a0a4b440859fa189d496d.html
├──bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html
├──bypass-93079abffe63d34f79ac4a511cd6b5e6.html
├──bypass-945822230d58d1ad4680d5dfbc470ecb.html
├──bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html
├──triaged-bypass.json
└──triaged-bypass.log0directories,8filesResults exportStarting fromMINIMALlevel, the results displayed by the program are saved in thetriaged-bypass.logfile.JSON exportWithPERTINENTandFULLsaving levels, the program additionally exports all results in thetriaged-bypass.jsonfile:{"url":"http://thinkloveshare.com/juicy_403_endpoint/","bypass_modes":"all","results":[{"request_curl_cmd":"/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -H 'X-BlueCoat-Via: localhos[...SNIP...]","request_curl_payload":"-H X-BlueCoat-Via: localhost http://thinkloveshare.com/juicy_403_endpoint/","response_headers":"HTTP/1.1 301 Moved Permanently\nConnection: keep-alive\nContent-Length: 162\nServer: GitHub.c[...SNIP...]","response_data":"<html>\n<head><title>301 Moved Permanently</title></head>\n<body>\n<center><h1>301 Moved Permane[...SNIP...]","response_status_code":301,"response_content_type":"text/html","response_content_length":162,"response_lines_count":7,"response_words_count":4,"response_title":"301 Moved Permanently","response_server_type":"GitHub.com","response_redirect_url":"https://thinkloveshare.com/juicy_403_endpoint/","response_html_filename":"bypass-e6118c315eea0e5b2ebc4fcafe0559c0.html"},{"request_curl_cmd":"/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is -X PROPFIND http://thinklove[...SNIP...]","request_curl_payload":"-X PROPFIND http://thinkloveshare.com/juicy_403_endpoint/","response_headers":"HTTP/1.1 405 Method Not Allowed\nConnection: close\nContent-Length: 131\nServer: Varnish\nRet[...SNIP...]","response_data":"<html>\n<head><title>405 Not Allowed</title></head>\n<body bgcolor=\"white\">\n<center><h1>405 N[...SNIP...]","response_status_code":405,"response_content_type":"","response_content_length":131,"response_lines_count":5,"response_words_count":5,"response_title":"405 Not Allowed","response_server_type":"Varnish","response_redirect_url":"","response_html_filename":"bypass-945822230d58d1ad4680d5dfbc470ecb.html"},{"...":"[...SNIP...]"},{"request_curl_cmd":"/usr/bin/curl -sS -kgi -H 'User-Agent: curl 7.74.0' --path-as-is 'http://thinkloveshare.com/j[...SNIP...]","request_curl_payload":"http://thinkloveshare.com/juicy_403_endpoint/\u00b0//","response_headers":"HTTP/1.1 400 Bad request\nConnection: keep-alive\nContent-Length: 90\nCache-Control: no-cache[...SNIP...]","response_data":"<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n","response_status_code":400,"response_content_type":"text/html","response_content_length":90,"response_lines_count":3,"response_words_count":7,"response_title":"","response_server_type":"","response_redirect_url":"","response_html_filename":"bypass-2469eecf6c38b5817d2248e911ad4382.html"}]}Making them easier to handle withjq:$jq-r'.results[] | [.request_curl_payload, .response_status_code, .response_content_type, .response_content_length] | join("|")'/tmp/bypass-res2/triaged-bypass.json

-HX-BlueCoat-Via:localhosthttp://thinkloveshare.com/juicy_403_endpoint/|301|text/html|162-XPROPFINDhttp://thinkloveshare.com/juicy_403_endpoint/|405||131http://thinkloveshare.com/%3b%2f%2e%2e%2f%2e%2e%2f%2fjuicy_403_endpoint/|400|text/html|9121-HHost:8.8.8.8http://thinkloveshare.com/juicy_403_endpoint/|404|text/html|9115-XCONNECThttp://thinkloveshare.com/juicy_403_endpoint/|400|text/plain|15http://thinkloveshare.com/juicy_403_endpoint/°//|400|text/html|90HTML filesWithPERTINENTandFULLsaving levels, curl commands and full HTTP responses are also stored in pseudo.htmlfiles:$echo/tmp/bypass-res2/*.html|xargsbatcat
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│File:/tmp/bypass-res2/bypass-2469eecf6c38b5817d2248e911ad4382.html
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────1│/usr/bin/curl-sS-kgi-H'User-Agent: curl 7.74.0'--path-as-is'http://thinkloveshare.com/juicy_403_endpoint/°//'2│3│HTTP/1.1400Badrequest4│Connection:keep-alive5│Content-Length:906│Cache-Control:no-cache7│Content-Type:text/html8│Accept-Ranges:bytes9│Date:Tue,25Apr202323:51:38GMT10│Via:1.1varnish11│X-Served-By:cache-par-lfpg1960025-PAR12│X-Cache:MISS13│X-Cache-Hits:014│X-Timer:S1682466698.230664,VS0,VE1015│Vary:Accept-Encoding16│X-Fastly-Request-ID:b6bbb82302420db4f101a316dca39cc283a4fd4417│18│<html><body><h1>400Badrequest</h1>19│Yourbrowsersentaninvalidrequest.20│</body></html>
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│File:/tmp/bypass-res2/bypass-6f7cce7caf0a0a4b440859fa189d496d.html
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────1│/usr/bin/curl-sS-kgi-H'User-Agent: curl 7.74.0'--path-as-is-XCONNECThttp://thinkloveshare.com/juicy_403_endpoint/2│3│HTTP/1.1400BadRequest4│Connection:close5│Content-Length:156│content-type:text/plain;charset=utf-87│x-served-by:cache-par-lfpg19600838│9│invalidrequest
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│File:/tmp/bypass-res2/bypass-80f4ab5d32b4e74c20630c7e67f2e42f.html
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────1│/usr/bin/curl-sS-kgi-H'User-Agent: curl 7.74.0'--path-as-ishttp://thinkloveshare.com/%3b%2f%2e%2e%2f%2e%2e%2f%2fjuicy_403_endpoint/2│3│HTTP/1.1400BadRequest4│Connection:keep-alive5│Content-Length:91216│Server:GitHub.com7│Content-Type:text/html;charset=utf-88│ETag:"64417b9f-23a1"9│Content-Security-Policy:default-src'none';style-src'unsafe-inline';img-srcdata:;connect-src'self'10│X-GitHub-Request-Id:598E:F13E:26EE27D:284F5D2:6448673111│Accept-Ranges:bytes12│Date:Tue,25Apr202323:50:11GMT13│Via:1.1varnish14│X-Served-By:cache-par-lfpg1960046-PAR15│X-Cache:MISS16│X-Cache-Hits:017│X-Timer:S1682466611.396077,VS0,VE10118│Vary:Accept-Encoding19│X-Fastly-Request-ID:30bd6af5892c40da130ee49bbeacd147a1a6b3c320│21│<!DOCTYPEhtml>22│<html>23│<head>24│<metahttp-equiv="Content-type"content="text/html; charset=utf-8">25│<metahttp-equiv="Content-Security-Policy"content="default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self'">26│<title>Badrequest&middot;GitHubPages</title>27│<styletype="text/css"media="screen">[...SNIP...]Non-Regression tests & Code Cleanup# Code Cleanuppre-commitrun--all-files# Ensure no regression is pushedbypass-url-parser-S0-v-uhttp://127.0.0.1:8000/foo/bar--dump-payloads>"tests-history/bup-payloads-$(date+'%Y-%m-%d').lst"# Compare /tmp/bup-payloads-YYYY-MM-DD.lst and the latest tests-history/bup-payloads-YYYY-MM-DD.lstgitdiff--no-index$(findtests-history-typef|sort-n|tail-n2)# Push your changesgitadd.
gitcommit-m"My cool feature or bugfix"gittag-avX.Y.Z"$COMMIT_HASH"-m"New release: vX.Y.Z"gitpush--tags# If X or Y is bumped, create new release on githubContributorsInitial release by@TheLalukaHuge refactoring & lib-mode with thanks to@jtop_fapLicenseCopyright (C) 2022 LalukaThis program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.You should have received a copy of the GNU Affero General Public License along with this program. If not, seehttps://www.gnu.org/licenses/.