Package: http-request-randomizer
Vietnamese versionA convenient way to implement HTTP requests is using Pythons’requestslibrary. One of requests’ most popular features is simple
proxying support. HTTP as a protocol has very well-defined semantics for
dealing with proxies, and this contributed to the widespread deployment
of HTTP proxiesProxying is very useful when conducting intensive web crawling/scrapping
or when you just want to hide your identity (anonymization).In this project I am using public proxies to randomise http requests
over a number of IP addresses and using a variety of known user agent
headers these requests look to have been produced by different
applications and operating systems.ProxiesProxies provide a way to use server P (the middleman) to contact server
A and then route the response back to you. In more nefarious circles,
it’s a prime way to make your presence unknown and pose as many clients
to a website instead of just one client. Often times websites will block
IPs that make too many requests, and proxies is a way to get around
this. But even for simulating an attack, you should know how it’s done.User AgentSurprisingly, the only thing that tells a server the application
triggered the request (like browser type or from a script) is a header
called a “user agent” which is included in the HTTP request.The source codeThe project code in this repository is crawlingfivedifferent
public proxy websites: *http://proxyfor.eu/geo.php*http://free-proxy-list.net*http://rebro.weebly.com/proxy-list.html*http://www.samair.ru/proxy/time-01.htm*https://www.sslproxies.orgAfter collecting the proxy data and filtering the slowest ones it is
randomly selecting one of them to query the target url. The request
timeout is configured at 30 seconds and if the proxy fails to return a
response it is deleted from the application proxy list. I have to
mention that for each request a different agent header is used. The
different headers are stored in the/data/user_agents.txtfile which
contains around 900 different agents.InstallationIf you wish to use this module as aCLI
tool, install it globally via pip:pip install http-request-randomizerOtherwise, you can clone the repository and use setup tools:python setup.py installDev testingClone repo, install requirements, develop and run tests:pip install -r requirements.txt
tox -e pyDevVerboseHow to useCommand-line interfaceLibrary APICommand-line interfaceAssuming that you havehttp-request-randomizerinstalled, you can
use the commands below:show help message:proxyList   -h, --helpspecify proxy provider(s) (required):-s {proxyforeu,rebro,samair,freeproxy,all}Specify output stream (default: sys.stdout), could also be a file:-o, --outfilespecify provider timeout threshold in seconds:-t, --timeoutspecify proxy bandwidth threshold in KBs:-bw, --bandwidthshow program’s version number:-v, --versionAPITo usehttp-request-randomizeras a library, include it in your
requirements.txt file. Then you can simply generate a proxied request
using a method call:importloggingimporttimefromhttp_request_randomizer.requests.proxy.requestProxyimportRequestProxyif__name__=='__main__':start=time.time()req_proxy=RequestProxy(log_level=logging.ERROR)print("Initialization took:{0}sec".format((time.time()-start)))print("Size:{0}".format(len(req_proxy.get_proxy_list())))print("ALL ={0}".format(list(map(lambdax:x.get_address(),req_proxy.get_proxy_list()))))test_url='http://ipv4.icanhazip.com'whileTrue:start=time.time()request=req_proxy.generate_proxied_request(test_url)print("Proxied Request Took:{0}sec => Status:{1}".format((time.time()-start),request.__str__()))ifrequestisnotNone:print("\tResponse: ip={0}".format(u''.join(request.text).encode('utf-8')))print("Proxy List Size:{0}".format(len(req_proxy.get_proxy_list())))print("-> Going to sleep..")time.sleep(10)Changing log levelsTheRequestProxyconstructor accepts an optional parameter oflog_levelthat can be used to change the level of logging. By
default, this is equal to 0, or NOTSET. The python logging levels are
documentedhere.
You can either use integers or their equivalent constant in the logging
module. (e.g.logging.DEBUG,logging.ERROR, etc)Documentationhttp-request-randomizer
documentationContributingMany thanks to the open-source community forcontributingto this project!Faced an issue?Open an issuehere, and
be as detailed as possible :)Feels like a feature is missing?Feel free to open a ticket! PRs are always welcome!LicenseThis project is licensed under the terms of the MIT license.