Package: requests-ip-rotator
requests-ip-rotatorA Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.This library will allow the user to bypass IP-based rate-limits for sites and services.X-Forwarded-For headers are automatically randomised and applied unless given. This is because otherwise, AWS will send the client's true IP address in this header.AWS' ApiGateway sends its requests from any available IP - and since the AWS infrastructure is so large, it is almost guarenteed to be different each time. By using ApiGateway as a proxy, we can take advantage of this to send requests from different IPs each time. Please note that these requests can be easily identified and blocked, since they are sent with unique AWS headers (i.e. "X-Amzn-Trace-Id").InstallationThis package is on pypi so you can install via any of the following:pip3 install requests-ip-rotatorpython3 -m pip install requests-ip-rotatorSimple Usageimportrequestsfromrequests_ip_rotatorimportApiGateway# Create gateway object and initialise in AWSgateway=ApiGateway("https://site.com")gateway.start()# Assign gateway to sessionsession=requests.Session()session.mount("https://site.com",gateway)# Send request (IP will be randomised)response=session.get("https://site.com/index.php",params={"theme":"light"})print(response.status_code)# Delete gatewaysgateway.shutdown()Alternate Usage (auto-start and shutdown)importrequestsfromrequests_ip_rotatorimportApiGatewaywithApiGateway("https://site.com")asg:session=requests.Session()session.mount("https://site.com",g)response=session.get("https://site.com/index.php")print(response.status_code)Please remember that if gateways are not shutdown via theshutdown()method when using method #1, you may be charged in future.CostsAPI Gateway is free for the first million requests per region, which means that for most use cases this should be completely free.At the time of writing, AWS charges ~$3 per million requests after the free tier has been exceeded.If your requests involve data stream, AWS would charge data transfer fee at $0.09 per GB.DocumentationAWS AuthenticationIt is recommended to setup authentication via environment variables. Withawscli, you can runaws configureto do this, or alternatively, you can simply set theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYvariables yourself.You can find your access key ID and secret by following theofficial AWS tutorial.Creating ApiGateway objectThe ApiGateway class can be created with the following optional parameters:NameDescriptionRequiredDefaultsiteThe site (without path) requests will be sent to.TrueregionsAn array of AWS regions to setup gateways in.Falseip_rotator.DEFAULT_REGIONSaccess_key_idAWS Access Key ID (will override env variables).FalseRelies on env variables.access_key_secretAWS Access Key Secret (will override env variables).FalseRelies on env variables.verboseInclude status and error messages.FalseTruefromip_rotatorimportApiGateway,EXTRA_REGIONS,ALL_REGIONS# Gateway to outbound HTTP IP and port for only two regionsgateway_1=ApiGateway("http://1.1.1.1:8080",regions=["eu-west-1","eu-west-2"])# Gateway to HTTPS google for the extra regions pack, with specified access key pairgateway_2=ApiGateway("https://www.google.com",regions=EXTRA_REGIONS,access_key_id="ID",access_key_secret="SECRET")Starting API gatewayAn ApiGateway object must then be started using thestartmethod.By default, if an ApiGateway already exists for the site, it will use the existing endpoint instead of creating a new one.This does not require any parameters, but accepts the following:NameDescriptionRequiredDefaultendpointsArray of pre-existing endpoints (i.e. from previous session).FalseforceCreate a new set of endpoints, even if some already exist.FalseFalserequire_manual_shutdownBool specifying whether Apigateways should persistshutdown()callsFalseFalse# Starts new ApiGateway instances for site, or locates existing endpoints if they already exist.gateway_1.start()# Starts new ApiGateway instances even if some already exist.gateway_2.start(force=True)Sending requestsRequests are sent by attaching the ApiGateway object to a requests Session object.The site given inmountmust match the site passed in theApiGatewayconstructor.importrequests# Posts a request to the site created in gateway_1. Will be sent from a random IP.session_1=requests.Session()session_1.mount("http://1.1.1.1:8080",gateway_1)session_1.post("http://1.1.1.1:8080/update.php",headers={"Hello":"World"})# Send 127.0.0.1 as X-Forwarded-For header in outbound request (otherwise X-Forwarded-For is randomised).session_1.post("http://1.1.1.1:8080/update.php",headers={"X-Forwarded-For","127.0.0.1"})# Execute Google search query from random IPsession_2=requests.Session()session_2.mount("https://www.google.com",gateway_2)session_2.get("https://www.google.com/search?q=test")Closing ApiGateway ResourcesIt's important to shutdown the ApiGateway resources once you have finished with them, to prevent dangling public endpoints that can cause excess charges to your account.This is done through theshutdownmethod of the ApiGateway object. It will close all resources for the regions specified in the ApiGateway object constructor.# This will shutdown all gateway proxies for "http://1.1.1.1:8080" in "eu-west-1" & "eu-west-2"gateway_1.shutdown()# This will shutdown all gatewy proxies for "https://www.google.com" for all regions in ip_rotator.EXTRA_REGIONSgateway_2.shutdown()Alternatively, you can selectively shutdown specific endpoints, if needed. To do this, simply pass in an array of endpoints to the shutdown() method, i.e:# This will force start a new gateway (i.e. create new endpoints even if some exist on the region already), and then delete the first 3 of them only.gateway_3=ApiGateway("http://1.1.1.1:8082",regions=ALL_REGIONS)endpoints=gateway_3.start(force=True)gateway_3.shutdown(endpoints[:3])Please bear in mind that any gateways started with therequire_manual_shutdownparameter set toTruewill not be deleted via theshutdownmethod, and must be deleted
manually through either the AWS CLI or Website.CreditThe core gateway creation and organisation code was adapter from RhinoSecurityLabs'IPRotate Burp Extension.The X-My-X-Forwarded-For header forwarding concept was originally conceptualised byustayreadyin hisfireproxproxy.