Package: pyfortified-requests
pyfortified-requestsExtension of Python HTTPrequestswith verbose
logging usingpyfortified-logging, and robust handling for
Downloading files containing JSON, CSV, and XML data formats and Streaming.Important NoteThis Python project is a refactoring ofrequests-mv-integrationfor the purpose of general usage intent.Work In ProgressThe following still needs to be performed for this Python project:Unit-testing: This project will be switching over to using Python native Unit testing frameworkunittest.More concise documentation is required.Travis CIBadgesBadgesInstallpipinstallpyfortified_requestsArchitecturepyfortified-requestsis an extension of thePython package requestsand it is used for handling all HTTP requests including APIs in REST and SOAP,
screen scrapping, and handling response downloads in JSON, XML, and CSV.UsageURL_REST_COUNTRIES=\'http://api.population.io/1.0/countries'frompyfortified_requestsimport(RequestsFortifiedDownload,)request_download=RequestsFortifiedDownload(logger_level=logging.DEBUG)result=\request_download.request(request_method='GET',request_url=URL_REST_COUNTRIES,request_params=None,request_retry=None,request_headers=HEADER_CONTENT_TYPE_APP_JSON,request_label="REST Countries")json_rest_countries=result.json()pprint(json_rest_countries)Example$makerun-example-countries======================================================run-examplespyfortified-requests======================================================rm-fR_tmp/*.json======================================================'Logger file path: ./tmp/log_1526043820.json'['{"asctime": "2018-05-14 10:40:54 -0700", "levelname": "NOTE", "name": "pyfortified_requests", "version": "0.3.3", "message": "debug"}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.3.3", "message": "START"}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Start"}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Start: Details", "Disk:": {"free": "91.54 GB", "percent": 60, "total": "232.62 GB", "path": "/", "used": "140.84 GB"}, "Mem": {"free": "133.81 KB", "cached": "0 B", "total": "16.00 GB", "used": "13.19 GB", "shared": "0 B", "buffers": "0 B"}, "allow_redirects": true, "build_request_curl": true, "cookie_payload": null, "request_auth": null, "request_cert": null, "request_data": null, "request_headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.3.3, Python/3.5.4)"}, "request_json": null, "request_label": "REST Countries", "request_method": "GET", "request_params": null, "request_retry": {"tries": 3, "delay": 10, "timeout": 60}, "request_url": "http://api.population.io/1.0/countries", "stream": false, "timeout": null, "verify": true}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Start", "request_label": "REST Countries", "request_retry_excps": ["ConnectTimeout", "ReadTimeout", "Timeout"], "request_retry_http_status_codes": [500, 502, 503, 504, 429], "timeout": null}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Attempt", "attempts": 1, "delay": 10, "request_label": "REST Countries", "request_url": "http://api.population.io/1.0/countries", "timeout": 60, "tries": 3}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Session: Details", "cookie_payload": {}, "request_label": "REST Countries"}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Details", "request_data": "", "request_headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.3.3, Python/3.5.4)"}, "request_label": "REST Countries", "request_method": "GET", "request_params": {}, "request_url": "http://api.population.io/1.0/countries", "timeout": 60}\n','{"asctime": "2018-05-14 10:40:55 -0700", "levelname": "NOTE", "name": "pyfortified_requests", "version": "0.3.3", "message": "REST Countries: Curl", "request_curl": "curl --verbose -X GET -H \'Content-Type:application/json\'-H\'User-Agent:(pyfortified-requests/0.3.3,Python/3.5.4)\'--connect-timeout60-L\'http://api.population.io/1.0/countries\'", "request_label": "RESTCountries", "request_method": "GET"}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests.support.requests_session_client", "version": "0.3.3", "message": "SessionRequest:Details", "allow_redirects": true, "headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.3.3,Python/3.5.4)"}, "method": "GET", "timeout": 60, "url": "http://api.population.io/1.0/countries", "verify": true}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:Response:Details", "http_status_code": 200, "http_status_desc": "Requestfulfilled,documentfollows", "http_status_type": "Successful", "response_headers": {"Vary": "Accept", "Allow": "OPTIONS,GET", "Cache-Control": "max-age=3600", "Via": "1.17bfcc2251021d9dc94a87ff179d69731.cloudfront.net(CloudFront)", "Expires": "Mon,14May201818:23:07GMT", "Connection": "keep-alive", "X-Amz-Cf-Id": "_Q_sioBJ9zkP0pFZztqnQJHLBXl5DWoAnGnb2HBGxJHDEmPX4jH82g==", "Date": "Mon,14May201817:23:07GMT", "Content-Type": "application/json", "Server": "nginx/1.10.2", "Content-Length": "1572", "X-Cache": "Hitfromcloudfront", "Age": "1039", "Content-Encoding": "gzip"}}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:CookiePayload", "cookie_payload": {}, "request_label": "RESTCountries"}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:TrySendRequest:IsReturnResponse:Checking", "request_url": "http://api.population.io/1.0/countries"}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:TrySendRequest:IsReturnResponse:Valid", "request_url": "http://api.population.io/1.0/countries"}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:Finished", "request_time_msecs": 96}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.3.3", "message": "RESTCountries:Usage", "Disk:": {"free": "91.54GB", "percent": 60, "total": "232.62GB", "path": "/", "used": "140.84GB"}, "Mem": {"free": "133.17KB", "cached": "0B", "total": "16.00GB", "used": "13.19GB", "shared": "0B", "buffers": "0B"}}\n',
    '{"asctime": "2018-05-1410:40:55-0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.3.3", "message": "COMPLETED", "connection": "<requests.adapters.HTTPAdapterobjectat0x104096be0>", "cookies": "<RequestsCookieJar[]>", "elapsed": "0:00:00.079948", "encoding": null, "headers": "{\'Allow\':\'OPTIONS,GET\',\'Cache-Control\':\'max-age=3600\',\'Content-Length\':\'1572\',\'Vary\':\'Accept\',\'Age\':\'1039\',\'Connection\':\'keep-alive\',\'X-Cache\':\'Hitfromcloudfront\',\'Content-Type\':\'application/json\',\'Server\':\'nginx/1.10.2\',\'Via\':\'1.17bfcc2251021d9dc94a87ff179d69731.cloudfront.net(CloudFront)\',\'Expires\':\'Mon,14May201818:23:07GMT\',\'Date\':\'Mon,14May201817:23:07GMT\',\'X-Amz-Cf-Id\':\'_Q_sioBJ9zkP0pFZztqnQJHLBXl5DWoAnGnb2HBGxJHDEmPX4jH82g==\',\'Content-Encoding\':\'gzip\'}", "history": [], "raw": "<urllib3.response.HTTPResponseobjectat0x1040a86d8>", "reason": "OK", "request": "<PreparedRequest[GET]>", "status_code": 200, "url": "http://api.population.io/1.0/countries"}\n',
]
'DEBUG'
{
    'countries': [
        'Afghanistan',
        'AFRICA',
        'Albania',
        'Algeria',
        'Angola',
        'Antigua and Barbuda',
        'Arab Rep of Egypt',
        'Argentina',
        'Armenia',
        'Aruba',
        'ASIA',
        'Australia',
        'Australia/New Zealand',
        'Austria',
        'Azerbaijan',
        'The Bahamas',
        ***
        'Uganda',
        'Ukraine',
        'United Arab Emirates',
        'United Kingdom',
        'United States',
        'US Virgin Islands',
        'Uruguay',
        'Uzbekistan',
        'Vanuatu',
        'Vietnam',
        'Western Africa',
        'Western Asia',
        'Western Europe',
        'Western Sahara',
        'World',
        'Zambia',
        'Zimbabwe',
    ],
}Classesclass RequestsFortified– Base class usingrequestswith retry functionality and verbose logging.class RequestsFortifiedDownload– Download file handling.class RequestsFortifiedUpload– Upload file handling.Requirementspyfortified-requestsmodule is built upon Python 3 and has dependencies upon
several Python modules available withinPython Package Index PyPI.makeinstallorpython3-mpipuninstall--yes--no-input-rrequirements.txtpython3-mpipinstall--upgrade-rrequirements.txtPackagesbeautifulsoup4:https://pypi.python.org/pypi/beautifulsoup4deepdiff:https://pypi.python.org/pypi/deepdiffpyfortified-logging:https://pypi.org/project/pyfortified-loggingpyhttpstatus-utils:https://pypi.python.org/pypi/pyhttpstatus-utilsrequests:https://pypi.python.org/pypi/requestssafe-cast:https://pypi.python.org/pypi/safe-cast