Package: get-free-proxy
#get_free_proxy
get_free_proxy is a tool to get free proxy from websiteinstallpip install get-free-proxyusageget_free_proxy depends ongen_browser_header.create gen_browser_header settingimport gen_browser_header.setting.Setting as gbh_settingimport gen_browser_header.self.SelfEnum as gbh_self_enumcur_gbh_setting = gbh_setting.GbhSetting()cur_gbh_setting.proxy_ip = ['10.11.12.13:8080']cur_gbh_setting.browser_type = {gbh_self_enum.BrowserType.All}cur_gbh_setting.firefox_ver = {'min': 74, 'max': 75}cur_gbh_setting.chrome_type = {gbh_self_enum.ChromeType.Stable}cur_gbh_setting.chrome_max_release_year = 1cur_gbh_setting.os_type = {gbh_self_enum.OsType.Win64}create get_free_proxy settingimport get_free_proxy.self.SelfEnum as gfp_self_enumimport get_free_proxy.setting.Setting as gfp_settingcur_gfp_setting = gfp_setting.GfpSetting()cur_gfp_setting.proxy_type = {gfp_self_enum.ProxyType.HIGH_ANON}cur_gfp_setting.protocol = {gfp_self_enum.ProtocolType.HTTP, gfp_self_enum.ProtocolType.HTTPS}cur_gfp_setting.country = {gfp_self_enum.Country.All}cur_gfp_setting.storage_type = {gfp_self_enum.StorageType.All}cur_gfp_setting.mysql = { 'host': '127.0.0.1', 'port': 3306, 'user': 'root', 'pwd': '1234', 'db_name': 'db_proxy', 'tbl_name': 'tbl_proxy', 'charset': 'utf8mb4'}cur_gfp_setting.redis = { 'host': '127.0.0.1', 'port': 6379, 'db': 0,  # 0~15 'pwd': None }cur_gfp_setting.result_file_path = os.path.join(tempfile.gettempdir(), 'result.json')cur_gfp_setting.valid_time_in_db = 86400cur_gfp_setting.site_max_page_no = 2cur_gfp_setting.site = {gfp_self_enum.SupportedWeb.Xici}start to get free proxymainOp = MainOp(cur_gfp_setting, cur_gbh_setting)首先清空数据库(反正都要全部重新读取网页)mainOp.del_proxy()检测url是否需要使用代理mainOp.check_if_site_need_proxy()从可以直连的网站获得代理tmp_proxies = mainOp.get_proxy_without_proxy()验证代理是否可用first_validate_proxies = mainOp.async_validate_proxies(tmp_proxies, 'https://www.baidu.com')有可用的代理，则使用这些代理，来连接需要代理连接的代理完整；否则，使用固定的cur_gbh_setting.proxy_ipif len(first_validate_proxies) > 0:tmp_proxies = mainOp.get_proxy_with_proxy(proxies=first_validate_proxies)else:tmp_proxies = mainOp.get_proxy_with_proxy(proxies=None)获得结果，再次进行验证，是否可以使用second_validate_proxies = mainOp.async_validate_proxies(tmp_proxies, 'https://www.baidu.com')合并所有可用的代理all_validate_proxies = first_validate_proxies+second_validate_proxiesprint('最终有效代理%s' % all_validate_proxies)保存代理mainOp.save_proxy(proxies=all_validate_proxies)gfp_settingproxy_typetype:set, element is enum=>gfp_self_enum.ProxyTypedefault:{gfp_self_enum.ProxyType.HIGH_ANON}description:  proxy has 3 type: transparent/anonymous/high_anonymous, TRANS/ANON/HIGH_ANON. There is an addition one All,
if set, will be replace by  TRANS+ANON+HIGH_ANONprotocoltype:set, element is enum=>gfp_self_enum.ProtocolTypedefault: {gfp_self_enum.ProtocolType.HTTP, gfp_self_enum.ProtocolType.HTTPS}description:  proxy protocol has 4 type: HTTP, HTTPS, SOCK4, SOCK5. There is an addition one All, is set, will be replace by
HTTP+HTTPS+SOCK4+SOCK5.countrytype:set, element is enum=>gfp_self_enum.Countrydefault: {gfp_self_enum.Country.China}description: some web provide proxy form all countries, the parameter will filter the country. There is an addition one
All, is set, will ignore country.storage_typetype:set, element is enum=>gfp_self_enum.StorageTypedefault: {gfp_self_enum.StorageType.All}description: current support 3 storage type: Mysql/Redis/File. There is an addition one All, is set, will be replace by
Mysql+Redis+Filemysqltype:dictdefault:{'host': '127.0.0.1','port': 3306,'user': 'root','pwd': '1234','db_name': 'db_proxy','tbl_name': 'tbl_proxy','charset': 'utf8mb4'}description: ifstorage_typeinclude Mysql, set this parameter to connect mysql.redistype:dictdefault:{'host': '127.0.0.1','port': 6379,'db': 0,  # 0~15'pwd': None}description: ifstorage_typeinclude Redis, set this parameter to connect redis.result_file_pathtype:stringdefault:os.path.join(tempfile.gettempdir(), 'result.json')description: ifstorage_typeinclude File, all get proxy will be save into the file defined byresult_file_pathvalid_time_in_dbtype:intdefault:86400unit:seconddescription: since all got proxy are free, not sure when these proxy will expire. So set this parameter, it a proxy expire this duration, will not delete/not_choosesite_max_page_notype:intdefault:2description: min:2, max:9. The web site which provide free proxy, the content are pagationed. This parameter determine how many page
will be handled to extract proxy.sitetype:set, enum=>gfp_self_enum.SupportedWebdefault:{gfp_self_enum.SupportedWeb.Xici}description: this parameter determine which site will be used to extract proxy. currently only support 4 site:https://www.xicidaili.com,https://www.kuaidaili.com/free,https://hidemy.name/en/proxy-list/#list,https://proxy-list.org/english.
and if All is set, will be replace by above 4 site.change history0.1.0  use requests-html replace requests0.1.1  match gen_browser 0.1.3: when gen_header, add host base on parameter url
0.1.2  add support for zh-cn in setup.py by add encoding="utf-8"