Package: site2hdd
$pipinstallsite2hddfromsite2hddimportdownload_url_list,get_proxies,download_webpagexlsxfile,pklfile=get_proxies(save_path_proxies_all_filtered='c:\\newfilepath\\myproxiefile\\proxy',#  path doesn't have to exist, it will be created, last# part (proxy) is the name of the file - pkl and xlsx will be added# important: There will be 2 files, in this case: c:\\newfilepath\\myproxiefile\\proxy.pkl and c:\\newfilepath\\myproxiefile\\proxy.xlsxhttp_check_timeout=4,# if proxy can't connect within 4 seconds to wikipedia, it is invalidthreads_httpcheck=50,# threads to check if the http connection is workingthreads_ping=100,# before the http test, there is a ping test to check if the server existssilent=False,# show results when a working server has been foundmax_proxies_to_check=2000,# stops the search at 2000)Downloading lists of free proxy serversChecking if the ip existsChecking if http works and own IP is hiddenurls=[r'''https://pandas.pydata.org/docs/#''',r'''https://pandas.pydata.org/docs/getting_started/index.html''',r'''https://pandas.pydata.org/docs/user_guide/index.html''',r'''https://pandas.pydata.org/docs/reference/index.html''',r'''https://pandas.pydata.org/docs/development/index.html''',r'''https://pandas.pydata.org/docs/whatsnew/index.html''',r'''https://pandas.pydata.org/docs/dev/index.html''',r'''https://pandas.pydata.org/docs/index.html''',r'''https://pandas.pydata.org/pandas-docs/version/1.4/index.html''',r'''https://pandas.pydata.org/pandas-docs/version/1.3/index.html''',r'''https://pandas.pydata.org/pandas-docs/version/1.2/index.html''',r'''https://pandas.pydata.org/pandas-docs/version/1.1/index.html''',r'''https://pandas.pydata.org/pandas-docs/version/1.0/index.html''',r'''https://github.com/pandas-dev/pandas''',r'''https://twitter.com/pandas_dev''',r'''https://pandas.pydata.org/docs/#pandas-documentation''',r'''https://pandas.pydata.org/docs/pandas.zip''',r'''https://pandas.pydata.org/''',r'''https://pypi.org/project/pandas''',r'''https://github.com/pandas-dev/pandas/issues''',r'''https://stackoverflow.com/questions/tagged/pandas''',r'''https://groups.google.com/g/pydata''',r'''https://pandas.pydata.org/docs/#module-pandas''',r'''https://www.python.org/''',r'''https://pandas.pydata.org/docs/getting_started/index.html#getting-started''',r'''https://pandas.pydata.org/docs/user_guide/index.html#user-guide''',r'''https://pandas.pydata.org/docs/reference/index.html#api''',r'''https://pandas.pydata.org/docs/development/index.html#development''',r'''https://pandas.pydata.org/docs/_sources/index.rst.txt''',r'''https://numfocus.org/''',r'''https://www.ovhcloud.com/''',r'''http://sphinx-doc.org/''',]download_url_list(urls,ProxyPickleFile='c:\\newfilepath\\myproxiefile\\proxyn.pkl',# The file you created using the function: get_proxiesSaveFolder='f:\\testdlpandaslinks',# where should the files be savedtry_each_url_n_times=5,# maximum retries for each urlProxyConfidenceLimit=10,# each link will be downloaded twice and then compared. If only one result is positive, it counts as a not successful download. But if     the ProxyConfidenceLimit is higher, then it will be acceptedThreadLimit=50,# downloads at the same timeRequestsTimeout=10,# Timeout for requestsThreadTimeout=12,# Should be a little higher than RequestsTimeoutSleepAfterKillThread=0.1,# Don't put 0.0 here - it will use too much CPUSleepAfterStartThread=0.1,# Don't put 0.0 here - it will use too much CPUIgnoreExceptions=True,)Downloading a url listNever close the app when this is the last message that was printed: "Batch done - writing files to HDD ..."# downloads only links from one domain! All others are ignored!starturls=[r'''https://pydata.org/upcoming-events/''',# if it can't find links on the starting page, pass a list of links from the site.r'''https://pydata.org/past-events/''',r'''https://pydata.org/organize-a-conference/''',r'''https://pydata.org/start-a-meetup/''',r'''https://pydata.org/volunteer/''',r'''https://pydata.org/code-of-conduct/''',r'''https://pydata.org/diversity-inclusion/''',r'''https://pydata.org/wp-content/uploads/2022/03/PyData-2022-Sponsorship-Prospectus-v4-1.pdf''',r'''https://pydata.org/sponsor-pydata/#''',r'''https://pydata.org/faqs/''',r'''https://pydata.org/''',r'''https://pydata.org/about/''',r'''https://pydata.org/sponsor-pydata/''',r'''https://pydata.org/wp-content/uploads/2022/03/PyData-2022-Sponsorship-Prospectus-v4.pdf''',]download_webpage(ProxyPickleFile='c:\\newfilepath\\myproxiefile\\proxyn.pkl',DomainName="pydata.org",DomainLink="https://pydata.org/",SaveFolder=r"f:\pandashomepagetest",ProxyConfidenceLimit=10,UrlsAtOnce=100,ThreadLimit=50,RequestsTimeout=10,ThreadTimeout=12,SleepAfterKillThread=0.1,SleepAfterStartThread=0.1,IgnoreExceptions=True,proxy_http_check_timeout=4,proxy_threads_httpcheck=65,proxy_threads_ping=100,proxy_silent=False,proxy_max_proxies_to_check=1000,starturls=starturls,)Downloading a whole page# Command line also works, but you can't use starturls, and the proxy.pkl has to exist already!# Best usage is to continue a download that hasn't been finished yet.# Existing files won't be downloaded again!importsubprocessimportsyssubprocess.run([sys.executable,r"C:\Users\USERNAME\anaconda3\envs\ENVNAME\Lib\site-packages\site2hdd\__init__.py",r"C:\Users\USERNAME\anaconda3\envs\ENVNAME\pandaspyd.ini",])# This is how ini files should look liker"""[GENERAL]ProxyPickleFile = c:\newfilepath\myproxiefile\proxyn.pklProxyConfidenceLimit = 10UrlsAtOnce = 100; ThreadLimit - 50% of UrlsAtOnce is a good numberThreadLimit = 50RequestsTimeout = 10; ThreadTimeout - Should be a little higher than RequestsTimeoutThreadTimeout = 12; SleepAfterKillThread - Don't put 0.0 hereSleepAfterKillThread = 0.1; SleepAfterStartThread - Don't put 0.0 hereSleepAfterStartThread = 0.1IgnoreExceptions = TrueSaveFolder = f:\pythonsiteDomainName = python.orgDomainLink = https://www.python.org/"""Downloading a whole page using the command line