Saravanan666/pypi-org huggingface dataset:
Package: downloader-light
Downloader lightLightweight python library that let's you download and process files concurrently.This package was developed to allows serverless deployment.DependenciesInstalled automatically withpiprequestspysftpInstallationUsage examplesInstallationpipinstalldownloader-lightUsage examplesDownload and upload files to AWS S3For this to work, AWS CLI must be configuredfromblackfeed.downloaderimportDownloaderfromblackfeed.adapter.s3importS3Adapterqueue=[{'url':'https://www.example.com/path/to/image.jpg',# Required'destination':'some/key/image.jpg'# S3 key - Required},{'url':'https://www.example.com/path/to/image2.jpg','destination':'some/key/image2.jpg'}]downloader=Downloader(S3Adapter(bucket='bucketname'),multi=True,# If true, uploads files to images to S3 with multithreadingstateless=False# If set to False, it generates and stores md5 hashes of files in a filestate_id='flux_states'# name of the file where hashes will be stored (states.txt) not requiredbulksize=200# Number of concurrent downloads)downloader.process(queue)stats=downloader.get_stats()# Returns a dict with information about the processDownload files with statesLoading states can be useful if you don't want to re-download the same file twice.fromblackfeed.downloaderimportDownloaderfromblackfeed.adapter.s3importS3Adapterqueue=[...]downloader=Downloader(S3Adapter(bucket='bucketname'),multi=True,stateless=False,state_id='filename')# You can add a callback function if needed# This function will be called after each bulk is processeddefcallback(responses):# response: {#    'destination': destination of the file can be local or can be S3 key,#    'url': URL from where the file was downloaded,#    'httpcode': HTTP code returned by the server,#    'status': True|False,#    'content-type': Mime type of the downloaded resource Example: image/jpeg# }# responses: response[]pass# Your logicdownloader.set_callback(callback)downloader.load_states('filename')# This will load states from "filename.txt"downloader.process(queue)stats=downloader.get_stats()# StatisticsElasticDownloaderLet's you to download/retrieve files from FTP, SFTP and HTTP/S servers easily.ExamplesDownloading file from FTPfromblackfeed.elasticdownloaderimportElasticDownloaderuri='ftp://user:password@ftp.server.com/path/to/file.csv'retriever=ElasticDownloader()res=retriever.download(uri,localpath='/tmp/myfile.csv')# localfile is optional# .download() function returns False if there was an error or return the local path of the downloaded file if it was a success.print(res)/tmp/myfile.csvRetrieving binary content of file from FTPfromblackfeed.elasticdownloaderimportElasticDownloaderuri='ftp://user:password@ftp.server.com/path/to/file.csv'retriever=ElasticDownloader()res=retriever.retrieve(uri)# Return type: io.BytesIO | Falsewithopen('/tmp/myfile.csv','wb')asf:f.write(res.getvalue())ElasticDownloadercan handle FTP, SFTP and HTTP URIs automatically.
Use the methoddownloadto download file locally and use theretrievemethod to get the binary content of a file.
Package: downloader-py3
Each Downloader maintains an sqlite3-based disk cache that utilizes zlib
compression. Network requests are only made if the cached version of the
resource has an age larger or equal to the stale_after value provided by the
programmer.Between network requests a throttling interval needs to elapse. This throttling
interval is randomly chosen, but lies within the throttle_bounds defined by the
programmer.HTML resources can be parsed using lxml and in this case an lxml ElementTree is
returned instead of a file object, with the links rewritten to be absolute in
order to facilitate following them. The parsing is done leniently in order to
not fail when invalid HTML is encountered.The programmer can also supply a function that decides whether the server has
banned the client (possibly by examining the returned resource). In this case
an exception will be raised.Downloader’s features make it ideal for writing scrapers, as it can keep its
network footprint small (due to the cache) and irregular (due to the random
throttling interval).To install, simply run:python setup.py installFor documentation, after installing, run:python -m pydoc downloader
Package: downloader-python
Downloader-pythonWhat can it do?You can use it to download images,text and vedios in the internet.Download moudles for your Python.Write python script and run it.Can we make mods for 'downloader-python'?Yes,we can.You can read the file:moudle path/downloader-python/TEXT FOR MOD MAKER/README.TXT,make mods and upload to PyPIAbout AuthorAuthor's e-mailemail:guoxiuchen20170402@163.com
Package: downloaders
Python package to handle the download of multiple types of files.How do I install this package?As usual, just download it using pip:pipinstalldownloadersUsage examplesfromdownloadersimportBaseDownloaderdownloader=BaseDownloader()urls=[...]downloader.download(urls)TroubleshootingMacOS multiprocessing nightmare fuelCupertino has a gift for us: somehow multiprocessing on MacOS in some specific
astral configurations that I have yet to fully understand, it will crash with the
following error:TheprocesshasforkedandyoucannotusethisCoreFoundationfunctionalitysafely.YouMUSTexec().Breakon__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__()todebug.Apparently, this can be easily fixed by changing the way multiprocessing spawns
processes, that is:importplatform,multiprocessingifplatform.system()=="Darwin":multiprocessing.set_start_method('spawn')The aforementioned solution was proposed onthis StackOverflow question.
Package: download-esm
download-esmDownload ESM modules from npm and jsdelivrInstallationInstall this tool usingpip:pip install download-esmWarningThis isalpha software. It works for downloading Observable Plot, but has not been tested against any other packages yet.Your help welcome in testing this further!UsageTo download Observable Plot and all of its dependencies as ECMAScript modules:download-esm @observablehq/plotThis will download around 40.jsfiles to the current directory.To put them in another directory, add that as an argument:download-esm @observablehq/plot ./jsEach file will have anyimportandexportstatements rewritten as relative paths.You can then use the library in your own HTML and JavaScript something like this:<divid="myplot"></div><scripttype="module">import*asPlotfrom"./js/observablehq-plot-0-6-6.js";constplot=Plot.rectY({length:10000},Plot.binX({y:"count"},{x:Math.random})).plot();constdiv=document.querySelector("#myplot");div.append(plot);</script>DevelopmentTo contribute to this tool, first checkout the code. Then create a new virtual environment:cd download-esm
python -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytest
Package: download-giab
download_giabUtility Python package to download Genome-in-a-Bottle (GIAB) data from theirindex files.This requires Python 3.6 or later.To install, run the following:pipinstalldownload_giabIf you're installing on a cluster, this might be more like:pipinstall--userdownload_giabTo use, run something like the following:download_giabhttps://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/AshkenazimTrio/sequence.index.AJtrio_Illumina300X_wgs_07292015.HG002This will download everything in the linked index to the directory the utility is run from.
It can also download from local index files.If you want to download lots of data and not have the program hang up upon session disconnect,
you can usenohupand&:nohupdownload_giabhttps://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/AshkenazimTrio/sequence.index.AJtrio_Illumina300X_wgs_07292015.HG002&If you are downloading paired-end reads and want to concatenate all FASTQ files into two files,
you can use the--cat-pairedflag. This will generate two files per sample:[sample]_1.fastq.gzand[sample]_2.fastq.gz. If a sample ID is not present, the literal textpairedwill be used.This will not work for some tools (e.g.bwa mem) if the FASTQ files in a pair-set are of
different lengths.If instead you want to store the read pairs + a suggested common name, use the--store-paired-namesflag. This will write to a file calledpaired_names.txt.To filter what files are downloaded, the--filterflag can be provided with a case insensitive
string or regular expression (in Python syntax.)
Package: download-gitignore
Download-gitignoreA Python CLI program to download .gitignore files fromGithub's repositoryeasily.How to Downloaddownload-gitignore requiresthat you have installed in your machine:Python >= 3.5Pip3Run the following command toinstall download-gitignore:pip3 install download-gitignoreHow to useDownload .gitignore fileUse the commanddownload-gitignore downloadwith the language you want to download the .gitignore file.Example:download-gitignore download pythonYou can also usedgias an alias fordownload-gitignore:dgi download c++List all available .gitignore filesUse the commanddownload-gitignore listto list all available .gitignore files to download:download-gitignore list
Package: downloadhelper
# Downloadhelper```Downloadhelper is an Apache2 Licensed HTTP Download library,written in Python, for human beings``````python>>> from downloadhelper import download>>> download('http://partner.myworld2015.org/media/chinese/HOWTO/MYWorld_HowTo_OnlineSurvey_Chinese.pdf', '我的世界.pdf')ID:[1/1] Dir: .File: 我的世界.pdf [ 379.82 KB ]24.00KB [  6.32%  ] [Speed:  8.00KB/s Timeleft: 00:00:44](388940, 388940, 0)```## Features--------------------* support: Resume broken downloads## Installation-------------------------```bash$ pip install Downloadhelperor$ pip install git+https://gitcafe.com/Potter/downloadhelper.git```
Package: downloadimages
downloadImagesGet image url from common websiteVersion0.0.1Download from wallhaven.ccInstallpip install "git+https://github.com/diepthuyhan/downloadImages#egg=downloadImages"Documentwaiting...Usagefrom downloadImages import WallhavenImage

whdl = WallhavenImage(limit=None, nsfw=True)
for img in whdl:
    print(img)
Package: download-jenkins-build-log
download-jenkins-build-log utilityThis is a simple tool that enables downloading console output files
from Jenkins jobs. Freestyle, pipeline and matrix jobs are supported.Pre-requisitesPython 3.3 or laterInstallationpip3 install download-jenkins-build-logBasic UsageUsedownload-jenkins-build-log-hto see all the available options.For example, to download the console log of a freestyle projectmy-freestyle-jobwith the build number123into the target directory oflogsfrom the Jenkins
instance athttps://myjenkins.example.com:8080, use the following command line:download-jenkins-build-log--urlhttps://myjenkins.example.com:8080--build123--directorylogsmy-freestyle-jobAuthenticationYou may use the--loginto set the user name and the--tokento set
the password or API token to access Jenkins. Jenkins documentation recommends
creating anAPI tokento use with tools.Alternatively, you can also provide credentials with the following environment
variables.Environment variable nameDescriptionDOWNLOAD_JENKINS_BUILD_LOG_LOGINlogin nameDOWNLOAD_JENKINS_BUILD_LOG_API_TOKENpassword or API tokenReturn CodesThe tool will return0in case of successful execution, and-1in case of an error.
Package: download-latest
Download LatestDownload a file only if the remote file has changed.ExampleURL='https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=en-US'ARCHIVE="firefox.tar.bz2"dl"$URL""$ARCHIVE"if[-f"$ARCHIVE.new"];thenecho"New version detected"tar-xjf"$ARCHIVE"fiInstallpipinstalldownload-latestSee the install guidefor more information.Usageusage: download-latest [ -h | --help ] [OPTIONS] URL [FILE]

Download URL to FILE only if remote file has changed.

positional arguments:
  URL                url to download
  FILE               path to output (deduced if not specified, see below)

options:
  -h, --help         show this help message and exit
  -V, --version      show program's version number and exit
  -n, --dry-run      do not download (default: false)
  -f, --force        do not check for changes (default: false)
  -q, --quiet        suppress output (default: false)
  -v, --verbose      increase output (default: false)
  -c, --color        enable colorized output
  -C, --no-color     disable colorized output
  -p, --progress     enable the progress meter
  -P, --no-progress  disable the progress meter
  --backend BACKEND  how to download (default: auto)

BACKEND can be one of 'auto', 'curl', 'wget', or 'python'. If 'auto' is
selected, 'curl' will be chosen if available, then 'wget', then 'python'.

If the color or progress options are not specified, they are determined
from the TTY.

If FILE is not specified, it will be deduced by the filename part of the
URL. If no filename can be deduce, e.g., https://example.com/, then the
program will exit with an error.

Additional files may be generated:

FILE.new       present when download occured, otherwise absent
FILE.download  in-progress download
Package: download-npo
Download-npo downloads videos from the Dutch npo.nl site. The rest ofthe documentation is in Dutch.Download-npo (voorheen download-gemist) download videos van de NPO sitevan de publieke omroep. In principe zouden alle sites die gebruik makenvan de zogeheten “NPOPlayer” zouden moeten werken, zoals bv. ncrv.nl ofnrc.nl (al zijn deze niet allemaal getest).
Package: download-oscar
Downloading language data from OSCAR automatedFeaturesAdds dodc and dodg as command line toolsdodc: command line variant provided with arguments to download datadodg: gui variant to download data, put arguments into input fieldsUsagedodcTo get help with the command line tool usedodc -hfrom a shell.The command line tool needs to be supplied with multiple arguments:user: The user used to login to the site providing OSCAR.password: The password used to login to the site providing OSCAR.base_url: The url where the language files are hosted.e.g.https://oscar-public.huma-num.fr/shuff-dedup/sv/orhttps://oscar-prive.huma-num.fr/2109/packaged/sv/out: The folder where files should be downloaded to.chunk_size(optional): Defaults to 4096. The size of the chunks files are downloaded in.dodgThe gui tool internally calls the command line tooldodc.
Instead of providing arguments to the command line you can enter these into input fields directly and they will be passed downward to the command line tool.InstallationSimple Installationpip install download-oscarwill install the requirements and the tool with one command.Installing from sourceRequirementsRequiresPythonin version 3.RequiresRequestsRequireshtml5libRequiresBeautifulSoupRequiresPySimpleGUIRequirestqdmBuildinginstallPythongit clone https://github.com/xamm/download_oscar.gitcd download_oscar(optional) create a virtual enironmentpip install -r requirements.txtpip install -e .will install the tool in development mode.Release a new versionAll pushed git commits and pull requests on themainbranch trigger an automatic build and packaging for pypicommits without a tag only trigger packaging forTestPyPicommits with a tag will also push toPyPiA new version number must be specified insetup.pyin order for publishing to workpublishing is trigerred on creation of atagon themainbranche.g.git tag -a v0.0.1 -m 'Release 0.1' andgit push origin v0.0.1`easiest procedure:work on your codeadd & commit changespush changescreate tagpush tagLicence
Package: downloadplus
A download manager that takes urls, json files as well as Notion as an input and can download files from an URL as well as via torrent.Want to support the development and stay updated?InstallationClone this repo.UsagefromdownloadplusimportDownloaderDownload from an urlDownloader(input_url='https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Brick_wall_close-up_view.jpg/440px-Brick_wall_close-up_view.jpg').process()Download from urls from a json fileMake sure the json file is formated like in the following example:[{"name":"bricks","target_subfolder":"InterestingStuff","progress":null,"url":"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Brick_wall_close-up_view.jpg/440px-Brick_wall_close-up_view.jpg"},{"name":"Big Buck Bunny","target_subfolder":null,"progress":null,"url":"magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c&dn=Big+Buck+Bunny&tr=udp%3A%2F%2Fexplodie.org%3A6969&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.empire-js.us%3A1337&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=wss%3A%2F%2Ftracker.btorrent.xyz&tr=wss%3A%2F%2Ftracker.fastcast.nz&tr=wss%3A%2F%2Ftracker.openwebtorrent.com&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2F&xs=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fbig-buck-bunny.torrent"}]then process it...Downloader(input_json_path='example.json').process()Download from urls from a Notion databaseMake sure you have a page with a "full page database" and the following fields:nameprogressurltarget_subfolderThen get your Notiontoken_v2from the cookies in your webbrowser and process the Notion page...Downloader(notion_token='{{token_v2}}',input_notion_url='{{notion_page_url}}').process()
Package: downloads
DownloadsEasier HTTP downloads in Python 3.5+FeaturesEasier to remember thanurllibFiles are not written unless download finishesProgress bar!Installationpip install downloads==1.0.0Usagefromdownloadsimportdownloaddownload("http://i.imgur.com/ij2h06p.png")# output path is automatically determined and returned# but you can specify it manually if that"s your thing:download("http://i.imgur.com/i5pJRxX.jpg",out_path="cheezburger.jpg")# or, if you want to be fancy:download("https://www.gutenberg.org/files/2600/2600-0.txt",progress=True)That's it!
Package: downloads3key
DescriptionUtility to download all version of s3 file.Usagepipinstalldownloads3key
downloads3key-b"{{s3  bucket  name}}"-k"{{s3  key}}"Parameters:-b : S3 bucket name.
-k : S3 key.Sampledownloads3key-bdev-static.dlgamer.com-kassets/117/69/packshots/super_street_fighter_4_arcade.gif
Package: download-sorter
DownloadSorterDownloadSorter is a Python module that organizes the files in your Downloads folder based on their file extensions. The module moves files into predefined folders according to a configuration file, making it easier to find and manage your downloaded files.FeaturesOrganizes files directly in the Downloads folder (non-recursive)Customizable folder organization using a JSON configuration fileAutomatically appends Unix timestamps to filenames to avoid conflictsCross-platform compatibilityGetting StartedPrerequisitesPython 3.xJSON configuration file (default:sort.json)InstallationClone the repository or download thesort_downloads.pyfile.Create a JSON configuration file, if not using the providedsort.json.ConfigurationThesort.jsonfile defines the folder organization for different file extensions. The structure of the JSON file is as follows:{"DestinationFolder1":["ext1","ext2"],"DestinationFolder2":["ext3","ext4"]}For example, to organize music into the "Music" folder and image files into the "Pictures" folder, thesort.jsonfile should look like this:{"Music":[".mp3",".wma",".ogg",".wav"],"Pictures":[".jpg",".jpeg",".png",".gif",".webm",".bmp"]}UsageTo use the DownloadSorter, simply run thesort_downloads.pyscript:pythonsort_downloads.pyBy default, the script will organize files in the~/Downloadsdirectory if you're using Windows, or~/downloadsfor non-Windows users. You can modify thedownloads_pathvariable in the script to point to a different directory if needed, or pass the directory path as a command line argument when runningsort_downloads.py.CustomizationIf you want to customize DownloadSorter, you can modify thesort_downloads.pyscript:Change theconfig_fileparameter in theDownloadSorterconstructor to use a different JSON configuration file.Update thedownloads_pathvariable in the__main__block to use a different directory for sorting files, or pass the directory path as a command line argument when runningsort_downloads.py.ContributingContributions are welcome! If you have any ideas, feature requests, or bug reports, feel free to submit an issue or create a pull request.LicenseThis project is licensed under the MIT License. See theLICENSEfile for details.
Package: download-station-api
Download Station APISmall python wrapper for interacting with the Synology Download Station.Free software: MIT licenseDocumentation:https://download-station-api.readthedocs.io.Synology documentation:https://global.download.synology.com/download/Document/Software/DeveloperGuide/Package/DownloadStation/All/enu/Synology_Download_Station_Web_API.pdfFeaturesTODO: Write this readmeCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.1 (2021-06-26)First release & update PyPI.
Package: download-stats
download-statsView your pypi download statscli$ download-stats shopyo$ download-stats --compare flask django fastapi$ download-stats --self # same as `download-stats download-stats`Total downloads$ download-stats --total hooman shopyo download-stats meteomoris newsmoris shopcube honeybot jamstackgeneral>>>fromdownload_statsimportstats>>>stats('shopyo'){"total":"41327","30_days":"1484","7_days":"506","by_version":[["Date","4.5.7","4.5.8","4.6.0","Sum","Total"],["2022-09-01","0","1","3","4","24"],["2022-08-31","2","1","17","20","126"],...["2022-06-04","0","0","0","0","9"],],}recent>>>fromdownload_statsimportrecent>>>recent('shopyo'){"data":{"last_day":23,"last_month":454,"last_week":142},"package":"shopyo","type":"recent_downloads",}version>>>fromdownload_statsimportversion>>>version('shopyo'){"data":[{"category":"3","date":"2022-03-06","downloads":1},...{"category":"null","date":"2022-08-31","downloads":55},],"package":"shopyo","type":"python_major_downloads",}>>>version('shopyo',version=3.9){"data":[{"category":"3.9","date":"2022-03-12","downloads":1},{"category":"3.9","date":"2022-03-18","downloads":15},...{"category":"3.9","date":"2022-08-29","downloads":2},],"package":"shopyo","type":"python_minor_downloads",}system>>>fromdownload_statsimportsystem>>>system('shopyo'){"data":[{"category":"Darwin","date":"2022-03-21","downloads":1},...{"category":"Windows","date":"2022-08-22","downloads":2},],"package":"shopyo","type":"system_downloads",}>>>system('shopyo',name='linux'){"data":[{"category":"Linux","date":"2022-03-08","downloads":1},...{"category":"Linux","date":"2022-08-31","downloads":3},],"package":"shopyo","type":"system_downloads",}
Package: download-tarball
Download TarballAutomates manual Python library installation through thePyPitar.gztarball. If no tarball is found, the program will look for next available.zipfile and install the package. I've used this in the past whenpiphasn't worked out, and I have to revert to manual installation of Python libraries. Works onPython 3.4+.Install with:pip install download-tarballOr:python setup.py installBasic UsageInstalling a single package from the command line:download -p <package name>File UsageInstalling via packages listed on separate lines in a.txtfile.Example File Format:xlrd
black
pyinstallerCommand:download -r <text file>DemoDevelopementWindowsInstalling Development dependenciesRunbuild installRunning tests:Runbuild test, which runs all the unit tests intests.py.All the other targets are for upload package to PyPi.LinuxInstalling Development dependenciesRunmake installMake sure you haveGNU Makeinstalled. You can install it withsudo apt-get install make.Running tests:Runmake test, which runs all the unit tests intests.py.All the other targets are for upload package to PyPi.Running tests:Runbuild test, which runs all the unit tests intests.py.All the other targets are for upload package to PyPi.NoteThis tool currently installs the latest version of your package.This package is currently is platform independent. It should work onWindows,LinuxandMacOSX.If your using aUnixenviorment, you might need to usepython3/pip3instead ofpython/pip.