Package: ctdl
content-downloadercontent-downloadera.k.actdlis a python package withcommand
line utilityanddesktop GUIto download files on any topic in
bulk!Featuresctdl can be used as a command line utility as well as a desktop GUI.ctdl fetches file links related to a search query fromGoogle
Search.Files can be downloaded parallely using multithreading.ctdl is Python 2 as well as Python 3 compatible.InstallationTo install content-downloader, simply,$ pip install ctdlThere seem to be some issues with parallel progress bars in tqdm
which have been resolved in thispull. Until this pull is
merged, please use my patch by running this command:$ pip install-Ugit+https://github.com/nikhilkumarsingh/tqdmDesktop GUI usageTo usectdldesktop GUI, open terminal and run this command:$ ctdl-guiCommand line usage$ ctdl [-h] [-f FILE_TYPE] [-l LIMIT] [-d DIRECTORY] [-p] [-a] [-t]
       [-minfs MIN_FILE_SIZE] [-maxfs MAX_FILE_SIZE] [-nr]
       [query]Optional arguments are:-f FILE_TYPE : set the file type. (can take values like ppt, pdf,
xml, etc.)Default value: pdf-l LIMIT : specify the number of files to download.Default value: 10-d DIRECTORY : specify the directory where files will be stored.Default: A directory with same name as the search query in the current directory.-p : for parallel downloading.-minfs MIN_FILE_SIZE : specify minimum file size to download in
Kilobytes (KB).Default: 0-maxfs MAX_FILE_SIZE : specify maximum file size to download in
Kilobytes (KB).Default: -1 (represents no maximum file size)-nr : prevent download redirects.Default: FalseExamplesTo get list of available filetypes:$ ctdl-aTo get list of potential high threat filetypes:$ ctdl-tTo download pdf files on topic ‘python’:$ ctdl pythonThis is the default behaviour which will download 10
pdf files in a folder named ‘python’ in current directory.To download 3 ppt files on ‘health’:$ ctdl-fppt-l3 healthTo explicitly specify download folder:$ ctdl-d/home/nikhil/Desktop/ml-pdfsmachine-learningTo download files parallely:$ ctdl-fpdf-ppythonTo search for and download in parallel 10 files in PDF format
containing the text “python” and “algorithm”, without allowing any
url redirects, and where the file size is between 10,000 KB (10 MB)
and 100,000KB (100 MB), where KB means Kilobytes, which has an
equivalent value expressed in Megabytes:$ ctdl-fpdf-l10-minfs10000-maxfs100000-nr-p"python algorithm"Usage in Python filesfromctdlimportctdlctdl.download_content(file_type='ppt',limit=5,directory='/home/nikhil/Desktop/ml-pdfs',query='machine learning using python')TODO[X] Prompt user before downloading potentially threatful files[X] Create ctdl GUI[ ] Implement unit testing[ ] Use DuckDuckgo API as an optionWant to contribute?Clone the repository$ git clonehttp://github.com/nikhilkumarsingh/content-downloaderInstall dependencies$ pip install-rrequirements.txtNote:There seem to be some issues with current version of tqdm. If
you do not get expected progress bar behaviour, try this patch:$ pip uninstall tqdm   $ pip installgit+https://github.com/nikhilkumarsingh/tqdmIn ctdl/ctdl.py, remove the.prefix from.downloaderand.utilsfor the following imports, so it changes from:python   from .downloader import download_series, download_parallel   from .utils import FILE_EXTENSIONS, THREAT_EXTENSIONSto:python   from downloader import download_series, download_parallel   from utils import FILE_EXTENSIONS, THREAT_EXTENSIONSRun the python file directlypython ctdl/ctdl.py ___(instead of
withctdl ___)