Package: yt-videos-list
Python Quick StartPython 3.6+ setup (required if not already installed)This package usesf-strings(morehere), and so requires Python 3.6+.If you have an older version of Python, you can download Python 3.9.1 (follow links below) and follow the instructions to set up Python for your machine. If you want to install a different version, visit thePython Downloads pageand select the version you want.macOS 64-bit installerWindows x86-64 executable installerWindows x86 executable installerGzipped source tarball(most useful for Linux)Permissions for first runThis is required to make sure you can download and install the required Selenium binary dependencies.On Windows: make sure you openCommand PromptorPowershell(both work) in "Run as Administrator" modeshortcut:⊞ Win+X+AOn Unix based machines (MacOS, Linux): make sure you have read and write access to/usr/local/bin/if you're not sure, open terminal and runsudo chown $USER /usr/local/bin/Usingvenv(optional)While creating a virtual environmentis not requiredto use this package, creating a virtual environment is useful for avoiding dependency conflicts with other projects. Ifyou are sure you do not need to worry about dependency conflicts with other projects, skip this step.Python hasmanyways to set up and use a virtual environment. The following instructions use thevenvprovided with the python standard library for simplicity. You do not need to use this particular implementation of a virtual environment, but virtual environments are outside of the scope of this project, soyou will need to figure out how to set up and use a different implementation of python virtual environments on your own if you choose a different implementation of a virtual environment, since there are too many different variations to cover here.### CREATING the virtual environment on MacOS/Linux ###

python3 -m venv ytvl-venv
source ytvl-venv/bin/activate
# python3 # enter the python shell inside this virtual environment
deactivate # exit this virtual environment

### USING the virtual environment on MacOS/Linux ###

# if ytvl-venv is in the directory you are currently in:
source ytvl-venv/bin/activate

# if ytvl-venv is NOT in the directory you are currently in:
source /absolute/path/to/ytvl-venv/bin/activate

deactivate # exit this virtual environment### CREATING the virtual environment on Windows (NOT FOR git BASH) ###

python -m venv ytvl-venv
ytvl-venv\Scripts\activate
# python # enter the python shell inside this virtual environment
deactivate # exit this virtual environment

### USING the virtual environment on Windows (NOT FOR git BASH) ###

# if ytvl-venv is in the directory you are currently in:
ytvl-venv\Scripts\activate

# if ytvl-venv is NOT in the directory you are currently in:
## you may need to
##    include the .ps1 extenstion (activate.ps1) in Powershell
## or include the .bat extension (activate.bat) in Command Prompt
\absolute\path\to\ytvl-venv\Scripts\activate

deactivate # exit this virtual environment### CREATING the virtual environment on Windows (FOR git BASH) ###

python -m venv ytvl-venv
source ytvl-venv/Scripts/activate
# python # enter the python shell inside this virtual environment
deactivate # exit this virtual environment

### USING the virtual environment on Windows (FOR git BASH) ###

# if ytvl-venv is in the directory you are currently in:
source ytvl-venv/Scripts/activate

# if ytvl-venv is NOT in the directory you are currently in:
source /absolute/path/to/ytvl-venv/Scripts/activate

deactivate # exit this virtual environmentInstalling the packageAfter you install Python 3.6+ and ensure you have the required permissions as needed and have activated your virtual environment as required (if you decide to use a virtual environment - you do notneedto use a virtual environment, but if you choose to usevenv, follow the instructions above), enter the following in your command line:# if something isn't working properly, try rerunning this# the problem may have been fixed with a newer versionpip3install-Uyt-videos-list# MacOS/Linuxpipinstall-Uyt-videos-list# Windows# if that doesn't work:python3-mpipinstall-Uyt-videos-list# MacOS/Linuxpython-mpipinstall-Uyt-videos-list# WindowsIf you're on Windows: make sure youalwaysopenCommand PromptorPowershell(both work) in "Run as Administrator" mode!shortcut:⊞ Win+X+Athis allowsyt_videos_listto update selenium webdriver binaries to be compatible with newer browser versions as browsers are updated (e.g. your Firefox browser updates from version 77 to version 82)to see the commands being run, see theyt_videos_list/docs/dependencies.jsonfileRunning the package from the python interpreterpython3# MacOS/Linuxpython# Windowsfromyt_videos_listimportListCreatormy_driver='firefox'# SUBSTITUTE DRIVER YOU WANT (options below)lc=ListCreator(driver=my_driver,scroll_pause_time=0.8)lc.create_list_for(url='https://www.youtube.com/user/schafer5')lc.create_list_for(url='https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ',log_silently=True)# Set `log_silently` to `True` to mute program logging to the console.# The program will log the prgram status and any program information# to only the log file for the channel being scraped# (this is useful when scraping multiple channels at once with multi-threading).# By default, the program logs to both the log file for the channel being scraped AND the console.# to name the file using the channel ID instead of the channel name, set file_name='id'# this is useful when scraping multiple channels with the same name:lc.create_list_for(url='https://www.youtube.com/channel/UCb2EYjrzI6WpNAmPZeihhag',file_name='id')lc.create_list_for(url='https://www.youtube.com/channel/UCDzYhlGOvGqsYw8IaTKDT8g',file_name='id')# for more details about this method:help(lc.create_list_for)# see the new files that were just created:importosos.system('ls -lt | head')# MacOS/Linuxos.system('dir /O-D | find "_videos_list"')# Windows# for more information on using the module:help(lc)driveroptions include:'firefox''opera''safari'(MacOS only)'chrome''brave''edge'(Windows only!)increasescroll_pause_timefor laggy internet and decreasescroll_pause_timefor fast internetIf you already scraped a channel and the channel uploaded a new video, simply rerun this program on that channel and this package updates your files to include the newer video(s)!Scraping multiple channels from a file simultaneously with multi-threadingAdd the url to every channel you want to extract information from in atxtfile with every url placed on a new line.example:channels.txt(NOTE this is a relative link, so thismightnot link properly on non-GitHub hosted sites!)Enter the python interpreter:python3     # MacOS/Linux
python      # Windowsfromyt_videos_listimportListCreatorlc=ListCreator(driver='firefox',scroll_pause_time=1.2)lc.create_list_from(path_to_channel_urls_file='channels.txt',number_of_threads=4)# configuring settings:lc.create_list_from(path_to_channel_urls_file='channels.txt',number_of_threads=4,min_sleep=1,max_sleep=5,after_n_channels_pause_for_s=(20,10),log_subthread_status_silently=False,log_subthread_info_silently=False)# defaults (keyword argument form)lc.create_list_from('channels.txt',4,1,5,(20,10),False,False)# defaults (positional argument form)lc.create_list_from('channels.txt',min_sleep=3,max_sleep=10)# modifying only min_sleep and max_sleephelp(lc.create_list_from)# see API method detailsExplicitly downloading all Selenium dependenciesIdeal if you use Selenium for other projects 😎Make sure you already have theyt-videos-listpackage installed (follow directions above for getting set up), then run the following:pip3install-Uyt-videos-list# MacOS/Linux: ensure latest packagepython3# MacOS/Linux: enter python interpreterpipinstall-Uyt-videos-list# Windows:     ensure latest packagepython# Windows:     enter python interpreterfromyt_videos_list.downloadimportselenium_webdriver_dependenciesselenium_webdriver_dependencies.download_all()That's all! 🤓More API informationNOTEthat you can also access all the information below from the Python interpreter by enteringimportyt_videos_listhelp(yt_videos_list)# default options for the ListCreator instanceListCreator(txt=True,csv=True,md=True,file_suffix=True,all_video_data_in_memory=False,video_data_returned=False,video_id_only=False,reverse_chronological=True,headless=False,scroll_pause_time=0.8,driver='firefox',cookie_consent=False,verify_page_bottom_n_times=3,file_buffering=-1,)There are a number of optional arguments you can specify during the instantiation of the ListCreator instance. The preceding arguments are run by default, but in case you want more flexibility, you can specify the:driverargument:Firefox (default)OperaSafari (MacOS only)ChromeBraveEdge (Windows only)driver='firefox'driver='opera'driver='safari'driver='chrome'driver='brave'driver='edge'cookie_consentargument:False(default) - block all cookie options if prompted by YouTube (at consent.youtube.com)True- accept all cookie options if prompted by YouTube (also at consent.youtube.com)cookie_consent=False(default) ORcookie_consent=Truetxt,csv,mdfile type argument:True(default) - create a file for the specified typeFalse- do not create a file for the specified typetxt=True(default) ORtxt=Falsecsv=True(default) ORcsv=Falsemd=True(default) ORmd=Falsefile_suffixargument:True(default) - add a file suffix to the output file nameChannelName_reverse_chronological_videos_list.csvChannelName_chronological_videos_list.csvFalse- do NOT add a file suffix to the output file namethis means if a reverse chronological file and a chronological file is made for the same channel, they will have the same name!ChannelName.csv(reverse chronological output file)ChannelName.csv(chronological output file)
->file_suffix=True(default) ORfile_suffix=Falseall_video_data_in_memoryargument:False(default) - do not scrape the entire pageTrue- scrape the entire page (must ALSO set thevideo_data_returnedattribute toTrueto return this data!)all_video_data_in_memory=False(default) ORall_video_data_in_memory=Truevideo_data_returnedargument:False(default) - do not return video data collected from the current scrape job (return dummy data instead:[[0, '', '', '']])True- return video data collected from the current scrape jobifall_video_data_in_memoryattribute set toFalse, the returned data MIGHT not be the full data, and video numbering MIGHT be incorrectsetall_video_data_in_memoryattribute toTrueto return ALL video data for channel (video number will then also ALWAYS be correct)video_data_returned=False(default) ORvideo_data_returned=Truevideo_id_onlyargument:False(default) - include      the full URL             to video:https://www.youtube.com/watch?v=ElevenCharsTrue- include only the identifier parameter to video:ElevenCharsvideo_id_only=False(default) ORvideo_id_only=Truereverse_chronologicalargument:True(default) - write the files in order from most recent video to the oldest videoFalse- write the files in order from oldest video to the most recent videoreverse_chronological=True(default) ORreverse_chronological=Falseheadlessargument:False(default) - run the driver with an open Selenium instance for viewingTrue- run the driver in "invisible" modeheadless=False(default) ORheadless=Truescroll_pause_timeargument:any float values greater than0(default0.8)The value you provide will be how long the program waits before trying to scroll the videos list page down for the channel you want to scrape. For fast internet connections, you may want to reduce the value, and for slow connections you may want to increase the value.scroll_pause_time=0.8(default)CAUTION: reducing this value too much will result in the program not capturing all the videos, so be careful! Experiment :)verify_page_bottom_n_timesargument:any int values greater than0(defaults to3)NOTE: this argument is only used when CREATING a new file for a new channel, and is unused when UPDATING an existing file for an already scraped channel.The value you provide will be how many times the program needs to verify it acually reached the bottom of the page before accepting it is the bottom of the page, and starting to write the information to the output file(s).For channels that have uploaded THOUSANDS of videos, increase this value to a large number that you think should be sufficient to verify the program reached the bottom of the page.To determine HOW large of a value you should provide, determine the length of time you'd like to wait before being reasonably sure that you reached the bottom of the page and it's not just YouTube's server trying to fetch the response from an old database entry, and divide the time you decided to wait by thescroll_pause_timeargument.For example, if you want to wait 45 seconds and you set thescrioll_pause_timevalue to1.0:
->your_time / scroll_pause_time->45 / 1.0->45-> therefore:verify_page_bottom_n_times=45For channels with only a couple hundred videos (or less), the default value of verify_page_bottom_n_times=3shouldbe sufficient.See commit a68f8f62e5c343cbb0641125e271bb96cc4f0750 for more details.file_bufferingargument:anyintvalues greater than0(default-1, which uses the default OS setting)LEAVE THIS ALONE IF YOU'RE UNSURE!Documentation:https://docs.python.org/3/library/functions.html#openDeep dive:https://stackoverflow.com/questions/3167494/how-often-does-python-flush-to-a-filehttps://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-methodhttps://stackoverflow.com/questions/230751/how-can-i-flush-the-output-of-the-print-functionhttps://en.wikipedia.org/wiki/Data_bufferhttps://stackoverflow.com/questions/1450551/buffered-vs-unbuffered-iohttps://www.quora.com/What-does-flushing-files-or-Stdin-do-in-Pythonhttps://www.quora.com/Whats-the-difference-between-buffered-I-O-and-unbuffered-I-Ohttps://stackoverflow.com/questions/8409050/unix-buffered-vs-unbuffered-i-ohttps://medium.com/@bramblexu/three-ways-to-close-buffer-for-stdout-stdin-stderr-in-python-8be694bd2737https://www.quora.com/In-C-what-does-buffering-I-O-or-buffered-I-O-meanscrapetubeintegrationscrapetubeis a much more efficient backend developer tool that loads the videos uploaded by a channel. This packagealsosupports loading information from playlists and searches, whichyt-videos-listcurrently does not do. Integration withscrapetubewill be available in a futureyt-videos-listrelease!To keep things backwards-compatible and maintainable, thescrapetubeintegration will be accessible through an almost identical,separateinterface as theListCreatorinterface, and the originalListCreatorinterface will continue to be available and continue to receive updates. 🤓Cloning and running locallyTo clone the repository and install the most updated version of the package that may not yet be available on the latest release throughPyPI, clone this repository and run:cd yt_videos_list/python # MacOS/Linux
python3 -m pip install . # MacOS/Linux

cd yt_videos_list\python # Windows
python -m pip install .  # WindowsTo make your own changes to theyt_videos_listpython package and run the changes locally:# make changes to the codebase in the
# ===> /dev <=== directory
python3 minifier.py           # MacOS/Linux
pip3 install .                # MacOS/Linux

python minifier.py            # Windows
pip install .                 # WindowsNOTE: make the changes to the codebase in theyt_videos_list/python/devdirectory!!the code in theyt_videos_list/python/yt-videos-listdirectory is minified withleading indents stipped to the minimum (1 space for each nested scope)whitespace for padding (e.g. extra spaces to align variable assignments) strippedcomments strippedas a result, the code in theyt_videos_list/python/yt-videos-listdirectory is NOT human readable, and theyt_videos_list/python/devdirectory should be used for development instead!theminifier.pymodule performs all the code preprocessing and packages the code fromyt_videos_list/python/devinto the final version seen in theyt_videos_list/python/yt-videos-listdirectoryso runningminifier.pybeforeinstalling the local package withpip install .(Windows) orpip3 install .is essential!Running testsThe tests use the customThreadWithResultsubclass ofthreading.Threadprovided by thesave-thread-resultpackage, so make sure you install that module usingpip3 install -U save-thread-result     # MacOS/Linux
pip  install -U save-thread-result     # Windows

# if that doesn't work:

python3 -m pip install -U save-thread-result     # MacOS/Linux
python  -m pip install -U save-thread-result     # WindowsThen, make sure you're in theyt_videos_list/pythondirectory, then run:tests\run_tests.bat     # Windows
####       Any shell on   MacOS/Linux
bash tests/run_tests.sh # this works
csh  tests/run_tests.sh # this works
dash tests/run_tests.sh # this works
ksh  tests/run_tests.sh # this also works
tcsh tests/run_tests.sh # this works too
zsh  tests/run_tests.sh # this works as well
# you can try other shells and
# they should work too, since
# there's no special syntax in
# the run_tests.sh file