Package: videolab-youtube-crawler
videolab-youtube-crawlerThis is an integrated tool for crawling YouTube data and decode JSON. See the use cases for examples.This tool is developed by Shuo Niu at Clark University.Pypi page: "https://pypi.org/project/videolab-youtube-crawler/"InstallingTo install,pip install videolab-youtube-crawlerThe crawler needs multiple other packages to function:You may need to install those packages:pip install youtube-transcript-apipip install google-api-python-clientpip install yt-dlppip install deepmultilingualpunctuationAlso install the packages if you don't have them in your ventpip install pytzpip install pandaspip install asynciopip install isodatepip install configparserpip install httplib2UpgradingTo upgradepip install videolab-youtube-crawler --upgradeGo to the project folder, delete config.ini if it is already there.YouTube API KeyGo tohttps://cloud.google.com/, click console, and create a project. Under Credentials, copy the API key.In your project folder, create a "DEVELOPER_KEY.txt" file (must be this file name) and paste your API key.You can use multiple API keys by putting them on different lines in DEVELOPER_KEY.txt.The crawler will use up all quotas of one key and try next one, until all quotas are used up.Example usageCase 1: crawl videos by keywords,import videolab_youtube_crawler as ytcrawler

# create folders for search videos
ytcrawler.init_config()
searcher = ytcrawler.VideoSearcher()
searcher.search("Generative AI",
                start_day=1,
                start_month=1,
                start_year=2023,
                end_day=31,
                end_month=12,
                end_year=2023,
                day_span=30)
searcher.search("Artificial Intelligence",
                start_day=1,
                start_month=1,
                start_year=2023,
                end_day=31,
                end_month=12,
                end_year=2023,
                day_span=30)
                
searcher.merge_to_workfile(destination="DATA/list_video.csv")

# Get all video and channel information
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile=f"DATA/list_video.csv")
video_crawler.json_to_csv()

# Get all channel information from the list generated by merge_to_workfile()
channel_crawler = ytcrawler.ChannelCrawler()
channel_crawler.crawl_channel_of_videos(video_list_workfile=f"DATA/list_video.csv")
channel_crawler.json_to_csv()Case 2: Search a list of channels by search keys, then crawl all videos belonging to those channels.import videolab_youtube_crawler as ytcrawler

# Create folders for search videos
ytcrawler.init_config()
searcher = ytcrawler.ChannelSearcher()
searcher.search("ADHD")
searcher.search("PTSD")
searcher.merge_to_workfile(destination="DATA/channel_list.csv")

# Get channel metadata in a given channel list
channel_crawller = ytcrawler.ChannelCrawler()
channel_crawller.crawl_channel_in_list(channel_list_workfile="DATA/channel_list.csv")
channel_crawller.json_to_csv()

# This step usually has too many videos. Filter the channel before crawling.
channel_crawller.crawl_all_videos_of_channels(channel_list_workfile='DATA/channel_list.csv')

# Merge the searched videos into a workfile
video_searcher = ytcrawler.VideoSearcher()
video_searcher.merge_to_workfile()

# Search all videos in the workfile
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile='DATA/list_video.csv')
video_crawler.json_to_csv()Case 3: You already have a list of channels. You want to crawl all videos of the channels in the list:import pandas as pd

import videolab_youtube_crawler as ytcrawler

ytcrawler.init_config()

# Get channel metadata in a given channel list
channel_crawller = ytcrawler.ChannelCrawler()
channel_crawller.crawl_channel_in_list(channel_list_workfile="speech_disability_channels.csv")
channel_crawller.json_to_csv()

# This step usually has too many videos. Filter the channel before crawling.
channel_crawller.crawl_all_videos_of_channels(channel_list_workfile='speech_disability_channels.csv')

# Merge the searched videos into a workfile
video_searcher = ytcrawler.VideoSearcher()
video_searcher.merge_to_workfile()

# Search all videos in the workfile
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile='DATA/list_video.csv')
video_crawler.json_to_csv()To collect comments, subtitles, and streams of the videos# Get all comments from the list generated by merge_to_workfile()
comment_crawler = ytcrawler.CommentCrawler()
comment_crawler.crawl_comments_of_videos(video_list_workfile=f"DATA/list_video.csv", comment_page=2)
comment_crawler.json_to_csv()

# Get all subtitles from the list generated by merge_to_workfile()
# Since the default YouTube cpations may not contain punctuations, use the split_caption_by_sentences to add punctuations and split 
# the captions by sentences. It will automatically align the timestamps.
subtitle_crawler = ytcrawler.SubtitleCrawler()
subtitle_crawler.crawl_subtitles_in_list(videos_to_collect="DATA/list_video.csv")

# Collect video streams.
video_downloader = ytcrawler.VideoDownloader()
video_downloader.download_videos_in_list(video_list_workfile="DATA/list_video.csv", quality='worst', audio=True)
video_downloader.clean_files()