Package: youte
youte: A command-line tool to retrieve and tidy YouTube metadata and comments from YouTube Data APIChanges from youte 1.3.0Several major changes are made in youte 2.0. To better correspond with YouTube API endpoints and avoid confusion, some commands have been changed.youte hydrateis now broken down toyoute videosandyoute channels.youte get-commentsis nowyoute commentsandyoute replies.youte most-popularis nowyoute chart.youte get-relatedis nowyoute related-to.Furthermore:Resuming search is no longer available. Instead, you can set a limit on the number of search pages returned to avoid exhausting your API quota.All youte commands that retrieve data from YouTube API now won't print results to the shell, but store them in a specified json or jsonl file. This is to avoid clogging up the shell.You can now tidy data into a CSV or a flat JSON array.Big thanks to @Lingomat (Mat Bettinson) for code review and suggestions.Installationpython-mpipinstallyouteYouTube API keyTo get data from YouTube API, you will need a YouTube API key. Follow YouTubeinstructionsto obtain a YouTube API key if you do not already have one.Configure API key (recommended)You can save your API key in the youte config file for reuse. To do so, run:youteconfigadd-keyThe interactive prompt will ask you to input your API key and name it. The name is used to identify the key, and can be anything you choose.The prompt will also ask if you want to set the given key as default.When running queries, if no API key or name is specified,youtewill automatically use the default key.Manually set a key as defaultIf you want to manually set an existing key as a default, run:youteconfigset-default<name-of-existing-key>Note that what is passed to this command is thenameof the API key, not the API key itself. It follows that the API key has to be first added to the config file usingyoute config add-key. If you use a name that has not been added to the config file, an error will be raised.See the list of all keysTo see the list of all keys, run:youteconfiglist-keysThe default key, if there is one, will have an asterisk next to it.Remove a keyTo remove a stored key, run:youteconfigremove<name-of-key>About the config fileyoute's config file is stored in a central place whose exact location depends on the running operating system:Linux/Unix: ~/.config/youte/Mac OS X: ~/Library/Application Support/youte/Windows: C:\Users\<user>\AppData\Roaming\youtesearchSearching can be as simple as:youtesearch<search-terms>--key<API-key>--outfile<name-of-file.json># ORyoutesearch<search-terms>--key<API-key>-o<name-of-file.json>If you have a default key set up usingyoute config, then there is no need to specify an API key using--key.This will return the maximum number of results pages (around 12-13) matching the search terms and store them in a JSON file. Unlike version 1.3, youte 2.0 does not print results to the terminal. Instead,--outfileis now a required option.  and--outfilemust be specified.In the search terms, you can also use the Boolean NOT (-) and OR (|) operators to exclude videos or to find videos that match one of several search terms. If the terms contain spaces, the entire search term value has to be wrapped in quotes.Use the flag--prettyto pretty format the JSON output.youtesearch<search-terms>--key<API-key>--outfile<name-of-file>--prettyLimit pages returnedSearching is very expensive in terms of API usage - a single results page uses up 100 points - 1% of your standard daily quota. Therefore, you can limit the maximum number of result pages returned, so that a search doesn't go on and exhaust your API quota.youtesearch<search-terms>--max-pages5# ORyoutesearch<search-terms>-m5Tidy dataRaw JSONs from YouTube API contain request metadata and nested fields. You can tidy these data into a CSV or a flat JSON using--tidy-to. The default format that youte will tidy raw JSON into will be CSV.youtesearch<search-terms>--tidy-to<file.csv>Specify--format jsonif you want to tidy raw data into an array of flat JSON objects.youtesearch<search-terms>--tidy-to<file-name.json>--formatjson--tidy-tooption is available for allyoutecommands that retrieve data, and works the same way.Advanced searchThere are multiple filters to refine your search. A full list of these are provided below:Options:
  --type TEXT                     Type of resource to search for  [default:
                                  video]
  --order [date|rating|relevance|title|videoCount|viewCount]
                                  Sort results  [default: date]
  --safe-search [none|moderate|strict]
                                  Include or exclude restricted content
                                  [default: none]
  --lang TEXT                     Return results most relevant to a language
                                  (ISO 639-1 two-letter code)
  --region TEXT                   Return videos viewable in the specified
                                  country (ISO 3166-1 alpha-2 code)  [default:
                                  US]
  --video-duration [any|long|medium|short]
                                  Include videos of a certain duration
  --channel-type [any|show]       Restrict search to a particular type of
                                  channel
  --video-type [any|episode|movie]
                                  Search a particular type of videos
  --caption [any|closedCaption|none]
                                  Filter videos based on if they have captions
  --definition, --video-definition [any|high|standard]
                                  Include videos by definition
  --dimension, --video-dimension [any|2d|3d]
                                  Search 2D or 3D videos
  --embeddable, --video-embeddable [any|true]
                                  Search only embeddable videos
  --license, --video-license [any|creativeCommon|youtube]
                                  Include videos with a certain license
  --location FLOAT...             Lat and long coordinates to restrict search
                                  to. --radius must be specified
  --radius TEXT                   Define the geographic area to restrict
                                  search. Must be a number with a unitRestrict by date rangeThe--fromand--tooptions allow you to restrict your search to a specific period. The input values have to be in ISO format (YYYY-MM-DD). Currently, all dates and times in youte are in UTC.Restrict by typeThe--typeoption specifies the type of results returned, which by default is videos. The accepted values arechannel,playlist,video, or a combination of these three. If more than one type is specified, separate each by a comma.youtesearch<search-terms>--limit5--typeplaylist,videoRestrict by language and regionThe--langreturns results most relevant to a language. Not all results will be in the specified language: results in other languages will still be returned if they are highly relevant to the search query term. To specify the language, useISO 639-1 two letter code, except that you should use the valueszh-Hansfor simplified Chinese andzh-Hantfor traditional Chinese.The--regionreturns results viewable in a region. To specify the region, useISO 3166-1 alpha-2. Note that this option doesnotfilter videos uploaded in that region, but rather videos that can beviewedin that region.The--locationand--radiusoptions define a circular geographic area to filter videos that specify, in their metadata, a location within this area. This isnota robust and reliable way to geolocate YouTube videos, and hence should be used with care.--locationtakes in 2 values - the latitude/longitude coordinates that represent the centre of the area--radiusspecifies the maximum distance that the location associated with a video can be from that point for the video to still be included in the search results. It must be a number followed by a unit. Valid units arem,km,ft, andmi. For example,1500m,5km,10000ft, and0.75mi.Both--locationand--radiushave to be specified if they are to be used, otherwise an API error will be thrown.Sort resultsThe--orderoption specifies how results will be sorted. The following values are accepted:date: Resources are sorted in reverse chronological order based on the date they were created (default value).rating: Resources are sorted from highest to lowest rating.relevance: Resources are sorted based on their relevance to the search query.title– Resources are sorted alphabetically by title.videoCount– Channels are sorted in descending order of their number of uploaded videos.viewCount– Resources are sorted from highest to lowest number of views. For live broadcasts, videos are sorted by number of concurrent viewers while the broadcasts are ongoing.videosyoute videostakes in one or multiple video IDs and retrieve all public metadata for those videos. This complements results returned fromyoute search, as they contain only limited information.youtehydrate<resource-id>....--outfile<file.json>You can put as many IDs as you need, separating each with a space.Like search, you can also tidy the data to a CSV using the--tidy-tooption.youtehydrate<resource-id>...--outfile<file.json>--tidy-to<file-name.csv>Use IDs from text fileYou can hydrate a list of video ids stored in a text file by using--file-pathor-f. The text file should contain a line-separated list of video ids, with no header.youtehydrate-f<id-file.csv>This option is often used in combination withyoute dehydrate, which retrieves the ids from raw JSON returned byyoute searchand stores them in a text file.channelsyoute channelsworks the same asyoute videos, except it retrieves channel metadata given channel ids.commentsyoute commentsretrieves top-level comments (comment threads) on one or multiple videos or channels. It takes in a list of ids and a flag indicating which type these ids are (i.e. videos or channels).To retrieve comments on videos, specify the video ids and pass the--by-video-idor-vflag.youtecomments<id>...--by-video-id--outfile<file.json>#ORyoutecomments<id>...-v--outfile<file.json>To retrieve comments on channels, specify channel ids and pass the--by-channel-idor-cflag.youtecomments<id>...--by-channel-id--outfile<file.json>
OR
youtecomments<id>...-c--outfile<file.json>If neither of the flags are specified,youte commentswill assume the ids are thread ids and retrieve the full metadata for those threads.You can search within the threads and filter threads that match the search terms, by using the--queryor-qoption.youtecomments<ids>...-v--outfile<file.json>-q"search term"repliesWhileyoute commentsonly retrieve top-level comment threads, if those threads have replies, they can be retrieved usingyoute replies.youte repliestakes a list of thread ids and return the replies to those threads.youtereplies<ids>...--outfile<file.json>related-torelated-toretrieves videos related to a video.youterelated-to<video-ids>...-o<file.json>If multiple video IDs are inputted, youte will iterate through each video ID separately, retrieving all related videos to each video, one by one.Other options include:--safe-search [none|moderate|strict]
                                  Include or exclude restricted content
                                  [default: none]
  --region TEXT                   Specify region the videos can be viewed in
                                  (ISO 3166-1 alpha-2 country code)
  --lang TEXT                     Return results most relevant to a language
                                  (ISO 639-1 two-letter code)chartyoute chartretrieves the most popular videos in a region, specified byISO 3166-1 alpha-2 country codes. If no argument or option is given, it retrieves the most popular videos in the United States.For example:youtechart<region-code>-o<file.json>full-archiveA new feature added inyoute2.1.0 is the ability to run a full archive workflow in one command.youte full-archiverunsyoute search, then retrieving video and channel metadata for the search results, as well as getting comments and replies for those videos as well. All data are then tidied and stored in multiple tables in an SQLite database.youtefull-archive<query>[options]-o<name-of-database-file>The search options are identical toyoute search. Name of the file given to-ohas to have SQLite extension (i.e..dbor.sqlite).Below are the list of tables and the corresponding YouTube resource that they contain:search_result: search results fromyoute searchvideo: videoschannel: channelscommment: comment threads and repliesWarning: sincefull-archivewill potentially run a large number of queries, it's important to ensure you have enough API quota. You can select which resources to retrieve by using the--selectoption.--selecttakes one or a comma-separated list of YouTube resource types, namelyvideo,channel,thread, andreply. Note that if you selectreply,threadalso has to be selected. This is because comment thread replies are retrieved using thread IDs, thus collecting comment threads is a must before getting replies. Because of that, if you want to archive the replies, both 'thread' and 'reply' will have to be specified.dehydratedehydrateextracts the IDs from a JSON file returned from YouTube API.youtedehydrate<file-name.json>Options:
  -o, --output FILENAME  Output text file to store IDs inYouTube API Quota system and youte handling of quotaMost often, there is a limit to how many requests you can make to YouTube API per day. YouTube Data API uses a quota system, whereby each request costs a number of units depending on the endpoint the request is made to.For example:search endpoint costs 100 units per requestvideo, channel, commentThread, and comment endpoints each costs 1 unit per requestFree accounts get an API quota cap of 10,000 units per project per day, which resets at midnight Pacific Time.At present, you can only check your quota usage on theQuotaspage in the API Console. It is not possible to monitor quota usage via metadata returned in the API response.youtedoes not monitor quota usage.