Package: caterpillar-hls
Caterpillarcaterpillaris a hardened HLS merger. It takes an HTTP Live Streaming VOD URL (typically an .m3u8 URL), downloads the video segments, and attempts to merge them into a single, coherent file. It is specially designed to combat timestamp discontinuities (symptom: a naive FFmpeg run spews tons of "Non-monotonous DTS in output stream" warning messages and ends up with a useless file with completely broken timestamps).caterpillarsupportsup to version 3of the HTTP Live Streaming protocol (VOD only; non-VOD playlists are treated as VOD, and may result in unexpected consequences).ContentsDependenciesInstallationFor end usersFor developers and beta testersFor application developersUsageBatch modeConfigurationNotes and limitationsEtymologyCopyrightDependenciesA recent version ofFFmpeg. FFmpeg 3.3.4 is known to work with caterpillar; FFmpeg 3.2.4 is known to NOT work.InstallationPython 3.6 or later is required.If in doubt, check out the detailed "Installation Guide for Novices".For end usersTo install,pip install caterpillar-hlsTo upgrade to the latest version,pip install -U caterpillar-hlsFor developers and beta testersTo install from the master branch,git clone https://github.com/zmwangx/caterpillar.git
cd caterpillar
python setup.py develop
caterpillar -hTo update to the latest master,cd /path/to/caterpillar
git pull origin masterFor application developersShort of callingcaterpillar.caterpillar.mainwithsys.argvset appropriately, you can access caterpillar's functionality throughcaterpillar.caterpillar.process_entryandcaterpillar.caterpillar.process_batch. Warning: there's no stability guarantee to these interfaces, although I won't break compatibility without a very compelling reason.process_entryandprocess_batchadditionally support event hooks (a feature not exposed to end users). Seecaterpillar.caterpillar.eventsfor types of events emitted and associated data attributes.Usage$caterpillar-husage: caterpillar [-h] [-b] [-e] [-f] [-j JOBS] [-k][-m {concat_demuxer,concat_protocol,0,1}] [-r RETRIES][--remove-manifest-on-success] [--workdir WORKDIR][--workroot WORKROOT] [--wipe] [-v] [--progress][--no-progress] [-q] [--debug] [-V]m3u8_url [output]positional arguments:m3u8_url              the VOD URL, or the batch mode manifest fileoutput                path to the final output file (default is a .ts filein the current directory with the basename of the VODURL)optional arguments:-h, --help            show this help message and exit-b, --batch           run in batch mode (see the "Batch Mode" section indocs)-e, --exist-ok        skip existing targets (only works in batch mode)-f, --force           overwrite the output file if it already exists-j JOBS, --jobs JOBS  maximum number of concurrent downloads (default istwice the number of CPU cores, including virtualcores)-k, --keep            keep intermediate files even after a successful merge-m {concat_demuxer,concat_protocol,0,1}, --concat-method {concat_demuxer,concat_protocol,0,1}method for concatenating intermediate files (defaultis 'concat_demuxer'); seehttps://github.com/zmwangx/caterpillar/#notes-and-limitationsfor details-r RETRIES, --retries RETRIESnumber of times to retry when a possibly recoverableerror (e.g. download issue) occurs; default is 2, and0 turns off retries--remove-manifest-on-successremove manifest file if all downloads are successful(only works in batch mode)--workdir WORKDIR     working directory to store downloaded segments andother intermediate files (default is automaticallydetermined based on URL and output file)--workroot WORKROOT   if nonempty, this path is used as the root directoryfor all processing, under which both the workingdirectory and final destination are mapped; aftermerging is done, the artifact is eventually moved tothe destination (use cases: destination on a slow HDDwith workroot on a fast SSD; destination on anetworked drive with workroot on a local drive)--wipe                wipe all downloaded files (if any) and start over-v, --verbose         increase logging verbosity (can be specified multipletimes)--progress            show download progress bar regardless of verbositylevel--no-progress         suppress download progress bar regardless of verbositylevel-q, --quiet           decrease logging verbosity (can be specified multipletimes)--debug               output debugging information (also implies highestverbosity)-V, --version         show program's version number and exitenvironment variables:CATERPILLAR_USER_CONFIG_DIRcustom directory for caterpillar.confCATERPILLAR_USER_DATA_DIRcustom directory for certain data cached bycaterpillarCATERPILLAR_NO_USER_CONFIGwhen set to a non-empty value, do not loadoptions from user config fileCATERPILLAR_NO_CACHE  when set to a non-empty value, do not read orwrite caterpillar's cacheconfiguration file:<an operating system and user-dependent path>See thewiki pagefor usage examples.Batch modeIn normal mode,caterpillardeals with only one stream. There is also a batch mode for downloading multiple streams at once. In this mode, you specify a manifest file on the command line in the place of the VOD URL, where the manifest file contains a VOD URL and a filename (or path) seperated by a tab on each line, e.g.,caterpillar manifest.txt, wheremanifest.txtcontainshttps://example.com/hls/1.m3u8	1.mp4
https://example.com/hls/2.m3u8	2.mp4
https://example.com/hls/3.m3u8	3.mp4The filenames (or paths) are relative to the parent directory of the manifest file. The tab character is not allowed in the filenames (or paths).Comments that start with#are allowed in the manifest file.Most options for normal mode are also allowed in the batch mode, as are options set in the configuration file.ConfigurationTo save some retyping,caterpillarsupports the configuration of default options in an operating system and user-dependent configuration file. The path is usually~/Library/Application Support/caterpillar/caterpillar.confon macOS,%AppData%\org.zhimingwang\caterpillar\caterpillar.confon Windows, and~/.config/caterpillar/caterpillar.confon Linux. Runcaterpillar -hto view the actual path.The syntax of the configuration file is documented in the template (automatically created for you if possible), duplicated below:# You may configure default options here so that you don't need to
# specify the same options on the command line every time.
#
# Each option, along with its argument (if any), should be on a separate
# line; unlike on the command line, you don't need to quote or escape
# whitespace or other special characters in an argument, e.g., a line
#
#     --workdir Temporary Directory
#
# is interpreted as two command line arguments "--workdir" and
# "Temporary Directory".
#
# Positional arguments are not allowed, i.e., option lines must begin
# with -.
#
# Blank lines and lines starting with a pound (#) are ignored.
#
# You can always override the default options here on the command line.
#
# Examples:
#
#     --jobs 32
#     --concat-method concat_protocolNotes and limitationsEXT-X-MEDIAfor alternative renditions are not supported since multiple playlists need to be downloaded, merged, and muxed, which is beyond the current scope of this tool.EXT-X-STREAM-INFvariant streams are only partially supported: when two or more variant streams are present, the best one (selected based on higher resolution, higher average bandwidth, and higher bandwidth, in that order) is automatically selected; there is no mechanism for users to select another variant at the moment. In addition,AUDIO,VIDEO,SUBTITLESorCLOSED-CAPTIONSattributes referencingEXT-X-MEDIAtags are not supported.Efforts could be made to extract the variant streams and show them to the user, and it is even feasible to proceed with the download if only one variant stream is present. Contribution is welcome for this feature.A note on-m, --concat-method: The final step ofcaterpillaris concatenating one or more parts (generated from splitted playlists with FFmpeg'shlsdemuxer) into a single output file. In this step we provide two methods of choice: theconcat demuxerand theconcat protocol(the former is the default). To pick the non-defaultconcat_protocol, specify--concat-method concat_protocolon the command line, or as a shortcut,-m 1(0is an alias forconcat_demuxer, and1is an alias forconcat_protocol).Each of these two methods may work better in certain cases. For instance, forthis stream, the concat demuxer simply fails with loads of error messages like "Application provided duration: 7980637472 / timestamp: 7994129672 is out of range for mov/mp4 format", whereas the concat protocol works fine. However, forthis stream, the concat protocol dumps a bunch of worrisome warnings like "DTS out of order" or "Non-monotonous DTS in output stream", whereas the concat demuxer doesn't.In short, if the default fails you (either doesn't work outright, or the generated video is bad in some way), try-m 1.Encoding-related issues.caterpillar, since version 1.0, reads and writes every on-disk file in UTF-8 encoding, period. In particular, user config file and batch mode manifests need to be in UTF-8. Legacy config file might need to be migrated to UTF-8.If you're unfamiliar with the concept of character encodings, you may find the classicJoel on Softwarearticle on charsets and Unicodeinformative (gosh, it's fifteen years old now). On Unix-like systems,uchardetis my go-to tool for encoding detection; it's not readily packaged on Windows, and it's not officially supported and a royal pain to compile by yourself (to a non-Windows developer like me, and probably to whomever is reading this note). I don't know a good alternative on Windows, but I have a small web app that does encoding detection entirely in the browser:https://chardet.tcl.sh/.Converting a file to UTF-8, once you know its contents, is actually pretty easy. Any text editor worth its salt (and still afloat) should be able to save as UTF-8; heck, even Windows Notepad can do this (it even appears to be the default).This note is mostly only relevant to Windows. I can't overstate how I absolutely fucking detest dealing with encodings surrounding Microsoft products. If you ever saw �'s on web pages in places where there should be quotation marks — that's Latin-1, or Windows-1252, being decoded as UTF-8. That's just the tip of the iceberg. If you ever deal with CJK, or another alphabet other than Latin, oh baby, you're in for a world of pain. I can't count the number of times I opened a file or unzipped an archive from a Windows user, only to find garbage filenames or garbage content or both, and had to guess the encoding to restore the meaning. Also, meet Windows cmd or PowerShell; you have to fuckingchcp 65001just to work with Unicode, and even then for whatever fucking reason I still get ??? for Chinese characters in input commands – oh and I use Cmder, which allows a secondary font for CJK, an infinite improvement over stock cmd or PowerShell already, but still no luck. You may also want to meet the latest and greatest Office for Mac, where you still can't import a UTF-8-encoded CSV file with non-ASCII characters (I know a workaround, for fuck's sake). Not sure about the Windows version, and not enthusiastic enough to find out. (In all fairness, Apple's HFS+ with its infamous NFD-based UTF-8-MAC encoding in filenames was hell, but other than that Mac OS X and its successors have been pretty good encoding-wise; and APFS is finally rid of the pseudo-NFD garbage — it might have had its own problems at first, but I've yet to run into any. HFS+ encoding problems are peanuts compared to Microsoft's encoding problems, by the way. I'm not even bringing fonts into the mix.)In addition, if you ever have Chinese filenames in the English version of Windows — gosh, they're so ugly they almost bring tears to my eyes.I had to swap the entire fucking system UI font via a registry hackto keep tears at bay. Last I heard there are plenty of Chinese folks working in Redmond — hell, I even have acquitances there. How they managed to tolerate this shit for so many years is beyond me.Pretty sure I can tell you a lot more if I spent more time with Microsoft products (which is close to zero at the moment).Windows is more than thirty years old now, and the UN is more than seventy years old. Yet Redmond still hasn't got the memo on globalization.This dissolved into another Microsoft rant. Oh well.EtymologyThe word "caterpillar" starts withcat(1), and the body of a caterpillar is segmented.CopyrightCopyright © 2017 Zhiming WangThis project is licensed underthe MIT license. SeeCOPYINGfor details.