Package: wnget
Rarely will web novel sites provide any means to read their contents off
line, and this is precisely whywngetcame to be. It is a tool to
scrape web novels from blogs, and optionally convert them to epub
format. It provides several options to configure the exact behaviour,
while at the same time trying to provide sane defaults. The strings for
the next/previous navigation links, as well as CSS class for
title/content containers can be configured, among other settings.Besides the mainwngetutility, the suite includewnbook, to
generate ebooks from the downloaded contents, andwnlocal, to rewrite
the links of html files that can be resolved to a local file using the
same rules applied bywnget.Installation$pipinstallwngetIf you happened to have cloned the repo and are playing with the code,
you probably want to installwngetin “editable” mode while you’re
working on it. This is so it becomes both installed and editable in
project form.Assuming you’re in the root of the project, just run:$pipinstall-e.ConfigurationAny argument that starts with ‘–’ can also be set in a config file
(./wnget.conf or ~/.wnget.conf or specified via -s). The
recognized syntax for setting (key, value) pairs is based on the INI and YAML
formats (e.g. key=value or foo=TRUE). If an argument is specified in more than
one  place, then commandline values override config file values which override
defaults.The default configuration looks like this:keeplinks:nofirsttitle:nonext:"NextChapter"previous:"PreviousChapter"titleclass:entry-titlecontentclass:entry-contentepub:default_ebooklimit:0# No limit if < 1Since every web novel site will have a unique theme, it is unlikely the
default configurations shipped with _wnget_ will work out of the box (unless
you happen to be addicted to the same ones as yours truly). Feel free to
extend data/sites.yaml for your favorite sites and issue a pull request.Usage$wnget--helpusage:wnget[-h][-sfile][-k][-f][-ncaption][-pcaption][-tcss_class][-ccss_class][-etitle][-lN][-v]first_url[last_url]positionalarguments:first_urlfirstURLtocrawllast_urloptionallastURLtocrawl(stopsafterreaching)optionalarguments:-h,--helpshowthishelpmessageandexit-sfile,--settingsfileconfigfilepath-k,--keeplinksrewriteandkeepnavigationlinksinHTMLcontent-f,--firsttitlekeepfirsttitlematchinsteadoftryingtobesmartaboutit-ncaption,--nextcaptionnextlinkcaption(default:'Next Chapter')-pcaption,--previouscaptionpreviouslinkcaption(default:'Previous Chapter')-tcss_class,--titleclasscss_classtitlecontainerclass(default:'entry-title')-ccss_class,--contentclasscss_classcontentcontainerclass(default:'entry-content')-etitle,--epubtitlecreateEpubwiththistitle(willusecover.jpg/pngiffound)-lN,--limitNcrawlatmostNpages-v,--versionshowprogram'sversionnumberandexitUsage examplesInvoke each command without arguments to display help information.The scraper looks for a given CSS class in the title and content containers,
and those can be set manually to suit your web novel site of choice. It also
looks for links with default strings to find the next and previous chapters,
and this can also be set by hand.To scrape all chapters of a given web novel, following links, and saving
each chapter in a different html file in the current directory:$wngethttp://example.com/first_chapter_linkOr, for more advanced uses, downloading all chapters until a given link
is retrieved, and generate an EPUB with the loot:$wnget-e"My Web Novel"\http://example.com/first_chapter_link\http://example.com/first_chapter_linkAdditionally, the ebook functionality can be used directly through thewnbookstandalone utility. Just provide the HTML index file and a
name for the book, and it will generate an ebook with all referenced
resources in the working directory:$wnbookindex.html"My Web Novel"Also, if a cover.png or cover.jpg file is present, it will be used as
cover page. Its use as standalone command will often prove more
flexible, as it exposes features not normally used by the main
application, while allowing some manual tweaking of the downloaded
contents and index files.Here, generating a book with relative paths, and custom filename, cover
image, and language/author metadata:$wnbook../index.html"I Shall Seal the Heavens (我欲封天)"\--filename=issth.epub--language=zh--author="Ergen (耳根)"\--cover~/images/MengHao.pngAnd finally, to rewrite links within a file so that they point to
already downloaded resources,wnlocalcan be used.Either to print the converted file to stdout:$wnlocalintroduction.htmlOr to write it back to disk:$wnlocalintroduction.htmlnewfile.htmlTODOMake content selection more flexible: by tag, class, caption, or xpath.Add option for elements to be removed during the parsing stage.Add interactive title selection mode (and ability to repeat choice).In-content image support (download, and store locally).