Package: colusa
colusaRender website to ebook to make it easier to read on devices.Installationpython3setup.pyinstallUsageStart from scratchFirst of all, we need to generate a configuration file forcolusato work on.colusahas builtin command to generate a template configuration as starter.
Run following command to generate configuration file:$colusainitnew_ebook.jsoncolusawill generate a configuration file as below:{"title":"__fill the title__","author":"__fill the author__","version":"v1.0","homepage":"__fill url to home page__","output_dir":"__fill output dir__","urls":[]}We have to modify the configuration file to fill up valid information.Add content to ebookAfter generating the configuration file, all we need to do is adding (blog posts, articles, webs)' urls tourlsfield in the configuration file.Example for final configuration:{"title":"The Great Mental Models","author":"Farnam Street Media Inc","version":"v1.0","homepage":"https://fs.blog","output_dir":"fsblog","urls":["https://fs.blog/2018/04/first-principles/","https://fs.blog/2016/04/second-order-thinking/","https://fs.blog/2017/06/thought-experiment/","https://fs.blog/2018/05/probabilistic-thinking/","https://fs.blog/2019/12/survivorship-bias/"]}Update ebook contentWe can update ebook content by modifying theurls, by adding or removing url in theurls, the result ebook will be changed.Generate ebook contentAfter adding or removing url inurls, we need to invokecolusato have it regenerate ebook content. Run following command at terminal:$colusageneratenew_ebook.jsonBy invoking above command,colusawill download webpages (specified inurls), parse, transform them to asciidoc format, and save them tooutput_dir.colusaalso create a neccessary information for ebook compilating at later steps.Compile ebook for consuming purposePrerequisitesBefore generating ebook, we need to install asciidoctor tools. Follow install guideline on following websites:for generating html:https://asciidoctor.orgfor generating epub:https://asciidoctor.org/docs/asciidoctor-epub3/for generating pdf:https://asciidoctor.org/docs/asciidoctor-pdf/Generating ebooksTo help with generating ebook,colusaalso create aMakefilein the root folder of the ebook. In theMakefile, there are three common targets that we can use to generate ebook in html, epub, pdf formats.# to generate html$makehtml# to generate epub$makeepub# to generate pdf$makepdfGenerated ebooks will be saved to./outputfolder.user:output/ $ ls                                                            [10:55:55]
total 3056
drwxr-xr-x  10 320B images
-rw-r--r--   1 699K index.epub
-rw-r--r--   1 131K index.html
-rw-r--r--@  1 694K index.pdfList of Supported WebsiteCurrentlycolusasupports only limited number of websites. The following list all support websites:https://untools.cohttps://unintendedconsequenc.eshttps://blog.acolyer.orghttps://fs.bloghttps://increment.comhttps://slack.engineeringhttps://medium.comhttps://www.cs.rutgers.edu/~pxk/https://www.preethikasireddy.comhttps://engineering.atspotify.comhttps://truyenfull.vnhttps://avikdas.comhttps://www.infoq.comContributionContribution is welcome. You can open issues to request for supporting more websites, open PR to help with those issues, or anything else like documentation, code contribution.Changelogv0.12.0 (2022-07-24)NewAdd new command to crawl an URL. [Nguyen Huu Hoa]Crawl an URL (website) to help generate list of URL, mostly story chaptersChangesAdd some debug capability. [Nguyen Huu Hoa]Add support for new websites. [Nguyen Huu Hoa]OtherMerge branch 'main' of github.com:huuhoa/colusa. [Nguyen Huu Hoa]Chg(plugins/truyenfull): clean ads content. [Nguyen Huu Hoa]Chg(plugins/truyenfull): clean ads content. [Nguyen Huu Hoa]v0.11.0 (2022-02-17)ChangesSupport for tangthuvien (#72) [Huu Hoa NGUYEN]FixGitchangelog ignore pattern. [Nguyen Huu Hoa]v0.10.0 (2021-10-16)ChangesUpdate dev requirements. [Nguyen Huu Hoa]Improve code coverage (#21) [Huu Hoa NGUYEN]Mock up two functions download_image and download_contentdownload_contentwill return existing cached file, so that we don't have to redownload every time
we run the testdownload_imagewill just return True, do nothing, so that we don't have to download imagesOtherAdd: support for techtarget.com (#32) [Huu Hoa NGUYEN]chg(asciidoc_visitor): support parsing datasrc and data-srcset for imgadd(web): support for techtarget.comv0.9.0 (2021-08-26)NewIntegration tests (#20) [Huu Hoa NGUYEN]chg: add tox.ini for running toxchg(colusa): move colusa source to src folderSupport parsing site xp123.com (#18) [Huu Hoa NGUYEN]OtherChore: update setup.cfg for version location. [Nguyen Huu Hoa]Prepare for next release. [Nguyen Huu Hoa]Refactor(etr): Rework on Extractor (#19) [Huu Hoa NGUYEN]refactor(etr): move _parse_yoast from a plugin extract to base Extractorrefactor(etr): rename methods for clarificationrenameinternal_initto_find_main_contentrenameget_authorto fieldauthorand_parse_authorfor parsing valuerenameget_publishedto fieldpublishedand_parse_publishedfor parsing valuerenameget_titleto fieldtitleand_parse_titlefor parsing valueadd_parse_metadatafor parsing all related metadata from htmlrefactor(etr): change signature of Extractor._find_main_content_find_main_contentis now return bs.Tag instead of setting value for fieldmain_content. The change make it more clear for purpose of_find_main_content, i.e. only to find the main content, does not modify anything_parse_metadatawill be executed after we found the main contentv0.8.0 (2021-08-22)NewSupport rendering additional book properties (#16) [Huu Hoa NGUYEN]In the book configuration file, add new arraybook_propertieswith content is list of strings. Those strings will be render as
book properties on master file (index.asciidoc)Example:"book_properties":["ifdef::backend-pdf[]",":front-cover-image: image:cover.pdf[]",":notitle:","endif::[]","ifdef::backend-epub3[]",":front-cover-image: image:cover.png[]","endif::[]"]Above example will instruct asciidoctor processor to use:cover.pdf as front cover image when generating pdfcover.png as front cover image when generating epub3ChangesRender html table as native asciidoc table (#17) [Huu Hoa NGUYEN]OtherPrepare for bump version 0.8. [Nguyen Huu Hoa]Add(plugins): support for scrumcrazy.wordpress.com. [Nguyen Huu Hoa]v0.7.0 (2021-08-20)ChangesImprove article parsing (#15) [Huu Hoa NGUYEN]add: agilethought support to get article's authoradd: support website tech.trivago.comMetadata rendering (#14) [Huu Hoa NGUYEN]render metadata in a more clean way, the format should beby **{author}** on {published_date} at {url | domain}OtherChore: refactor project's setup configurations. [Nguyen Huu Hoa]Setup codeql-analysis. [Huu Hoa NGUYEN]Setup dependabot. [Huu Hoa NGUYEN]Dev: update requirements_dev.txt. [Nguyen Huu Hoa]Docs: update CHANGELOG. [Nguyen Huu Hoa]v0.6.0 (2021-08-18)NewSupport new websitehttps://agilethought.com(#11) [Huu Hoa NGUYEN]ChangesEtr: improve metadata rendering for generated articles (#10) [Huu Hoa NGUYEN]v0.5.1 (2021-08-14)ChangesUpdate requirements for package and dev. [Nguyen Huu Hoa]v0.5.0 (2021-08-14)NewSupport new websitehttps://staffeng.com. [Nguyen Huu Hoa]ChangesInitial configuration for bumpversion. [Nguyen Huu Hoa]Add comments at the beginning of included files. [Nguyen Huu Hoa]new version of asciidoctor removes leading and trailing empty lines of included files
therefore the beginning of new section in the included files will not
be separated as expected. The work around is to add comment line
at the very beginning of included files.Support for parsing some common webblogs. [Nguyen Huu Hoa]Support for parsing hbr.org. [Nguyen Huu Hoa]Support parsing content for blog-content and wikipedia. [Nguyen Huu Hoa]Support parsing srcset dimension with 'x' specification. [Nguyen Huu Hoa]Add suffix to generated file name to prevent name colliding. [Nguyen Huu Hoa]Support detecting webpage content insidemaintag. [Nguyen Huu Hoa]Passthrough the table tag content. [Nguyen Huu Hoa]Special treatment for image inside an anchor tag. [Nguyen Huu Hoa]Cleanup code to get content class of a website. [Nguyen Huu Hoa]Heading level of generated asciidoc. [Nguyen Huu Hoa]Add support for parsing new websites. [Nguyen Huu Hoa]https://cadenceworkflow.iohttps://softwareengineeringdaily.comFixCorrect config for bumpversion. [Nguyen Huu Hoa]Slugify that import non existing unicode from idna. [Nguyen Huu Hoa]Get correct image suffix by parsing url first to get onlypathin URL. [Nguyen Huu Hoa]OtherDocs: add some documents. [Nguyen Huu Hoa]Add: proper configuration for packaging. [Nguyen Huu Hoa]add bump_version supportv0.4.0 (2020-10-14)NewSupport new websitehttps://www.infoq.com. [Nguyen Huu Hoa]Yaml configuration (#9) [Huu Hoa NGUYEN]feat: support configuration file in YAML formatConfiguration file format is determined by extension, i.e json or ymlchg: add help str to make error report more concisechg: add logs statements in various placechg: dev: add CHANGELOG.md to record changesChangesCleanup output for content from truyenfull.vn. [Nguyen Huu Hoa]Tolerate for some non-comforms htmls. [Nguyen Huu Hoa]Add customization options for generated ebook. [Nguyen Huu Hoa]metadata: type bool, default is True. Metadata such as published_date, source url are generated after article (chapter) title if Truetitle_prefix_trim: type string, default is empty. When specified, string value intitle_prefix_trimwill be removed from article (chapter) titleOtherDocs: Prepare for release v0.4.0. [Nguyen Huu Hoa]v0.3.1 (2020-09-26)ChangesUpdate meta information for pypi.org package (#8) [Huu Hoa NGUYEN]Add long descriptionAdd classifiersPrepare for release version 0.3.1v0.3.0 (2020-09-26)NewSupport book parts. [Nguyen Huu Hoa]Render additional information for book partsSupport website truyenfull.vn, avikdas.com. [Nguyen Huu Hoa]Support web engineering.atspotify.com. [Nguyen Huu Hoa]Support websitewww.preethikasireddy.com. [Nguyen Huu Hoa]Support website cs.rutgers.edu. [Nguyen Huu Hoa]Support website medium.com. [Nguyen Huu Hoa]Support website slack.engineering. [Nguyen Huu Hoa]Support website increment.com. [Nguyen Huu Hoa]Add pdf target to Makefile for generating pdf format. [Nguyen Huu Hoa]ChangesRename project from symphony to colusa (#7) [Huu Hoa NGUYEN]Colusa is not yet existed on pypi.org, so I rename the project
in order to be able to upload it to pypi.orghttps://pypi.org/project/colusaCleanup code. [Nguyen Huu Hoa]Cleanup etr.Transform to remove obsolete methods. [Nguyen Huu Hoa]Add coloring log for improved experiences in using app. [Nguyen Huu Hoa]Support to rendercodetag in asciidoc_visitor. [Nguyen Huu Hoa]Implement Visitor pattern for saving document to asciidoc (#6) [Huu Hoa NGUYEN]implement visitor pattern for writing asciidoc file formatimplement visit methods for various tagssuppress empty text in anchor tagadd support for knowledgegraph.todayall unknown PageElement are classified to visit_unknownsupport pre tagUpdate gitignore to exclude PyCharm IDE generated files. [Nguyen Huu Hoa]Update usage for symphony. [Nguyen Huu Hoa]to initialize new ebook: symphony initto generate ebook contents: symphony generateAdd setup.py for easier installation. [Nguyen Huu Hoa]Implement plugin archiecture for Extractor, Transformer (#4) [Huu Hoa NGUYEN]Update requirements.txt. [Nguyen Huu Hoa]Ignore empty heading. [Nguyen Huu Hoa]Support getting metadata from opengraph and getting main content from microformats - hentry. [Nguyen Huu Hoa]Report error when cannot understand a website. [Nguyen Huu Hoa]Update format for ul, ol. [Nguyen Huu Hoa]Default get_title for Extractor to get information from meta tag og:title or get from header>title. [Nguyen Huu Hoa]Support rendering code block. [Nguyen Huu Hoa]Don't use Renderer any more, since Transformer is enough to render asciidoc content. [Nguyen Huu Hoa]Transform tags: table, pre. [Nguyen Huu Hoa]Change encoding when saving content to cache and loading from cache. [Nguyen Huu Hoa]Correct rendering table code. [Nguyen Huu Hoa]Remove line break in heading tags. [Nguyen Huu Hoa]Add more information to README. [Nguyen Huu Hoa]Update README.md for Usage. [Nguyen Huu Hoa]FixError when check for existent ofparagraph-imageinfigureclass. [Nguyen Huu Hoa]Error occurs when tagfiguredoes not have any classes, the node.get('class') will return None.
Therefore when checking for existent of a text in None will throw exception.Remove extra line separators in truyenfull.vn. [Nguyen Huu Hoa]Generating new configuration. [Nguyen Huu Hoa]Rendering image in asciidoc. [Nguyen Huu Hoa]Parsing url and dimension in srcset attribute of img tag. [Nguyen Huu Hoa]OtherRefactor: remove classmethods inside Transformer to make it open for extension. [Nguyen Huu Hoa]Refactor: move etr register extractor, transformer and their factory functions from etr_factory to etr. [Nguyen Huu Hoa]Cleanup codeMake it more clean in using register decoratorsDocs: Update README.md. [Nguyen Huu Hoa]Docs: Update README.md for installing tools to generate ebooks. [Nguyen Huu Hoa]Tests: add skeleton for unit testing. [Nguyen Huu Hoa]Refactor: remove unused params in Transformer. [Nguyen Huu Hoa]Refactor: cleanup code. [Nguyen Huu Hoa]Refactor: create Symphony class to handle all business logics. [Nguyen Huu Hoa]downloading urlstransform and generating ebook contentsgenerating ebook master filegenerating MakefileRefactor: create symphony package and move all etr related to symphony. [Nguyen Huu Hoa]Docs: Update README.md. [Nguyen Huu Hoa]Wip: new way to transform html content to asciidoc format. [Nguyen Huu Hoa]Update README.md (#1) [Anh Le (Andy)]Add installation scriptCleanup ads on fsblog. [Nguyen Huu Hoa]Add support for fs.blog. [Nguyen Huu Hoa]Refactor code to make it easier to support new websites. [Nguyen Huu Hoa]Cleanup code and support rendering article's metadata. [Nguyen Huu Hoa]Change string formats to use f-string style. [Nguyen Huu Hoa]Add requirements.txt. [Nguyen Huu Hoa]Cleanup. [Nguyen Huu Hoa]Fix factory creator for unintendedconsequences. [Nguyen Huu Hoa]Generate makefile to generate ebook. [Nguyen Huu Hoa]Support render pre tag. [Nguyen Huu Hoa]Support config file to make it more general in supporting more repetitive works. [Nguyen Huu Hoa]Update ignote patterns. [Nguyen Huu Hoa]Generalize to accept new webblog. [hoanh]Update tool to generate index.asciidoc from template. [hoanh]Tools to download unintendedsequences. [hoanh]First version. [hoanh]Initial commit. [Huu Hoa NGUYEN]