Package: bikesanity
BikeSanity Cycle Journal Exporter for CrazyGuyOnABikeWhat is this?In this repository is the complete code needed to download, extract, interpret, and export - in an attractive, completely new andmobile friendlyHTML format - bicycle touring journals available in the popularcrazyguyonabike.com(CGOAB) website. You can also convert your journals to PDF format (new!) or to a JSON data structure for transfer to other cycle touring platforms.You can use it download and create backups of your journals, to interpret them into a form where the content is easier to extract, and to re-format them in various formats readable offline.All the authored content in journals will be retrieved, including all pages, text, images and maps. "Social" content such as guestbooks will not be retrieved.For technical users, it also populates an object model of all cycle journals. You can write your own Python programs, and linking to this library get hold ofJournalobjects containing sensibly structured content for you to work on as you wish.Can I see a sample of an extracted journal?Sure! Thesamples/htmlfolder contains a sample of the converted journal HTML format, generated using placeholder data. This shows off the responsive layout, cleaner design, and fully working map embedding. You can browse that sample live athttps://www.bikejournalbackup.com/journals/sample/index.html.All images used in the sample are in the public domain and freely licensed frompexels.How can I use this tool?This is a command line tool, and source code written in Python 3.6, to perform journal extraction.If you are comfortable using command line tools and have Python on your system, you can install this module and use it directly.Otherwise you may be better off using the user-friendly graphical toolavailable here. That is built on top of this library and so can perform the same actions, but will be easier for non-technical users.Who can use it?Anyone. All code is free and open source, and provided as a service to the cycle touring community.The code is licensed under the permissive Apache 2.0 license. Please feel free to fork this code, or use it for your own development.Installing this moduleThis is standard Python package, and you can install it from PyPi or directly usingpippip install bikesanityor, after cloning/downloading the repository, in the base directorypip install .The module is implemented in pure Python, with a few dependencies which will be automatically installed.After installation, thebikesanity-runscript will be made available on your path. Runbikesanity-run --helpto see options.Quick StartThe three simple commands will download, interpret, and re-output a journal in new HTMLbikesanity-run download <http://www.crazyguyonabike.com/doc/JournalNameHere>
bikesanity-run process <12345>
bikesanity-run publish <12345> --htmlReplace<>values with a link to the journal you want to download, and its ID (which will be shown to you upon download). The your re-formatted journal will be available inCycleSanityJournals/processed/12345/html/index.html!Retrieving and interpreting journal content using the scriptThebikesanity-runscript can perform four different operations:Download of a complete journal, as is, from crazyguyonabike.comInterpretation ("processing") of adownloadedjournal to extract  all the content into an understandable internal (object) model.Interpretation of anexportedjournal (i.e. one formally exported using the tool provided by CGOAB) to extract all the contentPublishing of a processed journal to create new, clean HTML content including all images, maps, and structure locally browseable. You can also publish to create a unified PDF of your journal, or a simplified JSON data structure for transfer to other platforms.By default all journals will be downloaded and processed into a folder created in your home directory (/home/<usr>/CycleSanityJournalson mac/linux,c:\Users\<usr\CycleSanityJournalson Windows). You can change this path using the options below.Downloading journalsDownload a journal by using thedownloadargument and providing a URL to the journal front page. This can be a permalink likehttp://www.crazyguyonabike.com/doc/JournalNameHere.bikesanity-run download http://www.crazyguyonabike.com/doc/JournalNameHereBecause of the slow rate of retrieval from CGOAB, this can take a few minutes or longer for very large journals. The default download path will beCycleSanityJournals/downloads/<journal_id>. You can change it with the options below:--locationchanges the location to download to.--from-pagepicks up the download from a provided page number, e.g.--from-page 8. This is useful for picking up downloads that failed, e.g. in the middle.--do-processalso performs an interpreting processing run once the download has completed (see below)--no-local-readability-postprocessingturns off post-processing to make journals locally readable and navigable (advanced)Processing journalsDownloaded or exported journals can be interpreted using theprocessorprocess-exportedarguments respectively. You should provide thejournal id, which will be the number attributed to the downloaded folder, e.g. 12345.bikesanity-run process 12345By default, the processor expects downloaded journals to be in theCycleSanityJournals/downloads/<journal_id>directory, and exported journals to be inCycleSanityJournals/exported/<journal_id>. It will output processed journals on the pathCycleSanityJournals/processed/<journal_id>.Options:--input_locationchanges the location to take downloaded/exported input from--out_locationchanges the location to send processed output toFollowing processing, a complete object model of the journal will be created and saved in as a serialized Python pickle asjournal.pickle- for technical users, you may wish to load and inspect this. All resources (images and maps) will be copied to the newprocessed/resourceslocation.Locally publishing to HTML, PDF and other formatsCurrently it possible to publish processed journals to HTML, PDF documents, or a JSON data structure. Publish any processed journal using thepublishargument and providing the samejournal id:bikesanity-run publish <12345> --html --pdf --jsonUse any combination of the optional flags to generate HTML, PDF or JSON output:--htmlwill produce attractive and clean HTML that can be fully-functionally browsed on the local machine, including dynamic maps.--pdfwill generate a collected PDF document of the journal, including all images. Large journals may be split into several PDF parts--jsonwill generate a simple JSON data structure that may be used to migrate the journal to other platformsBy default, HTML is output inside the processed directory, atCycleSanityJournals/processed/<journal_id>/html- openindex.htmlto browse the journal index.Options:--input_locationchanges the location to take processed input from--out_locationchanges the location to send HTML output to. Note links to resources may break if your change this.HTML is generated by populating templates. These are available inresources/templatesif you would like to adjust the exact appearance of the output. Publication to EPUB is work in progress - watch this space!Why is this necessary?CGOAB has for twenty years been a fantastic resource. What has made it so good has been the contributions of the wonderful bike touring community. It contains a wealth of material primarily in the form of thousands of journals describing bicycle tours.However, in recent years the site has seen significant fall in user activity. Since 2016, the rate of journal submission has roughly halved. A sometimes toxic environment on the site forum has driven away many users.The site is run and maintained by one person. It has become clear that the understandable work involved in running the site has become a significant burden, financial and otherwise, on the administrator.Nobody except the owner can administrate the site, or know how its bespoke and now very old internals work. The site is now extremely dated in appearance and operation.Because of the factors above, there is a significant risk that CGOAB is vulnerable to eventual outage. This means there is a serious consideration that years of work and thousands of touring journals may be lost forever. BikeSanity allows these journals to be retrieved for posterity and never lost.How do we know it works?The interpreter has been successfully tested against >10000 journals.What difficulties are there in retrieving journal content?To the great credit of CGOAB's administrator, two factors make retrieving journal content easier than it could be:All content is owned purely by their authors. The administrator does not own any of the journal content (that he has not authored).CGOAB provides an "export" function that can allow journals to be downloaded as-is from the site.However, even after export, the content remains in CGOAB's peculiar HTML format. This is highly non-standard and often faulty (see technical details below) and very laborious to extract content from manually.The administrator of CGOAB is extremely resistant to any attempts to liberate journal content from this old and unworkable format. To make it reliably readable into the future, a robust parser is needed. This library attempts to provide that facility.What technical deficiencies are there in CGOAB journal format?Unfortunately, it's clear that there are serious technical deficiencies in the CGOAB site and platform which are actively causing problems with maintaining and modernising the site, and are a risk factor for its long-term stability.The front end code has serious problems:No CSS is used across the sitewhatsoever. All style is baked directly into HTML, an extremely bad practice. This is one reason why the antiquated visual style of the site has never changed, and would be very difficult to change.JavaScript is baked in HTML across the site, and is repeated in every single page.All the HTML uses nonstandard markup and is a terrible mess. Cases of tags are mixed indiscriminately, e.g.<B>vs<b>. HTML tags are not closed properly, and sometimes are nested recursively, like theDDtags.These are very basic errors that have a direct impact on the difficulty for the administrator to maintain and update the site. A case in point is responsive layout for mobile devices, which has been requested for many years yet but is still not supported. Google terminating their free mapping API caused a major panic and months to resolve.These problems raise serious worries about the robustness of the server code. At the very least, it is likely to be impossible to work with for anyone other than their creator, who has developed the entire site in isolation from modern coding practices.Do you have any relationship with any other bicycle touring resources?No, none whatsoever. This tool has been provided completely independently andpro bonofor the benefit of the cycle touring community. If you plan to make copies or republish material retrieved by this tool, please ensure that you are, or have permission of, the author.Absolutely no property of CGOAB is included in this library whatsoever. All code is original, other than third-party libraries which have appropriate open-source licenses attached.