Package: bookstack-file-exporter
bookstack-file-exporterTable of Contentsbookstack-file-exporterBackgroundFeaturesUse CaseUsing This ApplicationRun via PipRun via DockerAuthenticationConfigurationBackup BehaviorGeneralImagesAttachmentsModify Markdown FilesObject StorageMinio BackupsFuture ItemsBackgroundIf you encounter any issues, want to request an additional feature, or provide assistance, feel free to open a Github issue.This tool provides a way to exportBookstackpages and their content (text, images, metadata, etc.) into a relational parent-child layout locally with an option to push to remote object storage locations. SeeBackup Behaviorsection for more details on how pages are organized.This small project was mainly created to run as a cron job in k8s but works anywhere. This tool allows me to export my docs in markdown, or other formats like pdf. I use Bookstack's markdown editor as default instead of WYSIWYG editor and this makes my notes portable anywhere even if offline.FeaturesWhat it does:Discover and build relationships between BookstackShelves/Books/Chapters/Pagesto create a relational parent-child layoutExport Bookstack pages and their content to a.tgzarchiveAdditional content for pages like their images, attachments, and metadata and can be exportedThe exporter can alsoModify Markdown Filesto replace image and/or attachment links with local exported paths for a more portable backupYAML configuration file for repeatable and easy runsCan be run viaPythonorDockerCan push archives to remote object storage likeMinioBasic housekeeping option (keep_last) to keep a tidy archive destinationSupported backup targets are:localminios3 (Not Yet Implemented)Supported backup formats are based on Bookstack API and shownhereand below:htmlpdfmarkdownplaintextUse CaseThe main use case is to backup all docs in a relational directory-tree format to cover the scenarios:Share docs with another person to keep locally.Offline copy wanted.Back up at a file level as an accessory or alternative to disk and volume backups.Migrate all Bookstack page contents to Markdown documenting for simplicity.Provide an easy way to do automated file backups locally, in docker, orkubernetesfor Bookstack page contents.Using This ApplicationEnsure a valid configuration is provided when running this application. SeeConfigurationsection for more details.Simple example configuration:# config.ymlhost:"https://bookstack.yourdomain.com"credentials:token_id:""token_secret:""formats:# md only example-markdown# - html# - pdf# - plaintextoutput_path:"bkps/"assets:export_images:falseexport_attachments:falsemodify_markdown:falseexport_meta:falseverify_ssl:trueRun via PipThe exporter can be installed via pip and run directly.Python VersionNote: This application is tested and developed on Python version3.12.X. The min required version is >=3.8but is recommended to install (or set up a venv) a3.12.Xversion.Examplespython-mpipinstallbookstack-file-exporter# if you prefer a specific version, example:python-mpipinstallbookstack-file-exporter==X.X.X# using pippython-mbookstack_file_exporter-c<path_to_config_file># if you already have python bin directory in your pathbookstack-file-exporter-c<path_to_config_file>OptionsCommand line options:optionrequireddescription-c,--config-fileTrueRelative or Absolute path to a valid configuration file. This configuration file is checked against a schema for validation.-v,--log-levelFalse, default: infoProvide a valid log level: info, debug, warning, error.Environment VariablesSeeValid Environment Variablesfor more options.Example:exportLOG_LEVEL=debug# using pippython-mbookstack_file_exporter-c<path_to_config_file>Run via DockerDocker images are provided forlinux/amd64andlinux/arm64variants only at the moment. If another variant is required, please request it via Github Issue.TagsUsers will generally want to use thelatesttag or a specific version tag. Themaintag is also provided but is not guaranteed to be stable.tagdescriptionlatestLatest stable release and is updated with each new stable release.X.X.XSemantic versioned releases are also provided if preferred for stability or other reasons.mainThis tag reflects themainbranch of this repository and may not be stableExamples# --user flag to override the uid/gid for created files. Set this to your uid/giddockerrun\--user${USER_ID}:${USER_GID}\-v$(pwd)/config.yml:/export/config/config.yml:ro\-v$(pwd)/bkps:/export/dump\homeylab/bookstack-file-exporter:latestMinimal example with object storage upload. A temporary filesystem will be used so archive will not be persistent locally.dockerrun\-v$(pwd)/config.yml:/export/config/config.yml:ro\homeylab/bookstack-file-exporter:latestEnvironment VariablesSeeValid Environment Variablesfor more options.Tokens and other options can be specified, example:# '-e' flag for env vars# --user flag to override the uid/gid for created files. Set this to your uid/giddockerrun\-eLOG_LEVEL='debug'\-eBOOKSTACK_TOKEN_ID='xyz'\-eBOOKSTACK_TOKEN_SECRET='xyz'\--user1000:1000\-v$(pwd)/config.yml:/export/config/config.yml:ro\-v$(pwd)/bkps:/export/dump\homeylab/bookstack-file-exporter:latestBind Mountspurposestatic docker pathdescriptionexampleconfig/export/config/config.ymlA valid configuration file-v /local/yourpath/config.yml:/export/config/config.yml:rodump/export/dumpDirectory to place exports.This is optional when using remote storage option(s). Omit if you don't need a local copy.-v /local/yourpath/bkps:/export/dumpAuthenticationNote visibility of pages is based on user, so use a user that has access to pages you want to back up.Ref:https://demo.bookstackapp.com/api/docs#authenticationProvide a tokenId and a tokenSecret as environment variables or directly in the configuration file.BOOKSTACK_TOKEN_IDBOOKSTACK_TOKEN_SECRETEnv variables for credentials will take precedence over configuration file options if both are set.For object storage authentication, find the relevant sections further down in their respective sections.ConfigurationEnsureAuthenticationhas been set up beforehand for required credentials.For a simple example to run quickly, refer to the one in theUsing This Applicationsection. A full example is also shown below with descriptions. Optionally, look atexamples/folder of the github repo for more examples.For object storage configuration, find more information in their respective sectionsMinioSchema and values are checked so ensure proper settings are provided. As mentioned, credentials can be specified as environment variables instead if preferred.Full ExampleBelow is an example configuration that shows example values for all possible options.host:"https://bookstack.yourdomain.com"credentials:token_id:""token_secret:""additional_headers:test:"test"test2:"test2"User-Agent:"test-agent"formats:-markdown-html-pdf-plaintextminio:host:"minio.yourdomain.com"access_key:""secret_key:""region:"us-east-1"bucket:"mybucket"path:"bookstack/file_backups"keep_last:5output_path:"bkps/"assets:export_images:trueexport_attachments:truemodify_markdown:falseexport_meta:falseverify_ssl:truekeep_last:5Options and DescriptionsMore descriptions can be found for each section below:Configuration ItemTypeRequiredDescriptionhoststrtrueIfhttp/httpsnot specified in the url, defaults tohttps. Useassets.verify_sslto disable certificate checking.credentialsobjectfalseOptional section where Bookstack tokenId and tokenSecret can be specified. Env variable for credentials may be supplied instead. SeeAuthenticationfor more details.credentials.token_idstrtrueifcredentialsIfcredentialssection is given, this should be a valid tokenIdcredentials.token_secretstrtrueifcredentialsIfcredentialssection is given, this should be a valid tokenSecretadditional_headersobjectfalseOptional section where key/value for pairs can be specified to use in Bookstack http request headers.formatslist<str>trueWhich export formats to use for Bookstack page content. Valid options are:["markdown", "html", "pdf", "plaintext"]output_pathstrfalseOptional (default:cwd) which directory (relative or full path) to place exports. User who runs the command should have access to read/write to this directory. If not provided, will use current run directory by defaultassetsobjectfalseOptional section to export additional assets from pages.assets.export_imagesboolfalseOptional (default:false), export all images for a page to animagedirectory within page directory. SeeBackup Behaviorfor more information on layoutassets.export_attachmentsboolfalseOptional (default:false), export all attachments for a page to anattachmentsdirectory within page directory. SeeBackup Behaviorfor more information on layoutassets.modify_markdownboolfalseOptional (default:false), modify markdown files to replace image links with local exported image paths. This requiresassets.export_imagesto betruein order to work. SeeModify Markdown Filesfor more information.assets.export_metaboolfalseOptional (default:false), export of metadata about the page in a json fileassets.verify_sslboolfalseOptional (default:true), whether or not to check ssl certificates when requesting content from Bookstack hostkeep_lastintfalseOptional (default:None), if exporter can delete older archives. valid values are:- set to-1if you want to delete all archives after each run (useful if you only want to upload to object storage)- set to1+if you want to retain a certain number of archives-0will result in no action doneminioobjectfalseOptionalMinioconfiguration options.Valid Environment VariablesGeneralLOG_LEVEL: default:info. Provide a valid log level: info, debug, warning, error.Bookstack CredentialsBOOKSTACK_TOKEN_IDBOOKSTACK_TOKEN_SECRETMinio CredentialsMINIO_ACCESS_KEYMINIO_SECRET_KEYBackup BehaviorGeneralBackups are exported in.tgzformat and generated based off timestamp. Export names will be in the format:%Y-%m-%d_%H-%M-%S(Year-Month-Day_Hour-Minute-Second).Files are first pulled locally to create the tarball and then can be sent to object storage if needed. Example file name:bookstack_export_2023-09-22_07-19-54.tgz.The exporter can also do housekeeping duties and keep a configured number of archives and delete older ones. Seekeep_lastproperty in theConfigurationsection. Object storage provider configurations include their ownkeep_lastproperty for flexibility.File NamingFor file names,slugnames (from Bookstack API) are used, as such certain characters like!,/will be ignored and spaces replaced from page names/titles. If your page has an emptyslugvalue for some reason (draft that was never fully saved), the exporter will use page name with theslugifyfunction from Django to generate a valid slug. Example:My Page.bin Name!will be converted tomy-page-bin-name.You may also notice some directories (books) and/or files (pages) in the archive have a random string at the end, example -nKA:user-and-group-management-nKA. This is expected and is because there were resources with the same name created in another shelve and bookstack adds a string at the end to ensure uniqueness.Directory LayoutAll sub directories will be created as required during the export process.Shelves --> Books --> Chapters --> Pages

## Example
kafka (shelf)
---> controller (book)
    ---> settings (chapter)
        ---> retention-settings.md (page)
        ---> retention-settings_meta.json
            ...
        ---> compression.html (page)
        ---> compression.pdf
        ---> compression_meta.json
            ...
        ---> optional-config.md (page)
            ...
        ---> main.md (page)
            ...
---> broker (book)
    ---> settings.md (page)
        ...
    ---> deploy.md (page)
        ...
kafka-apps (shelf)
---> schema-registry (book)
    ---> protobuf.md (page)
        ...
    ---> settings.md (page)
        ...

## Example with image and attachment layout
# unassigned dir is used for books with no shelf
unassigned (shelf)
---> test (book)
    ---> images (image_dir)
        ---> test_page (page directory)
            ---> img-001.png
            ---> img-002.png
        ---> rec-page
            ---> img-010.png
            ---> img-020.png
    --> attachments (attachment_dir)
        ---> test_page (page directory)
            ---> something.config
            ---> something_else.config
        ---> rec-page
            ---> test_output.log
            ---> actual_output.log
    ---> test_page.md (page)
            ...
    ---> rec_page (page)
        ---> rec_page.md
        ---> rec_page.pdfAnother example is shown below:## First example:
# programming = shelf
# book = react
# basics = page

bookstack_export_2023-11-28_06-24-25/programming/react/basics.md
bookstack_export_2023-11-28_06-24-25/programming/react/basics.pdf
bookstack_export_2023-11-28_06-24-25/programming/react/images/basics/YKvimage.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/basics/dwwimage.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/basics/NzZimage.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/nextjs/next1.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/nextjs/tips.png
bookstack_export_2023-11-28_06-24-25/programming/react/attachments/nextjs/sample.config
bookstack_export_2023-11-28_06-24-25/programming/react/attachments/nextjs/sample_output.log
bookstack_export_2023-11-28_06-24-25/programming/react/nextjs.md
bookstack_export_2023-11-28_06-24-25/programming/react/nextjs.pdfBooks without a shelf will be put in a shelve folder namedunassigned.Empty/New PagesEmpty/New Pages will be ignored since they have not been modified yet from creation and are empty but also do not have a valid slug.Example from Bookstack API:{
    ...
    "name": "New Page",
    "slug": "",
    ...
}ImagesImages will be dumped in a separate directory,imageswithin the page parent (book/chapter) directory it belongs to. The relative path will be{parent}/images/{page}/{image_name}. As shown earlier:bookstack_export_2023-11-28_06-24-25/programming/react/images/basics/dwwimage.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/basics/NzZimage.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/nextjs/next1.png
bookstack_export_2023-11-28_06-24-25/programming/react/images/nextjs/tips.pngNote you may see old images in your exports. This is because, by default, Bookstack retains images/drawings that are uploaded even if no longer referenced on an active page. Admins can runCleanup Imagesin the Maintenance Settings or viaCLIto remove them.AttachmentsAttachments will be dumped in a separate directory,attachmentswithin the page parent (book/chapter) directory it belongs to. The relative path will be{parent}/attachments/{page}/{attachment_name}. As shown earlier:bookstack_export_2023-11-28_06-24-25/programming/react/attachments/nextjs/sample.config
bookstack_export_2023-11-28_06-24-25/programming/react/attachments/nextjs/sample_package.json
...
...Note attachments that are just external links are ignored. Only attachments that are shown asexternal: Falsewill be exported.Referenceand excerpt from Bookstack API docs:Get a listing of attachments visible to the user. The external property indicates whether the attachment is simple a link. A false value for the external property would indicate a file upload.Modify Markdown FilesTo use this feature,assets.export_imagesshould be set totrueand/orassets.export_attachmentsThe configuration item,assets.modify_markdown, can be set totrueto modify markdown files to replace image and attachment url links with local exported image paths. This feature allows for you to make yourmarkdownexports much more portable.Page (parent) -> Images (Children) relationships are created and then each image/attachment url is replaced with its own respective local export path. Example:## before
[![pool-topology-1.png](https://demo.bookstack/uploads/images/gallery/2023-07/scaled-1680-/pool-topology-1.png)](https://demo.bookstack/uploads/images/gallery/2023-07/pool-topology-1.png)

## after
[![pool-topology-1.png](images/{page_name}/pool-topology-1.png)](https://demo.bookstack/uploads/images/gallery/2023-07/pool-topology-1.png)This allows the image or attachment to be found locally within the export files and allow yourmarkdowndocs to have all the assets display properly like it would normally would.Note: This will work properly if your pages are using the notation used by Bookstack for Markdown image links, example:[![image alt text](Bookstack Markdown image URL link)](anchor/url link)The(anchor/url link)is optional. For attachments the format is:[file](url link)Object StorageOptionally, target(s) can be specified to upload generated archives to a remote location. Supported object storage providers can be found below:MinioMinio BackupsOptionally, look atexamples/minio_config.ymlfolder of the github repo for more examples.AuthenticationCredentials can be specified directly in theminioconfiguration section or as environment variables. If specified in config and env, env variable will take precedence.Environment variables:MINIO_ACCESS_KEYMINIO_SECRET_KEYExampleminio:host:"minio.yourdomain.com"region:"us-east-1"bucket:"mybucket"access_key:""secret_key:""path:"bookstack/file_backups"keep_last:5ConfigurationItemTypeRequiredDescriptionhoststrtrueHostname for minio. A host/ip + port combination is also allowed, example:minio.yourdomain.com:8443regionstrtrueThis is required since minio api appears to require it. Set to the region your bucket resides in, if unsure, tryus-east-1bucketstrtrueBucket to upload toaccess_keystrfalseif specified through env var instead, otherwisetrueAccess key for the minio instancesecret_keystrfalseif specified through env var, otherwisetrueSecret key for the minio instancepathstrfalseOptional, path of the backup to use. Will use root bucket path if not set.<bucket_name>:/<path>/bookstack-<timestamp>.tgzkeep_lastintfalseOptional (default:None), if exporter can delete older archives in minio.- set to1+if you want to retain a certain number of archives-0will result in no action doneFuture ItemsBe able to pull images locally and place in their respective page folders for a more complete file level backup.Include the exporter in a maintained helm chart as an optional deployment. The helm chart ishere.Be able to modify markdown links of images to local exported images in their respective page folders for a more complete file level backup.Be able to pull attachments locally and place in their respective page folders for a more complete file level backup.Export S3 and more options.Filter shelves and books by name - for more targeted backups. Example: you only want to share a book about one topic with an external friend/user.Be able to pull media/photos from 3rd party providers likedrawio