Package: etherpump
etherpumpPumping text from etherpads into publicationsA command-line utility that extends the multi writing and publishing functionalities of theetherpadby exporting the pads in multiple formats.Many pads, many networksEtherpumpis a friendly fork ofetherdump, a command line tool written byMichael Murtaughthat converts etherpad pages to files. This fork is made out of curiosities in the tool, a wish to study it and shared sparks of enthusiasm to use it in different situations within Varia.Etherpump is a stretched version of etherdump. It is a playground in which we would like to add features to the initial tool that diffuse actions ofdumpingintopumping. So most of all, etherpump is a work-in-progress, exploring potential uses of etherpads to edit, structure and publish various types of content.Added features are:opt-in publishing with the__PUBLISH__magic wordthepublicationcommand, that listens to custom magic words such as__RELEARN__See theChange log / notessection for further changes.Etherpump is a tool that is used from the command line. It pumps all pads of one etherpad installation to a folder, saving them as different text files, such as plain text and HTML. It also creates an index file, that allows one to easily navigate through the list of pads. Etherpump follows a document-driven idea of publishing, which means that it converts pads as database entries into pads as files. This seems to be a redundant act of copying, but is actually an important in-between step that allows for many different publishing projects and experiments.We started to get to know etherpump through various editions of Relearn and/or the worksessions organized by Constant. Collaborative writing on an etherpad has been an important ingredient for these situations. The habit of using pads branched into the day-to-day practice of Varia, where we use etherpads for all sorts of things, ranging from organising remote-meetings with 10+ people, to writing and designing PDF documents collaboratively.After installing etherpump on the Varia server, we collectively decided to not want to publish pads by default. Discussions in the group around the use of etherpads, privacy and ideas of what publishing means, led to a need to have etherpump only start the indexing work after it recognizes a__PUBLISH__marker on a pad. We decided to work on a__PUBLISH__ vs. __NOPUBLISH__branch of etherdump, which we now fork intoetherpump.Change log / notesDecember 2020Added the--magicwordsflag. Parsing and indexing of magic words is now
supported. Seeetherpump.vvvvvvaria.orgfor
more. This is still a work in progress.Change--connectiondefault setting to 50 to avoid overpowering modestly
powered servers.November 2020Releasing Etherpump 0.0.18!Handled a bug that saved the same HTML content in multiple files. Disclaimer: resolved in a hacky way.October 2020Use the more friendly packaging toolPoetryfor publishing.Further performance tweaks, informative logging and miscellaneous bug fixing.Decolonize our Git praxis and use themainbranch.January 2020Added experimentaltrioandaskssupport for thepullcommand which enables pads to be processed concurrently. The default--connectionoption is set to 20 which may overpower the target server. If in
doubt, set this to a lower number (like 5). This functionality is experimental,
be cautious and please report bugs!Removed fancy progress bars for pulling because concurrent processing makes
that hard to track. For now, we simply output whichever padid we're finished
with.October 2019Improveetherpump --helphandling to make it easier for new users.Added thepython-dateutilandpypandocdependenciesAdded a fancy progress bar withtqdmfor long runningetherpump pull --allcallsStarted with theexperimental library API.September 2019Forkingetherpumpintoetherpump.https://git.vvvvvvaria.org/varia/etherpumpMigrating the source code to Python 3.Integrate PyPi publishing with setuptools.May - September 2019etherpump is used to produce theRuminating Relearnsection of the Network Of One's Own 2 (NOOO2) publication.A new command is added to make a web publication, based on the custom magic word__RELEARN__.June 2019Multiple conversations around etherpump emerged during Relearn Curved in Varia, Rotterdam.Including the idea of executable pads (etherhooks), custom magic words, a federated snippet protocol (etherstekje) and more.https://varia.zone/relearn-2019.htmlApril 2019Installation of etherpump on the Varia server.https://etherpump.vvvvvvaria.org/March 2019The__PUBLISH__ vs. __NOPUBLISH__was added to the etherpump repository bydecentral1se.https://gitlab.constantvzw.org/aa/etherpump/issues/3Originally designed for use at:Constant.More notes can be found in thegit repository of etherdump.Install etherpump$ pip install etherpumpEtherpump only supports Python >= 3.6.Command-line example$ mkdir mydump
$ cd myddump
$ etherpump initThe program then interactively asks some questions:Please type the URL of the etherpad (e.g.https://pad.vvvvvvaria.org):https://pad.vvvvvvaria.org/The APIKEY is the contents of the file APIKEY.txt in the etherpad folder.Please paste the APIKEY:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxThe settings are placed in a file called.etherpump/settings.jsonand are used (by default) by future commands.Common WorkflowsText+Meta performance wranglingIf you have a lot of pads, you might want to try the following to speed things
up. This example is something we do at Varia. Firstly, you download all the
pads text + metadata as the only formats. This is likely what you want when
you're trying to work directly with the text. You can do that like so:$etherpumppull--text--meta--publish-opt-inThe key here is to get the--metaso that etherpump is able to read quickly
skip it on the following run if there are no new revisions. So, in practice,
you get a slower first run and faster following runs as more pads are skipped
from actually doing a file system write to save the contents which we already
have.Library API ExampleEtherpump can be used as a library.All commands can be imported and run programmatically.>>>frometherpump.apiimportpull>>>pull(['--text','--meta','--publish-opt-in'])There is also a Magic Word interface. It supports the following API:magic_word(word, fresh)word: The magic word to match pad text against (e.g.__PUB_CLUB__)fresh(default:True): Whether or not run aetherpump pulleach timeHere is an example:frometherpump.apiimportmagic_word@magic_word("__PUB_CLUB__",fresh=False)defpub_club_texts(pads):fornameinpads:print(pads[name]["txt"])pub_club_texts()padsis a dictionary which contains pad names as keys and pad text as values.
Normally, thefresh=Falseis useful when you're hacking away and want to read
pad contents from the local file system and not over the network each time.SubcommandsTo see all available subcommands, run:$ etherpump --helpFor help on each individual subcommand, run:$ etherpump revisionscount --helpPublishingPlease use"semver"conventions for versions.Here are the steps to follow (e.g. for a0.1.3release):Change the version number in theetherpump/__init__.py__VERSION__to0.1.3Change the version number in thepyproject.tomlversionfield to0.1.3git add . && git commit -m "Publish new 0.1.3 version" && git tag 0.1.3 && git push --tagsRunpoetry publish --buildYou should have aPyPiaccount and be added as an owner/maintainer on theetherpump package.TestingIt can be quite handy to run a very temporary local Etherpad instance to test against. This is possible withDocker.$dockerrun-d--nameetherpad-p9001:9001etherpad/etherpad
$dockerexec-tietherpadcatAPIKEY.txt;echoThen you canetherpump initto that local Etherpad for experimentation and testing. You usehttp://localhost:9001as the pad URL.Later on, you can remove the Etherpad with:$dockerrm-f--volumesetherpadMaintenance utilitiesTools to help things stay tidy over time.$makePlease see the following links for further reading:flake8isortblackServer Systers Situation$ sudo -su systers
$ cd /var/www/etherpump
$ sh cron.shServed from/etc/nginx/sites-enabled/etherpump.vvvvvvaria.conf.Keeping track of Etherpad-liteEtherpad-lite API documentationEtherpad-lite releasesLicenseGNU AFFERO GENERAL PUBLIC LICENSE, Version 3.SeeLICENSE.