Package: orgmunge
#+title:Orgmunge* Motivation and scopeOrgmunge was born out of the desire to modify Org documentsprogrammatically from within Python. The wonderful [[https://github.com/karlicoss/orgparse][orgparse]] can readan Org document into a tree object but doesn't offer an interface tomodify the tree and write it back to file.The original use case was trying to sync Outlook calendar items withOrg: whenever someone rescheduled a meeting, my Python script wasunable to reschedule the Org heading it had originallycreated. Instead of forking =orgparse=, I decided to write an actualgrammar for an Org document and use [[https://github.com/dabeaz/ply][PLY]] to generate a parser for it.Now Org syntax is too sophisticated for me to claim that this firstattempt can parse everything. In fact, some folks way smarter than Iam (and with more formal training), have hinted that Orgsyntax can't be properly parsed with a context-free grammar. For suchreasons (and for my own lack of experience with writing grammars), Ihave restricted the scope of this module to the features I care about:for each heading, the headline components (the =COMMENT= keyword, thetodo state, priority, cookies, and tags) are all parsed, as well asany scheduling timestamps and all the drawers. The heading contentsare treated as a blob of text and the only thing the parser extractsfrom the contents are the timestamps. No attempts are made at parsingthings like tables or source code blocks further. =orgmunge= can alsoparse out the document's metadata and export options but the majorassumption it makes is that the document starts out with some optionalmetadata and export options, followed by some optional initial bodytext (not falling under any heading), and then a tree of headings. Anyexport options or metadata that come later within the document aretreated as text (some heading's content).* Use CasesIf you have built something on top of =orgmunge=, please open an issuehere and I'm happy to add your project to the use cases.** Redact an Org fileReplace important information in an Org file with random words inorder to share the structure of the file with someone withoutcompromising your information. See [[https://github.com/durableOne/redactOrg][redactOrg]]* Installation** From PyPi- =orgmunge= is now on PyPi- You can install =orgmunge= using =pip=:#+begin_src shellpython3 -m pip install orgmunge#+end_src** From the repo- The only dependency of =orgmunge= is =PLY=. So you need =PLY= installed.- Clone this repo- Add the directory where you cloned this repo to your =PYTHONPATH=* Usage** Specifying TODO keywords- The parser needs to know the set of valid keywords before it startsparsing your input. To do this, it uses the following steps1. If your input string/file contains [[https://orgmode.org/manual/Per_002dfile-keywords.html][per-file keywords]], these willtake precedence over anything else2. Failing to find any such keywords, it looks to see if you passedit the keywords using the =todos= argument3. If no todo keywords were passed, the parser looks for todo keywords by looking for a file named=todos.json= in one of 2 places (again in order of preference):1) The current directory2) The user's home directory4. Failing all the above, the keywords are assumed to be defined by:#+begin_src javascript{"todo_states":{"todo": "TODO","next": "NEXT","wait": "WAIT"},"done_states":{"cncl": "CNCL","done": "DONE"}}#+end_src- If you choose to supply your own keywords as an argument to theparser, you must follow the above structure: separate =todo_states=and =done_states= with pairs of =keyword_nickname: keyword= specifyingeach set of states.** Reading an Org tree- The =Org= class in =__init__.py= is the main entry point to =orgmunge=.It can be used to read an Org tree either from a string or from afile:#+begin_src pythonfrom orgmunge import Orgorg_1 = Org('* TODO Something important\n', from_file=False) # \n needed to signify end of documentorg_2 = Org('/path/to/my/file.org')org_3 = Org('/path/to/my/file.org', debug=True) # Print PLY debugging info#+end_src- The =Org= object has 3 main attributes you should care about:1. =Org.metadata= stores the metadata and export options found at thebeginning of the file. This is a dict mapping the option/keywordname to a list of its values (to allow for cumulative keywordssuch as =#+OPTION=). Example:#+begin_src pythonorg_1 = Org('#+title: Test\n')assert(org_1.metadata['title'] == ['Test'])#+end_src2. =Org.initial_body= stores any text between the metadata and thefirst heading.3. =Org.root= stores the root of the Org tree. This is a [[*Heading Objects][heading]] withthe [[*Headline Objects][headline]] =ROOT= whose only useful attribute is =children=, which is alist of all the [[*Heading Objects][headings]] in the given document.- The Org tree is a list of [[file:__init__.py::def _classify_headings(self, lst):][headings]] with parent, child and sibling relationships.*** Heading Objects- A heading object consists of:1. A [[*Headline Objects][headline]]2. Contents:1) [[*Scheduling Objects][Scheduling]], if any2) A list of [[*Drawer Objects][Drawers]], if any3) Body text, if any- Important attributes:1. =properties=. This is a dict mapping property names to theirvalues. The properties are parsed from the =PROPERTIES= drawer ifit exists. This attribute can also be set by the user (the valuesupplied must be a dict).2. =headline= returns the heading's headline. This attribute can alsobe set by a user (the value must be a [[*Headline Objects][Headline]] instance).3. =scheduling= is a [[*Scheduling Objects][Scheduling]] object containing information about=SCHEDULED/DEADLINE/CLOSED= timestamps of the heading, if any. Canalso be set by the user (the value must be a Scheduling instance).4. =drawers= is a list of [[*Drawer Objects][Drawer]] objects containing the drawersassociated with this heading. When you update the heading's=properties= attribute, the =PROPERTIES= drawer is updated the nexttime you access it.5. =children= returns a list of Heading objects that are the directchildren of this heading.6. =parent= returns the parent heading of the current one. If thecurrent heading is a top-level heading, the root heading will bereturned.7. =sibling= returns the sibling heading of the current one that comesbefore it in the tree, if any. The reason this is the siblingheading that is formally tracked is because it's the one thatwould adopt the current heading whenever the current heading isdemoted. If you want a list of all siblings of the currentheading, you can do this:#+begin_src pythonsiblings = [c for c in current_heading.parent.children if c is not current_heading]#+end_src8. =level= is the heading's level, with 1 being the top level and eachsub-level after that being incremented by 1 (the heading's levelis the number of "stars" before its headline).- Important methods:1. =clocking=. This returns a list of [[*Clocking Objects][Clocking]] objects, parsedfrom the heading's =LOGBOOK= drawer, if any. You can also pass theoptional boolean parameter =include_children=, which, when True,includes clocking information of this heading's children as well.2. =add_child= accepts a Heading object to add as a child to thecurrent heading. The optional boolean parameter =new= should be setto =True= when this is a new heading that was created and needs tobe assigned a parent. It should be set to =False= (default) whenthe addition of a child is due to a promotion/demotion operation.3. =remove_child= accepts a heading object and deletes it from thecurrent heading's children if it's a child of the currentheading.4. =promote= promotes the current heading one level. If the heading haschildren, they would be orphaned so this raises a=ValueError=. Technically, Org allows you to have, say, level 3headings under a level 1 heading, but =orgmunge= does not allowthis to make parsing the tree easier.5. =promote_tree= promotes the current heading and all itsdescendants. Use this if the heading you want to promote haschildren.6. =demote= demotes the current heading one level. If the currentheading has no sibling to adopt it, the demotion attempt failsand raises a =ValueError=.7. =demote_tree= is the equivalent of =promote_tree= for demotion.*** Headline Objects- Important attributes:1. =done= is a boolean attribute that determines whether the headlineis in one of the done states. You can't set this attribute directly.2. =level= is the headline's level (the number of "stars" before thetitle)3. =comment= is a boolean attribute that determines whether a headlineis commented out (by having the keyword =COMMENT= inserted beforethe title).4. =todo= returns/sets the headline's todo state. You can set ityourself but it has to be one of the values of =self._todo_states=or =self._done_states=.5. =cookie= returns/sets the headline's cookie. See [[*Cookie Objects][Cookie Objects]].6. =priority= returns/sets the headline's priority- Important methods:1. =promote= decreases the level by the number given by the parameter=n= (default 1).2. =demote= acts like =promote= but increases the level by =n= instead.3. =toggle_comment= toggles the state of whether or not a headline iscommented out using the =COMMENT= keyword.1. =comment_out= ensures the headline is commented out using=COMMENT=2. =uncomment= ensures the headline is not commented out using the=COMMENT= keyword.3. =raise_priority= increases the headline's priority by 14. =lower_priority= decreases the headline's priority by 1*** Scheduling Objects- Has 6 attributes for the 3 possible scheduling keywords (3 are aliases of the other 3):1. CLOSED, closed2. SCHEDULED, scheduled3. DEADLINE, deadline- Each attribute, when queried will return either =None= or a =TimeStamp=object representing the timestamp associated with this particularscheduling keyword. You can set the attributes directly but theyhave to be set to a =TimeStamp= object.*** Drawer Objects- A =Drawer= object has only 2 attributes: =name= and =contents=. The=contents= attribute is simply a list of lines making up the drawercontents. When you modify a heading's =properties= attribute, its=PROPERTIES= drawer gets updated accordingly.*** Clocking Objects- The =Clocking= objects have 3 attributes: =start_time=, =end_time= and=duration=. Only the first 2 can be set. When setting either, youshould pass a string following the Org time format; namely,'%Y-%m-%d %a %H:%M' (see the [[https://man7.org/linux/man-pages/man3/strftime.3.html][strftime(3)]] man page for an explanationof the format codes).- If =end_time= is =None=, the duration is calculated from the =start_time=up to the current moment.*** Priority Objects- The only attribute, =priority= can be set directly by the user and canbe one of only 3 strings: 'A', 'B' or 'C'. Set to =None= to remove itfrom the =Heading=.- The methods =_raise= and =_lower= will raise or lower the priority.- If the priority is =None=, raising it, sets it to 'A' and lowering itsets it to 'C'.*** TimeStamp Objects- Important attributes:1. =start_time= and =end_time= can be queried and set by the user. Youcan set them by supplying a string, a =datetime= object or =None=.2. =repeater= returns a timestamp repeater string such as '+1w'. Canalso be set by the user.3. =deadline_warn= acts similarly to =repeater= and represents thenumber of days before a deadline to warn the user of an upcomingdeadline.4. =active= is a boolean property and decides whether the time stampwill be printed with =[]= or =<>= delimiters. Can be set directly bythe user.*** Cookie Objects- =Cookie= objects represent progress on the current =Heading=.- They can be of type 'percent' (e.g. [50%]) or of type 'progress' (e.g. [2/4]).- Important attributes:1. =cookie_type=: can only be one of 'percent' or 'progress'. Can beset directly by the user.2. =m= and =n= represent the progress as the ratio =m/n=. If the cookietype is 'percent', =n= is 100. When changing =cookie_type=, =m= and =n=are converted accordingly.** Modifying an Org tree- The ability to modify the tree was the main reason I wrote thispackage. Most of the attributes of the tree objects can be modifieddirectly by the user.- Use the =promote*= and =demote*= methods of the =Heading= objects tochange =Heading= levels.- To rearrange headings, note that a =Heading's= =children=attribute is a list whose ordering is important: in other words, thetree will be written back to a file with the order each =Heading='schildren are in. So the user can rearrange the headings of the same levelby assigning the =children= attribute of their parent to a differentorder of child headings. It's up to the user to update the childheadings' =sibling= attributes appropriately.** Writing an Org tree- You can use the =Org= object's =write= method to write out the tree to afile whose name you supply to the method:#+begin_src pythonfrom orgmunge import Orgagenda = Org('/path/to/agenda.org')# Do something with agenda...agenda.write('/path/to/modified_agenda.org')#+end_src** Convenience Methods*** Getting All HeadingsThe convenience method =Org.get_all_headings= walks the Org treedepth-first and returns a generator of all the headings in the tree inthe order in which they occur.*** Filtering HeadingsYou can use =Org.filter_headings(func)= where =func= is any arbitrary[[https://en.wikipedia.org/wiki/Boolean-valued_function][predicate]] and get a generator of all headings satisfying the predicate.*** Search for Headings by TitleUse =Org.get_headings_by_title= to search for a heading with the given title:#+begin_src pythonOrg.get_headings_by_title(search_string, exact=False, re_flags=0)#+end_src=search_string= is what's searched in the title. It's interpreted as aregex unless =exact= is set to =True=, in which case, the function willreturn headings whose title matches the search stringexactly. =re_flags= are flags passed to =re.search=. This argument isignored if =exact= is =True=.Uses =filter_headings= under the hood so will return a generator ofmatching headings.* License#+INCLUDE: ./LICENSE* Contributors:PROPERTIES::ID:       d27f8cd9-4be9-4fa3-b54a-40b9d3807e90:END::LOGBOOK:CLOCK: [2023-07-08 Sat 15:54]--[2023-07-08 Sat 15:54] =>  0:00:END:Thanks to these wonderful people for contributing time and code:- [[https://github.com/Nalisarc][Nalisarc]]- [[https://github.com/ispringle][ispringle]]- [[https://github.com/crdoconnor][crdoconnor]]