lsegal/yard — Issue #453: Directives (new behavioural tag syntax)
Author: lsegal
Created: 2012-01-12 07:23:44+00:00
State: open
Labels: Feature

## Preface

Currently, tags in YARD can be used to associate meta-data with objects via comments attached to an object in code (known as a docstring). However, tags cannot influence parsing or object behaviour, since they are only data. The goal is to add a new type of tag that compliments (not replaces) tags to contain behaviour, not data, to mutate objects or parsing state.

RDoc already has support for directives, but not for meta-data (not arbitrarily, anyhow). The goal is to fill in the blanks here and provide a similar functionality to RDoc's directives.

## Syntax

The directive syntax will be similar to tags but will have an extra '!' prefix. The format will be:

```
@!directivename DATA ...
INDENTED DATA ...
```

This extra prefix will help differentiate behavioural tags from pure meta-data tags, both for the documentation reader/writers and to YARD. It should be clear when a tag is a directive or pure meta-data.

At this point, we have the opportunity to also support RDoc's `:directive: DATA...` syntax as an optional alternative, but we would still have to port over each builtin directive, which might be quite complex. I'm not sure if it is worth it to tie ourselves to RDoc in this way. Perhaps this is the place for a yard-rdoc compatibility plugin?

## Uses

Currently YARD has a few scenarios handled by tags that can (and should) be replaced by directives:

1. `@group` and `@endgroup` are currently hacked into the parser layer. Using directives, these can be moved into the handlers architecture and be supported without duplication across all languages. The implementation will also be much cleaner. Note that this would require adding a `@!group` and `@!endgroup` syntax and deprecating the old tags.
2. Macros work by using the `@tag` syntax, but they are not actually meta-data tags either. In fact, they are much more like directives. Moving macros over to the directive syntax would yield a much more natural implementation and would not flood the tag namespace with tags that do not actually behave as metadata. There are a number of supporting tags for macros (like `@method` and `@attribute`) that are also directive-like. These would be moved to the `@!` syntax.
3. Functionality like RDoc's `:stopdoc:` and `:startdoc:` could be added without also adding this behaviour directly to the parser

## Implementation Ideas & Preliminary Design

Because tags and directives can co-mingle inside of a docstring, docstring parsing should be modified to handle the `@!syntax`. These tags can be defined in the same way as meta-data tags, by registering them with Tags::Library (potentially via `define_directive`). It might be wise to allow the same tag parsing infrastructure so that developers can define directives that parse types/name/options arguments out of the directive data.

Because directives can modify parsing state, they will require access to the Processor class. One foreseeable design problem with parsing directives inside docstrings is that Docstring#parse_comments (which currently parses tags) has no coupling to the handlers layer (for good reason). This means that parsing of docstrings will likely have to be moved into an intermediary class (like DocstringParser) that can optionally contain a processor. Docstring#parse_comments would instead call DocstringParser to perform parsing with processor=nil. however this might cause directives to be processed twice, once with a processor, once without. Another option would be to have a special parser in the handlers layer that parsed and removed directives prior to being parsed by Docstring. Therefore, Docstring would be oblivious to these tags and ignore them. Parsing of directives would then *only* occur at the handlers layer. This would mean that macro expansion in the form `Docstring.new("@!macro foo")` would not work. **The pros and cons of these two designs should be weighed**. Note that there could be a happy medium between these two design choices.

There will also need to be a change so that unattached comments (comments that are not docstrings / not attached to an object) can also be parsed. This will require inserting Comment statements into the Ruby AST / legacy statement list to represent unattached comments.

## Compatibility Considerations

Since there are a number of tags that will be moved from the `@tag` to the `@!tag` syntax, compatibility must be considered. Given that there are uses of `@macro` and `@group` in the wild, these tags cannot be removed. They will be supported for the long term future, however, will be considered *deprecated* and not recommended for use. Instead, we will recommend documentation authors use the `@!macro` or `@!group` syntax instead. It might be possible to issue warnings in debug mode only.