We should be able to just get rid of `Directive` completely and just use the annotations directly. (Maybe next CL?)