The Stata Journal (2016)
16, Number 4, pp. 938–963

Rethinking literate programming in statistics
E. F. Haghish
Center for Medical Biometry and Medical Informatics
University of Freiburg
Freiburg, Germany
andDepartment of Mathematics and Computer Science
University of Odense
Odense, Denmarkhaghish@imada.sdu.dk
Abstract. Literate programming is becoming increasingly trendy for data analysis because it allows the generation of dynamic-analysis reports for communicating
data analysis and eliminates untraceable human errors in analysis reports. Traditionally, literate programming includes separate processes for compiling the codeand preparing the documentation. While this workﬂow might be satisfactory forsoftware documentation, it is not ideal for writing statistical analysis reports. Instead, these processes should run in parallel. In this article, I introduce the weaver
package, which examines this idea by creating a new log system in HTML or LATEX
that can be used simultaneously with the Stata log system. The new log systemprovides many features that the Stata log system lacks; for example, it can rendermathematical notations, insert ﬁgures, create publication-ready dynamic tables,and style text, and it includes a built-in syntax highlighter. The weaver package
also produces dynamic PDF documents by converting the HTML log to PDF or bytypesetting the LATEX log and thus provides a real-time preview of the document
without recompiling the code. I also discuss potential applications of the weaver
package.
Keywords: pr0063, weave, div, img, txt, tbl, weaver, weaving, literate programming, dynamic documents, reproducible research, log, HTML, LATEX

1Introduction

Statistical analyses are carried out to draw conclusions from data, gain knowledge, andcommunicate claims with the scientiﬁc community. However, the communication aspectof statistics is often ignored in statistical software.
Traditionally, the analysis report is written manually using a word processor suchas Microsoft Word by dragging and dropping ﬁgures, creating tables, and copying andpasting output from the statistical software into the document. This workﬂow is unreproducible and becomes more prone to human errors as the complexity of the reportincreases. More importantly, when ﬁgures and output are copied and pasted into theanalysis report, they become disconnected from the code that produced them, potentially leading to more human errors that are not traceable. Automating the writing ofanalysis documents can save a lot of time and human labor, reduce the chance of humanerror in writing the report, and make the report reproducible.
c 2016 StataCorp LP


pr0063

E. F. Haghish939

The common approach for creating automated analysis reports in statistics is “literate programming”, a programming paradigm introduced by Knuth (1983, 1984, 1992).
In literate programming, the documentation is written within the source ﬁles and is separated from the programming code with a special notation. Next, a literate-programming
software (for which the special notation is written) parses the string notations to separate the programming code from the documentation and continues to either compile
the software or prepare the documentation. Knuth names the process of compiling thesoftware “tangle” and the process of preparing the documentation “weave”.
Software engineers advocate literate programming because it eases the eﬀort needed
to update the software documentation and it makes the source more comprehensible formaintenance programmers. In addition, it preserves the connection between the codeand the documentation (Sametinger and Pomberger 1992; Ramsey and Marceau 1991;
Cordes and Brown 1991), although its applications are not merely limited to software
documentation (Shum and Cook 1994; Ruys and Brinksma 1998).
Literate programming is becoming a popular way to document data analysis andgenerate analysis reports, with literate-programming software that is independent of
the target programming language, such as noweb (Johnson and Johnson 1997; Ramsey1994) or any literate-programming software developed especially for producing dataanalysis reports (Murdoch and Carey 2001; Xie 2016, 2014; Leisch 2002; Allaire et al.
2016; Lenth 2012). For Stata users, markdoc1 (Haghish 2014a) is a convenient generalpurpose literate-programming package that supports a variety of markup languages
and document formats. This software package is inspired by Knuth’s (1983) literateprogramming software called WEB and thus applies a special notation for separating the
documentation text from the data-analysis code in the source ﬁles.
However, there is a major diﬀerence between software development and statistics
when it comes to code communication. Computer programs are primarily written to
be compiled by computers, and literate programming is the preferred method for software documentation. In contrast, the cornerstone of statistics lies in communicating
the statistical analysis and interpreting the results. For example, while software documentation is presumably optional, interpreting the statistical results is the natural stepafter any data analysis. Moreover, data analysis is often carried out interactively byexecuting consequent commands, and as a result, a dynamic report should also be developed gradually and updated automatically with every new executed command. In thisway, a user can obtain a live preview of his or her analysis report as the data analysis
develops.
In contrast to software development, literate-programming packages in statistics
should compile the analysis code either way because the analysis report must include
output and ﬁgures generated from the statistics software. In other words, the code
must be compiled whether the user wishes to create an analysis report or just carryout the data analysis without producing documentation. Thus, the question is why
statisticians insist on executing tangle and weave independently when these are better
1. markdoc is hosted on https://github.com/haghish/markdoc.

940

Rethinking literate programming in statistics

run in parallel to save on execution time and provide support for interactive analysis
report development.
To examine this idea, I developed the weaver package for Stata. It can be imagined
as a new kind of analysis log system with distinctive behavior—that I refer to as weaver
log—that can run in parallel with Stata Markup and Control Language (SMCL) or text
log system. Because the weaver log system can produce HTML and LATEX documents,
I will refer to a particular format as HTML log or LATEX log, and to avoid confusion
with the native log system of Stata, I will refer to the latter as Stata log or SMCL log.
The weaver log system includes a variety of features one could expect from a softwarethat generates analysis reports. I present these features in the current article as well
as discuss potential applications of weaver and how it can be better implemented in
Stata.

2

The weaver package

2.1

The main idea

The weaver log system can be used simultaneously with SMCL or text log system, but
for a very diﬀerent purpose. From the time the SMCL log is opened, it documents every
action that takes place during the analysis session, including commands, text-based
output, and also errors. Therefore, it is a reliable way of saving the history of the dataanalysis session. In contrast, the weaver log is not autonomous and remains indiﬀerent
to all Stata commands and outputs. It does not automatically register anything unlessthe user commands it otherwise. Such behavior allows the user to select what text,
ﬁgures, commands, and output should be included in the weaver log ﬁle while the
SMCL log documents the analysis in parallel.

2.2

Features

The weaver package supports HTML and LATEX for creating dynamic-analysis reports,
without requiring any knowledge of these markup languages. weaver includes a simpliﬁed JavaScript-based markup language called weaver markup that allows writing
text headings and paragraphs, styling text, changing text color, highlighting text, andadding hypertext links in the HTML log (see section 4.1). Moreover, weaver also comeswith a syntax highlighter for Stata code in the HTML and LATEX dynamic document.
For conveniently writing dynamic text, adding ﬁgures, and creating dynamic tables inboth the HTML and the LATEX log ﬁles, weaver includes three specialized commands.
Additionally, weaver supports rendering mathematical notations in HTML and LATEX
log ﬁles. Finally, both the HTML or LATEX log and the rendered PDF document can bepreviewed in real time while developing the dynamic report.

E. F. Haghish

2.3

941

Installation

weaver requires the statax (Haghish 2015) package, a syntax highlighter for HTML and
LATEX dynamic documents. Both packages are hosted on GitHub website. The github2(Haghish 2016a) command can install weaver and its dependency as shown below.
. github install haghish/weaver

In addition, the HTML log requires some third-party software: wkhtmltopdf and
MathJax. The wkhtmltopdf freeware provides command-line tools for converting HTMLto PDF and accurately renders CSS and JavaScript to PDF. MathJax is a JavaScript
engine for rendering mathematical notations in the HTML log ﬁle. Both MathJax andwkhtmltopdf are open-source freeware, available for Microsoft Windows, Mac OS X,
and Linux operating systems. Naturally, for typesetting a PDF document from the
LATEX log ﬁle, you should install a LATEX distribution on your machine. To use weaver
without any third-party software installation, include the nopdf option to cancel theprocess of rendering a PDF document. This option will avoid errors regarding MathJax, wkhtmltopdf, and pdfLaTeX but will still produce an HTML or a LATEX dynamic
document.
The GitHub repository hosts the ado-ﬁle and help ﬁles. The required third-party
software should be installed manually. To make the installation process more convenient,
weaver provides an optional automatic installation for MathJax and wkhtmltopdf. Bothmanual and automatic installation of the required software are described below.
Manual installation
Manual installation of wkhtmltopdf is straightforward. The software can be downloaded
from http://wkhtmltopdf.org and installed in any location on your machine. The pathto the executable wkhtmltopdf should be given to weaver by using the printer(string)
option. Similarly, to typeset the LATEX log to PDF, the proper LATEX distribution should
be downloaded from https://latex-project.org and the path to the executable pdfLaTeX
should be provided using the printer(string) option. The manual installation of MathJax is easier. The MathJax-master.zip ﬁle can be downloaded from http://mathjax.org
and unzipped in the ado\plus\Weaver\ directory (see the technical note below).
As an alternative to specifying the printer(string) option, users can set up thedefault ﬁle paths for PDF drivers or install MathJax in the Weaver directory permanently. The weave setup command opens weaversetup.ado in the Do-ﬁle Editor. It
deﬁnes global macros for memorizing the full paths to the executable wkhtmltopdf, theexecutable pdfLaTeX, and the MathJax master directory for future use. Note thatweaversetup.ado is not included in the package and is created the ﬁrst time the weave
setup command is executed.

2. To install the github command, typenet install github, from("https://haghish.github.io/github/").

942

Rethinking literate programming in statistics

Automatic installation
When opening a weaver log, the install option can be used to download wkhtmltopdf
and MathJax automatically and install them in the Weaver directory (see the technicalnote below). The automatic installation of the required software was successfully tested
on Mac OS X (10.9, 10.10, 10.11); 32-bit and 64-bit versions of Microsoft Windows (XP,7, 8); and 64-bit versions of Linux Ubuntu 14.04.4 LTS, Mint 17, and CentOS 7. However,manual installation is generally recommended, because it ensures the installation of thelatest version of the software.
Technical note
weaver creates a directory named Weaver in the plus directory to store the required
software. The path to the ado\plus\ directory can be found using the sysdir command,
which returns all the system directories. In Stata 13 and 14, the likely paths of the
Weaver directory on Microsoft Windows, Mac OS X, and Linux operating systems areshown below.
Windows: C:\ado\plus\Weaver
Mac OS X: ~/Library/Application Support/Stata/ado/plus/Weaver
Linux: /home/username/ado/plus/Weaver

3

Syntax and examples

weaver includes two sets of commands. The commands that begin with weave are
related to handling the weaver log, for example, opening a new log ﬁle, merging existing
HTML or LATEX documents into the current log, exporting a live preview PDF document,
closing the log, or temporarily turning the log on or oﬀ. The other commands areused for annotating the document, that is, inserting a ﬁgure, creating a dynamic table,echoing Stata commands and outputs, and writing and styling text in the log ﬁle.

3.1

The weave commands

The commands related to managing the weaver log ﬁle begin with the weave command
and are summarized in table 1. The syntax of these commands is similar to the Statalog command syntax.

E. F. Haghish

943
Table 1. The weave commands

Command

weave using ﬁlename , options
weave merge
 using ﬁlename
weave pdf , replace
weave close
weave query
weave {off | on}
weave {preserve | restore}
weave setupDescription
create a new log
merge a ﬁle into the log
create a PDF preview of the log ﬁleclose the log and create the PDF
report the status of the log ﬁleturn the log oﬀ or on
preserve and restore the log status
set the ﬁle path to the required software

Opening a new log
The dynamic report begins with the command weave using ﬁlename, which creates a
new log ﬁle. This log stays on until it is closed, turned oﬀ, or Stata is shut down. The
overall syntax for opening a log ﬁle is shown below.
weave using ﬁlename, install nopdf printer(string) markup(name){replace | append} papersize(name) landscape margin(int,int,int,int)
style(name) template(ﬁlename) toc font(string) title(string)author(string) affiliation(string) address(string) summary(string) date
synoff noisily
The weave using ﬁlename command accepts several options that determine thegeneral format of the log and the PDF dynamic report, such as paper size, margins,
and styles that can change the overall appearance of the document. These options arefurther described below.
install automatically downloads and installs the wkhtmltopdf and MathJax software
if they are not located in the Weaver directory or cannot be accessed by weaver.
nopdf avoids rendering PDF and mathematical notations (in HTML log). This option
allows the user to use weaver HTML or LATEX dynamic reports without installing
any third-party software.
printer(string) deﬁnes the ﬁle path of the executable wkhtmltopdf software or of
pdfLaTeX on your machine.
markup(name) deﬁnes the log type as html or latex. If the ﬁlename already includes
a ﬁle extension (that is, .html or .tex), then weaver will recognize the markup
language and this option is unnecessary. The default is markup(html).

944

Rethinking literate programming in statistics

replace overwrites an existing weaver log and PDF ﬁles with the identical name.
append speciﬁes that results be appended to an existing HTML or LATEX log ﬁle.
papersize(name) changes the default page size of the PDF document. The default
is papersize(A4), but weaver supports most of the widely used page sizes forgenerating printer-ready PDF documents from both the HTML and the LATEX log
ﬁles. The supported paper sizes are A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, B0, B1,
B2, B3, B4, B5, B6, B7, B8, B9, B10, C5E, Comm10E, DLE, Executive, Folio, Ledger,
Legal, Letter, and Tabloid. These names may also be speciﬁed in lowercase. In
addition, the weave setup command may be used to permanently change the default
paper size.
landscape ﬂips the page to landscape mode.
margin(int,int,int,int) changes the default margins of the PDF document and canrange between 3 to 50 millimeters. Each int indicates a page margin, beginning withthe top margin and moving clockwise; from left to right, the integers indicate the
top margin, right margin, bottom margin, and left margin.
style(name) changes the theme style of the weaver log. The available styles arestata (the default), classic, modern, elegant, minimal, and empty. The empty
style creates an empty log for HTML or LATEX to allow advanced users to manuallycustomize the dynamic document.
template(ﬁlename) imports an external CSS ﬁle for the HTML log or the heading of a
LATEX document for the LATEX log to overrule the existing style sheet of the dynamic
report.
toc adds an automatic table of contents and creates a reference link to the corresponding
headings. The table of contents will only appear in the PDF documents.
font(string) speciﬁes the font for all headings, subheadings, and paragraphs.
title(string) displays the title of the document on the title page.
author(string) displays the author’s name on the title page.
affiliation(string) displays the author’s aﬃliation (or any preferred relevant information) on the title page.
address(string) displays the author’s contact information (or any preferred relevant
information) on the title page.
summary(string) displays a summary or abstract on the title page.
date displays the current date on the title page.
synoff turns the statax syntax highlighter oﬀ in the dynamic document.
noisily displays the command’s output in the Results window.
As noted, weaver can create HTML or LATEX dynamic reports. The syntax for
creating a LATEX log is similar to that for HTML, however, the simpliﬁed weaver markup

E. F. Haghish

945(see section 4.1) is not supported in LATEX because it is written in JavaScript. Because
writing with weaver markup is simpler and more intuitive than LATEX, most of theexamples in this article are written for an HTML log.
Exampleset more off
weave using example, install markup(html) replace datetitle("Opening HTML Log") author("E. F. Haghish")affiliation("University of Freiburg")summary("This is an example of opening an HTML log in the ""[mono]weaver[#] package. The following examples should be ""run after opening the HTML log. Note that [mono]weaver[#] ""markup codes may be used wherever to annotate the document. ""However, the examples cover only the basic features of the ""[mono]weaver[#] package. All the examples in the current ""section may be downloaded from ""[-- https://github.com/haghish/weaver --][-GitHub-].")

///
///
///
///
///
///
///
///
///
///

Figure 1. HTML preview of the weave using example

Creating a PDF preview of the dynamic report
The weaver log ﬁle is the building block of the dynamic report in the weaver package.
However, the HTML log ﬁle is only interesting for sharing analysis on the web, andmost users are probably interested in PDF documents, especially if the log is written
in LATEX. The PDF document is more practical than HTML or LATEX because it is a

946

Rethinking literate programming in statistics

single ﬁle that includes the report and the graphical ﬁgures; thus, it is highly portableand easily sharable. Therefore, it is important to see how the document looks in PDF
format during the data analysis, without having to rerun whole codes to compile a PDF
document. The command below may be used anytime during the data-analysis session
to view the PDF document.
weave pdf

, replaceThis command will generate a dynamic PDF document by converting the HTML log
to PDF format using wkhtmltopdf or by converting the LATEX log to PDF format using
pdfLaTeX. The ﬁnal dynamic report is produced by weave close, which closes the log
ﬁle and renders the dynamic report.
Merging a ﬁle to the weaver log
Plenty of Stata packages can produce HTML and LATEX documents, such as texdoc (Jann
2016), markdoc (Haghish 2014a), synlight (Haghish 2014b), estout (Jann 2004), andtabout (Watson 2004). weaver can merge any web-based ﬁle or LATEX document to theHTML or LATEX log, respectively. The syntax for merging a document into the log is asfollows:
weave merge using ﬁlename
Both HTML and LATEX documents have a straightforward structure, beginning witha heading followed by the body of the document, which includes the main content of
the ﬁle. Therefore, merging two documents is not as simple as appending one ﬁle toanother because the template of the merged ﬁle can interfere with weaver’s template(in HTML) or corrupt the log (in LATEX).
weaver processes the given ﬁle and only merges the main body of the document toavoid any damage to the log ﬁle. Content that is omitted in the merging process will be
included in the log ﬁle as comments (which only can be seen in the source of the HTMLand LATEX logs), and the user will be notiﬁed about the ignored content.
Restoring the weaver log
When writing a dynamic document interactively, you might wish to undo an analysis
or remove a Stata outcome from the document. By using weave preserve and weave
restore, you can preserve the status of an HTML or a LATEX log and later restore thelog to undo changes. The commands do not, however, preserve or restore the data.

3.2

Annotating commands

The annotating commands, as the name suggests, are used for writing the dynamic report, adding mathematical notation, inserting Stata code and output, importing graphs,

E. F. Haghish

947

and creating dynamic tables. Table 2 lists the annotating commands with a brief description of each.
Table 2. The annotating commands
Command
div code | result command
txt code display directive
img using ﬁlename



  

tbl (* ,*...
\ * ,*...
\ ...
)

3.3

Description
print code, output, or bothin the log
print text, append code
insert a graphcreate a dynamic table

The div command

By default, weaver does not include any of the Stata commands and output run fromthe do-ﬁle or typed in the Command window. This allows you to perform exploratory
analysis without worrying that the dynamic report will contain output that is not essential or interesting. However, both Stata code and output can be echoed in the weaver
log by using the div command placed before any Stata command. The syntax of the
div command is shown below.
divcode | resultcommand

The code or result subcommand may be added to div to echo only the Stata
command or the output, respectively. These subcommands would be used when you
intend to include a chunk of code or only the output of a command in the log, making
the document less technical and more formal.
ExampleTo demonstrate the div command, we load auto.dta and list the ﬁrst ﬁve observations. Because the div command is not speciﬁed, weaver will not include them in the
document. Nevertheless, the output will be shown in the Results window and SMCL log
ﬁle, if it has been opened. The next three commands, which begin with the div command, allow Stata to communicate with the weaver log. Figure 2 shows the contentsof the weaver log after all ﬁve commands.

948

Rethinking literate programming in statistics
sysuse auto, clear
list in 1/5
div misstable summarize
div summarize price mpg
div tabulate foreign

Figure 2. Preview of the weaver log

The code subcommand modiﬁes the div command to echo only the command andnot any resulting output. In the following example, the code subcommand is usedsuccessively to create a code block as shown in ﬁgure 3.
div code misstable summarize
div code summarize price mpg
div code tabulate foreign

Figure 3. The weaver log after div code

E. F. Haghish

949

To echo only the Stata output in the weaver log, use the result subcommand, as
shown below; see ﬁgure 4.
div result regress price mpg

Figure 4. The weaver log after div result

3.4

The img command

Software that generates dynamic reports should handle inserting images and graphs inthe document with great care. With weaver, you can resize and style graphs in the
document, and you can also add a text description by using the title(string) option.
The img command has two options for inserting ﬁgures in the weaver log. The syntaxof the img command is shown below.
using ﬁlename
{left | center}

img

 

, title(string) width(int) height(int)In the ﬁrst option, if img using ﬁlename is speciﬁed, img adds the markup code
required for rendering the graphical ﬁle in the HTML or LATEX log. In this procedure,
the Stata ﬁgures should be ﬁrst exported and then imported in the weaver log. For an
HTML log, the graph should be in a format recognizable by web browsers (for example,
PNG, JPEG, or GIF). In addition to graphical formats supported in HTML, the LATEX log
also supports PDF ﬁgures without requiring the loading of an additional package.
The second option for importing a ﬁgure from Stata requires less coding. As shownin the syntax, specifying using ﬁlename is optional. If the ﬁlename is not speciﬁed,
the img command automatically exports the current graph from Stata to PNG format
in a directory named Weaver-figure, located in the same directory as the weaver log.
Therefore, the img command can be executed right after generating a graph in Stata toautomatically export it and import it into the dynamic document.
The width(int) and height(int) options allow the user to resize an image in the
document without changing the actual size of the image. The title(string) option canbe used to write a description about the ﬁgure, and the left and center options align950

Rethinking literate programming in statistics

the image to the left side or to the center of the document, respectively. The example
below demonstrates how to insert a graph with the img command; see ﬁgure 5.
Examplehistogram price, title("Price histogram")
img

01.0e−04

Density2.0e−043.0e−04

Price histogram

0

5,000

10,00015,000Price

Figure 5. Result of the img command

3.5

The txt command

The txt command acts similarly to the display command, and it can be used to display
dynamic text in weaver. Dynamic text includes scalars and macros, and thus its content
can vary. The syntax of this command is as follows:
txt



code



display directive

display directive may be a double-quoted or compound double-quoted string, a display format (for example, %5.2f), skip(#), column(#), newline(#), or dup(#).
By default, the txt command prints a text paragraph if no markup language is used toalter the documentation. The optional code suﬃx allows writing unformatted text on

E. F. Haghish

951

the weaver log, which is only useful for advanced users who wish to write HTML codeor paste a chunk of HTML code or a script in the HTML log. code does not change thefunction of the txt command in the LATEX log.
As shown in the example below (see ﬁgure 6), in the dynamic document the txt
command automatically displays scalars with two decimal digits, which is common inscientiﬁc writing. A display format value can be used to alter the way the scalar value
is displayed.
Examplequietly summarize pricetxt "*- Writing dynamic text with the txt command -*"///"The [mono]txt[#] command can be used for writing dynamic text, "///"(that is, displaying macros in the text paragraph). For example, " ///"the [mono]price[#] variable in [mono]auto.dta[#] includes "///"#_`r(N)´_# observations with a mean of #_" r(mean) "_# and a "///"standard deviation of #_" r(sd) "_#."
txt [box] Note that [mono]weaver[#] markup codes can be appliedin the [mono]txt[#] command to style text. For example, thetext color can be changed to [blue] blue[#], [red]red[#], or anyof the other supported colors. Similarly, text can be[-yellow]highlighted[#] in variety of colors.[#]

///
///
///
///

Figure 6. Dynamic text styled using weaver markup

The txt command can also be used to write mathematical notations, which is explained in section 5.3.

3.6

The tbl command

The tbl command can be used to create publication-ready dynamic tables in the dynamic report. The syntax is similar to that of Stata’s matrix input command, which
is used for manually deﬁning a matrix. When creating the dynamic table, the table can

952

Rethinking literate programming in statistics

be thought of as a matrix with its content deﬁned row by row, from top to bottom.
The options of the tbl command are identical to the options of the img command. The
syntax is as follows:

 

  
 
tbl (* , *...
\ * ,*...
\ ...
) , title(string) width(int)height(int) {left | center}
The * represents a display directive, which may be a double-quoted string, compound double-quoted string, macro, or scalar. This command also allows merging cells
horizontally with {col #} directives, where # represents the number of columns to
be merged. By default, tbl left-aligns the ﬁrst column and centers the other columns.
However, the content of each column may be left-aligned, centered, or right-aligned byusing the {l}, {c}, or {r} directive, respectively. Examples of column alignment canbe found in the help ﬁle.
In the example below, the summarize command is used twice to obtain the numberof observations, means, and standard deviations of two variables from auto.dta. To
demonstrate storing and displaying numbers in the dynamic table (see ﬁgure 6), thecontent of the ﬁrst row was saved in local macros and the content of the second row
was saved in scalars.
Examplequietly summarize pricelocal priceobs `r(N)´
local pricemean : display %7.2f `r(mean)´local pricesd: display %7.2f `r(sd)´quietly summarize mpg
tbl ("Variable", "Observations", "Mean", "SD" \"#__price__#", `priceobs´, `pricemean´, `pricesd´"#__mpg__#", r(N), r(mean), r(sd) )\Figure 7. A dynamic table

///
///

E. F. Haghish

953

Technical note
Like the txt command, the tbl command renders numeric scalars with two decimal
digits, which is commonly practiced. The tbl command deals with numeric macros andscalars diﬀerently, however, and the content of the macros is not reformatted automatically. Nevertheless, the content of any numeric macro or scalar can be reformatted (seethe example in section 8).

4Supported markup languages

In addition to HTML and LATEX, which are the native markup languages for annotating the content of the log ﬁle, the weaver package introduces a simpliﬁed JavaScript
markup language for the HTML log called weaver markup. Furthermore, weaver supports rendering LATEX mathematical notations in both the HTML and the LATEX log
ﬁles.

4.1

weaver markup

Inspired by Markdown (Haghish 2016b), which is a simple and intuitive markup language, a new markup language was developed for the weaver package to provide all
the text styling options that a basic text editor can provide, such as bolding or italicizing text, writing headings and subheadings, changing text color, highlighting text, andadding hypertext links. The weaver markup language was written in JavaScript andthus is only supported in the HTML log. It can be used along with HTML code to styletext anywhere in the document, including within the txt command and all options thatallow the display of descriptive text for graphs, tables, and even the title page of the
document. Wherever text appears in the HTML log, weaver markup can be used tostyle it. Table 3 describes the codes used in the weaver markup language.

954

Rethinking literate programming in statistics
Table 3. The weaver markup language
Markup code

ResultHeadings

Heading 1*- Heading 1 -**-- Heading 2 --**--- Heading 3 ---**---- Heading 4 ----*

Heading 4

Page and line breaks
line-break
page-break

add an empty line
begin a new page (in PDF)

Text styling#* Underscored *##_ Italic _##__ Bold __##___ Italic and Bold ___#
[center] text [#][right] text [#][box] text [#][mono] text [#]Underscored
Italic
Bold
Italic and Boldalign text to centeralign text to rightprint text in a colored box
display monospace text

Text color[blue] blue color text [#][green] green color text [#][red] red color text [#][purple] purple color text [#][pink] pink color text [#][orange] blue color text [#]blue color text
green color text
red color text
purple color text
pink color text
orange color text

Text highlight[-blue] blue color highlight [#][-green] green color highlight [#][-purple] purple color highlight [#][-pink] pink color highlight [#][-yellow] yellow color highlight [#][-gray] gray color highlight [#]blue color highlightgreen color highlight
purple color highlightpink color highlightyellow color highlightgray color highlightLink[-- "link url" --][- title -]title

Heading 2
Heading 3

E. F. Haghish

4.2

955

Writing mathematical notations in the HTML log

The ability to write mathematical notations can be essential when the weaver package
is used to develop educational materials for students or to develop technical documents
related to new statistical commands. Using the MathJax (Cervone 2012; Cervone et al.
2014; D’Ambrosia and Spitznagel 2013) software, weaver allows rendering LATEX mathematical notations in a paragraph or a separate line within the HTML log and, consequently, in the PDF document rendered from the HTML log.
In addition to displaying text, the txt command can also be used for printing mathematical notations in the log. Simply place the mathematical notations between \( . . . \)or \[ . . . \] signs, which render the notations in a text paragraph or on a separate line,
respectively. Moreover, a single or double dollar sign can also be applied in the LATEX
log. Examples of these notations are provided in the example in section 8.

5

ExamplesThe previous examples demonstrated how to communicate Stata commands and output
to the weaver log, write and style text using weaver markup, write mathematical notations, add graphs, and create dynamic tables. The following examples brieﬂy review
several features in both the HTML and the LATEX log ﬁles.

5.1

HTML example

In this example, a summary of the weaver commands is provided with more stylingdetails to demonstrate how weaver can be used to produce appealing educational materials, without any knowledge of HTML language. To make the dynamic report look
more formal, style(minimal) is speciﬁed. To keep the document short, the title page isomitted by excluding the title(), author(), affiliation(), address(), summary(),
and date() options from the weave using ﬁlename command.
Exampleweave using final.html, replace style(minimal) installsysuse auto, clear
summarize mpg
scalar mpgN
= r(N)
scalar mpgMean = r(mean)scalar mpgSD= r(sd)txt "*- Basic linear regression -*""[mono]auto.dta[#] includes the [mono]mpg[#] and [mono]weight[#] ""variables, which include data about mileage and weight of " r(N)" observations. The variables are summarized in table 1."summarize weight

///
///
///

956

Rethinking literate programming in statistics
tbl ({l}"Variable", "Observations", {l}"Mean", {l}"SD" \"[mono]mpg[#]", mpgN, mpgMean, mpgSD \"[mono]weight[#]", r(N), r(mean), r(sd)),
title("Table 1. Summary of the [mono]mpg[#] and [mono]weight[#] variables")center

///
///
///
///

correlate mpg weighttxt "There is a high and negative correlation of " r(rho)" between the [mono]mpg[#] and [mono]weight[#] variables. We can impose ""a regression line on the plotted data to examine the quality of fit ""of a classic linear model. To do so, type the following command in ""Stata. Figure 1 shows the fit regression line in the scatterplot."
div code graph twoway (scatter mpg weight) (lfit mpg weight)
graph export scatter.png, replace
img using scatter.png, center w(250) h(180)title("Figure 1. Regression line fit on the [mono]mpg[#] and""[mono]weight[#] variables")

///
///
///
///

///
///

txt The fit regression line can be modeled as follows:\[ \hat{\mu} (x) = \beta{_1} x + \beta{_0} \]///txt The \( \hat{\mu} (x) \) indicates the expected car mileage([mono]mpg[#]) for a given weight of \( x \) (lbs). Using the[mono]mpg[#] and [mono]weight[#] variables, the model can be updated
as follows:\[ \texttt{mpg} = \beta{_1}\texttt{weight} + \beta{_0} \]

///
///
///
///

txt We can fit a regression model in Stata by using the[mono][blue]regress[#][#] command as shown below. Table 2 presents
a summary of the regression analysis.

///
///

div c regress mpg weightmat reg = r(table)scalar Wcoe = reg[1,1]scalar Wsr = reg[2,1]scalar Wp= reg[4,1]scalar Wll = reg[5,1]scalar Wul = reg[6,1]
scalar Ccoe = reg[1,2]scalar Csr = reg[2,2]scalar Cp= reg[4,2]scalar Cll = reg[5,2]scalar Cul = reg[6,2]tbl ("Predictor", {l}"\( \\beta \)", {l}"SE",{l}{col 2} "95% Conf. Interval", {l}"p-value" \"Weight", Wcoe, Wsr, Wll, Wul, %7.3f Wp \"Intercept", Ccoe, Csr, Cll, Cul, %7.3f Cp ), center
title("Table 2. Summary of the regression analysis")

///
///
///
///

E. F. Haghish

txt "The two-sided p-value was found to be less than " %7.3f Wp". Based on the estimated coefficients, the fit regression ""equation for estimating the [mono]mpg[#] based on given ""[mono]weight[#] would be ""\[ \texttt{mpg} = " %7.3f Wcoe "\texttt{weight} + " Ccoe " \]"

957

///
///
///
///

weave close

5.2

LATEX example

Here, the previous example is repeated for LATEX to allow comparison of LATEX markup
with the more simpliﬁed weaver markup. Because the output of this example is similar
to the previous example, it is omitted from the article.
Exampleweave using example.tex, install replace margin(5,20,5,20)
sysuse auto, clear
summarize mpg
scalar mpgN
= r(N)
scalar mpgMean = r(mean)scalar mpgSD= r(sd)txt "\section{Basic linear regression}"txt "\texttt{auto.dta} includes the \texttt{mpg} and ""\texttt{weight} variables, which include data about mileage and""weight of " r(N) " observations. The variables are summarized in table 1."

///
///

summarize weighttbl ({l}"Variable", "Observations", {l}"Mean", {l}"SD" \"mpg", mpgN, mpgMean, mpgSD \"weight", r(N), r(mean), r(sd)),
title("Summary of the \texttt{mpg} and \texttt{weight} variables") center

///
///
///

correlate mpg weighttxt "There is a high and negative correlation of " r(rho)" between the \texttt{mpg} and \texttt{weight} variables. ""We can impose a regression line on the plotted data to examine the ""quality of fit of a classic linear model. To do so, type the following ""command in Stata. Figure 1 shows the fit regression ""line in the scatterplot."

///
///
///
///
///

div code graph twoway (scatter mpg weight) (lfit mpg weight)
graph export scatter.png, replace
img using scatter.png, center w(250) h(180)///title("Regression line fit on the \texttt{mpg} and \texttt{weight} variables")958

Rethinking literate programming in statistics

txt The fit regression line can be modeled as follows:$$ \widehat{\mu} (x) = \beta{_1} x + \beta{_0} $$///txt "The $ \widehat{\mu} (x) $ indicates the expected car mileage ""for a given weight of $ x $ (lbs). Using the \texttt{mpg} "///"and \texttt{weight} variables, the model can be updated as follows:""$$ mpg = \beta{_1}weight + \beta{_0} $$ "///txt "We can fit a regression model in Stata by using the \texttt{""{\color{blue}regress}} command as shown below. Table 2 presents ""a summary of the regression analysis."

///
///

///

div c regress mpg weightmat reg = r(table)scalar Wcoe = reg[1,1]scalar Wsr = reg[2,1]scalar Wp= reg[4,1]scalar Wll = reg[5,1]scalar Wul = reg[6,1]
scalar Ccoe = reg[1,2]scalar Csr = reg[2,2]scalar Cp= reg[4,2]scalar Cll = reg[5,2]scalar Cul = reg[6,2]tbl ("Predictor", {l}" $ \\beta $ ", {l}"SE",{l}{col 2} "95\\% Conf. Interval", {l}"p-value" \"Weight", Wcoe, Wsr, Wll, Wul, %7.3f Wp \"Intercept", Ccoe, Csr, Cll, Cul, %7.3f Cp ), center
title("Summary of the regression analysis")

///
///
///
///

txt "The two-sided p-value was found to be less than " %7.3f Wp". Based on the estimated coefficients, the fit regression ""equation for estimating the \texttt{mpg} based on given ""\texttt{weight} would be ""$$ \texttt{mpg} = " %7.3f Wcoe "\texttt{weight} " + " Ccoe " $$"

///
///
///
///

weave closeE. F. Haghish

959

Figure 8. Preview of the PDF example

960

Rethinking literate programming in statistics

Technical note
Note that the tbl command parses the rows with the backslash symbol. Therefore,
to include in a dynamic table any LATEX notations that begin with a backslash, suchas \beta or 95\%, a double backslash must be used to avoid conﬂict with the parsing
syntax (for example, \\beta and 95\\%).

6

Applications and conclusion

weaver is a dynamic document package for Stata that creates an HTML or LATEX log
ﬁle and a PDF dynamic document. Similarly to a Stata log, the weaver log is updated
interactively as the analysis commands are executed, and the dynamic document canbe previewed in real time. It also provides interesting features that are lacking in a
SMCL log because HTML and LATEX markups can render a variety of graphical featuresand provide numerous styling options. The weaver log can run parallel to the SMCL
log but will not register any input or output autonomously. While all commands andoutput are registered in the SMCL log, the weaver log instead allows the user to selectthe content of interest for the documentation.
The main advantage of this approach to dynamic reports is that users do not haveto create a separate script ﬁle for writing the report. When the user simply tells weaver
what to include in the HTML or LATEX log ﬁle, the document can be generated interactively while the analysis is developed or based on the same code ﬁles that are used
for the data analysis. Moreover, in contrast to literate-programming packages that require “weaving” the document from the source code, weaver catches output from Stata
as it is executed and then updates both the SMCL and the weaver log. weaver also
allows merging other ﬁles into the weaver log, and therefore, it can include outputs
from all other user-written packages that export LATEX or HTML ﬁles or tables, such as
the markdoc package. Thus, a time-consuming and long-term analysis can be written
in separate ﬁles using weaver or alternative packages, and then can be put together
dynamically in an HTML or LATEX log ﬁle.
The weaver package has many applications for Stata users. In addition to creating
a dynamic report, which is the main purpose of the package, it can also be used to write
tutorials for students. Anything from a mathematical equation to the way it can beapplied or programmed in Stata can be documented using the weaver package, making
it a useful tool for developing educational documents. It can also be used as an active
learning tool for statistics students. For example, Baumer et al. (2014) argue that
integrating reproducible analysis tools in teaching statistics, as early as introductory
statistics courses, improves students’ ability to express statistical results.
weaver also has several limitations. The package is mainly developed with the Statalanguage and partially supports the Mata language. In other words, Mata commands
cannot be directly included in the log using the div command unless they are executedwith the mata: preﬁx, which is the usual way of calling a Mata function in Stata without
entering the Mata environment. However, the markdoc literate-programming package
supports the Mata environment; thus, HTML or LATEX output generated by markdoc

E. F. Haghish

961

can be merged into the weaver log to include sections of data analysis carried out by
Mata.
Creating a dynamic-analysis document requires executing additional text commands
or programming preparations for creating dynamic tables (for example, saving estimations into local macros or scalars). This can potentially result in increasing thecomplexity of the code due to mixing two languages in one place. This is a problem
with any literate-programming software (not just the weaver package) that creates atrilingual source code including a programming language, a human language for interpretation, and a markup language for formatting and typesetting the documentation. If
the markup language is complex, such as LATEX or HTML, then the documentation becomes less distinguishable from the computer language, resulting in a ﬁle that is harderfor humans to read. weaver attempts to address this issue by creating the weaver
markup language, an easy-to-read and easy-to write markup language for annotating
the content of the dynamic document.
The weaver log is not necessarily limited to HTML or LATEX. The package canbe updated to support other markup languages such as markdown to further simplify
working with weaver package. Such a feature would allow Stata to store the output anddocumentation in a markdown ﬁle—that is running in parallel to the SMCL log—and
use the markdown log to generate a dynamic document in various formats, an approach
used by markdoc package.
Nonetheless, no tool or software can guarantee the clarity and cleanness of the codeﬁle, because writing a dirty code ﬁle is essentially a human problem. But keep inmind that the aim of writing code for running the analysis and compiling the dynamic
document simultaneously is not mainly sharing the HTML or PDF document. Instead,
it is commenting the procedure, interpreting the results, and sharing the code ﬁle anddata that compile the analysis and reproduce the dynamic document. Readability ofthe source code is essential: the document can be misleading by excluding part of theanalysis, giving a false description of the analysis procedure, or falsely interpreting theresults. Therefore, code should be written with great care about its appearance andsimplicity to facilitate others’ reading and comprehension of the code.

7

References

Allaire, J. J., J. Cheng, Y. Xie, J. McPherson, W. Chang, J. Allen, H. Wickham,
A. Atkins, and R. Hyndman. 2016. rmarkdown: Dynamic Documents for R. R package
version 0.5. https://cran.r-project.org/web/packages/rmarkdown/index.html.
Baumer, B., M. Cetinkaya-Rundel, A. Bray, L. Loi, and N. J. Horton. 2014. R markdown: Integrating a reproducible analysis tool into introductory statistics. Technology
Innovations in Statistics Education 8(1): 1–29.
http://escholarship.org/uc/item/90b2f5xh.
Cervone, D. 2012. MathJax: A platform for mathematics on the web. Notices of the
AMS 59: 312–316.

962

Rethinking literate programming in statistics

Cervone, D., C. Stark, R. Miner, and P. Topping. 2014. MathJax documentation:
Release 2.3. https://media.readthedocs.org/pdf/mathjax/v2.3-latest/mathjax.pdf.
Cordes, D., and M. Brown. 1991. The literate-programming paradigm. IEEE Computer
24: 52–61.
D’Ambrosia, B. K., and C. R. Spitznagel. 2013. MathJax: Seamless mathematics on theweb. In Proceedings of the 25th International Conference on Technology in Collegiate
Mathematics. Norfolk, VA: ICTCM.
Haghish, E. F. 2014a. MarkDoc: Literate programming in Stata.
http://www.haghish.com/statistics/stata-blog/reproducible-research/markdoc.php.
. 2014b. synlight: Stata module to highlight syntax in SMCL and translate toHTML format. Statistical Software Components S457894, Department of Economics,

Boston College. https://ideas.repec.org/c/boc/bocode/s457894.html.
. 2015. Statax: JavaScript syntax highlighter for Stata.
http://www.haghish.com/statax/statax.php.
. 2016a. github: Stata module for searching and installing Stata packages from
GitHub. GitHub. https://github.com/haghish.github.
. 2016b. markdown: A Stata module to convert Markdown to SMCL language.
http://github.com/haghish/markdown.
Jann, B. 2004. estout: Stata module to make regression tables. Statistical Software
Components S439301, Department of Economics, Boston College.
https://ideas.repec.org/c/boc/bocode/s439301.html.
. 2016. Creating LaTeX documents from within Stata using texdoc. Working
Paper 14, University of Bern Social Sciences.
http://repec.sowi.unibe.ch/ﬁles/wp14/jann-2015-texdoc.pdf.
Johnson, A. L., and B. C. Johnson. 1997. Literate programming using noweb. Linux
Journal 42: 64–69.
Knuth, D. E. 1983. The WEB system of structured documentation. Technical Report STAN-CS-83-980, Department of Computer Science, Stanford University.
http://infolab.stanford.edu/pub/cstr/reports/cs/tr/83/980/CS-TR-83-980.pdf.
. 1984. Literate programming. Computer Journal 27: 97–111.
. 1992. Literate Programming. Stanford, CA: CSLI Lecture Notes.
Leisch, F. 2002. Sweave: Dynamic generation of statistical reports using literate data
analysis. In COMPSTAT 2002, ed. W. Härdle and B. Rönz, 575–580. Physica Verlag:
Heidelberg.
Lenth, R. V. 2012. StatWeave Users’ Manual.
http://homepage.stat.uiowa.edu/∼rlenth/StatWeave/StatWeave-manual.pdf.

E. F. Haghish

963

Murdoch, D. J., and V. J. Carey. 2001. On the edge: Statistics & computing: Literatestatistical programming: Concepts and tools. Chance 14: 46–50.
Ramsey, N. 1994. Literate programming simpliﬁed. IEEE Software 11: 97–105.
Ramsey, N., and C. Marceau. 1991. Literate programming on a team project. Software—
Practice and Experience 21: 677–683.
Ruys, T. C., and E. Brinksma. 1998. Experience with literate programming in themodelling and validation of systems. In Tools and Algorithms for the Construction
and Analysis of Systems, ed. W. R. Cleaveland, 393–408. Stony Brook, NY: Springer.
Sametinger, J., and G. Pomberger. 1992. A hypertext system for literate C++ programming. Journal of Object-Oriented Programming 4: 24–29.
Shum, S., and C. Cook. 1994. Using literate programming to teach good programming
practices. ACM SIGCSE Bulletin 26(1): 66–70.
Watson, I. 2004. tabout: Stata module to export publication quality cross-tabulations.
Statistical Software Components S447101, Department of Economics, Boston College.
https://ideas.repec.org/c/boc/bocode/s447101.html.
Xie, Y. 2014. Dynamic Documents with R and knitr. Boca Raton, FL: Chapman &Hall/CRC.
. 2016. knitr: A General-Purpose Package for Dynamic Report Generation in R.
R package version 1.13. https://cran.r-project.org/web/packages/knitr/index.html.
About the author
E. F. Haghish is a PhD student in applied statistics at the Center for Medical Biometry andMedical Informatics at the University of Freiburg in Germany. He is also aﬃliated with the
University of Southern Denmark in Denmark.