Using Stata for data management
and reproducible research
Christopher F Baum
Boston College and DIW Berlin

March 15, 2021

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

1 / 161Overview of the Stata environment

Overview of the Stata environment

Stata is a full-featured statistical programming language for Windows,
macOS, Unix and Linux. It can be considered a “stat package,” likeSAS, SPSS, RATS, or eViews.
Stata is available in several versions: Stata/IC (the standard version),
Stata/SE (an extended version) and Stata/MP (for multiprocessing).
The major difference between the versions is the number of variables
allowed in memory, which is limited to 2,047 in standard Stata/IC, but
can be up to 32,767 in Stata/SE and 120,000 in Stata/MP. The numberof observations in any version is limited only by your computer’smemory.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

2 / 161Overview of the Stata environment

Stata/SE relaxes the Stata/IC constraint on the number of variables,
while Stata/MP is the multiprocessor version, capable of utilizing 2, 4,
8, 16, 24, 32... processors available on a single computer.
Stata/IC will meet most users’ needs. If you have access to Stata/SE
or Stata/MP, you can use that program to create a subset of a large
survey dataset with fewer than 2,047 variables. Stata runs on all 64-bit
operating systems.
All versions of Stata provide the full set of features and commands:
there are no special add-ons or ‘toolboxes’. Each copy of Stata
includes a complete set of manuals (over 11,000 pages) in PDF
format, hyperlinked to the on-line help.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

3 / 161Overview of the Stata environment

A Stata license may be used on any machine which supports Stata(Mac OS X, Windows, Linux): there are no machine-specific licenses
for Stata version 11 onward. You may install Stata on a home andoffice machine, as long as they are not used concurrently. Licenses
can be either annual or perpetual.
Stata works differently than some other packages in requiring that theentire dataset to be analyzed must reside in memory. This brings aconsiderable speed advantage, but implies that you may need more
RAM (memory) on your computer.
The latest release of Stata, version 16 of late June 2019, no longer
supports computers with a 32-bit operating system. All modern
computers are equipped with a 64-bit operating system. Only someolder, unsupported versions of Microsoft Windows are still using 32-bit
architecture.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

4 / 161Overview of the Stata environment

In some cases, the memory requirement may be of little concern.
Stata is capable of holding data very efficiently, and even a quite
sizable dataset (e.g., more than one million observations on 20–30
variables) may only require 500 Mb or so. You should take advantageof the compress command, which will check to see whether eachvariable may be held in fewer bytes than its current allocation.
For instance, indicator (dummy) variables and categorical variables
with fewer than 100 levels can be held in a single byte, and integers
less than 32,000 can be held in two bytes: see help datatypes fordetails. By default, floating-point numbers are held in four bytes,
providing about seven digits of accuracy. Some other statisticalprograms routinely use eight bytes to store all numeric variables.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

5 / 161Overview of the Stata environment

Portability

Stata is eminently portable, and its developers are committed tocross-platform compatibility. Stata runs the same way on Windows,
Mac OS X, Unix, and Linux systems. The only platform-specificaspects of using Stata are those related to native operating systemcommands: e.g. is the file to be accessed
C:\Stata\StataData\myfile.dta
or/users/baum/statadata/myfile.dta
Perhaps unique among statistical packages, Stata’s binary data files
may be freely copied from one platform to any other, or even accessedover the Internet from any machine that runs Stata. You may store
Stata’s binary datafiles on a webserver (HTTP server) and open themon any machine with access to that server.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

6 / 161Overview of the Stata environment

Stata’s user interface

Stata’s user interface

Stata has traditionally been a command-line-driven package that
operates in a graphical (windowed) environment. Stata version 14(released April 2015), version 15 (released June 2017) and version 16(released June 2019) contain a graphical user interface (GUI) for
command entry via menus and dialogs. Stata may also be used in acommand-line environment on a shared system (e.g., a Linux server) if
you do not have a graphical interface to that system.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

7 / 161Overview of the Stata environment

Stata’s user interface

A major advantage of Stata’s GUI system is that you always have theoption of reviewing the command that has been entered in Stata’s
Review panel. Thus, you may examine the syntax, revise it in theCommand panel and resubmit it. You may find that this is a more
efficient way of using the program than relying wholly on dialogs.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

8 / 161Overview of the Stata environment

Stata’s user interface

Stata version 15, default screen appearance:

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

9 / 161Overview of the Stata environment

Stata’s user interface

The Toolbar contains icons that allow you to Open and Save files, Print
results, control Logs, and manipulate windows. Some very important
tools allow you to open the Do-File Editor, the Data Editor and the Data
Browser.
The Data Editor and Data Browser present you with a spreadsheet-likeview of the data, no matter how large your dataset may be. The
Do-File editor, as we will discuss, allows you to construct a file of Stata
commands, or “do-file”, and execute it in whole or in part from theeditor.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

10 / 161

Overview of the Stata environment

Stata’s user interface

The foot of the screen also contains an important piece of information:
the Current Working Directory, or cwd. In the screenshot, it is listed as/users/baum/Documents. The cwd is the directory to which any
files created in your Stata session will be saved. Likewise, if you try to
open a file and give its name alone, it is assumed to reside in the cwd.
If it is in another location, you must change the cwd [File→Change
working directory] or qualify its name with the directory in which it
resides.
You generally will not want to locate or save files in the default cwd. A
common strategy is to set up a directory for each project or task in aconvenient location in the filesystem and change the cwd to that
directory when working on that task. This can be automated in a
do-file with the cd command.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

11 / 161

Overview of the Stata environment

Stata’s user interface

There are several panels in the default interface: the Review, Results,
Command, Variables and Properties panels. You may alter theappearance of any panel in the GUI using the Preferences→Generaldialog, and make those changes on a temporary or permanent basis.
As you might expect, you may type commands in the Command panel.
You may only enter one command in that panel, so you should not try
pasting a list of several commands. When a command is
executed—with or without error—it appears in the Review panel, andthe results of the command (or an error message) appears in the
Results panel. You may click on any command in the Review panel
and it will reappear in the Command panel, where it may be edited andresubmitted.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

12 / 161

Overview of the Stata environment

Stata’s user interface

Once you have loaded data into the program, the Variables panel will
be populated with information on each variable, as you can see in theexample. That information includes the variable name, its label (if any),
its type and its format. This is a subset of information available fromthe describe command.
Let’s look at the interface after I have loaded one of the datasets
provided with Stata, uslifeexp, with the sysuse command andgiven the describe and summarize commands.
As this dataset is already loaded on your machine, you
may now give the same commands to follow along.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

13 / 161

Overview of the Stata environment

Christopher F Baum (BC / DIW)

Stata’s user interface

Using Stata

March 15, 2021

14 / 161

Overview of the Stata environment

Stata’s user interface

Notice that the three commands are listed in the Review panel. If any
had failed, the _rc column would contain a nonzero number, in red,
indicating the error code.
The Variables panel contains the list of variables and their labels.
The Results panel shows the effects of summarize: for each variable,
the number of observations, their mean, standard deviation, minimum
and maximum. If there were any string variables in the dataset, theywould be listed as having zero observations.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

15 / 161

Overview of the Stata environment

Stata’s user interface

Try it out: type the commands

sysuse uslifeexp
describesummarize

Take note of a very important design feature of Stata. If you do not say
what to describe or summarize, Stata assumes you want toperform those commands for every variable in memory, as shown
here. As we shall see, this design principle holds throughout the
program. It is quite different from the underlying principle of a
spreadsheet or a statistical package like SAS.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

16 / 161

Overview of the Stata environment

Using the Do-File Editor

We may also write a do-file in the do-file editor and execute it. The
Do-File Editor icon on the Toolbar brings up a window in which we may
type those same three commands, as well as a few more:sysuse uslifeexp
describesummarizenotes// average life expectancy, 1900-1949
summarize le if year < 1950// average life expectancy, 1950-1999
summarize le if year >= 1950
After typing those commands into the window, the rightmost icon, withtooltip Do, may be used to execute them.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

17 / 161

Overview of the Stata environment

Christopher F Baum (BC / DIW)

Using the Do-File Editor

Using Stata

March 15, 2021

18 / 161

Overview of the Stata environment

Using the Do-File Editor

In this do-file, I have included the notes command to display the notes
saved with the dataset, and included two comment lines. There areseveral styles of comments available. In this style, anything on a line
following a double slash (//) is ignored. You may also place an asterisk(*) on the left margin to indicate a comment, or surround several
comment lines in a do-file with the /* . . . */ notation.
If a command is too long to fit comfortably on a single line, you may
continue it on successive lines by placing a triple slash (///) at theend of each line.
You may use the other icons in the Do-File Editor window to save yourdo-file (to the cwd or elsewhere), print it, or edit its contents. You may
also select a portion of the file with the mouse and execute only thosecommands.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

19 / 161

Overview of the Stata environment

Christopher F Baum (BC / DIW)

Using the Do-File Editor

Using Stata

March 15, 2021

20 / 161

Overview of the Stata environment

Using the Do-File Editor

Try it out: use the Do-File Editor to save and reopen
the do-file monday.do, and run the file.
Try selecting only those last four lines and run thosecommands.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

21 / 161

Overview of the Stata environment

Using the Do-File Editor

I suggest that you use the following template for do-files:

set more off
capture log close
log using dofilename, replace...
...
log closetranslate dofilename.smcl dofilename.pdf, replace

set more off will prevent Stata from stopping when the screen isfilled. The log commands will produce a logfile in both Stata’s own
SMCL format and as a PDF.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

22 / 161

Overview of the Stata environment

The help system

The rightmost menu on the menu bar is labeled Help. From that menu,
you can search for help on any command or feature. The Help
Browser, which opens in a Viewer window, provides hyperlinks, in blue,
to additional help pages. At the foot of each help screen, there are
hyperlinks to the full manuals, which are accessible in PDF format.
The links will take you directly to the appropriate page of the manual.
You can also search for help at the command line with help
command. But what if you don’t know the exact command name?
Then you may use the search command, which may be followed by
one or several words.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

23 / 161

Overview of the Stata environment

The help system

Results from search are presented in a Viewer window. Thosecommands will present results from a keyword database and from theInternet: for instance, FAQs from the Stata website, articles in the
Stata Journal and Stata Technical Bulletin, and downloadable routinesfrom the SSC Archive (about which more later) and user sites.
Try it out: when you are connected to the Internet, type the commands
search baum
Note the hyperlinks that appear on URLs for the books and journal
articles, and on the individual software packages (e.g., st0030_3,
archlm).

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

24 / 161

Overview of the Stata environment

Stata’s update facility

Stata’s update facility

One of Stata’s great strengths is that it can be updated over theInternet. Stata is actually a web browser, so it may contact Stata’s web
server and enquire whether there are more recent versions of either
Stata’s executable (the kernel) or the ado-files. This enables Stata’s
developers to distribute bug fixes, enhancements to existingcommands, and even entirely new commands during the lifetime of agiven major release, including ‘dot-releases’ such as Stata 15.1.
Updates during the life of the version you own are free. You need only
have a licensed copy of Stata and access to the Internet (which may
be by proxy server) to check for and, if desired, download the updates.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

25 / 161

Overview of the Stata environment

Extensibility

Extensibility of official Stata

Another advantage of the command-line driven environment involves
extensibility: the continual expansion of Stata’s capabilities. A
command, to Stata, is a verb instructing the program to perform someaction.
Commands may be “built in” commands—those elements sofrequently used that they have been coded into the “Stata kernel.” A
relatively small fraction of the total number of official Stata commands
are built in, but they are used very heavily.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

26 / 161

Overview of the Stata environment

Extensibility

The vast majority of Stata commands are written in Stata’s own
programming language–the “ado-file” language. If a command is not
built in to the Stata kernel, Stata searches for it along the adopath.
Like the PATH in Unix, Linux or DOS, the adopath indicates theseveral directories in which an ado-file might be located. This implies
that the “official” Stata commands are not limited to those coded into
the kernel.
Try it out: give the adopath command in Stata.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

27 / 161

Overview of the Stata environment

Extensibility

If Stata’s developers tomorrow wrote a new command named “foobar”,
they would make two files available on their web site: foobar.ado(the ado-file code) and foobar.sthlp (the associated help file). Bothare ordinary, readable ASCII text files. These files should be produced
in a text editor, not a word processing program.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

28 / 161

Overview of the Stata environment

Extensibility

The importance of this program design goes far beyond the limits ofofficial Stata. Since the adopath includes both Stata directories andother directories on your hard disk (or on a server’s filesystem), you
may acquire new Stata commands from a number of web sites.
The Stata Journal (SJ), a quarterly peer-reviewed journal, is theprimary method for distributing user contributions. Between 1991 and2001, the Stata Technical Bulletin played this role, and a complete setof issues of the STB are available on line at the Stata website.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

29 / 161

Overview of the Stata environment

Extensibility

The SJ is a subscription publication (articles more than three years oldfreely downloadable), but the ado- and sthlp-files may be freely
downloaded from Stata’s web site. The Stata help command
accesses help on all installed commands; the Stata search command
will locate commands that have been documented in the STB and theSJ, and with one click you may install them in your version of Stata.
Help for these commands will then be available in your own copy.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

30 / 161

Overview of the Stata environment

Extensibility

User extensibility: the SSC archive

But this is only the beginning. Stata users worldwide participate in the
StataList forum, and when a user has written and documented a newgeneral-purpose command to extend Stata functionality, they
announce it on the StataList forum. You may freely subscribe: see
https://www.statalist.org for information.
Since September 1997, all items posted to Statalist (over 1,500) havebeen placed in the Boston College Statistical Software Components(SSC) Archive in RePEc (Research Papers in Economics), available
from IDEAS (https://ideas.repec.org) and EconPapers(http://econpapers.repec.org).

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

31 / 161

Overview of the Stata environment

Extensibility

Any component in the SSC archive may be readily inspected with a
web browser, using IDEAS’ or EconPapers’ search functions, and if
desired you may install it with one command from the archive fromwithin Stata.
For instance, if you know there is a module in the archive named
mvsumm, you could use ssc describe mvsumm to learn more about
it, and ssc install mvsumm to install it if you wish. Anything in thearchive can be accessed via Stata’s ssc command: thus ssc
describe mvsumm will locate this module, and make it possible toinstall it with one click.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

32 / 161

Overview of the Stata environment

Extensibility

Windows users should not attempt to download the materials from a
web browser; it won’t work.
Try it out: typessc describe mvsumm
ssc install mvsumm
help mvsumm

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

33 / 161

Overview of the Stata environment

Extensibility

The command ssc new lists, in the Stata Viewer, all SSC packages
that have been added or modified in the last month. You may click on
their names for full details. The command ssc hot reports on themost popular packages on the SSC Archive.
The Stata command adoupdate checks to see whether all packages
you have downloaded and installed from the SSC archive, the Stata
Journal, or other user-maintained net from... sites are up to date.
adoupdate alone will provide a list of packages that have been
updated. You may then use adoupdate, update to refresh yourcopies of those packages, or specify which packages are to be
updated.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

34 / 161

Overview of the Stata environment

Extensibility

In this course, we will be making use of a number of datasets that areaccessible via the bcuse command. This is a variant of official
webuse which I wrote to provide access to instructional datasets.
Please install this command now on your copy of Stata with
ssc install bcuse
so that you will be able to access datasets with bcuse...

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

35 / 161

Overview of the Stata environment

Extensibility

The importance of all this is that Stata is infinitely extensible. Any
ado-file on your adopath is a full-fledged Stata command. Stata’s
capabilities thus extend far beyond the official, supported featuresdescribed in the Stata manual to a vast array of additional tools.
As the current directory is on the adopath, you can use the Do-file
Editor to create an ado-file, hello.ado:
program define hellodisplay "Stata says hello!"
end
exit
Stata will now respond to the command hello. It’s that easy.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

36 / 161

Working with the command line

Stata command syntax

Stata command syntax

Let’s consider the form of Stata commands. One of Stata’s greatstrengths, compared with many statistical packages, is that its
command syntax follows strict rules: in grammatical terms, there areno irregular verbs. This implies that when you have learned the way a
few key commands work, you will be able to use many more without
extensive study of the manual or even on-line help.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

37 / 161

Working with the command line

Stata command syntax

The fundamental syntax of all Stata commands follows a template. Not
all elements of the template are used by all commands, and someelements are only valid for certain commands. But where an element
appears, it will appear in the same place, following the same grammar.
Like Unix or Linux, Stata is case sensitive. Commands must be given
in lower case. For best results, keep all variable names in lower case(or upper case) to avoid confusion.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

38 / 161

Working with the command line

Command template

The general syntax of a Stata command is:[prefix_cmd:] cmdname [varlist] [=exp][if exp] [in range][weight] [using...] [,options]where elements in square brackets are optional for some commands.
In some cases, only the cmdname itself is required. describe without
arguments gives a description of the current contents of memory(including the identifier and timestamp of the current dataset), whilesummarize without arguments provides summary statistics for all(numeric) variables. Both may be given with a varlist specifying thevariables to be considered.
What are the other elements?

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

39 / 161

Working with the command line

The varlist

The varlist

varlist is a list of one or more variables on which the command is tooperate: the subject(s) of the verb. Stata works on the concept of asingle set of variables currently defined and contained in memory,each of which has a name. As the describe command will show you,each variable has a data type (various sorts of integers and reals, andstring variables of a specified maximum length). The varlist specifies
which of the defined variables are to be used in the command.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

40 / 161

Working with the command line

The varlist

The order of variables in the dataset matters, since you can use
hyphenated lists to include all variables between first and last. (Theorder and move commands can alter the order of variables.) You canalso use “wildcards” to refer to all variables with a certain prefix. If you
have variables pop60, pop70, pop80, pop90, you can refer to them in avarlist as pop* or pop?0.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

41 / 161

Working with the command line

The exp clause

The exp clause

The exp clause is used in commands such as generate andreplace where an algebraic expression is used to produce a new (orupdated) variable. In algebraic expressions, the operators V==, &, | and! are used as equal, AND, OR and NOT, respectively. The operator
is used to denote exponentiation. The + operator is overloaded todenote concatenation of character strings.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

42 / 161

Working with the command line

The if and in clauses

The if and in clauses

As mentioned earlier, Stata differs from several common programs inthat Stata commands will automatically apply to all observations
currently defined. You need not write explicit loops over theobservations. You can, but it is usually bad programming practice to do
so. As we shall see, it may also lead to misleading results.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

43 / 161

Working with the command line

The if and in clauses

Of course you may want not to refer to all observations, but to pick out
those that satisfy some criterion. This is the purpose of the if exp andin range clauses.
For instance, give the commands:

sysuse auto, clear
sort pricelist make price in 1/5
which list the five cheapest cars in the auto dataset. The 1/5 is anumlist: in this case, a list of observation numbers. ` is the last
observation, so that list make price in -5/` will list the five
most expensive cars in auto.dta.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

44 / 161

Working with the command line

The if and in clauses

Even more commonly, you may employ the if exp clause. This restricts
the set of observations to those for which the exp, a Boolean
expression, evaluates to true. Note the double equal in the exp. A
single equal sign, as in the C language, is used for assignment; double
equal for comparison.
Eitherlist make price if foreign==1orlist make price if foreign
lists only foreign cars.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

45 / 161

Working with the command line

The if and in clauses

Give the command
list make price if price > 10000 & !mi(price)
to list only expensive cars (in 1978 prices). Stata’s missing value
codes are greater than the largest positive number, so that the lastcommand would avoid listing cars for which the price is missing. You
could also use if price < . to select non-missing prices.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

46 / 161

Working with the command line

The using clause

The using clauseSome commands access files: reading data from external files, or
writing to files. These commands contain a using clause, in which thefilename appears. If a file is being written, you must specify the“replace” option to overwrite an existing file of that name.
Stata’s own binary file format, the .dta file, is cross-platform
compatible, even between machines with different byte orderings(low-endian and high-endian). A .dta file may be moved from one
computer to another using ftp (in binary transfer mode).

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

47 / 161

Working with the command line

The using clause

To bring the contents of an existing Stata file into memory, thecommand:use filename [, clear]is employed (clear will empty the current contents of memory). You
must have sufficient memory for Stata to load the entire file, as Stata’s
speed is largely derived from holding the entire data set in memory.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

48 / 161

Working with the command line

The using clause

In Stata version 16, you can also access a portion of an existing Stata
file:
use varlist [if] [in] using filename [, clear]
A subset of variables can be specified in the varlist, and the if exp orin range qualifiers can be used to read a subset of observations.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

49 / 161

Working with the command line

The using clause

Reading and writing binary (.dta) files is much faster than dealing withtext (ASCII) files (with the insheet or infile commands), andpermits variable labels, value labels, and other characteristics of thefile to be saved along with the file. To write a Stata binary file, thecommandsave file [, replace]is employed. The compress command can be used to economize on
the disk space and memory required to store variables and load thedataset.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

50 / 161

Working with the command line

The using clause

Stata’s version 10, 11 and 12 datasets cannot be read by version 8 or
9; to create a compatible dataset, use the saveold command.
Likewise, Stata 13 introduced a new dataset format to accommodate
long string variables (strLs), and Stata 14 and 15’s dataset format
accommodates Unicode storage of string variables.
The saveold command in Stata versions 14, 15 and 16 will create adataset usable (except for long strings and Unicode) in version 11, 12
or 13. The version() option may be used to specify one of thoseversion numbers.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

51 / 161

Working with the command line

The options clause

The options clause

Many commands make use of options (such as clear on use, or
replace on save). All options are given following a single comma,
and may be given in any order. Options, like commands, may generally
be abbreviated (with the notable exception of replace).

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

52 / 161

Working with the command line

Programmability of tasks

Programmability of tasks

Stata may be used in an interactive mode, and those learning the
package may wish to make use of the menu system. But when you
execute a command from a pull-down menu, it records the commandthat you could have typed in the Review window, and thus you may
learn that with experience you could type that command (or modify it
and resubmit it) more quickly than by use of the menus.
Stata makes reproducibility very easy through a log facility, the ability
to generate a command log (containing only the commands you haveentered), and the do-file editor which allows you to easily enter,
execute and save sequences of commands, or program fragments.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

53 / 161

Working with the command line

Programmability of tasks

Going one step further, if you use the do-file editor to create asequence of commands, you may save that do-file and reuse it
tomorrow, or use it as the starting point for a similar set of data
management or statistical operations.
Working in this way promotes reproducibility, which makes it very easy
to perform an alternate analysis of a particular model. Even if many
steps have been taken since the basic model was specified, it is easy
to go back and produce a variation on the analysis if all the work isrepresented by a series of programs.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

54 / 161

Working with the command line

Programmability of tasks

One of the implications of the concern for reproducible work: avoid
altering data in a non-auditable environment such as a spreadsheet.
Rather, you should transfer external data into the Stata environment as
early as possible in the process of analysis, and only make permanentchanges to the data with do-files that can give you an audit trail ofevery change made to the data.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

55 / 161

Working with the command line

Programmability of tasks

Programmable tasks are supported by prefix commands, as we willsoon discuss, that provide implicit loops, as well as explicit looping
constructs such as the forvalues and foreach commands.
To use these commands you must understand Stata’s concepts oflocal and global macros. Note that the term macro in Stata bears noresemblance to the concept of an Excel macro. A macro, in Stata, isan alias to an object, which may be a number or string.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

56 / 161

Working with the command line

Local macros and scalars

Local macros and scalars

In programming terms, local macros and scalars are the “variables” of
Stata programs (not to be confused with the variables of the data set).
The distinction: a local macro can contain a string, while a scalar cancontain a single number (at maximum precision).
You should use these constructs whenever possible to avoid creating
variables with constant values merely for the storage of those
constants. This is particularly important when working with large data
sets.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

57 / 161

Working with the command line

Local macros and scalars

When you want to work with a scalar object—such as a counter in a
foreach or forvalues command—it will involve defining andaccessing a local macro. As we will see, all Stata commands thatcompute results or estimates generate one or more objects to holdthose items, which are saved as numeric scalars, local macros (strings
or numbers) or numeric matrices.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

58 / 161

Working with the command line

Local macros and scalars

The local macro
The local macro is an invaluable tool for do-file authors. A local macrois created with the local statement, which serves to name the macroand provide its content. When you next refer to the macro, you extract
its value by dereferencing it, using the backtick (‘) and apostrophe (’)
on its left and right. Try it out:

local george 2
local paul = ‘george’ + 2
display "‘paul’"In this case, I use an equals sign in the second local statement as I
want to evaluate the right-hand side, as an arithmetic expression, andstore it in the macro paul. If I did not use the equals sign in thiscontext, the macro paul would contain the string 2 + 2.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

59 / 161

Working with the command line

forvalues and foreachforvalues and foreach
In other cases, you want to redefine the macro, not evaluate it, and you
should not use an equals sign. You merely want to take the contents ofthe macro (a character string) and alter that string. The two key
programming constructs for repetition, forvalues and foreach,make use of local macros as their “counter”. For instance:
forvalues i=1/10 {summarize PRweek‘i’}Note that the value of the local macro i is used within the body of theloop when that counter is to be referenced. Any Stata numlist may
appear in the forvalues statement. Note also the curly braces,
which must appear at the end of their respective lines.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

60 / 161

Working with the command line

forvalues and foreach

In many cases, the forvalues command will allow you to substituteexplicit statements with a single loop construct. By modifying the range
and body of the loop, you can easily rewrite your do-file to handle a
different case.
The foreach command is even more useful. It defines an iteration
over any one of a number of lists:
the contents of a varlist (list of existing variables)
the contents of a newlist (list of new variables)
the contents of a numlist (list of integers)
the separate words of a macro
the elements of an arbitrary list

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

61 / 161

Working with the command line

forvalues and foreachFor example, we might want to summarize each of these variables’
detailed statistics from this World Bank data set. Give the commands:

sysuse lifeexp, clear
foreach v of varlist popgrowth lexp gnppc {summarize ‘v’, detail}

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

62 / 161

Working with the command line

forvalues and foreachAs a more elaborate example, use this code to run a regression onvariables for each region, and graph the data and fitted line:

sysuse lifeexp, clear
levelsof region, local(regid)
foreach c of local regid {local rr : label region ‘c’regress lexp gnppc if region ==‘c’twoway (scatter lexp gnppc if region ==‘c’) ///(lfit lexp gnppc if region ==‘c’, ///ti(Region: ‘rr’) name(fig‘c’, replace))}

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

63 / 161

Working with the command line

forvalues and foreach

A local macro can be built up by redefinition:sysuse lifeexp, clear
levelsof region, local(regid)
local alleps
foreach c of local regid {regress lexp gnppc if region ==‘c’predict double eps‘c’ if e(sample), residual
local alleps "‘alleps’ eps‘c’"}

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

64 / 161

Working with the command line

forvalues and foreachWithin the loop we redefine the macro alleps (as a double-quoted
string) to contain itself and the name of the residuals from that region’s
regression. We could then use the macro alleps to generate a graphof all three regions’ residuals. These commands must be included in
the same do-file that defines alleps in order to access that local
macro.

gen cty = _nscatter `alleps´ cty, yline(0) scheme(s2mono) legend(rows(1)) ///ti("Residuals from model of life expectancy vs per capita GDP") ///t2("Fit separately for each region")

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

65 / 161

Working with the command line

forvalues and foreach

Residuals from model of life expectancy vs per capita GDP

-15

-10

-5

0

5

Fit separately for each region

0

20

40
cty
Eur & C.Asia

Christopher F Baum (BC / DIW)

60

N.A.

Using Stata

80

S.A.

March 15, 2021

66 / 161

Working with the command line

forvalues and foreachGlobal macros

Stata also supports global macros, which are referenced by a different
syntax ($country rather than ‘country’). Global macros are usefulwhen particular definitions (e.g., the default working directory for aparticular project) are to be referenced in several do-files that are to be
executed.
However, the creation of persistent objects of global scope can be
dangerous, as global macro definitions are retained for the entire Statasession. One of the advantages of local macros is that they disappear
when the do-file or ado-file in which they are defined finishes
execution.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

67 / 161

Working with the command line

Prefix commands

Prefix commands
A number of Stata commands can be used as prefix commands,
preceding a Stata command and modifying its behavior. The mostcommonly employed is the by prefix, which repeats a command over a
set of categories. The statsby: prefix repeats the command, butcollects statistics from each category. The rolling: prefix runs thecommand on moving subsets of the data (usually time series).
Other command prefixes: simulate:, which simulates a statisticalmodel; bootstrap:, allowing the computation of bootstrap statistics fromresampled data; and jackknife:, which runs a command over jackknife
subsets of the data. The svy: prefix can be used with many statisticalcommands to allow for survey sample design. The bayes: prefix runs
Bayesian regression models.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

68 / 161

Working with the command line

The by prefix

The by prefix
You can often save time and effort by using the by prefix. When acommand is prefixed with a bylist, it is performed repeatedly for eachelement of the variable or variables in that list, each of which must be
categorical.
Try it out:sysuse census
by region: summ pop medage

This one command provides descriptive statistics for each of four US
Census regions. If the data are not already sorted by the bylist
variables, the prefix bysort should be used.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

69 / 161

Working with the command line

The by prefix

This can be extended to include more than one by-variable.
Try it out:sysuse census
generate large = (pop > 5000000) if !mi(pop)bysort region large: summ popurban death

This is a very handy tool, which often replaces explicit loops that must
be used in other programs to achieve the same end.
The by-group logic will work properly even when some of the definedgroups have no observations. However, its limitation is that it can only
execute a single command for each category. If you want to estimate aregression for each group and save the residuals or predicted values,
you must use an explicit loop.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

70 / 161

Working with the command line

The by prefix

The by prefix should not be confused with the by option available onsome commands, which allows for specification of a grouping variable:
for instancettest price, by(foreign)

will run a t-test for the difference of sample means across domestic
and foreign cars.
Another useful aspect of by is the way in which it modifies themeanings of the observation number symbol. Usually _n refers to thecurrent observation number, which varies from 1 to _N, the maximum
defined observation. Under a bylist, _n refers to the observation within
the bylist, and _N to the total number of observations for that category.
This is often useful in creating new variables.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

71 / 161

Working with the command line

The by prefix

For instance, if you have data on individuals with a family identifier,these commands might be useful:sort famid age
by famid: generate famsize = _N
by famid: generate birthorder = _N - _n +1

Here the famsize variable is set to _N, the total number of records forthat family, while the birthorder variable is generated by sorting thefamily members’ ages within each family.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

72 / 161

Data management: principles of organization and transformation

Missing values

Missing values
Missing value codes in Stata appear as the dot (.) in printed output(and a string missing value code as well: “”, the null string). It takes on
the largest possible positive value, so in the presence of missing data
you do not want to say
generate hiprice = (price > 10000)

but rather

generate hiprice = (price > 10000) if price <.

orgenerate hiprice = (price > 10000) if !mi(price)which then generates an indicator (dummy) variable equal to 1 forhigh-priced cars. The indicator will be zero for low-priced cars andmissing for cars with missing prices.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

73 / 161

Data management: principles of organization and transformation

Missing values

Stata allows for multiple missing value codes (.a, .b, .c, ...,
.z). The standard missing value code (.) is the smallest among
them, so testing for < . will always work. You may also use the missing
function: mi(varname) will return 1 if the observation is a missing
value, 0 otherwise.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

74 / 161

Data management: principles of organization and transformation

Missing data handling

Missing data handling

An issue that often arises when importing data from external sources
is the proper handling of missing data codes. Spreadsheet files often
use NA to denote missing values, while in some datasets codes suchas -9, -999, or -0.001 are used. The latter instances are
particularly worrisome as they may not be detected unless thevariables’ values are carefully scrutinized.
Note also that there is a missing value for string variables—the null, or
zero-length string—which looks identical to a string of one or more
space characters.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

75 / 161

Data management: principles of organization and transformation

Missing data handling

To properly handle missing values so that they are understood as suchin Stata, use the mvdecode command. This command allows you to
map various numeric values into numeric missing, or into one of theextended missing value codes .a, .b, ..., .z.
The mvencode command provides the inverse operation: particularly
useful if you must transfer data to another package that uses someother convention for missing values.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

76 / 161

Data management: principles of organization and transformation

Missing data handlingNo matter what methods you have used to input external data to the
Stata workspace, you should immediately save the file in Stata format
and perform the describe and summarize commands. It is much
more efficient to read a Stata-format .dta file with use than torepeatedly input a text file with any of the commands discussed above.
If the file is large, you may want to use the compress command to
optimise Stata’s memory usage before saving it. compress is
non-destructive; it never reduces the stored precision of a variable.
Before any further use is made of this datafile, examine the results ofthe describe and summarize commands and ensure that eachvariable has been input properly, and that numeric variables havesensible values for their minima and maxima.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

77 / 161

Data management: principles of organization and transformation

Display formats

Display formats

Each variable may have its own default display format. This does notalter the contents of the variable, but only affects how it is displayed.
For instance, %9.2f would display a two-decimal-place real number.
The command
format varname %9.2f
will save that format as the default format of the variable, andformat date %tm
will format a Stata date variable into a monthly format (e.g., 1998m10).

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

78 / 161

Data management: principles of organization and transformation

Variable labels

Variable labels

Each variable may have its own variable label. The variable label is acharacter string (maximum 80 characters) which describes thevariable, associated with the variable via
label variable varname "text"

Variable labels, where defined, will be used to identify the variable inprinted output, space permitting.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

79 / 161

Data management: principles of organization and transformation

Value labels

Value labels
Value labels associate numeric values with character strings. They
exist separately from variables, so that the same mapping of numerics
to their definitions can be defined once and applied to a set ofvariables (e.g. 1=very satisfied...5=not satisfied may be applied to all
responses to questions about consumer satisfaction). Value labels are
saved in the dataset. For example:
label define sexlbl 0 male 1 female
label values sex sexlbl

The latter command associates the label sexlbl with the variable
sex. Unlike other packages, Stata’s value labels are independent ofvariables, and the same label may be attached to any number ofvariables. If value labels are defined, they will be displayed in printed
output instead of the numeric values.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

80 / 161

Data management: principles of organization and transformation

Generating new variables

Generating new variables

The command generate is used to produce new variables in thedataset, whereas replace must be used to revise an existing
variable—and the command replace must always be spelled out.
A full set of functions are available for use in the generate command,
including the standard mathematical functions, recode functions, string
functions, date and time functions, and specialized functions: help
functions for details. Note that the sum() function in generate is arunning or cumulative sum.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

81 / 161

Data management: principles of organization and transformation

Generating new variables

As mentioned earlier, generate operates on all observations in thecurrent data set, producing a result or a missing value for each. You
need not write explicit loops over the observations. You can, but it isusually bad programming practice to do so. You may restrict
generate or replace to operate on a subset of the observations
with the if exp or in range qualifiers.
The if exp qualifier is usually more useful, but the in range qualifier
may be used to list a few observations of the data to examine theirvalidity. To list observations at the end of the current data set,
use if -5/` to see the last five.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

82 / 161

Data management: principles of organization and transformation

Generating new variables

You can take advantage of the fact that the exp specified in generate
may be a logical condition rather than a numeric or string value. This
allows producing both the 0s and 1s of an indicator (dummy, orBoolean) variable in one command. For instance:
generate large = (pop > 5000000) if !mi(pop)The condition & !mi(pop) makes use of a logical operator: !, NOT
to add the qualifier that the result variable should be missing if pop is
missing, using the mi() function. Although numeric functions of
missing values are usually missing, creation of an indicator variablerequires this additional step for safety.
The second logical operator is the Boolean AND, written as &. The
third logical operator is the Boolean OR, written as |. Note also that atest for equality is specified with the == operator (as in C). The single =is used only for assignment.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

83 / 161

Data management: principles of organization and transformation

Generating new variables

Keep in mind the important difference between the if exp qualifier
and the if (or ‘programmer’s if’) command. Users of some alternative
software may be tempted to use a construct such as
generate raceid = .
if (race == "Black") {replace raceid = 2}
else if (race == "Hispanic") {replace raceid = 3}

which is perfectly valid syntactically. It is also useless, in that it willdefine the entire raceid variable based on the value of race of thefirst observation in the data set!

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

84 / 161

Data management: principles of organization and transformation

Generating new variables

This is properly written in Stata as
generate raceid = .
replace raceid = 2 if race == "Black"
replace raceid = 3 if race == "White"

The raceid variable will be missing if race does not equal either of
those values.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

85 / 161

Data management: principles of organization and transformation

Functions for generate, replace

Functions for generate and replace
A number of lesser-known functions may be helpful in performing datatransformations. For instance, the inlist() and inrange()functions return an indicator of whether each observation meets a
certain condition: matching a value from a list or lying in a particularrange.
generate byte newengland = ///inlist(state, "CT", "ME", "MA", "NH", "RI", "VT")generate byte middleage = inrange(age, 35, 49)

The generated variables will take a value of 1 if the condition is met
and 0 if it is not. To guard against definition of missing values of stateor age, add the clause if !missing(varname):
generate byte middleage = inrange(age, 35, 49) if !mi(age)Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

86 / 161

Data management: principles of organization and transformation

Functions for generate, replace

Another common data manipulation task involves extracting a part ofan integer variable. For instance, firms in the US are classified byfour-digit Standard Industrial Classification (SIC) codes. The first twodigits represent an industrial sector. To define an industry variablefrom the firm’s SIC,
generate ind2d = int(SIC/100)

To extract the third and fourth digits, you could use
generate code34 = mod(SIC, 100)

using the modulo function to produce the remainder.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

87 / 161

Data management: principles of organization and transformation

Functions for generate, replace

The cond() function may often be used to avoid more complicatedcoding. It evaluates its first argument, and returns the second
argument if true, the third argument if false:
generate endqtr = cond( mod(month, 3) == 0, ///"Filing month", "Non-filing month")

Notice that in this example the endqtr variable need not be definedas string in the generate statement.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

88 / 161

Data management: principles of organization and transformation

Functions for generate, replace

Stata contains both a recode command and a recode() function.
These facilities may be used in lieu of a number of generate andreplace statements. There is also a irecode function to create anumeric code for values of a continuous variable falling in particular
brackets. For example, using a dataset containing population andmedian age for a number of US states:

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

89 / 161

Data management: principles of organization and transformation

Functions for generate, replace. bcuse census2c. generate size=irecode(pop, 1000, 4000, 8000, 20000). label define popsize 0 "<1m" 1 "1-4m" 2 "4-8m" 3 ">8m". label values size popsize. tabstat pop, stat(mean min max) by(size)
Summary for variables: pop
by categories of: size
mean
min
max
size
<1m
1-4m
4-8m
>8m

744.5412215.915381.751
12181.64

511.4561124.664075.979262.078

947.1543107.576
7364.823
17558.07

Total

5142.903

511.456

17558.07

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

90 / 161

Data management: principles of organization and transformation

Functions for generate, replace

Rather than categorizing a continuous variable using threshold values,
we may want to group observations based on quantiles: quartiles,
quintiles, deciles, or any other percentiles of their empirical distribution.
We can readily create groupings of that sort with xtile:. bcuse census2c. xtile medagequart = medage, nq(4). tabstat medage, stat(n mean min max) by(medagequart)
Summary for variables: medageby categories of: medagequart (4 quantiles of medage)
medagequart
N
mean
min
max
1
2
3
4

7
4
5
5

29.02857
29.875
30.54
32

28.3
29.7
30.1
31.8

29.4
30
31.2
32.2

Total

21

30.25714

28.3

32.2

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

91 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa

String-to-numeric conversion
A problem that commonly arises with data transferred fromspreadsheets is the automatic classification of a variable as string
rather than numeric. This often happens if the first value of such avariable is NA, denoting a missing value. If Stata’s convention for
numeric missings—the dot, or full stop (.) is used, this will not occur. If
one or more variables are misclassified as string, how can they bemodified?
First, a warning. Do not try to maintain long numeric codes (such as
US Social Security numbers, with nine digits) in numeric form, as they
will generally be rounded off. Treat them as string variables, which may
contain up to 2,045 bytes.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

92 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa

If a variable has merely been misclassified as string, the brute-force
approach can be used:generate patid = real( patientid )Any values of patientid that cannot be interpreted as numeric will
be missing in patid. Note that this will also occur if numbers are
stored with commas separating thousands.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

93 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa

A more subtle approach is given by the destring command, which
can transform variables in place (with the replace option) and can beused with a varlist to apply the same transformation to a set ofvariables. Like the real() function, destring should only be usedon variables misclassified as strings.
If the variable truly has string content and you need a numericequivalent, for statistical analysis, you may use encode on thevariable. To illustrate, let us read in some tab-delimited data with
import delimited.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

94 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa. import delimited statedata, clear(4 vars, 7 obs)
. format pop2008 %7.3f. list, sep(0)

1.
2.
3.
4.
5.
6.
7.

state

abbrev

yearjo~dpop2008

Massachusetts
New Hampshire
Vermont
New Jersey
Michigan
Arizona
Alaska

MA
NH
VT
NJ
MI
AZ
AK

1788
1788
1791
1787
1837
1912
1959

6.498
1.316
0.621
8.683
10.0036.500
0.686

As the data are tab-delimited, I can read a file with embedded spacesin the state variable.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

95 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa

I want to create a categorical variable identifying each state with an(arbitrary) numeric code. This can be achieved with encode:. encode state, gen(stid). list state stid, sep(0)

1.
2.
3.
4.
5.
6.
7.

state

stid

Massachusetts
New Hampshire
Vermont
New Jersey
Michigan
Arizona
AlaskaMassachusetts
New Hampshire
Vermont
New Jersey
Michigan
Arizona
Alaska. summarize stid
Variable

Obs

Mean

Std. Dev.

Min

Max

7

4

2.160247

1

7

stid

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

96 / 161

Data management: principles of organization and transformation

String-to-numeric conversion and vice versa

Although stid is a numeric variable (as summarize shows) it isautomatically assigned a value label consisting of the contents ofstate. The variable stid may now be used in analyses requiring
numeric variables.
You may also want to make a variable into a string (for instance, toreinstate leading zeros in an id code variable). You may use thestring() function, the tostring command or the decodecommand to perform this operation.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

97 / 161

Data management: principles of organization and transformation

The egen command

The egen command

Stata is not limited to using the set of defined generate functions.
The egen (extended generate) command makes use of functions
written in the Stata ado-file language, so that _gzap.ado would define
the extended generate function zap(). This would then be invoked asegen newvar = zap(oldvar)

which would do whatever zap does on the contents of oldvar,
creating the new variable newvar.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

98 / 161

Data management: principles of organization and transformation

The egen command

A number of egen functions provide row-wise operations similar to
those available in a spreadsheet: row sum, row average, row standard
deviation, and so on. Users may write their own egen functions. In
particular, findit egenmore for a very useful collection.
Although the syntax of an egen statement is very similar to that ofgenerate, several differences should be noted. As only a subset ofegen functions allow a by varlist: prefix or by(varlist) option, thedocumentation should be consulted to determine whether a particularfunction is byable, in Stata parlance. Similarly, the explicit use of _n
and _N, often useful in generate and replace commands is notcompatible with egen.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

99 / 161

Data management: principles of organization and transformation

The egen command

Wildcards may be used in row-wise functions. If you have state-level
U.S. Census variables pop1890, pop1900, ..., pop2000 you
may use egen nrCensus = rowmean(pop*) to compute theaverage population of each state over those decennial censuses. The
row-wise functions operate in the presence of missing values. The
mean will be computed for all 50 states, although several were not partof the US in 1890.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

100 / 161

Data management: principles of organization and transformation

The egen command

The number of non-missing elements in the row-wise varlist may becomputed with rownonmiss(), with rowmiss() as thecomplementary value. Other official row-wise functions include
rowmax(), rowmin(), rowtotal() and rowsd() (row standard
deviation).
The functions rowfirst()and rowlast() give the first (last)
non-missing values in the varlist. You may find this useful if thevariables refer to sequential items: for instance, wages earned peryear over several years, with missing values when unemployed.
rowfirst() would return the earliest wage observation, androwlast() the most recent.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

101 / 161

Data management: principles of organization and transformation

The egen command

Official egen also provides a number of statistical functions which
compute a statistic for specified observations of a variable and placethat constant value in each observation of the new variable. Since
these functions generally allow the use of by varlist:, they may be
used to compute statistics for each by-group of the data. This facilitatescomputing statistics for each household for individual-level data or
each industry for firm-level data. The count(), mean(), min(),max() and total() functions are especially useful in this context.
As an illustration using our state-level data, we egen the average
population in each of the size groups defined above, and express
each state’s population as a percentage of the average population inthat size group. Size category 0 includes the smallest states in our
sample.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

102 / 161

Data management: principles of organization and transformation

The egen command

Try it out:
. bcuse census2c. bysort size: egen avgpop = mean(pop). generate popratio = 100 * pop / avgpop. format popratio %7.2f. list state pop avgpop popratio if size == 0, sep(0)
state
1.
2.
3.
4.
5.

Rhode Island
Vermont
N. Dakota
S. Dakota
New Hampshire

Christopher F Baum (BC / DIW)pop

avgpoppopratio

947.2
511.5
652.7
690.8
920.6

744.541744.541744.541744.541744.541

127.21
68.69
87.67
92.78
123.65Using Stata

March 15, 2021

103 / 161

Data management: principles of organization and transformation

The egen command

Other egen functions in this statistical category include iqr()(inter-quartile range), kurt() (kurtosis), mad() (median absolutedeviation), mdev() (mean absolute deviation), median(), mode(),pc() (percent or proportion of total), pctile(), p(n) (nthpercentile), rank(), sd() (standard deviation), skew() (skewness)and std() (z-score).
Many other egen functions are available; see help egen for details.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

104 / 161

Data management: principles of organization and transformation

Time series calendar

Time series calendar

Stata supports date (and time) variables and the creation of a timeseries calendar variable. Dates are expressed, as they are in Excel, as
the number of days from a base date. In Stata’s case, that date is
1 Jan 1960 (like Unix/Linux). You may set up data on an annual,
half-yearly, quarterly, monthly, weekly or daily calendar, as well as acalendar that merely uses the observation number.
You may also set the delta of the calendar variable to be other than
1: for instance, if you have data at five-year intervals, you may define
the data as annual with delta=5. This ensures that the lagged valueof the 2005 observation is that of 2000.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

105 / 161

Data management: principles of organization and transformation

Time series calendar

An observation-number calendar is generally necessary forbusiness-daily data where you want to avoid gaps for weekends,
holidays etc. which will cause lagged values and differences to containmissing values. However, you may want to create two calendar
variables for the same time series data: one for statistical purposes
and one for graphical purposes, which will allow the series to begraphed with calendar-date labels.
This is a moot point in Stata version 12 onward, which provide support
for custom business-daily calendars (or bcals). As we shall see, Stata
can construct the bcal from your dataset in versions 13, 14, 15 and16.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

106 / 161

Data management: principles of organization and transformation

Time series operators

Time series operators

The D., L., F. and S. operators may be used under a time series
calendar (including in the context of panel data) to specify firstdifferences, lags, leads, and seasonal differences, respectively. These
operators understand missing data, and numlists: e.g. L(1/4).x isthe first through fourth lags of x, while L2D.x is the second lag of thefirst difference of the x variable.
The seasonal difference may be used to produce year-over-year orquarter-over-quarter values. For instance, with quarterly data,
S4.sales is the difference between current sales and sales four
quarters ago.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

107 / 161

Data management: principles of organization and transformation

Time series operators

It is very important to use the time series operators to refer to lagged
or led values, rather than referring to the observation number (e.g.,_n-1). The time series operators respect the time series calendar, andwill not mistakenly compute a lag or difference from a prior period if it
is missing. This is particularly important when working with panel data
to ensure that references to one individual do not reach back into theprior individual’s data.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

108 / 161

Data management: principles of organization and transformation

Time series operators

Using time series operators, you may not only consistently generate
differences, lags, and leads, but may refer to them ‘on the fly’ instatistical and estimation commands. To estimate an AR(4) model, youneed not create the lagged variables.
Try it out!
webuse lutkepohl, clear
regress consumption L(1/4).consumption

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

109 / 161

Data management: principles of organization and transformation

Time series operators

To test Granger causality:
regress consumption L(-4/4).income

which would regress consumption on four leads, four lags and thecurrent value of income.
For a “Dickey–Fuller” style regression:
regress D.consumption L.consumption

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

110 / 161

Data management: principles of organization and transformationtsegentsegenA limitation of the egen command is that it does not accept time series
operators in a varlist. The user-written tsegen command, ofPicard and Cox, removes that limitation for any egen command that
accepts a varlist. For example:
ssc install tsegenwebuse grunfeld, clear
tsegen inv_m5 = rowmean(invest L(1/4).invest)

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

111 / 161

Data management: principles of organization and transformationtsegenOr, making use of the rall() function from Cox’s egenmore
package, we can generate an indicator that identifies observations thatreflect 3 consecutive years of growth in market value exceeding 5%:
ssc install egenmore
webuse grunfeld, clear
g lmvalue = log(mvalue)tsegen ggt5 = rall(L(0/2)D.lmvalue,3) , c(@ > 0.05)

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

112 / 161

Data management: principles of organization and transformation

Factor variables, margins, marginsplot

Factor variables
A useful feature introduced in Stata version 11 is the factor variable.
Stata has only one kind of numeric variable, although it supports
several different data types, which define the amount of storage
needed and possible range of values. However, if a variable iscategorical, taking on only non-negative integer values, it may be used
as a factor variable with the i. prefix.
The use of factor variables not only avoids explicit generation ofindicator (dummy) variables for each level of the categorical variable,
but it means that the necessary indicator variables are generated ‘onthe fly’, as needed. Thus, to include the variable region, a categoricalvariable in census.dta which takes on values 1–4, we need only
refer to i.region in an estimation command.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

113 / 161

Data management: principles of organization and transformation

Factor variables, margins, marginsplot

This in itself merely mimics a preexisting feature of Stata: the xi:
prefix. But factor variables are much more powerful, in that they can beused to define interactions, both with other factor variables and with
continuous variables. Traditionally, you would define interactions bycreating new variables representing the product of two indicators, orthe product of an indicator with a continuous variable.
There is a great advantage in using factor variables rather thancreating new interaction variables. if you define interactions with thefactor variable syntax, Stata can then interpret the expression inpostestimation commands such as margins. For instance, you cansay i.race#i.sex, or i.sex#c.bmi, or c.bmi#c.bmi, where c.
denotes a continuous variable, and # specifies an interaction.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

114 / 161

Data management: principles of organization and transformation

Factor variables, margins, marginsplot

With interactions between indicator and continuous variables specified
in this syntax, we can evaluate the total effect of a change without
further programming. For instance,
regress healthscore i.sex#c.bmi c.bmi#c.bmi
margins, dydx(bmi) at(sex = (0 1))
which will perform the calculation of ∂healthscore/∂bmi for each levelof categorical variable sex, taking into account the squared term inbmi. We will discuss margins more fully in later talks.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

115 / 161

Data management: principles of organization and transformation

Factor variables, margins, marginsplot

An important new feature introduced in Stata 12 is the marginsplot
command, which may be used after margins to automatically produce
a graph of the margins you have computed: for instance, at various
levels of one of the continuous variables.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

116 / 161

Reading external data

import delimited

Reading external data with import delimited

Comma-separated (CSV) files or tab-delimited data files may be readvery easily with the import delimited command. If your file hasvariable names in the first row that are valid for Stata, they will be
automatically used (rather than default variable names). You usuallyneed not specify whether the data are tab- or comma-delimited. Notethat import delimited cannot read space-delimited data, norcharacter strings with embedded spaces, unless they aretab-delimited.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

117 / 161

Reading external data

import delimited

The command presumes that the filetype is .csv:
import delimited using filename

You can also specify which rows and/or columns of the .csv file are to
be read.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

118 / 161

Reading external data

import excel

Reading external data with import excel

In Stata 12 onward, you may read data directly from Excel andExcel-compatible worksheets with the import excel command. You
may specify from which sheet the data are to be loaded, and the rangeof that sheet in which they are located. You may also specify that thefirst row contains valid Stata variable names that are to be used. For
instance:
import excel using weo_201204_FR.xls, firstrow clear

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

119 / 161

Reading external data

import excel

We may read only some of the columns, but to use the Stata variablenames rather than column letters, we must specify cellrange() toomit the first row rather than the firstrow option. For instance:

import excel using weo_201204_FR.xls, describe
import excel iso year NGDPPC PCPI using weo_201204_FR.xls, cellrange(A2:AW39) clear

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

120 / 161

Reading external data

infileReading external data with infile
A free-format ASCII text file with space-, tab-, or comma-delimited data
may be read with the infile command. The missing-data indicator(.) may be used to specify that values are missing.
The command must specify the variable names. Assuming auto.raw
contains numeric data,
infile price mpg displacement using auto

will read it. If a file contains a combination of string and numeric values
in a variable, it should be read as string, and encode used to convert itto numeric with string value labels.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

121 / 161

Reading external data

infileIf some of the data are string variables without embedded spaces, theymust be specified in the command:
infile str3 country price mpg displacement using auto2

would read a three-letter country of origin code, followed by thenumeric variables. The number of observations will be determinedfrom the available data.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

122 / 161

Reading external data

infile

The infile command may also be used with fixed-format data,
including data containing undelimited string variables, by creating a
dictionary file which describes the format of each variable andspecifies where the data are to be found. The dictionary may also
specify that more than one record in the input file corresponds to asingle observation in the data set.
Sometimes data fields are not delimited—for instance, the sequence‘102’ might actually represent three integer variables. A dictionary
must then be used to define the variables’ locations.
The byvariable() option allows a variable-wise dataset to be read,where one specifies the number of observations available for eachseries.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

123 / 161

Reading external data

FRED

Reading macro data with import fred

Stata can access FRED, the Federal Reserve Economic Databaseprovided by the Federal Reserve Bank of St. Louis, with the importfred command.
To access FRED, you must create an account at
https://research.stlouisfed.org/ and request an API key,which is a long encoded string. Once you receive this string, you use
the set fredkey key, permanent to enable use of importfred.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

124 / 161

Reading external data

FRED

You may then use commands such as. fredsearch us dollar yen exchange rate monthly. fredsearch us dollar yuan exchange rate monthly. import fred EXJPUS EXCHUS. generate ym = mofd(daten). tsset ym, monthly

to locate the FRED series IDs and bring them in to Stata. All FRED
series are associated with calendar dates. The mofd() function
extracts the year and month from these monthly series’ calendar dates,
and the tsset command declares them as monthly time series. They
are then available for use, or combination with other time series data.
You can also use File->Import->Federal Reserve Economic
Data to access a graphical interface to FRED data.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

125 / 161

Reading external data

Christopher F Baum (BC / DIW)

FRED

Using Stata

March 15, 2021

126 / 161

Reading external data

FRED

In addition to data produced by the Federal Reserve System and other
US government statistical departments, FRED contains 9 IMF
databases, including International Financial Statistics and World
Economic Outlook, as well as data from the BIS, a number of othercentral banks, Eurostat, the NBER and the World Bank series fromWorld Development Indicators and Global Financial Development.
At present, FRED contains over 590,000 time series from 87 sources.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

127 / 161

Reading external data

Stat/Transfer

Reading data with Stat/Transfer
If your data are already in the internal format of SAS, SPSS, Excel,
GAUSS, MATLAB, or a number of other packages, the third-party
product Stat/Transfer is often useful.
Stat/Transfer will preserve variable labels, value labels, and otheraspects of the data, and can be used to convert a Stata binary file into
other packages’ formats. It can also produce subsets of the data
(selecting variables, cases or both) so as to generate an extract filethat is more manageable. This is particularly important when the
2,047-variable limit of Stata/IC is encountered.
In Stata version 16, the import sas, import sasxport5, importsasxport8 and import spss commands are available to read thenative formats of those packages.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

128 / 161

Writing external data

outfileWriting external data: outfile

If you want to transfer data to another package, Stat/Transfer is very
useful. But if you just want to create an ASCII file from Stata, theoutfile command may be used. It takes a varlist, and the if or inclauses may be used to control the observations to be exported.
Applying sort prior to outfile will control the order of observations in
the external file. You may specify that the data are to be written incomma-separated format.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

129 / 161

Writing external data

export delimited and fileWriting external data: export delimited and file

The export delimited command can write a comma-delimited or
tab-delimited ASCII file, optionally placing the variable names in thefirst row. Such a file can be easily read by a spreadsheet program
such as Excel.
For customized output, the file command can write out information(including scalars, matrices and macros, text strings, etc.) in any ASCIIor binary format of your choosing.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

130 / 161

Writing external data

export excel

Writing external data: export excel
In Stata 12 onward, the export excel command may be used towrite into a sheet of an Excel or Excel-compatible workbook. You may
modify or replace an existing sheet, and specify which Stata variables
are to be written to the worksheet. For example, after importing the
French spreadsheet data:
export excel using myfrenchdata, firstrow(variables) replace
export excel iso year NGDPPC PCPI using myfrenchdata, replace

will create a new workbook, myfrenchdata.xls, containing all
variables in memory in the first example, and only the four specified
variables in the second example. You may also specify that Statavariable labels are to be output, rather than variable names.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

131 / 161

Writing external data

postfile and post

Writing external data: postfile and post

A very useful capability is provided by the postfile and postcommands, which permit a Stata data set to be created in the courseof a program. For instance, you may be simulating the distribution of astatistic, fitting a model over separate samples, or bootstrapping
standard errors. Within the looping structure, you may post certain
numeric values to the postfile. This will create a separate Statabinary data set, which may then be opened in a later Stata run andanalyzed. Note that the parens () given in the documentation,
surrounding each exp, are required.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

132 / 161

Combining data sets

appendCombining data sets
In many empirical research projects, the raw data to be utilized arestored in a number of separate files: separate “waves” of panel data,timeseries data extracted from different databases, and the like. Stata
15 and earlier versions only permit a single data set to be accessed inmemory.
This constraint has been removed in Stata 16, which supports multiple
datasets in memory held in frames. Going forward, some of thetechniques we will now discuss will no longer be necessary when you
are using Stata 16, as contents from multiple datasets can be
combined using frame management techniques.
In Stata 15, how do you work with multiple data sets? Several
commands are available, including append, merge, and joinby.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

133 / 161

Combining data sets

appendThe append command

The append command combines two Stata-format data sets thatpossess variables in common, adding observations to the existing
variables. The same variables need not be present in both files, aslong as a subset of the variables are common to the “master” and“using” data sets. It is important to note that “PRICE" and “price” aredifferent variables, and one will not be appended to the other.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

134 / 161

Combining data sets

append

You might have a dataset on the demographic characteristics in 2007of the largest municipalities in China, cityCN. If you were given asecond dataset containing the same variables for the largest
municipalities in Japan in 2007, cityJP, you might want to combine
those datasets with append. With the cityCN dataset in memory, you
would append using cityJP, which would add those records asadditional observations. You could then save the combined file under adifferent name. append can be used to combine multiple datasets, so
if you had the additional files cityPH and cityMY, you could list thosefilenames in the using clause as well.
Prior to using append, it is very important to create an identifier
variable in each dataset that takes on a constant value: e.g., gencountry = 1 in the CN dataset, gen country = 2 in the JP
dataset, etc.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

135 / 161

Combining data sets

appendFor instance, consider the append command with two stylized
datasets:
id var 1 var 2

..
.. 
 112

.
.


dataset1 : 
..
.. 
 216

.
.


..
..
449
.
.


id


 126


dataset2 : 
 309

 421

604
Christopher F Baum (BC / DIW)var 1 var 2
..
.. 
.
. 

..
.. 
.
. 

..
.. 
.
. 

..
..
.
.

Using Stata

March 15, 2021

136 / 161

Combining data sets

appendThese two datasets contain the same variables, as they must for
append to sensibly combine them. If dataset2 contained idcode,
Var1, Var2 the two datasets could not sensibly be appended without
renaming the variables (recall that in Stata, var1 and Var1 are two
separate variables). Appending these two datasets with common
variable names creates a single dataset containing all of theobservations:

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

137 / 161

Combining data sets



id


 112


 216


 449
combined : 

 126

 309


 421

604

appendvar 1 var 2
..
.. 
.
. 
..
.. 

.
. 
..
.. 

.
. 
..
.. 

.
. 
..
.. 
.
. 

..
.. 
.
. 

..
..
.
.


The rule for append, then, is that if datasets are to be combined, theyshould share the same variable names and datatypes (string vs.
numeric). In the above example, if var1 in dataset1 was a float
while that variable in dataset2 was a string variable, they
could not be appended.
Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

138 / 161

Combining data sets

appendIt is permissible to append two datasets with differing variable names inthe sense that dataset2 could also contain an additional variable orvariables (for example, var3, var4). The values of those variables in
the observations coming from dataset1 would then be set to missing.
Some care must be taken when appending datasets in which the same
variable may exist with different data types (string in one, numeric inanother). For details, see “Stata tip 73: append with care!”, Baum CF,
Stata Journal, 2008, 9:1, 166-168.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

139 / 161

Combining data setsmerge

The merge command

We now describe the merge command, which is Stata’s basic tool forworking with more than one dataset. Its syntax changed considerably
in Stata version 11.
The merge command takes a first argument indicating whether you are
performing a one-to-one, many-to-one, one-to-many or many-to-manymerge using specified key variables. It can also perform a one-to-one
merge by observation.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

140 / 161

Combining data setsmerge

Like the append command, the merge works on a “master”
dataset—the current contents of memory—and a single “using”
dataset (prior to Stata 11, you could specify multiple using datasets).
One or more key variables are specified, and you need not sort either
dataset prior to merging.
The distinction between “master” and “using” is important. When thesame variable is present in each of the files, Stata’s default behavior isto hold the master data inviolate and discard the using dataset’s copyof that variable. This may be modified by the update option, which
specifies that non-missing values in the using dataset should replace
missing values in the master, and the even stronger update
replace, which specifies that non-missing values in the using dataset
should take precedence.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

141 / 161

Combining data setsmerge

A “one-to-one” merge (written merge 1:1) specifies that each recordin the using data set is to be combined with one record in the masterdata set. This would be appropriate if you acquired additional variables
for the same observations.
In any use of merge, a new variable, _merge, takes on integer values
indicating whether an observation appears in the master only, theusing only, or appears in both. This may be used to determine whether
the merge has been successful, or to remove those observations
which remain unmatched (e.g. merging a set of households fromdifferent cities with a comprehensive list of postal codes; one would
then discard all the unused postal code records).
The _merge variable must be dropped before another merge isperformed on this data set.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

142 / 161

Combining data setsmerge

Consider these two stylized datasets:
id var 1 var 2


.
.
.
.
 112
.
. 


dataset1 : 
..
.. 
 216
.
. 


..
..
449
.
.





var 22 var 44 var 46


..
..
..
 112

.
.
.


dataset3 : 

..
..
..
 216

.
.
.


..
..
..
449
.
.
.

Christopher F Baum (BC / DIW)

id

Using Stata

March 15, 2021

143 / 161

Combining data setsmerge

We may merge these datasets on the common merge key: in thiscase, the id variable.

id var 1 var 2 var 22 var 44 var 46


.
.
.
.
.
.
.
.
.
.
 112

.
.
.
.
.


combined : 

..
..
..
..
..
 216

.
.
.
.
.


..
..
..
..
..
449
.
.
.
.
.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

144 / 161

Combining data setsmerge

The rule for merge, then, is that if datasets are to be combined on one
or more merge keys, they each must have one or more variables with acommon name and datatype (string vs. numeric). In the example
above, each dataset must have a variable named id. That variable canbe numeric or string, but that characteristic of the merge key variables
must match across the datasets to be merged. Of course, we need nothave exactly the same observations in each dataset: if dataset3
contained observations with additional id values, those observations
would be merged with missing values for var1 and var2.
This is the simplest kind of merge: the one-to-one merge. Stata
supports several other types of merges. But the key concept should beclear: the merge command combines datasets “horizontally”, adding
variables’ values to existing observations.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

145 / 161

Combining data sets

Match merge

The merge command can also do a “many-to-one"’ or “one-to-many”
merge. For instance, you might have a dataset named hospitals
and a dataset named discharges, both of which contain a hospital
ID variable hospid. If you had the hospitals dataset in memory,
you could merge 1:m hospid using discharges to match eachhospital with its prior patients. If you had the discharges dataset inmemory, you could merge m:1 hospid using hospitals to add
the hospital characteristics to each discharge record. This is a very
useful technique to combine aggregate data with disaggregate data
without dealing with the details.
Although “many-to-one"’ or “one-to-many” merges are commonplaceand very useful, you should never want to do a “many-to-many” (m:m)
merge, which will yield seemingly random results.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

146 / 161

Combining data sets

Match merge

The long-form dataset is very useful if you want to add aggregate-level
information to individual records. For instance, we may have panel
data for a number of companies for several years. We may want to
attach various macro indicators (interest rate, GDP growth rate, etc.)that vary by year but not by company. We would place those macro
variables into a dataset, indexed by year, and sort it by year.
We could then use the firm-level panel dataset and sort it by year. A
merge command can then add the appropriate macro variables toeach instance of year. This use of merge is known as a one-to-many
match merge, where the year variable is the merge key.
Note that the merge key may contain several variables: we might haveinformation specific to industry and year that should be merged onto
each firm’s observations.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

147 / 161

Frames in Stata 16

Frames in Stata 16

In Stata version 16, released in summer 2019, you can work withmultiple datasets in memory. The constraint still applies: you musthave enough RAM to hold those datasets (or subsets thereof) in thecomputer’s memory. That is becoming a less serious constraint as
recent machines often have 8 Gb or 16 Gb of RAM.
The basic concept by which this is implemented is the data frame.
When you start Stata 16 and load a dataset with the use command, itgoes into a frame named default. You can use Stata as you always
have, ignoring the concept of data frames.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

148 / 161

Frames in Stata 16

However, what you can do now is load a second (and third...) dataset
into separate data frames. For instance:sysuse auto
frame create otherframe change otherwebuse lutkepohl... (do something with these data)
frame change default... (now we’re back to the auto dataset)frame drop other

So you can work with the alternate dataset without disturbing what you
have in memory. This was always possible using preserve andrestore, but as that created datafiles on disk, it was slow if a large
dataset was involved.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

149 / 161

Frames in Stata 16

Working with linked datasets

Working with linked datasets
As an alternative to merging two datasets as we discussed earlier,frames can be used to combine data by links. This is especially usefulwhen the data are at different levels of aggregation: individuals andhouseholds, households and counties, firms and industries, cities andcountries, and the like.
Say that we have an individual-level dataset, persons, and acounty-level dataset, counties. Just as with merge, there arecounties without any persons in our sample, and there could bepersons who do not live in any of the counties (although this may be adata error). The county-level demographic data may contain many
variables which we do not plan to use in our analysis.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

150 / 161

Frames in Stata 16

Working with linked datasets

use persons
frame create counties
frame counties: use counties
The frame framename: command specifies the active frame. It thenexecutes the Stata command following the colon, and switches back to
the prior frame. In this example, the default frame contains the
persons dataset. The command thus loads the counties dataset
into the frame counties, and makes default the active frame.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

151 / 161

Frames in Stata 16

Working with linked datasets

To link the county-level data to the individuals,
frlink m:1 countyid, frame(counties)
The m:1 works just as it does in merge: there may be many persons
who live in the same county, but there should be a unique countyid
attached to each record in the counties dataset.
So far we have not combined the data. We have linked the twodatasets, so that variables in the counties dataset can be accessedwith the persons dataset in memory.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

152 / 161

Frames in Stata 16

Working with linked datasets

You copy variables to the persons data as you need them, one at atime, or in groups, using the frget command:
frget med_income nschools, from(counties)
You can now perform the desired analysis using persons.dta, thedataset in the current frame:
regress income med_income n_schools educ age
Unlike the merge command, which adds the variables to the dataset,using frames in this manner does not modify either dataset.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

153 / 161

Reconfiguring data sets

Reconfiguring data sets
Data are often provided in a different orientation than that required forstatistical analysis. The most common example of this occurs withpanel, or longitudinal, data, in which each observation conceptually
has both cross-section (i) and time-series (t) subscripts. Often one will
want to work with a “pure” cross-section or “pure” time-series. If the
microdata themselves are the objects of analysis, this can be handledwith sorting and a loop structure.
If you have data on N firms for T periods per firm, and want to fit thesame model to each firm, one could use the statsby command, or if
more complex processing of each model’s results was required, aforeach block could be used. If analysis of a cross-section was
desired, a bysort would do the job.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

154 / 161

Reconfiguring data setscollapse

But what if you want to use average values for each time period,
averaged over firms? The resulting dataset of T observations can beeasily created by the collapse command, which permits you togenerate a new data set comprised of summary statistics of specified
variables. More than one summary statistic can be generated per input
variable, so that both the number of firms per period and the average
return on assets could be generated. collapse can produce counts,
means, medians, percentiles, extrema, and standard deviations.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

155 / 161

Reconfiguring data sets

reshapereshapeDifferent models applied to longitudinal data require different
orientations of those data. For instance, seemingly unrelated
regressions (sureg) require the data to have T observations (“wide”),
with separate variables for each cross–sectional unit. Fixed–effects or
random-effects regression models xtreg, on the other hand, requirethat the data be stacked or “vec”’d in the “long” format. It is usuallymuch easier to generate transformations of the data in stacked format,
where a single variable is involved.
The reshape command allows you to transfer the data from theformer (“wide”) format to the latter (“long”) format or vice versa. It is acomplicated command, because of the many variations on thisprocess one might encounter, but it is very powerful.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

156 / 161

Reconfiguring data sets

reshape

When data have more than one identifier per record, they may beorganized in different ways. For instance, it is common to find on-line
displays or downloadable spreadsheets of data for individual units—for
instance, U.S. states—with the unit’s name labeling the row and theyear labeling the column. If these data were brought into Stata in thisform, they would be in the wide form, wide form with the samemeasurement (population) for different years denoted as separate
Stata variables:. list, noobs
state

pop1990

pop1995

pop2000

CT
MA
RI

329196760226391005995

33241446141445
1017002

3411750
6362076
1050664

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

157 / 161

Reconfiguring data sets

reshape

There are a number of Stata commands—such as egen row-wisefunctions—which work effectively on data stored in the wide form. It
may also be a useful form of data organization for producing graphs.
Alternatively, we can imagine stacking each year’s population figuresfrom this display into one variable, pop. In this format, known in Stataas the long form, each datum is identified by two variables: the statename and the year to which it pertains.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

158 / 161

Reconfiguring data sets

reshape

We use reshape to transform the data, indicating that state shouldbe the main row identifier (i) with year as the secondary identifier (j):. reshape long pop, i(state) j(year). list, noobs sepby(state)
state

year

pop

CT
CT
CT

1990
1995
2000

329196733241443411750

MA
MA
MA

1990
1995
2000

602263961414456362076

RI
RI
RI

1990
1995
2000

1005995
10170021050664

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

159 / 161

Reconfiguring data sets

reshapeThis data structure is required for many of Stata’s statisticalcommands, such as the xt suite of panel data commands. The longform is also very useful for data management using by-groups and thecomputation of statistics at the individual level, often implemented withthe collapse command.
Inevitably, you will acquire data (either raw data or Stata datasets) thatare stored in either the wide or the long form and will find thattranslation to the other format is necessary to carry out your analysis.
In statistical packages lacking a data-reshape feature, common
practice entails writing the data to one or more external text files andreading it back in.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

160 / 161

Reconfiguring data sets

reshapeWith the proper use of reshape, writing data out and reading themback in is not necessary in Stata. But reshape requires, first of all,that the data to be reshaped are labelled in such a way that they canbe handled by the mechanical rules that the command applies. In
situations beyond the simple application of reshape, it may require
some experimentation to construct the appropriate command syntax.
This is all the more reason for enshrining that code in a do-file as someday you are likely to come upon a similar application for reshape.
An illustration of advanced use of reshape on data from International
Financial Statistics is provided in Baum CF, Cox NJ, “Stata tip 45:
Getting those data into shape,” Stata Journal, 2007, 7:268–271.

Christopher F Baum (BC / DIW)

Using Stata

March 15, 2021

161 / 161