Retrieving and Unpacking SDTS Data

Tutorial and Users Manual

August 2003

Neither the U.S. Government nor any agency thereof nor any of their
employees make any warranty, expressed or implied, or assume any legal
responsibility for the accuracy, completeness, or usefulness of any
information, apparatus, product, or process disclosed herein or
represent that its use would not infringe privately owned rights.
Reference to any specific commercial product, process, or service by
trade name, trademark, manufacturer, or otherwise does not necessarily
constitute or imply its endorsement, recommendation, or favoring by the
U.S. Government or any agency thereof.

1\. Introduction 1

2\. Overview of SDTS DLG and SDTS DEM Data 2

2.1 SDTS Overview 2

2.2 Mass Conversion of USGS Data to SDTS 2

2.3 Sales and Distribution Policies 3

2.4 SDTS data organization 3

2.4.1 SDTS DLG Data 3

2.4.2 SDTS DEM Data 4

3\. Summary of Data Retrieval Procedures 4

3.1 SDTS DLG Data 4

3.2 SDTS DEM Data 5

4\. Detailed Instructions for Retrieving SDTS Data 5

4.1 Create Local Directory 6

4.2 Find and Retrieve Data 6

4.2.1 Find and Retrieve SDTS DLG Data 6

4.2.1.1 SDTS DLG Transportation Overlay: A Special Case 7

4.2.1.2 Retrieve the SDTS DLG Master Data Dictionary 7

4.2.2 Find and Retrieve SDTS DEM Data 8

4.3 Unzip and Untar 8

4.3.1 WinZip 9

4.3.2 Other tar and zip utilities 10

4.3.2.1 DOS tar and zip utilities 10

4.3.2.2 UNIX tar and zip utilities 10

4.3.2.3 NT tar and zip utilities 10

5\. Problems Downloading Data 10

5.1 FTP 10

6\. Conclusions and Points of Contact 11

1\. Introduction

Two digital cartographic products of the U.S. Geological Survey (USGS),
Digital Line Graphs (DLG) and Digital Elevation Models (DEM), are
available in the Spatial Data Transfer Standard (SDTS) format. The USGS
distributes these data sets over the Internet as a means of promoting
the standard. The SDTS is an ANSI approved data exchange standard that
has been mandated for federal use. Numerous software packages from
vendors such as ESRI, Integraph and ERDAS are able to accept data in
this format.

This document is a detailed explanation of procedures for downloading
and unpacking USGS SDTS data. DLG data sets were created by reformatting
digital line graph optional (DLG-O) data as SDTS. DEM data sets were
created by reformatting existing data sets to the SDTS format. These
conversions include adding metadata to each transfer, but do not
necessarily include any update or other content changes to the spatial
data.

USGS SDTS data are a reflection of USGS native data. For more
information on USGS digital data, visit
[http://edc.usgs.gov](http://edc.usgs.gov/).

This document can be used as a companion to DLGv32 Pro, a free USGS
Digital Data Viewer. See http://mcmcweb.er.usgs.gov/drc/dlgv32pro/ for
additional information about DLGv32 Pro. The procedures described here
also may be beneficial to users of other software.

2\. Overview of SDTS DLG and SDTS DEM Data

2.1 SDTS Overview

The SDTS was designed by a group of people representing government
agencies, universities, and private companies that saw a requirement for
a robust way of transferring earth-referenced spatial data between
dissimilar computer systems with the potential for no information loss.
In 1980, the U.S. Geological Survey (USGS) was designated the lead
agency in developing earth science data standards for the federal
government. The USGS worked with academic, industrial, and federal,
state, and local government users of computer mapping and GIS to develop
a standard for transfer and exchange of spatial data. In 1992, after 12
years of developing, reviewing, revising, and testing, the resulting
standard\--SDTS, was approved as Federal Information Processing Standard
(FIPS) Publication 173, known as **FIPSPUB 173-1, 1994**. The FIPS
version has been superseded by the current version, known as **ANSI
NCITS 320-1998** and was ratified by the American National Standards
Institute (ANSI) June 9, 1998. Compliance with SDTS is now mandatory for
all federal agencies. SDTS is a transfer standard that embraces the idea
of self-contained transfers; spatial data, attributes, georeferencing, a
data-quality report, a data dictionary, and other supporting metadata
are all included in the transfer.

The SDTS is a standard. Standards are documents that specify rules; they
are neither software nor databases.

The SDTS is a standard for data *transfer*, as opposed to a standard for
data *processing*. SDTS does not replace existing Geographic Information
System (GIS) processing formats.

The SDTS is designed specifically for spatial data. Other standards
exist for other kinds of data (digital images, documents, electronic
signals, etc.).

The National Mapping Division of the USGS is the maintenance agency for
SDTS. Please visit the SDTS web site at
**<http://mcmcweb.er.usgs.gov/sdts>** for more information.

A single-document overview of SDTS is available at:

**<ftp://sdts.er.usgs.gov/pub/sdts/articles/ascii/overview.doc>**
or**,**

**<ftp://sdts.er.usgs.gov/pub/sdts/articles/ps/overview.ps>**

2.2 Mass Conversion of USGS Data to SDTS

In 1995 the USGS began converting digital cartographic data holdings to
SDTS. 1:24,000-scale (or 7.5 minute) DEM data were converted in early
1998.

The 1:2,000,000-scale DLG data were revised in 1995 and are now
available in SDTS. This includes coverage of the conterminous United
States for the following categories of data: hydrography,
transportation, boundaries (county names are included with the
transfer), United States Public Lands Survey System, and manmade
features.

All 1:100,000-scale DLG files have been converted to SDTS format. This
includes national coverage of the hydrography and transportation
overlays, with partial coverage of other layers.

All 1:24,000-scale DLG files have been converted to SDTS format.
Coverage varies between overlays at this scale, but no overlay has
complete coverage.

\"All files\" means all files for which DLG data have been produced.
Coverage of 1:24,000 and 1:100,000- scale DLG data is not complete.

2.3 Sales and Distribution Policies

Most USGS data are neither online nor free. SDTS data are a significant
exception to this policy. Other exceptions include 1:100,000-scale DLG-O
and 1:250,000-scale native format DEM data.

Production, pricing, and distribution policies for native DLG-O and
native DEM data have not changed because of the availability of SDTS
data. Data in native format are still available from the same sources at
the same prices. Because of data conversion problems, there are a small
number of data sets that are available in native format but not
available in SDTS. All DEM data now offered in SDTS format are still
available in native format.

2.4 SDTS data organization

**2.4.1 SDTS DLG Data**

DLG-O data are derived from standard USGS topographic quadrangles. The
packaging of DLG-O data reflects this: each physical file covers the
geographic area of some part of a standard quadrangle, and contains a
feature set that approximates the ink colors of a standard published
map.

Similarly, SDTS DLG data are derived directly from DLG-O data, and
reflect the DLG‑O data packaging. Each SDTS transfer has the same
geographic coverage and same spatial data content as the DLG-O it was
derived from. This packaging is not dictated by the SDTS; the standard
would allow (for example) all road and hydrography data for a county to
be contained in one transfer, but reorganizing the data this way would
have required a large and costly production operation, instead of a
relatively simple data reformat.

A side effect is that the SDTS DLG data appear more fragmented and
complex than necessary. This is not a characteristic of the SDTS, but of
policy decisions made when the USGS mass converted DLG-O data to SDTS
format.

**2.4.2 SDTS DEM Data**

DEM data are derived from standard USGS topographic quadrangles. The
packaging of DEM data reflects this: each physical file covers the
geographic area of some part of a standard quadrangle,

Similarly, SDTS DEM data are derived directly from DEM data, and reflect
the DEM data packaging. Each SDTS Data transfer has the same geographic
coverage and same spatial content as the original source DEM.

3\. Summary of Data Retrieval Procedures

SDTS DLG data can be found on the following ftp site:

**<ftp://edcftp.cr.usgs.gov/pub/data/DLG>**

3.1 SDTS DLG Data

There are three sub-directories in the DLG FTP site; one for each of the
three data scales, 100K, 2M, and LARGESCALE.

**100K** holds 1:100,000-scale data. The directory contains an SDTS
master data dictionary for this scale, a readme file, and 26
subdirectories, one for each letter of the alphabet. Each of these
subdirectories holds the 1:100,000-scale quadrangles whose names start
with that letter.

**2M** holds 1:2,000,000-scale data. The directory contains an SDTS
master data dictionary for this scale, a readme file, and 50
subdirectories, one for each state.

**LARGE_SCALE** holds 1:24,000-scale data. The directory contains an
SDTS master data dictionary for this scale, a readme file, and 26
subdirectories, one for each letter of the alphabet. Each of these
subdirectories holds the 1:24,000-scale quadrangles whose names start
with that letter.

Each data sub-directory has several more levels of directories that
reflect DLG overlays, data versions, and physical packaging.

The USGS GeoData web page, at **<http://edc.usgs.gov/geodata>** provides
a user interface to the ftp site. The web page provides several
different \"views\" of the data, including lists by state, by name, and
searching by graphic index. All of these views eventually lead to the
lowest level of the ftp directory structure. This web page also links to
the SDTS Home Page.

An SDTS transfer is composed of multiple files. SDTS DLG transfers have
been combined into one physical file with the tar utility, and then
compressed using gzip. See section 4.3 for details.

3.2 SDTS DEM Data

USGS SDTS DEM files are available for download from any of several
private venders. Links to the venders can be found at

**<http://edc.usgs.gov/geodata>**

4\. Detailed Instructions for Retrieving SDTS Data

This section contains step-by-step instructions for retrieving,
unpacking, and organizing SDTS DLG and SDTS DEM data. The San Francisco
North, California 1:24,000-scale quadrangle is used as an example of DLG
and DEM data.

Procedures for the other scales are similar, but vary slightly because
of different product characteristics.

These examples show how to find data using "FTP via graphics" and "FTP
via state" capabilities on the GeoData web page. Users who know what
they are looking for and know how to use ftp client software may wish to
skip these procedures and go directly to the ftp site described in
section 3. Reading section 4.3 is strongly recommended before doing
this.

4.1 Create Local Directory

Create folders or directories on your local computer to hold downloaded
files. For the purposes of these examples, we will assume you are using
a computer with the Windows 2000 operating system installed (although
these examples should also work with other versions of windows). If you
plan to use multiple quadrangles, it is important to organize the data
in a file structure that will help you keep track of all data. Create an
additional folder or subdirectory for each quadrangle.

An SDTS transfer typically contains many files. Keeping transfers
separate by putting each quadrangle in its own folder or directory is
necessary because of the way files within a SDTS DLG or SDTS DEM
transfer are named. Storing multiple quadrangles in one folder or
directory may result in file collisions, where one file may overwrite
another with the same naming convention.

Create a folder/directory named **san_francisco_north**. Within the
**san_francisco_north** folder/directory create two more folders,
**sdts_dem_24k** and **sdts_dlg_24k**.

For the SDTS DLG example, the file path should be similar to
**c:\\san_francisco_north\\sdts_dlg_24k**.

For the SDTS DEM example, the file path/name should be similar to
**c:\\san_francisco_north\\sdts_dem_24k**.

4.2 Find and Retrieve Data

**4.2.1 Find and Retrieve SDTS DLG Data**

A Master Data Dictionary (MDD) must be downloaded for each scale of DLG
data (100K, 2M, and Large Scale) data you intend to use. Detailed
instructions are included in section 4.2.1.2, below. To download a
specific data set, follow these procedures:

1\. Go to the USGS GeoData page at **<http://edc.usgs.gov/geodata>**

2\. Click on 1:24K DLG (in blue, near the top of the page).

3\. Since the name of the quadrangle is known, the data could be found
by selecting either [FTP via]{.underline} Graphics, FTP via
[Alphabetical List]{.underline} or [FTP via State]{.underline}. However,
it is common not to know exact quadrangle names:

Select **FTP via Graphics**.

4\. Select **Conterminous 48 states**.

5\. A simple map of the U.S. appears. Click on the **State of
California**.

6\. A map of California appears, with 1-degree lines of latitude and
longitude. Click **near San Francisco**. The next display will show
approximately 30 7.5-minute quadrangles, centered on where you clicked
your mouse.

7\. Click on the 7.5-minute cell labeled San Francisco North, CA**.**

8\. A \"list\" of one quadrangle appears. (Other paths to this page may
produce lists of more quadrangles.) Select the link to San Francisco
North, CA**.**

A list of nine directories will then appear. Each directory corresponds
to one DLG overlay. San Francisco North is used in this example because
all nine overlays are available. This is unusual for 1:24,000-scale
data; most quadrangles have fewer than nine overlays finished.

9\. Select the second overlay (in this example), **hydrography**.

10\. Select the file listed under (not necessarily inside) the folder
**[1/]{.underline}**, this should be the most recent version.

The file **1525941.HY.sdts.tar.gz** contains the SDTS transfer for the
hydrography layer of the San Francisco North, CA quadrangle.

11\. Select the file **1525941.HY.sdts.tar.gz** for downloading

12\. Save it to the **c:\\san_francisco_north\\sdts_dlg_24k folder**.
The file can be renamed (for example, sanfra.tar.gz). Retaining the
.tar.gz extensions is strongly recommended. Some browsers will attempt
to rename this file for you. In some cases, this renaming causes
problems for decompression software. One common case is a rename from
\*.tar.gz to \*\_tar.gz, which confuses the WinZip decompression
program.

Repeat steps 9 through 12 for all overlays of interest.

**4.2.1.1 SDTS DLG Transportation Overlay: A Special Case**

While you are at the FTP site getting data for the San Francisco North,
CA quadrangle, pull up the transportation subdirectory.

You will see that it contains three SDTS DLG transfers: roads (RD),
railroads (RR), and miscellaneous transportation (MT). All DLG-O
1:24,000-scale sales units are packaged this way, and that packaging was
retained in the SDTS version of the data: three separate SDTS
transportation transfers were created for each quadrangle. The data for
these transfers ***must*** *be kept in separate folders/directories*.
Untaring two or more transportation transfers for the same quad in the
same folder/directory will cause file collisions and loss of data.

If you intend to use more than one of these three transportation layers,
create subdirectories for each of them. For example,
c:\\san_francisco_north\\sdts_dlg_24k\\railroad. This step need not be
done before downloading the data, but must be done before unzipping and
untaring the data (section 4.3).

In 1:100,000 and 1:2,000,000 DLG data, the three transportation overlays
are combined into one file and, therefore, comprise only one SDTS
transfer. Creation of additional transportation folders/subdirectories
is not necessary for these scales.

**4.2.1.2 Retrieve the SDTS DLG Master Data Dictionary (MDD)**

Each SDTS transfer is required to have a data dictionary, specific to
that particular transfer's scale. Because SDTS DLG data were created by
reformatting DLG-O data, the same data dictionary can be used for all
data sets of the same scale. Rather than include this dictionary with
each transfer, one data dictionary is stored for all transfers at the
top of the ftp directory tree.

To retrieve the MDD:

1\. Go to **<http://edc.usgs.gov/geodata>.**

2\. Click on 1:24K DLG.

3\. Click on "SDTS DLGs require Master Data Dictionary."

4\. Click on single tar file for 1:24K.

5\. Save it to the **c:\\san_francisco_north\\sdts_dlg_24k** folder.

This file must be unzipped and untared like other SDTS data. Placement
of the data dictionary files may be software dependent. If using DLGv32
Pro, the files may be placed in any directory on your hard drive,
including the directory that holds the other SDTS data files. If using
ESRI Arc/Info, place the data dictionary files in the MDD directory as
specified by ESRI documentation.

**4.2.2 Find and Retrieve SDTS DEM Data**

1\. Go to the USGS GeoData page at **<http://edc.usgs.gov/geodata>.**

2\. Click on 1:24K DEM (in blue, near the top of the page).

3\. Use links to navigate to the vender of your choice download free
SDTS DEMs.

4\. Download San Francisco North (NOTE: Both ten and 30 meter resolution
SDTS DEMs are available for San Francisco North. For a quicker download,
choose the 30 meter resolution file).

5\. Select the file 1663964.dem.sdts.tar.gz for downloading.

6\. Save it to the **c:\\san_francisco_north\\sdts.dem.24k** folder. The
file can be renamed (for example, sanfran.tar.gz). Retaining the.tar.gz
extensions is strongly recommended. Some browsers will attempt to rename
this file for you. In some cases, this renaming causes problems for
decompression software. One common case is a rename from \*.tar.gz to
\*\_tar.gz, which confuses the WinZip decompression program.

4.3 Unzip and Untar

The SDTS DLG and SDTS DEM data were designed in 1994, before the release
of Windows 95 and before most PCs were powerful enough to use GIS data.
The data were designed for the scientific computers of the time, which
were mostly UNIX workstations. The files in each SDTS transfer were
combined into one file with the UNIX tar utility, and this combined file
was then compressed using the gnu-zip (gzip) utility. This technique for
combining and compressing files is common on the UNIX operating system.

Tar and gzip are not as common in the PC world, but a variety of
freeware, shareware, and commercial implementations of these utilities
exist for all PC operating systems.

**4.3.1 WinZip**

One of the most common PC compression programs is the shareware program
WinZip (**[http://www.winzip.com](http://www.winzip.com/)** ). WinZip
will unzip and untar SDTS DLGand SDTS DEM files if proper procedures are
followed. These instructions are for WinZip 8.1 and Windows,

Ensure that the file has a name of the form \*.tar.gz *NOT* \*\_tar.gz

1\. Start WinZip

2\. From the toolbar, select **Options**

3\. Select **Configuration**.

4\. Select **Miscellaneous**.

5\. There is a checkbox labeled **"TAR file smart CR/LF conversion"**.
[Ensure that this box is ***not*** checked.]{.underline}

6\. Select **OK** to exit configuration (this procedure should only have
to be done the first time you use WinZip, although the option has been
known to recheck itself).

7\. Navigate to the local directory you created earlier
(**c:\\san_francisco_north\\sdts_dlg_24k** for SDTS DLG data or
**c:\\san_francisco_north\\sdts_dem_24k** for SDTS DEM data). A list of
the files downloaded from the USGS GeoData site should be visible.
Select one of these files and double-click.

8\. A window appears with the message "Archive contains one file.
Decompress the file to a temporary folder and open it?" Select **Yes**.

9\. A list of .ddf files will appear. These are the files contained in
the SDTS transfer

10\. Select **Extract** from the tool bar.

12\. A popup window labeled \"Extract\" appears. Enter, or navigate to,
the path and directory that you want to place the data into. The
extracted files are stored in the same directory as the compressed
files, so enter (or navigate to) the local directory you created earlier
(**c:\\san_francisco_north\\sdts_dlg_24k** for SDTS DLG data or
**c:\\san_francisco_north\\sdts_dem_24k**), then click the **Extract**
button. This will unzip and untar the files.

NOTE: Do not rename files extracted from the tar file. The files in an
SDTS transfer are referenced by name in other files of the transfer.
Renaming .ddf files will break these relations.

**4.3.2 Other tar and zip utilities**

Tar and gzip for DOS, UNIX, and NT operating systems are available in
the GNU utilities from the Free Software Foundation. Software locations
for each operating system listed below.

**4.3.2.1 DOS tar and zip utilities**

Copies of these can be retrieved from

**<ftp://ftpmcmc.er.usgs.gov/pub/sdts/software/tools/>**

These must be run in a DOS window, using the following syntax:

to Unzip: \<path\>\\gunzip \<filename\>

> to Untar: \<path\>\\tar xvf \<filename\>

where \<path\> is the path to the directory that contains the gunzip and
tar executables.

**4.3.2.2 UNIX tar and zip utilities**

Copies of these can be retrieved from

**<ftp://ftpmcmc.er.usgs.gov/pub/sdts/software/tools/>**

**4.3.2.3 NT tar and zip utilities**

Copies of these can be retrieved from

**<ftp://ftpmcmc.er.usgs.gov/pub/sdts/software/tools/>**

5\. Problems Downloading Data

Unfortunately, the USGS does not have the staff to provide tutorial
support for commercial software. If you cannot access these data through
your browser, we recommend either:

\- studying your software documentation for anonymous ftp configuration
procedures, or

\- using an ftp utility instead of a browser.

**5.1 FTP**

Different brands and versions of web browsers implement anonymous ftp
differently. The USGS has tested a number of browsers on the ftp site
described in this paper and believes that the data can be accessed
through most browsers. However, the default browser configurations are
not always correct for anonymous ftp access, and in some cases changing
those configurations is somewhat difficult.

Some sites where ftp utilities can be downloaded and purchased are:

**[http://www.cuteftp.com](http://www.cuteftp.com/)**

**[http://www.ftpx.com](http://www.ftpx.com/)**

As an alternative, users of Win95 or WinNT can ftp data using the Run
function on the Start Menu:

Type **[ftp://edcftp.cr.usgs.gov](ftp://edcftp.cr.usgs.gov/)** and
select **OK.**

Login: anonymous

Password: your e-mail address in the form **name@domain**

From there, make the following choices:
**pub/data/DLG/LARGE_SCALE/[s/]{.underline}san_francisco_north_CA**
(this will get you to the SDTS DLG dataset we used in our example
earlier)

At this point the user is able to navigate to the layer to be
downloaded.

To change the data transfer type to binary, type binary.

To specify which directory the data are to be downloaded to, type lcd
name of directory.

To download the data, type get name of file.

To exit, type quit.

6\. Conclusions and Points of Contact

SDTS DLG and SDTS DEM data are relatively new. The design of these data
could and will be improved for future data production. The data are
usable in their current form if proper procedures are followed and
appropriate software is available.

For more information about SDTS, see
[**http://mcmcweb.er.usgs.gov/sdts**](http://mcmcweb.er.usgs.gov/sdts).
To make comments or ask specific questions about SDTS or the USGS
implementation of SDTS DLG, send email to the SDTS Mailing List at
**<sdts@usgs.gov>.**

For information about DLGv32 Pro, USGS freeware that will display USGS
digital data see **<http://mcmcweb.er.usgs.gov/drc/dlgv32pro/>.**

For more information about USGS Digital Data products, visit
**[http://edc.usgs.gov](http://edc.usgs.gov/).**