2018 International Workshop: A Portal for Japanese Old Maps
Ritsumeikan University
Kyoto

Infrastructure to SupportCollaboration
Ben Lewis
Harvard Center for Geographic Analysis

Center for Geographic Analysis (CGA)• Founded in 2006 by Professor Peter Bol, ahistorian of medieval China• Center created to support research in allthings Geospatial, and available toscholars across the university, all schools
and departments• Based in the Institute for Quantitative
Social Science (IQSS)Small Team Supports Many Projects
CGA Service Project Count by Starting Year (by Dec. 31st 2016)
160
140
120
1008060
402002006

200720082009201020112012

2013

2014

20152016We Develop Platforms to ScaleOurselves• WorldMap (customized geonode)• HHypermap (wrote ourselves)• Warp.worldmap (mapwarper instance)• Dataverse / WorldMap integration• ArcGIS Server (installed instance)• Omeka Neatline (installed instance)• Billion Object Platform (wrote ourselves)• MapD (project started at CGA)

WorldMap In a Nutshell• Designed to lower barriers for researchers
who wish to use geospatial technology• Web-based, cloud hosted• Made available to the world to use• Service oriented architecture• Open source software with a global developercommunity.

Allows researchers to…
• Organize: their own (large) mapping datasets
and share them
• Visualize: maps with data-driven symbology• Publish: data to the world or to just a fewcollaborators• Discover: maps by searching across systems• Mashup / Combine: one’s own data with data
provided by others• Collaborate: by letting several people edit thesame map

Open standards-based input• ESRI REST Image services• OGC WMS map services• OGC Catalog services• ESRI Shapefile, zip compressed or not• GeoTIFF• Styled Layer Descriptors (SLD)Open standards-based output
Service Types:
• WMS
• WFS
Download Formats:• Zipped Shapefile
• KML
• GML
• GeoJSON• CSV• Excel• JPEG• PDF• PNG

Metadata Format:• ISO TC211

Interoperability with ArcGIS
Database• Postgres/PostGISWeb Services• Esri Open REST
• WMS
• WFS

Data Input• Esri Shapefile• GeoTIFF
Data Output• Esri Shapefile• JSON• GeoTIFF• KML, GML

Open but with Access Controls* The Research Lifecycle:• Scoping• Data gathering / exploration• Synthesis / analysis• Writing / communication• Publishing / delivery

The best data is
outside any single organization
Universities,
schools

Libraries,
Archives,
Museums

Scholars outside
the academy

Businesses

Government
agencies

Built on GeoNode
http://geonode.org/• WorldMap is part of the GeoNode development
community which includes:– World Bank
– United Nations World Food Program– European Commission– U.S. State Department– and many other organizations• WorldMap forked from GeoNode in 2012 andafter much work are now reuniting with the main
project, having contributed many new features toGeoNode.

WorldMap software stands on many strong
shouldersContributing Organizations(WorldMap and GeoNode)••WorldMap ••••••Ritsumeikan University – Japanese Old Maps Online
Zhejiang University – Chinese Academic Mapping Platform
Boston Area Research Initiative - BostonMap
Cornell University – Global Health Map
UN University – Wildlife Enforcement Monitoring
Virtue Foundation – Women in the World
Amazon - Hardware
Others…

••••
GeoNode •••World Bank – GFDRR, Dominode, Risiko
U.S. State Dept. – ROGUE, HIU, Syria Damage Assessment
NOAA - GeoCloud
UN World Food Program – WFP Geonode
Australian Govt. - AIFDR, TsuDAT
MapStory Foundation – MapStory AppOthers…

Hosted on a New Public Cloud• Massachusetts Open Cloud (MOC)• Led by Boston University, the MOC is acollaborative effort among– Boston University– Harvard– MIT– University of Massachusettes– Northeastern University– Massachusetts Green High-Performance Computing
Center (MGHPCC)– Oak Ridge National Laboratory (ORNL).

WorldMap Stats• Number of registered users: 23,000• Number of maps: 5,400• Number of local layers: 26,000• Number of remote layers: 90,000• Number of visitors: 1.5 million

The Problem of Search• Millions of geo-services exist a clickaway, but no good way to discoverthem.
• It’s still like the early days of the web
before search engines.

Are Maps a Web or a Haystack?
• Tim Berners Lee chose term “World Wide Web”
because hypertext (HTML) can link any page toany other page and create an information web.
• This enabled Google to create a self-curating
registry using PageRank (which it built an empire
around).
• Web pages point to each other but maps andgeospatial datasets don’t.
• So “geospatial haystack” is a better term.
• This is one reason we don’t yet have good searchengines for geospatial data.

HHypermap: Toward a solution• Platform to create, maintain, and deploy alarge registries of web map services• Support search by time and space• Scalable to millions or billions of objects• Open API to enable any mapping system tofind data and use it• Funded by U.S. National Endowment for theHumanities

Why Focus on Map Service Discovery
Rather Than Data/Metadata Discovery?
• Some advantages:– Everything in system has a live interactive map behind it– All layers can be instantly viewed in context and used atleast for display, overlaid, etc.
– A vast set of layers created by thousands of organizations,containing millions of layers becomes discoverable– As layers are discovered and used, and saved in map views,
crowd curation can begin, a sort of PageRank for maps.

• Some disadvantages such as:– Metadata may be minimal (we are not a library initiative)– The data behind the service may and may not be available– Maps which are not yet georeferenced are not included

Ways to populate a registry• Crawl web to find them – returns many
endpoints but quality varies greatly• User submissions – fewer but endpoints tend
to be more reliableCrawling the web• OGC Services– Look for "?request getcapabilities" and not “test” in the hrefURL• ESRI Rest Services– Look for “/arcgis/rest/services” in the target-DOMAIN-URI of
the WARC Response Header text

• KML or KMZ files– Look for an href URL ending in .kml or .kmz files• Compressed shapefiles– Look for “shape” or “shp” and string ending with “.zip” in thehref URL• Tile Servers– Look for “tile” or “tiles” and string ending with “.png” in the hrefURLSome of the larger known catalogs••••••••••••••David Rumsey Collection
Various mapwarpers
New York Public Library Collection
Geoblacklight - Platform developed by Stanford and other universities to provide
fast search access to geospatial library holdings.
OpenGeoPortal – Platform developed by Tufts and other universities to provide
fast search access to geospatial library holdings.
ArcGIS Open Data – Esri collection of 44,000 open datasets and growing.
Geodata.gov, Geoplatform.gov – The U.S. Federal government has built a data
sharing platform for U.S. data using CKAN software and ArcGIS Online.
INSPIRE Geoportal – Spatial data portal of the European Commission.
GEOSS registry – Group on Earth Observations registry of 850 map service
collections.
Geopole.org – CSW catalogue service providing access to 400,000 layers.
Geonetwork – Geospatial catalogue maintained by the Food and Agriculture
Organization of the United Nations.
Spatineo.com – Commercial service which is currently monitoring 40,000 web
services containing 899,000 layers.
Many CKAN portals
Many Thredds servers

Improving search experience throughvisualization of distributions• Temporal dimension offers huge opportunity
to improve search• Opportunity to provide visibility into thespatial distributions of collections

Taking advantageof the time dimension• Most spatial data describes events in time
though often not explicitly.
• When data does have a time component it isoften not easily accessed.
• An opportunity exists to improve search by:
1. Making latent temporal information explicit
using enrichment techniques
2. Implementing UI/backend enhancements on
existing systems

TimeMiner – to enrich metadata• Temporal metadata for geospatial datasets is often
weak.
• In a crowd-sourced data repository, data creators andcontributors often do not create detailed metadata.
• Many data sets have temporal properties, but time isoften ambiguously defined, mentioned as
unstructured text in the title, abstract, and elsewhere.
• Time is often not referred to using a standard
date/time format such as ISO 8601, but as descriptivetext.

Initial simple TimeMiner Logic
1. Look for date in the date range section of themetadata and choose the earlier date. (Date:from Metadata)
2. If there is no #1 above, look for 4 digit numbers
in title first, then abstract, which are less than orequal to 2018 (present year) (Date: Detected)
3. If there IS a date in #2 above, check to see
whether there is a CE or AD or BCE or BC after it
and apply math accordingly (Date: Detected)
4. If there IS NO #2 above, look for 1, 2, or 3 digit
numbers with associated CE, AD, BCE, BC, andapply math accordingly (Date: Detected)

Another simple technique: historic periods
Example: Chinese dynasty names•••••••••••Xia, Hsia
Shang
Zhou, Chou
Qin, Ch'inHan
SuiTang, T'ang
Song, Sung
Yuan
Ming
Qing, Ch'ing

ca. 2100-1600 BCE
ca. 1600-1050 BCE
ca. 1046-256 BCE
221-206 BCE
206 BCE-220 CE
581-618 CE
618-906
960-1279
1279-1368
1368-1644
1644-1912

Source: http://afe.easia.columbia.edu/timelines/china_timeline.htm

Perhaps a similar approach could be
used for Japanese Periods

Source: https://en.wikipedia.org/wiki/History_of_Japan• animation• anim

BOP Brexit

HHypermap for Exploring SpecificCollections

Layer Page for Kobe 1932

From Layer page one can get to source page on
remote server mapwarper.net

HHypermap Architecture
Built on open sourcesoftware:• Celery• RabbitMQ• Django• Lucene– Solr

• MapProxy• Memcached• OWSLib• PostgreSQL• PostGIS• pycsw

Automatic Georeferencing• These 1300 high quality layers from Stanford weregeoreferenced automatically using software written by Ryo
Kamata at Ritsumeikan U. on the mapwarper platform.

• This is a great example of how ungeoreferenced maps can flow
through an online enrichment pipeline. One can imagine otherforms such as place name extraction.
• This approach has application to the vast collections of large
scale paper map series held by map libraries around the world.

A couple other projects

Dataverse <–> WorldMap Integration

Table Geocoding Service• Dataverse is industrial-strengtharchive for social science data.
• WorldMap is scalable
geovisualization platform.
• Connect them to let researchers
visualize their data spatially withoutleaving the social science platform.

Spatial GPU Database Project• MapD is a database 1000 times faster than astandard RDBMS.
• MapD with origins at CGA in 2012, is now asuccessful startup.
• MapD has open sourced the GPU database and is
working with CGA to extend it.
• The project is happening within the Spatiotemporal Innovation Center composed ofHarvard, George Mason, and U.C. Santa Barbara.
• We will focus initially on testing hydrologic
models which have very large outputs.

In Conclusion• The amount of historical spatial informationon the web is large and continues to grow• But it is not well handled by traditional searchengines• It is not possible to perform a quick search
and find a particular dataset (or determine
one does not exist)• This lack of discoverability remains a major
challenge for our community but is one that issolvable if we continue to work together on it.

Some Links…
• Center for Geographic Analysis– http://gis.harvard.edu• WorldMap– http://worldmap.Harvard.edu– https://github.com/cga-harvard/worldmap• HHypermap– http://hhypermap.worldmap.harvard.edu– https://github.com/cga-harvard/HHypermap• Dataverse– https://dataverse.harvard.edu/– https://github.com/IQSS/dataverse• Open Geoportal http://opengeoportal.org• Geoblacklight http://geoblacklight.org/

Thank you
Benjamin Lewis
blewis@cga.harvard.edu

http://cga.harvard.edu
http://worldmap.harvard.edu