UESPWiki:Oblivion Map Design

This article covers various aspects of the design of UESP's online Oblivion map using the Google Maps API. This is currently a functional  , although improvements continue to be made.
 
The Basic Map
Google Maps With An Oblivion Map
By default Google Maps uses its own tile server to serve up images of various types from Earth (satellite, road maps, etc...). We will have to use a custom map in order to display the Oblivion map data instead of Earths. This tutorial (archived) on custom maps shows the basic steps needed. The main thing is the addition of a custom tile function which translates the requested coordinates and zoom level into a tile image, for example:

  CustomGetTileUrl = function(Coordinate, Zoom) {
       return "tile_" + Coordinate.x + "_" + Coordinate.y + "_" + Zoom + ".jpg"
  }
  
If the input coordinate was (22, 44) and zoom was 3, this function would return: tile_22_44_3.jpg

The input coordinate and zoom values are described as follows:
:* Coordinate origin at (0,0) with increasing values to the right and down.
:* Zoom levels start at 0 (the farthest zoomed out) and increase from there (arbitrary limit, as many as needed).
:* The expected number of tiles per zoom level is: 2 . So at zoom level 0 there should be only 1 tile while at zoom level 9 the map is split into 512x512 tiles.
:* If using the Mercator projection the tile origin (0,0) is at coordinate (90,-180). There are projection methods to convert to/from tile and latitude/longitude coordinates.

Creating the Oblivion Map
Using the information in the previous section on how the Google Maps API works is possible to figure out how to arrange our Oblivion map tiles in order to get them assembled into a map image. This will quickly become the most complicated step in the process of making a working map. Basic steps in creating the Oblivion map tiles are listed below:

Export Local Map Data
: Using the Create Local Maps command in the World menu of the Construction Set. Select Exterior Cells and choose Tamriel (we may have to export the other exterior cells later). The CS seems to have a problem exporting map data as it crashes as soon as it runs out of RAM. For example, on my machine with 2GB of RAM I could generally only export 500 cells or so before it crashed. This forces us to export the map in sections by using the NW CellX, NW CellY, SE CellX and SE CellY fields. The overall cell limits in Oblivion are (-64,59) in the NW to (69,-69) in the SE (134x129 cells). Note that a good amount of those cells are outside of the 'real' Oblivion area (as much as half perhaps). I exported cells going from left to right in manageable sections (i.e., from (-64, 59) to (-60, -69)). If the CS crashes make sure you are not missing any tiles. 

Exported Map Tiles
:The exported map data will be saved as uncompressed 256x256 DDS files in the Data\textures\Maps\Oblivion folder of your Oblivion path (unsure if you have to create this before hand or not). The output filename format is:
   [CellName].[CellX].[CellY].dds
 
   ex: Tamriel.-23.01.dds
:The cell values are a minimum of 2 characters in size, so 1 will be output zero padded as 01 while -1 will be output as just -1 without padding. If you output the entire range of cells you'll end up with 134x129=17286 images around 256kb in size each.
:The output images actually show slightly more than a complete cell each as they overlap with adjacent cells as follows:
::*Right edge contains 15 pixels of overlap with the cell to the right.
::*Top edge contains 14 pixels of overlap with the cell to the top.
::*Bottom edge contains 1 pixel of overlap with the cell to the bottom.
::*Left edge has no overlap
:If you eliminate this overlap you'll be left with a 241x241 image that will show the cell border on all 4 edges. It may be necessary to remove one pixel from the right and top edges to end up with a 240x240 image although the 241x241 image seems to tile fine. 

Converting Map Tiles
:A number of conversions steps must be performed in order to get the exported map tiles into a format compatible with the Goggle Maps API.
::#Convert to JPG with a quality level around 20%. This reduces the average image size from 256kb to 17kb. You can compress higher but it begins to degrade the image noticeably past 20%.
::#Trim the cell overlap in each image ending up with a 241x241 (or 240x240) image.
::#Resize each image back to 256x256. We could cut/paste images to get the 256x256 size needed by the API but it is much faster to just resize the images. Resizing also smooths out imperfections in the image and make it look slightly better anyways.
::#Rename the file to match that needed by the API. The Y coordinates between Oblivion and Goggle Maps are reversed and we need to set the origin to a non-negative value. For simplicity we'll set the origin to (0,0) although this may change later. 
:At this point we should have necessary images to test the map at the largest zoom level. We can test this locally by simply setting the custom map tile function to point to the images on our hard drive. 

Generating Zoom Levels
:We have got the highest zoom level from manipulating the exported Oblivion map tiles but we have to now generate all lower zoom level map tiles as well. Since zooms are just a power of two this is relative simple: just combine 4 adjacent map tiles and reduce the image size by 50%. Things only get tricky at the right and bottom edges since our overall tile size (134x129) is not a perfect power of two. This can be solved by using a default image, like an empty water tile, for any missing image. We just keep repeating the zoom process until we are left with a 1x1 map tile size.:
:For convienence we'll place each zoom level in its own directory using a tile filename format like: zoom15/tamriel-34-24-15.jpg. 

In the end we should have some 23,000 images in 8 different zoom levels totalling some 400MB which will allow us to make a working map able to zoom in and out.

Simple Test Map
The basic page for displaying the Oblivion map is shown below:
 
Things to note in this version:
:* The zoom levels of 9 (zoomed out) to 16 (maximum zoom) are used. 
:* Map tile origin is at (0,0)
:* The default projection (Mercator) results in some strange coordinates due to our choice of tile origin. The center of the map is roughly at (85.021, -179.6). The latitude coordinate (85.021) is particularly sensitive since we're near the North pole with this projection.
:* No checks for valid coordinates in the custom tile function will result in blank (gray) areas where image files are missing.

Flat Projection
As seen in the test map example we run into some minor problems due to the default mercator projection used by the Google Maps API. This is a standard projection for displaying flat maps of a spherical object, like Earth. Since the Oblivion game map is not based on a spherical world a flat map projection would be better. We'll use this tutorial which gives an implementation for the Euclidean map projection type. 

Using a Euclidean projection we find a more reasonable behavior of the Y coordinate (latitude). In the previous Mercator projection the map extents were roughly (84.99,-180) to (85.05, -179.25). Note the very small latitude range due to the proximity of the North pole. In the Euclidean projection our map extents are a more reasonable, and square, (89.62, -180) to (90, -179.25).

Another alternative is to place the map tile origin closer to the center of the map. While this is possible it complicates the tile naming since each zoom level will have a different origin index. However, we will very likely be using a non-zero tile origin in order to add other continents of Tamriel (like Daggerfall and Morrowind).

More Features
There are a number of other map features that are useful as described below:

Map Bounds
:We're already limiting the zoom levels by creating a custom map tile layer but we still need to limit the latitude/longitude to keep within the Oblivion map bounds. The custom map tutorial includes a section on how to accomplish this using a move listener event. There are two additional lines needed in the checkBounds function since we're near the origin:
   if (X > 0.0)   {X = AminX;}
   if (Y  = 9 && Zoom  = 0 && Coordinate.y >= 0) {
 	     return "zoom" + Zoom + "/tamriel-" + Coordinate.x + "-" + Coordinate.y + "-" + Zoom + ".jpg"; 
 	}
    }
    return "oorimage.jpg";
  };
:The oorimage.jpg is a 256x256 image to be displayed for any map tile outside of the Oblivion map bounds.

More Complete Map Example
See the source code for a complete map using the features explained above. The map itself is working well and now we just need to add some actual location markers to it.

Map Locations
Required Data
In order to show locations on the map the following data will be needed:
:* Location name
:* Oblivion map location (X/Y/Z)
:* Cell X/Y (may not be needed if it can be derived from the location X/Y)
:* Description (may be needed for some locations)
:* Editor and form ID (probably not required)
:* Location type (dungeon, house, landmark, shrine, etc...)
:* Latitude/Longitude (can likely be derived from the location X/Y)
Most of this information can be directly exported from Oblivion.esm using a custom utility (the Construction Set cannot export any of this information).

Exporting Data From Oblivion
The relevant location data needed from Oblivion.esm is found deep within the top level WRLD group.
WRLD Group
WRLD Children (match by FormID of world space, Tamriel=0x3C) 
CELL Child Group 
CELL Persist Group
REFR records for persistent locations (ones with XTEL subrecords)
Exterior Cell Groups
Exterior Sub-Cell Groups
CELL Children Groups
Persistent Children Group
REFR location records (ones with XTEL subrecords)
Temporary Children Group
REFR location records (ones with XTEL subrecords)

The location information is stored within a RERF (reference) record that has a XTEL (teleport) subrecord which gives the destination location.

REFR Record
NAME Subrecord (form ID)
EDID Subrecord (editor ID)
FULL Subrecord (full location name)
XTEL Subrecord (teleport information)
Destination Door Reference ID (formid)
Destination Location (float X, Y, Z)
Destination Angle in Radians (float AX, AY, AZ) 
DATA Subrecord (reference data)
Location (float X, Y, Z)
Angle in Radians (float AX, AY, AZ)

Exported Location Data
The exterior location data exported from Oblivion.esm using the ObEdit utility looks like (some columns removed):



Exporting exterior locations from all worldspaces gives us 920 records.

Map Markers
It turns out that Oblivion actually uses special map marker references to display the icons on the map. These markers are references of the MapMarker (0x10) static object and contain a full location name (FULL subrecord) as well as a location type (TNAM subrecord). Marker references are identified by the empty XMRK subrecord.

There are 451 exterior map markers in Oblivion.esm although some of these will duplicate locations exported using the teleport information. We can merge the two sets of location data by looking for locations very close to each other (within 1500 units) with the same, or similar, names.

Overall Design
There are a number of methods possible for serving up the map locations to us.

:*Internal JavaScript -- The locations are stored in an array in the JavaScript format on the actual map page. This is the simplest design and would work well for a small number of locations. Unfortunately, once the number of locations begins to increase it will result in a very large and hard to maintain file. Since we will likely have 1000s of locations in the future this is not a good design for us.
:*PHP Using Data Files -- This design has the map script query a PHP script on the uesp.net server for a list of locations within a particular area. The PHP script reads locations from a data file (XML, or CSV for example) and responds with the appropriate XML data. The map script then parses this data to get the marker information to display. This greatly increases the complexity since we're now dealing with ansynchronous data. It has the advantage of keeping the location data stored on the server which makes it easier to manage and only the necessary location data is transferred. This allows us to have virtually any amount of location data.
:*PHP Using Database -- The same as the previous method but using a database (MySQL) for storing the location data. This simplifies the PHP somewhat as we don't have to worry about the performance scaling of the file format used for the location data. This is the method we'll be using.

Table Design
The following are a description of the minimum fields needed for storing the location information.



Basic PHP Location Script
A simple PHP script for accessing the locations in the database and returning them in a XML format is not too complex (taken from MapKI):
 

This script simply returns all the locations in the database which is fine for testing purposes. A more complete version will only return locations within a particular map area or locations matching an input search string.

Converting Oblivion Locations
The Oblivion map data uses its own coordinate system while the map uses latitudes and longitudes. In order to display the locations on the map we will have to convert between the two systems. We'll need a number of settings in order to get the conversion functions:
:* Map tile origin (currently set to (0,0))
:* Number of map tiles at maximum zoom (134 x 129)
:* Map projection (currently using Euclidean)
:* Zoom levels used (currently 9 to 16)
:* Extents of the map tiles in Oblivion coordinates
::* Upper Left (-64,59) = (-262145, 245759) = 4096*(-64, 60)
::* Lower Right (69,-69) = (286719, -282625) = 4096*(70, -69)
:* Oblivion cell size (4096 Oblivion units)
:* Map bounds in Latitude/Longitude (not as accurate but makes it easier)
::* Lower Left  = (89.62,-179.9999)
::* Upper Right = (89.9999, -179.25)

Using these values the conversion functions are:

     function ConvertObLocToLng(X, Y) {
          return -180.0 + ((X/4096.0 + 64)/65536.0 * 360.0);
     }
     
     function ConvertObLocToLat(X, Y) {
          return 90.0 - ((59.0 - Y/4096.0)/65536.0 * 180.0);
     }

Basic Map Script Changes
The map script needs some additions in order to request the location data from the PHP script and read the resulting location XML data (taken from MapKI):
 

All this function does is to request all locations from the PHP script when it is called and to display them once they are received.

Versions
Software Updates
v0.3.1, 6 July 2008
:* All three UESP maps completely moved to new, merged code
:* Minor improvements:
:** Accept locx+locy as input instead of oblocx+oblocy
:** "Link to this location" now provides URL using locx+locy instead of lat+long
:** Zoom optional, instead of required, for locx+locy links (defaults to umMapLinkZoomedValue for all URLs where other params are provided)

v0.3.0, 23 June 2008
:* Merged code for all three UESP google API maps:  ,  , and  .
:**  
:** This also means that although the map editors for all three maps are now functional, any edits made to the Oblivion map at  will not take effect on the pre-existing Oblivion map (changes made at  partially take effect on the pre-existing SI map).
:* Fixed problem with Safari browser failing to view map
:* Introduced alphabetization to the location list
:* Introduced ability to customize label positions relative to map markers, and changed default position to middle right
:* Added SearchTags parameter
:* Added new categories for map markers (although new icons needed for most categories)

v0.2.1, 28 August 2006
:* Added 263 Nirnroot locations (display level 15, marker type 13)

v0.2.0, 23 August 2006
:* Added the centeron parameter to the map page which allows one to quickly zoom to a named location.
:* Added info popup windows when a location is clicked.
:* Added the WikiPage field to locations. Initial values for all locations have been set to the location name for all locations that have a matching wiki page (about half of them).

v0.1.1, 12 August 2006
:* Added smarter location loading handling. Loaded locations now stay on the map area unless they go outside the current visible area. This reduces the number of locations that have to be loaded each time the map is panned or zoomed.
:* Added 'divs' for the search results header and footer.

v0.1.0, 11 August 2006
:* First release

Data Updates
13 January 2008: general cleanup of SI map markers --Nephele 
Disabled all map markers for "Dementia Garden," "Mania Garden," and ordered Fringe worlds.
Disabled extra markers for Aichan Prison and Corpserot Prison (i.e., the teleportation markers that never appear on player's map).
Set DisplayLevel = 17 for all extra doors on houses.
Fixed most (if not all) wiki page links to now point to valid wiki pages.
Adjusted DisplayLevel for many other markers to make marker distribution more uniform and less cluttered at most zoom levels (most doors moved to zoom=16; settlements moved to zoom=13).
13 January 2008: updated all Fringe and New Sheoth tiles for the SI map.  Now using using higher resolution images from individual worlds, as created by Timenn --Nephele 
11 January 2008: added all exterior Obelisks of Order and Crystal Chests to SI map --Nephele 

General Map
The original version of the OB map was specifically created for OB.  As of version 0.3, the map code has been generalized so that it can be reused for all desired maps.  Specifications for the Morrowind map are provided at Morrowind Map Design.

Isolating Game Specifics
The following items are specific to the map being displayed:
:* Tiles -- The simplest method would simply override the custom map tile function used by the Google maps API.
:* Maps Bounds -- Both in game cells/coordinates, zoom levels, and in map latitude/longitudes.
:* Coordinate Conversion -- Converting to/from game coordinates to map latitude/longitudes.
:* Location Database -- Each map will likely use a different database/table of locations and which may possibly contain varied data.
:* Images -- Including 'null' images and map icons.
:* Marker Types -- Types of map locations will change from game to game.
:* Link to Wiki -- Default namespace and options to override default namespace.
:* Map Labels -- May require updating to display game specific data.

Oblivion Map Specifications
Marker Types


Shivering Isles Map Specifications
Worldspaces
The following worldspaces are to be considered as exterior in SI:
:* Bliss
:* Crucible
:* Dementia Garden
:* Mania Garden
:* Palace Grounds
:* Realm of Sheogorath
:* The Fringe

Marker Types

† This icon is only used on the in-game Shivering Isles map for the ordered version of the Door to Cyrodiil.  Although the in-game icon is probably a bug, might as well make it available for that one icon if desired.

ToDo
:* Copyright license
:* Proper marker types for all locations
:* More marker types? (guilds, store types)
:* Distinguish between door and map markers (eliminate map markers?)

Other Stuff
:* Initial Notes -- Some initial notes on the Google maps API
:* ConvertNames.py -- Python script to convert CS exported filenames to one used by the online map system.
:* CreateZoomLevels.py -- Python script to create all zoom levels from the base export.
:* ResizeObMap.pspScript -- PaintShopPro v8.0 macro for resizing the base layer map tiles to the proper size.
:* getmaplocs.php -- Full script for retrieving the map locations from the database.
:* Database Table -- Current MySQL map location table structure.