System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface

A system includes a visualization manager for assembling the group of data elements and for assigning a connection visual element in the visual representation between a first visual element representing a first data element of the group and a second visual element representing a second data element of the group. The system also has a spatial visualization component configured for generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions. The reference surface is for relating the first visual element to a first location of interest in the spatial reference frame and for relating the second visual element to a second location of interest in the spatial reference frame. The system also has a temporal visualization component configured for generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest. The temporal domain includes a first time track, such as a timeline, coupled to the first location of interest and a second time track coupled to the second location of interest, such that the first visual element is positioned on the first time track and the second visual element is positioned on the second time track. Each of the time tracks are configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface.

BACKGROUND OF THE INVENTION

The present invention relates to an interactive visual presentation of multidimensional data on a user interface.

Tracking and analyzing entities and streams of events, has traditionally been the domain of investigators, whether that be national intelligence analysts, police services or military intelligence. Business users also analyze events in time and location to better understand phenomenon such as customer behavior or transportation patterns. As data about events and objects become more commonly available, analyzing and understanding of interrelated temporal and spatial information is increasingly a concern for military commanders, intelligence analysts and business analysts. Localized cultures, characters, organizations and their behaviors play an important part in planning and mission execution. In situations of asymmetric warfare and peacekeeping, tracking relatively small and seemingly unconnected events over time becomes a means for tracking enemy behavior. For business applications, tracking of production process characteristics can be a means for improving plant operations. A generalized method to capture and visualize this information over time for use by business and military applications, among others, is needed.

Many visualization techniques and products for analyzing complex event interactions only display information along a single dimension, typically one of time, geography or a network connectivity diagram. Each of these types of visualizations is common and well understood. For example a Time-focused scheduling chart such as Microsoft (MS) Project displays various project events over the single dimension of time, and a Geographic Information System (GIS) product, such as MS MapPoint, or ESRI ArcView, is good for showing events in the single dimension of locations on a map. There are also link analysis tools, such as Netmap (www.netmapanalytics.com) or Visual Analytics (www.visualanalytics.com) that display events as a network diagram, or graph, of objects and connections between objects. Some of these systems are capable of using animation to display another dimension, typically time. Time is played back, or scrolled, and the related spatial image display changes to reflect the state of information at a moment in time. However this technique relies on limited human short term memory to track and then retain temporal changes and patterns in the spatial domain. Another visualization technique called “small multiples” uses repeated frames of a condition or chart, each capturing an increment moment in time, much like looking at sequence of frames from a film laid side by side. Each image must be interpreted separately, and side-by-side comparisons made, to detect differences. This technique is expensive in terms of visual space since an image must be generated for each moment of interest, which can be problematic when trying to simultaneously display multiple images of adequate size that contain complex data content.

A technique has been developed, as described in Interactive Visualization of Spatiotemporal Patterns using Spirals on a Geographical Map—by Hewagamage et al. that uses spiral shaped ribbons as timelines to show isolated sequences of events that have occurred at discrete locations on a geographical map. This technique is limited because it uses spiral timelines exclusively to show the periodic quality of certain types of events, while does not show connectivity between the temporal and spatial information of data objects at multi-locations within the spatial domain. Further, event data objects placed on the spirals can suffer from occlusion, thereby providing for only a limited number of events and locations viewable with the spiral timelines.

It is an object of the present invention to provide a system and method for the integrated, interactive visual representation of a plurality of events and objects with spatial and temporal properties to obviate or mitigate at least some of the above-mentioned disadvantages.

SUMMARY OF THE INVENTION

Tracking and analyzing entities and streams of events, has traditionally been the domain of investigators, whether that be national intelligence analysts, police services or military intelligence. Business users also analyze events in time and location to better understand phenomenon such as customer behavior or transportation patterns. As data about events and objects become more commonly available, analyzing and understanding of interrelated temporal and spatial information is increasingly a concern for military commanders, intelligence analysts and business analysts. Contrary to present analysis tools, a system and method is provided for creating a multidimensional visual representation of a group of data elements having integrated temporal and spatial properties. The data elements are included in the visual representation as corresponding visual elements, such that the data elements of the group linked by at least one association. The system includes a visualization manager for assembling the group of data elements using the at least one association and for assigning a connection visual element in the visual representation between a first visual element representing a first data element of the group and a second visual element representing a second data element of the group. The system also has a spatial visualization component, such as a sprite, configured for generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions. The reference surface is for relating the first visual element to a first location of interest in the spatial reference frame and for relating the second visual element to a second location of interest in the spatial reference frame. The system also has a temporal visualization component, such as a sprite, configured for generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest. The temporal domain includes a first time track, such as a timeline, coupled to the first location of interest and a second time track coupled to the second location of interest, such that the first visual element is positioned on the first time track and the second visual element is positioned on the second time track. Each of the time tracks configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface. In implementation of the method, the connection visual element represents a distributed association in at least one of the domains between the first visual element and the second visual element such that the visual representation is displayed on a user interface for subsequent interaction with user events, including animation of the visual elements to help in the analysis of the data contained in the visual representation.

According to the present invention there is provided a method for creating a multidimensional visual representation of a group of data elements having integrated temporal and spatial properties, the data elements being included in the visual representation as corresponding visual elements, the data elements of the group linked by at least one association, the method comprising the steps of: assembling the group of data elements using the at least one association; generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions, the reference surface for relating a first visual element representing a first data element of the group to a first location of interest in the spatial reference frame and relating a second visual element representing a second data element of the group to a second location of interest in the spatial reference frame; generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest, the temporal domain including a first time track coupled to the first location of interest and a second time track coupled to the second location of interest, the first visual element positioned on the first time track and the second visual element positioned on the second time track, each of the time tracks configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface; and assigning a connection visual element in the visual representation between the first visual element and the second visual element, the connection visual element for representing a distributed association in at least one of the domains between the first visual element and the second visual element; wherein the visual representation is displayed on a user interface for subsequent interaction with user events.

According to a further aspect of the present invention there is provided a system for creating a multidimensional visual representation of a group of data elements having integrated temporal and spatial properties, the data elements being included in the visual representation as corresponding visual elements, the data elements of the group linked by at least one association, the system comprising: a visualization manager for assembling the group of data elements using the at least one association and for assigning a connection visual element in the visual representation between a first visual element representing a first data element of the group and a second visual element representing a second data element of the group; a spatial visualization component configured for generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions, the reference surface for relating the first visual element to a first location of interest in the spatial reference frame and relating the second visual element to a second location of interest in the spatial reference frame; and a temporal visualization component configured for generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest, the temporal domain including a first time track coupled to the first location of interest and a second time track coupled to the second location of interest, the first visual element positioned on the first time track and the second visual element positioned on the second time track, each of the time tracks configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface; and wherein the connection visual element represents a distributed association in at least one of the domains between the first visual element and the second visual element such that the visual representation is displayed on a user interface for subsequent interaction with user events.

According to a still further aspect of the present invention there is provided a computer program product for creating a multidimensional visual representation of a group of data elements having integrated temporal and spatial properties, the data elements being included in the visual representation as corresponding visual elements, the data elements of the group linked by at least one association, the computer program product comprising: a computer readable medium; a visualization module stored on the computer readable medium for assembling the group of data elements using the at least one association and for assigning a connection visual element in the visual representation between a first visual element representing a first data element of the group and a second visual element representing a second data element of the group; a spatial visualization module stored on the computer readable medium for generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions, the reference surface for relating the first visual element to a first location of interest in the spatial reference frame and relating the second visual element to a second location of interest in the spatial reference frame; and a temporal visualization module stored on the computer readable medium for generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest, the temporal domain including a first time track coupled to the first location of interest and a second time track coupled to the second location of interest, the first visual element positioned on the first time track and the second visual element positioned on the second time track, each of the time tracks configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface; wherein the connection visual element represents a distributed association in at least one of the domains between the first visual element and the second visual element such that the visual representation is displayed on a user interface for subsequent interaction with user events.

It is noted that similar references are used in different figures to denote similar components.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description of the embodiments of the present invention does not limit the implementation of the invention to any particular computer programming language. The present invention may be implemented in any computer programming language provided that the OS (Operating System) provides the facilities that may support the requirements of the present invention. A preferred embodiment is implemented in the Java computer programming language (or other computer programming languages in conjunction with C/C++). Any limitations presented would be a result of a particular type of operating system, computer programming language, or data processing system and would not be a limitation of the present invention.

Visualization Environment

Referring toFIG. 1, a visualization data processing system100includes a visualization tool12for processing a collection of data objects14as input data elements to a user interface202. The data objects14are combined with a respective set of associations16by the tool12to generate an interactive visual representation18on the visual interface (VI)202. The data objects14include event objects20, location objects22, images23(seeFIG. 20) and entity objects24, as further described below. The set of associations16include individual associations26that associate together various subsets of the objects20,22,23,24, as further described below. Management of the data objects14and set of associations16are driven by user events109of a user (not shown) via the user interface108(seeFIG. 2) during interaction with the visual representation18. The representation18shows connectivity between temporal and spatial information of data objects14at multi-locations within the spatial domain400(seeFIG. 4).

Data Processing System

Referring toFIG. 2, the data processing system100has a user interface108for interacting with the tool12, the user interface108being connected to a memory102via a BUS106. The interface108is coupled to a processor104via the BUS106, to interact with user events109to monitor or otherwise instruct the operation of the tool12via an operating system110. The user interface108can include one or more user input devices such as but not limited to a QWERTY keyboard, a keypad, a trackwheel, a stylus, a mouse, and a microphone. The visual interface202is considered the user output device, such as but not limited to a computer screen display. If the screen is touch sensitive, then the display can also be used as the user input device as controlled by the processor104. Further, it is recognized that the data processing system100can include a computer readable storage medium46coupled to the processor104for providing instructions to the processor104and/or the tool12. The computer readable medium46can include hardware and/or software such as, by way of example only, magnetic disks, magnetic tape, optically readable medium such as CD/DVD ROMS, and memory cards. In each case, the computer readable medium46may take the form of a small disk, floppy diskette, cassette, hard disk drive, solid-state memory card, or RAM provided in the memory102. It should be noted that the above listed example computer readable mediums46can be used either alone or in combination.

Referring again toFIG. 2, the tool12interacts via link116with a VI manager112(also known as a visualization renderer) of the system100for presenting the visual representation18on the visual interface202. The tool12also interacts via link118with a data manager114of the system100to coordinate management of the data objects14and association set16from data files or tables122of the memory102. It is recognized that the objects14and association set16could be stored in the same or separate tables122, as desired. The data manager114can receive requests for storing, retrieving, amending, or creating the objects14and association set16via the tool12and/or directly via link120from the VI manager112, as driven by the user events109and/or independent operation of the tool12. The data manager114manages the objects14and association set16via link123with the tables122. Accordingly, the tool12and managers112,114coordinate the processing of data objects14, association set16and user events109with respect to the content of the screen representation18displayed in the visual interface202.

Tool Information Model

Referring toFIG. 1, a tool information model is composed of the four basic data elements (objects20,22,23,24and associations26) that can have corresponding display elements in the visual representation18. The four elements are used by the tool12to describe interconnected activities and information in time and space as the integrated visual representation18, as further described below.

Event Data Objects20

Events are data objects20that represent any action that can be described. The following are examples of events;

Bill was at Toms house at 3 pm,

Tom phoned Bill on Thursday,

A tree fell in the forest at 4:13 am, Jun. 3, 1993 and

Tom will move to Spain in the summer of 2004.

The Event is related to a location and a time at which the action took place, as well as several data properties and display properties including such as but not limited to; a short text label, description, location, start-time, end-time, general event type, icon reference, visual layer settings, priority, status, user comment, certainty value, source of information, and default+user-set color. The event data object20can also reference files such as images or word documents.

Locations and times may be described with varying precision. For example, event times can be described as “during the week of January 5th” or “in the month of September”. Locations can be described as “Spain” or as “New York” or as a specific latitude and longitude.

Entity Data Objects24

Entities are data objects24that represent any thing related to or involved in an event, including such as but not limited to; people, objects, organizations, equipment, businesses, observers, affiliations etc. Data included as part of the Entity data object24can be short text label, description, general entity type, icon reference, visual layer settings, priority, status, user comment, certainty value, source of information, and default+user-set color. The entity data can also reference files such as images or word documents. It is recognized in reference toFIGS. 6 and 7that the term Entities includes “People”, as well as equipment (e.g. vehicles), an entire organization (e.g. corporate entity), currency, and any other object that can be tracked for movement in the spatial domain400. It is also recognized that the entities24could be stationary objects such as but not limited to buildings. Further, entities can be phone numbers and web sites. To be explicit, the entities24as given above by example only can be regarded as Actors

Location Data Objects22

Locations are data objects22that represent a place within a spatial context/domain, such as a geospatial map, a node in a diagram such as a flowchart, or even a conceptual place such as “Shang-ri-la” or other “locations” that cannot be placed at a specific physical location on a map or other spatial domain. Each Location data object22can store such as but not limited to; position coordinates, a label, description, color information, precision information, location type, non-geospatial flag and user comments.

Associations

Event20, Location22and Entity24are combined into groups or subsets of the data objects14in the memory102(seeFIG. 2) using associations26to describe real-world occurrences. The association is defined as an information object that describes a pairing between 2 data objects14. For example, in order to show that a particular entity was present when an event occurred, the corresponding association26is created to represent that Entity X “was present at” Event A. For example, associations26can include such as but not limited to; describing a communication connection between two entities24, describing a physical movement connection between two locations of an entity24, and a relationship connection between a pair of entities24(e.g. family related and/or organizational related). It is recognised that the associations26can describe direct and indirect connections. Other examples can include phone numbers and web sites.

A variation of the association type26can be used to define a subclass of the groups27to represent user hypotheses. In other words, groups27can be created to represent a guess or hypothesis that an event occurred, that it occurred at a certain location or involved certain entities. Currently, the degree of belief/accuracy/evidence reliability can be modeled on a simple 1-2-3 scale and represented graphically with line quality on the visual representation18.

Image Data Objects23

Standard icons for data objects14as well as small images23for such as but not limited to objects20,22,24can be used to describe entities such as people, organizations and objects. Icons are also used to describe activities. These can be standard or tailored icons, or actual images of people, places, and/or actual objects (e.g. buildings). Imagery can be used as part of the event description. Images23can be viewed in all of the visual representation18contexts, as for example shown inFIGS. 20 and 21which show the use of images23in the time lines422and the time chart430views. Sequences of images23can be animated to help the user detect changes in the image over time and space.

Annotations21in Geography and Time (seeFIG. 22) can be represented as manually placed lines or other shapes (e.g. pen/pencil strokes) can be placed on the visual representation18by an operator of the tool12and used to annotate elements of interest with such as but not limited to arrows, circles and freeform markings. Some examples are shown inFIG. 21. These annotations21are located in geography (e.g. spatial domain400) and time (e.g. temporal domain422) and so can appear and disappear on the visual representation18as geographic and time contexts are navigated through the user input events109.

Referring toFIG. 3, the visualization tool12has a visualization manager300for interacting with the data objects14for presentation to the interface202via the VI manager112. The Data Objects14are formed into groups27through the associations26and processed by the Visualization Manager300. The groups27comprise selected subsets of the objects20,21,22,23,24combined via selected associations26. This combination of data objects14and association sets16can be accomplished through predefined groups27added to the tables122and/or through the user events109during interaction of the user directly with selected data objects14and association sets16via the controls306. It is recognized that the predefined groups27could be loaded into the memory102(and tables122) via the computer readable medium46(seeFIG. 2). The Visualization manager300also processes user event109input through interaction with a time slider and other controls306, including several interactive controls for supporting navigation and analysis of information within the visual representation18(seeFIG. 1) such as but not limited to data interactions of selection, filtering, hide/show and grouping as further described below. Use of the groups27is such that subsets of the objects14can be selected and grouped through associations26. In this way, the user of the tool12can organize observations into related stories or story fragments. These groupings27can be named with a label and visibility controls, which provide for selected display of the groups27on the representation18, e.g. the groups27can be turned on and off with respect to display to the user of the tool12.

The Visualization Manager300processes the translation from raw data objects14to the visual representation18. First, Data Objects14and associations16can be formed by the Visualization Manager300into the groups27, as noted in the tables122, and then processed. The Visualization Manager300matches the raw data objects14and associations16with sprites308(i.e. visual processing objects/components that know how to draw and render visual elements for specified data objects14and associations16) and sets a drawing sequence for implementation by the VI manager112. The sprites308are visualization components that take predetermined information schema as input and output graphical elements such as lines, text, images and icons to the computers graphics system. Entity24, event20and location22data objects each can have a specialized sprite308type designed to represent them. A new sprite instance is created for each entity, event and location instance to manage their representation in the visual representation18on the display.

The sprites308are processed in order by the visualization manager300, starting with the spatial domain (terrain) context and locations, followed by Events and Timelines, and finally Entities. Timelines are generated and Events positioned along them. Entities are rendered last by the sprites308since the entities depend on Event positions. It is recognised that processing order of the sprites308can be other than as described above.

The Visualization manager112renders the sprites308to create the final image including visual elements representing the data objects14and associates16of the groups27, for display as the visual representation18on the interface202. After the visual representation18is on the interface202, the user event109inputs flow into the Visualization Manager, through the VI manager112and cause the visual representation18to be updated. The Visualization Manager300can be optimized to update only those sprites308that have changed in order to maximize interactive performance between the user and the interface202.

Layout of the Visualization Representation18

The visualization technique of the visualization tool12is designed to improve perception of entity activities, movements and relationships as they change over time in a concurrent time-geographic or time-diagrammatical context. The visual representation18of the data objects14and associations16consists of a combined temporal-spatial display to show interconnecting streams of events over a range of time on a map or other schematic diagram space, both hereafter referred to in common as a spatial domain400(seeFIG. 4). Events can be represented within an X,Y,T coordinate space, in which the X,Y plane shows the spatial domain400(e.g. geographic space) and the Z-axis represents a time series into the future and past, referred to as a temporal domain402. In addition to providing the spatial context, a reference surface (or reference spatial domain)404marks an instant of focus between before and after, such that events “occur” when they meet the surface of the ground reference surface404.FIG. 4shows how the visualization manager300(seeFIG. 3) combines individual frames406(spatial domains400taken at different times Ti407) of event/entity/location visual elements410, which are translated into a continuous integrated spatial and temporal visual representation18. It should be noted connection visual elements412can represent presumed location (interpolated) of Entity between the discrete event/entity/location represented by the visual elements410. Another interpretation for connections elements412could be signifying communications between different Entities at different locations, which are related to the same event as further described below.

Referring toFIG. 5, an example visual representation18visually depicts events over time and space in an x, y, t space (or x, y, z, t space with elevation data). The example visual representation18generated by the tool12(seeFIG. 2) is shown having the time domain402as days in April, and the spatial domain400as a geographical map providing the instant of focus (of the reference surface404) as sometime around noon on April 23—the intersection point between the timelines422and the reference surface404represents the instant of focus. The visualization representation18represents the temporal402, spatial400and connectivity elements412(between two visual elements410) of information within a single integrated picture on the interface202(seeFIG. 1). Further, the tool12provides an interactive analysis tool for the user with interface controls306to navigate the temporal, spatial and connectivity dimensions. The tool12is suited to the interpretation of any information in which time, location and connectivity are key dimensions that are interpreted together. The visual representation18is used as a visualization technique for displaying and tracking events, people, and equipment within the combined temporal and spatial domains402,400display. Tracking and analyzing entities24and streams has traditionally been the domain of investigators, whether that be police services or military intelligence. In addition, business users also analyze events20in time and spatial domains400,402to better understand phenomenon such as customer behavior or transportation patterns. The visualization tool12can be applied for both reporting and analysis.

The visual representation18can be applied as an analyst workspace for exploration, deep analysis and presentation for such as but not limited to:Situations involving people and organizations that interact over time and in which geography or territory plays a role;Storing and reviewing activity reports over a given period. Used in this way the representation18could provide a means to determine a living history, context and lessons learned from past events; andAs an analysis and presentation tool for long term tracking and surveillance of persons and equipment activities.

The visualization tool12provides the visualization representation18as an interactive display, such that the users (e.g. intelligence analysts, business marketing analysts) can view, and work with, large numbers of events. Further, perceived patterns, anomalies and connections can be explored and subsets of events can be grouped into “story” or hypothesis fragments. The visualization tool12includes a variety of capabilities such as but not limited to:An event-based information architecture with places, events, entities (e.g. people) andrelationships;Past and future time visibility and animation controls;Data input wizards for describing single events and for loading many events from a table;Entity and event connectivity analysis in time and geography;Path displays in time and geography;Configurable workspaces allowing ad hoc, drag and drop arrangements of events;Search, filter and drill down tools;Creation of sub-groups and overlays by selecting events and dragging them into sets (along with associated spatial/time scope properties); andAdaptable display functions including dynamic show/hide controls.Example objects14with associations16

In the visualization tool12, specific combinations of associated data elements (objects20,22,24and associations26) can be defined. These defined groups27are represented visually as visual elements410in specific ways to express various types of occurrences in the visual representation18. The following are examples of how the groups27of associated data elements can be formed to express specific occurrences and relationships shown as the connection visual elements412.

Referring toFIGS. 6 and 7, example groups27(denoting common real world occurrences) are shown with selected subsets of the objects20,22,24combined via selected associations26. The corresponding visualization representation18is shown as well including the temporal domain402, the spatial domain400, connection visual elements412and the visual elements410representing the event/entity/location combinations. It is noted that example applications of the groups27are such as but not limited to those shown inFIGS. 6 and 7. In theFIGS. 6 and 7it is noted that event objects20are labeled as “Event1”, “Event2”, location objects22are labeled as “Location A”, “Location B”, and entity objects24are labeled as “Entity X”, “Entity Y”. The set of associations16are labeled as individual associations26with connections labeled as either solid or dotted lines412between two events, or dotted in the case of an indirect connection between two locations.

Visual Elements Corresponding to Spatial and Temporal Domains

The visual elements410and412, their variations and behavior facilitate interpretation of the concurrent display of events in the time402and space400domains. In general, events reference the location at which they occur and a list of Entities and their role in the event. The time at which the event occurred or the time span over which the event occurred are stored as parameters of the event.

Spatial Domain Representation

Referring toFIG. 8, the primary organizing element of the visualization representation18is the 2D/3D spatial reference frame (subsequently included herein with reference to the spatial domain400). The spatial domain400consists of a true 2D/3D graphics reference surface404in which a 2D or 3 dimensional representation of an area is shown. This spatial domain400can be manipulated using a pointer device (not shown—part of the controls306—seeFIG. 3) by the user of the interface108(seeFIG. 2) to rotate the reference surface404with respect to a viewpoint420or viewing ray extending from a viewer423. The user (i.e. viewer423) can also navigate the reference surface404by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way the user can specify the spatial dimensions of an area of interest the reference surface404in which to view events in time. The spatial domain400represents space essentially as a plane (e.g. reference surface404), however is capable of representing 3 dimensional relief within that plane in order to express geographical features involving elevation. The spatial domain400can be made transparent so that timelines422of the temporal domain402can extend behind the reference surface404are still visible to the user.FIG. 8shows how the viewer423facing timelines422can rotate to face the viewpoint420no matter how the reference surface404is rotated in 3 dimensions with respect to the viewpoint420.

The spatial domain400includes visual elements410,412(seeFIG. 4) that can represent such as but not limited to map information, digital elevation data, diagrams, and images used as the spatial context. These types of spaces can also be combined into a workspace. The user can also create diagrams using drawing tools (of the controls306—seeFIG. 3) provided by the visualization tool12to create custom diagrams and annotations within the spatial domain400.

Event Representation and Interactions

Referring toFIGS. 4 and 8, events are represented by a glyph, or icon as the visual element410, placed along the timeline422at the point in time that the event occurred. The glyph can be actually a group of graphical objects, or layers, each of which expresses the content of the event data object20(seeFIG. 1) in a different way. Each layer can be toggled and adjusted by the user on a per event basis, in groups or across all event instances. The graphical objects or layers for event visual elements410are such as but not limited to:

1. Text labelThe Text label is a text graphic meant to contain a short description of the event content.

This text always faces the viewer423no matter how the reference surface404is oriented. The text label incorporates a de-cluttering function that separates it from other labels if they overlap. When two events are connected with a line (see connections412below) the label will be positioned at the midpoint of the connection line between the events. The label will be positioned at the end of a connection line that is clipped at the edge of the display area.

2. Indicator—Cylinder, Cube or SphereThe indicator marks the position in time. The color of the indicator can be manually set by the user in an event properties dialog. Color of event can also be set to match the Entity that is associated with it. The shape of the event can be changed to represent different aspect of information and can be set by the user. Typically it is used to represent a dimension such as type of event or level of importance.

3. IconAn icon or image can also be displayed at the event location. This icon/image23may used to describe some aspect of the content of the event. This icon/image23may be user-specified or entered as part of a data file of the tables122(seeFIG. 2).

4. Connection elements412Connection elements412can be lines, or other geometrical curves, which are solid or dashed lines that show connections from an event to another event, place or target. A connection element412may have a pointer or arrowhead at one end to indicate a direction of movement, polarity, sequence or other vector-like property. If the connected object is outside of the display area, the connection element412can be coupled at the edge of the reference surface404and the event label will be positioned at the clipped end of the connection element412.

5. Time Range IndicatorA Time Range Indicator (not shown) appears if an event occurs over a range of time. The time range can be shown as a line parallel to the timeline422with ticks at the end points. The event Indicator (see above) preferably always appears at the start time of the event.

The Event visual element410can also be sensitive to interaction. The following user events109via the user interface108(seeFIG. 2) are possible, such as but not limited to:

Selects the visual element410of the visualization representation18on the VI202(seeFIG. 2) and highlights it, as well as simultaneously deselecting any previously selected visual element410, as desired.
Ctrl-Mouse-Left-Click and Shift-Mouse-Left-ClickAdds the visual element410to an existing selection set.
Mouse-Left-Double-Click:Opens a file specified in an event data parameter if it exists. The file will be opened in a system-specified default application window on the interface202based on its file type.
Mouse-Right-Click:Displays an in-context popup menu with options to hide, delete and set properties.
Mouse over Drilldown:When the mouse pointer (not shown) is placed over the indicator, a text window is displayed next to the pointer, showing information about the visual element410. When the mouse pointer is moved away from the indicator, the text window disappears.
Location Representation

Locations are visual elements410represented by a glyph, or icon, placed on the reference surface404at the position specified by the coordinates in the corresponding location data object22(seeFIG. 1). The glyph can be a group of graphical objects, or layers, each of which expresses the content of the location data object22in a different way. Each layer can be toggled and adjusted by the user on a per Location basis, in groups or across all instances. The visual elements410(e.g. graphical objects or layers) for Locations are such as but not limited to:

1. Text LabelThe Text label is a graphic object for displaying the name of the location. This text always faces the viewer422no matter how the reference surface404is oriented. The text label incorporates a de-cluttering function that separates it from other labels if they overlap.

2. IndicatorThe indicator is an outlined shape that marks the position or approximate position of the Location data object22on the reference surface404. There are, such as but not limited to, 7 shapes that can be selected for the locations visual elements410(marker) and the shape can be filled or empty. The outline thickness can also be adjusted. The default setting can be a circle and can indicate spatial precision with size. For example, more precise locations, such as addresses, are smaller and have thicker line width, whereas a less precise location is larger in diameter, but uses a thin line width.The Location visual elements410are also sensitive to interaction. The following interactions are possible:
Mouse-Left-Click:Selects the location visual element410and highlights it, while deselecting any previously selected location visual elements410.
Ctrl-Mouse-Left-Click and Shift-Mouse-Left-ClickAdds the location visual element410to an existing selection set.
Mouse-Left-Double-Click:Opens a file specified in a Location data parameter if it exists. The file will be opened in a system-specified default application window based on its file type.
Mouse-Right-Click:Displays an in-context popup menu with options to hide, delete and set properties of the location visual element410.
Mouseover Drilldown:When the Mouse pointer is placed over the location indicator, a text window showing information about the location visual element410is displayed next to the pointer. When the mouse pointer is moved away from the indicator, the text window disappears.
Mouse-Left-Click-Hold-and-Drag:Interactively repositions the location visual element410by dragging it across the reference surface404.
Non-Spatial Locations

Locations have the ability to represent indeterminate position. These are referred to as non-spatial locations. Locations tagged as non-spatial can be displayed at the edge of the reference surface404just outside of the spatial context of the spatial domain400. These non-spatial or virtual locations can be always visible no matter where the user is currently zoomed in on the reference surface404. Events and Timelines422that are associated with non-spatial Locations can be rendered the same way as Events with spatial Locations.

Entity Representation

Entity visual elements410are represented by a glyph, or icon, and can be positioned on the reference surface404or other area of the spatial domain400, based on associated Event data that specifies its position at the current Moment of Interest900(seeFIG. 9) (i.e. specific point on the timeline422that intersects the reference surface404). If the current Moment of Interest900lies between 2 events in time that specify different positions, the Entity position will be interpolated between the 2 positions. Alternatively, the Entity could be positioned at the most recent known location on he reference surface404. The Entity glyph is actually a group of the entity visual elements410(e.g. graphical objects, or layers) each of which expresses the content of the event data object20in a different way. Each layer can be toggled and adjusted by the user on a per event basis, in groups or across all event instances. The entity visual elements410are such as but not limited to:

1. Text LabelThe Text label is a graphic object for displaying the name of the Entity. This text always faces the viewer no matter how the reference surface404is oriented. The text label incorporates a de-cluttering function that separates it from other labels if they overlap.

2. IndicatorThe indicator is a point showing the interpolated or real position of the Entity in the spatial context of the reference surface404. The indicator assumes the color specified as an Entity color in the Entity data model.

3. Image IconAn icon or image is displayed at the Entity location. This icon may used to represent the identity of the Entity. The displayed image can be user-specified or entered as part of a data file. The Image Icon can have an outline border that assumes the color specified as the Entity color in the Entity data model. The Image Icon incorporates a de-cluttering function that separates it from other Entity Image Icons if they overlap.

4. Past TrailThe Past Trail is the connection visual element412, as a series of connected lines that trace previous known positions of the Entity over time, starting from the current Moment of Interest900and working backwards into past time of the timeline422. Previous positions are defined as Events where the Entity was known to be located. The Past Trail can mark the path of the Entity over time and space simultaneously.

5. Future TrailThe Future Trail is the connection visual element412, as a series of connected lines that trace future known positions of the Entity over time, starting from the current Moment of Interest900and working forwards into future time. Future positions are defined as Events where the Entity is known to be located. The Future Trail can mark the future path of the Entity over time and space simultaneously.

The Entity representation is also sensitive to interaction. The following interactions are possible, such as but not limited to:

Selects the entity visual element410and highlights it and deselects any previously selected entity visual element410.
Ctrl-Mouse-Left-Click and Shift-Mouse-Left-ClickAdds the entity visual element410to an existing selection set
Mouse-Left-Double-Click:Opens the file specified in an Entity data parameter if it exists. The file will be opened in a system-specified default application window based on its file type.
Mouse-Right-Click:Displays an in-context popup menu with options to hide, delete and set properties of the entity visual element410.
Mouseover Drilldown:When the Mouse pointer is placed over the indicator, a text window showing information about the entity visual element410is displayed next to the pointer. When the mouse pointer is moved away from the indicator, the text window disappears.
Temporal Domain including Timelines

Referring toFIGS. 8 and 9, the temporal domain provides a common temporal reference frame for the spatial domain400, whereby the domains400,402are operatively coupled to one another to simultaneously reflect changes in interconnected spatial and temporal properties of the data elements14and associations16. Timelines422(otherwise known as time tracks) represent a distribution of the temporal domain402over the spatial domain400, and are a primary organizing element of information in the visualization representation18that make it possible to display events across time within the single spatial display on the VI202(seeFIG. 1). Timelines422represent a stream of time through a particular Location visual element410apositioned on the reference surface404and can be represented as a literal line in space. Other options for representing the timelines/time tracks422are such as but not limited to curved geometrical shapes (e.g. spirals) including 2D and 3D curves when combining two or more parameters in conjuction with the temporal dimension. Each unique Location of interest (represented by the location visual element410a) has one Timeline422that passes through it. Events (represented by event visual elements410b) that occur at that Location are arranged along this timeline422according to the exact time or range of time at which the event occurred. In this way multiple events (represented by respective event visual elements410b) can be arranged along the timeline422and the sequence made visually apparent. A single spatial view will have as many timelines422as necessary to show every Event at every location within the current spatial and temporal scope, as defined in the spatial400and temporal402domains (seeFIG. 4) selected by the user. In order to make comparisons between events and sequences of event between locations, the time range represented by multiple timelines422projecting through the reference surface404at different spatial locations is synchronized. In other words the time scale is the same across all timelines422in the time domain402of the visual representation18. Therefore, it is recognised that the timelines422are used in the visual representation18to visually depict a graphical visualization of the data objects14over time with respect to their spatial properties/attributes.

For example, in order to make comparisons between events20and sequences of events between locations410of interest (seeFIG. 4), the time range represented by the timelines422can be synchronized. In other words, the time scale can be selected as the same for every timeline422of the selected time range of the temporal domain402of the representation18.

Representing Current, Past and Future

Three distinct strata of time are displayed by the timelines422, namely;

1. The “moment of interest”900or browse time, as selected by the user,

2. a range902of past time preceding the browse time called “past”, and

3. a range904of time after the moment of interest900, called “future”

On a 3D Timeline422, the moment of focus900is the point at which the timeline intersects the reference surface404. An event that occurs at the moment of focus900will appear to be placed on the reference surface404(event representation is described above). Past and future time ranges902,904extend on either side (above or below) of the moment of interest900along the timeline422. Amount of time into the past or future is proportional to the distance from the moment of focus900. The scale of time may be linear or logarithmic in either direction. The user may select to have the direction of future to be down and past to be up or vice versa.

There are three basic variations of Spatial Timelines422that emphasize spatial and temporal qualities to varying extents. Each variation has a specific orientation and implementation in terms of its visual construction and behavior in the visualization representation18(seeFIG. 1). The user may choose to enable any of the variations at any time during application runtime, as further described below.

FIG. 10shows how 3D Timelines422pass through reference surface404locations410a.3D timelines422are locked in orientation (angle) with respect to the orientation of the reference surface404and are affected by changes in perspective of the reference surface404about the viewpoint420(seeFIG. 8). For example, the 3D Timelines422can be oriented normal to the reference surface404and exist within its coordinate space. Within the 3D spatial domain400, the reference surface404is rendered in the X-Y plane and the timelines422run parallel to the Z-axis through locations410aon the reference surface404. Accordingly, the 3D Timelines422move with the reference surface404as it changes in response to user navigation commands and viewpoint changes about the viewpoint420, much like flag posts are attached to the ground in real life. The 3D timelines422are subject to the same perspective effects as other objects in the 3D graphical window of the VI202(seeFIG. 1) displaying the visual representation18. The 3D Timelines422can be rendered as thin cylindrical volumes and are rendered only between events410awith which it shares a location and the location410aon the reference surface404. The timeline422may extend above the reference surface404, below the reference surface404, or both. If no events410bfor its location410aare in view the timeline422is not shown on the visualization representation18.

Referring toFIG. 8, 3D Viewer-facing Timelines422are similar to 3D Timelines422except that they rotate about a moment of focus425(point at which the viewing ray of the viewpoint420intersects the reference surface404) so that the 3D Viewer-facing Timeline422always remain perpendicular to viewer423from which the scene is rendered. 3D Viewer-facing Timelines422are similar to 3D Timelines422except that they rotate about the moment of focus425so that they are always parallel to a plane424normal to the viewing ray between the viewer423and the moment of focus425. The effect achieved is that the timelines422are always rendered to face the viewer423, so that the length of the timeline422is always maximized and consistent. This technique allows the temporal dimension of the temporal domain402to be read by the viewer423indifferent to how the reference surface404many be oriented to the viewer423. This technique is also generally referred to as “billboarding” because the information is always oriented towards the viewer423. Using this technique the reference surface404can be viewed from any direction (including directly above) and the temporal information of the timeline422remains readable.

Referring toFIG. 11, showing how an overlay time chart430is connected to the reference surface404locations410aby timelines422. The timelines422of the Linked TimeChart430are timelines422that connect the 2D chart430(e.g. grid) in the temporal domain402to locations410amarked in the 3D spatial domain400. The timeline grid430is rendered in the visual representation18as an overlay in front of the 2D or 3D reference surface404. The timeline chart430can be a rectangular region containing a regular or logarithmic time scale upon which event representations410bare laid out. The chart430is arranged so that one dimension432is time and the other is location434based on the position of the locations410aon the reference surface404. As the reference surface404is navigated or manipulated the timelines422in the chart430move to follow the new relative location410apositions. This linked location and temporal scrolling has the advantage that it is easy to make temporal comparisons between events since time is represented in a flat chart430space. The position410bof the event can always be traced by following the timeline422down to the reference surface404to the location410a.

Referring toFIGS. 11 and 12, the TimeChart430can be rendered in 2 orientations, one vertical and one horizontal. In the vertical mode ofFIG. 11, the TimeChart430has the location dimension434shown horizontally, the time dimension432vertically, and the timelines422connect vertically to the reference surface404. In the horizontal mode ofFIG. 12, the TimeChart430has the location dimension434shown vertically, the time dimension432shown horizontally and the timelines422connect to the reference surface404horizontally. In both cases the TimeChart430position in the visualization representation18can be moved anywhere on the screen of the VI202(seeFIG. 1), so that the chart430may be on either side of the reference surface404or in front of the reference surface404. In addition, the temporal directions of past902and future904can be swapped on either side of the focus900.

Interaction Interface Descriptions

Referring toFIGS. 3 and 13, several interactive controls306support navigation and analysis of information within the visualization representation12, as monitored by the visualization manger300in connection with user events109. Examples of the controls306are such as but not limited to a time slider910, an instant of focus selector912, a past time range selector914, and a future time selector916. It is recognized that these controls306can be represented on the VI202(seeFIG. 1) as visual based controls, text controls, and/or a combination thereof.

Time and Range Slider901

The timeline slider910is a linear time scale that is visible underneath the visualization representation18(including the temporal402and spatial400domains). The control910contains sub controls/selectors that allow control of three independent temporal parameters: the Instant of Focus, the Past Range of Time and the Future Range of Time.

Continuous animation of events20over time and geography can be provided as the time slider910is moved forward and backwards in time. Example, if a vehicle moves from location A at t1to location B at t2, the vehicle (object23,24) is shown moving continuously across the spatial domain400(e.g. map). The timelines422can animate up and down at a selected frame rate in association with movement of the slider910.

Instant of Focus

The instant of focus selector912is the primary temporal control. It is adjusted by dragging it left or right with the mouse pointer across the time slider910to the desired position. As it is dragged, the Past and Future ranges move with it. The instant of focus900(seeFIG. 12) (also known as the browse time) is the moment in time represented at the reference surface404in the spatial-temporal visualization representation18. As the instant of focus selector912is moved by the user forward or back in time along the slider910, the visualization representation18displayed on the interface202(seeFIG. 1) updates the various associated visual elements of the temporal402and spatial400domains to reflect the new time settings. For example, placement of Event visual elements410animate along the timelines422and Entity visual elements410move along the reference surface404interpolating between known locations visual elements410(seeFIGS. 6 and 7). Examples of movement are given with reference toFIGS. 14,15, and16below.

Past Time Range

The Past Time Range selector914sets the range of time before the moment of interest900(seeFIG. 11) for which events will be shown. The Past Time range is adjusted by dragging the selector914left and right with the mouse pointer. The range between the moment of interest900and the Past time limit can be highlighted in red (or other colour codings) on the time slider910. As the Past Time Range is adjusted, viewing parameters of the spatial-temporal visualization representation18update to reflect the change in the time settings.

Future Time Range

The Future Time Range selector914sets the range of time after the moment of interest900for which events will be shown. The Future Time range is adjusted by dragging the selector916left and right with the mouse pointer. The range between the moment of interest900and the Future time limit is highlighted in blue (or other colour codings) on the time slider910. As the Future Time Range is adjusted, viewing parameters of the spatial-temporal visualization representation18update to reflect the change in the time settings.

The time range visible in the time scale of the time slider910can be expanded or contracted to show a time span from centuries to seconds. Clicking and dragging on the time slider910anywhere except the three selectors912,914,916will allow the entire time scale to slide to translate in time to a point further in the future or past. Other controls918associated with the time slider910can be such as a “Fit” button918for automatically adjusting the time scale to fit the range of time covered by the currently active data set displayed in the visualization representation18. A scale control918includes a Fit control919, a scale-expand-contract controls920, a step control923, and a play control922, which allow the user to expand or contract the time scale. A step control918increments the instant of focus900forward or back. The “playback” button920causes the instant of focus900to animate forward by a user-adjustable rate. This “playback” causes the visualization representation18as displayed to animate in sync with the time slider910.

Simultaneous Spatial and Temporal Navigation can be provided by the tool12using, for example, interactions such as zoom-box selection and saved views. In addition, simultaneous spatial and temporal zooming can be used to provide the user to quickly move to a context of interest. In any view of the representation18, the user may select a subset of events20and zoom to them in both time402and space400domains using a Fit Time and a Fit Space functions. These functions can happen simultaneously by dragging a zoom-box on to the time chart430itself. The time range and the geographic extents of the selected events20can be used to set the bounds of the new view of the representation18, including selected domain400,402view formats.

Association Analysis Tools

Referring toFIGS. 1 and 3, association analysis functions307have been developed that take advantage of the association-based connections between Events, Entities and Locations. These functions307are used to find groups of connected objects14during analysis. The associations16connect these basic objects20,22,24into complex groups27(seeFIGS. 6 and 7) representing actual occurrences. The functions307are used to follow the associations16from object14to object14to reveal connections between objects14that are not immediately apparent. Association analysis functions307are especially useful in analysis of large data sets where an efficient method to find and/or filter connected groups is desirable. For example, an Entity24maybe be involved in events20in a dozen places/locations22, and each of those events20may involve other Entities24. The association analysis function307can be used to display only those locations22on the visualization representation18that the entity24has visited or entities24that have been contacted.

The analysis functions307provide the user with different types of link analysis, such as but limited to:

1. Expanding SearchThe expanding search function307allows the user to start with a selected object(s)14and then incrementally show objects14that are associated with it by increasing degrees of separation. The user selects an object14or group of objects14of focus and clicks on the Expanding search button920—this causes everything in the visualization representation18to disappear except the selected items. The user then increments the search depth and objects14connected by the specified depth are made visible the display. In this way, sets of connected objects14are revealed as displayed using the visual elements410and412.

2. Connection SearchThe Connection Search function307allows the user to connect any two objects14by their web of associations26. The user selects any two objects14and clicks on a Connection Search tool (not shown). The connection search function307works by automatically scanning the extents of the web of associations26starting from one of the objects14. The search will continue until the second object14is found as one of the connected objects14or until there are no more connected objects14. If a path of associated objects14between the target objects14exists, all of the objects14along that path are displayed and the depth is automatically displayed showing the minimum number of links between the objects14.

It is recognized that the functions307can be used to implement filtering via such as but not limited to criteria matching, algorithmic methods and/or manual selection of objects14and associations16using the analytical properties of the tool12. This filtering can be used to highlight/hide/show (exclusively) selected objects14and associations16as represented on the visual representation18. The functions307are used to create a group (subset) of the objects14and associations16as desired by the user through the specified criteria matching, algorithmic methods and/or manual selection. Further, it is recognized that the selected group of objects14and associations16could be assigned a specific name which is stored in the table122.

Operation of Visual Tool to Generate Visualization Representation

Referring toFIG. 14, example operation1400shows communications1402and movement events1404(connection visual elements412—seeFIGS. 6 and 7) between Entities “X” and “Y” over time on the visualization representation18. ThisFIG. 14shows a static view of Entity X making three phone call communications1402to Entity Y from 3 different locations410aat three different times. Further, the movement events1404are shown on the visualization representation18indicating that the entity X was at three different locations410a(location A,B,C), which each have associated timelines422. The timelines422indicate by the relative distance (between the elements410band410a) of the events (E1,E2,E3) from the instant of focus900of the reference surface404that these communications1404occurred at different times in the time dimension432of the temporal domain402. Arrows on the communications1402indicate the direction of the communications1402, i.e. from entity X to entity Y. Entity Y is shown as remaining at one location410a(D) and receiving the communications1402at the different times on the same timeline422.

Referring toFIG. 15, example operation1500for shows Events140boccurring within a process diagram space domain400over the time dimension432on the reference surface404. The spatial domain400represents nodes1502of a process. ThisFIG. 14shows how a flowchart or other graphic process can be used as a spatial context for analysis. In this case, the object (entity) X has been tracked through the production process to the final stage, such that the movements1504represent spatial connection elements412(seeFIGS. 6 and 7).

Referring toFIGS. 3 and 19, operation800of the tool12begins by the manager300assembling802the group of objects14from the tables122via the data manager114. The selected objects14are combined804via the associations16, including assigning the connection visual element412(seeFIGS. 6 and 7) for the visual representation18between selected paired visual elements410corresponding to the selected correspondingly paired data elements14of the group. The connection visual element412represents a distributed association16in at least one of the domains400,402between the two or more paired visual elements410. For example, the connection element412can represent movement of the entity object24between locations22of interest on the reference surface404, communications (money transfer, telephone call, email, etc. . . . ) between entities24different locations22on the reference surface404or between entities24at the same location22, or relationships (e.g. personal, organizational) between entities24at the same or different locations22.

Next, the manager300uses the visualization components308(e.g. sprites) to generate806the spatial domain400of the visual representation18to couple the visual elements410and412in the spatial reference frame at various respective locations22of interest of the reference surface404. The manager300then uses the appropriate visualization components308to generate808the temporal domain402in the visual representation18to include various timelines422associated with each of the locations22of interest, such that the timelines422all follow the common temporal reference frame. The manager112then takes the input of all visual elements410,412from the components308and renders them810to the display of the user interface202. The manager112is also responsible for receiving812feedback from the user via user events109as described above and then coordinating814with the manager300and components308to change existing and/or create (via steps806,808) new visual elements410,412to correspond to the user events109. The modified/new visual elements410,412are then rendered to the display at step810.

Referring toFIG. 16, an example operation1600shows animating entity X movement between events (Event1and Event2) during time slider901interactions via the selector912. First, the Entity X is observed at Location A at time t. As the slider selector912is moved to the right, at time t+1the Entity X is shown moving between known locations (Event1and Event2). It should be noted that the focus900of the reference surface404changes such that the events1and2move along their respective timelines422, such that Event1moves from the future into the past of the temporal domain402(from above to below the reference surface404). The length of the timeline422for Event2(between the Event2and the location B on the reference surface404decreases accordingly. As the slider selector912is moved further to the right, at time t+2, Entity X is rendered at Event2(Location B). It should be noted that the Event1has moved along its respective timeline422further into the past of the temporal domain402, and event2has moved accordingly from the future into the past of the temporal domain402(from above to below the reference surface404), since the representation of the events1and2are linked in the temporal domain402. Likewise, the entity X is linked spatially in the spatial domain400between event1at location A and event2at location B. It is also noted that the Time Slider selector912could be dragged along the time slider910by the user to replay the sequence of events from time t to t+2, or from t+2to t, as desired.

Referring toFIG. 17, the visual representation18shows connection visual elements412between visual elements410situated on selected various timelines422. The timelines422are coupled to various locations22of interest on the geographical reference frame404. In this case, the elements412represent geographical movement between various locations22by entity24, such that all travel happened at some time in the future with respect to the instant of focus represented by the reference plane404.

Referring toFIG. 18, the spatial domain400is shown as a geographical relief map. The timechart430is superimposed over the spatial domain of the visual representation18, and shows a time period spanning from December 3rdto January 1stfor various events20and entities24situated along various timelines422coupled to selected locations22of interest. It is noted that in this case the user can use the presented visual representation to coordinate the assignment of various connection elements412to the visual elements410(seeFIG. 6) of the objects20,22,24via the user interface202(seeFIG. 1), based on analysis of the displayed visual representation18content. A time selection950is January 30, such that events20and entities24within the selection box can be further analysed. It is recognised that the time selection950could be used to represent the instant of focus900(seeFIG. 9).

Referring toFIG. 3, an Aggregation Module600is for, such as but not limited to, summarizing or aggregating the data objects14, providing the summarized or aggregated data objects14to the Visualization Manager300which processes the translation from data objects14and group of data elements27to the visual representation18, and providing the creation of summary charts200(seeFIG. 26) for displaying information related to summarised/aggregated data objects14as the visual representation18on the display108.

Referring toFIGS. 3 and 22, the spatial inter-connectedness of information over time and geography within a single, highly interactive 3-D view of the representation18is beneficial to data analysis (of the tables122). However, when the number of data objects14increases, techniques for aggregation become more important. Many individual locations22and events20can be combined into a respective summary or aggregated output603. Such outputs603of a plurality of individual events20and locations22(for example) can help make trends in time and space domains400,402more visible and comparable to the user of the tool12. Several techniques can be implemented to support aggregation of data objects14such as but not limited to techniques of hierarchy of locations, user defined geo-relations, and automatic LOD level selection, as further described below. The tool12combines the spatial and temporal domains400,402on the display108for analysis of complex past and future events within a selected spatial (e.g. geographic) context.

Referring toFIG. 22, the Aggregation Module600has an Aggregation Manager601that communicates with the Visualization Manager300for receiving aggregation parameters used to formulate the output603. The parameters can be either automatic (e.g. tool pre-definitions) manual (entered via events109) or a combination thereof. The manager601accesses all possible data objects14through the Data Manager114(related to the aggregation parameters—e.g. time and/or spatial ranges and/or object14types/combinations) from the tables122, and then applies aggregation tools or filters602for generating the output603. The Visualization Manager300receives the output603from the Aggregation Manager601, based on the user events109and/or operation of the Time Slider and other Controls306by the user for providing the aggregation parameters. As described above, once the output603is requested by the Visualization Manager114, the Aggregation Manager601communicates with the Data Manager114access all possible data objects14for satisfying the most general of the aggregation parameters and then applies the filters602to generate the output603. It is recognised however, that the filters602could be used by the manager601to access only those data objects14from the tables122that satisfy the aggregation parameters, and then copy those selected data objects14from the tables122for storing/mapping as the output603.

Accordingly, the Aggregation Manager601can make available the data elements14to the Filters602. The filters602act to organize and aggregate (such as but not limited to selection of data objects14from the global set of data in the tables122according to rules/selection criteria associated with the aggregation parameters) the data objects14according the instructions provided by the Aggregation Manager601. For example, the Aggregation Manager601could request that the Filters602summarize all data objects14with location data22corresponding to Paris. Or, in another example, the Aggregation Manager601could request that the Filters602summarize all data objects14with event data20corresponding to Wednesdays. Once the data objects14are selected by the Filters602, the aggregated data is summarised as the output603. The Aggregation Manager601then communicates the output603to the Visualization Manager300, which processes the translation from the selected data objects14(of the aggregated output603) for rendering as the visual representation18. It is recognised that the content of the representation18is modified to display the output603to the user of the tool12, according to the aggregation parameters.

Further, the Aggregation Manager601provides the aggregated data objects14of the output603to a Chart Manager604. The Chart Manager604compiles the data in accordance with the commands it receives from the Aggregation Manager601and then provides the formatted data to a Chart Output605. The Chart Output605provides for storage of the aggregated data in a Chart section606of the display (seeFIG. 25). Data from the Chart Output605can then be sent directly to the Visualization Renderer112or to the visualisation manager300for inclusion in the visual representation18, as further described below.

Referring toFIG. 23, an example aggregation of data objects14by the Aggregation Module601is shown. The event data20(for example) is aggregated according to spatial proximity (threshold) of the data objects14with respect to a common point (e.g. particular location410or other newly specified point of the spatial domain400), difference threshold between two adjacent locations410, or other spatial criteria as desired. For example, as depicted inFIG. 23a, the three data objects20at three locations410are aggregated to two objects20at one location410and one object at another location410(e.g. combination of two locations410) as a user-defined field202of view is reduced inFIG. 23b, and ultimately to one location410with all three objects20inFIG. 23c. It is recognised in this example of aggregated output603that timelines422of the locations410are combined as dictated by the aggregation of locations410.

For example, the user may desire to view an aggregate of data objects14related within a set distance of a fixed location, e.g., aggregate of events20occurring within 50 km of the Golden Gate Bridge. To accomplish this, the user inputs their desire to aggregate the data according to spatial proximity, by use of the controls306, indicating the specific aggregation parameters. The Visualization Manager300communicates these aggregation parameters to the Aggregation Module600, in order for filtering of the data content of the representation18shown on the display108. The Aggregation Module600uses the Filters602to filter the selected data from the tables122based on the proximity comparison between the locations410. In another example, a hierarchy of locations can be implemented by reference to the association data26which can be used to define parent-child relationships between data objects14related to specific locations within the representation18. The parent-child relationships can be used to define superior and subordinate locations that determine the level of aggregation of the output603.

Referring toFIG. 24, an example aggregation of data objects14by the Aggregation Module601is shown. The data14is aggregated according to defined spatial boundaries204. To accomplish this, the user inputs their desire to aggregate the data14according to specific spatial boundaries204, by use of the controls306, indicating the specific aggregation parameters of the filtering602. For example, a user may wish to aggregate all event20objects located within the city limits of Toronto. The Visualization Manager300then requests to the Aggregation Module600to filter the data objects14of the current representation according to the aggregation parameters. The Aggregation Module600provides implements or otherwise applies the filters602to filter the data based on a comparison between the location data objects14and the city limits of Toronto, for generating the aggregated output603. InFIG. 24a, within the spatial domain205the user has specified two regions of interest204, each containing two locations410with associated data objects14. InFIG. 24b, once filtering has been applied, the locations410of each region204have been combined such that now two locations410are shown with each having the aggregated result (output603) of two data objects14respectively. InFIG. 24c, the user has defined the region of interest to be the entire domain205, thereby resulting in the displayed output603of one location410with three aggregated data objects14(as compared toFIG. 24a). It is noted that the positioning of the aggregated location410is at the center of the regions of interest204, however other positioning can be used such as but not limited to spatial averaging of two or more locations410or placing aggregated object data14at one of the retained original locations410, or other positioning techniques as desired.

In addition to the examples in illustrated inFIGS. 21 and 22, the aggregation of the data objects can be accomplished automatically based on the geographic view scale provided in the visual representations. Aggregation can be based on level of detail (LOD) used in mapping geographical features at various scales. On a 1:25,000 map, for example, individual buildings may be shown, but a 1:500,000 map may show just a point for an entire city. The aggregation module600can support automatic LOD aggregation of objects14based on hierarchy, scale and geographic region, which can be supplied as aggregation parameters as predefined operation of the controls306and/or specific manual commands/criteria via user input events109. The module600can also interact with the user of the tool12(via events109) to adjust LOD behaviour to suit the particular analytical task at hand.

Referring toFIG. 25, an example of a spatial and temporal visual representation18with summary chart202depicting event data20is shown. For example, a user may wish to see the quantitative information relating to a specific event object. The user would request the creation of the chart using the controls306, which would submit the request to the Visualization Manager300. The Visualization Manager300would communicate with the Aggregation Module600and instruct the creation of the chart200depicting all of the quantitative information associated with the data objects14associated with the specific event object20, and represent that on the display108(seeFIG. 2) as content of the representation18. The Aggregation Module600would communicate with the Chart Manager604, which would list the relevant data and provide only the relevant information to the Chart Output605. The Chart Output605provides a copy of the relevant data for storage in the Chart Comparison Module, and the data output is communicated from the Chart Output605to the Visualization Renderer112before being included in the visual representation18. The output data stored in the Chart Comparison section606can be used to compare to newly created charts200when requested from the user. The comparison of data occurs by selecting particular charts200from the chart section606for application as the output603to the Visual Representation18.

The charts rendered by the Chart Manager604can be created in a number of ways. For example, all the data objects14from the Data Manager114can be provided in the chart. Or, the Chart Manager604can filter the data so that only the data objects14related to a specific temporal range will appear in the chart provided to the Visual Representation18. Or, the Chart Manager604can filter the data so that only the data objects14related to a specific spatial and temporal range will appear in the chart provided to the Visual Representation18.

In a further example of the aggregation module601, user-defined location boundaries204can provide for aggregation of data14across an arbitrary region. Referring toFIG. 26, to compare a summary of events along two separate routes210and212, aggregation output603of the data14associated with each route210,212would be created by drawing an outline boundary204around each route210,212and then assigning the boundaries204to the respective locations410contained therein, as depicted inFIG. 26a. By the user adjusting the aggregation level in the Filters602through specification of the aggregation parameters of the boundaries204and associated locations410, the data14is the aggregated as output603(seeFIG. 26b) within the outline regions into the newly created locations410, with the optional display of text214providing analysis details for those new aggregated locations410. For example, the text214could summarise that the number of bad events20(e.g. bombings) is greater for route210than route212and therefore route212would be the route of choice based on the aggregated output603displayed on the representation18.

It will be appreciated that variations of some elements are possible to adapt the invention for specific conditions or functions. The concepts of the present invention can be further extended to a variety of other applications that are clearly within the scope of this invention.

For example, one application of the tool12is in criminal analysis by the “information producer”. An investigator, such as a police officer, could use the tool12to review an interactive log of events20gathered during the course of long-term investigations. Existing reports and query results can be combined with user input data109, assertions and hypotheses, for example using the annotations21. The investigator can replay events20and understand relationships between multiple suspects, movements and the events20. Patterns of travel, communications and other types of events20can be analysed through viewing of the representation18of the data in the tables122to reveal such as but not limited to repetition, regularity, and bursts or pauses in activity.

Subjective evaluations and operator trials with four subject matter experts have been conducted using the tool12. These initial evaluations of the tool12were run against databases of simulated battlefield events and analyst training scenarios, with many hundreds of events20. These informal evaluations show that the following types of information can be revealed and summarised. What significant events happened in this area in the last X days? Who was involved? What is the history of this person? How are they connected with other people? Where are the activity hot spots? Has this type of event occurred here or elsewhere in the last Y period of time?

With respect to potential applications and the utility of the tool12, encouraging and positive remarks were provided by military subject matter experts in stability and support operations. A number of those remarks are provided here. Preparation for patrolling involved researching issues including who, where and what. The history of local belligerent commanders and incidents. Tracking and being aware of history, for example, a ceasefire was organized around a religious calendar event. The event presented an opportunity and knowing about the event made it possible. In one campaign, the head of civil affairs had been there twenty months and had detailed appreciation of the history and relationships. Keeping track of trends. What happened here? What keeps happening here? There are patterns. Belligerents keep trying the same thing with new rotations [a rotation is typically six to twelve months tour of duty]. When the attack came, it did come from the area where many previous earlier attacks had also originated. The discovery of emergent trends . . . persistent patterns . . . sooner rather than later could be useful. For example, the XXX Colonel that tends to show up in an area the day before something happens. For every rotation a valuable knowledge base can be created, and for every rotation, this knowledge base can be retained using the tool12to make the knowledge base a valuable historical record. The historical record can include events, factions, populations, culture, etc.

Having thus described the present invention with respect to preferred embodiments as implemented, it will be apparent to those skilled in the art that many modifications and enhancements are possible to the present invention without departing from the basic concepts as described in the preferred embodiment of the present invention. Therefore, what is intended to be protected by way of letters patent should be limited only by the scope of the following claims.