Methods and systems for simultaneous local and contextual display

Methods and systems for simultaneous local and contextual display comprise displaying a smaller, “local” image on a first surface using a first device having limited screen real estate. A second, larger, “contextual” image is simultaneously displayed on a larger second surface, which may comprise a projector screen or a separate computer monitor screen. The contextual image may be projected onto the second surface using the first device, or the first device may be coupled to a second device for displaying the contextual image. The local image comprises a subset of the contextual image such that a user may view both the fine detail of the local image using the first device and the context of the local image within the contextual image on the second surface.

TECHNICAL FIELD

The present disclosure relates generally to methods and systems for providing simultaneous local and contextual displays of digital images on handheld devices, computer monitors, projection screens, and other media.

BACKGROUND

Recent advances in handheld device technology, such as smartphones, have enabled users to access and view increasingly detailed and sophisticated forms of graphical data, such as maps, documents, file libraries, and medical images. However, due to the limited screen real estate of handheld devices, or even traditional desktop or laptop computer screen monitors, users are often forced to choose between viewing either the finer graphical detail that may be provided in the context of a zoomed-in focal area of an image or the less detailed graphical data that is available or visible when viewing a broader focal area for the purpose of determining the context of a zoomed-in image.

FIG. 1, for example, depicts a conventional approach to displaying images on a handheld device. A handheld device100, such as a smartphone, may include a screen110for displaying images and other graphical user interface controls. A user may interface with handheld device100through a touchscreen interface provided through device screen110or through various physical buttons or controls on the device. Device100may be capable of displaying images, such as maps, documents, or other graphical data on device screen110. Using the example of displaying maps, a user may use device100to display an image115of a map on device screen110. Device100may allow the user to scroll image115and thus focus on geographical regions adjacent to image115.

Device100may also allow the user to zoom in or out from image115to display a larger or smaller geographical region in device screen110. As the user zooms in or out, device100may display a new image115that contains more or less detail, such as roads or municipality labels. For example, if a user zooms out from the current geographical focus of image115, device100may display a larger geographical area in device screen110and may reduce the amount of low-level geographical data by omitting the display of smaller roads or municipality labels. Conversely, if the user zooms in from the current geographical focus of image115, device100may display a smaller geographical area in device screen110and may increase the amount of low-level geographical data by displaying more detail corresponding to individual roads, rivers, parks, neighborhoods, or labels identifying the same.

Although the above technique—namely, increasing or decreasing granularity, resolution, or graphical detail as an image is zoomed—allows a user to view both large and small geographical focus areas of a map, this technique does not enable the user to view both fine graphical detail and broader contextual graphical data at the same time. For example, if the user zooms in to focus on a particular neighborhood in a municipality, while the user may be able to view individual streets and street labels, the user will be unable to view the broader context of where within a larger area, such as a city or state, that particular neighborhood lies. However, if the user zooms out to the city or state level, the user will either not be able to see any fine detail regarding the neighborhood of interest, or such detail will be too small for the user to easily discern.

There is therefore a need for methods and systems for simultaneously viewing local and contextual images, such as maps, documents, libraries, or medical images such that a user may view both the fine detail of a particular area of interest as well as the broader context of that area of interest.

SUMMARY OF THE INVENTION

The present invention comprises methods and systems for providing simultaneous or concurrent local and contextual displays of digital images on handheld devices, computer monitors, projection screens, and other media. In one embodiment, a smaller, “local” image is displayed on a first surface using a first device having limited screen real estate. A second, larger, “contextual” image is simultaneously displayed on a larger second surface, which may comprise a projector screen or a separate computer monitor screen. The contextual image may be projected onto the second surface using the first device, or the first device may be coupled to a second device for displaying the contextual image. The local image comprises a subset of the contextual image such that a user may view both the fine detail of the local image using the first device and the context of the local image within the contextual image on the second surface.

In some embodiments, the local and contextual images may be visually aligned such that the local image corresponds to the dimensions and content of the contextual image when the first surface is placed in front of the second surface in a line of sight. In some embodiments, in response to a user scrolling the local image, the first device or second device may simultaneously scroll the contextual image to maintain visual alignment between the local and contextual images. Or, the contextual image may be displayed in a substantially constant manner while a visual indicator, such as a rectangle is scrolled within the contextual image to indicate the placement of the local image within the contextual image.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments and features of the invention are described herein, modifications, adaptations, and other implementations are possible, without departing from the spirit and scope of the invention. Accordingly, the following detailed description does not limit the invention. Instead, the proper scope of the invention is defined by the appended claims.

FIG. 2is a diagram depicting an exemplary device for simultaneously displaying local and contextual images, consistent with certain disclosed embodiments. Although depicted inFIG. 2as a handheld device, such as a smartphone or PDA, device200is not limited to handheld devices, but may comprise any device or device system capable of displaying graphical data, such as images. Using the example of a handheld device, device200may comprise a display screen210, such as a liquid crystal display (LCD), plasma, cathode-ray, pixel-based display, or other type of screen capable of rendering images in response to electronic signals. Device200may additionally include, for example, one or more microprocessors220of varying core configurations and clock frequencies; one or more memory devices or computer-readable media230of varying physical dimensions and storage capacities, such as flash drives, hard drives, random access memory, etc., for storing data, such as images, files, and program instructions for execution by one or more microprocessors220; one or more wireless transceivers240for communicating over wireless protocols such as wireless Ethernet, code divisional multiple access (CDMA), or other wireless media; one or more peripheral connections250, such as universal serial bus (USB) connections or video interfaces for communicating with or sending display data to an external device, such as a separate computer or computer monitor screen; one or more projection components260for projecting light and images onto a physical surface, such as a projector screen; and one or more range detectors270, such laser or ultrasonic range finders, for determining device200's physical proximity to a physical object or surface, such as a projector screen. Those skilled in the art will appreciate that the above-described componentry is exemplary only, as device200may comprise any type of hardware componentry, including any necessary accompanying firmware or software, for performing the disclosed embodiments.

FIG. 3is a flow diagram illustrating an exemplary method of simultaneously displaying local and contextual images, as depicted inFIGS. 4-8, consistent with certain disclosed embodiments. As depicted inFIG. 4, an image400, such as a map, may be available to device200for access and display. Image400may comprise any type of file format, such as gif, jpeg, bmp, tiff, or other format capable of storing image or other kinds of data. Image400, however, is not confined to any particular file format, or even a data file at all. Image400may comprise image or graphical data in any form or from any source capable of display on a device screen or physical surface.

For example, device200may retrieve graphical data comprising image400from a separate device, such as a server (not depicted), via serial, Ethernet, or other link. Device200may obtain portions of graphical data contained in image400in streaming, “push,” or “pull” form, for example, from the server to display such varying portions as needed. For example, device200may obtain a portion of graphical data representing one particular focal area of image400for display on device screen210. And, in response to a user performing a scrolling operation on device200in order to view a different focal area of image400, device200may retrieve different or additional data from the server in order to display the focal area of image400requested as a result of the user's scrolling operation.

Different focal areas of image400may include adjacent graphical areas within image400or different “zoom” areas within image400. Image400may also comprise multiple layers of one or more focal areas. For example, as depicted inFIG. 4, image400may include a first layer that depicts a particular geographical region of a map with only minor granularity, such that only major highways, rivers, or municipality labels are depicted. As depicted inFIG. 5, image400may include a second layer500that depicts the same geographical region as the first layer, but includes more granularity, such as street grids and names. Those skilled in the art will appreciate that image400is not limited to the embodiments depicted or described with respect toFIGS. 4 and 5, but may comprise any kind of data in any form, whether embodied in a file or computer-readable medium prior to display or dynamically generated for display.

Returning toFIG. 3, in some embodiments, device200may determine its proximity, such as physical distance605, from an external surface, such as projector screen600depicted inFIG. 6(step310). Device200may use range detector componentry270, such as a laser range finder or an ultrasonic range finder (not depicted) to determine device200's physical distance from projector screen600. An exemplary laser range finder may determine distance605by emitting light, such as an infrared light pulse, from device200to projector screen600and measuring the degree of red-shifting or blue-shifting in the light as it is reflected off of projector screen600back to device200to determine movement of device200toward or away from projector screen600. An exemplary ultrasonic range finder may determine distance605by emitting sound waves from device200to projector screen600and measuring the time it takes for such sound waves to bounce off of projector screen600and return to device200. Those skilled in the art will appreciate that there may be other techniques used by device200to detect its proximity to an external surface. In other embodiments, device200may be manually configured by a user or other device to specify a distance605, and in other embodiments, it may not be necessary for device200to operate with any knowledge of its proximity to any external surface.

In step320, device200may determine the scope of a local image. As used herein, the term “local image” may refer to an image to be displayed on a smaller screen, such as device screen210of device200. A local image may comprise a subset of a larger image, such as image400or image500. Device200may determine the scope of the local image by determining the dimensions of the graphical or focal area subset of image400that should be displayed on device screen210. Device200may also determine the scope of the local image in terms of the resolution or the granularity of the local image.FIG. 6depicts an exemplary local image610displayed on device screen210of device200. As can be seen, local image610comprises a subset of image400in that it comprises smaller focal area within image400. In this example, local image610also contains a greater granularity than the image layer400depicted inFIG. 4, which granularity may come from image layer500of image400, as depicted inFIG. 5.

In step330, device200may determine the location of the local image within the full image. As used herein, the term “full image” may refer to an image or collection of image data, such as image400, of which a local image comprises only a subset. For example, determining the scope and the location of local image610within image400may comprise determining a coordinate611on image400on which local image610is based, as well as a height dimension612aand a width dimension612b. Using coordinate611, height dimension612a, and width dimension612b, device200may determine the precise geographical or focal area within image400that local image610comprises. Thus, as depicted inFIG. 6, it can be seen that local image610comprises a focal area of image400that is just north of the center of image400.

In step340, device200may determine the scope of a contextual image based on the proximity of device200to an external surface, based on the scope of the local image, and based on the location of the local image within the full image. As used herein, the term “contextual image” may refer to an image that is both a subset of the full image and a superset of the local image. For example, as depicted inFIG. 6, once the scope of image610has been determined, and once the location of image610within image400has been determined, device200may use its proximity to projector screen600to determine the scope and content of a contextual image620such that the area comprised by local image610would be located in the center of the larger contextual image620when displayed on a projector screen600that lies behind device200by the calculated distance.

In step350, device200may display local image610on its screen210. And, in step360, device200may simultaneously display contextual image620on an external surface, such as a projector screen600. Device200may project contextual image620onto projector screen600, for example, using projector componentry260. In some embodiments, projector componentry260may be located on the side of device200opposite screen210, thus allowing a user to simultaneously view local image610and contextual image620.

In step370, device200may also project, along with contextual image620, a visual indication of the placement of local image610within contextual image620. For example, as depicted inFIG. 6, device200projects a rectangle625onto projector screen600indicating the scope and location of local image610within contextual image620. Thus, it can be seen that local image610comprises a subset of contextual image620corresponding to the “Winchester” suburb of Las Vegas, Nev. In other embodiments, the scope and location of local image610within contextual image620may be depicted or indicated in other manners, such as providing a blank space within rectangle625in which graphical data of contextual image620is not displayed.

By displaying contextual image620on projector screen600, while simultaneously displaying local image on screen210of device200, a user is able to view both the detail of local image610and the context of local image610within a fuller image400. Thus, for example, if the user were able to view only local image610, and the user desired to scroll local image610to move from a detailed view of the “Winchester” suburb to the “Sunrise Manor” suburb, the user would not know, solely by viewing local image610, in which direction the user should scroll local image610. However, by being able to view both local image610and contextual image620at the same time, the user may determine, by looking at contextual image620, that the user need only scroll “up and to the right” in order to view the “Sunrise Manor” suburb as a new local image on device200. Understanding where local image610currently fits within contextual image620is also enhanced by the display of rectangle625on contextual image620.

In some embodiments, as depicted inFIG. 6, local image610may contain greater detail or more granularity than the same area625that is simultaneously displayed on contextual image620corresponding to the scope of local image610within contextual image620. For example, as can be seen inFIG. 6, contextual image620contains only modest granularity in the form of displaying only major highways, rivers, and municipality names, and thus individual street grids or other more granular information cannot be seen in area625of contextual image620. By contrast, local image610, which corresponds to the same geographical area as area625, contains more granular detail than area625, including street grids and additional labels. This approach allows users to view fine detail for only a particular area of interest—i.e., local image610—while using contextual image620only (or mostly) to see the broader context of local image610without displaying unnecessary detail in that context. In other embodiments, as depicted inFIG. 8, both a local image810and a contextual image820may display the same amount of granular detail.

In some embodiments, it may be preferable to display a local, image on a handheld device such that it is visually aligned with or within the contextual image. As depicted inFIG. 7, a local image710is displayed on device200and a contextual image720is projected by device200onto an external surface700such that when device200is placed directly in front of external surface700, local image710is visually aligned with contextual image720. That is, when a user views both local image710and contextual image720, local image710is embedded within contextual image720such that local image710is at the same zoom level as the area of contextual image720that is obscured by device200within a line of sight. Put differently, the details—in this example, roads, highways, rivers, and labels—displayed at the boundaries of local image710visually align with the details in the areas of contextual image720that abut local image710or device200within a line of sight, such that there is continuity of details between local image710and contextual image720within a line of sight.FIG. 8also depicts such a visual alignment between a local image810and a contextual image820, such that the trajectories of streets, highways, and labels are maintained continuously in the visual transition between local image810and contextual image820.

In some embodiments, it may be necessary for device200or other devices to determine the proximity between device200and a surface800—e.g., using the above described laser or ultrasonic measurement techniques—or to analyze other relationships between the scopes and locations of local image810and contextual image820in order to achieve visual alignment. In other embodiments, a user may adjust the dimensions or zoom levels of local image810or contextual image820, either together or in isolation, or the ratio therebetween, in order to achieve a desired visual alignment, or simply to display each image at desired zoom levels irrespective of visual alignment. For example, device200may include physical or graphical user interface controls that allow users to adjust such properties.

Device200may also provide functionality to allow the user to adjust the amount of detail displayed in either local image610or contextual image620. For example, the user may adjust contextual image620to display the same level of detail (e.g., down to the street grid level) as local image610, similar that shown inFIG. 8. Or, the user may adjust local image610to match the level of detail displayed by contextual image620or to display even less detail. Thus, the level of detail displayed in either local image610or contextual image620may be adjust along a fine gradient of levels of detail, either together or in isolation.

In addition to simultaneously displaying both a local image as well as a contextual image to allow a user to see the broader context of the local image, disclosed embodiments also permit a user to focus on other areas for local display—e.g., on a handheld device—while continuing to maintain a visual relationship between a new local image and the broader contextual image.FIG. 9is a flow diagram illustrating an exemplary method of maintaining synchronization between a local image and a contextual image while allowing a user to scroll the local image, as further illustrated inFIGS. 6,10, and11, consistent with certain disclosed embodiments.

In step910, device200may receive a scrolling command indicating that a user (or program) wishes to display a different subset of image400as a new local image. For example, a user holding device200may instruct device200to focus on an area within image400that is to the south of area625. The user may instruct device200to shift focus by, for example, sliding the user's finger upward on device screen210. Those skilled in the art will appreciate that a user may initiate a scrolling action in any number of different ways, including using physical buttons or controls located on device200, using voice commands, etc. Device200may also be instructed to perform a scrolling action by receiving a command from another device or program. In one embodiment, a user may instruct device200to scroll local image610by physically moving device200to visually align with a different sub-area within image400.

In step920, device200displays a new local image in response to the scrolling command. For example, if the user slid the user's finger upward along device screen210, device200may display a new subset of image400as a new local image1010. In step930, device200may determine the scope and location of the new local image1010within image400in order to determine how to display a corresponding contextual image related to the new local image1010.

In some embodiments, depending on the configuration of device200, device200may take two different approaches to displaying a corresponding contextual image. If device200has been configured to scroll the contextual image (step940, Yes), then device200may display a new contextual image1020such that new local image1010maintains the same location or placement625within new contextual image on the display surface600(step950). That is, prior to the scrolling operation, as depicted inFIG. 6, local image610corresponded to a subset of contextual image620at the center625of contextual image620. Similarly, after the scrolling operations, as depicted inFIG. 10, new local image1010corresponds to a subset of a new contextual1010image that is similarly at the center625of contextual image1010. In other words, if device200has been configured to scroll the contextual image, then, as device200scrolls the local image on device screen210, device200concurrently scrolls the corresponding contextual image such that the local image maintains a centered, or other constant, placement within the scrolling contextual image.

In this embodiment, for example, the constant placement or geographic relationship between new local image1010and new contextual image1020may be indicated by maintaining projection of rectangle625onto projector screen600, which indicates the scope and location of local image1010within contextual image1020and depicts a consistent center alignment. This embodiment may be preferred, for example, if it is desired to maintain visual alignment between new local image1010and new contextual image1020in a manner similar to that depicted inFIG. 7orFIG. 8.

In another embodiment, as depicted inFIG. 11, if device200has not been configured to scroll the contextual image (step940, No), then device200may continue to display the same contextual image620that was displayed before the scrolling operation. And device200may instead display a new visual indicator—e.g., rectangle1125—indicating the new location or placement of new local image1110within the same contextual image620as a result of the scrolling operation (step960). In other words, if device200has not been configured to scroll the contextual image, then, as device200scrolls the local image on device screen210, device200maintains a constant contextual image while concurrently scrolling a visual indicator (e.g., rectangle625moves along display surface600to become rectangle1125) corresponding to the changing location or position of the local image within the contextual image. This embodiment may be preferred, for example, if it is not necessary to maintain visual alignment between new local image1110and contextual image620, or in the event that a user physically moves device200to visually align with a different section of display surface600.

The above-described scrolling techniques are also not limited to scrolling operations in which the dimensions of the local image remain constant, as these techniques may also be applied to scrolling operations that additionally or exclusively involve zooming operations. For example, a user could instruct device200to zoom in on a smaller subset of local image610, such that the smaller subset is magnified to fill device screen210and thus becomes the new local image. As a result, device200may similarly zoom in on a smaller subset of contextual image620, such that the smaller subset is magnified to fill projector screen600and thus becomes the new contextual image. Alternatively, if device200is not configured to “scroll” the local image (i.e., wherein “scrolling” may also refer additionally or exclusively to zooming), then, rather than zooming in on a smaller subset of contextual image620, visual indicator625may instead by zoomed (e.g., contracted) to reflect the new relationship between the new, zoomed local image and contextual image620. These zooming techniques may also be applied in addition to any horizontal and/or vertical scrolling operations. The above-described scrolling and/or zooming techniques may additionally or alternatively comprise similar operations such as panning, jumping, etc.

Although described above primarily in the context of a single device200both displaying a local image and projecting a contextual image to an external surface, other device configurations are possible. For example, given potential projection limitations of a handheld device, such as device200, device200may connect to one or more additional devices, such as a traditional table-top projector, to project contextual image620while device200continues to display local image610. In this embodiment, device200may continue to monitor its proximity to the display surface600, and may communicate such proximity information to the projector, or an intermediate device coupled to the projector, to maintain visual alignment between device200and display surface600, or to otherwise determine appropriate dimensional or resolution relationships between local image610and contextual image620.

Contextual image620may also be displayed using techniques other than projection onto a physical surface. For example, device200may be coupled with or communicate with one or more other devices for the purpose of displaying contextual image620on a separate device screen, such as an LCD, plasma, or cathode-ray screen. For example, device200could connect to a separate computer monitor screen for the purpose of displaying—either directly or indirectly through the use of one or more intermediate devices, such as a traditional desktop or laptop computer—contextual image620on the separate computer monitor screen. In this embodiment, device200may monitor its proximity to the separate computer monitor screen, and may communicate such proximity information to the computer monitor screen, or an intermediate device coupled to the computer monitor screen, to maintain visual alignment between device200and the computer monitor screen or to otherwise determine appropriate dimensional or resolution relationships between local image610and contextual image620.

Device200need also not be a handheld, or even necessarily a small, device. For example, device200could comprise a conventional desktop or laptop computer, either coupled to or integrated with an LCD, cathode-ray, or other type of computer monitor screen. A user could display local image610on a desktop computer monitor screen while contextual image620is simultaneously projected onto an external surface, such as an office or cubical wall behind the computer monitor screen. In this manner, the user could use his or her computer to view images, such as maps, documents, or other data, in a normal fashion. And, at any point, if the user wishes to view the broader context of the image data being depicted on his or her computer monitor screen, could instruct his or her desktop or laptop computer to project or otherwise display a broader contextual image on a surface that lies behind the computer monitor screen.

In other embodiments, the relationship between the image displayed on device200and an external surface600, or a separate device screen, may be reversed from that described above. In particular, rather than displaying a contextual image on the external surface or screen and displaying a subset of that contextual image as a local image on device200, device200may instead display a contextual image while projecting, displaying, or causing to be projected or displayed a local image that is a subset of the contextual image displayed on device200. In this embodiment, device200may be used to view the broader context of an externally displayed projected local image, while the local image may be displayed on or projected onto a larger screen or surface to allow for an improved view of fine details, for example in a group setting, within the local image. In this embodiment, device200may similarly display a rectangle or other visual indicator to indicate the relationship between the contextual image and the local image. And, the user may scroll one or both images using the same techniques as described above with respect toFIGS. 9-11.

Although the above-described embodiments have been described in the context of displaying map information for purposes of illustration only, other applications of the invention are possible. For example, the foregoing techniques may also be used to display and explore medical images, such as x-rays and MRIs. Details of an x-ray or MRI could be explored by displaying on a smaller device, such as a handheld device, a higher resolution local image that focuses on a particular area in the x-ray, while the full x-ray, or a broader subset of the x-ray image, could be displayed for context on an external screen or surface. In a further enhancement of this technique, the local image could display MRI information while the contextual image displays x-ray information, or vice-versa.

The foregoing techniques may also be used to explore documents, photographs, or document or photograph libraries. For example, a handheld device could display a single image—for example, using a high resolution display—while the external contextual image could display a library of images (or a subset of the library of images) to which the locally displayed image belongs. This application of the invention, for example, may not require precise visual alignment or any alignment at all. Likewise, a handheld device could display a particular section of a document or file in sufficient detail to allow the user of the handheld device to read or study certain details, while the external contextual image could display the document as a whole, or a larger subset of the document, in a lower resolution or granularity to allow the user to ascertain the context of the locally displayed section.

The foregoing techniques are also not limited to two-dimensional data, but may also applied to three-dimensional or other multi-dimensional data in order to provide simultaneous local and contextual display. For example, the foregoing techniques could be used to simultaneously display local and contextual astronomical data, such as star maps, which may conform to a spherical, rather than flat or two-dimensional, organizational structure.

The foregoing description of the invention, along with its associated embodiments, has been presented for purposes of illustration only. It is not exhaustive and does not limit the invention to the precise form disclosed. Those skilled in the art will appreciate from the foregoing description that modifications and variations are possible in light of the above teachings or may be acquired from practicing the invention. The steps described need not be performed in the same sequence discussed or with the same degree of separation. Likewise various steps may be omitted, repeated, or combined, as necessary, to achieve the same or similar objectives. Accordingly, the invention is not limited to the above-described embodiments, but instead is defined by the appended claims in light of their full scope of equivalents.