Display application and perspective views of virtual space

A display management resource associated with a mobile device controls display of images on a respective display screen of the mobile device. The display management resource receives location information indicating a location of the mobile device in a geographical region. Additionally, the display management resource receives input from a user operating the mobile device. The input can be any suitable information such as a command to play back images on the display screen of the mobile device. The display management resource maps the input to content such as virtual images such as images associated with an historical event that occurred in the past. Using the virtual images, the display management resource initiates display of a rendition of the virtual images from different perspectives depending on an orientation and location of the mobile device and corresponding display screen.

BACKGROUND

In general, augmented reality is a view of a physical, real-world environment whose elements are augmented in some manner. For example, it is well known that a camera disposed on a respective mobile communication device can be used to capture a respective image and display it on a display screen. Augmented reality can include creating a supplemental image (that is not in the field of view of the camera) and overlaying the supplemental image onto the respective image captured by the camera. Via the overlay of the supplemental image onto a real captured image, the image of reality is augmented by the supplemental image.

BRIEF DESCRIPTION OF EMBODIMENTS

Embodiments herein deviate with respect to conventional display techniques. For example, embodiments herein include novel ways of producing viewable renditions of content for display on a respective display screen of a mobile device depending on a location and orientation of the display screen in a three-dimensional space. In one embodiment, the techniques as described herein enable a respective user to use a display screen of a mobile device as a window into history.

More specifically, in accordance with one embodiment, a display management resource associated with a mobile device controls display of images on a respective display screen of the mobile device. In one embodiment, the display management resource receives location information indicating a location of the mobile device in a geographical region. Additionally, the display management resource receives input from a user operating the mobile device. The input can be any suitable information such as a command to play back images on the display screen of the mobile device. The display management resource maps the input to content such as virtual images of an historical event (which, in one embodiment, occurred at the location in the past). Using the virtual images, the display management resource initiates display of a rendition of the virtual images (such as historical event) from different perspectives depending on an orientation of a display screen of the mobile device at the location

In accordance with more specific embodiments, the mobile device can be configured to include a location detection resource that detects a location of the mobile device in a geographical region. The mobile device also can be configured to include an orientation detection resource that detects an angular orientation (in any number of dimensions) of the display screen/mobile device at the mobile device's current location. The display management resource in the mobile device receives the location information from the location detection resource. The display management resource receives orientation information (such as an orientation vector) from the orientation detection resource. The orientation information (such as orientation vector) indicates a direction that an image sensor device (such as a camera) on the mobile device points in the geographical region. The display management resource utilizes the vector to define a viewing window through a first image layer and a second image layer. The viewing window defines a region in the first image layer to be superimposed onto a region in the second image layer. Each of the image layers can include one or more virtual objects. The display management resource utilizes the region in the first image layer (and one or more corresponding virtual objects) and the region in the second image layer (and one or more corresponding virtual objects) to produce a viewable rendition of the historical event for display on the display screen of the mobile device. Renditions of virtual objects in the image layer nearest the user of the mobile device can be given highest priority for display. For example, virtual objects in the first image layer can be displayed over virtual objects in the second farther layer if the objects occupy a same line of view.

Embodiments herein further include providing perspective views of the virtual objects in different layers depending on an orientation and location of the mobile device in a physical space.

One or more embodiments herein are useful over conventional techniques because they provide the user with a more realistic viewing experience of an historical event than conventional techniques. For example, according to certain embodiments herein, via updating of a perspective virtual view, a user is able to move about a geographical and view around renditions of virtual objects associated with the historical event in a viewing window.

These and other more specific embodiments are disclosed in more detail below.

Note that any of the resources as discussed herein can include one or more computerized devices, mobile devices, servers, base stations, wireless communication equipment, communication management systems, workstations, handheld or laptop computers, or the like to carry out and/or support any or all of the method operations disclosed herein. In other words, one or more computerized devices or processors can be programmed and/or configured to operate as explained herein to carry out different embodiments of the invention.

One or more embodiments herein include a computer readable storage medium and/or system having instructions stored thereon. The instructions, when executed by computer processor hardware, cause the computer processor hardware (such as in a mobile computer device) to: receive location information, the location information indicating a location of a mobile device in a geographical region; receive input from a user operating the mobile device; map the input to an historical event that occurred at the location; and initiate display of a rendition of the historical event from different perspectives depending on an orientation of a display screen of the mobile device.

One or more embodiments herein include a computer readable storage medium and/or system having instructions stored thereon. The instructions, when executed by computer processor hardware, cause the computer processor hardware (such as in a security system) to: receive location information indicating a location of a mobile device in a geographical region; receive a vector indicating an orientation of a display screen of the mobile device; utilize the vector to define a viewing window through a first image layer and a second image layer, the viewing window defining a region in the first image layer to be superimposed onto a region in the second image layer; and utilize the region in the first image layer and the region in the second image layer to produce a viewable rendition for display on the display screen of the mobile device.

The ordering of the operations above has been added for clarity sake. Note that any of the processing steps as discussed herein can be performed in any suitable order.

It is to be understood that the system, method, apparatus, instructions on computer readable storage media, etc., as discussed herein also can be embodied strictly as a software program, firmware, as a hybrid of software, hardware and/or firmware, or as hardware alone such as within a processor, or within an operating system or a within a software application.

As discussed herein, techniques herein are well suited for facilitating installation of corresponding wireless access points in a network environment. However, it should be noted that embodiments herein are not limited to use in such applications and that the techniques discussed herein are well suited for other applications as well.

Also, note that this preliminary discussion of embodiments herein purposefully does not specify every embodiment and/or incrementally novel aspect of the present disclosure or claimed invention(s). Instead, this brief description only presents general embodiments and corresponding points of novelty over conventional techniques. For additional details and/or possible perspectives (permutations) of the invention(s), the reader is directed to the Detailed Description section and corresponding figures of the present disclosure as further discussed below.

DETAILED DESCRIPTION AND FURTHER SUMMARY OF EMBODIMENTS

FIG. 1is an example diagram illustrating a mobile device configured to execute operations according to embodiments herein. Note that in addition to the resources as shown, mobile device120can include any additional hardware and software resources to carry out operations as discussed herein.

More specifically, as shown in this example embodiment, mobile device120(such as a IPAD™, IPOD™, mobile phone device, laptop computer, tablet computer, etc.) includes a display screen130, image sensor device150, input resource102, orientation detection resource195, location detection resource196, display management resource140, communication interface155, etc.

Each of the resources in mobile device120operate in a cooperative manner to display perspective views of virtual images (which may include a portion of real-time captured images) such as historical events.

For example, input resource102can be any suitable type of resource such as a keypad, touchscreen, mouse, microphone, etc., to receive control input105from user108. Via input resource102, user108is able to control operations of mobile device120.

During operation, display screen130initiates display of corresponding images (virtual images) in accordance with input from display management resource140.

Note that the execution of display management resource in mobile device120is shown by way of non-limiting example only and that all or a portion of the functionality associated with display management resource140can be executed at one or more remote locations with respect to the mobile device120.

In this example embodiment, mobile device120includes one or more image sensor devices to capture images of objects in a surrounding geographical region. For example, mobile device120can include an image sensor device150on a front facing of the mobile device120(opposite a facing of the display screen130) to view images in front of user108and mobile device120in the geographical region. In accordance with further embodiments, the mobile device120can include an additional image sensor device to receive images from an opposite direction such as from a direction of the user108viewing display screen130. Thus, display management resource140can receive images of objects located in front of the mobile device120as well as receive images of user108and objects located in back of the mobile device120.

As its name suggests, the orientation detection resource195constantly monitors an orientation of the mobile device120and corresponding display screen130. Orientation detection resource195can include any suitable circuitry to detect a respective orientation of the mobile device120and corresponding display screen130in three-dimensional space. In one embodiment, the orientation detection resource includes any suitable resources such as accelerometer circuitry to produce orientation information (such as compass data). The orientation detection resource195produces orientation information or compass data (indicating an angular orientation of the mobile device120in three-dimensional space) and forwards it to display management resource140.

Via the orientation information received from orientation detection resource195, the display manager140is apprised of a current orientation of the mobile device120and corresponding display screen130. As will be discussed later in the specification, the orientation information will be useful to generate an appropriate perspective of virtual images for display on the display screen130for viewing by user108.

The orientation detection resource195can be configured to detect an orientation of the mobile device120and corresponding display screen130in any suitable manner. For example, in one non-limiting example embodiment, the orientation detection resource195can be configured to receive input from image sensor device150disposed on front facing of the mobile device120opposite the display screen130. Given a current location of the mobile device120, the orientation detection resource195can be configured to use image recognition techniques to determine orientation based on real-time images from the image sensor device150.

In other words, while the mobile device120is at a given location in three-dimensional space, the orientation detection resource195can be configured to receive an real image captured by image sensor device150(such as a camera) on the mobile device120; detect a reference item such as a building, landmark, etc., captured in the real image; and utilize the detected reference item in the captured real image relative to the current location of the mobile device120as a basis to determine the directional orientation of the mobile device120/display screen130in the geographical region. Thus, when the current location of the mobile device120is known, the orientation detection resource195can be configured to identify renditions of one or more known objects (items) in the geographical region to derive the orientation of the display screen130.

In accordance with further embodiments, the orientation detection resource195can include appropriate circuitry (such as one or more gravity sensors) to detect an angular orientation of the mobile device120in three-dimensional space. In such an instance, there is no need to rely on use of captured images to detect an orientation of the mobile device120.

Also, as its name suggests, the location detection resource196monitors a whereabouts of the mobile device120in a respective geographical region. In one embodiment, the location detection resource196constantly monitors a current location of the mobile device120in the geographical region. The location detection resource196forwards the location information to display management resource140. The display manager140executing in the mobile device120(or other suitable resource) utilizes the received location information to keep track of the current location of the mobile device120in the geographical region.

Note that the location detection resource196can be or include any resources to detect the current location of the mobile device120. In one non-limiting example embodiment, the location detection resource196includes or relies on use of GPS (Global Positioning System) resources to determine a current location of the mobile device120in the geographical region.

In accordance with further embodiments, the location of the mobile device120can be detected using satellite navigation systems such as GPS (used in the U.S.), Galileo (used in Europe), GLONASS (used in Russia), IRNSS (used in India), Compass (used in China), or the like. Further embodiments herein can include use of terrestrial-based location services such as LORAN, cell tower triangulation, inertial navigation services, WLAN-SSID-based approaches, WiFi™ triangulation methods, etc., to determine a location of the mobile device120in the geographical region.

Thus, location detection resource195can rely on use of any suitable location detection services or methods to produce location information for use by display management resource140.

In accordance with one embodiment, the display management resource140enables the respective user108to play back images of an historical event that occurred in a vicinity of the location where the user and corresponding mobile device120currently resides.

More specifically, in one embodiment, a user108can select one of multiple historical events that have occurred at the current location of the mobile device120. As a more specific example, the user108can provide input105specifying a time in history when a selected historical event occurred. In response to receiving selection of a particular historical event or time, the display manager140utilizes map information170to map the selected historical event or time to corresponding virtual image data175(such as still images, video images and audio, etc.) stored in repository108. Via processing of the virtual image data175and generation of one or more perspective views, the display management resource140initiates display of a reenactment of the corresponding historical events on display screen130for viewing by respective user108of mobile device120.

As will be discussed further in this specification, embodiments herein include producing the rendition of the historical event on display screen130to be from a viewing perspective as defined by the directional orientation and current location of the display screen130of the mobile device120.

Just as the processing capability associated with display management resource140can be located in any suitable one or more resources, note that the data stored in repository180can be stored in any suitable location. For example, in one embodiment, the repository180can be located in the mobile device120. In such an instance, the display management resource140performs local READ accesses to the repository180to retrieve appropriate portions of virtual image data175to assemble images (and corresponding audio) for playback on the mobile device120.

Alternatively, the repository108can be disposed in a respective network190accessible to the mobile device120. In this latter instance, the display management resource140can be configured to utilize communication interface155to establish respective communication link128(such as a wired or wireless communication link) with base station135. The station135supports communications through network190to server resource165. The server resource165accesses virtual image data175stored in repository180and transmits the retrieved data over network190to display management resource140.

Network190can include any number of different types of networks such as the Internet, cellular phone networks, WiFi™ networks, etc.

In yet further embodiments, note that virtual image data175can be stored in multiple locations. For example, a first portion of the virtual image data175can be stored locally in mobile device120; a second portion of virtual image data175can be remotely stored in repository180, and so on. In such an instance, the display management resource140performs appropriate local and remote accesses to retrieve appropriate portions of the virtual image data175to generate images for display on display screen130.

As previously discussed, as further discussed below, the display management resource140receives location information and orientation information from location detection resource196and orientation detection resource195. Depending on the location and orientation of the display screen130, the display management resource140initiates display of a rendition of content such as an historical event from different perspectives. In one non-limiting example embodiment, the rendition of the historical event displayed on the display screen130of the mobile device120is a reenactment of a historical event that occurred in a vicinity of the location in the past. As an example, a user using the mobile device120in the year 2014 can utilize the display screen120as window into the past to view events that occurred in the year 1905. Thus, the user108can utilize the mobile device120and corresponding display screen130to view virtual images of a reenactment of a historical even that occurred minutes, days, years, etc., ago with respect to the current time. To provide a more realistic reenactment to the user108, the rendition of the historical event derived from virtual image data175can include one or more actual images of the historical event.

FIG. 2is an example diagram illustrating use and calibration of a mobile device in a geographical region according to embodiments herein.

Embodiments herein can include calibrating an orientation of the mobile device120to a reference.

As shown in this example, landscape200in geographical region210includes object260-1(such as tree), object260-2(such as a landmark), reference object270(such as tree), etc. The user108operates mobile device120in region of operation250including multiple locations where the user can stand to view the landscape200. In this example embodiment, region of operation250includes multiple different locations (L1, L2, L3, L4, L5, etc.) where the user108can stand. In this instance, assume that the user108and corresponding mobile device120are disposed at location, LC (such as a center location of region of operation250).

To calibrate an orientation of the mobile device120in the geographical region including landscape200, the display management resource initiates display of a reference image at a fixed location on display screen130(as shown inFIG. 3). As discussed below, the user108points image sensor device150on a front facing of the mobile device120to capture landscape200and corresponding objects. Display management resource140receives the signals produced by the image sensor device150and initiates display of a respective rendition of the objects on display screen130depending on the particular direction that the image sensor device150is pointed. The display management resource140can be configured to initiate display of reference image220(such as cross-hairs on a scope, silhouette or outline of a landmark=, etc.) at a fixed location on display screen130.

FIG. 3is an example diagram illustrating calibration of a mobile device according to embodiments herein.

To calibrate an angular orientation of the display screen130and corresponding mobile device120in physical three-dimensional space, the display management resource140initiates display of image information captured by a camera disposed on a front facing of the mobile device120. As a user108changes an angular orientation of the display screen, the image sensor device150displays the detected real captured images (e.g., objects such as object260-1, object260-2, reference object270, etc.) on display screen130.

The display screen130displays real-time images captured by image sensor device150. Because the display management resource140displays current captured images on display screen130, the rendition of the corresponding objects on display screen130moves about display screen130as the user108adjusts the angular orientation of the display screen130.

To perform the calibration, the user108adjusts an angular orientation of the mobile device120such that the reference image220aligns with the rendition of the reference object270-R as captured by the image sensor device150. The display management resource140continuously receives updated orientation input from orientation detection resource195. At a time when the reference image220and rendition of the reference object270-R are aligned, the display management resource140defines the corresponding angular orientation of the mobile device120to be a reference orientation such as zero degrees. This calibration enables the display management resource140to precisely know an angular orientation of the mobile device120at subsequent times of use. The orientation information generated by the orientation detection resource195indicates an angular orientation of the mobile device120and/or corresponding display screen130with respect to the reference object270in the physical three-dimensional space. I

As will be discussed later in this specification, the display management resource140will use the orientation information produced by the orientation detection resource195to map the orientation of the mobile device120to virtual images defined in a three-dimensional space. That is, in one non-limiting example embodiment, instead of displaying images captured by the image sensor device150, the display management resource140initiates display of virtual images derived from a three-dimensional virtual image space.

FIG. 4is an example diagram illustrating multiple image layers including respective virtual objects in a three-dimensional virtual image space according to embodiments herein.

As shown, virtual image data175defines images (still or moving) of one or more virtual objects at multiple different image layers440in a three-dimensional virtual image space. The three-dimensional virtual image space (such as based on an historical event) corresponds in location to the actual physical three-dimensional space in geographical region210. Assume that the virtual object VO35 (at least location wise) corresponds to the reference object270(such as a landmark) in physical reality of geographical region210.

In this example embodiment, the first image layer440-1includes a first set of virtual objects such as virtual object VO11, virtual object VO12, etc., which are located nearest to user108in the three-dimensional virtual image space; the second image layer440-2includes a second set of virtual objects including virtual objects VO21, VO22, VO23, etc., which are located next further out from user108in the three-dimensional virtual image space; the third image layer440-3includes a third set of virtual objects VO21, VO22, VO23, VO35, etc., which are located next further out from user108in the three-dimensional virtual image space; and so on.

In one non-limiting example embodiment, each of the layers represents a virtual dome. Thus, virtual image data175can define domes of virtual objects present in virtual three-dimensional space corresponding to the users physical space.

Each of the image layers440can represent objects within a range of distance from the centroid location LC. By way of non-limiting example, the first image layer440-1includes virtual objects residing within a first range or distance band such as a distance band between 10 and 30 feet away from location LC. In such an instance, each of the virtual objects VO11, VO12, etc., appear to reside within 10 and 30 feet from location LC. The second image layer440-2includes virtual objects residing within a second range or distance band such as a distance band between 30 and 150 feet away from location LC. In such an instance, each of the virtual objects VO21, VO22, etc., appear to reside within 30 and 150 feet from location LC. The third image layer440-3includes virtual objects residing within a third range or distance band such as a distance band between 150 and 1000 feet away from location LC. In such an instance, each of the virtual objects VO31, VO32, etc., appear to reside within 150 and 1000 feet from location LC.

Thus, the first image layer440-1can be configured to define a first set of virtual objects residing within a first distance band with respect to the location; the second image layer440-2can be configured to define a second set of virtual objects residing within a second distance band; the third image layer440-3can be configured to define a third set of virtual objects residing within a third distance band; and so on.

In accordance with further embodiments, each of the virtual objects in a layer is defined by one or more display elements (such as pixels). Any suitable resolution can be used to define the virtual images and/or virtual objects disposed in the different layers.

Each of the different image layers440can define moving pictures such as video images. In such an instance, the virtual image data175defines settings of the display elements (such as pixels) in each of the image layers440over time. Thus, the virtual image data175can be configured to define a virtual three-dimensional space.

As previously discussed, the user108operates the mobile device120and corresponding display screen130as a window to view virtual objects in the different image layers440. The display management resource140utilizes the directional orientation vector230(to determine an orientation) of the mobile device120to define a viewing window through each of the virtual image layers440. Depending on the angular orientation of the mobile device120, the display management resource140initiates display of a window of different sets of one or more virtual objects for viewing by user108on display screen130.

As a more specific example, the orientation detection resource195can be configured to constantly provide display management resource140an update of the mobile device's current orientation. The orientation is defined by directional orientation vector230. The display management resource140utilizes the directional orientation vector230to derive the rendition of images (potentially from a selected historical event) to be displayed on display screen130for viewing by user108. The display management resource140uses the directional orientation vector230to identify which of the virtual objects (at different distances from the user108) in the different image layers440are to be used to produce a respective image on display screen130. The display management resource140dynamically displays a rendition of virtual images of the historical event on the display screen130from a viewing perspective of the directional orientation of the mobile device120.

By panning the mobile device120(such as changing its angular orientation in physical space), the user108is able to view different portions of the historical event on the display screen130in accordance with a panning motion of the mobile device. As further described herein, the panning motion of the mobile device120(and corresponding image sensor device150) enables the user108to view different actions associated with the historical event that occur at different angular directions with respect to the user's current location. In certain instances, in addition to panning, the user108of the mobile device120may move from one location to another in the region of operation250. Based on the orientation and location of the mobile device120, the display management resource140provides updates perspective view of the virtual three-dimensional space for viewing by respective user108on display screen130.

Note that further embodiments herein can include utilizing real-time or pseudo real-time captured images as a basis for a respective background as opposed to pre-recorded virtual images derived from the furthest image layer (such as image layer440-3) as the background. In such an instance, one or more image layers such as image layers440-1and440-2represent pre-recorded content that is superimposed onto a background image (real-time captured images). In a manner as previously discussed, the image layers440-1and440-2capture movement of virtual objects (such as people, animals, etc.) in a three-dimensional virtual space. The virtual images as defined by the image layers440-1,440-2, etc., represent virtual objects that are superimposed over the real-time captured background.

In accordance with yet further embodiments, the user's viewing experience can be interactive or conditional. For example, one or more image layers can be configured to capture an historical event such as a person (such as Paul Revere) riding a horse down a path. The virtual events displayed on display screen130are conditional. For example, the display management resource140can be configured to monitor input such as a voice of user108. If the display management resource140detects that the user108utters a phrase such as “HI” or “STOP,” the display management resource140selects a rendition of video in virtual image data in which the horse stops, enabling the rider of the horse to engage speaking with the user108. Alternatively, if the user108is silent (e.g., the user108does not utter an attention grabbing phrase) as the rider of the horse approaches, the display management resource140alternative detects this condition and selects playback of virtual image data in which the rider of the horse continues riding past the user108without stopping. Accordingly, in one embodiment, the (virtual) historical events played back on the display screen130can vary depending on further input from the user108.

In accordance with further embodiments, using light coding techniques, the display management resource140can be configured to match the vectors in: a photorealistic, 180 degree-image-dome-composite of images (a version of image layers440). The image layers440can be made up of three or more 1080p output resolution background plates and three or more layers of live action assets, such as actors and cars, filmed against a green screen and three or more layers of three dimensional computer generated image assets, to: existing landmarks in the user's location, such as buildings, streets and sidewalks.

As the user108moves, the location detection resource196and orientation detection resource195in the mobile device120rotate and shift on X, Y, and Z axis all of the nine or more layers of video, photo and CGI assets, keeping both virtual and actual landmark positions, over-laid (geographically-registered) from the user's perspective, looking through the display screen130of the mobile device120and beyond it, to the actual landscape. The display screen130of the mobile device120then becomes a seamless window into a matching world that appears to be a parallel universe. So, as the user108moves within a limited space and turns around three hundred and sixty degrees, the user108experiences the illusion that he or she is looking through a window into another world that is exactly over-laid (such as geographically-registered) with his or her own physical world.

In the context of certain embodiments herein, historic scenes can be filmed and finished using the above technique to create a window into the past and an apparent view of the user's surroundings, from long ago. This synchronous parallel universe, that is, a WINDOW TO ANOTHER WORLD can be created when the light coding data collected from the user's mobile device of the user's surroundings is matched to tracking marks (and filmed land marks) in the live-action, video-composite, virtual dome (such as image layers440).

In accordance with still further embodiments, assets can be downloaded via any suitable wireless communication link such as from a WiFi™ hotspot or pre-loaded onto the mobile device120with download media packages that accompany the display management resource140. Eventually, as wireless bandwidth increases and becomes more available to a respective user108to receive display information, neither of the above asset delivery methods may be necessary as all assets could potentially be downloaded to and uploaded from the mobile device120via digital a wireless communication link such as a cellular wireless signal, WiFi™ link, etc.

Using a light-coding, infrared strobe on the face-side of the mobile device120as well as on the back-side of the mobile device120, the user108can enter the environment as a CG avatar and interact with the avatars of other users, who may be experiencing the environment in a Virtual Reality suit and Oculus Rift goggles or similar immersive interfaces. In one embodiment, if other people in the same geographical region are holding up their mobile devices and experiencing an historical event as described herein, each user can see the other users as custom-chosen characters from that time period.

In yet further embodiments, users would have the ability to interact with each other as well as potentially interact (in limited ways) with the live-action actors, vehicles and other assets in the historical recreation. The user108will have a specific radius (such as region of operation250) in which he or she is able to move about within the virtual world. The size of region of operation250can be determined by the user-perceived parallax authenticity parameters of the video dome composite (image layers440).

Additionally, note that as processor power in mobile devices grows, image layers can be added to the video dome composite (image layers440). The greater number of layers440associated with virtual image data170to support more complex parallax shift views that are rendered in real-time on respective mobile devices. If there is more than one user present with a mobile device in the immediate geographic area, processing can be distributed amongst the mobile devices via a wireless communication link such as WiFi™, cellular digital, etc., across available devices, so all users within a given geographical region, experience the immersive re-creation, with parallax shift renders running at the same speed across every device. In other words, their experiences are in sync, so if one user looks at the device of another user which happens to be pointed in a different direction, the user could observe an asset on the screen of device B, that travels in a life-like, believable way to device A.

Yet further, multiple WINDOWS (mobile devices), increasing in number, will theoretically create a video dome within the virtual video dome and a malleable and potentially amorphous view of the entire parallel world of the historical or other recreation.

Multiple user experiences could take place anywhere that vehicular traffic or other risk-to-life elements are not present. For example, in a large park that long ago was farm land, users could see and interact with avatars (selected characters) of each other and investigate and explore different facets of a scripted story, being played out by actors who are elements in the immersive re-creation. As an example, ten different users could experience the scripted story in ten different ways, while being completely immersed in the world of the narrative, which is taking place around them, complete with sets, actors, props and sounds.

Re-creation zones (such as region of operation250) can include strategically placed sources of re-created smells from the period depicted in the immersive, interactive experience. In such an instance, users would see, hear and to some small degree, smell the environment from long ago, brought to life around them.

As further described herein, using a front-camera-light-coding sensor and the camera itself, the display management resource140in the mobile device120can be configured to create a real-time single or multiple frame composite of the user and the environment, thus allowing the user to take a “selfie” picture, while time traveling through our augmented reality app, into the past.

FIG. 5is an example diagram illustrating multiple image layers of virtual objects according to embodiments herein.

Note that as an alternative to producing the virtual image space such as domes, embodiments herein can include producing planes of virtual image layers540-1,540-2,540-3, etc. In a similar manner as previously discussed, each of the virtual image layers540can define objects at different distances from the user's current location. For example, the image layer540-1can include virtual objects present between 10 and 30 feet away from a user, the image layer540-2can include virtual objects present between 30 and 150 feet away from a user, the image layer540-3can include virtual objects present between 150 and 1000 feet away from a user, and so on.

FIG. 6is an example top-view diagram illustrating a mobile device located at a first location and oriented in a first orientation in a physical domain according to embodiments herein.

As shown, assume in this example that the display management resource140receives location information from location detection resource196indicating that the mobile device120is located at location LC in geographical region210. The display management resource140also receives a vector230from orientation detection resource195indicating the orientation of the display screen130and corresponding image sensor device150.

The display management resource140utilizes the vector230to define a viewing window650for creating a virtual image for display on display screen130. In reality, the actual viewing window650associated with corresponding image sensor device150captures a view of reference object270and respective surroundings in the geographical region210. However, as further shown in the followingFIG. 7, based on the current location and orientation of the mobile device120, the display management resource140produces a virtual viewing window through each of one or more virtual image layers to produce a viewable rendition of content for display on display screen130. As previously discussed, the images displayed on display screen130can be completely or only partially virtual.

More specifically,FIG. 7is an example diagram illustrating construction of a virtual image in accordance with the first location and the first orientation of a mobile device according to embodiments herein.

As shown, in the virtual reconstruction domain, the display management resource140utilizes virtual viewing window750(corresponding to the actual viewing window650or view as defined by image sensor device150) as a basis to determine which if any virtual objects present in the different image layers440of the virtual image three-dimensional space are to be displayed on corresponding display screen130for viewing by user108.

In this instance, when the user points the image sensor device150of the mobile device120towards reference object270(the reference point), the display management resource processes virtual image data175. Assume that during processing, the display management resource140detects that virtual object VO16 resides in the virtual viewing window750in image layer440-1; the display management resource140detects that virtual object VO25 resides in the virtual viewing window750in image layer440-2; the display management resource140detects that virtual object VO35 (rendition of reference object270) resides in virtual viewing window750in image layer440-3.

When producing a respective image for display on display screen130, the display management resource140gives precedence to any virtual objects in the nearest image layers440because they would occlude a view of virtual objects disposed at further distances from the user's current location. For example, as further shown inFIG. 8, virtual object VO16 (circle) detects in image layer440-1is displayed as being in front of virtual objects VO25 (square) and VO35 (landmark); virtual object VO25 (square) detected in image layer440-2is displayed as being in front of virtual object virtual object VO35 (landmark).

Accordingly, the display management resource140initiates display of the rendition of virtual object VO16 (circle) in front of the rendition of virtual object virtual object VO25 (square). The display management resource140displays the rendition of the virtual object VO25 (square) in front the rendition of virtual object VO35 (building).

As previously discussed, note again that the viewable rendition (virtual images) displayed on display screen130can be represent an historical event that occurred at the geographical region in the past; each of the image layers440include virtual objects associated with the historical event.

In one embodiment, the display management resource can be configured to display a portion of virtual images derived from virtual image data175as well as real images captured by a respective image sensor device of the mobile device120. For example, as previously discussed, the mobile device120can be configured to include a image sensor device to capture images of a user108viewing display screen130. Embodiments herein can include detecting the user108and overlaying an image of the user108onto the virtual image shown on the display screen130inFIG. 8.

FIG. 9is an example top-view diagram illustrating a mobile device located at a second location and oriented in a second orientation in a physical domain according to embodiments herein.

In this example, assume that the user108operating the mobile device120physically moves from location LC to location L2 in region of operation250. The location detection resource196detects this condition and notifies display management resource140that the mobile device120is now located at location L2 instead of location LC. Assume further that the user108changes an orientation of the mobile device120to a new direction as shown. The display management resource140receives a directional orientation vector230indicating the new orientation of mobile device120.

In a manner as previously discussed, the display management resource140utilizes the vector230to define a viewing window650for creating a virtual image from virtual image data175. As shown inFIG. 9, in reality, the actual viewing window650associated with corresponding image sensor device150(if activated) captures a partial view of reference object270(represented by virtual object VO35) and respective surroundings in the geographical region210. In this example, the image sensor device150of mobile device120is pointed to a location to the right of the reference object270in the geographical region210.

As further shown in the followingFIG. 10, to account for the change in location and orientation of the mobile device120to the second location and second orientation, the display management resource140produces a virtual image from the perspective of the viewing window750(corresponding to viewing window650) inFIG. 10. In such an instance, virtual viewing window750captures virtual objects in each of one or more virtual image layers to produce a viewable rendition of images for display on display screen130. Thus,FIG. 10shows the virtual image that should be created for display on display screen130.

Rather than performing complex processing of virtual image data175with respect to location L2 and corresponding second orientation as shown inFIG. 10, embodiments herein include producing a respective virtual image for the mobile device120at the second location and second orientation using the technique shown inFIG. 11. As shown inFIG. 11, the display management resource140assumes the mobile device120is located at the location LC. However, the display management resource140rotationally offsets one or more of the image layers440to produce a corresponding virtual image for display on display screen130.

In other words, based on detecting the second location and second orientation, the display management resource140detects that the mobile device120is pointed at virtual object VO16 in layer440-1as shown inFIG. 11. Assume that location L2 is 10 feet from location LC and that the angular view of mobile device120is oriented or turned to be clockwise by 18.43 degrees with respect to the reference orientation of 0 degrees. Assume further that image layer440-1represents a layer of virtual objects 30 feet from location LC; image layer440-2represents a layer of virtual objects 60 feet from location LC; image layer440-3represents a layer of virtual objects 90 feet from location LC; and so on. In such an instance, virtual object VO16 becomes the center of the virtual image to be displayed on display screen130. Applying trigonometry (such as in related U.S. Provisional Patent Application Ser. No. 62/006,581 entitled “Parallax Equations for Use in Mobile Devices”) to the second location and second orientation information, the display management resource140rotates an original orientation of the image layer440-2by approximately 9 degrees counterclockwise; the display management resource140rotates the original orientation of image layer440-3clockwise by approximately 12.1 degrees in order to create an appropriate virtual image for the second location and second orientation. There is no need to rotationally offset layer image layer440-1because the center of viewing window750points directly at virtual object VO16.

As shown, the rotation of the one or more virtual image layers440makes it possible for the display management resource140to produce a virtual image from the perspective of the mobile device120at the location LC and corresponding orientation even though the mobile device120is located at location L2.

Subsequent to applying the rotational offset to one or more image layers440based on the user's current location and orientation, in a similar manner as previously discussed, the display management resource140identifies which of the virtual objects resides in the viewing window750from the perspective of location LC (instead of location L2) and produces the corresponding virtual image as shown inFIG. 12. Because the user108has moved to the second location and changed the orientation of the mobile device120, the display management resource140displays a different view from the virtual three-dimensional space as defined by virtual image data175.

In accordance with further embodiments herein, when a user108changes a location and orientation of the mobile device120, the display management resource140constantly and quickly provides the user108with a proper parallax view of the virtual window even though the virtual image data175is taken from the perspective of a viewer at a single point (such as location LC). In other words, as previously discussed, the amount of rotation of the one or more image layers440depends on the current angular orientation and location of the mobile device120. The simulated virtual view derived from offsetting one or more image layers440gives the user108different perspective views in a three-dimensional space as a user moves from one location to another and/or changes an orientation of the mobile device120.

Accordingly, embodiments herein include detecting movement of the mobile device120from a first location (such as location LC) to a second location (such as location L2). Depending on the orientation and/or location of the of the display screen130mobile device120at the second location, the display management resource140one or more offsets the multiple image layers140in a direction substantially orthogonal to the direction of viewing to derive a rendition of a virtual event (such as a historical event) for display on the display screen while at the second location, L2. The parallax view on of real-time and/or virtual objects on display screen130enables the user108to peer around objects. For example, because the user108moved to new location L2, the user108is able to view around a rendition of virtual object VO16 to get a better view of the rendition of virtual object VO25.

In addition to providing different parallax views, embodiments herein can include providing a user108adjusting respective views depending on movement of a respective mobile device120closer or further away from a respective virtual object. For example, the user108operating mobile device120can move 10 or more feet closer to the landscape200inFIG. 2. In such an instance, because the display management resource140detects movement closer to reference object270, the display management resource140applies appropriate magnification to the virtual objects and displays them on display screen130for viewing. Accordingly, as the user108and corresponding mobile device120move closer to reference object270, the images of corresponding virtual objects are magnified a proportional amount to accommodate the movement. Conversely, the display management resource140reduces the size of respective virtual objects in the display screen130when the user108moves further away from the landscape200.

FIG. 13is an example top-view diagram illustrating a mobile device located at a third location and oriented in a third orientation in a physical domain according to embodiments herein.

In this example, assume that the user108operating the mobile device120is physically located at location L2 in region of operation250. The location detection resource196detects this condition and notifies display management resource140that the mobile device120is located at location L2. Assume further that the user108changes an orientation of the mobile device120to a new direction as shown. The display management resource140receives a directional orientation vector230indicating the new orientation of mobile device120.

In a manner as previously discussed, the display management resource140utilizes the vector230to define a viewing window650for creating a virtual image from virtual image data175. As shown inFIG. 13, in reality, the actual viewing window650associated with corresponding image sensor device150(if activated) captures a partial view of reference object270(represented by virtual object VO35) and respective surroundings in the geographical region210. In this example, the image sensor device150of mobile device120is pointed to a location to the left of the reference object270in the geographical region210. Assume that the mobile device120and corresponding display screen130is disposed at a same angular orientation as discussed above inFIG. 6. In such an example embodiment, the display management resource140displays a different virtual view on display screen130than as shown inFIG. 8because the mobile device120is now located at location L2 instead of location LC.

As further shown in the followingFIG. 14, to account for the change in location and orientation of the mobile device120to location L2 and corresponding zero degree orientation, the display management resource140produces a virtual image from the perspective of the viewing window750(corresponding to viewing window650). In such an instance, virtual viewing window750captures virtual objects in each of one or more virtual image layers440to produce a viewable rendition of images for display on display screen130. Thus,FIG. 14shows the virtual image that the display management resource140will create and display on display screen130.

As previously discussed, rather than performing complex processing of virtual image data175with respect to location L2 and the corresponding orientation as shown inFIG. 13, embodiments herein include producing a respective virtual image for the mobile device120at the second location and orientation using the technique shown inFIG. 15. As shown inFIG. 15, the display management resource140assumes the mobile device120is located at the location LC. However, the display management resource140rotationally offsets one or more of the image layers440to produce a corresponding virtual image for display on display screen130based on the mobile device's current location and orientation.

In other words, based on the current location and orientation of the mobile device120inFIG. 13, the display management resource140detects that the mobile device120is pointed to left side of virtual object VO16 in layer440-1as shown inFIG. 15. In such an instance, the left side of virtual object VO16 becomes the center of the virtual image to be displayed on display screen130. Applying trigonometric calculations based on an orientation and location of the mobile device120, the display management resource140rotates an original orientation of the image layer440-1clockwise by a first offset value; the display management resource140rotates an original orientation of the image layer440-2clockwise by a second offset value; the display management resource140rotates an original orientation of the image layer440-3clockwise by a third offset value; and so on.

As shown, the rotation of the one or more virtual image layers440makes it possible for the display management resource140to produce a virtual image from the perspective of the mobile device120at the location LC and corresponding orientation even though the mobile device120is actually located at location L2.

Subsequent to applying the rotational offsets to one or more image layers440based on the user's current location and orientation, in a similar manner as previously discussed, the display management resource140identifies which of the virtual objects resides in the viewing window750from the perspective of location LC (instead of location L2) and produces the corresponding virtual image as shown inFIG. 16. Because the user108resides at the second location and changed the orientation of the mobile device120, the display management resource140displays a different perspective view derived from the virtual three-dimensional space (virtual image data175).

FIG. 20is an example diagram illustrating overlaying of a user image on a virtual image according to embodiments herein.

As previously discussed, the mobile device120can include any number of image sensor devices such as a first image sensor device and a second image sensor device. The first image sensor device (on a first facing of the mobile device120) can be configured to detect images of landscape200as previously discussed. Also, in a manner as previously discussed, the display management resource140creates virtual images for display on display screen130depending on a location and orientation of the mobile device120and corresponding image sensor devices. In one embodiment, a second image sensor device (such as an image sensor device disposed on a second facing of the mobile device120) captures an image of the user108. Thus, the second image sensor device captures real-time images of objects opposite the landscape205. Display management resource140overlays a rendition of the user2108onto the generated virtual images to make it appear as though the user108is present in the virtual image (as derived from virtual three-dimensional space) displayed on the display screen130.

Note that any suitable technique can be used to extract display elements representing an object (such as user108) from the captured image and overlay the image on the display screen130. The display management resource140can be further configured to store the composite image (visual indication and real-time rendition of the user108) on display screen130for subsequent transmission to other users (such as friends of the user108).

Accordingly, in one embodiment, the display management resource140receives an image of the user108operating the mobile device120. The display management resource140overlays the image of the user108on the rendition of one or more virtual images displayed on the display screen130for viewing.

FIG. 17is an example block diagram of a computer device for implementing any of the operations as discussed herein.

As shown, computer system850of the present example includes an interconnect811, a processor813(such as one or more processor devices, computer processor hardware, etc.), computer readable storage medium812(such as hardware storage to store data), I/O interface814, and communications interface817.

I/O interface814provides connectivity to a repository880and, if present, other devices such as a playback device, display screen, input resource102, a computer mouse, etc.

Computer readable storage medium812(such as a non-transitory hardware medium) can be any hardware storage resource or device such as memory, optical storage, hard drive, rotating disk, etc. In one embodiment, the computer readable storage medium812stores instructions executed by processor813.

Communications interface817enables the computer system850and processor813to communicate over a resource such as network190to retrieve information from remote sources and communicate with other computers. I/O interface814enables processor813to retrieve stored information such as from repository180.

As shown, computer readable storage media812is encoded with display manager application140-1(e.g., software, firmware, etc.) executed by processor813. Display manager application140-1can be configured to include instructions to implement any of the operations as discussed herein.

During operation of one embodiment, processor813(e.g., computer processor hardware) accesses computer readable storage media812via the use of interconnect811in order to launch, run, execute, interpret or otherwise perform the instructions in display manager application140-1stored on computer readable storage medium812.

Execution of the display manager application140-1produces processing functionality such as display manager in processor813. In other words, the display manager process140-2associated with processor813represents one or more aspects of executing display manager application140-1within or upon the processor813in the computer system850.

In accordance with different embodiments, note that computer system may be any of various types of devices, including, but not limited to, a wireless access point, a mobile computer, a personal computer system, a wireless device, base station, phone device, desktop computer, laptop, notebook, netbook computer, mainframe computer system, handheld computer, workstation, network computer, application server, storage device, a consumer electronics device such as a camera, camcorder, set top box, mobile device, video game console, handheld video game device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device. The computer system850may reside at any location or can be included in any suitable resource in network environment100to implement functionality as discussed herein.

Functionality supported by the different resources will now be discussed via flowcharts inFIGS. 18 and 19. Note that the steps in the flowcharts below can be executed in any suitable order.

FIG. 18is a flowchart1800illustrating an example method according to embodiments herein. Note that there will be some overlap with respect to concepts as discussed above.

In processing block1810, the display manager application140receives location information from location detection resource195. The location information indicates a current location of the mobile device120-1in a geographical region.

In processing block1820, the display manager application140receives input105from a user108-1operating the mobile device120-1.

In processing block1830, the display manager application140maps the input105to virtual images associated with an historical event that occurred at the location.

In processing block1840, the display manager application140initiates display of a rendition of the historical event from different perspectives depending on an orientation of a display screen of the mobile device at the current location.

FIG. 19is a flowchart1900illustrating an example method according to embodiments herein. Note that there will be some overlap with respect to concepts as discussed above.

In processing block1910, the display manager application140receives location information from location detection resource195indicating a location of the mobile device in a geographical region.

In processing block1920, the display manager application140receives a vector indicating an orientation of a display screen130of the mobile device120-1at the location.

In processing block1930, the display manager application140utilizes the vector to define a viewing window through a first image layer and a second image layer. The viewing window defines a region in the first image layer to be superimposed onto a region in the second image layer.

In processing block1940, the display manager application140utilizes the region in the first image layer and the region in the second image layer to produce a viewable rendition for display on the display screen130of the mobile device120.

Note again that techniques herein are well suited for use in providing realistic viewing of virtual images (such as recreation of historical events) in a respective geographical region depending on location and orientation of a respective display screen of the mobile device. However, it should be noted that embodiments herein are not limited to use in such applications and that the techniques discussed herein are well suited for other applications as well.