Previewing a graphic in an environment

A method includes defining a surface within a first captured image of an environment. The defined surface is identified in a second captured image of the environment. A graphic is overlaid on the surface identified in the second captured image. The second captured image is caused to be displayed to preview the graphic in the environment.

BACKGROUND

Products such as wallpaper can be custom designed for a specific wall in a room. Often, one may fail to account for door or window placement on a wall and how those objects may interfere with the pattern of the wallpaper. Like most custom products, once ordered, manufactured, and shipped, custom wall-paper often cannot be reused if it is later determined that mistakes were made in the design or if it simply proves not to be aesthetically pleasing.

DETAILED DESCRIPTION

Introduction

Various embodiments described below were developed in an effort to allow a customer to preview a graphic on a surface in an environment. The term graphic, as used herein, is used to mean a virtual representation of a physical object. A graphic may be a digital image of that object. Wall paper is just one example of such an object. The term environment is used to refer to a location that can be captured in an. The term surface, as used herein, is used to refer to any generally to an area of an image. That area may, but need not, represent an actual surface such as a wall or floor in the environment depicted in the image. In a particular example, that graphic to be previewed may be a representation of a wall paper design. A preview is accomplished by overlaying the graphic on a pre-defined surface within an image or a series of images that make up a video.

The following description is broken into sections. The first, labeled “Environment,” describes an exemplary environment in which various embodiments may be implemented. The second section, labeled “Components,” describes examples of various physical and logical components for implementing various embodiments. The third section, labeled as “Operation,” describes steps taken to implement various embodiments.

FIG. 1Adepicts an environment10in which various embodiments may be implemented. Environment10is shown to include device12and graphic service14. While environment10is shown to include one device12and one graphic service14, environment10may include any number of such components.

Device10represents generally any computing device capable of capturing images, modifying those images, and displaying the modified images for viewing by a user. Examples include, but are not limited to, smart phones and tablets. Graphic service14represents generally any network service configured to supply a graphic to device12. Graphic service14may also be configured to enable a user of device12to customize the graphic by, for example, adding text and images. Thus, a graphic, may take the form of a digital image.

Components12and14are interconnected via link16. Link16represents generally one or more of a cable, wireless, fiber optic, or remote connections via a telecommunication link, an infrared link, a radio frequency link, or any other connectors or systems that provide electronic communication. Link16may include, at least in part, an intranet, the Internet, or a combination of both. Link16may also include intermediate proxies, routers, switches, load balancers, and the like. The paths followed by link16between components12and14as depicted inFIG. 1Arepresent the logical communication paths between these devices, not necessarily the physical paths between the devices.

In the example ofFIG. 1A, device12is shown to include screen18. Here, device12is causing screen18to display a captured image of an environment18. As depicted that environment is a room with a wall22. InFIG. 1B, three additional captured images of the same wall22in the environment20are shown at different points in time and from different angles. Graphic24is overlaid on wall22in each captured image inFIG. 1Bproviding a preview of graphic24in environment20. The images displayed by device12inFIG. 1B, may be frames from a video in which a user is previewing graphic24in real-time.

FIG. 2depicts examples of physical and logical components for implementing various embodiments. InFIG. 2, device12is shown as a system that includes screen18, camera26, capture engine28, surface engine30, graphic engine31, display engine32, zone engine33, and action engine34. While shown as being integrated into one device, components18, and26-34may be distributed across two or more devices. For example, camera26may be a peripheral coupled to device12.

Screen18represents generally any screen that can be caused to display an image for viewing by a user. Camera26represents generally any camera than can be caused to capture an image that can then be displayed on screen18. Camera26may be used to capture both still images and motion video. Capture engine28represents generally any combination of hardware and programming configured to cause camera26to capture still images and motion video. In other words, capture engine28is responsible for placing device12in a mode where it captures still images and in a mode in which it captures video—both of which are ultimately caused to be displayed by screen18. The mode in which capture engine28places device12, as discussed below can be guided by surface engine30.

Surface engine30represents generally any combination of hardware and programming configured to define a surface of an image caused to be captured by capture engine28. Surface engine30may do so automatically or manually. A manual process can involve a user selecting the corners of a surface in the captured image as displayed by screen18. Where screen18is a touch-screen, this can involve the user touching the four corners of the surface such as a wall, floor, table, or other object to be defined. Alternatively another input device could be used to select the corers. Surface engine30can then use the coordinates of the corners selected within the image to define the surface. Surface engine30may implement an automatic process by detecting lines in a captured image using a Sobel or similar filter. Intersections can then be calculated. The detected lines are screened to identify those that belong to a surface such as a wall based on line length and known wall geometry.

Initially, surface engine30guides capture engine30to place device in a still image capture mode. Then, at the guidance of a user for example, a still image is captured and caused to be displayed by screen18, and surface engine30defines a surface. With the surface defined, surface engine30guides capture engine to place device12in a video mode in which camera26continually captures a series of sequence of images at a relatively high rate while screen18is caused to display the resulting video from those images in real time. Surface engine30is responsible for identifying or otherwise locating the defined surface within the series of images captured to generate the video. Surface engine30may accomplish this task, for example, by implementing a feature detection algorithm such as Scale Invariant Feature Transform (SIFT).

Graphic engine31represents generally any combination of hardware and programming configured to overlay a graphic on the image identified in each of the series of images. In doing so, graphic engine31may acquire the graphic from graphic service14. In each given image of the series, graphic engine31identifies a geometry of that image. As device12moves within the environment, the vantage point from which an image is captured changes as does the geometry of the surface. Examples of the changing geometry of a surface are discussed below in connection withFIG. 6. Graphic engine31modifies the graphic to match the identified geometry of the surface for a given image and then overlays the modified graphic on the surface identified in that image. Graphic engine31may also adjust an opacity of the graphic so that features on the identified surface such as doors and windows on a wall bleed though the graphic when the images of the series are displayed as the video.

Display engine32represents generally any combination of hardware and programming configured to capable of causing screen18to display still images and motion videos captured b camera16at the direction of capture engine28. Where screen18is a touch screen, display engine32may also be responsible for overlaying controls on the images caused to e displayed. Such controls may be for causing device12to capture an image and to direct the adjustment of the opacity of a graphic. Examples of such controls can be seen inFIG. 6.

Zone engine33represents generally any combination of hardware and programming configured to detect a user's interaction with a zone in an image being displayed by screen18. In particular, that interaction may be with a predefined zone of a graphic overlaying the surface identified in the displayed image. A pre-defined zone is a closed area of a graphic. The zone is defined by discernable coordinates within that graphic. A graphic may include an element. A pre-defined zone may be an area of the graphic bounding that element or a portion thereof. Examples of predefined zones are discussed below in connection withFIG. 7where the elements bounded by the zones are depicted as animals.

Zone engine33, may detect user interaction by determining that at least a portion of a zone falls within a predetermined position within a captured image. In one example, that position may be the center of the image. When a user positions device12such that the zone within a graphic overlay is positioned, for a period of time, in the center of images captured by camera26, the user can be presumed to be interacting with that zone. As noted above, a graphic is a digital image. The predefined zones of a graphic can be identified by metadata included in that digital image. The metadata, may, for example, be coordinates defining the zones. Zone engine33may perform this function by determining that the coordinates of the zone within the graphic lie at the center of the captured image on which the graphic is overlaid. In another example, the metadata may be data identifying an element bounded by a zone. Zone engine33may perform its function by examining the captured image and recognizing that the element of the graphic bounded by the zone is positioned at the center of the captured image.

User interaction can take other forms. For example, where screen18is a touch screen, user interaction can include touching screen18. Where the user's touches the zone within the displayed image, the user can be presumed to be interacting with the zone.

Action engine34represents generally any combination of hardware and programming configured to trigger an action associated with a predefined zone. Action engine34does so once zone engine33detects user interaction with the zone. Again, it is noted that the graphic can take the form of a digital image having metadata. The metadata defines a zone within the graphic as well as an action associated with the zone. An associated action can, for example, be any action that can be performed by device12in response to being triggered by action engine34. In the example ofFIG. 7, an action associated with a zone that bounds a lion in a graphic can include playing an audio clip of a lion's roar. In another example where device12is a smart phone, an action may include dialing a contact, sending a message, or accessing a web page. Further, an action may include executing a specified application or opening a file within an application.

It was noted above that a graphic is a virtual representation of a physical object. Such an object may be wall paper having interactive zones. Device12may capture images of a wall on which the wall paper has been installed. Zone engine33can detect user interaction with a zone in the portion of a captured image representing the wall paper. In an example, zone engine33examines a captured image and identifies a graphic representation of the wall paper. That graphic can be a digital image that includes metadata defining the interactive zones. The metadata may define relative coordinates of a zone within the graphic that map to a relative area of the wallpaper. Zone engine33detects interaction with the zone when that area of the wall paper appears within a predetermined location within a captured image. Instead, the metadata may identify an element that is bounded by the zone. Upon detecting that element in the predetermined location of the captured image, zone engine33detects or otherwise presumes user interaction with the corresponding zone. Thus, user interaction with a zone of a graphic can be detected both with respect to a captured image that includes the graphic as an overlay and a captured image of the actual wallpaper represented by the graphic.

Graphic service14, inFIG. 2, is shown to include job engine36, graphic store38, job store40, and application store42. Job engine36represents generally any combination of hardware and programming configured to communicate a graphic to device12. In doing so, job engine36may present device12with a selection of graphics from which to choose, obtaining those graphics from graphic store38. Graphic service14may save a user's selections in job store40. Device12may not initially be configured with engines28-34. In such a case, job engine42may communicate an installation package from application store42to device12—the installation package containing a software representation of engines28-34. Device12then executes the installation package to enable the functionality discussed above.

In foregoing discussion, various components were described as combinations of hardware and programming. Such components may be implemented in a number of fashions. One example is depicted inFIG. 3where, in addition to screen18and camera26, device12is shown to include interface44, processor46and memory48. Interface44represents hardware that device12can use to communicate data to and from graphic service14via link16. Such communications may, for example, employ a wireless protocol.

Processor46represents generally any device for executing program instructions stored in memory48. Memory48represents generally any memory configured to store data and program instructions (programming) that, when executed, cause processor48to implement the functionality of engines28-34ofFIG. 2. Thus, the hardware portions of engines28-34may be implemented though processor46. The programming elements may be instructions stored in memory48.

Graphic service14, inFIG. 3, is shown to include interface50, processor52, and memory54. Interface50represents hardware that graphic service14can use to communicate data to and from device12via link16. Processor52represents generally any device for executing program instructions stored in memory54. Memory54represents generally any memory configured to store data and program instructions (programming) that, when executed, cause processor52to implement the functionality of job engine36ofFIG. 2. Thus, the hardware portion of job engine36may be implemented though processor52. The programming elements may be instructions stored in memory54.

Memory54may further function as graphic store,38, job store40, and application store42. As previously discussed, application store42may maintain an installation package for an application or applications that when installed on device12and executed by processor46enables device12to function as a system that includes engines28-34ofFIG. 2.

FIGS. 4-5are exemplary flow diagrams of steps taken to implement various embodiments in which a graphic is overlaid on a surface. In discussingFIGS. 4-5, reference may be made to the elements ofFIGS. 1-3to provide contextual examples. Implementation, however, is not limited to those examples. Additional reference will also be made toFIGS. 6-7which depict, as examples, sequences of screen views (A-G) in which a user selects a graphic and previews that graphic overlaying a wall and (K-L) in which a user is interacting with displayed image to trigger an action.

Starting withFIG. 4, a surface is defined within a first captured image of an environment (step56). As noted above, the wall may be defined automatically or manually. In a manual approach, the coordinates within the image may be identified and used to define the surface. Looking at screen view C inFIG. 6, a user is manually selecting the corners of a wall, the coordinates of which can be used to define the wall. With surface defined in step56, that surface is identified in a second captured image of the same environment (step58). That second image may be one of a series of images defining a video. In which case, step58can involve identifying the surface in each image of that series. Referring toFIG. 2, surface engine30may be responsible for implementing steps56and58.

A graphic is overlaid on the surface identified in the second captured image (step60). Screen view D ofFIG. 6provides an example. As noted above, the second captured image may be one of a series of images defining a video. In this case, step60can include overlaying the graphic in each image of the series. Step60can involve identifying geometry of the identified surface and then modifying the graphic to match that geometry. Screen views H and G ofFIG. 6provide examples. Step60can also involve adjusting opacity of the graphic and overlaying the graphic such that at least a portion of the identified wall bleeds through the graphic and is visible when the second captured image is displayed. Referring toFIG. 2, graphic engine31may be responsible for implementing step60.

The second captured image with the graphic overlaying the identified surface is caused to be displayed (step62). This allows the graphic to be previewed in the environment. Screen views F, G, and H ofFIG. 6provide examples. Again, the second captured image may be one of a series of images defining a video. Thus step62can involve causing a display of the series of images where each image includes the graphic overlaying the surface identified in that image. In this fashion, the graphic can be previewed in real time or near real time. Referring toFIG. 2, display engine34may be responsible for implementing step62.

The graphic may include a predefined zone associated with an action. User interaction with a zone in the displayed second captured image is detected (step63). That zone corresponds to the predefined zone in the graphic. In response to the detection, the associated action is triggered. Referring back toFIG. 2, zone engine33and action engine34may be responsible for implementing step63. Looking atFIG. 7, a series of screen views (J-L) are depicted in which an action is triggered in response to detecting user interaction with a zone. In screen view J, the captured image being displayed by device12includes wall78of environment20. A graphic80has been overlaid on wall78in the captured image. Alternatively, wallpaper represented by the graphic80has been installed on wall78. Graphic80includes pre-defined zones82shown in broken lined in screen view K. A user has selected an interactive preview mode causing cross-hairs84to appear in screen view L. The user has positioned device12so that crosshairs84are centered on a zone in the captured image that corresponds to a predefined zone82of the graphic. This user interaction results in an action being triggered—that action being the playing of an audio clip of a lion's roar.

Moving toFIG. 5, a device is caused to capture and display an image of en environment (step64). Capture engine28ofFIG. 2may implement step64. Referring toFIG. 6, screen view B depicts a user interacting with device12. The interaction results in capture engine28causing device12to capture an image of environment20. The corners of a wall are identified in the captured image (step66). Referring to screen view C ofFIG. 6, a user is interacting with device12. Based on the interactions, surface engine30identifies the coordinates of the corners of wall22. The wall is defined in the captured image using the coordinates of the corners identified in step66(step68). Surface engine30ofFIG. 2uses the coordinates of the identified corners to define the wall.

The device is then caused to capture a series of images of the environment (step70). The series of images define a video that can be displayed in real time or near real time by the device. Once surface engine30is able to define the wall, capture engine28ofFIG. 2may automatically place device in a video mode in which it captures the series of images. The wall defined in step68is identified in each image of the series captured in step70(step72). A graphic is overlaid on the wall identified in each image of the series (step74). The particular graphic may be one selected by a user of device. Screen view A ofFIG. 6provides an example of a user interacting with device12to select graphic24.

The device is caused to sequentially display the series of images each including the graphic overlay to allow the graphic to be previewed in the environment (step76). Screen views F, G, and H ofFIG. 6depict three different screen views of a series being displayed. Where the display of each image of the series occurs substantially simultaneously with the capture of that image, the graphic can be previewed in real time or near real time. Referring toFIG. 6, one can see that the camera has moved between the capture of each image causing geometry of wall22to differ. The graphic24has been adjusted to match the geometry of wall22in each image. One can also see that the opacity of graphic24has been adjusted between screen views E and F to allow features of wall22to bleed though.

CONCLUSION

FIGS. 1-3aid in depicting the architecture, functionality, and operation of various embodiments. In particular,FIGS. 2-3depict various physical and logical components. Various components illustrated inFIGS. 2and3are defined at least in part as programs or programming. Each such component, portion thereof, or various combinations thereof may represent in whole or in part a module, segment, or portion of code that comprises one or more executable instructions to implement any specified logical function(s). Each component or various combinations thereof may represent a circuit or a number of interconnected circuits to implement the specified logical function(s).

Also, the present invention can be embodied in any computer-readable media for use by or in connection with an instruction execution system such as a computer/processor based system or an ASIC (Application Specific Integrated Circuit) or other system that can fetch or obtain the logic from computer-readable media and execute the instructions contained therein. “Computer-readable media” can be any media that can contain, store, or maintain programs and data for use by or in connection with the instruction execution system. Computer readable media can comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, or semiconductor media. More specific examples of suitable computer-readable media include, but are not limited to, a portable magnetic computer diskette such as floppy diskettes or hard drives, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, or a portable compact disc.

Although the flow diagrams ofFIGS. 4-5show specific orders of execution, the orders of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be scrambled relative to the order shown. Also, two or more blocks shown in succession may be executed concurrently or with partial concurrence. All such variations are within the scope of the present invention.

The present invention has been shown and described with reference to the foregoing exemplary embodiments. It is to be understood, however, that other forms, details and embodiments may be made without departing from the spirit and scope of the invention that is defined in the following claims.